@sun-asterisk/sungen 2.6.14 → 2.7.0-beta.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/cli/index.js +3 -1
- package/dist/cli/index.js.map +1 -1
- package/dist/exporters/feature-parser.d.ts +9 -2
- package/dist/exporters/feature-parser.d.ts.map +1 -1
- package/dist/exporters/feature-parser.js +12 -4
- package/dist/exporters/feature-parser.js.map +1 -1
- package/dist/orchestrator/ai-rules-updater.d.ts.map +1 -1
- package/dist/orchestrator/ai-rules-updater.js +10 -0
- package/dist/orchestrator/ai-rules-updater.js.map +1 -1
- package/dist/orchestrator/templates/ai-instructions/claude-cmd-review.md +13 -12
- package/dist/orchestrator/templates/ai-instructions/claude-config.md +1 -1
- package/dist/orchestrator/templates/ai-instructions/claude-skill-delivery.md +1 -1
- package/dist/orchestrator/templates/ai-instructions/claude-skill-gherkin-syntax.md +14 -0
- package/dist/orchestrator/templates/ai-instructions/claude-skill-tc-generation.md +371 -324
- package/dist/orchestrator/templates/ai-instructions/claude-skill-tc-review.md +268 -90
- package/dist/orchestrator/templates/ai-instructions/claude-skill-test-design-techniques.md +23 -49
- package/dist/orchestrator/templates/ai-instructions/claude-skill-viewpoint-group-a-data-entry.md +203 -0
- package/dist/orchestrator/templates/ai-instructions/claude-skill-viewpoint-group-b-data-ops.md +179 -0
- package/dist/orchestrator/templates/ai-instructions/claude-skill-viewpoint-group-c-data-explore.md +233 -0
- package/dist/orchestrator/templates/ai-instructions/claude-skill-viewpoint-group-d-display.md +226 -0
- package/dist/orchestrator/templates/ai-instructions/claude-skill-viewpoint-group-e-identity.md +177 -0
- package/dist/orchestrator/templates/ai-instructions/claude-skill-viewpoint.md +69 -240
- package/dist/orchestrator/templates/ai-instructions/copilot-cmd-review.md +13 -12
- package/dist/orchestrator/templates/ai-instructions/copilot-config.md +1 -1
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-delivery.md +1 -1
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-gherkin-syntax.md +15 -21
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-tc-generation.md +371 -324
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-tc-review.md +262 -102
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-test-design-techniques.md +23 -49
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-viewpoint-group-a-data-entry.md +203 -0
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-viewpoint-group-b-data-ops.md +179 -0
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-viewpoint-group-c-data-explore.md +233 -0
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-viewpoint-group-d-display.md +226 -0
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-viewpoint-group-e-identity.md +177 -0
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-viewpoint.md +69 -240
- package/package.json +9 -1
- package/src/cli/index.ts +4 -1
- package/src/exporters/feature-parser.ts +12 -4
- package/src/orchestrator/ai-rules-updater.ts +10 -0
- package/src/orchestrator/templates/ai-instructions/claude-cmd-review.md +13 -12
- package/src/orchestrator/templates/ai-instructions/claude-config.md +1 -1
- package/src/orchestrator/templates/ai-instructions/claude-skill-delivery.md +1 -1
- package/src/orchestrator/templates/ai-instructions/claude-skill-gherkin-syntax.md +14 -0
- package/src/orchestrator/templates/ai-instructions/claude-skill-tc-generation.md +371 -324
- package/src/orchestrator/templates/ai-instructions/claude-skill-tc-review.md +268 -90
- package/src/orchestrator/templates/ai-instructions/claude-skill-test-design-techniques.md +23 -49
- package/src/orchestrator/templates/ai-instructions/claude-skill-viewpoint-group-a-data-entry.md +203 -0
- package/src/orchestrator/templates/ai-instructions/claude-skill-viewpoint-group-b-data-ops.md +179 -0
- package/src/orchestrator/templates/ai-instructions/claude-skill-viewpoint-group-c-data-explore.md +233 -0
- package/src/orchestrator/templates/ai-instructions/claude-skill-viewpoint-group-d-display.md +226 -0
- package/src/orchestrator/templates/ai-instructions/claude-skill-viewpoint-group-e-identity.md +177 -0
- package/src/orchestrator/templates/ai-instructions/claude-skill-viewpoint.md +69 -240
- package/src/orchestrator/templates/ai-instructions/copilot-cmd-review.md +13 -12
- package/src/orchestrator/templates/ai-instructions/copilot-config.md +1 -1
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-delivery.md +1 -1
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-gherkin-syntax.md +15 -21
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-tc-generation.md +371 -324
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-tc-review.md +262 -102
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-test-design-techniques.md +23 -49
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-viewpoint-group-a-data-entry.md +203 -0
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-viewpoint-group-b-data-ops.md +179 -0
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-viewpoint-group-c-data-explore.md +233 -0
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-viewpoint-group-d-display.md +226 -0
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-viewpoint-group-e-identity.md +177 -0
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-viewpoint.md +69 -240
|
@@ -1,141 +1,319 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: sungen-tc-review
|
|
3
|
-
description: 'Test case review —
|
|
3
|
+
description: 'Test case review — 7-dimension rubric (100 pts), Syntax gate + Coverage matrix. Auto-loaded by review command.'
|
|
4
4
|
user-invocable: false
|
|
5
5
|
---
|
|
6
6
|
|
|
7
|
-
##
|
|
7
|
+
## How to use this skill
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
Review one `.feature` (plus `selectors.yaml` + `test-data.yaml` if present) in 3 steps:
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
1. **Syntax Gate (Layer A)** — check Sungen syntax. On a hard-fail, return to the author; do NOT proceed to detailed scoring.
|
|
12
|
+
2. **Coverage Matrix (Layer B)** — map against Sun Common Checklist viewpoints; list the gaps.
|
|
13
|
+
3. **Scoring (Layer C)** — score the 7-dimension/100-pt rubric. Layers A and B are *how* to evaluate each dimension, not separate bonus points.
|
|
14
|
+
|
|
15
|
+
> Scope: Sungen tests run through Playwright — **UI-only**. No DB/API/email-body/server-side. Out-of-scope viewpoints (see Out-of-scope) are NOT penalized; if required, mandate `@manual` + a technical reason.
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## Scoring (7 dimensions, 100 points)
|
|
20
|
+
|
|
21
|
+
**≥ 70**: PASS | **50–69**: CONDITIONAL (fix before execution) | **< 50**: FAIL (revise & re-review)
|
|
22
|
+
|
|
23
|
+
**Scoring convention**: Each row in a dimension table is scored independently. Full points if satisfied across all in-scope TCs; otherwise deduct ~1 pt per distinct violation, floor 0 per row (never negative). A row whose technique has no relevant case → 0 for that row only, it does not zero the whole dimension. Sum the rows for the dimension total.
|
|
24
|
+
|
|
25
|
+
| Dimension | Pts |
|
|
26
|
+
|---|---|
|
|
27
|
+
| Structure & Format | 15 |
|
|
28
|
+
| Coverage (6 sub-dimensions) | 30 |
|
|
29
|
+
| Assertion Quality | 20 |
|
|
30
|
+
| Test Data | 10 |
|
|
31
|
+
| Security & Permission | 10 |
|
|
32
|
+
| Automation Readiness | 10 |
|
|
33
|
+
| Maintainability | 5 |
|
|
34
|
+
| **Total** | **100** |
|
|
35
|
+
|
|
36
|
+
---
|
|
37
|
+
|
|
38
|
+
## Layer A — Syntax Gate (against `sungen-gherkin-syntax`)
|
|
39
|
+
|
|
40
|
+
Hard-fail gate. Any ✗ is a syntax error — must be fixed; do not score further while errors remain.
|
|
41
|
+
|
|
42
|
+
### A1. Structure & Keyword
|
|
43
|
+
|
|
44
|
+
- [ ] Correct shape: `User <Action> [Target] <Type> [in [Parent] pType] [with {{v}}] [is State]`
|
|
45
|
+
- [ ] Actor is always `User`, active voice
|
|
46
|
+
- [ ] `Given` → only `is on` · `When` → action (click/fill/select/press/clear/check/uncheck/hover/wait) · `Then` → only `see` · `And` inherits the preceding keyword
|
|
47
|
+
- [ ] NO `Given User see…` (→ `Then`, or entry assertion `Given User is on [X] page`)
|
|
48
|
+
- [ ] NO `When … And User is on [X] dialog` (→ `And User see` or a separate `Given`)
|
|
49
|
+
- [ ] NO literal-URL navigation (`navigate to "/x"`) → `User is on [X] page` + declare in `selectors.yaml`
|
|
50
|
+
|
|
51
|
+
### A2. Action → Element type
|
|
52
|
+
|
|
53
|
+
- [ ] checkbox/toggle → `check`/`uncheck` (NOT `click`)
|
|
54
|
+
- [ ] dropdown → `select … with {{v}}` · field/textarea/search → `fill … with {{v}}` · button/tab/link/icon → `click`
|
|
55
|
+
- [ ] radio: select another option via `check [Other] radio`, NOT `uncheck` a radio
|
|
56
|
+
- [ ] `press` only for keys (`press Enter key`), NOT `press [Submit] button`
|
|
57
|
+
- [ ] Element type is in the valid type table (Context/Input/Trigger/Data/Feedback/System)
|
|
58
|
+
- [ ] `wait for` only for Spinner/Modal, minimize usage
|
|
59
|
+
|
|
60
|
+
### A3. Value — State — click-rule
|
|
61
|
+
|
|
62
|
+
- [ ] Dynamic data: `with {{snake_case}}`, NO hardcoding (`with {{admin@mail.com}}` → ✗ `with {{invalid_email}}`)
|
|
63
|
+
- [ ] State: `is <state>` (hidden/visible/disabled/checked/empty/loading…), NOT `{{disabled}}`, NOT missing `is`
|
|
64
|
+
- [ ] **click + value**: static (`button/link/icon/tab`) takes NO value; dynamic (`row/item/card/option`) takes `with {{v}}`
|
|
65
|
+
- [ ] `fill` always has a target type (`fill [email] field with {{v}}`, not `fill [email] with {{v}}`)
|
|
66
|
+
|
|
67
|
+
### A4. Assertion (8 patterns → determines the Playwright assertion)
|
|
68
|
+
|
|
69
|
+
- [ ] Visibility `see [T] type` → NO redundant `is visible`; hidden uses `is hidden`
|
|
70
|
+
- [ ] Correct pattern: text(`message/header/label with {{v}}`) · partial(`text contains {{v}}`) · input(`field/dropdown with {{v}}`) · state(`is …`) · attribute(`image/link with {{v}}`) · count(`row with {{count}}`) · page(`[T] page`)
|
|
71
|
+
- [ ] Table: `[Col] column in [Table] table`, `[Ref] row in [Table] table with {{v}}`, `table with {{count}}`/`is empty`, `table match data:`; row scope used correctly
|
|
72
|
+
|
|
73
|
+
### A5. Alert / Scope / Background
|
|
74
|
+
|
|
75
|
+
- [ ] Alert step appears **before** the action that triggers the dialog
|
|
76
|
+
- [ ] Scope-dependent flow (dialog/frame) → use `@steps`/`@extend`, do NOT cram into `Background`
|
|
77
|
+
- [ ] `@extend`: entry assertion must be `Given User is on [X] type`, NOT `Given User see`; name `@steps:module__action` (snake/kebab)
|
|
78
|
+
- [ ] Scenario name matches the step's element type (a scenario that says "modal" must use `[X] dialog/modal` consistently)
|
|
79
|
+
|
|
80
|
+
### A6. YAML & Variables
|
|
81
|
+
|
|
82
|
+
- [ ] Selector keys: lowercase, keep Unicode, use **spaces** (no underscores/dots); same label → `--type`/`--N` suffix; names >30 chars → shorten to 1–3 words
|
|
83
|
+
- [ ] Types requiring explicit YAML: `date-picker`, `uploader`, `overlay`, `frame`, `step`
|
|
84
|
+
- [ ] Every `{{var}}` in `.feature` exists in `test-data.yaml`; NO orphan keys
|
|
85
|
+
- [ ] Dynamic vars (`{{$timestamp}}`/`{{$uuid}}`/`{{$random:a:b}}`) used correctly for run-unique data (CRUD)
|
|
86
|
+
- [ ] `locator` (CSS) only as a last resort
|
|
87
|
+
|
|
88
|
+
### A7. Tags & Flow
|
|
89
|
+
|
|
90
|
+
- [ ] Exactly **one** priority tag per scenario (`@high`/`@normal`/`@low`…)
|
|
91
|
+
- [ ] `@manual`/`@auth:role`/`@no-auth`/`@cleanup:*`/`@parallel`/`@flow` used in the right context
|
|
92
|
+
- [ ] `@parallel` is REQUIRED when a feature mixes auth groups (`@auth:user` + `@no-auth`)
|
|
93
|
+
- [ ] `@flow`: `[Screen:Element]` namespace is consistent; YAML keys quoted with the colon (`"login:submit":`)
|
|
94
|
+
|
|
95
|
+
---
|
|
96
|
+
|
|
97
|
+
## Layer B — Coverage Matrix (against Sun Common Checklist for QA)
|
|
98
|
+
|
|
99
|
+
Build a mapping table: for each applicable group, does the feature have a matching scenario → list the **gaps**. Each group maps to a Sungen VP classification.
|
|
100
|
+
|
|
101
|
+
| Category | Sub-viewpoint to check | VP map | Sungen representation |
|
|
102
|
+
|---|---|---|---|
|
|
103
|
+
| **Check Accessing** | Not logged in → redirect login; logged in with correct role; wrong role → access denied/hide menu | VP-SEC | `@no-auth` (→ login), `@auth:role` (right/wrong role) |
|
|
104
|
+
| **Check Initial** | Default field state; autofocus; tab order; display per spec | VP-UI | `Then User see [T] field is empty/focused`, `see [T] page` |
|
|
105
|
+
| **Check Required** | Submit with one / multiple required fields blank | VP-VAL | empty `fill` → `see [Err] message with {{v}}` |
|
|
106
|
+
| **Check Format** | Email/phone… invalid format; leading/trailing space; uppercase | VP-VAL | `with {{invalid_email}}` → error |
|
|
107
|
+
| **Check Maxlength** | Exceed maxlength per field | VP-VAL | `with {{over_max_name}}` → error/blocked input |
|
|
108
|
+
| **Check Exist/Duplicate** | Duplicate email/username | VP-VAL | `with {{existing_email}}` → "already exists" |
|
|
109
|
+
| **Check Valid** | Happy path: all valid fields; required-only | VP-UI/LOGIC | real data → success message + post-action state |
|
|
110
|
+
| **Check Expire** | OTP/token/link expired | VP-LOGIC | usually `@manual` (time-dependent) |
|
|
111
|
+
| **Behavior Handling** | Rapid double-click; slow network; reload keeps/loses data; copy-paste; Enter in field | VP-LOGIC | `double click`, `wait for [Spinner]`, `press Enter key` |
|
|
112
|
+
| **Account Status** | Deleted / blocked / inactive account login | VP-LOGIC | corresponding account data → error message |
|
|
113
|
+
| **Security** | SQL Injection; XSS; Data Integrity (edit dropdown/remove `disabled` via DevTools → server rejects). SQL on LIKE/search fields → 2 TCs per the `sungen-viewpoint` SQL 2-layer rule. | VP-SEC | `with {{xss_*}}`/`{{sql_*}}`; data-integrity usually `@manual` |
|
|
114
|
+
| **Cross-surface outcomes** | Admin action → outcome on user-facing surface (portal, mobile, widget); spec defines display condition on another URL | VP-LOGIC-CS | `@high @manual` per surface per business rule — at minimum 1 per cross-surface rule; use `@auth:role` + `@extend` if surface reachable in same test run |
|
|
115
|
+
|
|
116
|
+
**Tier-aware**: if the suite only has `@high` (Tier 1) → do NOT penalize missing pure VP-UI (deferred to Tier 2). Require full VP coverage only on a Full review.
|
|
117
|
+
|
|
118
|
+
### EP/BVA rules when mapping
|
|
119
|
+
|
|
120
|
+
- **EP**: keep only **one representative** per invalid class; same-class duplicates → flag as redundant.
|
|
121
|
+
- **BVA**: spec defines min/max → cover `min-1`, `min`, `max`, `max+1` (Maxlength, counts…).
|
|
122
|
+
- Error messages must match the spec **word-for-word**, not generic.
|
|
123
|
+
|
|
124
|
+
---
|
|
125
|
+
|
|
126
|
+
## Layer C — Dimension details
|
|
127
|
+
|
|
128
|
+
### Structure & Format (15)
|
|
12
129
|
|
|
13
130
|
| Check | Pts |
|
|
14
131
|
|---|---|
|
|
15
|
-
|
|
|
16
|
-
|
|
|
17
|
-
|
|
|
18
|
-
|
|
|
19
|
-
|
|
|
132
|
+
| ID format `VP-<CAT>-<NNN>` (e.g. `VP-LOGIC-001`, `VP-VAL-001`); deleted IDs never reused | 3 |
|
|
133
|
+
| Clear title: `[object] + [result/behavior] + [condition]` | 3 |
|
|
134
|
+
| Correct module/screen + exactly **one** priority tag | 2 |
|
|
135
|
+
| `Given` = state (`User is on [T] page`), not an action | 3 |
|
|
136
|
+
| `When`/`And` = atomic steps (1 action), no vague words, last step is the trigger | 2 |
|
|
137
|
+
| `Then` is specific & measurable (`see [T] type with {{v}}`/`is state`), not mere existence | 2 |
|
|
20
138
|
|
|
21
|
-
### Coverage (
|
|
139
|
+
### Coverage (30) — 6 sub-dimensions
|
|
22
140
|
|
|
23
|
-
|
|
|
141
|
+
| Sub | Technique | Pts | What to check |
|
|
24
142
|
|---|---|---|---|
|
|
25
|
-
| Happy paths | — |
|
|
26
|
-
| Negative
|
|
27
|
-
| Edge
|
|
28
|
-
| Boundary
|
|
29
|
-
| State
|
|
30
|
-
| Condition
|
|
143
|
+
| Happy paths | — | 5 | ≥1 happy path per core function; realistic data; full result assertions |
|
|
144
|
+
| Negative | EP | 6 | one representative per invalid class, no duplicates; error matches spec |
|
|
145
|
+
| Edge | EP | 5 | empty/whitespace, special chars (XSS/SQLi), long strings, abnormal file uploads |
|
|
146
|
+
| Boundary | BVA | 6 | `min-1/min/max/max+1` per range |
|
|
147
|
+
| State transition | ST | 4 | all valid transitions + ≥3 blocked transitions (**UI-only**; full points if all spec states covered) |
|
|
148
|
+
| Condition combo | DT | 4 | decision table for ≥2 dependent conditions; test only rules with distinct outcomes |
|
|
31
149
|
|
|
32
|
-
|
|
150
|
+
> Use the quick estimate `(VPs_covered / 6) * 30` ONLY when cases can't be inspected in depth (portfolio scan); never combine the two methods in one score.
|
|
33
151
|
|
|
34
|
-
###
|
|
152
|
+
### Assertion Quality (20)
|
|
35
153
|
|
|
36
154
|
| Check | Pts |
|
|
37
155
|
|---|---|
|
|
38
|
-
|
|
|
39
|
-
|
|
|
40
|
-
| `
|
|
41
|
-
|
|
|
42
|
-
| Assertion quality (see rules below) | 4 |
|
|
156
|
+
| Specific, not generic — "if this assertion PASSES, am I sure the feature works?" must be a confident yes | 6 |
|
|
157
|
+
| Avoids anti-patterns: re-asserting input just typed; exact-match on dynamic content; missing negative assertions | 6 |
|
|
158
|
+
| **Syntax compliance** — steps match `sungen-gherkin-syntax` patterns, correct `[Ref] type with {{v}}`, no bare `is visible` | 4 |
|
|
159
|
+
| Grouping — each case has 2–7 related assertions for the same action | 4 |
|
|
43
160
|
|
|
44
|
-
|
|
161
|
+
### Test Data (10)
|
|
45
162
|
|
|
46
|
-
|
|
163
|
+
| Check | Pts |
|
|
164
|
+
|---|---|
|
|
165
|
+
| No hardcoded env values in `.feature` — use `{{variables}}` | 3 |
|
|
166
|
+
| **Allow test credentials** in `test-data.yaml` if: (a) `.feature` uses `{{var}}` not hardcoded, (b) dedicated test accounts (not prod/personal), (c) env-specific values via `<screen>.<env>.yaml` overrides | 3 |
|
|
167
|
+
| Idempotent — has teardown or unique data (`{{$timestamp}}`/`{{$uuid}}`) so it can run repeatedly | 2 |
|
|
168
|
+
| Realistic data — accented names, real formats, not `abc`/`1` | 2 |
|
|
47
169
|
|
|
48
|
-
|
|
170
|
+
### Security & Permission (10) — UI-level
|
|
49
171
|
|
|
50
|
-
|
|
172
|
+
| Check | Pts |
|
|
173
|
+
|---|---|
|
|
174
|
+
| Authentication — `@no-auth` → redirect login; expired/forged token tests are `@manual` + reason | 3 |
|
|
175
|
+
| Authorization — via `@auth:role`: each role sees the correct elements/pages | 3 |
|
|
176
|
+
| Viewpoint classification — VP-UI/VAL/LOGIC/SEC assigned correctly | 2 |
|
|
177
|
+
| Input security — XSS/SQLi via `{{xss_*}}`/`{{sql_*}}` | 2 |
|
|
51
178
|
|
|
52
|
-
###
|
|
179
|
+
### Automation Readiness (10)
|
|
53
180
|
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
181
|
+
| Check | Pts |
|
|
182
|
+
|---|---|
|
|
183
|
+
| `@manual` has a valid **technical** reason (not "don't know how to automate" — document as `# TODO: automate when...` comment instead) | 3 |
|
|
184
|
+
| Stable selectors — per `sungen-selector-keys`: keys use spaces, standard types, `locator` only as last resort | 3 |
|
|
185
|
+
| Each step names a concrete element by name/label/role (enough to implement without asking) | 2 |
|
|
186
|
+
| Idempotent + no fixed waits (`wait for [T] dialog` instead of `wait N seconds`) | 2 |
|
|
57
187
|
|
|
58
|
-
###
|
|
188
|
+
### Maintainability (5)
|
|
59
189
|
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
190
|
+
| Check | Pts |
|
|
191
|
+
|---|---|
|
|
192
|
+
| Title reflects the **assertion**, not the action (reading the title tells you when it FAILS) | 2 |
|
|
193
|
+
| No duplicate TCs; remove obsolete tests when a feature is removed; no same-class VP/EP redundancy | 1 |
|
|
194
|
+
| Short enough: `Background` ≤2 lines, scenario ≤10 steps, 2–7 assertions | 1 |
|
|
195
|
+
| Comments where needed: `# --- Section:` for VP grouping; `@manual` explains why | 1 |
|
|
65
196
|
|
|
66
|
-
|
|
197
|
+
---
|
|
67
198
|
|
|
68
|
-
|
|
69
|
-
- Real files on disk, network simulation, backend-only state, timing-dependent
|
|
199
|
+
## Out-of-scope (UI-only Sungen)
|
|
70
200
|
|
|
71
|
-
Do NOT
|
|
201
|
+
Do NOT deduct points when a `.feature` lacks the following viewpoints (Playwright UI cannot verify them). If required → `@manual` + reason, or move to another test layer:
|
|
72
202
|
|
|
73
|
-
|
|
203
|
+
- `API Behavior` (direct status/body response)
|
|
204
|
+
- `Email Verification` content (header/from/to/body/link) — only the post-click UI is verifiable
|
|
205
|
+
- Server-side `Data Integrity` (DevTools edits payload → server rejects)
|
|
206
|
+
- `Performance/Load`, `Rate limit`
|
|
207
|
+
- DB/record verification after an action
|
|
74
208
|
|
|
75
209
|
---
|
|
76
210
|
|
|
77
|
-
##
|
|
211
|
+
## Quick Scan (run first, ~2 minutes)
|
|
212
|
+
|
|
213
|
+
10 questions; many "No" → return to author, skip detailed review.
|
|
214
|
+
|
|
215
|
+
1. ID unique + correct format `VP-<CAT>-<NNN>`?
|
|
216
|
+
2. Title understandable immediately, no context needed?
|
|
217
|
+
3. Every step uses the right keyword (`Given`=is on / `When`=action / `Then`=see)?
|
|
218
|
+
4. Every `{{var}}` exists in `test-data.yaml`?
|
|
219
|
+
5. At least one happy path?
|
|
220
|
+
6. Negative cases for invalid classes (EP)?
|
|
221
|
+
7. Boundary covers the 4 points (`min-1/min/max/max+1`)?
|
|
222
|
+
8. `Then` specific, not generic, no redundant `is visible`?
|
|
223
|
+
9. Test data doesn't hardcode env/static values?
|
|
224
|
+
10. Exactly one priority tag + valid functional tags?
|
|
78
225
|
|
|
79
|
-
|
|
80
|
-
2. **Misclassified VP** — UI tests behavior? Move to LOGIC. Logic tests appearance? Move to UI
|
|
81
|
-
3. **Dynamic content** — exact match on counters/timestamps? Use `contains` instead
|
|
82
|
-
4. **Duplicate across sections** — SEC scenario identical to UI? Remove duplicate
|
|
83
|
-
5. **Missing/wrong priority tag** — every non-`@steps` scenario needs exactly one of `@high`/`@normal`/`@low`. SEC/CRUD happy path/auth→`@high`, validation/secondary features→`@normal`, presence/cosmetic→`@low`
|
|
84
|
-
6. **Always-enabled elements** — `is enabled` on never-disabled element? Remove
|
|
85
|
-
7. **Test-data completeness** — every `{{var}}` must exist in test-data.yaml
|
|
86
|
-
8. **Missing BVA boundaries** — spec defines min/max range but scenarios only test midpoint? Add `min-1`, `min`, `max`, `max+1`
|
|
87
|
-
9. **Missing state transitions** — spec defines lifecycle states but only happy path tested? Add blocked transitions
|
|
88
|
-
10. **Uncovered condition combos** — spec has 2+ dependent conditions but only partial combinations tested? Build decision table
|
|
226
|
+
**Interpret**: 9–10 Yes → detailed review | 6–8 Yes → review with caveats | <6 Yes → return for revision.
|
|
89
227
|
|
|
90
228
|
---
|
|
91
229
|
|
|
92
|
-
##
|
|
230
|
+
## Auto-fix (on detection → suggest fix)
|
|
231
|
+
|
|
232
|
+
> **Never renumber existing `VP-<CAT>-<NNN>` IDs while fixing** (e.g. after removing an EP duplicate). IDs are the dashboard's tracking key — renumbering survivors loses their pass/fail history. Leave gaps; only assign the next unused number to genuinely new scenarios.
|
|
233
|
+
|
|
234
|
+
1. **Wrong keyword/action-type** → correct it (`click checkbox` → `check checkbox`; `Given … see` → `Then …`).
|
|
235
|
+
2. **Hardcoded data** → `with {{snake_case}}` + add the key to `test-data.yaml`.
|
|
236
|
+
3. **Redundant `is visible` / state as value** → drop `is visible`; `{{disabled}}` → `is disabled`.
|
|
237
|
+
4. **Missing target type** → add the type (`fill [email] field with {{v}}`).
|
|
238
|
+
5. **click static with value** → remove value; **click dynamic missing value** → add `with {{v}}`.
|
|
239
|
+
6. **Same-class EP duplicates** → keep one representative.
|
|
240
|
+
7. **Generic expected result** → rewrite to assert specific field/element/state.
|
|
241
|
+
8. **Missing BVA** → add `min-1/min/max/max+1`.
|
|
242
|
+
9. **Missing negative path** → map invalid classes, add scenarios.
|
|
243
|
+
10. **All `@high`** → reset by user impact per the `sungen-gherkin-syntax` priority table (auth/CRUD/security/required → `@high`; format/standard-range boundary/search → `@normal`; cosmetic → `@low`).
|
|
244
|
+
11. **`@manual` without reason** → add a technical reason comment explaining why automation is not possible.
|
|
245
|
+
12. **Orphan var in test-data** → delete or reconnect.
|
|
246
|
+
13. **Unnecessary CSS/locator selector** → switch to role+name/label/text; keys use spaces.
|
|
247
|
+
14. **Scope crammed into `Background`** → split into `@steps`/`@extend`.
|
|
248
|
+
15. **Mixed auth groups missing `@parallel`** → add `@parallel`.
|
|
249
|
+
16. **Missing secondary behaviors** — spec defines tiebreaker, fallback rule, or secondary sort but no scenario tests it? Add 1 `@high` TC per rule.
|
|
250
|
+
17. **Missing concurrency scenarios** — spec or test-viewpoint mentions multi-tab, multi-user, or simultaneous actions but no `@manual` scenario exists? Add 1 `@manual` TC per risk (`@normal` by default; `@high` if data integrity at risk).
|
|
93
251
|
|
|
94
|
-
|
|
252
|
+
---
|
|
95
253
|
|
|
96
|
-
|
|
97
|
-
- `[Screen:Element]` format used consistently (not mixing bare `[Element]` refs)
|
|
98
|
-
- YAML keys quoted with colon: `"login:submit":` not `login:submit:`
|
|
99
|
-
- `@flow` tag present at feature level
|
|
254
|
+
## Unverified Selectors (non-scoring metric)
|
|
100
255
|
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
| Auth persistence | (part of Happy paths) | Auth state maintained across transitions |
|
|
106
|
-
| Error recovery mid-flow | (part of Negative cases) | Invalid input at each phase → fix → continue |
|
|
107
|
-
| Cross-screen data | (part of Edge cases) | Data entered on screen A asserted on screen B |
|
|
108
|
-
|
|
109
|
-
### Viewpoint — flow-specific classification
|
|
110
|
-
- **VP-LOGIC** — screen transitions, navigation flow, auth persistence
|
|
111
|
-
- **VP-VAL** — cross-screen data consistency, form data carried across pages
|
|
112
|
-
- **VP-SEC** — auth state across redirects, permission changes mid-flow
|
|
113
|
-
- VP-UI is typically minimal in flows (focus on functionality over layout)
|
|
114
|
-
|
|
115
|
-
### Checklist — flow-specific items
|
|
116
|
-
11. **Missing screen transitions** — flow visits 4 screens but only 2 transitions tested? Add missing
|
|
117
|
-
12. **Orphan scenarios** — scenario doesn't connect to previous/next phase? Flag broken flow
|
|
118
|
-
13. **Selector namespace consistency** — mixing `[Submit]` and `[Login:Submit]` in same flow? Standardize
|
|
256
|
+
If `selectors/<name>.yaml` exists, count lines matching `@needs-live-verify` (provisional selectors from Figma, not yet checked against a live page).
|
|
257
|
+
- Report the count in the review output; **never** deduct points — the suite can PASS with unverified selectors.
|
|
258
|
+
- If the count > 0, add to the summary: `⚠ <N> selectors flagged @needs-live-verify — run run-test against a live URL to verify.`
|
|
259
|
+
- If `selectors.yaml` does not exist yet (not generated) → omit this metric entirely.
|
|
119
260
|
|
|
120
261
|
---
|
|
121
262
|
|
|
122
263
|
## Output Format
|
|
123
264
|
|
|
124
265
|
```markdown
|
|
125
|
-
## Review: <screen>
|
|
126
|
-
|
|
127
|
-
|
|
266
|
+
## Review: <screen / feature>
|
|
267
|
+
**Date**: YYYY-MM-DD | **Total TCs**: <n> | **Tier**: <1/2> | **Unverified selectors**: <n / n/a>
|
|
268
|
+
|
|
269
|
+
| Dimension | Score | Max | Notes |
|
|
270
|
+
|------------------------|-------|-----|-------|
|
|
271
|
+
| Structure & Format | | 15 | |
|
|
272
|
+
| Coverage | | 30 | x/6 sub-dimensions |
|
|
273
|
+
| Assertion Quality | | 20 | |
|
|
274
|
+
| Test Data | | 10 | |
|
|
275
|
+
| Security & Permission | | 10 | |
|
|
276
|
+
| Automation Readiness | | 10 | |
|
|
277
|
+
| Maintainability | | 5 | |
|
|
278
|
+
| **Total** | | 100 | **PASS / CONDITIONAL / FAIL** |
|
|
279
|
+
|
|
280
|
+
### Syntax Gate (Layer A — hard-fail)
|
|
281
|
+
- [ ] Keyword→Action correct
|
|
282
|
+
- [ ] Action→Type correct; click-rule correct
|
|
283
|
+
- [ ] Assertions follow the 8 patterns, no redundant `is visible`
|
|
284
|
+
- [ ] Every {{var}} exists in test-data.yaml; no orphan keys
|
|
285
|
+
- [ ] Selector keys follow sungen-selector-keys; locator only as last resort
|
|
286
|
+
- [ ] Tags valid; @parallel when mixing auth groups
|
|
287
|
+
- Syntax errors found: <n>
|
|
288
|
+
|
|
289
|
+
### Coverage Matrix (Layer B)
|
|
290
|
+
| Category | Has scenario? | Gap |
|
|
128
291
|
|---|---|---|
|
|
129
|
-
|
|
|
130
|
-
|
|
|
131
|
-
|
|
|
132
|
-
|
|
|
292
|
+
| Check Accessing | ✓/✗ | ... |
|
|
293
|
+
| Check Required/Format/Maxlength/Duplicate/Valid | ... | ... |
|
|
294
|
+
| Behavior Handling / Account Status | ... | ... |
|
|
295
|
+
| Security (XSS/SQLi) | ... | ... |
|
|
296
|
+
| Cross-surface outcomes | ... | ... |
|
|
133
297
|
|
|
134
|
-
###
|
|
135
|
-
1. [
|
|
136
|
-
|
|
137
|
-
3. [VIEWPOINT] ...
|
|
298
|
+
### ✗ Must-fix
|
|
299
|
+
1. [DIMENSION] VP-XXX issue description
|
|
300
|
+
→ Fix: ...
|
|
138
301
|
|
|
139
|
-
###
|
|
302
|
+
### ⚠ Suggested
|
|
303
|
+
1. [TEST-DATA] ...
|
|
304
|
+
|
|
305
|
+
### ✓ Strengths
|
|
306
|
+
- ...
|
|
307
|
+
|
|
308
|
+
### Recommendations (if CONDITIONAL/FAIL)
|
|
140
309
|
- ...
|
|
141
310
|
```
|
|
311
|
+
|
|
312
|
+
---
|
|
313
|
+
|
|
314
|
+
## Scope & Maintenance
|
|
315
|
+
|
|
316
|
+
- **When NOT to apply**: exploratory charters, performance/load scripts, pure NFR specs — this rubric grades functional test cases.
|
|
317
|
+
- **Adapting weights**: a project may rebalance the 7 dimensions (e.g. security-critical service: Security→15, Maintainability→0) as long as the total stays 100 and the PASS/CONDITIONAL/FAIL thresholds are restated. Record any change in this section.
|
|
318
|
+
- **References**: detailed syntax → `sungen-gherkin-syntax`; selector keys → `sungen-selector-keys`; VP classification → `sungen-viewpoint`; generation workflow → `sungen-tc-generation`.
|
|
319
|
+
- **Owner / version**: Owner: `<QA Lead>` · Version: `2.0` · Last updated: `2026-06-03`.
|
|
@@ -6,12 +6,14 @@ user-invocable: false
|
|
|
6
6
|
|
|
7
7
|
## When to Apply
|
|
8
8
|
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
|
12
|
-
|
|
13
|
-
|
|
|
14
|
-
|
|
|
9
|
+
Apply selectively — not every screen needs all four techniques. Use the technique only when the spec provides the trigger condition.
|
|
10
|
+
|
|
11
|
+
| Technique | Apply when spec mentions | Skip when |
|
|
12
|
+
|---|---|---|
|
|
13
|
+
| EP (Equivalence Partitioning) | Input types, categories, roles, valid/invalid ranges | No validation or only a required-field check |
|
|
14
|
+
| BVA (Boundary Value Analysis) | Numeric range, string length, date range, count limit | No boundary defined in spec |
|
|
15
|
+
| Decision Table | 2+ mutually dependent conditions with different outcomes | Conditions are independent of each other |
|
|
16
|
+
| State Transition | Entity lifecycle, workflow states, status changes | No state machine in spec |
|
|
15
17
|
|
|
16
18
|
**Rule:** These techniques determine **how many** and **which** scenarios to generate. `sungen-viewpoint` determines **which viewpoints** to cover.
|
|
17
19
|
|
|
@@ -53,7 +55,7 @@ Scenario: VP-VAL-003 Above maximum is rejected # value = 101
|
|
|
53
55
|
| Mode | Values | Use when |
|
|
54
56
|
|---|---|---|
|
|
55
57
|
| **Compact (default)** | `min-1`, `min`, `max`, `max+1` | Most fields |
|
|
56
|
-
| **Full 6-point** | `min-1`, `min`, `min+1`, `max-1`, `max`, `max+1` |
|
|
58
|
+
| **Full 6-point** | `min-1`, `min`, `min+1`, `max-1`, `max`, `max+1` | High-risk fields with `@high` priority |
|
|
57
59
|
|
|
58
60
|
**How to apply** (example: "quantity must be 1-10"):
|
|
59
61
|
- `min-1` = 0 -> invalid
|
|
@@ -62,50 +64,17 @@ Scenario: VP-VAL-003 Above maximum is rejected # value = 101
|
|
|
62
64
|
- `max+1` = 11 -> invalid
|
|
63
65
|
- Midpoint (e.g., 5) already covered by EP valid class
|
|
64
66
|
|
|
65
|
-
**BVA scenarios** (example: quantity 1-10):
|
|
66
|
-
-
|
|
67
|
-
-
|
|
68
|
-
-
|
|
69
|
-
-
|
|
70
|
-
|
|
71
|
-
### Expected outcome contract (MANDATORY)
|
|
72
|
-
|
|
73
|
-
When generating BVA scenarios, the **Then** assertion **MUST** match the scenario name's outcome verb:
|
|
74
|
-
|
|
75
|
-
| Name verb | Expected `Then` assertion |
|
|
76
|
-
|---|---|
|
|
77
|
-
| "is accepted" / "passes validation" | NO validation error visible. Form proceeds OR field is in normal state. Use `User see [Field Error] message is hidden` if uncertain. |
|
|
78
|
-
| "is rejected" / "fails validation" / "shows error" | Specific validation error visible with exact message text from spec. |
|
|
79
|
-
|
|
80
|
-
**Anti-pattern (do NOT generate):**
|
|
67
|
+
**BVA scenarios** (example: quantity 1-10) — 4 TCs per range:
|
|
68
|
+
- `VP-VAL-010 Below minimum (0) is rejected`
|
|
69
|
+
- `VP-VAL-011 Minimum boundary (1) is accepted`
|
|
70
|
+
- `VP-VAL-012 Maximum boundary (10) is accepted`
|
|
71
|
+
- `VP-VAL-013 Above maximum (11) is rejected`
|
|
81
72
|
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
When User fill [Email] field with {{email_max_254}}
|
|
86
|
-
And User fill [Password] field with {{valid_password}}
|
|
87
|
-
And User click [Login] button
|
|
88
|
-
Then User see [Auth Error] message with {{auth_error_message}}
|
|
89
|
-
```
|
|
90
|
-
|
|
91
|
-
**Correct pattern:**
|
|
73
|
+
**Priority by field risk:**
|
|
74
|
+
- Critical field / inclusive operator (`<=`, `>=`) / business-rule boundary → `@high` (Tier 1)
|
|
75
|
+
- Standard field with a spec-defined range → `@normal` (Tier 2)
|
|
92
76
|
|
|
93
|
-
|
|
94
|
-
# GOOD: name and assertion aligned
|
|
95
|
-
@normal
|
|
96
|
-
Scenario: VP-VAL-005 Email at maximum boundary (254 chars) passes validation
|
|
97
|
-
When User fill [Email] field with {{email_max_254}}
|
|
98
|
-
And User fill [Password] field with {{valid_password}}
|
|
99
|
-
And User click [Login] button
|
|
100
|
-
Then User see [Email Error] message is hidden
|
|
101
|
-
|
|
102
|
-
@normal
|
|
103
|
-
Scenario: VP-VAL-006 Email at maximum+1 (255 chars) fails validation
|
|
104
|
-
When User fill [Email] field with {{email_over_max_255}}
|
|
105
|
-
And User fill [Password] field with {{valid_password}}
|
|
106
|
-
And User click [Login] button
|
|
107
|
-
Then User see [Email Error] message with {{email_too_long_error}}
|
|
108
|
-
```
|
|
77
|
+
**Low-risk fields:** Apply EP only (valid class + invalid class) — skip BVA entirely. Reserve BVA for fields where off-by-one errors have user impact (quantity limits, file size, password length, date ranges).
|
|
109
78
|
|
|
110
79
|
---
|
|
111
80
|
|
|
@@ -123,6 +92,11 @@ Scenario: VP-VAL-006 Email at maximum+1 (255 chars) fails validation
|
|
|
123
92
|
- `@normal` Has permission + form invalid → disabled
|
|
124
93
|
- `@high` Form valid + has permission → succeeds
|
|
125
94
|
|
|
95
|
+
**Tier mapping for DT rows:**
|
|
96
|
+
- Row with a **distinct outcome not covered by any other row** → Tier 1 `@high`
|
|
97
|
+
- Row whose outcome is already tested by another row (same result, different condition path) → Tier 2 `@normal`
|
|
98
|
+
- When >3 boolean conditions (>8 rows): keep all distinct-outcome rows as Tier 1, move redundant-outcome rows to Tier 2 or `@manual`.
|
|
99
|
+
|
|
126
100
|
---
|
|
127
101
|
|
|
128
102
|
## 4. State Transition
|