@roopesh.yadava/qa-pack 1.5.0 → 1.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/claude/skills/SKILLS_CONTEXT.md +15 -4
- package/claude/skills/manual-testing/SKILL.md +97 -24
- package/claude/skills/manual-testing/TEST_DESIGN_GUIDE.md +200 -0
- package/claude/skills/mobile-automation/BDD_TEMPLATES.md +221 -0
- package/claude/skills/mobile-automation/LOCATOR_PATTERNS.md +205 -0
- package/claude/skills/mobile-automation/MOBILE_MCP_REFERENCE.md +234 -0
- package/claude/skills/mobile-automation/SKILL.md +593 -0
- package/claude/skills/qa-agent/toolkit/qa-toolkit.cjs +62 -9
- package/package.json +1 -1
|
@@ -50,6 +50,8 @@ A 10-test run with `browser_snapshot()` per test = ~100k wasted tokens.
|
|
|
50
50
|
| `BDD_TEMPLATES.md` | Once at Phase 1 start | Never reload in the same run |
|
|
51
51
|
| `LOCATOR_PATTERNS.md` | Once at Phase 1 start | Never reload in Phase 2 or 3 |
|
|
52
52
|
| `WCAG_CHECKS.md` | Once at skill start | Never reload |
|
|
53
|
+
| `manual-testing/TEST_DESIGN_GUIDE.md` | Once at Phase 1c (manual-testing) | Never reload — reference sections by number (§1–§9) |
|
|
54
|
+
| `mobile-automation/MOBILE_MCP_REFERENCE.md`, `BDD_TEMPLATES.md`, `LOCATOR_PATTERNS.md` | Once at Phase 2 start (mobile-automation) | Never reload in Phase 3 or 4 |
|
|
53
55
|
| `test-charter.md` | Once when charter starts | Never reload |
|
|
54
56
|
| `context.md` | Once at qa-agent Step 0 | Never reload |
|
|
55
57
|
|
|
@@ -86,14 +88,14 @@ node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs <command> [--flags]
|
|
|
86
88
|
| Command | Used by | Purpose |
|
|
87
89
|
|---------|---------|---------|
|
|
88
90
|
| `get-bugs` / `get-runs` / `get-selectors` --product P | any skill | Cheap reads of one context.md table, pipe-delimited rows |
|
|
89
|
-
| `fingerprint` --product P --url U --testids "a,b" | manual-testing, automation, roam-testing | DOM fingerprint cache — `UNCHANGED` means skip re-discovery |
|
|
91
|
+
| `fingerprint` --product P --url U --testids "a,b" | manual-testing, automation, mobile-automation, roam-testing | DOM/screen fingerprint cache — `UNCHANGED` means skip re-discovery. Mobile uses a synthetic `mobile:{bundleId}/{Screen}` string as `--url` and resource-id/accessibility-id values as `--testids` — the command itself is platform-agnostic |
|
|
90
92
|
| `pii-scan` (stdin or `--file`) | bug-reporting, manual-testing, roam-testing | Flags emails/keys/tokens before anything is posted to Jira |
|
|
91
93
|
| `dup-bug` --product P --summary S | bug-reporting, manual-testing, roam-testing | Fuzzy-matches a new bug against Known Bugs, no LLM comparison needed |
|
|
92
94
|
| `cost-estimate` --product P [--phase N] | qa-agent | One-line run-cost estimate from Runs Log history |
|
|
93
95
|
| `risk-score` --product P --modules "a,b" | manual-testing | Orders tests by git churn + past bug density, not AC order |
|
|
94
|
-
| `trust-record` / `trust-status` --product P | automation (
|
|
95
|
-
| `locator-record` / `locator-query` --product P --page U | automation | Self-improving locator memory across self-heal fixes |
|
|
96
|
-
| `dashboard` / `digest [--days N]` / `roi` | qa-insights | Cross-product reports — 100% script-generated, zero synthesis |
|
|
96
|
+
| `trust-record` / `trust-status` --product P [--namespace N] | automation, mobile-automation (`--namespace mobile` — routes to a separate `trust.mobile.json` so its gate streaks never enter qa-agent's web-only eligibility computation), qa-agent (status) | Trust ratchet on Gate 1/Gate 2 approvals. `--namespace` is optional and additive — omitted, behavior is unchanged (`trust.json`); `trust-status` takes `min(streak)` across every gate in whichever file it reads, so gates from two automation surfaces must never share one file |
|
|
97
|
+
| `locator-record` / `locator-query` --product P --page U | automation, mobile-automation (page keys prefixed `mobile:` to avoid colliding with web page URLs) | Self-improving locator memory across self-heal fixes |
|
|
98
|
+
| `dashboard` / `digest [--days N]` / `roi` | qa-insights | Cross-product reports — 100% script-generated, zero synthesis. Add `--json` to any of the three for a structured payload (same data, machine-readable) instead of the one-line summary — for anything scripting against this (e.g. an external dashboard app) rather than chatting with it. |
|
|
97
99
|
|
|
98
100
|
This file is versioned logic (always overwritten on `npm update`, like every other skill
|
|
99
101
|
file) — never store product data inside it. Its outputs live under
|
|
@@ -148,12 +150,18 @@ test-charter │
|
|
|
148
150
|
this diagram — they don't take a Jira card, and qa-insights doesn't touch Playwright at all.
|
|
149
151
|
Both are driven by the toolkit rather than by each other.
|
|
150
152
|
|
|
153
|
+
`mobile-automation` also sits outside this diagram — it's a standalone, directly-triggered
|
|
154
|
+
skill (same status as `roam-testing`/`k6-framework-scaffold`), not wired into qa-agent's
|
|
155
|
+
Phase 2 dispatch, which remains web/Playwright-only via `automation`. Trigger it by name
|
|
156
|
+
("automate mobile PROJ-123", "mobile test PROJ-123") rather than through the qa-agent phase menu.
|
|
157
|
+
|
|
151
158
|
## Skills — One-Line Summary
|
|
152
159
|
|
|
153
160
|
| Skill | Input | Output | MCP Needed |
|
|
154
161
|
|-------|-------|--------|------------|
|
|
155
162
|
| `qa-agent` | Jira card ID or menu choice | Dispatches to correct skill | Atlassian |
|
|
156
163
|
| `automation` | Jira card ID | Reuse audit + Gherkin + Step Defs + POM + real run (reuse % reported) | Atlassian, Playwright |
|
|
164
|
+
| `mobile-automation` | Jira card ID | Reuse audit + mobile Gherkin + Step Defs + Screen POM + real device run (reuse % reported) | Atlassian, Mobile MCP, MobileWright/mobilecli |
|
|
157
165
|
| `manual-testing` | Jira card ID + app URL | Execution report + bugs + charter | Atlassian, Playwright |
|
|
158
166
|
| `ui-test-figma` | Figma URL + app URL | UI mismatch report, Jira comment | Playwright (CLI+MCP), Figma (optional) |
|
|
159
167
|
| `accessibility-testing` | Full page URL + Jira card (optional) | WCAG 2.1 A/AA report + Jira bugs | Playwright (CLI+MCP), Atlassian |
|
|
@@ -244,6 +252,7 @@ python3 $PROJECT/track_tokens.py session
|
|
|
244
252
|
|
|
245
253
|
**Phase names by skill:**
|
|
246
254
|
- automation: `start` → `jira_fetch` → `reuse_audit` → `gherkin_generation` → `step_definitions` → `pom_generation` → `test_run` → `end`
|
|
255
|
+
- mobile-automation: `start` → `jira_fetch` → `reuse_audit` → `gherkin_generation` → `step_definitions` → `pom_generation` → `test_run` → `end`
|
|
247
256
|
- manual-testing: `start` → `jira_fetch` → `ui_testing` → `test_execution` → `end`
|
|
248
257
|
- ui-test-figma: `start` → `login` → `comparison` → `end`
|
|
249
258
|
- accessibility-testing: `start` → `login` → `a11y_checks` → `end`
|
|
@@ -280,6 +289,7 @@ First run will always be 0% (cold cache). Second and subsequent runs should cach
|
|
|
280
289
|
| Auth session | `.playwright-session.json` (gitignored) |
|
|
281
290
|
| Secrets / environment | `.env` at repo root (gitignored) |
|
|
282
291
|
| Step catalog (reuse audit) | `.claude/skills/qa-agent/product_context/[PRODUCT]/step-catalog.md` |
|
|
292
|
+
| Mobile step catalog (reuse audit) | `.claude/skills/qa-agent/product_context/[PRODUCT]/mobile-step-catalog.md` |
|
|
283
293
|
| Automation hints | `outputs/automation-hints-[CARD]-[date].md` |
|
|
284
294
|
| Token analytics | `~/.claude/token_analytics.png` |
|
|
285
295
|
| Knowledge graph | `graphify-out/graph.html` (open in browser) |
|
|
@@ -292,3 +302,4 @@ First run will always be 0% (cold cache). Second and subsequent runs should cach
|
|
|
292
302
|
| QA weekly digest | `outputs/qa-weekly-digest-[date].md` |
|
|
293
303
|
| Token-spend / ROI view | `outputs/roi-report.md` |
|
|
294
304
|
| Shared toolkit CLI | `.claude/skills/qa-agent/toolkit/qa-toolkit.cjs` |
|
|
305
|
+
| Manual-testing coverage/scenario reference | `.claude/skills/manual-testing/TEST_DESIGN_GUIDE.md` |
|
|
@@ -2,8 +2,12 @@
|
|
|
2
2
|
name: manual-testing
|
|
3
3
|
description: >
|
|
4
4
|
Manual Testing branch orchestrator. Follows the flowchart:
|
|
5
|
-
Jira Card Input → UI Testing (Figma MCP, optional) →
|
|
6
|
-
→ Bug Reporting (Atlassian MCP) → Test Charter →
|
|
5
|
+
Jira Card Input → Requirement Gap Analysis → UI Testing (Figma MCP, optional) →
|
|
6
|
+
Manual Testing (Playwright MCP) → Bug Reporting (Atlassian MCP) → Test Charter →
|
|
7
|
+
Automation Agent (optional handoff).
|
|
8
|
+
Test planning is risk-scoped and coverage-dimension-driven (see TEST_DESIGN_GUIDE.md) —
|
|
9
|
+
not just "1-3 tests per AC line" — so a payment/auth card gets materially deeper coverage
|
|
10
|
+
than a copy-change card.
|
|
7
11
|
Captures element selectors and interaction data during execution and saves them to an
|
|
8
12
|
automation-hints file for the automation skill to reuse — skipping DOM re-discovery.
|
|
9
13
|
Uses Playwright CLI for zero-token screenshots. Token tracking enabled.
|
|
@@ -99,6 +103,13 @@ Derive `PRODUCT_FOLDER` now (uppercase `PROJECT_KEY`'s product name, spaces →
|
|
|
99
103
|
normalisation qa-agent Step 6a uses) so Phases 3 and 4 below can call the toolkit without
|
|
100
104
|
re-deriving it. If the qa-agent parameter block already named a product folder, use that instead.
|
|
101
105
|
|
|
106
|
+
If a product folder was resolved, pull known bugs cheaply for the retest check used in 3a/§9
|
|
107
|
+
— one script call, not a `context.md` read:
|
|
108
|
+
```bash
|
|
109
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs get-bugs --product {PRODUCT_FOLDER}
|
|
110
|
+
```
|
|
111
|
+
Store non-`Closed`/`Done` rows as `OPEN_KNOWN_BUGS`. `NO_CONTEXT_FILE`/`NO_BUGS` → `OPEN_KNOWN_BUGS = []`, first run for this product.
|
|
112
|
+
|
|
102
113
|
Run token tracking `jira_fetch` checkpoint.
|
|
103
114
|
|
|
104
115
|
### 1b — Truncate Jira data if card is verbose
|
|
@@ -112,14 +123,33 @@ After fetching, check `CARD_DESCRIPTION` length:
|
|
|
112
123
|
|
|
113
124
|
Do NOT tell the user the description was truncated. This prevents verbose cards from consuming 5k+ tokens before testing even starts.
|
|
114
125
|
|
|
115
|
-
If no AC found (and qa-agent didn't already resolve this):
|
|
116
|
-
> "No Acceptance Criteria found on this card. What should be tested?"
|
|
117
|
-
Wait for user response before continuing.
|
|
118
|
-
|
|
119
126
|
Set from Phase 0 values (all input was collected there — do not re-ask):
|
|
120
127
|
- `RUN_UI_TEST` — true when a Figma URL is available (from parameter block, card, or user)
|
|
121
128
|
- `FIGMA_URL`, `APP_URL`, `USERNAME`, `PASSWORD`, `OTP_CODE`
|
|
122
129
|
|
|
130
|
+
### 1c — Load Test Design Guide (once)
|
|
131
|
+
|
|
132
|
+
Read `.claude/skills/manual-testing/TEST_DESIGN_GUIDE.md` now. Per the Static File Load
|
|
133
|
+
Rules in `SKILLS_CONTEXT.md`, load it once here and never reload it later in this run —
|
|
134
|
+
reference its sections by number (e.g. "per §4 High risk") for the rest of the pipeline.
|
|
135
|
+
|
|
136
|
+
### 1d — Requirement Gap Analysis
|
|
137
|
+
|
|
138
|
+
Run the checklist in `TEST_DESIGN_GUIDE.md` §1 against `CARD_TITLE` + `ACCEPTANCE_CRITERIA`.
|
|
139
|
+
This **replaces** the old bare "no AC found" check — the missing-AC row is §1's first row and
|
|
140
|
+
is the most severe: if it fires, skip evaluating the rest of the table and ask only that.
|
|
141
|
+
|
|
142
|
+
Fold every gap that actually applies into **one** consolidated question — the same message as
|
|
143
|
+
any unresolved Phase 0 gaps, not a second round:
|
|
144
|
+
> "No Acceptance Criteria found on this card. What should be tested?" (only if AC is entirely
|
|
145
|
+
> missing), or, when AC exists but has gaps: one line per applicable §1 row, e.g. "AC mentions
|
|
146
|
+
> a 'reference code' but doesn't state its length/format — what's the exact rule?"
|
|
147
|
+
|
|
148
|
+
Skip this step's question entirely if the AC is already precise and complete on every §1
|
|
149
|
+
row — do not manufacture questions to fill the message.
|
|
150
|
+
|
|
151
|
+
Wait for the response before proceeding to Phase 2.
|
|
152
|
+
|
|
123
153
|
---
|
|
124
154
|
|
|
125
155
|
## Phase 2 — UI Testing (optional)
|
|
@@ -162,20 +192,44 @@ HINTS = {
|
|
|
162
192
|
}
|
|
163
193
|
```
|
|
164
194
|
|
|
165
|
-
### 3a — Generate Test Plan
|
|
195
|
+
### 3a — Generate Test Plan (risk-scoped, coverage-dimension-driven)
|
|
166
196
|
|
|
167
197
|
Before drafting test ideas, check each AC item for a stated entry point (menu path, URL,
|
|
168
198
|
button/link name). Per the **Test Data & Entity Selection Rules** above, do not scan `src/`
|
|
169
199
|
to infer a missing one — collect a single navigation question per missing entry point and
|
|
170
200
|
fold it into the confirmation prompt below instead of guessing.
|
|
171
201
|
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
202
|
+
**Step 1 — Classify risk tier and select dimensions.** Using `TEST_DESIGN_GUIDE.md` §2 (Feature
|
|
203
|
+
Signal → Coverage Dimension Trigger Table), scan `CARD_TITLE` + `ACCEPTANCE_CRITERIA` for
|
|
204
|
+
signal keywords and turn on the matched dimensions. No signal matched → default set only
|
|
205
|
+
(Functional Positive/Negative + UI/UX States + one Boundary check). Then classify the card's
|
|
206
|
+
risk tier per §4 (High/Medium/Low) from the same signals — this sets how many tests each
|
|
207
|
+
triggered dimension gets, not just which dimensions run.
|
|
208
|
+
|
|
209
|
+
**Step 2 — Draft numbered test ideas per dimension.** Walk the §5 Test Scenario Identification
|
|
210
|
+
Framework (the 10 "what can/can't/who/what happens" questions) once, and for each dimension
|
|
211
|
+
turned on in Step 1, draft the number of tests §4's tier calls for. Concretely, for a Medium-risk
|
|
212
|
+
card this typically looks like: 1–2 Functional Positive, 1–2 Functional Negative, one Boundary
|
|
213
|
+
or Equivalence set per §6 if the AC states or implies a limit, one UI/UX States pass (§8) on the
|
|
214
|
+
card's key screen, plus one test per other triggered dimension (Permissions, API, Database,
|
|
215
|
+
State/Workflow, etc.). High-risk cards double the per-dimension count and add Security-oriented
|
|
216
|
+
+ Database checks unconditionally per §4. Low-risk cards (copy/label-only) skip straight to
|
|
217
|
+
Functional (positive + one negative) plus one regression spot-check — do not force the other
|
|
218
|
+
dimensions onto a low-risk card.
|
|
219
|
+
|
|
220
|
+
Cap exploratory ideas (§5 Q10) at one per plan, clearly tagged `[exploratory]`, so they don't
|
|
221
|
+
inflate the "planned" test count used for the Exit Criteria check later (§9).
|
|
222
|
+
|
|
223
|
+
If `OPEN_KNOWN_BUGS` (from Phase 1) contains a row whose Title overlaps this card's module/page
|
|
224
|
+
keywords, add one retest test tagged `[retest: {BUG-ID}]` — this is what §9's retest check
|
|
225
|
+
verifies actually happened.
|
|
226
|
+
|
|
227
|
+
Do not draft dedicated regression tests here — that's qa-agent Step 5's job (Covered Flows
|
|
228
|
+
overlap), triggered automatically after this skill returns. §5 Q9 exists only to flag *that*
|
|
229
|
+
regression may apply, not to author it now.
|
|
230
|
+
|
|
231
|
+
**Risk-based execution ordering** — pull 2–5 module/page keywords straight out of the AC/card
|
|
232
|
+
title you already read (e.g. "login", "checkout"; no extra fetching), then run one script call:
|
|
179
233
|
|
|
180
234
|
```bash
|
|
181
235
|
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs risk-score --product {PRODUCT_FOLDER} --modules "kw1,kw2,kw3"
|
|
@@ -183,11 +237,13 @@ node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs risk-score --product {PRODUC
|
|
|
183
237
|
|
|
184
238
|
Order the T-01, T-02... list so tests touching the highest-risk module come first. On a
|
|
185
239
|
product's first run (no git history / no known bugs yet) every module scores 0 — keep the
|
|
186
|
-
natural
|
|
240
|
+
natural drafting order in that case, no need to mention it. (This reorders the tests drafted in
|
|
241
|
+
Step 2 — it doesn't decide how many exist; that's §4's job, run once, not repeated per module.)
|
|
187
242
|
|
|
188
|
-
Show as a compact table (T-01, T-02 ... with name and expected outcome),
|
|
189
|
-
Ask: `"Ready to run these tests? (yes / no or edit)"` — include any missing
|
|
190
|
-
questions in this same message
|
|
243
|
+
Show as a compact table (T-01, T-02 ... with name, dimension tag, and expected outcome),
|
|
244
|
+
risk-ordered. Ask: `"Ready to run these tests? (yes / no or edit)"` — include any missing
|
|
245
|
+
navigation questions from above in this same message (§1 gap questions were already asked
|
|
246
|
+
and resolved in Phase 1d — don't re-ask them here).
|
|
191
247
|
Proceed on confirmation.
|
|
192
248
|
|
|
193
249
|
### 3b — Setup Browser + Capture Login Selectors
|
|
@@ -305,10 +361,13 @@ For each test T-01, T-02, ...:
|
|
|
305
361
|
outputs/screenshots/T-[N]-[pass|fail|observation].png
|
|
306
362
|
```
|
|
307
363
|
|
|
308
|
-
5. Assert expected result via targeted `browser_evaluate` on specific selectors
|
|
364
|
+
5. Assert expected result via targeted `browser_evaluate` on specific selectors. If this
|
|
365
|
+
test's dimension tag (from 3a) is UI/UX States, note which of the `TEST_DESIGN_GUIDE.md`
|
|
366
|
+
§8 states (Initial/Loading/Empty/Success/Error/Disabled) were actually observed and their
|
|
367
|
+
result — skip states genuinely unreachable in this test, don't force them.
|
|
309
368
|
|
|
310
369
|
6. Log test result (to file only, not chat):
|
|
311
|
-
`T-N | title | status | expected | actual | screenshot path`
|
|
370
|
+
`T-N | title | dimension | status | expected | actual | screenshot path`
|
|
312
371
|
|
|
313
372
|
7. Append to `HINTS.testCases`:
|
|
314
373
|
```
|
|
@@ -407,13 +466,19 @@ this keeps context contiguous and avoids a second skill invocation.
|
|
|
407
466
|
|
|
408
467
|
**Step 4a — Build bug payload for each ❌ FAIL and ⚠️ OBSERVATION**
|
|
409
468
|
|
|
410
|
-
Derive all fields from test execution data — no user input needed
|
|
469
|
+
Derive all fields from test execution data — no user input needed. Fields follow
|
|
470
|
+
`TEST_DESIGN_GUIDE.md` §7:
|
|
411
471
|
|
|
412
472
|
| Field | Source |
|
|
413
473
|
|-------|--------|
|
|
414
474
|
| Summary | `"T-{N}: {test name} — {actual outcome in one line}"` |
|
|
415
|
-
|
|
|
475
|
+
| Environment | `APP_URL` + environment name from product context (staging/dev/prod) |
|
|
476
|
+
| Preconditions | Login state + any setup steps from the test case (e.g. role, prior state) |
|
|
477
|
+
| Description (Steps/Expected/Actual) | Steps from test case + expected vs actual from execution log |
|
|
416
478
|
| Severity | AC explicitly failed → High · Assertion failed → Medium · Observation → Low |
|
|
479
|
+
| Priority | High-risk dimension (§4) failing → High · otherwise matches Severity |
|
|
480
|
+
| Test data used | The specific values/entities used for this test (never credentials) |
|
|
481
|
+
| Build/version | `CARD_ID` + run date (stands in for a separate build number) |
|
|
417
482
|
| Screenshot | Match `outputs/screenshots/T-{N}-*.png` by test ID — use exact filename |
|
|
418
483
|
|
|
419
484
|
**Step 4a.1 — Duplicate check (one script call per failure, no user input needed)**
|
|
@@ -496,14 +561,22 @@ Run token tracking `end + report + session` close-out after charter publishes.
|
|
|
496
561
|
|
|
497
562
|
## Phase 6 — Automation Handoff
|
|
498
563
|
|
|
499
|
-
After charter completes,
|
|
564
|
+
After charter completes, run the `TEST_DESIGN_GUIDE.md` §9 Exit Criteria Quick Check:
|
|
565
|
+
confirm every planned test executed (BLOCKED is fine, silently-skipped is not), check whether
|
|
566
|
+
any bug just filed is High severity on a High-risk dimension (§4) and if so flag it as the
|
|
567
|
+
first line of the summary below, and — only if `OPEN_KNOWN_BUGS` (Phase 1) was non-empty and
|
|
568
|
+
overlapped this card — confirm the `[retest: {BUG-ID}]` test from 3a actually ran and note its
|
|
569
|
+
outcome. This does not block or gate anything — it only makes sure nothing gets buried in the
|
|
570
|
+
summary.
|
|
571
|
+
|
|
572
|
+
Show the final summary:
|
|
500
573
|
|
|
501
574
|
```
|
|
502
575
|
Manual Testing Complete — [CARD_ID]
|
|
503
576
|
|
|
504
577
|
UI Testing (Figma) : [completed / skipped]
|
|
505
578
|
Manual Testing : X Pass | X Fail | X Observation | X Blocked
|
|
506
|
-
Bug Reporting : [N] bug(s) filed — [keys]
|
|
579
|
+
Bug Reporting : [N] bug(s) filed — [keys] [+ "⚠ High-severity on {dimension}" if applicable]
|
|
507
580
|
Test Charter : [published URL or "saved locally"]
|
|
508
581
|
Automation hints : outputs/automation-hints-[CARD_ID]-[YYYYMMDD].md
|
|
509
582
|
```
|
|
@@ -0,0 +1,200 @@
|
|
|
1
|
+
# Test Design Guide — Requirement Analysis, Coverage Dimensions, Scenario Framework
|
|
2
|
+
|
|
3
|
+
Loaded once by the manual-testing skill at Phase 1 (Requirement Analysis). Reference sections
|
|
4
|
+
by number afterward (e.g. "per §2 trigger table") — never re-quote this file in chat or in
|
|
5
|
+
saved reports.
|
|
6
|
+
|
|
7
|
+
Purpose: manual testing should verify a feature is functionally correct, usable, reliable,
|
|
8
|
+
secure, compatible, and aligned with business expectations — not just that the happy path
|
|
9
|
+
works. This guide operationalizes that without inflating every card into the same fixed
|
|
10
|
+
checklist: coverage is scoped by what the card actually touches (§2) and by risk (§4), so a
|
|
11
|
+
copy-change card and a payment card get proportionally different depth.
|
|
12
|
+
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
## §1 — Requirement Gap Checklist
|
|
16
|
+
|
|
17
|
+
Run once per card, right after Phase 1 fetch, before the test plan is drafted. For each row
|
|
18
|
+
whose signal is present, fold the question into the **one** consolidated gap-question message
|
|
19
|
+
(same message as any Phase 0 gaps — never a second round of questions). Skip a row entirely
|
|
20
|
+
if the AC already answers it — do not manufacture questions on a precise, complete AC.
|
|
21
|
+
|
|
22
|
+
| Gap type | Signal to look for | Question to ask if missing |
|
|
23
|
+
|---|---|---|
|
|
24
|
+
| No AC at all | Card has no AC/COS section | "No Acceptance Criteria found — what should be tested?" (most severe — if this fires, skip the rest of this table and just ask this) |
|
|
25
|
+
| Ambiguous requirement | Vague terms in AC ("appropriate", "as needed", "similar to", "etc.") | Quote the exact phrase, ask for the precise rule |
|
|
26
|
+
| Missing validation rule | A field/form is named with no stated format or limit | "What's the valid format/limit for {field}?" |
|
|
27
|
+
| Missing negative/error case | AC describes only the happy path, no failure behavior | "What should happen on {plausible failure mode}?" |
|
|
28
|
+
| Missing permission rule | Roles/access mentioned but AC doesn't say who can do what | "Which roles can perform this action?" |
|
|
29
|
+
| Missing boundary condition | A numeric/length limit is implied ("a short code", "a few items") but not exact | "What's the exact min/max for {field}?" |
|
|
30
|
+
| Missing state/workflow rule | A status/workflow field exists but transitions aren't listed | "What are the valid status transitions?" |
|
|
31
|
+
| Missing data relationship | AC needs a specific paired entity (parent+child, account+card) without naming which | Already covered by the skill's Test Data & Entity Selection Rules — ask once or mark `🔒 BLOCKED`, don't duplicate the question here |
|
|
32
|
+
|
|
33
|
+
---
|
|
34
|
+
|
|
35
|
+
## §2 — Feature Signal → Coverage Dimension Trigger Table
|
|
36
|
+
|
|
37
|
+
Scan the card title + AC (already read in Phase 1 — no extra fetching, no grepping `src/`)
|
|
38
|
+
for these signals. A matched signal turns its dimension(s) ON for test-plan generation in
|
|
39
|
+
Phase 3a. Untriggered dimensions are not drafted — this is what keeps test count proportional
|
|
40
|
+
to the card instead of forcing all 19 dimensions onto every run.
|
|
41
|
+
|
|
42
|
+
| Signal in title/AC | Dimensions turned ON |
|
|
43
|
+
|---|---|
|
|
44
|
+
| login, auth, OTP, session, token, password, logout | Authentication, Security-oriented, State/Workflow (session) |
|
|
45
|
+
| role, permission, admin, restricted, access level | Authorization / Permissions |
|
|
46
|
+
| payment, price, amount, invoice, refund, subscription, billing, calculation | Boundary/Equivalence, Business Rules, Database, Security-oriented |
|
|
47
|
+
| delete, remove, deactivate, cancel, archive | Negative, State/Workflow, Database (soft vs hard delete) |
|
|
48
|
+
| upload, download, attachment, file, import, export | File Upload/Download |
|
|
49
|
+
| search, filter, sort, paginate, list, table, grid | Search/Filter/Sort/Pagination |
|
|
50
|
+
| API, endpoint, integration, webhook, third-party, external service, sync | API, Integration |
|
|
51
|
+
| status, state, workflow, transition, approve, submit, draft, review | State/Workflow |
|
|
52
|
+
| any form or input field named in the AC | Validation, Boundary/Equivalence |
|
|
53
|
+
| notification, email, SMS, push, alert, toast, reminder | Notifications |
|
|
54
|
+
| multi-step flow, wizard, redirect, navigation between screens | End-to-End |
|
|
55
|
+
| *(always on, every card)* | Functional Positive, Functional Negative, UI/UX States |
|
|
56
|
+
|
|
57
|
+
No specific signal matched → draft Functional (positive + one negative) + UI/UX States + one
|
|
58
|
+
Boundary check only. Do not force File Upload tests on a card with no file field, API tests on
|
|
59
|
+
a pure front-end copy change, etc.
|
|
60
|
+
|
|
61
|
+
---
|
|
62
|
+
|
|
63
|
+
## §3 — Coverage Dimension Catalog
|
|
64
|
+
|
|
65
|
+
What each dimension actually means when drafting a test idea:
|
|
66
|
+
|
|
67
|
+
- **Functional Positive** — valid input/action produces the documented expected result.
|
|
68
|
+
- **Functional Negative** — invalid input, missing mandatory field, unauthorized action, wrong
|
|
69
|
+
state transition, or a dependency failure (API/network) is handled without a crash and with
|
|
70
|
+
a correct, user-facing message.
|
|
71
|
+
- **Boundary/Equivalence** — see §6 templates. Values at/around a stated limit, and one
|
|
72
|
+
representative value per valid/invalid partition — not exhaustive enumeration.
|
|
73
|
+
- **UI/UX States** — see §8. Initial, loading, empty, success, error, disabled states of the
|
|
74
|
+
components the card actually touches.
|
|
75
|
+
- **Validation** — mandatory/optional fields, character/numeric limits, special characters,
|
|
76
|
+
whitespace, duplicate values, invalid format, field interdependencies.
|
|
77
|
+
- **Business Rules** — role-based/status-based/date-based behavior, calculations, eligibility
|
|
78
|
+
and workflow-transition rules stated in the AC.
|
|
79
|
+
- **Authentication** — valid/invalid credentials, empty credentials, OTP flow, session
|
|
80
|
+
expiry, logout, token expiration.
|
|
81
|
+
- **Authorization / Permissions** — each distinct role named in the AC gets one test for
|
|
82
|
+
allowed and one for denied behavior; direct-URL/API access with an unauthorized session
|
|
83
|
+
should not succeed just because the UI hid the button.
|
|
84
|
+
- **API** (only when the card's flow visibly depends on a request) — correct status code for
|
|
85
|
+
the scenario (200/201/204 success; 400 invalid; 401 unauthenticated; 403 unauthorized; 404
|
|
86
|
+
not found; 409 conflict/duplicate; 500 server failure), response shape sanity, not full
|
|
87
|
+
contract testing.
|
|
88
|
+
- **Database** (only when the UI exposes persisted state) — does the value shown after a
|
|
89
|
+
reload match what was submitted; does a delete actually remove/soft-remove the record;
|
|
90
|
+
spot-check only, not a DB audit.
|
|
91
|
+
- **Integration** — cross-system handoff visible from the UI (payment gateway redirect,
|
|
92
|
+
auth-provider callback, file-storage upload) — verify the round trip completes and failures
|
|
93
|
+
are surfaced, not the integration's internals.
|
|
94
|
+
- **End-to-End** — the full journey named in the card (e.g. Login → create entity → submit →
|
|
95
|
+
see it reflected downstream), not just the single screen in the AC.
|
|
96
|
+
- **File Upload/Download** — supported/unsupported type, size limit, empty/corrupted file,
|
|
97
|
+
duplicate filename, correct downloaded filename/contents.
|
|
98
|
+
- **Search/Filter/Sort/Pagination** — exact match, no-match, empty query, filter
|
|
99
|
+
combination, clear filters, sort direction, first/last page, empty result page.
|
|
100
|
+
- **Notifications** — correct message, correct recipient/trigger, not duplicated, and (if
|
|
101
|
+
observable within the run) sent after both success and retry-after-failure.
|
|
102
|
+
- **State/Workflow** — every transition named in the AC, one attempt at an invalid transition,
|
|
103
|
+
and what the UI shows on reopening/refreshing after a transition.
|
|
104
|
+
- **Security-oriented** (only for high-risk signals in §4) — direct URL access while logged
|
|
105
|
+
out or as the wrong role, sensitive data not exposed in the URL or in error text, password
|
|
106
|
+
masked, session actually ends on logout.
|
|
107
|
+
- **Accessibility spot-check** — one keyboard-only pass through the card's critical path
|
|
108
|
+
(tab order reaches every control, focus is visible, no dead-end). This is a lightweight
|
|
109
|
+
spot-check, not the full WCAG audit — hand off to the `accessibility-testing` skill if the
|
|
110
|
+
card needs that level of coverage.
|
|
111
|
+
- **Compatibility** — only when the card explicitly calls out a browser/viewport; otherwise
|
|
112
|
+
out of scope for a single-card manual run (covered separately, not by default here).
|
|
113
|
+
|
|
114
|
+
---
|
|
115
|
+
|
|
116
|
+
## §4 — Risk-Based Depth
|
|
117
|
+
|
|
118
|
+
| Risk tier | Signals (from §2, or explicit in AC) | Depth |
|
|
119
|
+
|---|---|---|
|
|
120
|
+
| **High** | payment, auth/authz, delete, financial calculation, subscription/billing, critical third-party integration, sensitive data | Each triggered dimension gets 2–3 tests; also add Security-oriented and Database checks even if not explicitly triggered by §2 |
|
|
121
|
+
| **Medium** | ordinary business feature, none of the High signals | Each triggered dimension gets 1–2 tests |
|
|
122
|
+
| **Low** | copy/label/minor UI-only change | Functional (positive + one negative) plus one regression spot-check — nothing else |
|
|
123
|
+
|
|
124
|
+
This tier decides **how many tests to draft per dimension**. It's separate from the
|
|
125
|
+
`risk-score` toolkit call, which orders the **execution sequence** of whatever was drafted, by
|
|
126
|
+
git churn + known-bug density. Run both — they answer different questions.
|
|
127
|
+
|
|
128
|
+
---
|
|
129
|
+
|
|
130
|
+
## §5 — Test Scenario Identification Framework
|
|
131
|
+
|
|
132
|
+
While turning triggered dimensions into numbered T-01, T-02... ideas, walk these 10 questions
|
|
133
|
+
once — they map straight onto §3's dimensions and stop scenario drafting from stalling out at
|
|
134
|
+
"just the AC's happy path":
|
|
135
|
+
|
|
136
|
+
1. What can the user do? → Functional Positive
|
|
137
|
+
2. What can the user enter? → Validation, Boundary/Equivalence
|
|
138
|
+
3. What can go wrong? → Functional Negative
|
|
139
|
+
4. What are the limits? → Boundary
|
|
140
|
+
5. Who can perform the action? → Authorization/Permissions
|
|
141
|
+
6. What happens to the data? → Database/API
|
|
142
|
+
7. What happens after the action? → End-to-End, Notifications
|
|
143
|
+
8. What happens when the system fails? → Functional Negative (dependency failure)
|
|
144
|
+
9. What existing functionality could break? → flag for qa-agent's post-phase regression
|
|
145
|
+
suggestion (Covered Flows overlap) — don't draft duplicate regression tests here
|
|
146
|
+
10. What unusual behavior could a real user perform? → at most one exploratory idea, tagged
|
|
147
|
+
clearly (`[exploratory]`) so it doesn't inflate the planned-test count
|
|
148
|
+
|
|
149
|
+
---
|
|
150
|
+
|
|
151
|
+
## §6 — Boundary & Equivalence Quick Templates
|
|
152
|
+
|
|
153
|
+
If the AC (or the answer to a §1 gap question) states a numeric/length range `[MIN, MAX]`:
|
|
154
|
+
|
|
155
|
+
- Full set (High-risk fields — payments, quotas, security limits): `MIN-1` (invalid), `MIN`
|
|
156
|
+
(valid), `MIN+1` (valid), `MAX-1` (valid), `MAX` (valid), `MAX+1` (invalid) — 6 tests.
|
|
157
|
+
- Collapsed set (Medium/Low-risk, wide range): `MIN` (valid), one mid-range value (valid),
|
|
158
|
+
`MAX` (valid), one just-out-of-range value (invalid) — 4 tests.
|
|
159
|
+
|
|
160
|
+
Equivalence partitioning (no exact boundary stated, just a described valid category): one
|
|
161
|
+
representative value below the valid range, one within it, one above it — 3 tests, not an
|
|
162
|
+
enumeration of every possible value.
|
|
163
|
+
|
|
164
|
+
---
|
|
165
|
+
|
|
166
|
+
## §7 — Defect Record Fields (Phase 4 payload)
|
|
167
|
+
|
|
168
|
+
Every bug filed from this skill should carry: Title, Environment, Preconditions, Steps to
|
|
169
|
+
Reproduce, Expected Result, Actual Result, Severity, Priority, Screenshot, Test data used, and
|
|
170
|
+
Build/version (the card ID + run date stands in when the product has no separate build
|
|
171
|
+
number). Severity/Priority derivation stays as defined in the skill's Phase 4 table — this
|
|
172
|
+
section only adds the two fields (Preconditions, Test data used) that weren't previously
|
|
173
|
+
captured.
|
|
174
|
+
|
|
175
|
+
---
|
|
176
|
+
|
|
177
|
+
## §8 — State Validation Checklist
|
|
178
|
+
|
|
179
|
+
For each key screen/component the card's test plan actually reaches, note which of these
|
|
180
|
+
states were observed and their result — skip states that are genuinely unreachable within the
|
|
181
|
+
test rather than forcing them:
|
|
182
|
+
|
|
183
|
+
Initial · Loading · Empty · Success · Error · Disabled
|
|
184
|
+
|
|
185
|
+
---
|
|
186
|
+
|
|
187
|
+
## §9 — Exit Criteria Quick Check
|
|
188
|
+
|
|
189
|
+
Before handing off to Phase 5 (test charter), confirm:
|
|
190
|
+
|
|
191
|
+
- Every planned test executed — `🔒 BLOCKED` is an acceptable outcome, a silently skipped test
|
|
192
|
+
is not.
|
|
193
|
+
- No open Critical/Blocker-severity bug among those just filed goes unmentioned — flag it
|
|
194
|
+
prominently in the final summary. This pack does not gate the pipeline on it (no release
|
|
195
|
+
authority here), it only makes sure the flag isn't buried.
|
|
196
|
+
- If the product context's `Known Bugs` table (already loaded in Phase 1) lists a bug on this
|
|
197
|
+
same flow with a status other than Closed/Done, one retest of that specific bug was included
|
|
198
|
+
in the plan.
|
|
199
|
+
- The regression suggestion in qa-agent Step 5 runs after this skill returns — nothing to do
|
|
200
|
+
here, just don't skip past it.
|