@roopesh.yadava/qa-pack 1.5.0 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +22 -0
- package/claude/skills/SKILLS_CONTEXT.md +23 -4
- package/claude/skills/accessibility-testing/SKILL.md +16 -0
- package/claude/skills/automation/SKILL.md +18 -0
- package/claude/skills/bug-reporting/SKILL.md +18 -1
- package/claude/skills/impacted-tests/SKILL.md +16 -1
- package/claude/skills/k6-framework-scaffold/SKILL.md +14 -0
- package/claude/skills/manual-testing/SKILL.md +116 -24
- package/claude/skills/manual-testing/TEST_DESIGN_GUIDE.md +200 -0
- package/claude/skills/mobile-automation/BDD_TEMPLATES.md +221 -0
- package/claude/skills/mobile-automation/LOCATOR_PATTERNS.md +205 -0
- package/claude/skills/mobile-automation/MOBILE_MCP_REFERENCE.md +234 -0
- package/claude/skills/mobile-automation/SKILL.md +611 -0
- package/claude/skills/qa-agent/SKILL.md +15 -0
- package/claude/skills/qa-agent/toolkit/qa-toolkit.cjs +192 -11
- package/claude/skills/roam-testing/SKILL.md +19 -0
- package/claude/skills/test-charter/SKILL.md +14 -0
- package/claude/skills/ui-test-figma/SKILL.md +15 -0
- package/package.json +1 -1
- package/templates/.env.example +13 -0
package/README.md
CHANGED
|
@@ -107,6 +107,28 @@ token beyond the one line of output. This is also why `qa-insights` (dashboard/d
|
|
|
107
107
|
across every product you've ever tested) costs about the same whether you have 2 products or
|
|
108
108
|
200 — the script does the aggregation, not the model.
|
|
109
109
|
|
|
110
|
+
### Team dashboard (optional, off by default)
|
|
111
|
+
|
|
112
|
+
`qa-insights`' dashboard is local and per-repo — it only sees runs on your own machine.
|
|
113
|
+
To see the whole team's runs across every product, set `QA_DASHBOARD_REPO` and
|
|
114
|
+
`QA_DASHBOARD_TOKEN` in `.env` (see `.env.example`). Once set, **every skill** reports its
|
|
115
|
+
own execution as its last step — one small JSON file pushed to that shared GitHub repo via
|
|
116
|
+
the GitHub Contents API. No local `git commit`/`git push`, so it never touches this repo's
|
|
117
|
+
`.git` or shows up in `git status`. Leave the variables unset and nothing changes; the
|
|
118
|
+
call is a silent no-op, same as the optional `QA_TRACKING_DIR` token tracking.
|
|
119
|
+
|
|
120
|
+
One record is one *skill execution*, not one card: a single card taken through `qa-agent` →
|
|
121
|
+
`manual-testing` → `bug-reporting` reports three. The dashboard counts both ("skill runs"
|
|
122
|
+
and "cards covered") so neither number is misleading.
|
|
123
|
+
|
|
124
|
+
What each record contains: product, card (when the skill has one), skill name, outcome, bug
|
|
125
|
+
IDs, reuse %, date, calling repo name, and an author label taken from `git config user.name`
|
|
126
|
+
— never `user.email`, since the dashboard repo is typically public and its history is
|
|
127
|
+
permanent. Set `QA_DASHBOARD_AUTHOR` in `.env` to override the label. A push that fails
|
|
128
|
+
(expired PAT, wrong repo, no network) stays silent and is appended to
|
|
129
|
+
`outputs/report-run-errors.log`; add `--verbose` to a manual `report-run` call to see the
|
|
130
|
+
result directly, which is the way to verify a freshly issued token.
|
|
131
|
+
|
|
110
132
|
## What postinstall does
|
|
111
133
|
|
|
112
134
|
| File | Behaviour |
|
|
@@ -50,6 +50,8 @@ A 10-test run with `browser_snapshot()` per test = ~100k wasted tokens.
|
|
|
50
50
|
| `BDD_TEMPLATES.md` | Once at Phase 1 start | Never reload in the same run |
|
|
51
51
|
| `LOCATOR_PATTERNS.md` | Once at Phase 1 start | Never reload in Phase 2 or 3 |
|
|
52
52
|
| `WCAG_CHECKS.md` | Once at skill start | Never reload |
|
|
53
|
+
| `manual-testing/TEST_DESIGN_GUIDE.md` | Once at Phase 1c (manual-testing) | Never reload — reference sections by number (§1–§9) |
|
|
54
|
+
| `mobile-automation/MOBILE_MCP_REFERENCE.md`, `BDD_TEMPLATES.md`, `LOCATOR_PATTERNS.md` | Once at Phase 2 start (mobile-automation) | Never reload in Phase 3 or 4 |
|
|
53
55
|
| `test-charter.md` | Once when charter starts | Never reload |
|
|
54
56
|
| `context.md` | Once at qa-agent Step 0 | Never reload |
|
|
55
57
|
|
|
@@ -86,14 +88,15 @@ node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs <command> [--flags]
|
|
|
86
88
|
| Command | Used by | Purpose |
|
|
87
89
|
|---------|---------|---------|
|
|
88
90
|
| `get-bugs` / `get-runs` / `get-selectors` --product P | any skill | Cheap reads of one context.md table, pipe-delimited rows |
|
|
89
|
-
| `fingerprint` --product P --url U --testids "a,b" | manual-testing, automation, roam-testing | DOM fingerprint cache — `UNCHANGED` means skip re-discovery |
|
|
91
|
+
| `fingerprint` --product P --url U --testids "a,b" | manual-testing, automation, mobile-automation, roam-testing | DOM/screen fingerprint cache — `UNCHANGED` means skip re-discovery. Mobile uses a synthetic `mobile:{bundleId}/{Screen}` string as `--url` and resource-id/accessibility-id values as `--testids` — the command itself is platform-agnostic |
|
|
90
92
|
| `pii-scan` (stdin or `--file`) | bug-reporting, manual-testing, roam-testing | Flags emails/keys/tokens before anything is posted to Jira |
|
|
91
93
|
| `dup-bug` --product P --summary S | bug-reporting, manual-testing, roam-testing | Fuzzy-matches a new bug against Known Bugs, no LLM comparison needed |
|
|
92
94
|
| `cost-estimate` --product P [--phase N] | qa-agent | One-line run-cost estimate from Runs Log history |
|
|
93
95
|
| `risk-score` --product P --modules "a,b" | manual-testing | Orders tests by git churn + past bug density, not AC order |
|
|
94
|
-
| `trust-record` / `trust-status` --product P | automation (
|
|
95
|
-
| `locator-record` / `locator-query` --product P --page U | automation | Self-improving locator memory across self-heal fixes |
|
|
96
|
-
| `dashboard` / `digest [--days N]` / `roi` | qa-insights | Cross-product reports — 100% script-generated, zero synthesis |
|
|
96
|
+
| `trust-record` / `trust-status` --product P [--namespace N] | automation, mobile-automation (`--namespace mobile` — routes to a separate `trust.mobile.json` so its gate streaks never enter qa-agent's web-only eligibility computation), qa-agent (status) | Trust ratchet on Gate 1/Gate 2 approvals. `--namespace` is optional and additive — omitted, behavior is unchanged (`trust.json`); `trust-status` takes `min(streak)` across every gate in whichever file it reads, so gates from two automation surfaces must never share one file |
|
|
97
|
+
| `locator-record` / `locator-query` --product P --page U | automation, mobile-automation (page keys prefixed `mobile:` to avoid colliding with web page URLs) | Self-improving locator memory across self-heal fixes |
|
|
98
|
+
| `dashboard` / `digest [--days N]` / `roi` | qa-insights | Cross-product reports — 100% script-generated, zero synthesis. Add `--json` to any of the three for a structured payload (same data, machine-readable) instead of the one-line summary — for anything scripting against this (e.g. an external dashboard app) rather than chatting with it. |
|
|
99
|
+
| `report-run` --skill S --outcome Y [--product P] [--card C] [--bugs "a,b"] [--reuse N] [--phase X] [--verbose] | **every skill, as its own last step** — `qa-agent`, `automation`, `manual-testing`, `mobile-automation`, `accessibility-testing`, `ui-test-figma`, `roam-testing`, `bug-reporting`, `impacted-tests`, `test-charter`, `k6-framework-scaffold` | Optional team dashboard: pushes one JSON file per **skill execution** to a shared GitHub repo (Contents API — no local git commit/push). Only `--skill`/`--outcome` are required: `--product` defaults to the calling repo's name and `--card` is omitted entirely by the skills that have no Jira card. Silent no-op unless `QA_DASHBOARD_REPO`/`QA_DASHBOARD_TOKEN` are set in `.env`, same convention as `QA_TRACKING_DIR`. A failed push never prints or fails the run — it is appended to `outputs/report-run-errors.log`; `--verbose` prints the result, which is how you verify a new PAT once. No `--notes` flag — only short, structurally-constrained values are safe to inline. |
|
|
97
100
|
|
|
98
101
|
This file is versioned logic (always overwritten on `npm update`, like every other skill
|
|
99
102
|
file) — never store product data inside it. Its outputs live under
|
|
@@ -101,6 +104,11 @@ file) — never store product data inside it. Its outputs live under
|
|
|
101
104
|
`locator-learnings.md` — all new, all gitignored the same way `context.md` already is) or
|
|
102
105
|
`outputs/` (`dashboard.html`, `qa-weekly-digest-*.md`, `roi-report.md`).
|
|
103
106
|
|
|
107
|
+
The toolkit reads `.env` from the repo root itself (real environment variables win), so
|
|
108
|
+
every `QA_*` value documented for `.env` reaches it without the calling skill exporting
|
|
109
|
+
anything. Before this, `QA_DASHBOARD_*` and `QA_TRACKING_DIR` were read straight off
|
|
110
|
+
`process.env` and were therefore permanently unset in normal skill use.
|
|
111
|
+
|
|
104
112
|
**Never inline arbitrary text into a toolkit shell call.** Bug descriptions, locator
|
|
105
113
|
strings, and anything else that isn't a short agent-controlled token (a card ID, a product
|
|
106
114
|
folder name, a URL) can contain quotes, `` ` ``, `$(...)`, or `|` — inlined into a bash
|
|
@@ -148,12 +156,18 @@ test-charter │
|
|
|
148
156
|
this diagram — they don't take a Jira card, and qa-insights doesn't touch Playwright at all.
|
|
149
157
|
Both are driven by the toolkit rather than by each other.
|
|
150
158
|
|
|
159
|
+
`mobile-automation` also sits outside this diagram — it's a standalone, directly-triggered
|
|
160
|
+
skill (same status as `roam-testing`/`k6-framework-scaffold`), not wired into qa-agent's
|
|
161
|
+
Phase 2 dispatch, which remains web/Playwright-only via `automation`. Trigger it by name
|
|
162
|
+
("automate mobile PROJ-123", "mobile test PROJ-123") rather than through the qa-agent phase menu.
|
|
163
|
+
|
|
151
164
|
## Skills — One-Line Summary
|
|
152
165
|
|
|
153
166
|
| Skill | Input | Output | MCP Needed |
|
|
154
167
|
|-------|-------|--------|------------|
|
|
155
168
|
| `qa-agent` | Jira card ID or menu choice | Dispatches to correct skill | Atlassian |
|
|
156
169
|
| `automation` | Jira card ID | Reuse audit + Gherkin + Step Defs + POM + real run (reuse % reported) | Atlassian, Playwright |
|
|
170
|
+
| `mobile-automation` | Jira card ID | Reuse audit + mobile Gherkin + Step Defs + Screen POM + real device run (reuse % reported) | Atlassian, Mobile MCP, MobileWright/mobilecli |
|
|
157
171
|
| `manual-testing` | Jira card ID + app URL | Execution report + bugs + charter | Atlassian, Playwright |
|
|
158
172
|
| `ui-test-figma` | Figma URL + app URL | UI mismatch report, Jira comment | Playwright (CLI+MCP), Figma (optional) |
|
|
159
173
|
| `accessibility-testing` | Full page URL + Jira card (optional) | WCAG 2.1 A/AA report + Jira bugs | Playwright (CLI+MCP), Atlassian |
|
|
@@ -244,6 +258,7 @@ python3 $PROJECT/track_tokens.py session
|
|
|
244
258
|
|
|
245
259
|
**Phase names by skill:**
|
|
246
260
|
- automation: `start` → `jira_fetch` → `reuse_audit` → `gherkin_generation` → `step_definitions` → `pom_generation` → `test_run` → `end`
|
|
261
|
+
- mobile-automation: `start` → `jira_fetch` → `reuse_audit` → `gherkin_generation` → `step_definitions` → `pom_generation` → `test_run` → `end`
|
|
247
262
|
- manual-testing: `start` → `jira_fetch` → `ui_testing` → `test_execution` → `end`
|
|
248
263
|
- ui-test-figma: `start` → `login` → `comparison` → `end`
|
|
249
264
|
- accessibility-testing: `start` → `login` → `a11y_checks` → `end`
|
|
@@ -280,10 +295,13 @@ First run will always be 0% (cold cache). Second and subsequent runs should cach
|
|
|
280
295
|
| Auth session | `.playwright-session.json` (gitignored) |
|
|
281
296
|
| Secrets / environment | `.env` at repo root (gitignored) |
|
|
282
297
|
| Step catalog (reuse audit) | `.claude/skills/qa-agent/product_context/[PRODUCT]/step-catalog.md` |
|
|
298
|
+
| Mobile step catalog (reuse audit) | `.claude/skills/qa-agent/product_context/[PRODUCT]/mobile-step-catalog.md` |
|
|
283
299
|
| Automation hints | `outputs/automation-hints-[CARD]-[date].md` |
|
|
284
300
|
| Token analytics | `~/.claude/token_analytics.png` |
|
|
285
301
|
| Knowledge graph | `graphify-out/graph.html` (open in browser) |
|
|
286
302
|
| Product QA context | `.claude/skills/qa-agent/product_context/[PRODUCT]/context.md` |
|
|
303
|
+
| Team QA dashboard data (shared, cross-repo, not in this repo) | `runs/[PRODUCT]/[date]-[card]-[skill]-[id].json` inside the repo named by `QA_DASHBOARD_REPO` — see `report-run` |
|
|
304
|
+
| Team dashboard push failures (local, gitignored) | `outputs/report-run-errors.log` — the only place a failed `report-run` is visible |
|
|
287
305
|
| DOM fingerprint cache | `.claude/skills/qa-agent/product_context/[PRODUCT]/dom-fingerprints.json` |
|
|
288
306
|
| Trust ratchet state | `.claude/skills/qa-agent/product_context/[PRODUCT]/trust.json` |
|
|
289
307
|
| Locator learnings | `.claude/skills/qa-agent/product_context/[PRODUCT]/locator-learnings.md` |
|
|
@@ -292,3 +310,4 @@ First run will always be 0% (cold cache). Second and subsequent runs should cach
|
|
|
292
310
|
| QA weekly digest | `outputs/qa-weekly-digest-[date].md` |
|
|
293
311
|
| Token-spend / ROI view | `outputs/roi-report.md` |
|
|
294
312
|
| Shared toolkit CLI | `.claude/skills/qa-agent/toolkit/qa-toolkit.cjs` |
|
|
313
|
+
| Manual-testing coverage/scenario reference | `.claude/skills/manual-testing/TEST_DESIGN_GUIDE.md` |
|
|
@@ -315,3 +315,19 @@ Accessibility Testing Complete
|
|
|
315
315
|
| Jira bug creation fails | Note failure, continue remaining bugs, report failures at end |
|
|
316
316
|
| OTP screen appears unexpectedly | Type the `QA_OTP_CODE` value from `.env`, proceed |
|
|
317
317
|
| Page requires further navigation after login | Follow redirect, confirm TARGET_URL loads |
|
|
318
|
+
|
|
319
|
+
---
|
|
320
|
+
|
|
321
|
+
## Final Step — Report to team dashboard (optional, silent)
|
|
322
|
+
|
|
323
|
+
Always the last action of the run, after everything else has printed. This is a no-op
|
|
324
|
+
unless `QA_DASHBOARD_REPO`/`QA_DASHBOARD_TOKEN` are set in `.env` — never print anything
|
|
325
|
+
about it, never retry, never treat a non-zero exit as a run failure.
|
|
326
|
+
|
|
327
|
+
```
|
|
328
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs report-run \
|
|
329
|
+
--skill accessibility-testing --card {JIRA_CARD_ID, omit if none} \
|
|
330
|
+
--outcome {Pass|Fail|Partial} --bugs "{comma-separated bug IDs, or omit if none}"
|
|
331
|
+
```
|
|
332
|
+
This skill has no `PRODUCT_FOLDER`; omit `--product` and the record is filed under the
|
|
333
|
+
calling repo's name automatically. Omit `--card` too when the run had no Jira card.
|
|
@@ -531,3 +531,21 @@ Print one line only (omit entirely if file not found):
|
|
|
531
531
|
```
|
|
532
532
|
Selectors written to product context ({N} elements, {N} flows).
|
|
533
533
|
```
|
|
534
|
+
|
|
535
|
+
---
|
|
536
|
+
|
|
537
|
+
## Final Step — Report to team dashboard (optional, silent)
|
|
538
|
+
|
|
539
|
+
Always the last action of the run, after everything else has printed. This is a no-op
|
|
540
|
+
unless `QA_DASHBOARD_REPO`/`QA_DASHBOARD_TOKEN` are set in `.env` — never print anything
|
|
541
|
+
about it, never retry, never treat a non-zero exit as a run failure.
|
|
542
|
+
|
|
543
|
+
```
|
|
544
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs report-run \
|
|
545
|
+
--skill automation --product {PRODUCT_FOLDER} --card {CARD_ID} \
|
|
546
|
+
--outcome {Pass|Fail|Partial} --reuse {reuse % number} --phase Automation
|
|
547
|
+
```
|
|
548
|
+
`report-run` takes only short, structurally-constrained values (product folder, card ID,
|
|
549
|
+
skill name, outcome, bug IDs). It has no `--notes` flag — never inline freeform text such as
|
|
550
|
+
a finding, a page title, or a bug description (see "Never inline arbitrary text" in
|
|
551
|
+
`SKILLS_CONTEXT.md`).
|
|
@@ -293,4 +293,21 @@ Show a summary:
|
|
|
293
293
|
- Notified: [Confirmed person's name]
|
|
294
294
|
- Screenshots attached: [list of filenames, or "none" / "credentials not set"]
|
|
295
295
|
|
|
296
|
-
---
|
|
296
|
+
---
|
|
297
|
+
|
|
298
|
+
## Final Step — Report to team dashboard (optional, silent)
|
|
299
|
+
|
|
300
|
+
Always the last action of the run, after everything else has printed. This is a no-op
|
|
301
|
+
unless `QA_DASHBOARD_REPO`/`QA_DASHBOARD_TOKEN` are set in `.env` — never print anything
|
|
302
|
+
about it, never retry, never treat a non-zero exit as a run failure.
|
|
303
|
+
|
|
304
|
+
```
|
|
305
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs report-run \
|
|
306
|
+
--skill bug-reporting --product {PRODUCT_FOLDER} --card {the card just filed/updated} \
|
|
307
|
+
--outcome {Pass if the bug posted, Fail if it did not} \
|
|
308
|
+
--bugs "{the bug key(s) filed}"
|
|
309
|
+
```
|
|
310
|
+
`report-run` takes only short, structurally-constrained values (product folder, card ID,
|
|
311
|
+
skill name, outcome, bug IDs). It has no `--notes` flag — never inline freeform text such as
|
|
312
|
+
a finding, a page title, or a bug description (see "Never inline arbitrary text" in
|
|
313
|
+
`SKILLS_CONTEXT.md`).
|
|
@@ -108,4 +108,19 @@ Rules for each entry:
|
|
|
108
108
|
|
|
109
109
|
## Step 5 — Stop
|
|
110
110
|
|
|
111
|
-
The report is the deliverable. Do NOT run any test, do NOT ask whether to run them — the QA reviews the report and runs files manually using the footer command. End the turn after printing the report.
|
|
111
|
+
The report is the deliverable. Do NOT run any test, do NOT ask whether to run them — the QA reviews the report and runs files manually using the footer command. End the turn after printing the report (the dashboard report below is the one exception — it prints nothing).
|
|
112
|
+
|
|
113
|
+
---
|
|
114
|
+
|
|
115
|
+
## Final Step — Report to team dashboard (optional, silent)
|
|
116
|
+
|
|
117
|
+
Run this after the report is printed and before ending the turn. This is a no-op
|
|
118
|
+
unless `QA_DASHBOARD_REPO`/`QA_DASHBOARD_TOKEN` are set in `.env` — never print anything
|
|
119
|
+
about it, never retry, never treat a non-zero exit as a run failure.
|
|
120
|
+
|
|
121
|
+
```
|
|
122
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs report-run \
|
|
123
|
+
--skill impacted-tests --outcome Pass
|
|
124
|
+
```
|
|
125
|
+
No product folder and no card in this skill — omit both flags; the record is filed under
|
|
126
|
+
the calling repo's name.
|
|
@@ -56,3 +56,17 @@ After it runs, tell the tester the next steps:
|
|
|
56
56
|
- Keep real tokens out of version control (the scaffold's `.gitignore` covers `secrets.local.*`).
|
|
57
57
|
- If the user wants the structure tailored (their user types, real endpoints from a Postman
|
|
58
58
|
collection), scaffold first, then edit the generated templates to match.
|
|
59
|
+
|
|
60
|
+
---
|
|
61
|
+
|
|
62
|
+
## Final Step — Report to team dashboard (optional, silent)
|
|
63
|
+
|
|
64
|
+
Always the last action of the run, after everything else has printed. This is a no-op
|
|
65
|
+
unless `QA_DASHBOARD_REPO`/`QA_DASHBOARD_TOKEN` are set in `.env` — never print anything
|
|
66
|
+
about it, never retry, never treat a non-zero exit as a run failure.
|
|
67
|
+
|
|
68
|
+
```
|
|
69
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs report-run \
|
|
70
|
+
--skill k6-framework-scaffold --outcome {Pass|Fail}
|
|
71
|
+
```
|
|
72
|
+
No product folder and no card — omit both flags.
|
|
@@ -2,8 +2,12 @@
|
|
|
2
2
|
name: manual-testing
|
|
3
3
|
description: >
|
|
4
4
|
Manual Testing branch orchestrator. Follows the flowchart:
|
|
5
|
-
Jira Card Input → UI Testing (Figma MCP, optional) →
|
|
6
|
-
→ Bug Reporting (Atlassian MCP) → Test Charter →
|
|
5
|
+
Jira Card Input → Requirement Gap Analysis → UI Testing (Figma MCP, optional) →
|
|
6
|
+
Manual Testing (Playwright MCP) → Bug Reporting (Atlassian MCP) → Test Charter →
|
|
7
|
+
Automation Agent (optional handoff).
|
|
8
|
+
Test planning is risk-scoped and coverage-dimension-driven (see TEST_DESIGN_GUIDE.md) —
|
|
9
|
+
not just "1-3 tests per AC line" — so a payment/auth card gets materially deeper coverage
|
|
10
|
+
than a copy-change card.
|
|
7
11
|
Captures element selectors and interaction data during execution and saves them to an
|
|
8
12
|
automation-hints file for the automation skill to reuse — skipping DOM re-discovery.
|
|
9
13
|
Uses Playwright CLI for zero-token screenshots. Token tracking enabled.
|
|
@@ -99,6 +103,13 @@ Derive `PRODUCT_FOLDER` now (uppercase `PROJECT_KEY`'s product name, spaces →
|
|
|
99
103
|
normalisation qa-agent Step 6a uses) so Phases 3 and 4 below can call the toolkit without
|
|
100
104
|
re-deriving it. If the qa-agent parameter block already named a product folder, use that instead.
|
|
101
105
|
|
|
106
|
+
If a product folder was resolved, pull known bugs cheaply for the retest check used in 3a/§9
|
|
107
|
+
— one script call, not a `context.md` read:
|
|
108
|
+
```bash
|
|
109
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs get-bugs --product {PRODUCT_FOLDER}
|
|
110
|
+
```
|
|
111
|
+
Store non-`Closed`/`Done` rows as `OPEN_KNOWN_BUGS`. `NO_CONTEXT_FILE`/`NO_BUGS` → `OPEN_KNOWN_BUGS = []`, first run for this product.
|
|
112
|
+
|
|
102
113
|
Run token tracking `jira_fetch` checkpoint.
|
|
103
114
|
|
|
104
115
|
### 1b — Truncate Jira data if card is verbose
|
|
@@ -112,14 +123,33 @@ After fetching, check `CARD_DESCRIPTION` length:
|
|
|
112
123
|
|
|
113
124
|
Do NOT tell the user the description was truncated. This prevents verbose cards from consuming 5k+ tokens before testing even starts.
|
|
114
125
|
|
|
115
|
-
If no AC found (and qa-agent didn't already resolve this):
|
|
116
|
-
> "No Acceptance Criteria found on this card. What should be tested?"
|
|
117
|
-
Wait for user response before continuing.
|
|
118
|
-
|
|
119
126
|
Set from Phase 0 values (all input was collected there — do not re-ask):
|
|
120
127
|
- `RUN_UI_TEST` — true when a Figma URL is available (from parameter block, card, or user)
|
|
121
128
|
- `FIGMA_URL`, `APP_URL`, `USERNAME`, `PASSWORD`, `OTP_CODE`
|
|
122
129
|
|
|
130
|
+
### 1c — Load Test Design Guide (once)
|
|
131
|
+
|
|
132
|
+
Read `.claude/skills/manual-testing/TEST_DESIGN_GUIDE.md` now. Per the Static File Load
|
|
133
|
+
Rules in `SKILLS_CONTEXT.md`, load it once here and never reload it later in this run —
|
|
134
|
+
reference its sections by number (e.g. "per §4 High risk") for the rest of the pipeline.
|
|
135
|
+
|
|
136
|
+
### 1d — Requirement Gap Analysis
|
|
137
|
+
|
|
138
|
+
Run the checklist in `TEST_DESIGN_GUIDE.md` §1 against `CARD_TITLE` + `ACCEPTANCE_CRITERIA`.
|
|
139
|
+
This **replaces** the old bare "no AC found" check — the missing-AC row is §1's first row and
|
|
140
|
+
is the most severe: if it fires, skip evaluating the rest of the table and ask only that.
|
|
141
|
+
|
|
142
|
+
Fold every gap that actually applies into **one** consolidated question — the same message as
|
|
143
|
+
any unresolved Phase 0 gaps, not a second round:
|
|
144
|
+
> "No Acceptance Criteria found on this card. What should be tested?" (only if AC is entirely
|
|
145
|
+
> missing), or, when AC exists but has gaps: one line per applicable §1 row, e.g. "AC mentions
|
|
146
|
+
> a 'reference code' but doesn't state its length/format — what's the exact rule?"
|
|
147
|
+
|
|
148
|
+
Skip this step's question entirely if the AC is already precise and complete on every §1
|
|
149
|
+
row — do not manufacture questions to fill the message.
|
|
150
|
+
|
|
151
|
+
Wait for the response before proceeding to Phase 2.
|
|
152
|
+
|
|
123
153
|
---
|
|
124
154
|
|
|
125
155
|
## Phase 2 — UI Testing (optional)
|
|
@@ -162,20 +192,44 @@ HINTS = {
|
|
|
162
192
|
}
|
|
163
193
|
```
|
|
164
194
|
|
|
165
|
-
### 3a — Generate Test Plan
|
|
195
|
+
### 3a — Generate Test Plan (risk-scoped, coverage-dimension-driven)
|
|
166
196
|
|
|
167
197
|
Before drafting test ideas, check each AC item for a stated entry point (menu path, URL,
|
|
168
198
|
button/link name). Per the **Test Data & Entity Selection Rules** above, do not scan `src/`
|
|
169
199
|
to infer a missing one — collect a single navigation question per missing entry point and
|
|
170
200
|
fold it into the confirmation prompt below instead of guessing.
|
|
171
201
|
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
202
|
+
**Step 1 — Classify risk tier and select dimensions.** Using `TEST_DESIGN_GUIDE.md` §2 (Feature
|
|
203
|
+
Signal → Coverage Dimension Trigger Table), scan `CARD_TITLE` + `ACCEPTANCE_CRITERIA` for
|
|
204
|
+
signal keywords and turn on the matched dimensions. No signal matched → default set only
|
|
205
|
+
(Functional Positive/Negative + UI/UX States + one Boundary check). Then classify the card's
|
|
206
|
+
risk tier per §4 (High/Medium/Low) from the same signals — this sets how many tests each
|
|
207
|
+
triggered dimension gets, not just which dimensions run.
|
|
208
|
+
|
|
209
|
+
**Step 2 — Draft numbered test ideas per dimension.** Walk the §5 Test Scenario Identification
|
|
210
|
+
Framework (the 10 "what can/can't/who/what happens" questions) once, and for each dimension
|
|
211
|
+
turned on in Step 1, draft the number of tests §4's tier calls for. Concretely, for a Medium-risk
|
|
212
|
+
card this typically looks like: 1–2 Functional Positive, 1–2 Functional Negative, one Boundary
|
|
213
|
+
or Equivalence set per §6 if the AC states or implies a limit, one UI/UX States pass (§8) on the
|
|
214
|
+
card's key screen, plus one test per other triggered dimension (Permissions, API, Database,
|
|
215
|
+
State/Workflow, etc.). High-risk cards double the per-dimension count and add Security-oriented
|
|
216
|
+
+ Database checks unconditionally per §4. Low-risk cards (copy/label-only) skip straight to
|
|
217
|
+
Functional (positive + one negative) plus one regression spot-check — do not force the other
|
|
218
|
+
dimensions onto a low-risk card.
|
|
219
|
+
|
|
220
|
+
Cap exploratory ideas (§5 Q10) at one per plan, clearly tagged `[exploratory]`, so they don't
|
|
221
|
+
inflate the "planned" test count used for the Exit Criteria check later (§9).
|
|
222
|
+
|
|
223
|
+
If `OPEN_KNOWN_BUGS` (from Phase 1) contains a row whose Title overlaps this card's module/page
|
|
224
|
+
keywords, add one retest test tagged `[retest: {BUG-ID}]` — this is what §9's retest check
|
|
225
|
+
verifies actually happened.
|
|
226
|
+
|
|
227
|
+
Do not draft dedicated regression tests here — that's qa-agent Step 5's job (Covered Flows
|
|
228
|
+
overlap), triggered automatically after this skill returns. §5 Q9 exists only to flag *that*
|
|
229
|
+
regression may apply, not to author it now.
|
|
230
|
+
|
|
231
|
+
**Risk-based execution ordering** — pull 2–5 module/page keywords straight out of the AC/card
|
|
232
|
+
title you already read (e.g. "login", "checkout"; no extra fetching), then run one script call:
|
|
179
233
|
|
|
180
234
|
```bash
|
|
181
235
|
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs risk-score --product {PRODUCT_FOLDER} --modules "kw1,kw2,kw3"
|
|
@@ -183,11 +237,13 @@ node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs risk-score --product {PRODUC
|
|
|
183
237
|
|
|
184
238
|
Order the T-01, T-02... list so tests touching the highest-risk module come first. On a
|
|
185
239
|
product's first run (no git history / no known bugs yet) every module scores 0 — keep the
|
|
186
|
-
natural
|
|
240
|
+
natural drafting order in that case, no need to mention it. (This reorders the tests drafted in
|
|
241
|
+
Step 2 — it doesn't decide how many exist; that's §4's job, run once, not repeated per module.)
|
|
187
242
|
|
|
188
|
-
Show as a compact table (T-01, T-02 ... with name and expected outcome),
|
|
189
|
-
Ask: `"Ready to run these tests? (yes / no or edit)"` — include any missing
|
|
190
|
-
questions in this same message
|
|
243
|
+
Show as a compact table (T-01, T-02 ... with name, dimension tag, and expected outcome),
|
|
244
|
+
risk-ordered. Ask: `"Ready to run these tests? (yes / no or edit)"` — include any missing
|
|
245
|
+
navigation questions from above in this same message (§1 gap questions were already asked
|
|
246
|
+
and resolved in Phase 1d — don't re-ask them here).
|
|
191
247
|
Proceed on confirmation.
|
|
192
248
|
|
|
193
249
|
### 3b — Setup Browser + Capture Login Selectors
|
|
@@ -305,10 +361,13 @@ For each test T-01, T-02, ...:
|
|
|
305
361
|
outputs/screenshots/T-[N]-[pass|fail|observation].png
|
|
306
362
|
```
|
|
307
363
|
|
|
308
|
-
5. Assert expected result via targeted `browser_evaluate` on specific selectors
|
|
364
|
+
5. Assert expected result via targeted `browser_evaluate` on specific selectors. If this
|
|
365
|
+
test's dimension tag (from 3a) is UI/UX States, note which of the `TEST_DESIGN_GUIDE.md`
|
|
366
|
+
§8 states (Initial/Loading/Empty/Success/Error/Disabled) were actually observed and their
|
|
367
|
+
result — skip states genuinely unreachable in this test, don't force them.
|
|
309
368
|
|
|
310
369
|
6. Log test result (to file only, not chat):
|
|
311
|
-
`T-N | title | status | expected | actual | screenshot path`
|
|
370
|
+
`T-N | title | dimension | status | expected | actual | screenshot path`
|
|
312
371
|
|
|
313
372
|
7. Append to `HINTS.testCases`:
|
|
314
373
|
```
|
|
@@ -407,13 +466,19 @@ this keeps context contiguous and avoids a second skill invocation.
|
|
|
407
466
|
|
|
408
467
|
**Step 4a — Build bug payload for each ❌ FAIL and ⚠️ OBSERVATION**
|
|
409
468
|
|
|
410
|
-
Derive all fields from test execution data — no user input needed
|
|
469
|
+
Derive all fields from test execution data — no user input needed. Fields follow
|
|
470
|
+
`TEST_DESIGN_GUIDE.md` §7:
|
|
411
471
|
|
|
412
472
|
| Field | Source |
|
|
413
473
|
|-------|--------|
|
|
414
474
|
| Summary | `"T-{N}: {test name} — {actual outcome in one line}"` |
|
|
415
|
-
|
|
|
475
|
+
| Environment | `APP_URL` + environment name from product context (staging/dev/prod) |
|
|
476
|
+
| Preconditions | Login state + any setup steps from the test case (e.g. role, prior state) |
|
|
477
|
+
| Description (Steps/Expected/Actual) | Steps from test case + expected vs actual from execution log |
|
|
416
478
|
| Severity | AC explicitly failed → High · Assertion failed → Medium · Observation → Low |
|
|
479
|
+
| Priority | High-risk dimension (§4) failing → High · otherwise matches Severity |
|
|
480
|
+
| Test data used | The specific values/entities used for this test (never credentials) |
|
|
481
|
+
| Build/version | `CARD_ID` + run date (stands in for a separate build number) |
|
|
417
482
|
| Screenshot | Match `outputs/screenshots/T-{N}-*.png` by test ID — use exact filename |
|
|
418
483
|
|
|
419
484
|
**Step 4a.1 — Duplicate check (one script call per failure, no user input needed)**
|
|
@@ -496,14 +561,22 @@ Run token tracking `end + report + session` close-out after charter publishes.
|
|
|
496
561
|
|
|
497
562
|
## Phase 6 — Automation Handoff
|
|
498
563
|
|
|
499
|
-
After charter completes,
|
|
564
|
+
After charter completes, run the `TEST_DESIGN_GUIDE.md` §9 Exit Criteria Quick Check:
|
|
565
|
+
confirm every planned test executed (BLOCKED is fine, silently-skipped is not), check whether
|
|
566
|
+
any bug just filed is High severity on a High-risk dimension (§4) and if so flag it as the
|
|
567
|
+
first line of the summary below, and — only if `OPEN_KNOWN_BUGS` (Phase 1) was non-empty and
|
|
568
|
+
overlapped this card — confirm the `[retest: {BUG-ID}]` test from 3a actually ran and note its
|
|
569
|
+
outcome. This does not block or gate anything — it only makes sure nothing gets buried in the
|
|
570
|
+
summary.
|
|
571
|
+
|
|
572
|
+
Show the final summary:
|
|
500
573
|
|
|
501
574
|
```
|
|
502
575
|
Manual Testing Complete — [CARD_ID]
|
|
503
576
|
|
|
504
577
|
UI Testing (Figma) : [completed / skipped]
|
|
505
578
|
Manual Testing : X Pass | X Fail | X Observation | X Blocked
|
|
506
|
-
Bug Reporting : [N] bug(s) filed — [keys]
|
|
579
|
+
Bug Reporting : [N] bug(s) filed — [keys] [+ "⚠ High-severity on {dimension}" if applicable]
|
|
507
580
|
Test Charter : [published URL or "saved locally"]
|
|
508
581
|
Automation hints : outputs/automation-hints-[CARD_ID]-[YYYYMMDD].md
|
|
509
582
|
```
|
|
@@ -554,3 +627,22 @@ Do NOT stream: full charter text, all test results (save to file), full bug repo
|
|
|
554
627
|
| Figma URL missing / user says no | Skip UI Testing, note in final summary |
|
|
555
628
|
| `browser_evaluate` returns null for element | Log "element not captured" in hints, continue |
|
|
556
629
|
| Bug creation fails | Note failure, continue with remaining bugs, report at end |
|
|
630
|
+
|
|
631
|
+
---
|
|
632
|
+
|
|
633
|
+
## Final Step — Report to team dashboard (optional, silent)
|
|
634
|
+
|
|
635
|
+
Always the last action of the run, after everything else has printed. This is a no-op
|
|
636
|
+
unless `QA_DASHBOARD_REPO`/`QA_DASHBOARD_TOKEN` are set in `.env` — never print anything
|
|
637
|
+
about it, never retry, never treat a non-zero exit as a run failure.
|
|
638
|
+
|
|
639
|
+
```
|
|
640
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs report-run \
|
|
641
|
+
--skill manual-testing --product {PRODUCT_FOLDER} --card {CARD_ID} \
|
|
642
|
+
--outcome {Pass|Fail|Partial} --bugs "{comma-separated bug IDs, or omit if none}" \
|
|
643
|
+
--phase Manual
|
|
644
|
+
```
|
|
645
|
+
`report-run` takes only short, structurally-constrained values (product folder, card ID,
|
|
646
|
+
skill name, outcome, bug IDs). It has no `--notes` flag — never inline freeform text such as
|
|
647
|
+
a finding, a page title, or a bug description (see "Never inline arbitrary text" in
|
|
648
|
+
`SKILLS_CONTEXT.md`).
|