@roopesh.yadava/qa-pack 1.4.0 → 1.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +15 -1
- package/claude/skills/SKILLS_CONTEXT.md +61 -5
- package/claude/skills/automation/SKILL.md +53 -6
- package/claude/skills/bug-reporting/SKILL.md +38 -0
- package/claude/skills/manual-testing/SKILL.md +67 -2
- package/claude/skills/qa-agent/SKILL.md +30 -0
- package/claude/skills/qa-agent/product_context/README.md +16 -0
- package/claude/skills/qa-agent/toolkit/qa-toolkit.cjs +705 -0
- package/claude/skills/qa-insights/SKILL.md +55 -0
- package/claude/skills/roam-testing/SKILL.md +187 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -92,12 +92,26 @@ and `/write-acceptance-criteria`, which are also available as explicit slash com
|
|
|
92
92
|
| `/write-acceptance-criteria PROJ-123` | write-acceptance-criteria | Generates AC, appends to the Jira card description |
|
|
93
93
|
| `/impacted-tests` / `which tests are impacted by this pull` | impacted-tests | After pulling dev changes into a test branch, reports which Cucumber feature files are at risk — report-only, no card needed |
|
|
94
94
|
| `set up k6` / `scaffold performance tests` | k6-framework-scaffold | Scaffolds a `k6-performance-tests/` framework (Grafana Cloud, protocol + optional browser layers) with commented templates to fill in — no card needed |
|
|
95
|
+
| `roam mode` / `explore the app` | roam-testing | Card-free exploratory testing — capped breadth-first crawl, report + optional bug filing + charter |
|
|
96
|
+
| `qa dashboard` / `weekly digest` / `token roi` | qa-insights | Cross-product health dashboard, weekly digest, or token-spend/ROI view — 100% script-generated, no card needed |
|
|
97
|
+
|
|
98
|
+
## The toolkit — why these features stay cheap
|
|
99
|
+
|
|
100
|
+
Several features above (self-improving locator memory, DOM fingerprint caching, duplicate-bug
|
|
101
|
+
detection, PII/secrets scanning, risk-based test ordering, the trust ratchet, and every
|
|
102
|
+
`qa-insights` report) are powered by one dependency-free script:
|
|
103
|
+
`.claude/skills/qa-agent/toolkit/qa-toolkit.cjs`. Skills shell out to it and read back a
|
|
104
|
+
single line — parsing markdown tables, hashing DOM snapshots, scoring bug-title similarity,
|
|
105
|
+
and aggregating every product's history are pure computation, so none of it costs a model
|
|
106
|
+
token beyond the one line of output. This is also why `qa-insights` (dashboard/digest/ROI
|
|
107
|
+
across every product you've ever tested) costs about the same whether you have 2 products or
|
|
108
|
+
200 — the script does the aggregation, not the model.
|
|
95
109
|
|
|
96
110
|
## What postinstall does
|
|
97
111
|
|
|
98
112
|
| File | Behaviour |
|
|
99
113
|
|---|---|
|
|
100
|
-
| `.claude/skills/*/SKILL.md` + companion `.md` files | Always overwritten (versioned logic) |
|
|
114
|
+
| `.claude/skills/*/SKILL.md` + companion `.md`/`.cjs`/`.sh` files (e.g. the toolkit script) | Always overwritten (versioned logic) |
|
|
101
115
|
| `.claude/skills/SKILLS_CONTEXT.md` | Always overwritten + stamped with the installed pack version |
|
|
102
116
|
| `.claude/commands/*.md` | Always overwritten |
|
|
103
117
|
| `.claude/skills/qa-agent/product_context/**` | **Never touched** after first seed |
|
|
@@ -70,6 +70,45 @@ Structure every skill invocation in this order so the stable prefix can be cache
|
|
|
70
70
|
The stable block qualifies for Anthropic prompt caching when it exceeds 1024 tokens.
|
|
71
71
|
Cache TTL is 5 minutes. Keep the stable block identical across runs for the same product.
|
|
72
72
|
|
|
73
|
+
### The Toolkit — the low-token path for anything product_context-shaped
|
|
74
|
+
|
|
75
|
+
`.claude/skills/qa-agent/toolkit/qa-toolkit.cjs` is a dependency-free Node script, not a
|
|
76
|
+
skill. It exists because parsing a markdown table, hashing a DOM snapshot, scoring string
|
|
77
|
+
similarity, or aggregating dozens of products' history is pure computation — routing it
|
|
78
|
+
through the model (Read the file → reason over it → maybe Edit it back) burns tokens on work
|
|
79
|
+
that doesn't need a model at all. **Any skill needing one of the things below calls the
|
|
80
|
+
toolkit and reads only its one-line result — never Read a whole context.md for this.**
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs <command> [--flags]
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
| Command | Used by | Purpose |
|
|
87
|
+
|---------|---------|---------|
|
|
88
|
+
| `get-bugs` / `get-runs` / `get-selectors` --product P | any skill | Cheap reads of one context.md table, pipe-delimited rows |
|
|
89
|
+
| `fingerprint` --product P --url U --testids "a,b" | manual-testing, automation, roam-testing | DOM fingerprint cache — `UNCHANGED` means skip re-discovery |
|
|
90
|
+
| `pii-scan` (stdin or `--file`) | bug-reporting, manual-testing, roam-testing | Flags emails/keys/tokens before anything is posted to Jira |
|
|
91
|
+
| `dup-bug` --product P --summary S | bug-reporting, manual-testing, roam-testing | Fuzzy-matches a new bug against Known Bugs, no LLM comparison needed |
|
|
92
|
+
| `cost-estimate` --product P [--phase N] | qa-agent | One-line run-cost estimate from Runs Log history |
|
|
93
|
+
| `risk-score` --product P --modules "a,b" | manual-testing | Orders tests by git churn + past bug density, not AC order |
|
|
94
|
+
| `trust-record` / `trust-status` --product P | automation (record), qa-agent (status) | Trust ratchet on Gate 1/Gate 2 approvals |
|
|
95
|
+
| `locator-record` / `locator-query` --product P --page U | automation | Self-improving locator memory across self-heal fixes |
|
|
96
|
+
| `dashboard` / `digest [--days N]` / `roi` | qa-insights | Cross-product reports — 100% script-generated, zero synthesis |
|
|
97
|
+
|
|
98
|
+
This file is versioned logic (always overwritten on `npm update`, like every other skill
|
|
99
|
+
file) — never store product data inside it. Its outputs live under
|
|
100
|
+
`product_context/{PRODUCT}/` (`dom-fingerprints.json`, `trust.json`,
|
|
101
|
+
`locator-learnings.md` — all new, all gitignored the same way `context.md` already is) or
|
|
102
|
+
`outputs/` (`dashboard.html`, `qa-weekly-digest-*.md`, `roi-report.md`).
|
|
103
|
+
|
|
104
|
+
**Never inline arbitrary text into a toolkit shell call.** Bug descriptions, locator
|
|
105
|
+
strings, and anything else that isn't a short agent-controlled token (a card ID, a product
|
|
106
|
+
folder name, a URL) can contain quotes, `` ` ``, `$(...)`, or `|` — inlined into a bash
|
|
107
|
+
argument, that is a command-injection bug, not just an escaping nuisance. Write the text to
|
|
108
|
+
a file with the Write tool first and pass the path: `pii-scan --file`, `dup-bug
|
|
109
|
+
--summary-file`, `locator-record --file <json>`. Only short, structurally-constrained values
|
|
110
|
+
(URLs, card IDs, product folder names) are safe to inline directly.
|
|
111
|
+
|
|
73
112
|
---
|
|
74
113
|
|
|
75
114
|
## Pipeline Overview
|
|
@@ -105,6 +144,10 @@ test-charter │
|
|
|
105
144
|
End-to-End Testing complete
|
|
106
145
|
```
|
|
107
146
|
|
|
147
|
+
`roam-testing` (card-free exploratory) and `qa-insights` (dashboard/digest/roi) sit outside
|
|
148
|
+
this diagram — they don't take a Jira card, and qa-insights doesn't touch Playwright at all.
|
|
149
|
+
Both are driven by the toolkit rather than by each other.
|
|
150
|
+
|
|
108
151
|
## Skills — One-Line Summary
|
|
109
152
|
|
|
110
153
|
| Skill | Input | Output | MCP Needed |
|
|
@@ -116,14 +159,17 @@ test-charter │
|
|
|
116
159
|
| `accessibility-testing` | Full page URL + Jira card (optional) | WCAG 2.1 A/AA report + Jira bugs | Playwright (CLI+MCP), Atlassian |
|
|
117
160
|
| `bug-reporting` | Bug description | Bug filed on Jira card | Atlassian |
|
|
118
161
|
| `test-charter` | Execution report MD file | Charter MD + published to API | Playwright (login) |
|
|
162
|
+
| `roam-testing` | App URL, no card required | Roam report + optional bugs + charter | Playwright, Atlassian (optional) |
|
|
163
|
+
| `qa-insights` | Nothing (reads all products) | Dashboard HTML / digest MD / ROI MD | None — pure toolkit |
|
|
119
164
|
|
|
120
165
|
## God Nodes (highest connectivity — touch these carefully)
|
|
121
166
|
|
|
122
|
-
1. `automation` —
|
|
123
|
-
2. `manual-testing` —
|
|
124
|
-
3. `qa-
|
|
125
|
-
4. `
|
|
126
|
-
5. `
|
|
167
|
+
1. `automation` — 15 edges (Gherkin→StepDefs→POM chain, token tracking, Playwright MCP, locator memory, trust ratchet)
|
|
168
|
+
2. `manual-testing` — 15 edges (orchestrates ui-test-figma, bug-reporting, test-charter, risk-score, fingerprint, dup-bug)
|
|
169
|
+
3. `qa-toolkit.cjs` — 6 skills call into it (qa-agent, automation, manual-testing, bug-reporting, roam-testing, qa-insights) — not a skill itself, but the single highest-fan-in file in the pack
|
|
170
|
+
4. `qa-agent` — 9 edges (dispatches all paths, owns the Phase 3 pipeline, cost estimate + trust status)
|
|
171
|
+
5. `test-charter` — 7 edges (reads execution report, publishes to Decision Record API; also used by roam-testing)
|
|
172
|
+
6. `ui-test-figma` — 6 edges (Figma MCP preferred, Playwright CLI fallback)
|
|
127
173
|
|
|
128
174
|
## Dispatch Map (qa-agent routes)
|
|
129
175
|
|
|
@@ -135,6 +181,8 @@ Phase 3 / "full QA" → manual-testing THEN automation (hints file reused
|
|
|
135
181
|
"accessibility test" → accessibility-testing (standalone, URL + optional Jira card)
|
|
136
182
|
"file bug" → bug-reporting (standalone)
|
|
137
183
|
"charter" → test-charter (standalone)
|
|
184
|
+
"roam mode" / "explore" → roam-testing (standalone, no card — routed outside qa-agent)
|
|
185
|
+
"dashboard"/"digest"/"roi" → qa-insights (standalone, no card — routed outside qa-agent)
|
|
138
186
|
```
|
|
139
187
|
|
|
140
188
|
**Input collection happens ONCE in qa-agent.** Sub-skills receive their parameters in the
|
|
@@ -236,3 +284,11 @@ First run will always be 0% (cold cache). Second and subsequent runs should cach
|
|
|
236
284
|
| Token analytics | `~/.claude/token_analytics.png` |
|
|
237
285
|
| Knowledge graph | `graphify-out/graph.html` (open in browser) |
|
|
238
286
|
| Product QA context | `.claude/skills/qa-agent/product_context/[PRODUCT]/context.md` |
|
|
287
|
+
| DOM fingerprint cache | `.claude/skills/qa-agent/product_context/[PRODUCT]/dom-fingerprints.json` |
|
|
288
|
+
| Trust ratchet state | `.claude/skills/qa-agent/product_context/[PRODUCT]/trust.json` |
|
|
289
|
+
| Locator learnings | `.claude/skills/qa-agent/product_context/[PRODUCT]/locator-learnings.md` |
|
|
290
|
+
| Roam session report | `outputs/roam-report-[timestamp].md` |
|
|
291
|
+
| Product health dashboard | `outputs/dashboard.html` |
|
|
292
|
+
| QA weekly digest | `outputs/qa-weekly-digest-[date].md` |
|
|
293
|
+
| Token-spend / ROI view | `outputs/roi-report.md` |
|
|
294
|
+
| Shared toolkit CLI | `.claude/skills/qa-agent/toolkit/qa-toolkit.cjs` |
|
|
@@ -269,14 +269,21 @@ Present Gherkin + reuse report and ask:
|
|
|
269
269
|
>
|
|
270
270
|
> **Type "looks good" or "confirmed" to proceed to step definitions.**
|
|
271
271
|
|
|
272
|
-
Do not write any `.cjs` files until confirmed. Iterate until approved.
|
|
272
|
+
Do not write any `.cjs` files until confirmed. Iterate until approved. Once approved, record
|
|
273
|
+
the trust ratchet — `clean` if the user approved on first presentation with no revision
|
|
274
|
+
requests this gate, `edited` if they asked for any change before approving:
|
|
275
|
+
|
|
276
|
+
```bash
|
|
277
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs trust-record --product {PRODUCT_FOLDER} --gate gate1 --result clean|edited
|
|
278
|
+
```
|
|
273
279
|
|
|
274
280
|
**If AUTO_APPROVE = true (called from qa-agent full pipeline):**
|
|
275
281
|
|
|
276
282
|
Display the reuse report + a compact summary — do NOT wait:
|
|
277
283
|
> "Gherkin generated for [CARD-ID]: {N} Rules, {N} scenarios, {X}% step reuse. Auto-approved — proceeding to step definitions."
|
|
278
284
|
|
|
279
|
-
Immediately move to Phase 2 without waiting for any input.
|
|
285
|
+
Immediately move to Phase 2 without waiting for any input. Do not call `trust-record` here —
|
|
286
|
+
there was no human review to score.
|
|
280
287
|
|
|
281
288
|
After either path: run `gherkin_generation` token checkpoint, then move to Phase 2.
|
|
282
289
|
|
|
@@ -329,14 +336,19 @@ Present step definitions and ask:
|
|
|
329
336
|
>
|
|
330
337
|
> **Type "looks good" or "confirmed" to proceed to the POM.**
|
|
331
338
|
|
|
332
|
-
Do not write the POM until confirmed.
|
|
339
|
+
Do not write the POM until confirmed. Once approved, record the trust ratchet the same way
|
|
340
|
+
as Gate 1:
|
|
341
|
+
|
|
342
|
+
```bash
|
|
343
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs trust-record --product {PRODUCT_FOLDER} --gate gate2 --result clean|edited
|
|
344
|
+
```
|
|
333
345
|
|
|
334
346
|
**If AUTO_APPROVE = true (called from qa-agent full pipeline):**
|
|
335
347
|
|
|
336
348
|
Display a compact summary only — do NOT wait:
|
|
337
349
|
> "Step definitions generated for [CARD-ID]: {N} steps across {N} files. Auto-approved — proceeding to POM."
|
|
338
350
|
|
|
339
|
-
Immediately move to Phase 3 without waiting for any input.
|
|
351
|
+
Immediately move to Phase 3 without waiting for any input. Do not call `trust-record` here.
|
|
340
352
|
|
|
341
353
|
After either path: run `step_definitions` token checkpoint, then move to Phase 3.
|
|
342
354
|
|
|
@@ -356,7 +368,31 @@ class. Only create a new POM class for a page that has none.
|
|
|
356
368
|
Use the POM class template from BDD_TEMPLATES.md for new classes; match the existing
|
|
357
369
|
class's style when extending.
|
|
358
370
|
|
|
359
|
-
|
|
371
|
+
**Locator memory check — once per page, before opening Playwright MCP for it:**
|
|
372
|
+
```bash
|
|
373
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs locator-query --product {PRODUCT_FOLDER} --page "PAGE_URL"
|
|
374
|
+
```
|
|
375
|
+
`NONE` → nothing learned yet for this page, proceed as usual. Rows returned → for each
|
|
376
|
+
listed element, avoid its recorded **Failed Locator** even if it looks like the obvious
|
|
377
|
+
choice; that pattern already broke a real test on this page. Prefer its recorded **Working
|
|
378
|
+
Locator** if the element matches.
|
|
379
|
+
|
|
380
|
+
**DOM fingerprint check — once per distinct page URL this run:**
|
|
381
|
+
```javascript
|
|
382
|
+
browser_evaluate({ expression: `
|
|
383
|
+
Array.from(document.querySelectorAll('[data-testid]')).map(el => el.getAttribute('data-testid')).join(',')
|
|
384
|
+
` })
|
|
385
|
+
```
|
|
386
|
+
```bash
|
|
387
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs fingerprint --product {PRODUCT_FOLDER} --url "PAGE_URL" --testids "RESULT_FROM_ABOVE"
|
|
388
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs get-selectors --product {PRODUCT_FOLDER} --url "PAGE_URL"
|
|
389
|
+
```
|
|
390
|
+
If fingerprint says `UNCHANGED` and `get-selectors` returns rows covering the elements this
|
|
391
|
+
Rule needs, write the POM locators straight from those rows — skip the per-element DOM
|
|
392
|
+
inspection below entirely for this page. Otherwise (`NEW`/`CHANGED`, or a needed element is
|
|
393
|
+
missing from `get-selectors`) fall through to the full inspection:
|
|
394
|
+
|
|
395
|
+
For every locator not already resolved above:
|
|
360
396
|
1. Navigate to the real page in the live app via Playwright MCP
|
|
361
397
|
2. Inspect the target element in the DOM
|
|
362
398
|
3. Check if `data-testid` already exists
|
|
@@ -420,7 +456,18 @@ On failure, diagnose and fix — **never hand off a silently failing test**:
|
|
|
420
456
|
|
|
421
457
|
1. Read the failure: locator timeout? assertion mismatch? navigation/auth issue?
|
|
422
458
|
2. Locator failures → re-inspect that element via Playwright MCP, fix the POM locator
|
|
423
|
-
(respect the locator priority table).
|
|
459
|
+
(respect the locator priority table). Once the fix passes, record it so no future run on
|
|
460
|
+
this product repeats the same wrong guess. Locator strings routinely contain quotes and
|
|
461
|
+
`|` (xpath unions) — never inline them into a shell command; write a small JSON file
|
|
462
|
+
instead (Write tool) and pass its path:
|
|
463
|
+
```
|
|
464
|
+
Write outputs/.locator-tmp.json containing:
|
|
465
|
+
{"product": "{PRODUCT_FOLDER}", "page": "PAGE_URL", "element": "ELEMENT_LABEL",
|
|
466
|
+
"failed": "OLD_LOCATOR", "fixed": "NEW_LOCATOR", "reason": "ONE_LINE_WHY", "card": "{CARD_ID}"}
|
|
467
|
+
```
|
|
468
|
+
```bash
|
|
469
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs locator-record --file outputs/.locator-tmp.json
|
|
470
|
+
```
|
|
424
471
|
3. Assertion failures → check whether expected text/behavior on the card matches the app.
|
|
425
472
|
If the app appears genuinely wrong, this is a **bug, not a test fix** — stop healing
|
|
426
473
|
that scenario, mark it failing, and note it as a bug candidate in the hand-off.
|
|
@@ -27,6 +27,12 @@ Display the list and ask:
|
|
|
27
27
|
|
|
28
28
|
Wait for the user to select a project by number or board key. Save the selected project key for the rest of the flow.
|
|
29
29
|
|
|
30
|
+
**Resolve `PRODUCT_FOLDER` (best-effort — only used by the duplicate check in Step 2a):**
|
|
31
|
+
check whether `.claude/skills/qa-agent/product_context/{KEY}/context.md` exists, or any
|
|
32
|
+
folder under `product_context/` starts with `{KEY}`. Found → store as `PRODUCT_FOLDER`.
|
|
33
|
+
Not found → leave `PRODUCT_FOLDER` unset and skip the duplicate half of Step 2a silently;
|
|
34
|
+
this is not an error, most exploratory-bug filing has no prior product context yet.
|
|
35
|
+
|
|
30
36
|
---
|
|
31
37
|
|
|
32
38
|
## Step 1A/1B — Ask card type
|
|
@@ -110,6 +116,38 @@ Do NOT ask the user for any missing fields. Use what was given and proceed.
|
|
|
110
116
|
|
|
111
117
|
---
|
|
112
118
|
|
|
119
|
+
## Step 2a — Duplicate + PII/secrets check (script calls, no extra questions unless flagged)
|
|
120
|
+
|
|
121
|
+
The user typed this text freely — it can contain quotes, `$`, backticks, anything. **Never**
|
|
122
|
+
interpolate it directly into a shell command; write it to a file with the Write tool first
|
|
123
|
+
and pass the file path instead.
|
|
124
|
+
|
|
125
|
+
**Duplicate check — only if `PRODUCT_FOLDER` was resolved in Step 1:**
|
|
126
|
+
```
|
|
127
|
+
Write outputs/.dupcheck-tmp.txt containing: {Bug Title}
|
|
128
|
+
```
|
|
129
|
+
```bash
|
|
130
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs dup-bug --product {PRODUCT_FOLDER} --summary-file outputs/.dupcheck-tmp.txt
|
|
131
|
+
```
|
|
132
|
+
`NO_DUPLICATE_FOUND` → proceed silently. `POSSIBLE_DUPLICATE: {BUG-ID} (...) — "{title}"` →
|
|
133
|
+
mention it once above the formatted report in Step 3: `"Heads up — this looks similar to
|
|
134
|
+
{BUG-ID}: \"{title}\" ({status}). File as a new bug anyway, or add to that one instead?"`
|
|
135
|
+
Follow whichever the user picks.
|
|
136
|
+
|
|
137
|
+
**PII/secrets scan — always, regardless of PRODUCT_FOLDER:**
|
|
138
|
+
```
|
|
139
|
+
Write outputs/.piicheck-tmp.txt containing: {Bug Title}\n{Expected Outcome}\n{Actual Outcome}\n{Steps to Reproduce}
|
|
140
|
+
```
|
|
141
|
+
```bash
|
|
142
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs pii-scan --file outputs/.piicheck-tmp.txt
|
|
143
|
+
```
|
|
144
|
+
Delete both temp files once this bug is posted (Step 6). `CLEAN` → proceed silently.
|
|
145
|
+
`FLAGGED: ...` → show the flagged pattern types (never the raw
|
|
146
|
+
match) and ask once: `"This bug report may contain real {types}. Redact before posting, or
|
|
147
|
+
file as-is? (redact / as-is)"`. Apply the choice, then continue to Step 3.
|
|
148
|
+
|
|
149
|
+
---
|
|
150
|
+
|
|
113
151
|
## Step 3 — Display the formatted bug report
|
|
114
152
|
|
|
115
153
|
Show the bug report in this exact format before posting:
|
|
@@ -95,6 +95,10 @@ Use `getJiraIssue` to fetch `CARD_ID`. Extract and store:
|
|
|
95
95
|
- `FIGMA_URL_FROM_CARD` — any Figma link found in description or comments
|
|
96
96
|
- `PROJECT_KEY` — for bug filing later
|
|
97
97
|
|
|
98
|
+
Derive `PRODUCT_FOLDER` now (uppercase `PROJECT_KEY`'s product name, spaces → `_` — same
|
|
99
|
+
normalisation qa-agent Step 6a uses) so Phases 3 and 4 below can call the toolkit without
|
|
100
|
+
re-deriving it. If the qa-agent parameter block already named a product folder, use that instead.
|
|
101
|
+
|
|
98
102
|
Run token tracking `jira_fetch` checkpoint.
|
|
99
103
|
|
|
100
104
|
### 1b — Truncate Jira data if card is verbose
|
|
@@ -170,7 +174,18 @@ From the AC, generate numbered test ideas:
|
|
|
170
174
|
- 2+ negative/edge case tests
|
|
171
175
|
- 1+ error state test
|
|
172
176
|
|
|
173
|
-
|
|
177
|
+
**Risk-based ordering** — pull 2–5 module/page keywords straight out of the AC/card title
|
|
178
|
+
you already read (e.g. "login", "checkout"; no extra fetching), then run one script call:
|
|
179
|
+
|
|
180
|
+
```bash
|
|
181
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs risk-score --product {PRODUCT_FOLDER} --modules "kw1,kw2,kw3"
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
Order the T-01, T-02... list so tests touching the highest-risk module come first. On a
|
|
185
|
+
product's first run (no git history / no known bugs yet) every module scores 0 — keep the
|
|
186
|
+
natural AC order in that case, no need to mention it.
|
|
187
|
+
|
|
188
|
+
Show as a compact table (T-01, T-02 ... with name and expected outcome), risk-ordered.
|
|
174
189
|
Ask: `"Ready to run these tests? (yes / no or edit)"` — include any missing navigation
|
|
175
190
|
questions in this same message.
|
|
176
191
|
Proceed on confirmation.
|
|
@@ -241,6 +256,21 @@ For each test T-01, T-02, ...:
|
|
|
241
256
|
1. Navigate to the feature area via Playwright MCP (preserves session)
|
|
242
257
|
2. Add current URL to `HINTS.pages` if not already present
|
|
243
258
|
|
|
259
|
+
2a. **DOM fingerprint check — once per distinct URL, the first time you land on it this run:**
|
|
260
|
+
```javascript
|
|
261
|
+
browser_evaluate({ expression: `
|
|
262
|
+
Array.from(document.querySelectorAll('[data-testid]')).map(el => el.getAttribute('data-testid')).join(',')
|
|
263
|
+
` })
|
|
264
|
+
```
|
|
265
|
+
```bash
|
|
266
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs fingerprint --product {PRODUCT_FOLDER} --url "CURRENT_URL" --testids "RESULT_FROM_ABOVE"
|
|
267
|
+
```
|
|
268
|
+
- `UNCHANGED` → this page's selectors haven't moved since a prior run. Skip step 3's
|
|
269
|
+
per-interaction element capture for this page entirely (still perform the actual test
|
|
270
|
+
interactions and assertions — only the *hint-recording* is skipped). Add one line to
|
|
271
|
+
`HINTS.notes`: `"Elements on {url}: unchanged — reused prior fingerprint, capture skipped."`
|
|
272
|
+
- `NEW` or `CHANGED` → proceed with full per-interaction capture in step 3 below, same as always.
|
|
273
|
+
|
|
244
274
|
3. For each browser interaction (`browser_fill`, `browser_click`, `browser_select_option`):
|
|
245
275
|
- Execute the interaction
|
|
246
276
|
- Immediately run element capture (zero-token, targeted JS):
|
|
@@ -346,7 +376,7 @@ Run token tracking `test_execution` checkpoint.
|
|
|
346
376
|
|
|
347
377
|
After saving the automation hints file, silently update the product context.
|
|
348
378
|
|
|
349
|
-
|
|
379
|
+
`PRODUCT_FOLDER` was already derived in Phase 1 — reuse it, don't re-derive.
|
|
350
380
|
|
|
351
381
|
```
|
|
352
382
|
CONTEXT_FILE = .claude/skills/qa-agent/product_context/{PRODUCT_FOLDER}/context.md
|
|
@@ -386,6 +416,41 @@ Derive all fields from test execution data — no user input needed:
|
|
|
386
416
|
| Severity | AC explicitly failed → High · Assertion failed → Medium · Observation → Low |
|
|
387
417
|
| Screenshot | Match `outputs/screenshots/T-{N}-*.png` by test ID — use exact filename |
|
|
388
418
|
|
|
419
|
+
**Step 4a.1 — Duplicate check (one script call per failure, no user input needed)**
|
|
420
|
+
|
|
421
|
+
Test names are derived from the AC, not typed by a person, but AC text copied from Jira can
|
|
422
|
+
still contain quotes or symbols — never inline free text into a shell command. Write it with
|
|
423
|
+
the Write tool first, then reference the file:
|
|
424
|
+
|
|
425
|
+
```
|
|
426
|
+
Write outputs/.dupcheck-tmp.txt containing: T-{N}: {test name}
|
|
427
|
+
```
|
|
428
|
+
```bash
|
|
429
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs dup-bug --product {PRODUCT_FOLDER} --summary-file outputs/.dupcheck-tmp.txt
|
|
430
|
+
```
|
|
431
|
+
|
|
432
|
+
`NO_DUPLICATE_FOUND` → proceed normally. `POSSIBLE_DUPLICATE: {BUG-ID} (...) — "{title}"` →
|
|
433
|
+
still file the bug (a regression is a real, separately-trackable failure) but prepend one
|
|
434
|
+
line to the Description: `"⚠ Possibly related to {BUG-ID}: {title}"` for the triager.
|
|
435
|
+
|
|
436
|
+
**Step 4a.2 — PII / secrets scan (one script call per bug, before it leaves the machine)**
|
|
437
|
+
|
|
438
|
+
Same rule — the Description includes live-app text (error messages, field values) that can
|
|
439
|
+
contain anything. Write it to a file, never interpolate it into the command:
|
|
440
|
+
|
|
441
|
+
```
|
|
442
|
+
Write outputs/.piicheck-tmp.txt containing the composed Summary + Description text
|
|
443
|
+
```
|
|
444
|
+
```bash
|
|
445
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs pii-scan --file outputs/.piicheck-tmp.txt
|
|
446
|
+
```
|
|
447
|
+
|
|
448
|
+
`CLEAN` → proceed silently. `FLAGGED: ...` → this is rare (test data occasionally captures a
|
|
449
|
+
real value) — pause only this one bug, show the flagged pattern types (never the raw match),
|
|
450
|
+
and ask once: `"This bug's description may contain real {types}. Redact and continue, or file as-is? (redact / as-is)"`.
|
|
451
|
+
Apply the user's choice, then continue to the next failure — do not stop the whole batch.
|
|
452
|
+
Delete both `outputs/.dupcheck-tmp.txt` and `outputs/.piicheck-tmp.txt` once the batch is done.
|
|
453
|
+
|
|
389
454
|
**Step 4b — File each bug via Atlassian MCP (parallelise where possible)**
|
|
390
455
|
|
|
391
456
|
For each failure:
|
|
@@ -76,6 +76,8 @@ If the user's message contains `--reset-context`:
|
|
|
76
76
|
Extract the project key prefix from the card ID (`QE-89` → `QE`). Check
|
|
77
77
|
`.claude/skills/qa-agent/product_context/{PREFIX}/context.md`; if not found by exact
|
|
78
78
|
prefix, check whether any folder under `product_context/` starts with that prefix.
|
|
79
|
+
Whichever folder name matches, store it as `PRODUCT_FOLDER` for the rest of this run (Steps
|
|
80
|
+
1d and 3 below use it) — it will be re-derived/confirmed from the Jira project name in Step 6a.
|
|
79
81
|
|
|
80
82
|
**If found:** read it once and store `CTX_APP_URL`, `CTX_LOGIN_URL`, `CTX_OTP`,
|
|
81
83
|
`CTX_ENVIRONMENT`, plus the Covered Flows and Known Bugs tables (needed for Steps 2 and 5).
|
|
@@ -112,6 +114,17 @@ only — qa-pack has no way to force a mid-run model switch, so proceed on whate
|
|
|
112
114
|
active regardless of the user's choice. See `SKILLS_CONTEXT.md` → "Model Routing" for the
|
|
113
115
|
opt-in subagent-override pattern some environments can wire up instead.
|
|
114
116
|
|
|
117
|
+
### 1d — Pre-run Cost Estimate
|
|
118
|
+
|
|
119
|
+
**Only if Step 1a found an existing product folder.** This is a plain script call, not a
|
|
120
|
+
reasoning step — run it and print its one line verbatim, prefixed `"Cost estimate: "`:
|
|
121
|
+
|
|
122
|
+
```bash
|
|
123
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs cost-estimate --product {PRODUCT_FOLDER}
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
Skip entirely on a product's first run (no context file yet — nothing to estimate from).
|
|
127
|
+
|
|
115
128
|
---
|
|
116
129
|
|
|
117
130
|
## Step 2 — Ask Relevant Questions (gaps only)
|
|
@@ -176,6 +189,23 @@ Set `AUTO_APPROVE = true` only when ALL hold:
|
|
|
176
189
|
Set `AUTO_APPROVE = false` otherwise, or when the user said "manual gates" / "confirm each
|
|
177
190
|
step" / "don't auto-approve", or when `--reset-context` was passed.
|
|
178
191
|
|
|
192
|
+
### Trust Ratchet (only when AUTO_APPROVE would otherwise be false, and a product folder exists)
|
|
193
|
+
|
|
194
|
+
One script call, before showing the phase table:
|
|
195
|
+
|
|
196
|
+
```bash
|
|
197
|
+
node .claude/skills/qa-agent/toolkit/qa-toolkit.cjs trust-status --product {PRODUCT_FOLDER}
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
- Prints `ELIGIBLE`: add one line to the phase-selection message — "This product has a clean
|
|
201
|
+
approval streak — reply **auto** to skip manual gates for this run." If the user replies
|
|
202
|
+
`auto`, set `AUTO_APPROVE = true` for this run only (does not change future runs or other
|
|
203
|
+
products).
|
|
204
|
+
- Prints `NOT_ELIGIBLE`: say nothing — do not mention trust status at all.
|
|
205
|
+
|
|
206
|
+
The streak itself is recorded by the `automation` skill at each gate (see its SKILL.md) —
|
|
207
|
+
qa-agent only reads the status here.
|
|
208
|
+
|
|
179
209
|
---
|
|
180
210
|
|
|
181
211
|
## Step 4 — Pre-flight Check + Dispatch
|
|
@@ -17,3 +17,19 @@ The QA Agent writes to `{PRODUCT}/context.md` automatically at the end of every
|
|
|
17
17
|
- Known bugs accumulated across all runs
|
|
18
18
|
- Covered test flows
|
|
19
19
|
- Environment notes (URLs, quirks — never credentials; those live only in `.env`)
|
|
20
|
+
|
|
21
|
+
## Other files that may appear in a product folder
|
|
22
|
+
|
|
23
|
+
These are written and read exclusively by `.claude/skills/qa-agent/toolkit/qa-toolkit.cjs`
|
|
24
|
+
— never edited by hand, never read in full by a skill (only queried through the toolkit).
|
|
25
|
+
They exist so features like duplicate-bug detection, DOM-change skipping, and locator
|
|
26
|
+
learning don't have to re-derive their state from `context.md` every run.
|
|
27
|
+
|
|
28
|
+
| File | Written by | Purpose |
|
|
29
|
+
|------|-----------|---------|
|
|
30
|
+
| `dom-fingerprints.json` | `fingerprint` command | One hash per page URL — lets a run skip DOM re-discovery when nothing changed |
|
|
31
|
+
| `trust.json` | `trust-record` command | Gate 1 / Gate 2 clean-approval streaks — feeds the auto-approve trust ratchet |
|
|
32
|
+
| `locator-learnings.md` | `locator-record` command | Locators that broke before, and what fixed them — read before writing new POM code |
|
|
33
|
+
|
|
34
|
+
All three are covered by the same `.claude/skills/qa-agent/` gitignore entry as
|
|
35
|
+
`context.md` — they never leave the machine that generated them.
|