qualiow-exploratory-testing 1.3.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +22 -0
- package/.env.example +27 -0
- package/CHANGELOG.md +194 -0
- package/README.md +362 -205
- package/agents/qa-gather-agent.md +24 -0
- package/{scripts → bin}/doctor-mobile.sh +57 -18
- package/bin/mcli +15 -9
- package/bin/mobile-cli.mjs +574 -286
- package/{scripts → bin}/setup-mobile.sh +56 -19
- package/bin/wadb +10 -4
- package/bin/wk-ios +29 -17
- package/bin/wkeval.mjs +7 -1
- package/data/domains/_default.yml +1 -1
- package/data/knowledge/changelog.yml +30 -12
- package/data/knowledge/manifest.yml +333 -127
- package/data/knowledge/releases/v0.1.0/entries/technique-business-logic-race-conditions.yml +1 -1
- package/data/knowledge/releases/v0.1.0/release.yml +8 -1
- package/data/knowledge/releases/v0.6.0/entries/technique-async-callback-contracts.yml +98 -0
- package/data/knowledge/releases/v0.6.0/entries/technique-exactly-once-verification.yml +116 -0
- package/data/knowledge/releases/v0.6.0/release.yml +30 -0
- package/data/security/SECURITY-POLICY.md +273 -0
- package/data/targets/_default.yml +1 -1
- package/data/targets/_example-api-only.yml +80 -0
- package/data/targets/_example-mobile-emulation.yml +2 -2
- package/data/targets/_example-native-mobile.yml +2 -2
- package/data/targets/_example-sim-android-chrome.yml +1 -1
- package/data/targets/_example-sim-ios-safari.yml +1 -1
- package/data/templates/api-probe-matrix.md +3 -3
- package/data/templates/bug-report.md +29 -151
- package/data/templates/coverage-map.md +9 -6
- package/data/templates/expected-behaviour.md +3 -2
- package/data/templates/session-charter.md +3 -0
- package/data/templates/session-report.md +57 -80
- package/dist/cli/index.js +1452 -700
- package/dist/index.d.ts +589 -81
- package/dist/index.js +1612 -227
- package/docs/BACKEND-VERIFICATION.md +39 -9
- package/docs/GETTING-STARTED.md +188 -128
- package/docs/MOBILE-SETUP.md +149 -18
- package/package.json +32 -18
- package/skills/qa-explore/SKILL.md +31 -25
- package/skills/qa-explore/phases/00-setup.md +67 -41
- package/skills/qa-explore/phases/01-auth.md +32 -15
- package/skills/qa-explore/phases/02-charter.md +16 -5
- package/skills/qa-explore/phases/03-discovery.md +50 -26
- package/skills/qa-explore/phases/04-journeys.md +2 -2
- package/skills/qa-explore/phases/05-features.md +39 -24
- package/skills/qa-explore/phases/06-edge-cases.md +24 -13
- package/skills/qa-explore/phases/07-reporting.md +79 -140
- package/skills/qa-explore/references/context-integration.md +1 -1
- package/skills/qa-explore/references/output-contract.md +187 -0
- package/skills/qa-explore/references/paths.md +84 -0
- package/skills/qa-explore/references/playwright-agents-integration.md +61 -84
- package/skills/qa-explore/references/security-rules.md +66 -25
- package/skills/qa-explore/references/severity-guide.md +31 -4
- package/skills/qa-explore-cleanup/SKILL.md +17 -6
- package/skills/qa-explore-feedback/SKILL.md +18 -5
- package/skills/qa-explore-mobile/SKILL.md +86 -64
- package/skills/qa-explore-mobile/phases/00-setup.md +71 -62
- package/skills/qa-explore-mobile/phases/01-auth.md +28 -31
- package/skills/qa-explore-mobile/phases/02-charter.md +8 -4
- package/skills/qa-explore-mobile/phases/03-discovery.md +17 -11
- package/skills/qa-explore-mobile/phases/04-journeys.md +6 -6
- package/skills/qa-explore-mobile/phases/05-features.md +9 -9
- package/skills/qa-explore-mobile/phases/06-edge-cases.md +35 -40
- package/skills/qa-explore-mobile/phases/07-reporting.md +82 -56
- package/skills/qa-explore-mobile/references/mobile-edges.md +26 -26
- package/skills/qa-explore-quick/SKILL.md +60 -25
- package/skills/qa-explore-report/SKILL.md +40 -34
- package/skills/qa-gather/SKILL.md +17 -2
- package/skills/qa-knowledge-add/SKILL.md +20 -9
- package/skills/qa-knowledge-list/SKILL.md +12 -3
- package/skills/qa-target-setup/SKILL.md +63 -41
- package/skills/qa-verify-backend/SKILL.md +26 -17
- package/skills/qa-verify-backend/phases/00-setup.md +57 -12
- package/skills/qa-verify-backend/phases/01-ac-decomposition.md +2 -2
- package/skills/qa-verify-backend/phases/03-live-verification.md +3 -3
- package/skills/qa-verify-backend/phases/03b-api-verification.md +22 -9
- package/skills/qa-verify-backend/phases/04-e2e-trigger.md +18 -2
- package/skills/qa-verify-backend/phases/05-reporting.md +38 -10
- package/skills/qa-verify-backend/references/api-probes.md +28 -1
- package/skills/qa-verify-backend/references/payload-verification.md +1 -1
- package/skills/qa-verify-backend/references/release-readiness.md +1 -1
- package/skills/qa-verify-backend/references/safety-rules.md +13 -8
- package/data/domains/_default.md +0 -181
- package/data/domains/ecommerce.md +0 -192
- package/data/domains/fintech.md +0 -200
- package/data/domains/identity.md +0 -88
- package/data/domains/marketing.md +0 -194
- package/data/domains/saas.md +0 -197
- package/dist/cli/index.d.ts +0 -1
- package/dist/cli/index.js.map +0 -1
- package/dist/index.js.map +0 -1
|
@@ -1,27 +1,43 @@
|
|
|
1
|
-
# Phase 7: Reflection & Reporting (
|
|
1
|
+
# Phase 7: Reflection & Reporting (4 min)
|
|
2
|
+
|
|
3
|
+
Every command carries `-s=<sid>` (omitted here). The browser stays open until the evidence is captured; it is closed in the last step.
|
|
2
4
|
|
|
3
5
|
## Stop Recording
|
|
4
6
|
|
|
5
7
|
```bash
|
|
8
|
+
playwright-cli video-chapter "Reporting"
|
|
9
|
+
playwright-cli video-stop
|
|
6
10
|
playwright-cli tracing-stop
|
|
7
|
-
playwright-cli close
|
|
8
11
|
```
|
|
9
12
|
|
|
10
|
-
### Trace
|
|
13
|
+
### Trace analysis (Playwright 1.59+, optional)
|
|
14
|
+
|
|
15
|
+
`tracing-stop` writes a trace under `traces/` (or the path it prints). To inspect it from
|
|
16
|
+
the command line: `npx playwright trace open <trace>` then `npx playwright trace actions --errors-only`,
|
|
17
|
+
`npx playwright trace requests --failed`, `npx playwright trace console`, and `npx playwright trace close`
|
|
18
|
+
(check `npx playwright trace --help` for the exact subcommands of the installed version).
|
|
19
|
+
Copy the trace into `output/sessions/<session-dir>/` if it is evidence for a bug.
|
|
20
|
+
|
|
21
|
+
### Annotated video for Critical / High bugs (optional)
|
|
11
22
|
|
|
12
|
-
|
|
23
|
+
The session recording from phase 2 already has chapters. For a polished reproduction of a
|
|
24
|
+
Critical or High bug, record a second clip with action callouts:
|
|
13
25
|
|
|
14
|
-
|
|
15
|
-
-
|
|
16
|
-
-
|
|
17
|
-
|
|
26
|
+
```bash
|
|
27
|
+
playwright-cli video-start output/sessions/<session-dir>/videos/BUG-NNN.webm
|
|
28
|
+
playwright-cli video-show-actions
|
|
29
|
+
# ...reproduce the bug...
|
|
30
|
+
playwright-cli video-hide-actions
|
|
31
|
+
playwright-cli video-stop
|
|
32
|
+
```
|
|
18
33
|
|
|
19
|
-
|
|
34
|
+
Or write a scripted "hero" reproduction with `page.screencast.start()` / `showChapter()` /
|
|
35
|
+
`showOverlay()` (Playwright 1.59) and run it with `playwright-cli run-code --filename=<script.js>`.
|
|
36
|
+
A screenshot remains sufficient for Medium / Low.
|
|
20
37
|
|
|
21
38
|
## Reflect Before Writing
|
|
22
39
|
|
|
23
|
-
|
|
24
|
-
1. **What was the biggest risk I found?** (not the most bugs -- the biggest RISK)
|
|
40
|
+
1. **What was the biggest risk I found?** (not the most bugs; the biggest RISK)
|
|
25
41
|
2. **What would I tell the CEO in one sentence?**
|
|
26
42
|
3. **What did I NOT test that still worries me?**
|
|
27
43
|
4. **Which heuristic was most useful? Which was useless?**
|
|
@@ -29,145 +45,68 @@ Before writing the report, answer these questions:
|
|
|
29
45
|
|
|
30
46
|
## Write Bug Reports
|
|
31
47
|
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
## Write Session Report
|
|
68
|
-
|
|
69
|
-
`session-report.md` with:
|
|
70
|
-
- **Executive summary** (3 sentences max -- what was tested, what was found, what's the risk)
|
|
71
|
-
- Business context and charter summary
|
|
72
|
-
- Bug summary table (severity, component, business impact)
|
|
73
|
-
- Data integrity results
|
|
74
|
-
- What's MISSING (negative space findings)
|
|
75
|
-
- Coverage map with risk ranking
|
|
76
|
-
- Cross-feature journey results
|
|
77
|
-
- Heuristic effectiveness (which worked, which didn't)
|
|
78
|
-
- Areas NOT tested and why
|
|
79
|
-
- Recommendations for next session
|
|
80
|
-
- Reflection answers
|
|
81
|
-
|
|
82
|
-
## Coverage Map
|
|
83
|
-
|
|
84
|
-
Document what was tested and what was not:
|
|
85
|
-
|
|
86
|
-
| Feature | Risk | Tested? | Findings | Notes |
|
|
87
|
-
|---------|------|---------|----------|-------|
|
|
88
|
-
| [feature] | P0 | Yes | 2 bugs | [notes] |
|
|
89
|
-
| [feature] | P1 | Partial | 1 bug | [notes] |
|
|
90
|
-
| [feature] | P2 | No | -- | Time ran out |
|
|
48
|
+
One file per bug, `bugs/BUG-NNN.md`, in exactly the format of
|
|
49
|
+
`${CLAUDE_SKILL_DIR}/references/output-contract.md` (the template
|
|
50
|
+
`<data>/templates/bug-report.md` is the same format with guidance):
|
|
51
|
+
|
|
52
|
+
- the confidentiality header as the **first two lines**, verbatim:
|
|
53
|
+
|
|
54
|
+
```
|
|
55
|
+
> CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
|
|
56
|
+
> and application details. Do not share outside your organization without review.
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
- then `# BUG-NNN: [Component] fails [Condition] causing [Impact]`
|
|
60
|
+
- `**Severity:**` (per `severity-guide.md`; when in doubt go lower), `**Priority:**`,
|
|
61
|
+
`**Component:**`, `**URL:**`, `**Environment:** Playwright CLI, Chromium, <viewport>`,
|
|
62
|
+
`**Reproduction rate:**`
|
|
63
|
+
- `## Summary`, `## Expected Behavior`, `## Actual Behavior`, `## Steps to Reproduce`
|
|
64
|
+
- `## Business Impact` — MANDATORY; answer at least one of Revenue / Trust / Regulatory /
|
|
65
|
+
Data / Scale, write "none identified" for the rest
|
|
66
|
+
- `## Evidence` — `Screenshot: screenshots/BUG-NNN.png` (take it with
|
|
67
|
+
`playwright-cli screenshot --filename=output/sessions/<session-dir>/screenshots/BUG-NNN.png`),
|
|
68
|
+
console errors, network failures (`playwright-cli request <n>` output, redacted)
|
|
69
|
+
- `## Recommended Fix Priority`
|
|
70
|
+
|
|
71
|
+
Redact per `security-rules.md` before writing.
|
|
72
|
+
|
|
73
|
+
## Write the Session Report
|
|
74
|
+
|
|
75
|
+
`session-report.md` in the format of `output-contract.md`: header, `# Session Report — <target>`,
|
|
76
|
+
`## Session Metadata`, `## Executive Summary` (3 sentences: what was tested, what was found,
|
|
77
|
+
the biggest risk), `## Summary Stats`, `## Coverage Map` with the header
|
|
78
|
+
`| Area | Risk | Status | Bugs | Notes |` (Status: tested / partial / not-tested /
|
|
79
|
+
code-verified-only), `## Bugs Found`, `## Observations` (including what's MISSING and the
|
|
80
|
+
data-integrity results), `## Areas Not Tested` with reasons, `## Recommendations` for the
|
|
81
|
+
next session, `## Reflection` (the five answers above), `## Session Stats`.
|
|
91
82
|
|
|
92
83
|
## Session Stats
|
|
93
84
|
|
|
94
|
-
Write `output/sessions/<session-dir>/stats.json`
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
"target": "<target_url>",
|
|
100
|
-
"domain": "<domain>",
|
|
101
|
-
"started_at": "<from progress.json>",
|
|
102
|
-
"completed_at": "<ISO timestamp now>",
|
|
103
|
-
"duration_min": "<calculated>",
|
|
104
|
-
"phases_completed": "<count of phases with status=complete>",
|
|
105
|
-
"total_phases": 8,
|
|
106
|
-
"bugs": {
|
|
107
|
-
"total": "<count>",
|
|
108
|
-
"critical": "<count>",
|
|
109
|
-
"high": "<count>",
|
|
110
|
-
"medium": "<count>",
|
|
111
|
-
"low": "<count>"
|
|
112
|
-
},
|
|
113
|
-
"coverage": {
|
|
114
|
-
"pages_explored": "<count>",
|
|
115
|
-
"checklist_total": "<from domain config>",
|
|
116
|
-
"checklist_verified": "<count checked>",
|
|
117
|
-
"checklist_percentage": "<calculated>",
|
|
118
|
-
"data_integrity_checks": "<count>",
|
|
119
|
-
"data_integrity_passed": "<count>",
|
|
120
|
-
"personas_tested": ["<list>"],
|
|
121
|
-
"purchase_flows_tested": ["<list>"]
|
|
122
|
-
},
|
|
123
|
-
"evidence": {
|
|
124
|
-
"screenshots": "<count>",
|
|
125
|
-
"console_errors_found": "<count>",
|
|
126
|
-
"network_failures_found": "<count>"
|
|
127
|
-
},
|
|
128
|
-
"areas_not_tested": ["<list with reasons>"],
|
|
129
|
-
"blocked_by": "<if something blocked testing, note it>"
|
|
130
|
-
}
|
|
131
|
-
```
|
|
132
|
-
|
|
133
|
-
Also append a **stats summary** at the bottom of `session-report.md`:
|
|
85
|
+
Write `output/sessions/<session-dir>/stats.json` exactly as `output-contract.md` defines it
|
|
86
|
+
(`session_id`, `kind: "explore"`, `target`, `date`, `duration_min`, `bugs_found`,
|
|
87
|
+
`severity_counts`, `pages_explored`, plus the optional `domain`, `started_at`,
|
|
88
|
+
`completed_at`, `phases_completed`, `total_phases: 8`, `coverage`, `evidence`,
|
|
89
|
+
`areas_not_tested`, `blocked_by`). No other top-level keys.
|
|
134
90
|
|
|
135
|
-
|
|
136
|
-
---
|
|
91
|
+
Append the same numbers as the `## Session Stats` table at the end of `session-report.md`.
|
|
137
92
|
|
|
138
|
-
##
|
|
93
|
+
## Update progress.json — Final
|
|
139
94
|
|
|
140
|
-
|
|
141
|
-
|--------|-------|
|
|
142
|
-
| Duration | <X> min |
|
|
143
|
-
| Phases completed | <N>/8 |
|
|
144
|
-
| Bugs found | <N> (Critical: <N>, High: <N>, Medium: <N>, Low: <N>) |
|
|
145
|
-
| Pages explored | <N> |
|
|
146
|
-
| Screenshots taken | <N> |
|
|
147
|
-
| Console errors found | <N> |
|
|
95
|
+
Set `status` to `complete`, `current_phase` to `reporting`, every phase to its final status.
|
|
148
96
|
|
|
149
|
-
|
|
97
|
+
## Update Indexes
|
|
150
98
|
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
| <area> | P0 | Yes | Deep | <N> | <notes> |
|
|
154
|
-
| <area> | P1 | Partial | Shallow | <N> | <blocked by...> |
|
|
155
|
-
| <area> | P2 | No | — | — | Time ran out |
|
|
99
|
+
Append the session row to `output/sessions/INDEX.md` and one row per bug to
|
|
100
|
+
`output/bugs/all-bugs.md`, in the column order defined in `paths.md`.
|
|
156
101
|
|
|
157
|
-
|
|
158
|
-
<N>/<N> items verified (<X>%)
|
|
102
|
+
## Close the Browser Session
|
|
159
103
|
|
|
160
|
-
|
|
161
|
-
<N>/<N> checks passed
|
|
104
|
+
Last, once every screenshot, trace and video is on disk:
|
|
162
105
|
|
|
163
|
-
|
|
164
|
-
-
|
|
106
|
+
```bash
|
|
107
|
+
playwright-cli close
|
|
108
|
+
playwright-cli delete-data
|
|
165
109
|
```
|
|
166
110
|
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
Set status to "complete", current_phase to "reporting", all phases to their final status.
|
|
170
|
-
|
|
171
|
-
## Update Indexes
|
|
172
|
-
|
|
173
|
-
Append to `output/sessions/INDEX.md` and `output/bugs/all-bugs.md`.
|
|
111
|
+
(both with `-s=<sid>`; also close and `delete-data` any extra session such as `-s=<sid>-race`).
|
|
112
|
+
Append to `session-log.md`: `[<timestamp>] [PHASE] Reporting complete — <N> bugs, session closed`.
|
|
@@ -4,7 +4,7 @@ How the `--context` parameter changes the session.
|
|
|
4
4
|
|
|
5
5
|
## Mode 1: Context File (`--context <file>`)
|
|
6
6
|
|
|
7
|
-
Read the file. It may be a
|
|
7
|
+
Read the file. It may be a `/qa-gather` context file (structured markdown with ticket data, specs, MR diffs, design notes) or any document describing what to test.
|
|
8
8
|
|
|
9
9
|
Extract and use:
|
|
10
10
|
|
|
@@ -0,0 +1,187 @@
|
|
|
1
|
+
# Output Contract
|
|
2
|
+
|
|
3
|
+
One format for every session kind (explore, quick, mobile, backend). The templates in
|
|
4
|
+
`<data>/templates/` follow it, `qualiow report` parses it, and the feedback and cleanup
|
|
5
|
+
skills read it. A session that deviates from it is invisible to all of them.
|
|
6
|
+
|
|
7
|
+
## Confidentiality header
|
|
8
|
+
|
|
9
|
+
The first two lines of **every** markdown artefact (charter, phase files, bug reports,
|
|
10
|
+
session report, coverage map, AC matrix, expected-behaviour spec):
|
|
11
|
+
|
|
12
|
+
```
|
|
13
|
+
> CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
|
|
14
|
+
> and application details. Do not share outside your organization without review.
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
## Session files
|
|
18
|
+
|
|
19
|
+
| File | Written by |
|
|
20
|
+
|---|---|
|
|
21
|
+
| `charter.md` | phase 2 |
|
|
22
|
+
| `session-log.md` | every phase, append-only |
|
|
23
|
+
| `progress.json` | phase 0, updated after every phase |
|
|
24
|
+
| `phase-3-discovery.md` … `phase-6-edge-cases.md` | phases 3–6 (file number = phase number) |
|
|
25
|
+
| `screenshots/BUG-NNN.png`, other `screenshots/*.png` | as taken |
|
|
26
|
+
| `videos/*.webm` (web) / `videos/*.mp4` (mobile) | recording |
|
|
27
|
+
| `bugs/BUG-NNN.md` | phase 7 |
|
|
28
|
+
| `session-report.md` | phase 7 |
|
|
29
|
+
| `stats.json` | phase 7 |
|
|
30
|
+
| mobile: `logs/BUG-NNN.log` · backend: `ac-matrix.md`, `evidence/`, `probes/`, `expected-behaviour.md` | as produced |
|
|
31
|
+
|
|
32
|
+
The coverage map lives inside `session-report.md`. There is no separate session-level
|
|
33
|
+
`coverage-map.md`; `/qa-explore-report` can export one on request.
|
|
34
|
+
|
|
35
|
+
## Bug report — `bugs/BUG-NNN.md`
|
|
36
|
+
|
|
37
|
+
```markdown
|
|
38
|
+
> CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
|
|
39
|
+
> and application details. Do not share outside your organization without review.
|
|
40
|
+
|
|
41
|
+
# BUG-NNN: [Component] fails [Condition] causing [Impact]
|
|
42
|
+
|
|
43
|
+
**Severity:** Critical | High | Medium | Low
|
|
44
|
+
**Priority:** P0 | P1 | P2 | P3
|
|
45
|
+
**Component:** <Component>
|
|
46
|
+
**URL:** <exact URL — mobile: screen name or deep link — backend: endpoint or resource>
|
|
47
|
+
**Environment:** <browser + viewport — mobile: mode / platform / device / OS — backend: env kind + build>
|
|
48
|
+
**Reproduction rate:** Always | Intermittent (~X%) | Once
|
|
49
|
+
|
|
50
|
+
## Summary
|
|
51
|
+
<two sentences: what breaks and why it matters>
|
|
52
|
+
|
|
53
|
+
## Expected Behavior
|
|
54
|
+
<from the user's perspective>
|
|
55
|
+
|
|
56
|
+
## Actual Behavior
|
|
57
|
+
<be specific; quote error messages verbatim>
|
|
58
|
+
|
|
59
|
+
## Steps to Reproduce
|
|
60
|
+
1. <exact steps a developer can follow>
|
|
61
|
+
|
|
62
|
+
## Business Impact
|
|
63
|
+
- **Revenue impact:** <or "none identified">
|
|
64
|
+
- **Trust impact:** …
|
|
65
|
+
- **Regulatory risk:** …
|
|
66
|
+
- **Data risk:** …
|
|
67
|
+
- **Scale:** <who is affected, under what conditions>
|
|
68
|
+
|
|
69
|
+
## Evidence
|
|
70
|
+
- Screenshot: `screenshots/BUG-NNN.png`
|
|
71
|
+
- Video: `videos/<file>` (if recorded)
|
|
72
|
+
- Log: `logs/BUG-NNN.log` (mobile) / `evidence/<file>` (backend)
|
|
73
|
+
- Console errors: <or "none">
|
|
74
|
+
- Network failures: <or "none">
|
|
75
|
+
|
|
76
|
+
## Recommended Fix Priority
|
|
77
|
+
<why this priority, relative to the other findings>
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Rules: numbered from `BUG-001` per session; one bug per file; Business Impact is mandatory
|
|
81
|
+
(answer at least one bullet, write "none identified" for the rest); severity per
|
|
82
|
+
`severity-guide.md`, and when in doubt go one level lower.
|
|
83
|
+
|
|
84
|
+
## Session report — `session-report.md`
|
|
85
|
+
|
|
86
|
+
```markdown
|
|
87
|
+
> CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
|
|
88
|
+
> and application details. Do not share outside your organization without review.
|
|
89
|
+
|
|
90
|
+
# Session Report — <target>
|
|
91
|
+
|
|
92
|
+
## Session Metadata
|
|
93
|
+
| Field | Value |
|
|
94
|
+
|---|---|
|
|
95
|
+
| Session ID | <session-dir> |
|
|
96
|
+
| Kind | explore / quick / mobile / backend |
|
|
97
|
+
| Target | <target id or name> |
|
|
98
|
+
| URL | <base url> |
|
|
99
|
+
| Date | <YYYY-MM-DD> |
|
|
100
|
+
| Duration | <N> min |
|
|
101
|
+
| Charter | charter.md |
|
|
102
|
+
|
|
103
|
+
## Executive Summary
|
|
104
|
+
<three sentences: what was tested, what was found, the biggest risk>
|
|
105
|
+
|
|
106
|
+
## Summary Stats
|
|
107
|
+
| Metric | Count |
|
|
108
|
+
|---|---|
|
|
109
|
+
| Pages explored | |
|
|
110
|
+
| Actions performed | |
|
|
111
|
+
| Bugs — Critical | |
|
|
112
|
+
| Bugs — High | |
|
|
113
|
+
| Bugs — Medium | |
|
|
114
|
+
| Bugs — Low | |
|
|
115
|
+
| **Total bugs** | |
|
|
116
|
+
|
|
117
|
+
## Coverage Map
|
|
118
|
+
| Area | Risk | Status | Bugs | Notes |
|
|
119
|
+
|---|---|---|---|---|
|
|
120
|
+
| <area> | P0 | tested | 2 | |
|
|
121
|
+
| <area> | P1 | partial | 1 | keyboard covered submit |
|
|
122
|
+
| <area> | P2 | not-tested | — | time ran out |
|
|
123
|
+
|
|
124
|
+
## Bugs Found
|
|
125
|
+
| # | ID | Title | Severity | Report |
|
|
126
|
+
|---|---|---|---|---|
|
|
127
|
+
| 1 | BUG-001 | <title> | High | bugs/BUG-001.md |
|
|
128
|
+
|
|
129
|
+
## Observations
|
|
130
|
+
## Areas Not Tested
|
|
131
|
+
| Area | Reason |
|
|
132
|
+
|---|---|
|
|
133
|
+
|
|
134
|
+
## Recommendations
|
|
135
|
+
## Reflection
|
|
136
|
+
## Session Stats
|
|
137
|
+
| Metric | Value |
|
|
138
|
+
|---|---|
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
Coverage `Status` ∈ `tested` · `partial` · `not-tested` · `code-verified-only` (believed
|
|
142
|
+
correct from reading source, never observed: this is UNVERIFIABLE, not a pass, and must not
|
|
143
|
+
be skimmed as green).
|
|
144
|
+
|
|
145
|
+
Mobile appends `## Mobile Context` and `## Deferred Tests`; backend inserts
|
|
146
|
+
`## AC Matrix Summary` (all six verdicts counted) after the executive summary.
|
|
147
|
+
|
|
148
|
+
## `stats.json`
|
|
149
|
+
|
|
150
|
+
```json
|
|
151
|
+
{
|
|
152
|
+
"session_id": "<session-dir>",
|
|
153
|
+
"kind": "explore",
|
|
154
|
+
"target": "<target id or url>",
|
|
155
|
+
"date": "<YYYY-MM-DD>",
|
|
156
|
+
"duration_min": 42,
|
|
157
|
+
"bugs_found": 5,
|
|
158
|
+
"severity_counts": { "critical": 1, "high": 2, "medium": 2, "low": 0 },
|
|
159
|
+
"pages_explored": 14,
|
|
160
|
+
"domain": "ecommerce",
|
|
161
|
+
"started_at": "<ISO timestamp>",
|
|
162
|
+
"completed_at": "<ISO timestamp>",
|
|
163
|
+
"phases_completed": 8,
|
|
164
|
+
"total_phases": 8,
|
|
165
|
+
"coverage": { "checklist_total": 22, "checklist_verified": 15, "data_integrity_checks": 6, "data_integrity_passed": 5 },
|
|
166
|
+
"evidence": { "screenshots": 9, "console_errors_found": 2, "network_failures_found": 1 },
|
|
167
|
+
"areas_not_tested": ["settings — time ran out"],
|
|
168
|
+
"blocked_by": null
|
|
169
|
+
}
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
The first eight keys are required; the rest are optional. No other top-level keys (the
|
|
173
|
+
schema is strict). Mobile puts its device block under `coverage.mobile`; backend puts the
|
|
174
|
+
verdict counts under `coverage.verdicts`.
|
|
175
|
+
|
|
176
|
+
## Index rows
|
|
177
|
+
|
|
178
|
+
Every session ends by appending one row to `output/sessions/INDEX.md` and one row per bug
|
|
179
|
+
to `output/bugs/all-bugs.md`. The column order (defined once in `paths.md`) is:
|
|
180
|
+
|
|
181
|
+
```
|
|
182
|
+
| Date | Kind | Target | Bugs | Duration | Status | Report |
|
|
183
|
+
| ID | Session | Title | Severity | Status | Report |
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
A session that skips these rows is invisible to `/qa-explore-report`,
|
|
187
|
+
`/qa-explore-feedback`, `/qa-explore-cleanup` and `qualiow list sessions`.
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
# Paths and Resolution Order
|
|
2
|
+
|
|
3
|
+
Every qualiow skill resolves its inputs in the same order, whichever way the skills were
|
|
4
|
+
installed (`qualiow init` copy into a project, a Claude Code plugin, or this repository
|
|
5
|
+
checked out). Resolve once in the setup phase, then **write the resolved literal prefix in
|
|
6
|
+
every command**. Never put a shell variable as the first token of a command line: Claude
|
|
7
|
+
Code's permission rules match on the first literal token, so a command that begins with a
|
|
8
|
+
variable expansion matches no rule and prompts every time. Write `qa/bin/mcli snapshot`,
|
|
9
|
+
not a variable holding that prefix.
|
|
10
|
+
|
|
11
|
+
## Target config
|
|
12
|
+
|
|
13
|
+
| Input | Resolution order (first existing wins) |
|
|
14
|
+
|---|---|
|
|
15
|
+
| `--target <name>` | `$PWD/data/targets/<name>.yml` → `${CLAUDE_PLUGIN_ROOT}/data/targets/<name>.yml` |
|
|
16
|
+
| no `--target` | `$PWD/qa/target.yml` → `<data>/targets/_default.yml` |
|
|
17
|
+
|
|
18
|
+
`qa/target.yml` is the **project-local** target: it lives next to the code under test, is
|
|
19
|
+
written by `/qa-target-setup`, and is the right place for anything with internal hostnames.
|
|
20
|
+
`data/targets/` holds shared and example targets.
|
|
21
|
+
|
|
22
|
+
## Data directory (`<data>`)
|
|
23
|
+
|
|
24
|
+
First existing of: `$PWD/data` → `${CLAUDE_PLUGIN_ROOT}/data` →
|
|
25
|
+
`${CLAUDE_SKILL_DIR}/../../data` → `${CLAUDE_SKILL_DIR}/../../../data`.
|
|
26
|
+
|
|
27
|
+
Read from `<data>`: `domains/<domain>.yml`, `knowledge/manifest.yml`,
|
|
28
|
+
`knowledge/releases/<version>/entries/*.yml`, `knowledge/learned-patterns.md`,
|
|
29
|
+
`templates/*.md`, `security/SECURITY-POLICY.md`.
|
|
30
|
+
|
|
31
|
+
**Writes** (learned patterns from `/qa-explore-feedback`, custom entries from
|
|
32
|
+
`/qa-knowledge-add`) always go to `$PWD/data/…`; create the directory if it is missing.
|
|
33
|
+
Never write into a plugin directory.
|
|
34
|
+
|
|
35
|
+
## Credentials
|
|
36
|
+
|
|
37
|
+
`$PWD/qa/.env` → `$PWD/.env`. Target configs name env vars; the values live only in these
|
|
38
|
+
files. Read a value when you need it, pass it to the command, and never echo it into a
|
|
39
|
+
log, a report, or a command line that will be recorded.
|
|
40
|
+
|
|
41
|
+
## Mobile driver
|
|
42
|
+
|
|
43
|
+
First existing of: `$PWD/qa/bin/mcli` → `$PWD/bin/mcli` → `mcli` on `PATH` →
|
|
44
|
+
`${CLAUDE_PLUGIN_ROOT}/bin/mcli` → `${CLAUDE_SKILL_DIR}/../../bin/mcli`. Same order for
|
|
45
|
+
`wadb`, `wk-ios` and `doctor-mobile.sh`. The mobile phase files show `qa/bin/mcli` (the
|
|
46
|
+
default after `qualiow init`); in a checkout of this repository it is `bin/mcli`.
|
|
47
|
+
Substitute the prefix you resolved.
|
|
48
|
+
|
|
49
|
+
## Output
|
|
50
|
+
|
|
51
|
+
Always `$PWD/output/`. Never write session output anywhere else.
|
|
52
|
+
|
|
53
|
+
### Session directory
|
|
54
|
+
|
|
55
|
+
`output/sessions/<YYYY-MM-DD-HHmm>-<kind>-<slug>/`
|
|
56
|
+
|
|
57
|
+
- `kind` ∈ `explore` | `quick` | `mobile` | `backend`
|
|
58
|
+
- `slug` = target id without a leading `_`; mobile: `<target>-<ios|android>`; backend: the
|
|
59
|
+
ticket id or the target id; an ad-hoc URL: the hostname with dots replaced by dashes.
|
|
60
|
+
Lowercase `[a-z0-9-]`, at most 40 characters.
|
|
61
|
+
- Date first, so alphabetical order is chronological and "latest" is unambiguous.
|
|
62
|
+
|
|
63
|
+
`--session <dir>`: when `qualiow explore` (pre-flight) already created the directory, reuse
|
|
64
|
+
it instead of creating a second one.
|
|
65
|
+
|
|
66
|
+
### playwright-cli session id
|
|
67
|
+
|
|
68
|
+
`-s=<kind>-<HHmm>-<slug>` (for example `-s=explore-1420-parabank`). Every `playwright-cli`
|
|
69
|
+
command in the session carries it; the phase files omit the prefix for brevity. The session
|
|
70
|
+
ends with `playwright-cli -s=<sid> close` and then `playwright-cli -s=<sid> delete-data`.
|
|
71
|
+
|
|
72
|
+
### Index rows
|
|
73
|
+
|
|
74
|
+
Append to `output/sessions/INDEX.md` (columns `| Date | Kind | Target | Bugs | Duration | Status | Report |`):
|
|
75
|
+
|
|
76
|
+
```
|
|
77
|
+
| <YYYY-MM-DD> | <kind> | <target> | <bugs> | <N> min | complete | <session-dir>/session-report.md |
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Append one row per bug to `output/bugs/all-bugs.md` (columns `| ID | Session | Title | Severity | Status | Report |`):
|
|
81
|
+
|
|
82
|
+
```
|
|
83
|
+
| BUG-NNN | <session-dir> | <title> | <severity> | open | <session-dir>/bugs/BUG-NNN.md |
|
|
84
|
+
```
|