qualiow-exploratory-testing 1.3.0 → 2.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +26 -0
- package/.claude-plugin/plugin.json +22 -0
- package/.env.example +27 -0
- package/CHANGELOG.md +259 -0
- package/README.md +472 -204
- package/agents/qa-diff-indexer-agent.md +81 -0
- package/agents/qa-gather-agent.md +41 -0
- package/agents/qa-page-mapper-agent.md +74 -0
- package/agents/qa-reporting-agent.md +103 -0
- package/{scripts → bin}/doctor-mobile.sh +57 -18
- package/bin/mcli +15 -9
- package/bin/mobile-cli.mjs +574 -286
- package/bin/qualiow +30 -0
- package/{scripts → bin}/setup-mobile.sh +56 -19
- package/bin/wadb +10 -4
- package/bin/wk-ios +29 -17
- package/bin/wkeval.mjs +7 -1
- package/data/domains/_default.yml +1 -1
- package/data/knowledge/changelog.yml +30 -12
- package/data/knowledge/manifest.yml +355 -127
- package/data/knowledge/releases/v0.1.0/entries/technique-business-logic-race-conditions.yml +1 -1
- package/data/knowledge/releases/v0.1.0/release.yml +8 -1
- package/data/knowledge/releases/v0.6.0/entries/technique-async-callback-contracts.yml +98 -0
- package/data/knowledge/releases/v0.6.0/entries/technique-exactly-once-verification.yml +116 -0
- package/data/knowledge/releases/v0.6.0/release.yml +30 -0
- package/data/security/SECURITY-POLICY.md +273 -0
- package/data/targets/_default.yml +1 -1
- package/data/targets/_example-api-only.yml +80 -0
- package/data/targets/_example-mobile-emulation.yml +2 -2
- package/data/targets/_example-native-mobile.yml +2 -2
- package/data/targets/_example-sim-android-chrome.yml +1 -1
- package/data/targets/_example-sim-ios-safari.yml +1 -1
- package/data/templates/api-probe-matrix.md +3 -3
- package/data/templates/bug-report.md +29 -151
- package/data/templates/coverage-map.md +9 -6
- package/data/templates/expected-behaviour.md +3 -2
- package/data/templates/session-charter.md +3 -0
- package/data/templates/session-report.md +57 -80
- package/dist/cli/index.js +2286 -701
- package/dist/index.d.ts +637 -81
- package/dist/index.js +2058 -228
- package/docs/BACKEND-VERIFICATION.md +39 -9
- package/docs/GETTING-STARTED.md +300 -121
- package/docs/MOBILE-SETUP.md +149 -18
- package/hooks/hooks.json +27 -0
- package/hooks/scripts/read-guard.mjs +95 -0
- package/hooks/scripts/secret-patterns.d.mts +8 -0
- package/hooks/scripts/secret-patterns.mjs +136 -0
- package/hooks/scripts/write-guard.mjs +87 -0
- package/package.json +33 -18
- package/skills/qa-explore/SKILL.md +32 -25
- package/skills/qa-explore/phases/00-setup.md +76 -42
- package/skills/qa-explore/phases/01-auth.md +32 -15
- package/skills/qa-explore/phases/02-charter.md +16 -5
- package/skills/qa-explore/phases/03-discovery.md +93 -27
- package/skills/qa-explore/phases/04-journeys.md +2 -2
- package/skills/qa-explore/phases/05-features.md +39 -24
- package/skills/qa-explore/phases/06-edge-cases.md +24 -13
- package/skills/qa-explore/phases/07-reporting.md +102 -142
- package/skills/qa-explore/references/context-integration.md +1 -1
- package/skills/qa-explore/references/delegation-rules.md +82 -0
- package/skills/qa-explore/references/output-contract.md +192 -0
- package/skills/qa-explore/references/paths.md +108 -0
- package/skills/qa-explore/references/playwright-agents-integration.md +61 -84
- package/skills/qa-explore/references/security-rules.md +70 -25
- package/skills/qa-explore/references/session-rules.md +2 -1
- package/skills/qa-explore/references/severity-guide.md +31 -4
- package/skills/qa-explore-cleanup/SKILL.md +51 -15
- package/skills/qa-explore-feedback/SKILL.md +23 -5
- package/skills/qa-explore-mobile/SKILL.md +86 -64
- package/skills/qa-explore-mobile/phases/00-setup.md +71 -62
- package/skills/qa-explore-mobile/phases/01-auth.md +28 -31
- package/skills/qa-explore-mobile/phases/02-charter.md +8 -4
- package/skills/qa-explore-mobile/phases/03-discovery.md +17 -11
- package/skills/qa-explore-mobile/phases/04-journeys.md +6 -6
- package/skills/qa-explore-mobile/phases/05-features.md +9 -9
- package/skills/qa-explore-mobile/phases/06-edge-cases.md +35 -40
- package/skills/qa-explore-mobile/phases/07-reporting.md +108 -60
- package/skills/qa-explore-mobile/references/mobile-edges.md +26 -26
- package/skills/qa-explore-quick/SKILL.md +71 -26
- package/skills/qa-explore-report/SKILL.md +65 -35
- package/skills/qa-gather/SKILL.md +44 -8
- package/skills/qa-knowledge-add/SKILL.md +25 -9
- package/skills/qa-knowledge-list/SKILL.md +37 -40
- package/skills/qa-target-setup/SKILL.md +63 -41
- package/skills/qa-verify-backend/SKILL.md +31 -19
- package/skills/qa-verify-backend/phases/00-setup.md +57 -12
- package/skills/qa-verify-backend/phases/01-ac-decomposition.md +2 -2
- package/skills/qa-verify-backend/phases/02-static-review.md +11 -0
- package/skills/qa-verify-backend/phases/03-live-verification.md +5 -3
- package/skills/qa-verify-backend/phases/03b-api-verification.md +24 -9
- package/skills/qa-verify-backend/phases/04-e2e-trigger.md +18 -2
- package/skills/qa-verify-backend/phases/05-reporting.md +72 -12
- package/skills/qa-verify-backend/references/api-probes.md +28 -1
- package/skills/qa-verify-backend/references/payload-verification.md +1 -1
- package/skills/qa-verify-backend/references/release-readiness.md +1 -1
- package/skills/qa-verify-backend/references/safety-rules.md +13 -8
- package/data/domains/_default.md +0 -181
- package/data/domains/ecommerce.md +0 -192
- package/data/domains/fintech.md +0 -200
- package/data/domains/identity.md +0 -88
- package/data/domains/marketing.md +0 -194
- package/data/domains/saas.md +0 -197
- package/dist/cli/index.d.ts +0 -1
- package/dist/cli/index.js.map +0 -1
- package/dist/index.js.map +0 -1
|
@@ -1,27 +1,21 @@
|
|
|
1
1
|
# Phase 6: Edge Cases & Mobile-Specific (mode-aware)
|
|
2
2
|
|
|
3
|
-
Follow
|
|
3
|
+
Follow `${CLAUDE_SKILL_DIR}/../qa-explore/phases/06-edge-cases.md` for the saboteur mindset. Edges are tagged
|
|
4
4
|
**[both]**, **[native]**, or **[web]** — run the ones that fit the resolved mode.
|
|
5
5
|
|
|
6
|
-
Set up once per Bash block:
|
|
7
|
-
```bash
|
|
8
|
-
export MOBILE_CLI_STATE=/tmp/<target>-state.json
|
|
9
|
-
MCLI="$PWD/bin/mcli" # run from the repo root
|
|
10
|
-
```
|
|
11
|
-
|
|
12
6
|
## Dropped / limited (no DOM, no JS console)
|
|
13
7
|
|
|
14
8
|
- **XSS execution assertions via `fill` [web]** — without DOM/console you can't confirm execution. You can still submit payloads and watch for gross breakage (crash, broken render, reflected unescaped text in the a11y tree).
|
|
15
9
|
- **`route ... --status=500` network simulation [both]** — needs mitmproxy. **Deferred** — log scenarios that would benefit, run them once a proxy exists.
|
|
16
|
-
- **Multi-session race [both]** — needs two emulators. **Deferred
|
|
10
|
+
- **Multi-session race [both]** — needs two emulators. **Deferred.**
|
|
17
11
|
- **`resize` viewport tests** — sim viewport is fixed. Use **rotation** instead (below).
|
|
18
12
|
|
|
19
13
|
## Kept (still useful on mobile)
|
|
20
14
|
|
|
21
|
-
- **Unicode / emoji [both]** in inputs:
|
|
22
|
-
- **Whitespace-only [both]:**
|
|
15
|
+
- **Unicode / emoji [both]** in inputs: `qa/bin/mcli fill <ref> "naïve café 🤑 العربية"`
|
|
16
|
+
- **Whitespace-only [both]:** `qa/bin/mcli fill <ref> " "`
|
|
23
17
|
- **Extremely long [both]:** see the truncation caveat in `05-features.md`
|
|
24
|
-
- **State manipulation via deep link / URL [both]:** NATIVE
|
|
18
|
+
- **State manipulation via deep link / URL [both]:** NATIVE `qa/bin/mcli deep-link "<app-scheme>://<path>?<tampered>"`; WEB `qa/bin/mcli open-url "<base_url>/<deep view>?<tampered params>"`
|
|
25
19
|
- **Replay [both]:** double-tap submit; submit then back then re-submit
|
|
26
20
|
- **Brute force / rate limits [both]:** repeated failed submissions/logins — lockout / cooldown?
|
|
27
21
|
|
|
@@ -31,21 +25,20 @@ MCLI="$PWD/bin/mcli" # run from the repo root
|
|
|
31
25
|
|
|
32
26
|
```bash
|
|
33
27
|
# Background mid-flow
|
|
34
|
-
|
|
35
|
-
# (iOS: use the home gesture/button via the Simulator; press HOME is Android-only)
|
|
28
|
+
qa/bin/mcli press HOME # both platforms (iOS via Maestro pressKey Home)
|
|
36
29
|
sleep 30
|
|
37
|
-
|
|
30
|
+
qa/bin/mcli launch # foreground — did state restore or reload-and-lose-state?
|
|
38
31
|
|
|
39
32
|
# Kill mid-flow
|
|
40
|
-
|
|
41
|
-
|
|
33
|
+
qa/bin/mcli stop
|
|
34
|
+
qa/bin/mcli launch
|
|
42
35
|
# WEB only: re-open the view you were on
|
|
43
|
-
#
|
|
36
|
+
# qa/bin/mcli open-url <current view>
|
|
44
37
|
|
|
45
38
|
# OS-initiated suspend (Android)
|
|
46
|
-
bin/wadb shell am broadcast -a android.intent.action.SCREEN_OFF
|
|
39
|
+
qa/bin/wadb shell am broadcast -a android.intent.action.SCREEN_OFF
|
|
47
40
|
sleep 5
|
|
48
|
-
bin/wadb shell am broadcast -a android.intent.action.SCREEN_ON
|
|
41
|
+
qa/bin/wadb shell am broadcast -a android.intent.action.SCREEN_ON
|
|
49
42
|
```
|
|
50
43
|
|
|
51
44
|
For each: did state survive? Was auth still valid? Did an in-progress action complete or
|
|
@@ -55,8 +48,8 @@ duplicate?
|
|
|
55
48
|
|
|
56
49
|
NATIVE — deny / revoke / re-grant runtime permissions and check graceful handling:
|
|
57
50
|
```bash
|
|
58
|
-
bin/wadb shell pm revoke <app.package> android.permission.CAMERA # then try the QR/camera feature
|
|
59
|
-
bin/wadb shell pm grant <app.package> android.permission.CAMERA
|
|
51
|
+
qa/bin/wadb shell pm revoke <app.package> android.permission.CAMERA # then try the QR/camera feature
|
|
52
|
+
qa/bin/wadb shell pm grant <app.package> android.permission.CAMERA
|
|
60
53
|
# iOS sim: xcrun simctl privacy <udid> reset|grant camera <app.bundle_id>
|
|
61
54
|
```
|
|
62
55
|
Test camera / notifications / contacts / location. In WEB mode the same applies to the
|
|
@@ -69,7 +62,7 @@ If the app uses biometric auth: approve → succeeds; deny/cancel → password f
|
|
|
69
62
|
disable biometrics in OS mid-flow → graceful fallback. Android emulator can simulate a
|
|
70
63
|
fingerprint touch:
|
|
71
64
|
```bash
|
|
72
|
-
bin/wadb -e emu finger touch 1
|
|
65
|
+
qa/bin/wadb -e emu finger touch 1
|
|
73
66
|
```
|
|
74
67
|
|
|
75
68
|
### Push notifications [native]
|
|
@@ -81,11 +74,13 @@ works; trigger a notification and tap it — does it deep-link to the right scre
|
|
|
81
74
|
|
|
82
75
|
```bash
|
|
83
76
|
# Android — toggle network
|
|
84
|
-
bin/wadb shell svc wifi disable && bin/wadb shell svc data disable
|
|
77
|
+
qa/bin/wadb shell svc wifi disable && qa/bin/wadb shell svc data disable
|
|
85
78
|
# Run a submit/save flow — does it surface "offline" cleanly or hang/blank?
|
|
86
|
-
bin/wadb shell svc wifi enable && bin/wadb shell svc data enable
|
|
79
|
+
qa/bin/wadb shell svc wifi enable && qa/bin/wadb shell svc data enable
|
|
87
80
|
|
|
88
|
-
# Slow network
|
|
81
|
+
# Slow network (Android emulator console via adb, no telnet):
|
|
82
|
+
qa/bin/wadb emu network speed gsm && qa/bin/wadb emu network delay umts
|
|
83
|
+
# restore: qa/bin/wadb emu network speed full && qa/bin/wadb emu network delay none
|
|
89
84
|
```
|
|
90
85
|
|
|
91
86
|
Ask: does the app surface offline cleanly, queue or fail loudly, auto-retry, or just spin?
|
|
@@ -94,10 +89,10 @@ Ask: does the app surface offline cleanly, queue or fail loudly, auto-retry, or
|
|
|
94
89
|
|
|
95
90
|
```bash
|
|
96
91
|
# Android
|
|
97
|
-
bin/wadb shell settings put system accelerometer_rotation 0
|
|
98
|
-
bin/wadb shell settings put system user_rotation 1 # landscape
|
|
99
|
-
|
|
100
|
-
bin/wadb shell settings put system user_rotation 0 # portrait
|
|
92
|
+
qa/bin/wadb shell settings put system accelerometer_rotation 0
|
|
93
|
+
qa/bin/wadb shell settings put system user_rotation 1 # landscape
|
|
94
|
+
qa/bin/mcli snapshot # does the layout reflow? keyboard cover input?
|
|
95
|
+
qa/bin/wadb shell settings put system user_rotation 0 # portrait
|
|
101
96
|
```
|
|
102
97
|
|
|
103
98
|
iOS sim: Cmd+Right/Left Arrow in the Simulator to rotate. Check reflow, clipping, and that
|
|
@@ -113,15 +108,15 @@ and submit correctly?
|
|
|
113
108
|
|
|
114
109
|
```bash
|
|
115
110
|
# NATIVE — exercise the app's own scheme/intent:
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
bin/wadb shell am start -a android.intent.action.VIEW -d "<app-scheme>://<path>?x=$(python3 -c 'print("A"*5000)')"
|
|
111
|
+
qa/bin/mcli deep-link "<app-scheme>://<a real path>"
|
|
112
|
+
qa/bin/mcli deep-link "<app-scheme>://<path>?id=&n=NaN"
|
|
113
|
+
qa/bin/wadb shell am start -a android.intent.action.VIEW -d "<app-scheme>://<path>?x=$(python3 -c 'print("A"*5000)')"
|
|
119
114
|
|
|
120
115
|
# WEB — exercise the target URL:
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
116
|
+
qa/bin/mcli open-url "<base_url>/<a real deep view>"
|
|
117
|
+
qa/bin/mcli open-url "<base_url>/<view>?id=&n=NaN"
|
|
118
|
+
qa/bin/mcli open-url "<base_url>/<view>?next=javascript:alert(1)"
|
|
119
|
+
qa/bin/mcli open-url "<base_url>/<view>?q=$(python3 -c 'print("A"*5000)')"
|
|
125
120
|
```
|
|
126
121
|
|
|
127
122
|
Crash, freeze, blank screen, open-redirect, or accept-as-valid are all findings.
|
|
@@ -130,8 +125,8 @@ Crash, freeze, blank screen, open-redirect, or accept-as-valid are all findings.
|
|
|
130
125
|
|
|
131
126
|
```bash
|
|
132
127
|
# Navigate to a view with sensitive data, then open Recents and screenshot the preview
|
|
133
|
-
bin/wadb shell input keyevent KEYCODE_APP_SWITCH # Android Recents
|
|
134
|
-
|
|
128
|
+
qa/bin/wadb shell input keyevent KEYCODE_APP_SWITCH # Android Recents
|
|
129
|
+
qa/bin/mcli screenshot output/sessions/<session-dir>/screenshots/recents-preview.png
|
|
135
130
|
```
|
|
136
131
|
|
|
137
132
|
Is sensitive data (account info, tokens-in-URL, personal data) visible in the Recents preview?
|
|
@@ -140,7 +135,7 @@ finding where it matters.
|
|
|
140
135
|
|
|
141
136
|
### Storage / data wipe
|
|
142
137
|
|
|
143
|
-
- **NATIVE:** force-clear app data mid-flow (Android
|
|
138
|
+
- **NATIVE:** force-clear app data mid-flow (Android `qa/bin/mcli relaunch-clean` = force-stop + `pm clear` + launch) — does it land on a clean welcome state or get stuck on an intermediate screen? Also test fresh-install vs upgrade if you have both builds.
|
|
144
139
|
- **WEB:** **do NOT use `relaunch-clean`** — on Android it `pm clear`s the *browser* and wipes the logged-in profile, forcing a fresh SSO+MFA login. If you need a cleared-cookie test, do it knowingly and expect to re-login.
|
|
145
140
|
- Returning vs fresh state [both]: correct first-ever behavior (empty states, first-run interstitials) AND returning behavior.
|
|
146
141
|
|
|
@@ -151,7 +146,7 @@ extremes (very large numbers, long names, long localized strings at mobile width
|
|
|
151
146
|
|
|
152
147
|
## After Edge Cases
|
|
153
148
|
|
|
154
|
-
Write `phase-
|
|
149
|
+
Write `phase-6-edge-cases.md` (confidentiality header first) with one section per category. Each deferred test gets a single
|
|
155
150
|
line so it's visible.
|
|
156
151
|
|
|
157
152
|
**Update progress.**
|
|
@@ -1,84 +1,112 @@
|
|
|
1
1
|
# Phase 7: Reflection & Reporting (Mobile)
|
|
2
2
|
|
|
3
|
-
Follow
|
|
4
|
-
questions,
|
|
5
|
-
|
|
3
|
+
Follow `${CLAUDE_SKILL_DIR}/../qa-explore/phases/07-reporting.md` for the reflection
|
|
4
|
+
questions, the `phase-7-notes.md` sections and the hand-off to the reporting agent; every
|
|
5
|
+
artefact uses the format in
|
|
6
|
+
`${CLAUDE_SKILL_DIR}/../qa-explore/references/output-contract.md` (confidentiality header
|
|
7
|
+
first). The mobile differences (both modes):
|
|
6
8
|
|
|
7
9
|
## Stop recording
|
|
8
10
|
|
|
9
|
-
Replace `tracing-stop` with:
|
|
11
|
+
Replace `video-stop` / `tracing-stop` with:
|
|
10
12
|
|
|
11
13
|
```bash
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
$MCLI stop # close the app/browser — leaves the sim running for next time
|
|
14
|
+
qa/bin/mcli record-stop # closes the .mp4 (if record-start was used)
|
|
15
|
+
qa/bin/mcli stop # close the app/browser — leaves the sim running for next time
|
|
16
|
+
qa/bin/wk-ios --stop # iOS WEB mode only, if the DOM bridge was used
|
|
16
17
|
```
|
|
17
18
|
|
|
18
|
-
Do NOT `relaunch-clean` / wipe the device at session end
|
|
19
|
-
persisted login; in NATIVE mode it preserves any account/state created this session.
|
|
19
|
+
Do NOT `relaunch-clean` / wipe the device at session end: in WEB mode this preserves the
|
|
20
|
+
persisted login; in NATIVE mode it preserves any account/state created this session. There
|
|
21
|
+
is no playwright-cli session to `close` / `delete-data` on mobile.
|
|
20
22
|
|
|
21
|
-
## Bug
|
|
23
|
+
## Bug report — mobile fields
|
|
22
24
|
|
|
23
|
-
|
|
25
|
+
Same file and headings as the contract — `bugs/BUG-NNN.md`, starting with the two-line
|
|
26
|
+
header, verbatim:
|
|
27
|
+
|
|
28
|
+
```
|
|
29
|
+
> CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
|
|
30
|
+
> and application details. Do not share outside your organization without review.
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
then `# BUG-NNN: [Component] fails [Condition] causing [Impact]`, `**Severity:**`,
|
|
34
|
+
`**Priority:**`, `**Component:**`, `**URL:**`, `**Environment:**`,
|
|
35
|
+
`**Reproduction rate:**`, then `## Summary`, `## Expected Behavior`, `## Actual Behavior`,
|
|
36
|
+
`## Steps to Reproduce`, `## Business Impact`, `## Evidence` and
|
|
37
|
+
`## Recommended Fix Priority`. Two fields change on mobile:
|
|
24
38
|
|
|
25
39
|
```markdown
|
|
40
|
+
**URL:** <WEB: the page URL loaded via open-url | NATIVE: the screen name or deep link>
|
|
26
41
|
**Environment:**
|
|
27
|
-
- Mode:
|
|
28
|
-
- Platform:
|
|
29
|
-
- Device:
|
|
30
|
-
- OS version:
|
|
31
|
-
- App under test:
|
|
32
|
-
- Target URL:
|
|
33
|
-
- Environment:
|
|
42
|
+
- Mode: native | web
|
|
43
|
+
- Platform: android | ios
|
|
44
|
+
- Device: emulator-5554 | qa-iphone (UDID …)
|
|
45
|
+
- OS version: <android: qa/bin/wadb shell getprop ro.build.version.release — ios: xcrun simctl list runtimes>
|
|
46
|
+
- App under test: <native: app id + version + build type | web: browser bundle + version>
|
|
47
|
+
- Target URL: <web mode only: base_url>
|
|
48
|
+
- Environment: <preprod | staging | production>
|
|
34
49
|
```
|
|
35
50
|
|
|
36
|
-
And
|
|
51
|
+
And `## Evidence` carries the mobile artefacts:
|
|
37
52
|
|
|
38
53
|
```markdown
|
|
39
|
-
|
|
40
|
-
- Screenshot: `screenshots/
|
|
41
|
-
- Video
|
|
42
|
-
-
|
|
54
|
+
## Evidence
|
|
55
|
+
- Screenshot: `screenshots/BUG-NNN.png`
|
|
56
|
+
- Video: `videos/BUG-NNN.mp4` (if recorded)
|
|
57
|
+
- Log: `logs/BUG-NNN.log` (logcat / simctl excerpt around the failure — iOS is scoped to the app/browser process, Android is whole-device logcat, and neither is a page JS console)
|
|
58
|
+
- Console errors: n/a on mobile (or the wk-ios observation, iOS WEB)
|
|
59
|
+
- Network failures: n/a (no proxy) — or "needs proxy capture"
|
|
43
60
|
```
|
|
44
61
|
|
|
45
|
-
|
|
62
|
+
Capture the log at the moment of failure (the `logs/` directory was created in setup):
|
|
46
63
|
|
|
47
64
|
```bash
|
|
48
|
-
|
|
65
|
+
qa/bin/mcli screenshot output/sessions/<session-dir>/screenshots/BUG-NNN.png
|
|
66
|
+
qa/bin/mcli logs --since 30 --errors > output/sessions/<session-dir>/logs/BUG-NNN.log
|
|
49
67
|
```
|
|
50
68
|
|
|
51
|
-
|
|
69
|
+
Redact the log excerpt per `security-rules.md` before it is saved: process logs routinely
|
|
70
|
+
carry tokens and emails.
|
|
71
|
+
|
|
72
|
+
## phase-7-notes.md — mobile additions
|
|
52
73
|
|
|
53
|
-
|
|
74
|
+
Write `phase-7-notes.md` with the sections the web phase lists (`## Executive Summary`,
|
|
75
|
+
`## Coverage Map`, `## Observations`, `## Areas Not Tested`, `## Recommendations`,
|
|
76
|
+
`## Reflection`), then these two — they are copied into `session-report.md` after
|
|
77
|
+
`## Session Stats`:
|
|
54
78
|
|
|
55
79
|
```markdown
|
|
56
80
|
## Mobile Context
|
|
57
81
|
|
|
58
82
|
| Field | Value |
|
|
59
83
|
|---|---|
|
|
60
|
-
| Mode |
|
|
61
|
-
| Platform |
|
|
62
|
-
| Device |
|
|
63
|
-
| OS version |
|
|
64
|
-
| App under test |
|
|
65
|
-
| Target URL |
|
|
66
|
-
|
|
|
84
|
+
| Mode | native / web |
|
|
85
|
+
| Platform | android / ios |
|
|
86
|
+
| Device | <device label> |
|
|
87
|
+
| OS version | <version> |
|
|
88
|
+
| App under test | <native: app id + version + build type | web: browser bundle + version> |
|
|
89
|
+
| Target URL | <web mode only: base_url + environment> |
|
|
90
|
+
| Source state | <native: branch + sha + clean/dirty, if rebuilt> |
|
|
91
|
+
| Driver version | <output of `qa/bin/mcli version`> |
|
|
67
92
|
|
|
68
93
|
## Deferred Tests (require additional tooling)
|
|
69
94
|
|
|
70
95
|
| Test | Why deferred | What's needed |
|
|
71
96
|
|---|---|---|
|
|
72
|
-
| Network failure simulation | mobile-cli
|
|
73
|
-
| Multi-device race conditions |
|
|
74
|
-
| Server-side parameter tampering (non-deep-link) |
|
|
97
|
+
| Network failure simulation | mobile-cli has no proxy | mitmproxy + cert install |
|
|
98
|
+
| Multi-device race conditions | single device | second emulator + parallel sessions |
|
|
99
|
+
| Server-side parameter tampering (non-deep-link) | no script eval on device | proxy + request modification |
|
|
75
100
|
```
|
|
76
101
|
|
|
102
|
+
Every deferred test also appears in the coverage map as `not-tested` with the reason, so it
|
|
103
|
+
shows up as a known gap.
|
|
104
|
+
|
|
77
105
|
## Requirements coverage cross-check (optional)
|
|
78
106
|
|
|
79
|
-
If the session was scoped to gathered requirements / tickets, add a cross-check table
|
|
80
|
-
|
|
81
|
-
designed?":
|
|
107
|
+
If the session was scoped to gathered requirements / tickets, add a cross-check table to the
|
|
108
|
+
notes; it is carried into the report with the other sections. This is **not** test execution;
|
|
109
|
+
it is "did exploration touch the areas those requirements designed?":
|
|
82
110
|
|
|
83
111
|
```markdown
|
|
84
112
|
## Requirements Cross-Check
|
|
@@ -90,34 +118,54 @@ Source: <ticket / acceptance criteria / gathered context>
|
|
|
90
118
|
| ... | Yes/No/Partial | BUG-NNN or — | ... |
|
|
91
119
|
```
|
|
92
120
|
|
|
93
|
-
The point is to surface coverage gaps the team can decide to fill manually or with automation.
|
|
94
|
-
|
|
95
121
|
## stats.json — mobile fields
|
|
96
122
|
|
|
97
|
-
|
|
123
|
+
`stats.json` exactly as the contract defines it, with `"kind": "mobile"`, and the device
|
|
124
|
+
block under `coverage.mobile` (the schema is strict; no other top-level keys):
|
|
98
125
|
|
|
99
126
|
```json
|
|
100
|
-
"
|
|
101
|
-
"
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
127
|
+
"coverage": {
|
|
128
|
+
"mobile": {
|
|
129
|
+
"mode": "web",
|
|
130
|
+
"platform": "ios",
|
|
131
|
+
"device": "qa-iphone",
|
|
132
|
+
"app_under_test": "com.apple.mobilesafari",
|
|
133
|
+
"target_url": "https://staging.m.example.com",
|
|
134
|
+
"environment": "staging",
|
|
135
|
+
"videos_recorded": 1,
|
|
136
|
+
"log_errors_observed": 0,
|
|
137
|
+
"lifecycle_tests_run": 0,
|
|
138
|
+
"rotation_tests_run": 0,
|
|
139
|
+
"deferred_tests": []
|
|
140
|
+
}
|
|
112
141
|
}
|
|
113
142
|
```
|
|
114
143
|
|
|
115
144
|
(For native mode set `"mode": "native"`, `app_under_test` to the app id, and `target_url` to null.)
|
|
116
145
|
|
|
117
|
-
##
|
|
146
|
+
## Assemble and Finalize
|
|
147
|
+
|
|
148
|
+
Invoke the `qa-reporting-agent` sub-agent (`qualiow:qa-reporting-agent` under a plugin
|
|
149
|
+
install) with the session directory and `kind: mobile`. It assembles `session-report.md` from
|
|
150
|
+
`phase-7-notes.md`, the bugs, `stats.json` and the phase files — keeping `## Mobile Context`
|
|
151
|
+
and `## Deferred Tests` after `## Session Stats` — and then runs
|
|
152
|
+
`qualiow session finalize`, which validates `stats.json` against the strict schema, checks the
|
|
153
|
+
confidentiality header on every artefact and scans the directory against the redaction list —
|
|
154
|
+
the logcat / simctl excerpts under `logs/` are scanned like everything else — then appends the
|
|
155
|
+
session row to `output/sessions/INDEX.md` (`Kind` = `mobile`) and one row per bug to
|
|
156
|
+
`output/bugs/all-bugs.md`, records the session metrics and sets `progress.json` to
|
|
157
|
+
`complete`. It is idempotent.
|
|
158
|
+
|
|
159
|
+
Read its return (at most 8 lines). If it reports a missing notes section or a violation it
|
|
160
|
+
could not fix, fix that yourself and re-run the check:
|
|
118
161
|
|
|
119
|
-
|
|
162
|
+
```bash
|
|
163
|
+
qualiow session finalize output/sessions/<session-dir> --check
|
|
164
|
+
```
|
|
120
165
|
|
|
121
|
-
|
|
166
|
+
`--check` writes nothing and prints a numbered list of violations naming each file. Resolve
|
|
167
|
+
the `qualiow` prefix per `${CLAUDE_SKILL_DIR}/../qa-explore/references/paths.md`; if the CLI
|
|
168
|
+
is unavailable, write the rows by hand in the column order defined there.
|
|
122
169
|
|
|
123
|
-
|
|
170
|
+
If sub-agents are unavailable, do the step yourself as in 2.1.0: write `session-report.md`
|
|
171
|
+
from the notes in the contract's format and run `qualiow session finalize output/sessions/<session-dir>`.
|
|
@@ -3,18 +3,18 @@
|
|
|
3
3
|
Treat as a thinking tool, not a checkbox sheet. Pick what matches the feature and the resolved
|
|
4
4
|
**mode**. Each entry is tagged **[both]**, **[native]**, or **[web]**, with the adb / simctl
|
|
5
5
|
trigger where possible. `<app>` = the app under test's package/bundle in NATIVE mode, or the
|
|
6
|
-
browser bundle (`com.android.chrome` / `com.apple.mobilesafari`) in WEB mode.
|
|
7
|
-
`bin/mcli`
|
|
6
|
+
browser bundle (`com.android.chrome` / `com.apple.mobilesafari`) in WEB mode. `qa/bin/mcli` is the
|
|
7
|
+
driver prefix resolved in phase 0 (`bin/mcli` in a checkout); use `qa/bin/wadb` for raw adb.
|
|
8
8
|
|
|
9
9
|
## Lifecycle [both]
|
|
10
10
|
|
|
11
11
|
| Edge | Android trigger | iOS sim trigger |
|
|
12
12
|
|---|---|---|
|
|
13
|
-
| Background app/browser | `bin/
|
|
14
|
-
| Kill app/browser | `bin/wadb shell am force-stop <app>` | `xcrun simctl terminate <udid> <app>` |
|
|
15
|
-
| Cold relaunch |
|
|
16
|
-
| Clear data | NATIVE:
|
|
17
|
-
| Screen off / on | `bin/wadb shell input keyevent KEYCODE_POWER` (toggle) | n/a (sim doesn't lock realistically) |
|
|
13
|
+
| Background app/browser | `qa/bin/mcli press HOME` | `qa/bin/mcli press HOME` (Maestro `pressKey: Home`) |
|
|
14
|
+
| Kill app/browser | `qa/bin/wadb shell am force-stop <app>` | `xcrun simctl terminate <udid> <app>` |
|
|
15
|
+
| Cold relaunch | `qa/bin/mcli launch` (WEB: then `qa/bin/mcli open-url <url>`) | `qa/bin/mcli launch` (WEB: then `qa/bin/mcli open-url <url>`) |
|
|
16
|
+
| Clear data | NATIVE: `qa/bin/mcli relaunch-clean` (= force-stop + pm clear + launch) — clean-state test. WEB: same command **wipes the logged-in profile; avoid unless intended** | NATIVE: `xcrun simctl uninstall` + reinstall. WEB: clear Safari data in sim Settings (drops the login) |
|
|
17
|
+
| Screen off / on | `qa/bin/wadb shell input keyevent KEYCODE_POWER` (toggle) | n/a (sim doesn't lock realistically) |
|
|
18
18
|
|
|
19
19
|
**Ask:** does state survive? Does an in-progress action complete? Does auth/session persist, or is everything reloaded?
|
|
20
20
|
|
|
@@ -26,7 +26,7 @@ geolocation, notifications. Use `<app>` = app package (native) or browser (web).
|
|
|
26
26
|
|
|
27
27
|
| Permission | Android revoke | Android grant | iOS sim |
|
|
28
28
|
|---|---|---|---|
|
|
29
|
-
| Camera | `bin/wadb shell pm revoke <app> android.permission.CAMERA` | `pm grant` | `xcrun simctl privacy <udid> reset camera <app>` (then re-prompt) |
|
|
29
|
+
| Camera | `qa/bin/wadb shell pm revoke <app> android.permission.CAMERA` | `pm grant` | `xcrun simctl privacy <udid> reset camera <app>` (then re-prompt) |
|
|
30
30
|
| Location | `pm revoke <app> android.permission.ACCESS_FINE_LOCATION` | `pm grant` | `xcrun simctl privacy <udid> reset location <app>` |
|
|
31
31
|
| Notifications | `pm revoke <app> android.permission.POST_NOTIFICATIONS` | `pm grant` | `xcrun simctl privacy <udid> reset notifications <app>` |
|
|
32
32
|
| Contacts [native] | `pm revoke <app> android.permission.READ_CONTACTS` | `pm grant` | `xcrun simctl privacy <udid> reset contacts <app>` |
|
|
@@ -37,7 +37,7 @@ geolocation, notifications. Use `<app>` = app package (native) or browser (web).
|
|
|
37
37
|
|
|
38
38
|
App-level biometric auth only (a real sim browser doesn't expose a biometric flow to web).
|
|
39
39
|
Approve → succeeds; deny/cancel → password fallback; disable in OS mid-flow → graceful
|
|
40
|
-
fallback. Android emulator can simulate a fingerprint: `bin/wadb -e emu finger touch 1`.
|
|
40
|
+
fallback. Android emulator can simulate a fingerprint: `qa/bin/wadb -e emu finger touch 1`.
|
|
41
41
|
|
|
42
42
|
## Push notifications [native]
|
|
43
43
|
|
|
@@ -48,9 +48,9 @@ notification and tap it — does it deep-link to the right screen with context?
|
|
|
48
48
|
|
|
49
49
|
| Edge | Android | iOS sim |
|
|
50
50
|
|---|---|---|
|
|
51
|
-
| Airplane mode | `bin/wadb shell svc wifi disable && bin/wadb shell svc data disable` (re-enable to undo) | Network Link Conditioner (manual) |
|
|
52
|
-
| Slow network |
|
|
53
|
-
| Captive portal | hard — skip
|
|
51
|
+
| Airplane mode | `qa/bin/wadb shell svc wifi disable && qa/bin/wadb shell svc data disable` (re-enable to undo) | Network Link Conditioner (manual) |
|
|
52
|
+
| Slow network | `qa/bin/wadb emu network speed gsm` (restore with `speed full`) | Network Link Conditioner |
|
|
53
|
+
| Captive portal | hard — skip | hard — skip |
|
|
54
54
|
|
|
55
55
|
**Ask:** does the app surface offline cleanly? Does it queue actions or fail loudly? Does it
|
|
56
56
|
auto-retry? Does it leak retries (battery drain)?
|
|
@@ -59,10 +59,10 @@ auto-retry? Does it leak retries (battery drain)?
|
|
|
59
59
|
|
|
60
60
|
| Edge | Android | iOS sim |
|
|
61
61
|
|---|---|---|
|
|
62
|
-
| Rotation portrait/landscape | `bin/wadb shell settings put system user_rotation 0/1/2/3` | Cmd+Right/Left Arrow in Simulator |
|
|
63
|
-
| Dark mode | `bin/wadb shell "cmd uimode night yes"` (and `no`) | `xcrun simctl ui <udid> appearance dark/light` |
|
|
64
|
-
| Dynamic type / large font | `bin/wadb shell settings put system font_scale 1.5` | Settings app inside sim |
|
|
65
|
-
| Low battery | `bin/wadb shell dumpsys battery set level 10` (and `reset`) | n/a |
|
|
62
|
+
| Rotation portrait/landscape | `qa/bin/wadb shell settings put system user_rotation 0/1/2/3` | Cmd+Right/Left Arrow in Simulator |
|
|
63
|
+
| Dark mode | `qa/bin/wadb shell "cmd uimode night yes"` (and `no`) | `xcrun simctl ui <udid> appearance dark/light` |
|
|
64
|
+
| Dynamic type / large font | `qa/bin/wadb shell settings put system font_scale 1.5` | Settings app inside sim |
|
|
65
|
+
| Low battery | `qa/bin/wadb shell dumpsys battery set level 10` (and `reset`) | n/a |
|
|
66
66
|
| Split-screen | manual via Recents | n/a |
|
|
67
67
|
|
|
68
68
|
**Ask:** does the layout reflow, clip, or hide content? Does it honor the OS dark-mode /
|
|
@@ -74,15 +74,15 @@ height (`100vh` jumps), momentum/overscroll bounce, and whether the page respect
|
|
|
74
74
|
|
|
75
75
|
```bash
|
|
76
76
|
# NATIVE — the app's own scheme/intent
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
bin/wadb shell am start -W -a android.intent.action.VIEW -d "<app-scheme>://<path>" -f 0x10000000
|
|
77
|
+
qa/bin/mcli deep-link "<app-scheme>://<a real path>"
|
|
78
|
+
qa/bin/mcli deep-link "<app-scheme>://<path>?id=&n=NaN"
|
|
79
|
+
qa/bin/wadb shell am start -W -a android.intent.action.VIEW -d "<app-scheme>://<path>" -f 0x10000000
|
|
80
80
|
|
|
81
81
|
# WEB — the target URL
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
82
|
+
qa/bin/mcli open-url "<base_url>/<a real deep view>"
|
|
83
|
+
qa/bin/mcli open-url "<base_url>/<view>?id=&n=NaN"
|
|
84
|
+
qa/bin/mcli open-url "<base_url>/<view>?next=javascript:alert(1)"
|
|
85
|
+
qa/bin/mcli open-url "<base_url>/<view>?q=$(python3 -c 'print("A"*5000)')"
|
|
86
86
|
```
|
|
87
87
|
|
|
88
88
|
**Ask:** crash, freeze, blank screen, open-redirect, or unsanitized reflection?
|
|
@@ -90,7 +90,7 @@ $MCLI open-url "<base_url>/<view>?q=$(python3 -c 'print("A"*5000)')"
|
|
|
90
90
|
## Privacy / leak surface [both]
|
|
91
91
|
|
|
92
92
|
- **Recents preview:** background while on a sensitive screen, screencap Recents — is personal data / a token-in-URL visible? A native app can mask its preview (FLAG_SECURE); web pages generally can't.
|
|
93
|
-
- **Process log:**
|
|
93
|
+
- **Process log:** `qa/bin/mcli logs --errors` — grep for PII or credentials (emails, tokens, secrets) leaking into the app/browser process log.
|
|
94
94
|
- **URL leakage [web]:** sensitive values in query strings end up in history and Recents — flag where they shouldn't be.
|
|
95
95
|
- **Clipboard:** if the app copies sensitive text, does it auto-clear/expire?
|
|
96
96
|
|
|
@@ -103,10 +103,10 @@ $MCLI open-url "<base_url>/<view>?q=$(python3 -c 'print("A"*5000)')"
|
|
|
103
103
|
|
|
104
104
|
## Native a11y [both] (driven through the device a11y tree)
|
|
105
105
|
|
|
106
|
-
- TalkBack (Android): `bin/wadb shell settings put secure enabled_accessibility_services com.google.android.marvin.talkback/.TalkBackService`
|
|
106
|
+
- TalkBack (Android): `qa/bin/wadb shell settings put secure enabled_accessibility_services com.google.android.marvin.talkback/.TalkBackService`
|
|
107
107
|
- VoiceOver (iOS sim): mostly limited on sim — manual via the accessibility menu
|
|
108
108
|
- **Focus order:** swipe right repeatedly — does focus land on every interactive element in a sensible order?
|
|
109
|
-
- **Aggregation:** does each control announce its own label, or does the view dump many fields into one comma-joined label? (Known iOS issue — see
|
|
109
|
+
- **Aggregation:** does each control announce its own label, or does the view dump many fields into one comma-joined label? (Known iOS issue — see `${CLAUDE_SKILL_DIR}/phases/03-discovery.md`.)
|
|
110
110
|
- **Contrast:** spot-check critical text — sufficient contrast in both light and dark mode?
|
|
111
111
|
- **Touch targets:** are tap targets finger-sized, or desktop-sized and hard to hit on a phone?
|
|
112
112
|
|