qualiow-exploratory-testing 1.2.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +22 -0
- package/.env.example +27 -0
- package/CHANGELOG.md +194 -0
- package/README.md +428 -179
- package/agents/qa-gather-agent.md +24 -0
- package/{scripts → bin}/doctor-mobile.sh +57 -18
- package/bin/mcli +15 -9
- package/bin/mobile-cli.mjs +574 -286
- package/{scripts → bin}/setup-mobile.sh +56 -19
- package/bin/wadb +10 -4
- package/bin/wk-ios +29 -17
- package/bin/wkeval.mjs +7 -1
- package/data/domains/_default.yml +1 -1
- package/data/domains/identity.yml +138 -0
- package/data/knowledge/changelog.yml +104 -0
- package/data/knowledge/learned-patterns.md +172 -86
- package/data/knowledge/manifest.yml +481 -70
- package/data/knowledge/releases/v0.1.0/entries/technique-business-logic-race-conditions.yml +1 -1
- package/data/knowledge/releases/v0.1.0/release.yml +8 -1
- package/data/knowledge/releases/v0.2.0/entries/technique-contract-narrowing.yml +150 -0
- package/data/knowledge/releases/v0.2.0/entries/technique-test-suite-audit.yml +116 -0
- package/data/knowledge/releases/v0.2.0/release.yml +20 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-functional-diff-analysis.yml +119 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-llm-output-verification.yml +102 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-verification-mode-selection.yml +153 -0
- package/data/knowledge/releases/v0.3.0/release.yml +25 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-authenticated-api-probing.yml +137 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-environment-fingerprinting.yml +113 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-silent-failure-audit.yml +86 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-ui-api-differential.yml +97 -0
- package/data/knowledge/releases/v0.4.0/release.yml +28 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-config-surface-verification.yml +77 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-derived-value-verification.yml +81 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-expected-behaviour-specification.yml +97 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-presentation-integrity.yml +78 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-release-readiness-verification.yml +110 -0
- package/data/knowledge/releases/v0.5.0/release.yml +33 -0
- package/data/knowledge/releases/v0.6.0/entries/technique-async-callback-contracts.yml +98 -0
- package/data/knowledge/releases/v0.6.0/entries/technique-exactly-once-verification.yml +116 -0
- package/data/knowledge/releases/v0.6.0/release.yml +30 -0
- package/data/security/SECURITY-POLICY.md +273 -0
- package/data/targets/_default.yml +1 -1
- package/data/targets/_example-api-only.yml +80 -0
- package/data/targets/_example-backend.yml +123 -0
- package/data/targets/_example-mobile-emulation.yml +2 -2
- package/data/targets/_example-native-mobile.yml +2 -2
- package/data/targets/_example-sim-android-chrome.yml +1 -1
- package/data/targets/_example-sim-ios-safari.yml +1 -1
- package/data/templates/api-probe-matrix.md +99 -0
- package/data/templates/bug-report.md +29 -151
- package/data/templates/coverage-map.md +23 -6
- package/data/templates/expected-behaviour.md +94 -0
- package/data/templates/session-charter.md +3 -0
- package/data/templates/session-report.md +57 -80
- package/dist/cli/index.js +1500 -739
- package/dist/index.d.ts +673 -75
- package/dist/index.js +1664 -270
- package/docs/BACKEND-VERIFICATION.md +307 -0
- package/docs/GETTING-STARTED.md +188 -128
- package/docs/MOBILE-SETUP.md +149 -18
- package/package.json +32 -16
- package/skills/qa-explore/SKILL.md +31 -25
- package/skills/qa-explore/phases/00-setup.md +67 -41
- package/skills/qa-explore/phases/01-auth.md +32 -15
- package/skills/qa-explore/phases/02-charter.md +16 -5
- package/skills/qa-explore/phases/03-discovery.md +50 -26
- package/skills/qa-explore/phases/04-journeys.md +2 -2
- package/skills/qa-explore/phases/05-features.md +39 -24
- package/skills/qa-explore/phases/06-edge-cases.md +24 -13
- package/skills/qa-explore/phases/07-reporting.md +79 -140
- package/skills/qa-explore/references/context-integration.md +1 -1
- package/skills/qa-explore/references/output-contract.md +187 -0
- package/skills/qa-explore/references/paths.md +84 -0
- package/skills/qa-explore/references/playwright-agents-integration.md +61 -84
- package/skills/qa-explore/references/security-rules.md +66 -25
- package/skills/qa-explore/references/severity-guide.md +31 -4
- package/skills/qa-explore-cleanup/SKILL.md +17 -6
- package/skills/qa-explore-feedback/SKILL.md +18 -5
- package/skills/qa-explore-mobile/SKILL.md +86 -64
- package/skills/qa-explore-mobile/phases/00-setup.md +71 -62
- package/skills/qa-explore-mobile/phases/01-auth.md +28 -31
- package/skills/qa-explore-mobile/phases/02-charter.md +8 -4
- package/skills/qa-explore-mobile/phases/03-discovery.md +17 -11
- package/skills/qa-explore-mobile/phases/04-journeys.md +6 -6
- package/skills/qa-explore-mobile/phases/05-features.md +9 -9
- package/skills/qa-explore-mobile/phases/06-edge-cases.md +35 -40
- package/skills/qa-explore-mobile/phases/07-reporting.md +82 -56
- package/skills/qa-explore-mobile/references/mobile-edges.md +26 -26
- package/skills/qa-explore-quick/SKILL.md +60 -25
- package/skills/qa-explore-report/SKILL.md +40 -34
- package/skills/qa-gather/SKILL.md +17 -2
- package/skills/qa-knowledge-add/SKILL.md +20 -9
- package/skills/qa-knowledge-list/SKILL.md +12 -3
- package/skills/qa-target-setup/SKILL.md +63 -41
- package/skills/qa-verify-backend/SKILL.md +172 -0
- package/skills/qa-verify-backend/phases/00-setup.md +142 -0
- package/skills/qa-verify-backend/phases/01-ac-decomposition.md +58 -0
- package/skills/qa-verify-backend/phases/02-static-review.md +71 -0
- package/skills/qa-verify-backend/phases/03-live-verification.md +62 -0
- package/skills/qa-verify-backend/phases/03b-api-verification.md +170 -0
- package/skills/qa-verify-backend/phases/04-e2e-trigger.md +82 -0
- package/skills/qa-verify-backend/phases/05-reporting.md +163 -0
- package/skills/qa-verify-backend/references/api-probes.md +168 -0
- package/skills/qa-verify-backend/references/aws-readonly-probes.md +156 -0
- package/skills/qa-verify-backend/references/environment-fingerprinting.md +104 -0
- package/skills/qa-verify-backend/references/payload-verification.md +112 -0
- package/skills/qa-verify-backend/references/release-readiness.md +97 -0
- package/skills/qa-verify-backend/references/safety-rules.md +95 -0
- package/dist/cli/index.d.ts +0 -1
- package/dist/cli/index.js.map +0 -1
- package/dist/index.js.map +0 -1
|
@@ -1,12 +1,12 @@
|
|
|
1
1
|
# Phase 2: Business Context & Charter (Mobile)
|
|
2
2
|
|
|
3
|
-
Follow
|
|
3
|
+
Follow `${CLAUDE_SKILL_DIR}/../qa-explore/phases/02-charter.md` for the structure (use the app as a user,
|
|
4
4
|
identify journeys, risk-rank, select heuristics, write charter). The mobile additions below
|
|
5
5
|
apply to both modes.
|
|
6
6
|
|
|
7
7
|
## Use the app as a real user — for ~3 min
|
|
8
8
|
|
|
9
|
-
Drive the system under test via
|
|
9
|
+
Drive the system under test via `qa/bin/mcli snapshot` → `click` → `snapshot` loops to reach entry
|
|
10
10
|
points (NATIVE: navigate the app's screens; WEB: `open-url` plus in-page links). Don't test
|
|
11
11
|
yet — just observe what it exists to do, on a phone-sized screen, the way a real mobile user
|
|
12
12
|
would. Note the primary task, where money/data/trust is at stake, and what a frustrated user
|
|
@@ -44,13 +44,17 @@ dimension** appropriate to the feature under charter, e.g.:
|
|
|
44
44
|
Replace qa-explore's `tracing-start` with a video:
|
|
45
45
|
|
|
46
46
|
```bash
|
|
47
|
-
|
|
47
|
+
qa/bin/mcli record-start output/sessions/<session-dir>/videos/session.mp4 # a real .mp4 on both platforms
|
|
48
48
|
```
|
|
49
49
|
|
|
50
|
+
On **Android** one recording is capped at 180 s and everything after it is lost with no
|
|
51
|
+
error: `record-stop` / `record-start` per journey instead of one session-long clip. iOS has
|
|
52
|
+
no cap.
|
|
53
|
+
|
|
50
54
|
If recording fails or is too heavy, take frequent screenshots instead. Recording is
|
|
51
55
|
nice-to-have, not required.
|
|
52
56
|
|
|
53
|
-
## Charter additions (append to the qa-explore charter template)
|
|
57
|
+
## Charter additions (append to the qa-explore charter template; confidentiality header first)
|
|
54
58
|
|
|
55
59
|
```markdown
|
|
56
60
|
## Mobile Context
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Phase 3: Discovery & Mapping (Mobile)
|
|
2
2
|
|
|
3
|
-
Follow
|
|
3
|
+
Follow `${CLAUDE_SKILL_DIR}/../qa-explore/phases/03-discovery.md` for goal and method (map structure,
|
|
4
4
|
refine risk). The substitutions below apply to both modes.
|
|
5
5
|
|
|
6
6
|
## Map screens/views by name + nav path (not URLs)
|
|
@@ -10,9 +10,9 @@ the nav path through the app; WEB: the URL that `open-url`/a link loaded, record
|
|
|
10
10
|
For each reachable screen/view:
|
|
11
11
|
|
|
12
12
|
```bash
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
13
|
+
qa/bin/mcli snapshot # refs expire after 60 s — re-snapshot before acting
|
|
14
|
+
qa/bin/mcli screenshot output/sessions/<session-dir>/screenshots/discovery-<view>.png
|
|
15
|
+
qa/bin/mcli logs --errors --since 30
|
|
16
16
|
```
|
|
17
17
|
|
|
18
18
|
Log each: `[DISCOVERED] <view-name> — <one-line description> — Risk: <P0/P1/P2/P3> — entry: <how-to-reach>`
|
|
@@ -22,18 +22,23 @@ Log each: `[DISCOVERED] <view-name> — <one-line description> — Risk: <P0/P1/
|
|
|
22
22
|
`snapshot` reads the device **accessibility tree, not the DOM**, so you identify elements by
|
|
23
23
|
their **visible text and accessibility labels** (headings, links, buttons, inputs), not by
|
|
24
24
|
CSS selectors or a fixed testID map. Read the live snapshot tree on each view. When a control
|
|
25
|
-
has no usable accessible label, fall back to
|
|
25
|
+
has no usable accessible label, fall back to `snapshot --full`, which assigns a ref to every
|
|
26
|
+
bounded node — `click <ref>` taps that node's centre on both platforms. A raw coordinate tap
|
|
27
|
+
exists only on Android (`qa/bin/wadb shell input tap <x> <y>`); the driver has no iOS
|
|
28
|
+
equivalent.
|
|
26
29
|
|
|
27
30
|
## App/browser log instead of console
|
|
28
31
|
|
|
29
32
|
Replace `playwright-cli console error` with:
|
|
30
33
|
|
|
31
34
|
```bash
|
|
32
|
-
|
|
35
|
+
qa/bin/mcli logs --errors --since 60
|
|
33
36
|
```
|
|
34
37
|
|
|
35
|
-
Caveat: this is the **process log
|
|
36
|
-
app/browser process
|
|
38
|
+
Caveat: this is the **process log**, and only iOS is process-scoped: `simctl spawn log show`
|
|
39
|
+
filters on the app/browser process, while Android's `logcat` is the **whole device** at `*:E`
|
|
40
|
+
/ `*:W` (last 400 lines), so lines from other packages appear — correlate by tag/pid before
|
|
41
|
+
attributing one to the app. In WEB mode it is the BROWSER's log, NOT the page's JS console, so it
|
|
37
42
|
catches gross failures (renderer crashes) but misses most page-level `console.error`. In
|
|
38
43
|
NATIVE mode it is the app's own log (crashes, native exceptions, bridge errors). Treat a clean
|
|
39
44
|
log as "no gross crash", not "no errors". Still watch for:
|
|
@@ -65,12 +70,13 @@ Beyond the qa-explore "what's not here" check, ask:
|
|
|
65
70
|
On iOS, some views **aggregate child elements into one parent accessibility label** (e.g. a
|
|
66
71
|
form returns one big comma-joined label of several fields). When you see a single ref with a
|
|
67
72
|
comma-joined label of many controls, that's an iOS a11y aggregation issue — file as a Medium
|
|
68
|
-
bug (it blocks automation and hurts VoiceOver users) and
|
|
69
|
-
|
|
73
|
+
bug (it blocks automation and hurts VoiceOver users) and drive that view with
|
|
74
|
+
`snapshot --full` refs for the rest of the session (on Android you can also fall back to
|
|
75
|
+
`qa/bin/wadb shell input tap <x> <y>`; iOS has no raw-coordinate tap).
|
|
70
76
|
|
|
71
77
|
## After Discovery
|
|
72
78
|
|
|
73
|
-
Write findings to `phase-
|
|
79
|
+
Write findings to `phase-3-discovery.md` (confidentiality header first) including:
|
|
74
80
|
- View map (table: view → key visible labels/controls → entry path → URL if known)
|
|
75
81
|
- Log error summary (count by level) with the browser-log caveat noted
|
|
76
82
|
- iOS a11y aggregation hits (if any)
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Phase 4: End-to-End User Journeys (Mobile)
|
|
2
2
|
|
|
3
|
-
Follow
|
|
3
|
+
Follow `${CLAUDE_SKILL_DIR}/../qa-explore/phases/04-journeys.md` for goal and the data-integrity
|
|
4
4
|
discipline. Build the journey templates from the charter's top user goals. Generic
|
|
5
5
|
mobile journey shapes to instantiate (apply to both modes):
|
|
6
6
|
|
|
@@ -10,7 +10,7 @@ mobile journey shapes to instantiate (apply to both modes):
|
|
|
10
10
|
2. **Create / edit / save** — create or edit a record, save it, navigate away and back, confirm the change is reflected (not just an optimistic UI that drops on reload/relaunch).
|
|
11
11
|
3. **Multi-step flow / wizard** — go through a multi-step flow start to finish; then go partway, back out, and resume — does it preserve or correctly reset state?
|
|
12
12
|
4. **Cross-screen consistency** — a value shown on one screen (a count, total, status, balance, name) matches everywhere else it appears.
|
|
13
|
-
5. **Deep entry** — reach a deep screen directly (NATIVE: deep link via
|
|
13
|
+
5. **Deep entry** — reach a deep screen directly (NATIVE: deep link via `qa/bin/mcli deep-link`; WEB: `open-url` a deep view) rather than navigating from home — does it load with auth/context intact, or break?
|
|
14
14
|
|
|
15
15
|
Concrete examples:
|
|
16
16
|
- **WEB:** login/SSO → landing; the core transactional journey end-to-end (e.g. search → product → basket → checkout); any onboarding wizard; editing settings and confirming persistence; exercising the mobile form elements (pickers, toggles, date/number inputs).
|
|
@@ -31,16 +31,16 @@ Log every check: `[DATA CHECK] <oracle> — expected: <X>, actual: <Y> — MATCH
|
|
|
31
31
|
## Lifecycle interleaving (mobile-only addition)
|
|
32
32
|
|
|
33
33
|
For at least ONE journey, interrupt mid-flow and verify recovery:
|
|
34
|
-
- Background the app/browser via
|
|
35
|
-
- Kill via
|
|
36
|
-
- Toggle airplane mode mid-submit (`bin/wadb shell svc wifi disable; bin/wadb shell svc data disable`) — does the app surface the failure cleanly or hang/silently drop the action? Re-enable to undo.
|
|
34
|
+
- Background the app/browser via `qa/bin/mcli press HOME` (both platforms — iOS goes through Maestro `pressKey: Home`), wait 30s, return — does state restore, or reload-and-lose-input?
|
|
35
|
+
- Kill via `qa/bin/mcli stop`, relaunch via `qa/bin/mcli launch` (WEB: + `open-url`) — does in-progress work survive? Does auth survive?
|
|
36
|
+
- Toggle airplane mode mid-submit (`qa/bin/wadb shell svc wifi disable; qa/bin/wadb shell svc data disable`) — does the app surface the failure cleanly or hang/silently drop the action? Re-enable to undo.
|
|
37
37
|
|
|
38
38
|
These lifecycle results often surface the most impactful bugs (data loss, stuck states,
|
|
39
39
|
silent failures, duplicate submissions).
|
|
40
40
|
|
|
41
41
|
## After Journeys
|
|
42
42
|
|
|
43
|
-
Write findings to `phase-
|
|
43
|
+
Write findings to `phase-4-journeys.md` (confidentiality header first) with:
|
|
44
44
|
- Journey-by-journey result (passed / failed / partial — and the WHY)
|
|
45
45
|
- Data integrity table (oracle, expected, actual, pass/fail)
|
|
46
46
|
- Lifecycle test results
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Phase 5: Deep Feature Testing (Mobile)
|
|
2
2
|
|
|
3
|
-
Follow
|
|
3
|
+
Follow `${CLAUDE_SKILL_DIR}/../qa-explore/phases/05-features.md` for the SFDIPOT structure, boundary
|
|
4
4
|
values, business-logic stress, and FEW HICCUPPS oracles. The mobile changes (both modes):
|
|
5
5
|
|
|
6
6
|
## Substitutions
|
|
@@ -11,15 +11,15 @@ values, business-logic stress, and FEW HICCUPPS oracles. The mobile changes (bot
|
|
|
11
11
|
tooling (mitmproxy)"**. Do NOT invent a result. (You CAN still tamper values that travel in
|
|
12
12
|
a deep link / URL via `deep-link`/`open-url` — see Phase 6.)
|
|
13
13
|
|
|
14
|
-
- **`playwright-cli fill <ref> <long-string>`** — use the same patterns
|
|
15
|
-
|
|
16
|
-
|
|
14
|
+
- **`playwright-cli fill <ref> <long-string>`** — use the same patterns (`fill` types through a
|
|
15
|
+
Maestro `inputText` flow on both platforms), but very long strings can still be truncated by
|
|
16
|
+
the platform IME, so a "field accepted N chars" result may be a driver limit, not the app's.
|
|
17
17
|
Note the truncation when reporting, and for true very-long tests prefer pasting via the
|
|
18
18
|
clipboard:
|
|
19
19
|
```bash
|
|
20
|
-
bin/wadb shell "echo '<very long string>' | xargs -I {} am broadcast -a clipper.set -e text '{}'"
|
|
21
|
-
|
|
22
|
-
# then long-press to paste — or use
|
|
20
|
+
qa/bin/wadb shell "echo '<very long string>' | xargs -I {} am broadcast -a clipper.set -e text '{}'"
|
|
21
|
+
qa/bin/mcli click <input ref>
|
|
22
|
+
# then long-press to paste — or use qa/bin/mcli fill with the actual string and note truncation
|
|
23
23
|
```
|
|
24
24
|
|
|
25
25
|
## Generic mobile feature stress recipes
|
|
@@ -51,7 +51,7 @@ screens and web views alike — they replace the desktop-only Playwright specifi
|
|
|
51
51
|
### Navigation & routing
|
|
52
52
|
|
|
53
53
|
- Hardware/gesture back (Android `press BACK`) vs in-screen back control — same destination?
|
|
54
|
-
- Deep-link straight into a mid-flow screen (NATIVE:
|
|
54
|
+
- Deep-link straight into a mid-flow screen (NATIVE: `qa/bin/mcli deep-link <app-scheme>`; WEB: `open-url`) — does it load with context or break?
|
|
55
55
|
- After a successful action, does back land somewhere sane (not re-submitting)?
|
|
56
56
|
|
|
57
57
|
### Submit / save actions
|
|
@@ -65,7 +65,7 @@ Concrete high-value targets:
|
|
|
65
65
|
|
|
66
66
|
## After Feature Testing
|
|
67
67
|
|
|
68
|
-
Write `phase-
|
|
68
|
+
Write `phase-5-features.md` (confidentiality header first) with feature-by-feature notes. For each deferred test
|
|
69
69
|
(proxy-blocked, multi-device-blocked), record it explicitly in the coverage map so it shows
|
|
70
70
|
up as a known gap in the report.
|
|
71
71
|
|
|
@@ -1,27 +1,21 @@
|
|
|
1
1
|
# Phase 6: Edge Cases & Mobile-Specific (mode-aware)
|
|
2
2
|
|
|
3
|
-
Follow
|
|
3
|
+
Follow `${CLAUDE_SKILL_DIR}/../qa-explore/phases/06-edge-cases.md` for the saboteur mindset. Edges are tagged
|
|
4
4
|
**[both]**, **[native]**, or **[web]** — run the ones that fit the resolved mode.
|
|
5
5
|
|
|
6
|
-
Set up once per Bash block:
|
|
7
|
-
```bash
|
|
8
|
-
export MOBILE_CLI_STATE=/tmp/<target>-state.json
|
|
9
|
-
MCLI="$PWD/bin/mcli" # run from the repo root
|
|
10
|
-
```
|
|
11
|
-
|
|
12
6
|
## Dropped / limited (no DOM, no JS console)
|
|
13
7
|
|
|
14
8
|
- **XSS execution assertions via `fill` [web]** — without DOM/console you can't confirm execution. You can still submit payloads and watch for gross breakage (crash, broken render, reflected unescaped text in the a11y tree).
|
|
15
9
|
- **`route ... --status=500` network simulation [both]** — needs mitmproxy. **Deferred** — log scenarios that would benefit, run them once a proxy exists.
|
|
16
|
-
- **Multi-session race [both]** — needs two emulators. **Deferred
|
|
10
|
+
- **Multi-session race [both]** — needs two emulators. **Deferred.**
|
|
17
11
|
- **`resize` viewport tests** — sim viewport is fixed. Use **rotation** instead (below).
|
|
18
12
|
|
|
19
13
|
## Kept (still useful on mobile)
|
|
20
14
|
|
|
21
|
-
- **Unicode / emoji [both]** in inputs:
|
|
22
|
-
- **Whitespace-only [both]:**
|
|
15
|
+
- **Unicode / emoji [both]** in inputs: `qa/bin/mcli fill <ref> "naïve café 🤑 العربية"`
|
|
16
|
+
- **Whitespace-only [both]:** `qa/bin/mcli fill <ref> " "`
|
|
23
17
|
- **Extremely long [both]:** see the truncation caveat in `05-features.md`
|
|
24
|
-
- **State manipulation via deep link / URL [both]:** NATIVE
|
|
18
|
+
- **State manipulation via deep link / URL [both]:** NATIVE `qa/bin/mcli deep-link "<app-scheme>://<path>?<tampered>"`; WEB `qa/bin/mcli open-url "<base_url>/<deep view>?<tampered params>"`
|
|
25
19
|
- **Replay [both]:** double-tap submit; submit then back then re-submit
|
|
26
20
|
- **Brute force / rate limits [both]:** repeated failed submissions/logins — lockout / cooldown?
|
|
27
21
|
|
|
@@ -31,21 +25,20 @@ MCLI="$PWD/bin/mcli" # run from the repo root
|
|
|
31
25
|
|
|
32
26
|
```bash
|
|
33
27
|
# Background mid-flow
|
|
34
|
-
|
|
35
|
-
# (iOS: use the home gesture/button via the Simulator; press HOME is Android-only)
|
|
28
|
+
qa/bin/mcli press HOME # both platforms (iOS via Maestro pressKey Home)
|
|
36
29
|
sleep 30
|
|
37
|
-
|
|
30
|
+
qa/bin/mcli launch # foreground — did state restore or reload-and-lose-state?
|
|
38
31
|
|
|
39
32
|
# Kill mid-flow
|
|
40
|
-
|
|
41
|
-
|
|
33
|
+
qa/bin/mcli stop
|
|
34
|
+
qa/bin/mcli launch
|
|
42
35
|
# WEB only: re-open the view you were on
|
|
43
|
-
#
|
|
36
|
+
# qa/bin/mcli open-url <current view>
|
|
44
37
|
|
|
45
38
|
# OS-initiated suspend (Android)
|
|
46
|
-
bin/wadb shell am broadcast -a android.intent.action.SCREEN_OFF
|
|
39
|
+
qa/bin/wadb shell am broadcast -a android.intent.action.SCREEN_OFF
|
|
47
40
|
sleep 5
|
|
48
|
-
bin/wadb shell am broadcast -a android.intent.action.SCREEN_ON
|
|
41
|
+
qa/bin/wadb shell am broadcast -a android.intent.action.SCREEN_ON
|
|
49
42
|
```
|
|
50
43
|
|
|
51
44
|
For each: did state survive? Was auth still valid? Did an in-progress action complete or
|
|
@@ -55,8 +48,8 @@ duplicate?
|
|
|
55
48
|
|
|
56
49
|
NATIVE — deny / revoke / re-grant runtime permissions and check graceful handling:
|
|
57
50
|
```bash
|
|
58
|
-
bin/wadb shell pm revoke <app.package> android.permission.CAMERA # then try the QR/camera feature
|
|
59
|
-
bin/wadb shell pm grant <app.package> android.permission.CAMERA
|
|
51
|
+
qa/bin/wadb shell pm revoke <app.package> android.permission.CAMERA # then try the QR/camera feature
|
|
52
|
+
qa/bin/wadb shell pm grant <app.package> android.permission.CAMERA
|
|
60
53
|
# iOS sim: xcrun simctl privacy <udid> reset|grant camera <app.bundle_id>
|
|
61
54
|
```
|
|
62
55
|
Test camera / notifications / contacts / location. In WEB mode the same applies to the
|
|
@@ -69,7 +62,7 @@ If the app uses biometric auth: approve → succeeds; deny/cancel → password f
|
|
|
69
62
|
disable biometrics in OS mid-flow → graceful fallback. Android emulator can simulate a
|
|
70
63
|
fingerprint touch:
|
|
71
64
|
```bash
|
|
72
|
-
bin/wadb -e emu finger touch 1
|
|
65
|
+
qa/bin/wadb -e emu finger touch 1
|
|
73
66
|
```
|
|
74
67
|
|
|
75
68
|
### Push notifications [native]
|
|
@@ -81,11 +74,13 @@ works; trigger a notification and tap it — does it deep-link to the right scre
|
|
|
81
74
|
|
|
82
75
|
```bash
|
|
83
76
|
# Android — toggle network
|
|
84
|
-
bin/wadb shell svc wifi disable && bin/wadb shell svc data disable
|
|
77
|
+
qa/bin/wadb shell svc wifi disable && qa/bin/wadb shell svc data disable
|
|
85
78
|
# Run a submit/save flow — does it surface "offline" cleanly or hang/blank?
|
|
86
|
-
bin/wadb shell svc wifi enable && bin/wadb shell svc data enable
|
|
79
|
+
qa/bin/wadb shell svc wifi enable && qa/bin/wadb shell svc data enable
|
|
87
80
|
|
|
88
|
-
# Slow network
|
|
81
|
+
# Slow network (Android emulator console via adb, no telnet):
|
|
82
|
+
qa/bin/wadb emu network speed gsm && qa/bin/wadb emu network delay umts
|
|
83
|
+
# restore: qa/bin/wadb emu network speed full && qa/bin/wadb emu network delay none
|
|
89
84
|
```
|
|
90
85
|
|
|
91
86
|
Ask: does the app surface offline cleanly, queue or fail loudly, auto-retry, or just spin?
|
|
@@ -94,10 +89,10 @@ Ask: does the app surface offline cleanly, queue or fail loudly, auto-retry, or
|
|
|
94
89
|
|
|
95
90
|
```bash
|
|
96
91
|
# Android
|
|
97
|
-
bin/wadb shell settings put system accelerometer_rotation 0
|
|
98
|
-
bin/wadb shell settings put system user_rotation 1 # landscape
|
|
99
|
-
|
|
100
|
-
bin/wadb shell settings put system user_rotation 0 # portrait
|
|
92
|
+
qa/bin/wadb shell settings put system accelerometer_rotation 0
|
|
93
|
+
qa/bin/wadb shell settings put system user_rotation 1 # landscape
|
|
94
|
+
qa/bin/mcli snapshot # does the layout reflow? keyboard cover input?
|
|
95
|
+
qa/bin/wadb shell settings put system user_rotation 0 # portrait
|
|
101
96
|
```
|
|
102
97
|
|
|
103
98
|
iOS sim: Cmd+Right/Left Arrow in the Simulator to rotate. Check reflow, clipping, and that
|
|
@@ -113,15 +108,15 @@ and submit correctly?
|
|
|
113
108
|
|
|
114
109
|
```bash
|
|
115
110
|
# NATIVE — exercise the app's own scheme/intent:
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
bin/wadb shell am start -a android.intent.action.VIEW -d "<app-scheme>://<path>?x=$(python3 -c 'print("A"*5000)')"
|
|
111
|
+
qa/bin/mcli deep-link "<app-scheme>://<a real path>"
|
|
112
|
+
qa/bin/mcli deep-link "<app-scheme>://<path>?id=&n=NaN"
|
|
113
|
+
qa/bin/wadb shell am start -a android.intent.action.VIEW -d "<app-scheme>://<path>?x=$(python3 -c 'print("A"*5000)')"
|
|
119
114
|
|
|
120
115
|
# WEB — exercise the target URL:
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
116
|
+
qa/bin/mcli open-url "<base_url>/<a real deep view>"
|
|
117
|
+
qa/bin/mcli open-url "<base_url>/<view>?id=&n=NaN"
|
|
118
|
+
qa/bin/mcli open-url "<base_url>/<view>?next=javascript:alert(1)"
|
|
119
|
+
qa/bin/mcli open-url "<base_url>/<view>?q=$(python3 -c 'print("A"*5000)')"
|
|
125
120
|
```
|
|
126
121
|
|
|
127
122
|
Crash, freeze, blank screen, open-redirect, or accept-as-valid are all findings.
|
|
@@ -130,8 +125,8 @@ Crash, freeze, blank screen, open-redirect, or accept-as-valid are all findings.
|
|
|
130
125
|
|
|
131
126
|
```bash
|
|
132
127
|
# Navigate to a view with sensitive data, then open Recents and screenshot the preview
|
|
133
|
-
bin/wadb shell input keyevent KEYCODE_APP_SWITCH # Android Recents
|
|
134
|
-
|
|
128
|
+
qa/bin/wadb shell input keyevent KEYCODE_APP_SWITCH # Android Recents
|
|
129
|
+
qa/bin/mcli screenshot output/sessions/<session-dir>/screenshots/recents-preview.png
|
|
135
130
|
```
|
|
136
131
|
|
|
137
132
|
Is sensitive data (account info, tokens-in-URL, personal data) visible in the Recents preview?
|
|
@@ -140,7 +135,7 @@ finding where it matters.
|
|
|
140
135
|
|
|
141
136
|
### Storage / data wipe
|
|
142
137
|
|
|
143
|
-
- **NATIVE:** force-clear app data mid-flow (Android
|
|
138
|
+
- **NATIVE:** force-clear app data mid-flow (Android `qa/bin/mcli relaunch-clean` = force-stop + `pm clear` + launch) — does it land on a clean welcome state or get stuck on an intermediate screen? Also test fresh-install vs upgrade if you have both builds.
|
|
144
139
|
- **WEB:** **do NOT use `relaunch-clean`** — on Android it `pm clear`s the *browser* and wipes the logged-in profile, forcing a fresh SSO+MFA login. If you need a cleared-cookie test, do it knowingly and expect to re-login.
|
|
145
140
|
- Returning vs fresh state [both]: correct first-ever behavior (empty states, first-run interstitials) AND returning behavior.
|
|
146
141
|
|
|
@@ -151,7 +146,7 @@ extremes (very large numbers, long names, long localized strings at mobile width
|
|
|
151
146
|
|
|
152
147
|
## After Edge Cases
|
|
153
148
|
|
|
154
|
-
Write `phase-
|
|
149
|
+
Write `phase-6-edge-cases.md` (confidentiality header first) with one section per category. Each deferred test gets a single
|
|
155
150
|
line so it's visible.
|
|
156
151
|
|
|
157
152
|
**Update progress.**
|
|
@@ -1,83 +1,107 @@
|
|
|
1
1
|
# Phase 7: Reflection & Reporting (Mobile)
|
|
2
2
|
|
|
3
|
-
Follow
|
|
4
|
-
questions
|
|
5
|
-
|
|
3
|
+
Follow `${CLAUDE_SKILL_DIR}/../qa-explore/phases/07-reporting.md` for the reflection
|
|
4
|
+
questions and the report structure; every artefact uses the format in
|
|
5
|
+
`${CLAUDE_SKILL_DIR}/../qa-explore/references/output-contract.md` (confidentiality header
|
|
6
|
+
first). The mobile differences (both modes):
|
|
6
7
|
|
|
7
8
|
## Stop recording
|
|
8
9
|
|
|
9
|
-
Replace `tracing-stop` with:
|
|
10
|
+
Replace `video-stop` / `tracing-stop` with:
|
|
10
11
|
|
|
11
12
|
```bash
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
$MCLI stop # close the app/browser — leaves the sim running for next time
|
|
13
|
+
qa/bin/mcli record-stop # closes the .mp4 (if record-start was used)
|
|
14
|
+
qa/bin/mcli stop # close the app/browser — leaves the sim running for next time
|
|
15
|
+
qa/bin/wk-ios --stop # iOS WEB mode only, if the DOM bridge was used
|
|
16
16
|
```
|
|
17
17
|
|
|
18
|
-
Do NOT `relaunch-clean` / wipe the device at session end
|
|
19
|
-
persisted login; in NATIVE mode it preserves any account/state created this session.
|
|
18
|
+
Do NOT `relaunch-clean` / wipe the device at session end: in WEB mode this preserves the
|
|
19
|
+
persisted login; in NATIVE mode it preserves any account/state created this session. There
|
|
20
|
+
is no playwright-cli session to `close` / `delete-data` on mobile.
|
|
20
21
|
|
|
21
|
-
## Bug
|
|
22
|
+
## Bug report — mobile fields
|
|
22
23
|
|
|
23
|
-
|
|
24
|
+
Same file and headings as the contract — `bugs/BUG-NNN.md`, starting with the two-line
|
|
25
|
+
header, verbatim:
|
|
26
|
+
|
|
27
|
+
```
|
|
28
|
+
> CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
|
|
29
|
+
> and application details. Do not share outside your organization without review.
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
then `# BUG-NNN: [Component] fails [Condition] causing [Impact]`, `**Severity:**`,
|
|
33
|
+
`**Priority:**`, `**Component:**`, `**URL:**`, `**Environment:**`,
|
|
34
|
+
`**Reproduction rate:**`, then `## Summary`, `## Expected Behavior`, `## Actual Behavior`,
|
|
35
|
+
`## Steps to Reproduce`, `## Business Impact`, `## Evidence` and
|
|
36
|
+
`## Recommended Fix Priority`. Two fields change on mobile:
|
|
24
37
|
|
|
25
38
|
```markdown
|
|
39
|
+
**URL:** <WEB: the page URL loaded via open-url | NATIVE: the screen name or deep link>
|
|
26
40
|
**Environment:**
|
|
27
|
-
- Mode:
|
|
28
|
-
- Platform:
|
|
29
|
-
- Device:
|
|
30
|
-
- OS version:
|
|
31
|
-
- App under test:
|
|
32
|
-
- Target URL:
|
|
33
|
-
- Environment:
|
|
41
|
+
- Mode: native | web
|
|
42
|
+
- Platform: android | ios
|
|
43
|
+
- Device: emulator-5554 | qa-iphone (UDID …)
|
|
44
|
+
- OS version: <android: qa/bin/wadb shell getprop ro.build.version.release — ios: xcrun simctl list runtimes>
|
|
45
|
+
- App under test: <native: app id + version + build type | web: browser bundle + version>
|
|
46
|
+
- Target URL: <web mode only: base_url>
|
|
47
|
+
- Environment: <preprod | staging | production>
|
|
34
48
|
```
|
|
35
49
|
|
|
36
|
-
And
|
|
50
|
+
And `## Evidence` carries the mobile artefacts:
|
|
37
51
|
|
|
38
52
|
```markdown
|
|
39
|
-
|
|
40
|
-
- Screenshot: `screenshots/
|
|
41
|
-
- Video
|
|
42
|
-
-
|
|
53
|
+
## Evidence
|
|
54
|
+
- Screenshot: `screenshots/BUG-NNN.png`
|
|
55
|
+
- Video: `videos/BUG-NNN.mp4` (if recorded)
|
|
56
|
+
- Log: `logs/BUG-NNN.log` (logcat / simctl excerpt around the failure — iOS is scoped to the app/browser process, Android is whole-device logcat, and neither is a page JS console)
|
|
57
|
+
- Console errors: n/a on mobile (or the wk-ios observation, iOS WEB)
|
|
58
|
+
- Network failures: n/a (no proxy) — or "needs proxy capture"
|
|
43
59
|
```
|
|
44
60
|
|
|
45
|
-
|
|
61
|
+
Capture the log at the moment of failure (the `logs/` directory was created in setup):
|
|
46
62
|
|
|
47
63
|
```bash
|
|
48
|
-
|
|
64
|
+
qa/bin/mcli screenshot output/sessions/<session-dir>/screenshots/BUG-NNN.png
|
|
65
|
+
qa/bin/mcli logs --since 30 --errors > output/sessions/<session-dir>/logs/BUG-NNN.log
|
|
49
66
|
```
|
|
50
67
|
|
|
68
|
+
Redact the log excerpt per `security-rules.md` before it is saved: process logs routinely
|
|
69
|
+
carry tokens and emails.
|
|
70
|
+
|
|
51
71
|
## Session report — mobile additions
|
|
52
72
|
|
|
53
|
-
|
|
73
|
+
`session-report.md` per the contract with `Kind: mobile`, then append after `## Session Stats`:
|
|
54
74
|
|
|
55
75
|
```markdown
|
|
56
76
|
## Mobile Context
|
|
57
77
|
|
|
58
78
|
| Field | Value |
|
|
59
79
|
|---|---|
|
|
60
|
-
| Mode |
|
|
61
|
-
| Platform |
|
|
62
|
-
| Device |
|
|
63
|
-
| OS version |
|
|
64
|
-
| App under test |
|
|
65
|
-
| Target URL |
|
|
66
|
-
|
|
|
80
|
+
| Mode | native / web |
|
|
81
|
+
| Platform | android / ios |
|
|
82
|
+
| Device | <device label> |
|
|
83
|
+
| OS version | <version> |
|
|
84
|
+
| App under test | <native: app id + version + build type | web: browser bundle + version> |
|
|
85
|
+
| Target URL | <web mode only: base_url + environment> |
|
|
86
|
+
| Source state | <native: branch + sha + clean/dirty, if rebuilt> |
|
|
87
|
+
| Driver version | <output of `qa/bin/mcli version`> |
|
|
67
88
|
|
|
68
89
|
## Deferred Tests (require additional tooling)
|
|
69
90
|
|
|
70
91
|
| Test | Why deferred | What's needed |
|
|
71
92
|
|---|---|---|
|
|
72
|
-
| Network failure simulation | mobile-cli
|
|
73
|
-
| Multi-device race conditions |
|
|
74
|
-
| Server-side parameter tampering (non-deep-link) |
|
|
93
|
+
| Network failure simulation | mobile-cli has no proxy | mitmproxy + cert install |
|
|
94
|
+
| Multi-device race conditions | single device | second emulator + parallel sessions |
|
|
95
|
+
| Server-side parameter tampering (non-deep-link) | no script eval on device | proxy + request modification |
|
|
75
96
|
```
|
|
76
97
|
|
|
98
|
+
Every deferred test also appears in the coverage map as `not-tested` with the reason, so it
|
|
99
|
+
shows up as a known gap.
|
|
100
|
+
|
|
77
101
|
## Requirements coverage cross-check (optional)
|
|
78
102
|
|
|
79
|
-
If the session was scoped to gathered requirements / tickets, add a cross-check table
|
|
80
|
-
is **not** test execution
|
|
103
|
+
If the session was scoped to gathered requirements / tickets, add a cross-check table. This
|
|
104
|
+
is **not** test execution; it is "did exploration touch the areas those requirements
|
|
81
105
|
designed?":
|
|
82
106
|
|
|
83
107
|
```markdown
|
|
@@ -90,25 +114,26 @@ Source: <ticket / acceptance criteria / gathered context>
|
|
|
90
114
|
| ... | Yes/No/Partial | BUG-NNN or — | ... |
|
|
91
115
|
```
|
|
92
116
|
|
|
93
|
-
The point is to surface coverage gaps the team can decide to fill manually or with automation.
|
|
94
|
-
|
|
95
117
|
## stats.json — mobile fields
|
|
96
118
|
|
|
97
|
-
|
|
119
|
+
`stats.json` exactly as the contract defines it, with `"kind": "mobile"`, and the device
|
|
120
|
+
block under `coverage.mobile` (the schema is strict; no other top-level keys):
|
|
98
121
|
|
|
99
122
|
```json
|
|
100
|
-
"
|
|
101
|
-
"
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
123
|
+
"coverage": {
|
|
124
|
+
"mobile": {
|
|
125
|
+
"mode": "web",
|
|
126
|
+
"platform": "ios",
|
|
127
|
+
"device": "qa-iphone",
|
|
128
|
+
"app_under_test": "com.apple.mobilesafari",
|
|
129
|
+
"target_url": "https://staging.m.example.com",
|
|
130
|
+
"environment": "staging",
|
|
131
|
+
"videos_recorded": 1,
|
|
132
|
+
"log_errors_observed": 0,
|
|
133
|
+
"lifecycle_tests_run": 0,
|
|
134
|
+
"rotation_tests_run": 0,
|
|
135
|
+
"deferred_tests": []
|
|
136
|
+
}
|
|
112
137
|
}
|
|
113
138
|
```
|
|
114
139
|
|
|
@@ -116,8 +141,9 @@ Append a `mobile` block to the qa-explore stats schema:
|
|
|
116
141
|
|
|
117
142
|
## Update progress.json — final
|
|
118
143
|
|
|
119
|
-
Same as qa-explore. Set `status: complete`, all phases to final status.
|
|
144
|
+
Same as qa-explore. Set `status: complete`, all phases to their final status.
|
|
120
145
|
|
|
121
146
|
## Update indexes
|
|
122
147
|
|
|
123
|
-
Append the session to `output/sessions/INDEX.md
|
|
148
|
+
Append the session row to `output/sessions/INDEX.md` (`Kind` = `mobile`) and one row per
|
|
149
|
+
bug to `output/bugs/all-bugs.md`, in the column order defined in `paths.md`.
|