qualiow-exploratory-testing 1.1.0 → 1.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (79) hide show
  1. package/README.md +177 -0
  2. package/bin/mcli +16 -0
  3. package/bin/mobile-cli.mjs +573 -0
  4. package/bin/wadb +6 -0
  5. package/bin/wk-ios +58 -0
  6. package/bin/wkeval.mjs +44 -0
  7. package/data/domains/_default.md +181 -0
  8. package/data/domains/ecommerce.md +192 -0
  9. package/data/domains/fintech.md +200 -0
  10. package/data/domains/identity.md +88 -0
  11. package/data/domains/identity.yml +138 -0
  12. package/data/domains/marketing.md +194 -0
  13. package/data/domains/saas.md +197 -0
  14. package/data/knowledge/changelog.yml +86 -0
  15. package/data/knowledge/learned-patterns.md +172 -1
  16. package/data/knowledge/manifest.yml +210 -4
  17. package/data/knowledge/releases/v0.2.0/entries/technique-contract-narrowing.yml +150 -0
  18. package/data/knowledge/releases/v0.2.0/entries/technique-test-suite-audit.yml +116 -0
  19. package/data/knowledge/releases/v0.2.0/release.yml +20 -0
  20. package/data/knowledge/releases/v0.3.0/entries/technique-functional-diff-analysis.yml +119 -0
  21. package/data/knowledge/releases/v0.3.0/entries/technique-llm-output-verification.yml +102 -0
  22. package/data/knowledge/releases/v0.3.0/entries/technique-verification-mode-selection.yml +153 -0
  23. package/data/knowledge/releases/v0.3.0/release.yml +25 -0
  24. package/data/knowledge/releases/v0.4.0/entries/technique-authenticated-api-probing.yml +137 -0
  25. package/data/knowledge/releases/v0.4.0/entries/technique-environment-fingerprinting.yml +113 -0
  26. package/data/knowledge/releases/v0.4.0/entries/technique-silent-failure-audit.yml +86 -0
  27. package/data/knowledge/releases/v0.4.0/entries/technique-ui-api-differential.yml +97 -0
  28. package/data/knowledge/releases/v0.4.0/release.yml +28 -0
  29. package/data/knowledge/releases/v0.5.0/entries/technique-config-surface-verification.yml +77 -0
  30. package/data/knowledge/releases/v0.5.0/entries/technique-derived-value-verification.yml +81 -0
  31. package/data/knowledge/releases/v0.5.0/entries/technique-expected-behaviour-specification.yml +97 -0
  32. package/data/knowledge/releases/v0.5.0/entries/technique-presentation-integrity.yml +78 -0
  33. package/data/knowledge/releases/v0.5.0/entries/technique-release-readiness-verification.yml +110 -0
  34. package/data/knowledge/releases/v0.5.0/release.yml +33 -0
  35. package/data/targets/_example-backend.yml +123 -0
  36. package/data/targets/_example-mobile-emulation.yml +52 -0
  37. package/data/targets/_example-native-mobile.yml +61 -0
  38. package/data/targets/_example-sim-android-chrome.yml +81 -0
  39. package/data/targets/_example-sim-ios-safari.yml +80 -0
  40. package/data/templates/api-probe-matrix.md +99 -0
  41. package/data/templates/coverage-map.md +14 -0
  42. package/data/templates/expected-behaviour.md +93 -0
  43. package/dist/cli/index.js +137 -5
  44. package/dist/cli/index.js.map +1 -1
  45. package/dist/index.d.ts +350 -527
  46. package/dist/index.js +144 -5
  47. package/dist/index.js.map +1 -1
  48. package/docs/BACKEND-VERIFICATION.md +277 -0
  49. package/docs/MOBILE-SETUP.md +92 -0
  50. package/package.json +26 -12
  51. package/scripts/doctor-mobile.sh +143 -0
  52. package/scripts/setup-mobile.sh +213 -0
  53. package/skills/qa-explore/phases/00-setup.md +8 -1
  54. package/skills/qa-explore/phases/01-auth.md +4 -0
  55. package/skills/qa-explore-mobile/SKILL.md +146 -0
  56. package/skills/qa-explore-mobile/phases/00-setup.md +203 -0
  57. package/skills/qa-explore-mobile/phases/01-auth.md +127 -0
  58. package/skills/qa-explore-mobile/phases/02-charter.md +70 -0
  59. package/skills/qa-explore-mobile/phases/03-discovery.md +79 -0
  60. package/skills/qa-explore-mobile/phases/04-journeys.md +49 -0
  61. package/skills/qa-explore-mobile/phases/05-features.md +72 -0
  62. package/skills/qa-explore-mobile/phases/06-edge-cases.md +157 -0
  63. package/skills/qa-explore-mobile/phases/07-reporting.md +123 -0
  64. package/skills/qa-explore-mobile/references/mobile-edges.md +117 -0
  65. package/skills/qa-explore-quick/SKILL.md +6 -0
  66. package/skills/qa-verify-backend/SKILL.md +163 -0
  67. package/skills/qa-verify-backend/phases/00-setup.md +97 -0
  68. package/skills/qa-verify-backend/phases/01-ac-decomposition.md +58 -0
  69. package/skills/qa-verify-backend/phases/02-static-review.md +71 -0
  70. package/skills/qa-verify-backend/phases/03-live-verification.md +62 -0
  71. package/skills/qa-verify-backend/phases/03b-api-verification.md +157 -0
  72. package/skills/qa-verify-backend/phases/04-e2e-trigger.md +66 -0
  73. package/skills/qa-verify-backend/phases/05-reporting.md +135 -0
  74. package/skills/qa-verify-backend/references/api-probes.md +141 -0
  75. package/skills/qa-verify-backend/references/aws-readonly-probes.md +156 -0
  76. package/skills/qa-verify-backend/references/environment-fingerprinting.md +104 -0
  77. package/skills/qa-verify-backend/references/payload-verification.md +112 -0
  78. package/skills/qa-verify-backend/references/release-readiness.md +97 -0
  79. package/skills/qa-verify-backend/references/safety-rules.md +90 -0
package/README.md CHANGED
@@ -74,6 +74,90 @@ npm test
74
74
  /qa-explore-quick https://app.example.com/checkout
75
75
  ```
76
76
 
77
+ ### Mobile session (simulator/emulator)
78
+
79
+ Runs a full exploratory session on an iOS Simulator or Android Emulator — either a
80
+ **native app** (the installed app is the system under test) or a **mobile web app in the
81
+ real device browser** (iOS Simulator Safari / Android Emulator Chrome — the genuine
82
+ engines, which Playwright cannot drive). The mode comes from the target config.
83
+
84
+ #### 1. One-time machine setup
85
+
86
+ ```bash
87
+ # installs everything scriptable: Maestro, JDK 17, Android SDK + Google-Play AVD
88
+ # (qa_pixel_api35), ios-webkit-debug-proxy, and the qa-iphone simulator device
89
+ scripts/setup-mobile.sh # add --android or --ios to scope; --yes for non-interactive
90
+ ```
91
+
92
+ Two iOS steps are genuinely manual (the script prints them if needed):
93
+
94
+ ```bash
95
+ # a) install Xcode from the App Store, then:
96
+ sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
97
+ sudo xcodebuild -license accept
98
+ # b) if no iOS runtime is installed:
99
+ xcodebuild -downloadPlatform iOS
100
+ # then re-run scripts/setup-mobile.sh --ios (creates the qa-iphone simulator)
101
+ ```
102
+
103
+ #### 2. Verify readiness (read-only, run before every first session on a machine)
104
+
105
+ ```bash
106
+ scripts/doctor-mobile.sh # or --ios / --android; fix each ✗ until READY
107
+ ```
108
+
109
+ #### 3. Create a target and run
110
+
111
+ Copy a template from `data/targets/`, rename it, and point it at your app:
112
+
113
+ | Template | What it tests | Driver |
114
+ |----------|---------------|--------|
115
+ | `_example-native-mobile.yml` | An installed iOS/Android app (native mode) | mobile-cli (Maestro) |
116
+ | `_example-sim-ios-safari.yml` | A mobile web app in REAL iOS Simulator Safari | mobile-cli (Maestro) |
117
+ | `_example-sim-android-chrome.yml` | A mobile web app in REAL Android Emulator Chrome | mobile-cli (Maestro) |
118
+ | `_example-mobile-emulation.yml` | A mobile web app via Playwright device emulation (no simulator) | `/qa-explore` (playwright-cli) |
119
+
120
+ ```bash
121
+ /qa-explore-mobile --target my-mobile-target # real sim/emulator (native or web)
122
+ /qa-explore --target my-emulation-target # Playwright mobile emulation
123
+ ```
124
+
125
+ #### 4. Driving the device manually (what the skill does under the hood)
126
+
127
+ ```bash
128
+ # Android: boot the standard AVD # iOS: boot the standard simulator
129
+ ~/Library/Android/sdk/emulator/emulator \
130
+ -avd qa_pixel_api35 -no-snapshot & xcrun simctl boot qa-iphone && open -a Simulator
131
+
132
+ export MOBILE_CLI_STATE=/tmp/my-target-state.json # isolate this session's device/app state
133
+ bin/mcli set-device emulator-5554 # or the iOS sim UDID (xcrun simctl list devices)
134
+ bin/mcli set-app com.android.chrome # or com.apple.mobilesafari, or your app's id
135
+ bin/mcli launch
136
+ bin/mcli open-url https://staging.m.example.com # web mode only
137
+ bin/mcli snapshot # page/app as tappable refs e1, e2, ...
138
+ bin/mcli fill e4 my-username
139
+ bin/mcli snapshot # ALWAYS re-snapshot after the keyboard appears
140
+ bin/mcli click e7
141
+ bin/mcli screenshot /tmp/evidence.png
142
+ bin/mcli logs --errors --since 60
143
+ bin/mcli --help # full command surface
144
+ ```
145
+
146
+ iOS Safari web targets additionally get real JS/DOM access via the WebKit bridge:
147
+
148
+ ```bash
149
+ bin/wk-ios 'document.title' # eval JS in the sim's Safari
150
+ bin/wk-ios 'document.querySelectorAll(".cart_item").length' # exact-DOM assertions
151
+ bin/wk-ios --stop # stop the proxy after the session
152
+ ```
153
+
154
+ Notes that save time:
155
+ - **Never `relaunch-clean` a web target** — it wipes the logged-in browser profile.
156
+ - SSO/MFA logins are done **once, by hand, on the device**; the browser profile persists
157
+ the session. There is no transferable `storage_state` on a real device browser.
158
+ - Use `bin/wadb` for raw adb commands (logcat, `pm`, `am`) — it sets `ANDROID_HOME` for you.
159
+ - Full setup guide + troubleshooting: `docs/MOBILE-SETUP.md`.
160
+
77
161
  ### Generate or regenerate a report
78
162
 
79
163
  ```bash
@@ -95,6 +179,96 @@ npm test
95
179
  /qa-target-setup
96
180
  ```
97
181
 
182
+ ## How to Verify Backend Acceptance Criteria
183
+
184
+ Not every acceptance criterion is visible in a browser. `/qa-verify-backend` covers tickets
185
+ whose ACs live below the UI — tables and streams, queue consumers, Lambda triggers, IAM
186
+ policies, webhooks, IaC, and the service's own HTTP endpoints — and produces an **AC
187
+ traceability matrix** instead of a pass/fail claim.
188
+
189
+ ```bash
190
+ # Copy the template, point it at your service, then verify a ticket branch
191
+ cp data/targets/_example-backend.yml data/targets/local-my-service.yml
192
+
193
+ /qa-verify-backend --target local-my-service --context output/context/TICKET-123-context.md
194
+ /qa-verify-backend --target local-my-service --static-only # no cloud credentials
195
+ /qa-verify-backend --target local-my-service --api-only --parity local-my-service-staging
196
+ ```
197
+
198
+ It runs four lanes:
199
+
200
+ | Lane | What it does |
201
+ |------|--------------|
202
+ | **Static** | Reads the implementation branch against each AC with `git show` (never checks out) — spec drift, scope creep, failure paths, identity propagation, producer/consumer contract breaks |
203
+ | **Live** | Read-only `aws-cli` probes, one per AC, raw output saved as evidence. Never mutates; hard stop on production |
204
+ | **API** | Calls the service's own endpoints from the already-authenticated page, with a written case matrix covering everything the browser's guards prevent. Read-only, limited to the target's `api.probe_allowlist` |
205
+ | **End-to-end** | Drives the real write path in a non-production environment, then re-probes the data layer — including same-tick writes, deletes, bulk saves, a second identity, and DLQ depth |
206
+
207
+ Each AC gets a verdict with cited evidence — `PASS`, `PARTIAL`, `FAIL`, `BLOCKED`,
208
+ `NOT-REACHABLE` or `UNVERIFIABLE` — and the verdict **names the mode it was reached by and the
209
+ environment it holds in**. `PASS (direct request, raw response attached)` and `PASS (read the
210
+ diff)` are different claims, and a verdict backed only by a code reading is `UNVERIFIABLE`,
211
+ never `PASS`. `BLOCKED` is a first-class outcome: when credentials are missing, the probe
212
+ commands are written out ready to run rather than the verdict being inferred from source.
213
+
214
+ ### Two things it will not let you get away with
215
+
216
+ **Believing a green result from an environment that does not run your code.** Before the first
217
+ probe, the skill fingerprints every component of the request path: which build is deployed,
218
+ whether the changed path is *selected* here, and whether the commit under test is genuinely an
219
+ ancestor of what is running. One identical build can hold two implementations of the same
220
+ feature with a flag choosing between them — *same build, different behaviour ⇒ configuration,
221
+ not deploy lag*. An environment that does not run the change gets `NOT-REACHABLE`.
222
+
223
+ **Recording a client-side guard as the endpoint's behaviour.** "The search box refuses to fire
224
+ when empty" describes the browser. Every other client — mobile, partner integration, script —
225
+ sends that request, and it is frequently the one that fails. When a ticket has both a screen
226
+ and an endpoint, the same cases run at both surfaces and each finding is sorted into *both*,
227
+ *API only* (a real defect the guard is hiding), *UI only* (the client invents or masks
228
+ behaviour the service does not have) or *neither*.
229
+
230
+ The API lane needs no token handling: the request runs inside the page that is already logged
231
+ in, so it carries the same session cookie, CSRF token and interceptors as the UI — which also
232
+ means it works with SSO and MFA that no scripted login can pass. Credentials stay in the
233
+ browser profile under `.auth/` and never enter a script, a report or the repository.
234
+
235
+ ### Correct shape is not a correct answer
236
+
237
+ Every derived value — a percentage, total, ratio, delta or aggregate — is recomputed from the
238
+ raw figures in the same response, with the formula taken from the specification rather than from
239
+ the code under test, and with cases chosen to stress sign, zero, scale and cardinality. Then the
240
+ report states the limitation plainly: when both sides of the check come from one payload, the
241
+ derivation is verified and the inputs are not, so the **independent oracle** that would close
242
+ the gap is named along with whether it was run.
243
+
244
+ The payload is also held next to the screen, because a correct response can still reach the user
245
+ as a wrong number — a formatter that guesses what a value is, a unit applied twice, a truncated
246
+ figure presented as a total. That defect is invisible from either surface alone, and it usually
247
+ belongs to a different change than the one under test.
248
+
249
+ ### Findings with no acceptance criterion become a spec
250
+
251
+ Most of what an API probe turns up has nothing to be filed against, so it turns into an argument
252
+ rather than a fix. `data/templates/expected-behaviour.md` converts a pile of observations into
253
+ one reviewable decision: observed against expected, grouped by cause, with the decisions the fix
254
+ forces made explicit — reject rather than clamp, an error rather than a silent zero, validation
255
+ at the layer that covers every implementation — ranked by what real users can reach today, and
256
+ closed with a plain-English reply for whoever decides to fund the work.
257
+
258
+ ### Verifying a release rather than a ticket
259
+
260
+ `.claude/skills/qa-verify-backend/references/release-readiness.md` changes the shape of the
261
+ session: the deployment table first
262
+ for every ticket, so a ticket whose backend is not in the build is marked *not testable here*
263
+ rather than tested against a UI that will render convincing nonsense; a result vocabulary that
264
+ keeps *not tested* visible; a coverage map where 🔍 *code-verified only* is marked as
265
+ `UNVERIFIABLE` rather than green, with one sentence naming the untested item that carries the
266
+ most risk; carry-over defects in their own section; and a disposition with the condition that
267
+ would reverse it.
268
+
269
+ Full guide: **`docs/BACKEND-VERIFICATION.md`**. Safety rules (read-only discipline, the
270
+ production hard stop, redaction): `.claude/skills/qa-verify-backend/references/safety-rules.md`.
271
+
98
272
  ## How to Generate Reports
99
273
 
100
274
  qualiow produces three report formats from session data.
@@ -134,7 +308,9 @@ const csv = await generateJiraExport('output/sessions/2026-03-28-parabank');
134
308
  | Skill | Purpose |
135
309
  |-------|---------|
136
310
  | `/qa-explore` | Full exploratory testing session (45 min) |
311
+ | `/qa-explore-mobile` | Exploratory session on a simulator/emulator — native apps or mobile web in the real device browser (iOS Safari / Android Chrome) |
137
312
  | `/qa-explore-quick` | Quick focused session on a single page or feature (15 min) |
313
+ | `/qa-verify-backend` | Verify backend/API/infra acceptance criteria with no UI surface — branch review, read-only cloud probes, direct API probes, AC traceability matrix |
138
314
  | `/qa-explore-report` | Generate or regenerate report from existing session |
139
315
  | `/qa-explore-feedback` | Post-session feedback capture (false positives, missed bugs) |
140
316
  | `/qa-explore-cleanup` | Session cleanup and archival |
@@ -178,6 +354,7 @@ qualiow-exploratory-testing-skills/
178
354
  benchmarks/ # performance benchmarks
179
355
  docs/
180
356
  GETTING-STARTED.md # detailed setup and usage guide
357
+ BACKEND-VERIFICATION.md # verifying ACs with no UI surface
181
358
  ARCHITECTURE-DECISIONS.md # architecture decision records
182
359
  ```
183
360
 
package/bin/mcli ADDED
@@ -0,0 +1,16 @@
1
+ #!/bin/bash
2
+ # mcli — env-setting wrapper around mobile-cli.mjs so subagents can call a single
3
+ # bash command without needing to prefix every call with export JAVA_HOME=... etc.
4
+ # (Compound `export ...; cmd ...` calls hit the Claude Code permission system
5
+ # which matches on the first token of the line.)
6
+ # Resolve JDK 17 portably: honor an existing JAVA_HOME, else ask macOS' java_home,
7
+ # else fall back to the Homebrew prefixes (arm64 then Intel). Avoids hardcoding one path.
8
+ if [ -z "${JAVA_HOME:-}" ]; then
9
+ JAVA_HOME="$(/usr/libexec/java_home -v 17 2>/dev/null || true)"
10
+ [ -z "$JAVA_HOME" ] && [ -d /opt/homebrew/opt/openjdk@17 ] && JAVA_HOME=/opt/homebrew/opt/openjdk@17
11
+ [ -z "$JAVA_HOME" ] && [ -d /usr/local/opt/openjdk@17 ] && JAVA_HOME=/usr/local/opt/openjdk@17
12
+ fi
13
+ export JAVA_HOME
14
+ export ANDROID_HOME="${ANDROID_HOME:-$HOME/Library/Android/sdk}"
15
+ export PATH="$JAVA_HOME/bin:$ANDROID_HOME/platform-tools:$ANDROID_HOME/emulator:$HOME/.maestro/bin:$PATH"
16
+ exec node "$(dirname "$0")/mobile-cli.mjs" "$@"