qualiow-exploratory-testing 1.1.0 → 1.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (79) hide show
  1. package/README.md +177 -0
  2. package/bin/mcli +16 -0
  3. package/bin/mobile-cli.mjs +573 -0
  4. package/bin/wadb +6 -0
  5. package/bin/wk-ios +58 -0
  6. package/bin/wkeval.mjs +44 -0
  7. package/data/domains/_default.md +181 -0
  8. package/data/domains/ecommerce.md +192 -0
  9. package/data/domains/fintech.md +200 -0
  10. package/data/domains/identity.md +88 -0
  11. package/data/domains/identity.yml +138 -0
  12. package/data/domains/marketing.md +194 -0
  13. package/data/domains/saas.md +197 -0
  14. package/data/knowledge/changelog.yml +86 -0
  15. package/data/knowledge/learned-patterns.md +172 -1
  16. package/data/knowledge/manifest.yml +210 -4
  17. package/data/knowledge/releases/v0.2.0/entries/technique-contract-narrowing.yml +150 -0
  18. package/data/knowledge/releases/v0.2.0/entries/technique-test-suite-audit.yml +116 -0
  19. package/data/knowledge/releases/v0.2.0/release.yml +20 -0
  20. package/data/knowledge/releases/v0.3.0/entries/technique-functional-diff-analysis.yml +119 -0
  21. package/data/knowledge/releases/v0.3.0/entries/technique-llm-output-verification.yml +102 -0
  22. package/data/knowledge/releases/v0.3.0/entries/technique-verification-mode-selection.yml +153 -0
  23. package/data/knowledge/releases/v0.3.0/release.yml +25 -0
  24. package/data/knowledge/releases/v0.4.0/entries/technique-authenticated-api-probing.yml +137 -0
  25. package/data/knowledge/releases/v0.4.0/entries/technique-environment-fingerprinting.yml +113 -0
  26. package/data/knowledge/releases/v0.4.0/entries/technique-silent-failure-audit.yml +86 -0
  27. package/data/knowledge/releases/v0.4.0/entries/technique-ui-api-differential.yml +97 -0
  28. package/data/knowledge/releases/v0.4.0/release.yml +28 -0
  29. package/data/knowledge/releases/v0.5.0/entries/technique-config-surface-verification.yml +77 -0
  30. package/data/knowledge/releases/v0.5.0/entries/technique-derived-value-verification.yml +81 -0
  31. package/data/knowledge/releases/v0.5.0/entries/technique-expected-behaviour-specification.yml +97 -0
  32. package/data/knowledge/releases/v0.5.0/entries/technique-presentation-integrity.yml +78 -0
  33. package/data/knowledge/releases/v0.5.0/entries/technique-release-readiness-verification.yml +110 -0
  34. package/data/knowledge/releases/v0.5.0/release.yml +33 -0
  35. package/data/targets/_example-backend.yml +123 -0
  36. package/data/targets/_example-mobile-emulation.yml +52 -0
  37. package/data/targets/_example-native-mobile.yml +61 -0
  38. package/data/targets/_example-sim-android-chrome.yml +81 -0
  39. package/data/targets/_example-sim-ios-safari.yml +80 -0
  40. package/data/templates/api-probe-matrix.md +99 -0
  41. package/data/templates/coverage-map.md +14 -0
  42. package/data/templates/expected-behaviour.md +93 -0
  43. package/dist/cli/index.js +137 -5
  44. package/dist/cli/index.js.map +1 -1
  45. package/dist/index.d.ts +350 -527
  46. package/dist/index.js +144 -5
  47. package/dist/index.js.map +1 -1
  48. package/docs/BACKEND-VERIFICATION.md +277 -0
  49. package/docs/MOBILE-SETUP.md +92 -0
  50. package/package.json +26 -12
  51. package/scripts/doctor-mobile.sh +143 -0
  52. package/scripts/setup-mobile.sh +213 -0
  53. package/skills/qa-explore/phases/00-setup.md +8 -1
  54. package/skills/qa-explore/phases/01-auth.md +4 -0
  55. package/skills/qa-explore-mobile/SKILL.md +146 -0
  56. package/skills/qa-explore-mobile/phases/00-setup.md +203 -0
  57. package/skills/qa-explore-mobile/phases/01-auth.md +127 -0
  58. package/skills/qa-explore-mobile/phases/02-charter.md +70 -0
  59. package/skills/qa-explore-mobile/phases/03-discovery.md +79 -0
  60. package/skills/qa-explore-mobile/phases/04-journeys.md +49 -0
  61. package/skills/qa-explore-mobile/phases/05-features.md +72 -0
  62. package/skills/qa-explore-mobile/phases/06-edge-cases.md +157 -0
  63. package/skills/qa-explore-mobile/phases/07-reporting.md +123 -0
  64. package/skills/qa-explore-mobile/references/mobile-edges.md +117 -0
  65. package/skills/qa-explore-quick/SKILL.md +6 -0
  66. package/skills/qa-verify-backend/SKILL.md +163 -0
  67. package/skills/qa-verify-backend/phases/00-setup.md +97 -0
  68. package/skills/qa-verify-backend/phases/01-ac-decomposition.md +58 -0
  69. package/skills/qa-verify-backend/phases/02-static-review.md +71 -0
  70. package/skills/qa-verify-backend/phases/03-live-verification.md +62 -0
  71. package/skills/qa-verify-backend/phases/03b-api-verification.md +157 -0
  72. package/skills/qa-verify-backend/phases/04-e2e-trigger.md +66 -0
  73. package/skills/qa-verify-backend/phases/05-reporting.md +135 -0
  74. package/skills/qa-verify-backend/references/api-probes.md +141 -0
  75. package/skills/qa-verify-backend/references/aws-readonly-probes.md +156 -0
  76. package/skills/qa-verify-backend/references/environment-fingerprinting.md +104 -0
  77. package/skills/qa-verify-backend/references/payload-verification.md +112 -0
  78. package/skills/qa-verify-backend/references/release-readiness.md +97 -0
  79. package/skills/qa-verify-backend/references/safety-rules.md +90 -0
@@ -0,0 +1,123 @@
1
+ # Phase 7: Reflection & Reporting (Mobile)
2
+
3
+ Follow `../../qa-explore/phases/07-reporting.md` for the report structure, reflection
4
+ questions, bug-report template, session-report template, and stats.json schema. The
5
+ mobile additions (both modes):
6
+
7
+ ## Stop recording
8
+
9
+ Replace `tracing-stop` with:
10
+
11
+ ```bash
12
+ export MOBILE_CLI_STATE=/tmp/<target>-state.json
13
+ MCLI="$PWD/bin/mcli" # run from the repo root
14
+ $MCLI record-stop # closes the video (if record-start was used)
15
+ $MCLI stop # close the app/browser — leaves the sim running for next time
16
+ ```
17
+
18
+ Do NOT `relaunch-clean` / wipe the device at session end — in WEB mode this preserves the
19
+ persisted login; in NATIVE mode it preserves any account/state created this session.
20
+
21
+ ## Bug-report — mobile fields
22
+
23
+ In each `bugs/BUG-NNN.md`, replace the qa-explore `Environment` line with:
24
+
25
+ ```markdown
26
+ **Environment:**
27
+ - Mode: [native / web]
28
+ - Platform: [android / ios]
29
+ - Device: [emulator-5554 / iPhone 17 sim UDID]
30
+ - OS version: [from `bin/wadb shell getprop ro.build.version.release` / `xcrun simctl`]
31
+ - App under test: [native: app id + version + build type | web: browser bundle + version]
32
+ - Target URL: [web mode only: base_url]
33
+ - Environment: [preprod / staging / production]
34
+ ```
35
+
36
+ And add an **Evidence** block enriched for mobile:
37
+
38
+ ```markdown
39
+ ### Evidence
40
+ - Screenshot: `screenshots/bug-NNN.png`
41
+ - Video clip: `videos/bug-NNN.mp4` (if recorded)
42
+ - Process log: `logs/bug-NNN.log` (logcat / simctl excerpt around the time of failure — app process in native mode, browser process in web mode, not a page JS console)
43
+ ```
44
+
45
+ When you file a bug, capture the log at the moment of failure:
46
+
47
+ ```bash
48
+ $MCLI logs --since 30 --errors > output/sessions/<session-dir>/logs/bug-NNN.log
49
+ ```
50
+
51
+ ## Session report — mobile additions
52
+
53
+ Append a "Mobile context" section to `session-report.md`:
54
+
55
+ ```markdown
56
+ ## Mobile Context
57
+
58
+ | Field | Value |
59
+ |---|---|
60
+ | Mode | [native / web] |
61
+ | Platform | [android / ios] |
62
+ | Device | [device label] |
63
+ | OS version | [version] |
64
+ | App under test | [native: app id + version + build type | web: browser bundle + version] |
65
+ | Target URL | [web mode only: base_url + environment] |
66
+ | mobile-cli version | [git sha of this repo] |
67
+
68
+ ## Deferred Tests (require additional tooling)
69
+
70
+ | Test | Why deferred | What's needed |
71
+ |---|---|---|
72
+ | Network failure simulation | mobile-cli v1 has no proxy | mitmproxy + cert install |
73
+ | Multi-device race conditions | mobile-cli v1 single-device | Second emulator + parallel sessions |
74
+ | Server-side parameter tampering (non-deep-link) | No script eval on device | Proxy + request modification |
75
+ ```
76
+
77
+ ## Requirements coverage cross-check (optional)
78
+
79
+ If the session was scoped to gathered requirements / tickets, add a cross-check table — this
80
+ is **not** test execution, it's "did exploration touch the areas those requirements
81
+ designed?":
82
+
83
+ ```markdown
84
+ ## Requirements Cross-Check
85
+
86
+ Source: <ticket / acceptance criteria / gathered context>
87
+
88
+ | Requirement / Area | Covered? | Found bug? | Notes |
89
+ |---|---|---|---|
90
+ | ... | Yes/No/Partial | BUG-NNN or — | ... |
91
+ ```
92
+
93
+ The point is to surface coverage gaps the team can decide to fill manually or with automation.
94
+
95
+ ## stats.json — mobile fields
96
+
97
+ Append a `mobile` block to the qa-explore stats schema:
98
+
99
+ ```json
100
+ "mobile": {
101
+ "mode": "web",
102
+ "platform": "ios",
103
+ "device": "iPhone 17",
104
+ "app_under_test": "com.apple.mobilesafari",
105
+ "target_url": "https://staging.m.example.com",
106
+ "environment": "staging",
107
+ "videos_recorded": 1,
108
+ "log_errors_observed": 0,
109
+ "lifecycle_tests_run": 0,
110
+ "rotation_tests_run": 0,
111
+ "deferred_tests": []
112
+ }
113
+ ```
114
+
115
+ (For native mode set `"mode": "native"`, `app_under_test` to the app id, and `target_url` to null.)
116
+
117
+ ## Update progress.json — final
118
+
119
+ Same as qa-explore. Set `status: complete`, all phases to final status.
120
+
121
+ ## Update indexes
122
+
123
+ Append the session to `output/sessions/INDEX.md`.
@@ -0,0 +1,117 @@
1
+ # Mobile Edge Cases — Checklist (native + web)
2
+
3
+ Treat as a thinking tool, not a checkbox sheet. Pick what matches the feature and the resolved
4
+ **mode**. Each entry is tagged **[both]**, **[native]**, or **[web]**, with the adb / simctl
5
+ trigger where possible. `<app>` = the app under test's package/bundle in NATIVE mode, or the
6
+ browser bundle (`com.android.chrome` / `com.apple.mobilesafari`) in WEB mode. `$MCLI` is the
7
+ `bin/mcli` wrapper; use `bin/wadb` for raw adb.
8
+
9
+ ## Lifecycle [both]
10
+
11
+ | Edge | Android trigger | iOS sim trigger |
12
+ |---|---|---|
13
+ | Background app/browser | `bin/wadb shell input keyevent KEYCODE_HOME` | home gesture/button in Simulator |
14
+ | Kill app/browser | `bin/wadb shell am force-stop <app>` | `xcrun simctl terminate <udid> <app>` |
15
+ | Cold relaunch | `$MCLI launch` (WEB: then `$MCLI open-url <url>`) | `$MCLI launch` (WEB: then `$MCLI open-url <url>`) |
16
+ | Clear data | NATIVE: `$MCLI relaunch-clean` (= force-stop + pm clear) — clean-state test. WEB: same command **wipes the logged-in profile; avoid unless intended** | NATIVE: `xcrun simctl uninstall` + reinstall. WEB: clear Safari data in sim Settings (drops the login) |
17
+ | Screen off / on | `bin/wadb shell input keyevent KEYCODE_POWER` (toggle) | n/a (sim doesn't lock realistically) |
18
+
19
+ **Ask:** does state survive? Does an in-progress action complete? Does auth/session persist, or is everything reloaded?
20
+
21
+ ## Permissions [native] (and browser prompts in [web])
22
+
23
+ NATIVE: the app requests OS runtime permissions directly. WEB: the *page* requests them via
24
+ the BROWSER, so grant/revoke on the browser package controls `getUserMedia` camera,
25
+ geolocation, notifications. Use `<app>` = app package (native) or browser (web).
26
+
27
+ | Permission | Android revoke | Android grant | iOS sim |
28
+ |---|---|---|---|
29
+ | Camera | `bin/wadb shell pm revoke <app> android.permission.CAMERA` | `pm grant` | `xcrun simctl privacy <udid> reset camera <app>` (then re-prompt) |
30
+ | Location | `pm revoke <app> android.permission.ACCESS_FINE_LOCATION` | `pm grant` | `xcrun simctl privacy <udid> reset location <app>` |
31
+ | Notifications | `pm revoke <app> android.permission.POST_NOTIFICATIONS` | `pm grant` | `xcrun simctl privacy <udid> reset notifications <app>` |
32
+ | Contacts [native] | `pm revoke <app> android.permission.READ_CONTACTS` | `pm grant` | `xcrun simctl privacy <udid> reset contacts <app>` |
33
+
34
+ **Ask:** does the app explain WHY before prompting? Does denial crash or degrade gracefully? Can the user recover/re-grant?
35
+
36
+ ## Biometrics [native]
37
+
38
+ App-level biometric auth only (a real sim browser doesn't expose a biometric flow to web).
39
+ Approve → succeeds; deny/cancel → password fallback; disable in OS mid-flow → graceful
40
+ fallback. Android emulator can simulate a fingerprint: `bin/wadb -e emu finger touch 1`.
41
+
42
+ ## Push notifications [native]
43
+
44
+ If the app registers for push: deny the prompt and confirm the app still works; deliver a
45
+ notification and tap it — does it deep-link to the right screen with context?
46
+
47
+ ## Connectivity [both]
48
+
49
+ | Edge | Android | iOS sim |
50
+ |---|---|---|
51
+ | Airplane mode | `bin/wadb shell svc wifi disable && bin/wadb shell svc data disable` (re-enable to undo) | Network Link Conditioner (manual) |
52
+ | Slow network | telnet to emulator console: `telnet localhost 5554`, then `network speed gsm` | Network Link Conditioner |
53
+ | Captive portal | hard — skip for v1 | hard — skip |
54
+
55
+ **Ask:** does the app surface offline cleanly? Does it queue actions or fail loudly? Does it
56
+ auto-retry? Does it leak retries (battery drain)?
57
+
58
+ ## Hardware / display [both]
59
+
60
+ | Edge | Android | iOS sim |
61
+ |---|---|---|
62
+ | Rotation portrait/landscape | `bin/wadb shell settings put system user_rotation 0/1/2/3` | Cmd+Right/Left Arrow in Simulator |
63
+ | Dark mode | `bin/wadb shell "cmd uimode night yes"` (and `no`) | `xcrun simctl ui <udid> appearance dark/light` |
64
+ | Dynamic type / large font | `bin/wadb shell settings put system font_scale 1.5` | Settings app inside sim |
65
+ | Low battery | `bin/wadb shell dumpsys battery set level 10` (and `reset`) | n/a |
66
+ | Split-screen | manual via Recents | n/a |
67
+
68
+ **Ask:** does the layout reflow, clip, or hide content? Does it honor the OS dark-mode /
69
+ font-scale, or break? **[web]** also: address-bar show/hide on scroll changing the viewport
70
+ height (`100vh` jumps), momentum/overscroll bounce, and whether the page respects the
71
+ `viewport` meta (no desktop-width zoom-out, no horizontal scroll).
72
+
73
+ ## Deep links / URL handling [both]
74
+
75
+ ```bash
76
+ # NATIVE — the app's own scheme/intent
77
+ $MCLI deep-link "<app-scheme>://<a real path>"
78
+ $MCLI deep-link "<app-scheme>://<path>?id=&n=NaN"
79
+ bin/wadb shell am start -W -a android.intent.action.VIEW -d "<app-scheme>://<path>" -f 0x10000000
80
+
81
+ # WEB — the target URL
82
+ $MCLI open-url "<base_url>/<a real deep view>"
83
+ $MCLI open-url "<base_url>/<view>?id=&n=NaN"
84
+ $MCLI open-url "<base_url>/<view>?next=javascript:alert(1)"
85
+ $MCLI open-url "<base_url>/<view>?q=$(python3 -c 'print("A"*5000)')"
86
+ ```
87
+
88
+ **Ask:** crash, freeze, blank screen, open-redirect, or unsanitized reflection?
89
+
90
+ ## Privacy / leak surface [both]
91
+
92
+ - **Recents preview:** background while on a sensitive screen, screencap Recents — is personal data / a token-in-URL visible? A native app can mask its preview (FLAG_SECURE); web pages generally can't.
93
+ - **Process log:** `$MCLI logs --errors` — grep for PII or credentials (emails, tokens, secrets) leaking into the app/browser process log.
94
+ - **URL leakage [web]:** sensitive values in query strings end up in history and Recents — flag where they shouldn't be.
95
+ - **Clipboard:** if the app copies sensitive text, does it auto-clear/expire?
96
+
97
+ ## Storage / state [both]
98
+
99
+ - NATIVE: force-clear app data mid-flow (`pm clear`) — clean restart or stuck intermediate screen? Fresh-install vs upgrade: did state migrate?
100
+ - WEB: clearing browser cookies/data drops the login session — test the cleared-state behavior knowingly, not by accident.
101
+ - Returning vs fresh state: correct first-run interstitials, empty states, onboarding?
102
+ - Long-lived session: does a session that expires while the app/page is open prompt a graceful re-auth, or break into a blank/broken state?
103
+
104
+ ## Native a11y [both] (driven through the device a11y tree)
105
+
106
+ - TalkBack (Android): `bin/wadb shell settings put secure enabled_accessibility_services com.google.android.marvin.talkback/.TalkBackService`
107
+ - VoiceOver (iOS sim): mostly limited on sim — manual via the accessibility menu
108
+ - **Focus order:** swipe right repeatedly — does focus land on every interactive element in a sensible order?
109
+ - **Aggregation:** does each control announce its own label, or does the view dump many fields into one comma-joined label? (Known iOS issue — see `phases/03-discovery.md`.)
110
+ - **Contrast:** spot-check critical text — sufficient contrast in both light and dark mode?
111
+ - **Touch targets:** are tap targets finger-sized, or desktop-sized and hard to hit on a phone?
112
+
113
+ ## Empty / boundary states [both]
114
+
115
+ - Zero items / one item / many items in lists.
116
+ - First-ever visit vs returning visit.
117
+ - Formatter extremes: very large numbers, long names, long localized strings at mobile width — truncation, overflow, `NaN`/`undefined`?
@@ -19,6 +19,12 @@ You are a **Principal QA Engineer** doing a focused 15-minute exploratory sessio
19
19
  ## Flow
20
20
 
21
21
  ### 1. Setup (~1 min)
22
+
23
+ Before opening the browser, check for project-local config:
24
+ 1. If `qa/target.yml` exists in CWD, read it for base URL, auth strategy, and scope hints
25
+ 2. If `qa/.env` exists in CWD, load credentials from it (never log these values)
26
+ 3. If the target requires auth (storage_state or token strategy), apply it before navigating
27
+
22
28
  ```bash
23
29
  playwright-cli open <url>
24
30
  playwright-cli snapshot
@@ -0,0 +1,163 @@
1
+ ---
2
+ name: qa-verify-backend
3
+ description: >
4
+ Verify backend, API and infrastructure acceptance criteria that have NO user-interface
5
+ surface — tables and streams, queue consumers, Lambda triggers, IAM policies, IaC, and
6
+ the service's own HTTP endpoints. Reviews the implementation branch against each AC,
7
+ probes the live environment read-only via aws-cli, calls the API directly from the
8
+ authenticated session, and produces an AC-by-AC traceability matrix plus bug reports
9
+ with business impact. Use when user says: "verify this ticket", "check the ACs",
10
+ "review the infra change", "does this meet the acceptance criteria", "test the backend",
11
+ "test the API", "check the endpoint", or gives a backend/API/infra ticket whose ACs
12
+ cannot be seen in a browser.
13
+ allowed-tools: Read, Write, Glob, Grep, Bash(git:*), Bash(aws:*), Bash(jq:*), Bash(curl:*), Bash(playwright-cli:*), Bash(terraform validate:*), Bash(terraform fmt:*)
14
+ ---
15
+
16
+ # Backend & Infrastructure AC Verification
17
+
18
+ ## Your Identity
19
+
20
+ You are a **Principal QA Engineer** who does not accept "it deployed" as evidence.
21
+ Your job is to answer one question per acceptance criterion: **is it actually true,
22
+ and what is my evidence?**
23
+
24
+ **Your mindset:**
25
+ - "The ticket says NEW_IMAGE. What does the resource actually say?"
26
+ - "The happy path wrote a record. What does the failure path do?"
27
+ - "This satisfies the letter of the AC. Does it satisfy the intent?"
28
+ - "The producer works. Does it match the contract the consumer ticket will rely on?"
29
+ - "The screen is guarded. What does the endpoint do when nothing guards it?"
30
+ - "It passes here. Does this environment even run the code I am judging?"
31
+
32
+ **You are NOT a checkbox ticker.** An AC is `PASS` only when you can point at
33
+ evidence. `PARTIAL`, `FAIL` and `BLOCKED` are respectable verdicts. Guessing is not.
34
+
35
+ ## Why This Skill Exists
36
+
37
+ `/qa-explore` drives a browser. Backend tickets — a stream, a table, an IAM policy, a
38
+ search endpoint — have most of their acceptance criteria below the UI, where a browser
39
+ cannot see. Those ACs get waved through on "it works in the eph env", and the gaps
40
+ surface months later during an incident. This skill closes that lane.
41
+
42
+ Two failure modes it is built against specifically:
43
+
44
+ - **A confident `PASS` backed by a code reading looks exactly like a `PASS` backed by a
45
+ measurement**, and is worth far less. Every verdict here names the mode that produced it.
46
+ - **A client-side guard is not the endpoint's behaviour.** "The button is disabled until
47
+ you type" describes the browser. Every other client — mobile, integration, script —
48
+ sends the request the guard was preventing.
49
+
50
+ ## Quick Start
51
+
52
+ ```
53
+ # Verify a ticket against a target that declares a `backend:` block
54
+ /qa-verify-backend --target my-service-dev --context output/context/TICKET-123-context.md
55
+
56
+ # Verify from an inline ticket paste
57
+ /qa-verify-backend --target my-env
58
+ > "AC1: table X exists. AC2: stream enabled with NEW_IMAGE. AC3: write-only IAM."
59
+
60
+ # Static review only (no cloud credentials available)
61
+ /qa-verify-backend --target my-env --static-only
62
+
63
+ # API contract/behaviour verification, and the same matrix against a second environment
64
+ /qa-verify-backend --target my-service-dev --api-only --parity my-service-staging
65
+ ```
66
+
67
+ ## Session Phases
68
+
69
+ Execute in order. Read and follow the linked file.
70
+
71
+ | Phase | File | Summary |
72
+ |-------|------|---------|
73
+ | **Setup** | `phases/00-setup.md` | Resolve target, ticket context, repo branch, AWS access. Decide which lanes are runnable. |
74
+ | **AC Decomposition** | `phases/01-ac-decomposition.md` | Turn each AC into a falsifiable check with a named evidence source. Flag untestable ACs. |
75
+ | **Static Review** | `phases/02-static-review.md` | Read the branch diff against every AC. Producer/consumer contract. Spec drift. Negative space. |
76
+ | **Live Verification** | `phases/03-live-verification.md` | Read-only aws-cli probes of the cloud resources. One probe per AC. Capture raw evidence. |
77
+ | **API Verification** | `phases/03b-api-verification.md` | Fingerprint the environment, then call the endpoints directly from the authenticated session. Case matrix, cross-environment parity, UI-vs-API differential. |
78
+ | **End-to-End Trigger** | `phases/04-e2e-trigger.md` | Drive the real write path (UI or API), then re-probe the data layer. Covers "verified in env" ACs. |
79
+ | **Reporting** | `phases/05-reporting.md` | AC traceability matrix, bug reports with business impact, DoD gaps. |
80
+
81
+ ## References
82
+
83
+ - **`references/aws-readonly-probes.md`** — canonical probe commands per resource type, and how to read their output
84
+ - **`references/api-probes.md`** — getting an authenticated request context without handling a token, the case families worth probing on every endpoint, and how to read the results
85
+ - **`references/environment-fingerprinting.md`** — proving which build and which implementation an environment actually runs, before any verdict is written
86
+ - **`references/payload-verification.md`** — recomputing derived values, saying what a single-source check does not prove, and comparing the rendered value against the payload it came from
87
+ - **`references/release-readiness.md`** — verifying many tickets against one build: the deployment table, the result vocabulary, carry-overs, disposition
88
+ - **`references/safety-rules.md`** — read-only discipline, production guardrails, redaction, confidentiality
89
+
90
+ ## Knowledge Base
91
+
92
+ Load these from `data/knowledge/` before the static and live lanes. They are the
93
+ BE/API verification layer — techniques derived from real sessions with this skill,
94
+ not from published literature.
95
+
96
+ | Entry | Use it when |
97
+ |-------|-------------|
98
+ | `technique-verification-mode-selection` | **Every session, at AC decomposition.** Routes each AC to the channel that can actually falsify it — API-behind-the-screen, direct request to a no-UI endpoint, LLM output judgement, or needs-a-human. Defines when `UNVERIFIABLE` is the correct verdict. |
99
+ | `technique-functional-diff-analysis` | The static lane. Eight passes that read a diff for behaviour at the service boundary instead of code quality, and output falsifiable hypotheses attached to ACs. |
100
+ | `technique-contract-narrowing` | Any ticket that swaps a data source on a read path — "replace X with Y", "migrate to", "switch to". |
101
+ | `technique-test-suite-audit` | Whenever the change rewrites the tests that are meant to prove it works. |
102
+ | `technique-llm-output-verification` | The change touches a prompt, tool definition, model version, retrieval step, or agent orchestration. |
103
+ | `technique-environment-fingerprinting` | **Every session, before the first probe.** Establishes which build and which implementation the environment runs, so a verdict is about code rather than about luck. |
104
+ | `technique-authenticated-api-probing` | Any AC about an endpoint's own behaviour — validation, filters, pagination, error contract, payload shape. |
105
+ | `technique-ui-api-differential` | The ticket has both a screen and an endpoint. Sorts findings into UI-only, API-only and both. |
106
+ | `technique-silent-failure-audit` | Any read path with an error branch — which is all of them. Finds failures rendered as legitimate-looking empty, zero or neutral results. |
107
+ | `technique-derived-value-verification` | The AC concerns a calculated value — a percentage, total, ratio, delta or aggregate. |
108
+ | `technique-presentation-integrity` | The value is also shown on a screen. The payload can be right and the render wrong. |
109
+ | `technique-expected-behaviour-specification` | The session found behaviour no AC covers — which is most of what an API probe turns up. |
110
+ | `technique-config-surface-verification` | The change under test *is* the configuration mechanism — a flag endpoint, a settings surface, a shared config library. |
111
+ | `technique-release-readiness-verification` | The ask is "is this release good to go", not "does this ticket meet its ACs". |
112
+
113
+ The mode selection entry governs the others: it decides *how* an AC gets observed,
114
+ and the rest supply *what to look for* once the channel is chosen.
115
+
116
+ ## Core Rules
117
+
118
+ 1. **Evidence or it did not happen.** Every `PASS` cites a command output, a file and
119
+ line, or a screenshot. "Looks right" is not a verdict.
120
+ 2. **Read-only against anything you did not create.** Probes are `describe-*`,
121
+ `get-*`, `list-*`, `scan`, `query`, `simulate-principal-policy`. Never mutate.
122
+ See `references/safety-rules.md` for the production hard stop.
123
+ 3. **The ticket text is the spec, the code is the claim.** Where they differ, that
124
+ is a finding — even when the code is arguably better. Say which you think is right.
125
+ 4. **Verify the contract, not just the resource.** If another ticket will read this
126
+ data, check attribute names, types and key shape against what that consumer needs.
127
+ Producer/consumer drift is the most expensive bug this skill can catch.
128
+ 5. **Test the failure path.** Retries, DLQs, partial batch failures, missing claims,
129
+ size limits. Audit and event systems fail silently by default.
130
+ 6. **Absence is a finding.** An AC that nothing implements, a cleanup that was written
131
+ but never run, a doc that still describes the deleted architecture.
132
+ 7. **One bug = one report**, each with Business Impact, per `data/templates/bug-report.md`.
133
+ 8. **Name the observation mode in every verdict.** `PASS (direct request, raw
134
+ response attached)` and `PASS (read the diff)` are different claims. A verdict
135
+ whose only evidence is a code reading is `UNVERIFIABLE`, not `PASS` — see
136
+ `technique-verification-mode-selection`.
137
+ 9. **Fingerprint before you judge.** A verdict is a claim about code; a probe is an
138
+ observation of an environment. Establish which build is deployed and whether the
139
+ changed path is even selected here — a flag can pick between two implementations
140
+ inside one identical build. An environment that does not run the change gets
141
+ `NOT-REACHABLE`, never `PASS`.
142
+ 10. **A guard is not a behaviour.** Never record a client-side restriction as a passing
143
+ AC. Call the endpoint the way the guard is preventing, then judge the answer on the
144
+ consequence of the call succeeding — not on how hard it was to make.
145
+ 11. **Compare shapes, not counts.** Across environments, status codes and response shapes
146
+ compare; absolute numbers do not. Different data, drifting under you.
147
+ 12. **Recompute every derived value, and say what that does not prove.** Take the formula
148
+ from the spec, never from the code under test. When both sides of the check come from
149
+ one payload you have verified the derivation, not the inputs — write that down and name
150
+ the independent oracle that would close it.
151
+ 13. **Hold the payload next to the screen.** A correct response can still reach the user as
152
+ a wrong number, and that defect is invisible from either surface alone. When it happens,
153
+ attribute it to the change that owns it — usually not the one under test.
154
+ 14. **Carry-overs get their own section.** A defect that pre-dates the change under test is
155
+ reported as pre-existing and tracked separately. Mixing it in inflates the apparent risk
156
+ of shipping and buries what actually belongs to this change.
157
+ 15. **BLOCKED is honest.** If credentials for the account are missing, say so, leave the
158
+ probe commands ready to run, and do not infer the verdict from the code.
159
+ 16. **Never widen the blast radius.** No `terraform apply`, no deploys, no writes to a
160
+ shared table, no destructive cleanup — not even when a runbook in the repo says to.
161
+ Propose it; let a human run it.
162
+ 17. **Redact.** Account IDs beyond what the target config already holds, tokens, emails
163
+ of real people, and consumer PII get `[REDACTED]` before anything is written to disk.
@@ -0,0 +1,97 @@
1
+ # Phase 0: Setup
2
+
3
+ ## Step 1: Resolve Inputs
4
+
5
+ Extract from the user message:
6
+ - **target**: id under `data/targets/` (with a `backend:` and/or `api:` block) — or inline
7
+ resource names
8
+ - **context**: path to a gathered ticket context file, or inline ticket text
9
+ - **flags**: `--static-only` (skip live probes), `--api-only` (skip the cloud lane),
10
+ `--no-e2e` (skip phase 4), `--parity <target-id>` (run the API matrix in a second
11
+ environment too)
12
+
13
+ Read the target config. Read `data/domains/<domain>.md` for the domain's data-integrity
14
+ checks — they become extra checks beyond the ticket's own ACs.
15
+
16
+ ## Step 2: Resolve the Source Branch
17
+
18
+ From `target.source`, confirm the repo and branch exist:
19
+
20
+ ```bash
21
+ git -C <repo_path> rev-parse --verify <branch>
22
+ git -C <repo_path> diff --stat <base_branch>...<branch>
23
+ ```
24
+
25
+ If the branch is not fetched, ask before running `git fetch` — it touches the user's repo.
26
+ Never check the branch out; read it with `git show <branch>:<path>` so the working tree
27
+ is untouched.
28
+
29
+ ## Step 3: Establish AWS Access (skip if `--static-only`)
30
+
31
+ ```bash
32
+ aws sts get-caller-identity --profile "$<aws_profile_env>"
33
+ ```
34
+
35
+ Compare the returned account to `backend.account_id`.
36
+
37
+ | Result | Action |
38
+ |--------|--------|
39
+ | Matches | Live lane is GO |
40
+ | Different account | **STOP the live lane.** Report it. Do not probe an account the ticket does not concern. |
41
+ | No credentials | Live lane is **BLOCKED**, not failed. Continue with static + write the probe commands into the report so a human can run them. |
42
+ | Account looks like production | Read `references/safety-rules.md` and apply the production hard stop. |
43
+
44
+ ## Step 3b: Establish the API Lane (skip if the ticket has no HTTP surface)
45
+
46
+ From `target.api`, confirm the endpoints under test and the probe allowlist. Open an
47
+ authenticated session once — headed, against a persistent profile, because an
48
+ interactive identity provider cannot complete headless:
49
+
50
+ ```bash
51
+ playwright-cli -s=<target-id> open <base_url> --headed --profile .auth/<profile>
52
+ ```
53
+
54
+ Confirm the session is live before building any matrix — a probe run against a logged-out
55
+ page returns a login page with a `200`, which reads exactly like a passing case.
56
+
57
+ If the target declares `api.parity_targets` or `--parity` was passed, prime that session
58
+ too. Same matrix, both environments, or the comparison is not one.
59
+
60
+ ## Step 3c: Fingerprint the Environment
61
+
62
+ **Before the first probe of any lane.** Read
63
+ `references/environment-fingerprinting.md` and answer its three questions: which build is
64
+ deployed in every component of the path, whether the changed code path is *selected*
65
+ here, and whether the commit under test is genuinely an ancestor of what is running.
66
+
67
+ Write the answers to `evidence/fingerprint.md`. If the changed path is not reachable in
68
+ this environment, every AC that depends on it is `NOT-REACHABLE` — record that now, and
69
+ say which environment *would* answer the question.
70
+
71
+ Record the outcome. A blocked live lane changes verdicts, not effort — every AC that
72
+ needed a probe becomes `BLOCKED` with the exact command attached.
73
+
74
+ ## Step 4: Create the Session Directory
75
+
76
+ `output/sessions/<YYYY-MM-DD-HHmm>-<ticket>-backend/` containing:
77
+ `charter.md`, `ac-matrix.md`, `evidence/`, `bugs/`, `probes/`, `session-log.md`
78
+
79
+ `probes/` holds the probe scripts and case files, so the run is repeatable by someone
80
+ else — and by you, against the next environment.
81
+
82
+ ## Step 5: Write the Charter
83
+
84
+ ```markdown
85
+ # Charter — <TICKET>
86
+ **Mission:** Verify <n> acceptance criteria for <ticket summary>
87
+ **Target:** <target id> (<environment kind>)
88
+ **Branch:** <branch> vs <base_branch>
89
+ **Lanes:** static: GO | live: GO/BLOCKED (<reason>) | api: GO/SKIPPED | e2e: GO/SKIPPED
90
+ **Build under test:** <component: build id / commit>, <component: build id / commit>
91
+ **Changed path selected here:** yes / no (<flag or config that decides it>)
92
+ **Out of scope:** <what the ticket defers to other tickets>
93
+ **Time box:** 45 min
94
+ ```
95
+
96
+ Append to `session-log.md` after this and every later phase:
97
+ `[<ts>] [PHASE] setup complete — lanes: static GO, live BLOCKED (no creds for <account>)`
@@ -0,0 +1,58 @@
1
+ # Phase 1: AC Decomposition
2
+
3
+ Turn prose acceptance criteria into falsifiable checks **before** you look at the
4
+ implementation. Doing this first is what stops you from rationalising whatever the
5
+ code happens to do.
6
+
7
+ ## For Each AC, Write
8
+
9
+ | Field | Meaning |
10
+ |-------|---------|
11
+ | **ID** | AC1, AC2… as numbered in the ticket |
12
+ | **Claim** | The AC restated as a single falsifiable statement |
13
+ | **Evidence source** | `static` (file+line), `live` (cloud probe command), `api` (request + raw response), `e2e` (trigger + re-probe), or a combination |
14
+ | **Falsifier** | What observation would prove this AC false — name it now |
15
+ | **Consumer** | Which downstream ticket/system depends on this being true in a particular shape |
16
+
17
+ Example:
18
+
19
+ ```
20
+ AC4: "DynamoDB Stream (NEW_IMAGE) is enabled on the existing config table"
21
+ Claim: The config table's stream is enabled with view type exactly NEW_IMAGE.
22
+ Evidence: static → the table resource in IaC; live → describe-table StreamSpecification
23
+ Falsifier: StreamViewType is anything other than NEW_IMAGE, or StreamEnabled is false
24
+ Consumer: the stream Lambda's mapper — if OldImage is absent, delete handling changes
25
+ ```
26
+
27
+ ## Flag Untestable ACs
28
+
29
+ An AC is untestable when it has no observable falsifier. Common shapes:
30
+ - **Unbounded removal** — "all X are removed". Removed from where? Define the search
31
+ space (this repo, this module, this account) and say which you checked.
32
+ - **Manual data work** — "garbage entries are cleaned up". Cleaned up in which
33
+ environments? A merged runbook is not executed work.
34
+ - **Vague quality words** — "clean", "maintainable", "simple". Convert to something
35
+ countable or mark it as not independently verifiable.
36
+ - **A guard described as a behaviour** — "the user cannot submit an empty search". That is
37
+ a statement about the client. Restate it as the endpoint's contract ("an empty query
38
+ returns 400 with an error body") and route it to the `api` lane. If the ticket only ever
39
+ meant the client, say so and add the endpoint's behaviour as a negative-space check.
40
+
41
+ ## Name the Environment in the Claim
42
+
43
+ Every claim is a claim about an environment. Write it in: *"in staging, an empty query
44
+ returns 400"*. An AC verified in one environment and asserted for another is the single
45
+ most common false `PASS` this skill exists to prevent — see
46
+ `references/environment-fingerprinting.md`.
47
+
48
+ Report these as **DoR gaps**, not as failures of the developer. They are ticket-writing
49
+ defects and they belong in the report — an untestable AC will be marked done by whoever
50
+ is under the least time pressure.
51
+
52
+ ## Add Domain Checks
53
+
54
+ From `data/domains/<domain>.md`, pull the data-integrity checks and add any that this
55
+ change touches but the ticket never mentions. These become findings in the
56
+ **negative space** — the things the ACs forgot to require.
57
+
58
+ Write the skeleton matrix to `ac-matrix.md` with every verdict set to `PENDING`.
@@ -0,0 +1,71 @@
1
+ # Phase 2: Static Review
2
+
3
+ Read the branch against every AC. Use `git show <branch>:<path>` and
4
+ `git diff <base>...<branch> -- <path>` — never check out.
5
+
6
+ ## Step 1: Map the Diff to the ACs
7
+
8
+ ```bash
9
+ git -C <repo> diff --stat <base>...<branch>
10
+ ```
11
+
12
+ Group changed files by AC. Any file group that maps to **no** AC is either scope creep
13
+ or an undocumented dependency — both worth a line in the report. Any AC that maps to
14
+ **no** file is a candidate `FAIL` before you have read a single line.
15
+
16
+ ## Step 2: Read the Implementation Against Each Claim
17
+
18
+ Work AC by AC. For each, open the actual resource or handler and compare it to the
19
+ claim you wrote in phase 1. Record file and line for evidence.
20
+
21
+ Read with these lenses:
22
+
23
+ **Spec drift** — names, types, key shapes, enum values. The ticket says one thing, the
24
+ code says another. Even a benign-looking rename (`newImage` → `newConfig`, `PK` →
25
+ `entityId`, Map → JSON string) is a finding, because the consumer ticket was written
26
+ against the ticket's names.
27
+
28
+ **Scope** — is anything here that the ticket put explicitly **out of scope**? Storing
29
+ data the design said was not needed is not a bonus; it is unreviewed data collection
30
+ with privacy and size consequences.
31
+
32
+ **Least privilege** — enumerate the actions each policy grants and compare to what the
33
+ component provably needs. Note permissions created but unattached, and permissions
34
+ attached but unused.
35
+
36
+ **Failure path** — for every asynchronous component: what happens on retry exhaustion,
37
+ partial batch failure, oversized payload, missing field, or a downstream throttle? Find
38
+ where the error surfaces. If it surfaces nowhere, that is your highest-value finding.
39
+
40
+ **Uniqueness and ordering** — for anything keyed by time: what is the actual timestamp
41
+ granularity, and what happens to two events inside one tick? Collisions in a
42
+ write-once store mean silent overwrites.
43
+
44
+ **Identity propagation** — trace the actor from the token, through the handler, into
45
+ the persisted item, into the derived record. Find every place it can silently default
46
+ to a placeholder. Check create, update **and delete** separately — deletes routinely
47
+ attribute to the wrong person because the only image available is the pre-delete one.
48
+
49
+ **Documentation** — does any README, diagram or runbook still describe the architecture
50
+ this ticket removed? Stale docs fail the DoD line "documentation is created/updated"
51
+ and mislead the next engineer.
52
+
53
+ ## Step 3: Verify the Consumer Contract
54
+
55
+ If the ticket names a downstream ticket, write out the shape that consumer will need
56
+ and diff it against what this branch produces. Attribute names, types, key format,
57
+ sort semantics. Producer/consumer drift discovered here costs a rename; discovered in
58
+ the consumer's sprint it costs a migration of live data.
59
+
60
+ ## Step 4: Check the Tests
61
+
62
+ ```bash
63
+ git -C <repo> diff --stat <base>...<branch> -- '*.spec.*' '*.test.*'
64
+ ```
65
+
66
+ For each AC, is there a test that would fail if the AC broke? Note tests that were
67
+ **deleted** by this branch and not replaced — especially tests whose names described
68
+ behaviour that still ought to hold. A deleted test is a removed guardrail.
69
+
70
+ Record every finding with file, line, AC reference and severity. Write findings to
71
+ `evidence/static-review.md` as you go — do not carry them all in context.