qualiow-exploratory-testing 1.2.0 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (111) hide show
  1. package/.claude-plugin/plugin.json +22 -0
  2. package/.env.example +27 -0
  3. package/CHANGELOG.md +194 -0
  4. package/README.md +428 -179
  5. package/agents/qa-gather-agent.md +24 -0
  6. package/{scripts → bin}/doctor-mobile.sh +57 -18
  7. package/bin/mcli +15 -9
  8. package/bin/mobile-cli.mjs +574 -286
  9. package/{scripts → bin}/setup-mobile.sh +56 -19
  10. package/bin/wadb +10 -4
  11. package/bin/wk-ios +29 -17
  12. package/bin/wkeval.mjs +7 -1
  13. package/data/domains/_default.yml +1 -1
  14. package/data/domains/identity.yml +138 -0
  15. package/data/knowledge/changelog.yml +104 -0
  16. package/data/knowledge/learned-patterns.md +172 -86
  17. package/data/knowledge/manifest.yml +481 -70
  18. package/data/knowledge/releases/v0.1.0/entries/technique-business-logic-race-conditions.yml +1 -1
  19. package/data/knowledge/releases/v0.1.0/release.yml +8 -1
  20. package/data/knowledge/releases/v0.2.0/entries/technique-contract-narrowing.yml +150 -0
  21. package/data/knowledge/releases/v0.2.0/entries/technique-test-suite-audit.yml +116 -0
  22. package/data/knowledge/releases/v0.2.0/release.yml +20 -0
  23. package/data/knowledge/releases/v0.3.0/entries/technique-functional-diff-analysis.yml +119 -0
  24. package/data/knowledge/releases/v0.3.0/entries/technique-llm-output-verification.yml +102 -0
  25. package/data/knowledge/releases/v0.3.0/entries/technique-verification-mode-selection.yml +153 -0
  26. package/data/knowledge/releases/v0.3.0/release.yml +25 -0
  27. package/data/knowledge/releases/v0.4.0/entries/technique-authenticated-api-probing.yml +137 -0
  28. package/data/knowledge/releases/v0.4.0/entries/technique-environment-fingerprinting.yml +113 -0
  29. package/data/knowledge/releases/v0.4.0/entries/technique-silent-failure-audit.yml +86 -0
  30. package/data/knowledge/releases/v0.4.0/entries/technique-ui-api-differential.yml +97 -0
  31. package/data/knowledge/releases/v0.4.0/release.yml +28 -0
  32. package/data/knowledge/releases/v0.5.0/entries/technique-config-surface-verification.yml +77 -0
  33. package/data/knowledge/releases/v0.5.0/entries/technique-derived-value-verification.yml +81 -0
  34. package/data/knowledge/releases/v0.5.0/entries/technique-expected-behaviour-specification.yml +97 -0
  35. package/data/knowledge/releases/v0.5.0/entries/technique-presentation-integrity.yml +78 -0
  36. package/data/knowledge/releases/v0.5.0/entries/technique-release-readiness-verification.yml +110 -0
  37. package/data/knowledge/releases/v0.5.0/release.yml +33 -0
  38. package/data/knowledge/releases/v0.6.0/entries/technique-async-callback-contracts.yml +98 -0
  39. package/data/knowledge/releases/v0.6.0/entries/technique-exactly-once-verification.yml +116 -0
  40. package/data/knowledge/releases/v0.6.0/release.yml +30 -0
  41. package/data/security/SECURITY-POLICY.md +273 -0
  42. package/data/targets/_default.yml +1 -1
  43. package/data/targets/_example-api-only.yml +80 -0
  44. package/data/targets/_example-backend.yml +123 -0
  45. package/data/targets/_example-mobile-emulation.yml +2 -2
  46. package/data/targets/_example-native-mobile.yml +2 -2
  47. package/data/targets/_example-sim-android-chrome.yml +1 -1
  48. package/data/targets/_example-sim-ios-safari.yml +1 -1
  49. package/data/templates/api-probe-matrix.md +99 -0
  50. package/data/templates/bug-report.md +29 -151
  51. package/data/templates/coverage-map.md +23 -6
  52. package/data/templates/expected-behaviour.md +94 -0
  53. package/data/templates/session-charter.md +3 -0
  54. package/data/templates/session-report.md +57 -80
  55. package/dist/cli/index.js +1500 -739
  56. package/dist/index.d.ts +673 -75
  57. package/dist/index.js +1664 -270
  58. package/docs/BACKEND-VERIFICATION.md +307 -0
  59. package/docs/GETTING-STARTED.md +188 -128
  60. package/docs/MOBILE-SETUP.md +149 -18
  61. package/package.json +32 -16
  62. package/skills/qa-explore/SKILL.md +31 -25
  63. package/skills/qa-explore/phases/00-setup.md +67 -41
  64. package/skills/qa-explore/phases/01-auth.md +32 -15
  65. package/skills/qa-explore/phases/02-charter.md +16 -5
  66. package/skills/qa-explore/phases/03-discovery.md +50 -26
  67. package/skills/qa-explore/phases/04-journeys.md +2 -2
  68. package/skills/qa-explore/phases/05-features.md +39 -24
  69. package/skills/qa-explore/phases/06-edge-cases.md +24 -13
  70. package/skills/qa-explore/phases/07-reporting.md +79 -140
  71. package/skills/qa-explore/references/context-integration.md +1 -1
  72. package/skills/qa-explore/references/output-contract.md +187 -0
  73. package/skills/qa-explore/references/paths.md +84 -0
  74. package/skills/qa-explore/references/playwright-agents-integration.md +61 -84
  75. package/skills/qa-explore/references/security-rules.md +66 -25
  76. package/skills/qa-explore/references/severity-guide.md +31 -4
  77. package/skills/qa-explore-cleanup/SKILL.md +17 -6
  78. package/skills/qa-explore-feedback/SKILL.md +18 -5
  79. package/skills/qa-explore-mobile/SKILL.md +86 -64
  80. package/skills/qa-explore-mobile/phases/00-setup.md +71 -62
  81. package/skills/qa-explore-mobile/phases/01-auth.md +28 -31
  82. package/skills/qa-explore-mobile/phases/02-charter.md +8 -4
  83. package/skills/qa-explore-mobile/phases/03-discovery.md +17 -11
  84. package/skills/qa-explore-mobile/phases/04-journeys.md +6 -6
  85. package/skills/qa-explore-mobile/phases/05-features.md +9 -9
  86. package/skills/qa-explore-mobile/phases/06-edge-cases.md +35 -40
  87. package/skills/qa-explore-mobile/phases/07-reporting.md +82 -56
  88. package/skills/qa-explore-mobile/references/mobile-edges.md +26 -26
  89. package/skills/qa-explore-quick/SKILL.md +60 -25
  90. package/skills/qa-explore-report/SKILL.md +40 -34
  91. package/skills/qa-gather/SKILL.md +17 -2
  92. package/skills/qa-knowledge-add/SKILL.md +20 -9
  93. package/skills/qa-knowledge-list/SKILL.md +12 -3
  94. package/skills/qa-target-setup/SKILL.md +63 -41
  95. package/skills/qa-verify-backend/SKILL.md +172 -0
  96. package/skills/qa-verify-backend/phases/00-setup.md +142 -0
  97. package/skills/qa-verify-backend/phases/01-ac-decomposition.md +58 -0
  98. package/skills/qa-verify-backend/phases/02-static-review.md +71 -0
  99. package/skills/qa-verify-backend/phases/03-live-verification.md +62 -0
  100. package/skills/qa-verify-backend/phases/03b-api-verification.md +170 -0
  101. package/skills/qa-verify-backend/phases/04-e2e-trigger.md +82 -0
  102. package/skills/qa-verify-backend/phases/05-reporting.md +163 -0
  103. package/skills/qa-verify-backend/references/api-probes.md +168 -0
  104. package/skills/qa-verify-backend/references/aws-readonly-probes.md +156 -0
  105. package/skills/qa-verify-backend/references/environment-fingerprinting.md +104 -0
  106. package/skills/qa-verify-backend/references/payload-verification.md +112 -0
  107. package/skills/qa-verify-backend/references/release-readiness.md +97 -0
  108. package/skills/qa-verify-backend/references/safety-rules.md +95 -0
  109. package/dist/cli/index.d.ts +0 -1
  110. package/dist/cli/index.js.map +0 -1
  111. package/dist/index.js.map +0 -1
@@ -1,7 +1,7 @@
1
1
  # Template target — WEB-MODE mobile app in the REAL Android Emulator Chrome.
2
2
  # Copy this file, rename it, and point web.base_url at your mobile web app.
3
3
  # For NATIVE mode (an installed app under test), see _example-native-mobile.yml.
4
- id: example-sim-android-chrome
4
+ id: _example-sim-android-chrome
5
5
  name: "Example Mobile Web App — Android Emulator Chrome"
6
6
  platform: android
7
7
  domain: _default
@@ -1,7 +1,7 @@
1
1
  # Template target — WEB-MODE mobile app in the REAL iOS Simulator Safari.
2
2
  # Copy this file, rename it, and point web.base_url at your mobile web app.
3
3
  # For NATIVE mode (an installed app under test), see _example-native-mobile.yml.
4
- id: example-sim-ios-safari
4
+ id: _example-sim-ios-safari
5
5
  name: "Example Mobile Web App — iOS Simulator Safari"
6
6
  platform: ios
7
7
  domain: _default
@@ -0,0 +1,99 @@
1
+ > CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
2
+ > and application details. Do not share outside your organization without review.
3
+
4
+ # API Probe Matrix
5
+
6
+ ## Session Reference
7
+
8
+ | Property | Value |
9
+ |-----------------|-----------------------------------------|
10
+ | Session ID | [e.g., SESSION-2026-03-28-001] |
11
+ | Ticket | [ticket id, or "none"] |
12
+ | Endpoint | [`POST /api/v1/…` — note the verb] |
13
+ | Date | [YYYY-MM-DD] |
14
+ | Observation mode| [in-page fetch / out-of-browser request / unauthenticated] |
15
+ | Identity | [role or persona used — never a credential] |
16
+
17
+ ---
18
+
19
+ ## Environment Fingerprint
20
+
21
+ Fill this in **before** the matrix. Without it the results below are uninterpretable.
22
+
23
+ | Component | Environment A `[name]` | Environment B `[name]` |
24
+ |------------------|------------------------|------------------------|
25
+ | Frontend build | [build id / date] | [build id / date] |
26
+ | API build | [build id / commit] | [build id / commit] |
27
+ | Service under test | [build id / commit] | [build id / commit] |
28
+ | Changed path selected? | [yes / no — name the flag or config that decides it] | [yes / no] |
29
+ | Commit under test deployed? | [ancestor / not an ancestor] | [ancestor / not an ancestor] |
30
+
31
+ > **Same build, different behaviour ⇒ configuration, not deploy lag.**
32
+ > If the changed path is not selected in an environment, every AC that depends on it is
33
+ > `NOT-REACHABLE` there — not `PASS`, not `FAIL`.
34
+
35
+ ---
36
+
37
+ ## Cases
38
+
39
+ One row per case. The **label** is the stable identifier the report, the bug and the
40
+ re-run all refer to. Record passes as well as failures — a matrix of only failures
41
+ cannot show a regression later.
42
+
43
+ | # | Label | Request | Expected | Env A: status / count / rows | Env B: status / count / rows | Verdict |
44
+ |---|-------|---------|----------|------------------------------|------------------------------|---------|
45
+ | 1 | identity | `{"query":"<known value>"}` | 200, stable non-zero | 200 / 77 / 20 | 200 / 91 / 20 | ✅ |
46
+ | 2 | empty string | `{"query":""}` | 400 + error body | | | |
47
+ | 3 | body absent | `{}` | 400 + error body | | | |
48
+ | 4 | metachar — lone `(` | `{"query":"("}` | 400 or escaped 200 | | | |
49
+ | 5 | limit above cap | `?limit=<cap+1>` | 400, or capped | | | |
50
+ | 6 | invalid enum | `{"filter":["<nonsense>"]}` | 400 + error body | | | |
51
+ | 7 | second identity | same request, other user | scoped result | | | |
52
+
53
+ > **Compare status codes and shapes. Never compare absolute counts** — different
54
+ > environments hold different data and it drifts between runs.
55
+
56
+ ### Case families to cover
57
+
58
+ Boundary lengths · tokenisation and whitespace · metacharacters of the underlying query
59
+ language · empty / null / absent / whitespace-only · enum values valid, invalid, empty,
60
+ wildcard, injected · pagination bounds and types · type confusion · casing and
61
+ normalisation · second identity and second scope.
62
+
63
+ ---
64
+
65
+ ## Findings
66
+
67
+ One row per finding, not one per failing case — a family of inputs failing on the same
68
+ character is **one** bug with several examples.
69
+
70
+ | ID | Cases | Finding | Surface | Severity | Bug report |
71
+ |----|-------|---------|---------|----------|------------|
72
+ | F1 | 4, 9–14 | [what is wrong, in one line] | API / UI / both | High | `bugs/BUG-0xx-….md` |
73
+
74
+ **Surface** classifies where the defect lives, from the differential pass:
75
+
76
+ | Surface | Meaning | Owner |
77
+ |---------|---------|-------|
78
+ | **Both** | The UI faithfully reflects the endpoint | Service |
79
+ | **API only** | Real defect the client's guard is hiding — reachable by every other client | Service |
80
+ | **UI only** | The client invents or masks behaviour the service does not have | Client |
81
+ | **Neither** | Environment or access noise, not a product defect | — |
82
+
83
+ ---
84
+
85
+ ## Evidence
86
+
87
+ | File | Contents |
88
+ |------|----------|
89
+ | `evidence/fingerprint.md` | Build ids, flag values, commit ancestry |
90
+ | `evidence/<case-set>-raw.json` | Raw per-case output, redacted |
91
+ | `probes/<probe>.js` | The probe script, so this is repeatable |
92
+
93
+ ---
94
+
95
+ ## Notes and Open Questions
96
+
97
+ - [What could not be checked, and exactly what access would unblock it]
98
+ - [Which environment would answer a `NOT-REACHABLE` verdict]
99
+ - [Counts recorded here are valid for the date above only]
@@ -1,178 +1,56 @@
1
- # Bug Report
1
+ > CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
2
+ > and application details. Do not share outside your organization without review.
2
3
 
3
- ## Title
4
+ # BUG-NNN: [Component] fails [Condition] causing [Impact]
4
5
 
5
- `[Component] fails [Condition] causing [Impact]`
6
+ > _Title example: `[Login Form] fails [when password contains special characters] causing [authentication error and user lockout]`. Number bugs from BUG-001 per session; one file per bug at `bugs/BUG-NNN.md`._
6
7
 
7
- > _Example: [Login Form] fails [when password contains special characters] causing [authentication error and user lockout]_
8
+ **Severity:** Critical | High | Medium | Low
9
+ **Priority:** P0 | P1 | P2 | P3
10
+ **Component:** [Component]
11
+ **URL:** [exact URL where the bug was found — mobile: screen name or deep link — backend: endpoint or resource]
12
+ **Environment:** [e.g. Playwright CLI, Chromium, 1280x720, logged in as X — mobile: mode / platform / device / OS — backend: env kind + build]
13
+ **Reproduction rate:** Always | Intermittent (~X%) | Once
8
14
 
9
- ---
15
+ > _Severity per `severity-guide.md`: Critical = data loss, security breach, financial loss, system unusable; High = major feature broken, no reasonable workaround; Medium = partial break, workaround exists, or an important missing feature; Low = cosmetic, rare edge case. **When in doubt, go one level LOWER.** Priority = severity adjusted by workaround and blast radius (P0 hours · P1 this sprint · P2 next sprint · P3 backlog)._
10
16
 
11
- ## Severity
17
+ ## Summary
12
18
 
13
- > Select one. See definitions below.
14
-
15
- - [ ] **Critical**
16
- - [ ] **High**
17
- - [ ] **Medium**
18
- - [ ] **Low**
19
-
20
- ### Severity Level Definitions
21
-
22
- | Level | Definition | Examples |
23
- |----------|----------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------|
24
- | Critical | System down, data loss, security breach. Blocks all users immediately. | Payment gateway crashes on all transactions; user passwords exposed in API response; database corruption losing customer orders. |
25
- | High | Major feature broken, many users affected. No reasonable workaround. | Checkout fails for all credit card payments; search returns zero results for every query; login broken on mobile devices. |
26
- | Medium | Partial break, workaround exists. Some users affected under certain paths. | Discount code applies twice if user clicks fast; filter resets after pagination; profile photo upload fails on Safari only. |
27
- | Low | Cosmetic issue, rare edge case. Minimal user or business impact. | Footer alignment off by 2px on wide screens; tooltip text truncated in German locale; hover state missing on disabled button. |
28
-
29
- > **Note: When in doubt, go one level LOWER. Severity inflation destroys credibility.** A bug report rated "Critical" that turns out to be cosmetic undermines every future report you file. Reserve Critical for genuine emergencies. If you hesitate between two levels, pick the lower one and let the evidence speak for itself.
30
-
31
- ---
32
-
33
- ## URL
34
-
35
- `[URL where the bug was found]`
36
-
37
- ## Environment
38
-
39
- | Property | Value |
40
- |-----------|---------------------------|
41
- | Browser | [e.g., Chrome 125.0] |
42
- | Viewport | [e.g., 1920x1080] |
43
- | OS | [e.g., macOS 15.x] |
44
- | Auth | [e.g., logged in as X] |
45
- | Device | [e.g., Desktop / Mobile] |
46
-
47
- ---
19
+ [Two sentences: what breaks, and why it matters to the business.]
48
20
 
49
21
  ## Expected Behavior
50
22
 
51
- _What should happen according to requirements, conventions, or reasonable user expectations._
52
-
53
- > [Describe the expected outcome clearly.]
23
+ [What should happen according to requirements, conventions, or reasonable user expectations.]
54
24
 
55
25
  ## Actual Behavior
56
26
 
57
- _What actually happens instead._
58
-
59
- > [Describe the actual outcome. Be precise -- include error messages verbatim if applicable.]
60
-
61
- ---
27
+ [What actually happens instead. Be precise; include error messages verbatim.]
62
28
 
63
29
  ## Steps to Reproduce
64
30
 
65
31
  1. Navigate to `[URL]`
66
32
  2. [Action 2]
67
33
  3. [Action 3]
68
- 4. [Action 4]
69
- 5. Observe: [what goes wrong]
70
-
71
- > _Reproduction rate: [Always / Intermittent (~X%) / Once]_
72
-
73
- ---
34
+ 4. Observe: [what goes wrong]
74
35
 
75
36
  ## Business Impact
76
37
 
77
- > This section is **mandatory**. Every bug exists in a business context. Articulating impact helps engineering prioritize correctly and helps stakeholders understand the risk without needing to read technical details.
78
-
79
- ### Revenue Impact
80
-
81
- > [Does this bug prevent purchases, cause incorrect charges, block upgrades, or reduce conversion? Quantify if possible -- e.g., "Blocks all guest checkout, which represents ~30% of orders."]
82
-
83
- ### Trust Impact
84
-
85
- > [Would a user lose confidence in the product? Would they contact support, leave a negative review, or churn? e.g., "User sees a $0.00 balance briefly before correct amount loads -- creates panic."]
86
-
87
- ### Regulatory Risk
88
-
89
- > [Does this violate any compliance requirements? Specify the regulation. e.g., "Credit card number visible in URL query string -- PCI DSS violation." If none, write "No known regulatory risk."]
90
- >
91
- > Common frameworks to consider: SOX, PCI DSS, GDPR, CCPA, HIPAA, WCAG 2.1 AA, CAN-SPAM, KYC/AML, Regulation E.
92
-
93
- ### Data Risk
94
-
95
- > [Is data corrupted, lost, or exposed? e.g., "Order total saved as $0 in database when coupon equals cart value -- data integrity failure." If none, write "No data risk identified."]
96
-
97
- ### Scale
98
-
99
- > [How many users are affected and under what conditions? e.g., "All users on iOS Safari 17+ (~18% of mobile traffic)" or "Only affects accounts created before the migration on 2025-01-15 (~200 accounts)."]
100
-
101
- ---
38
+ > _Mandatory. Answer at least one bullet; write "none identified" for the rest. Quantify where you can ("blocks all guest checkout, ~30% of orders")._
102
39
 
103
- ## Risk Priority
104
-
105
- > Use this matrix to communicate urgency objectively.
106
-
107
- | Factor | Rating | Justification |
108
- |-------------|------------------|----------------------------------------------------|
109
- | Likelihood | High / Med / Low | [How likely is a user to hit this? Every time? Rare edge case?] |
110
- | Impact | High / Med / Low | [How bad is it when they do? Data loss? Mild annoyance?] |
111
- | **Risk** | **[Result]** | **Risk = Likelihood x Impact** (see matrix below) |
112
-
113
- ### Risk Matrix
114
-
115
- | | **Impact: Low** | **Impact: Med** | **Impact: High** |
116
- |--------------------|-----------------|-----------------|-------------------|
117
- | **Likelihood: High** | Medium | High | Critical |
118
- | **Likelihood: Med** | Low | Medium | High |
119
- | **Likelihood: Low** | Low | Low | Medium |
120
-
121
- ---
122
-
123
- ## Recommended Fix Priority
124
-
125
- > **Priority: [P0 / P1 / P2 / P3]**
126
-
127
- | Priority | Meaning | Typical SLA |
128
- |----------|--------------------------------------|----------------------|
129
- | P0 | Fix immediately, interrupt sprint | Hours |
130
- | P1 | Fix this sprint | Days |
131
- | P2 | Fix next sprint | 1-2 weeks |
132
- | P3 | Backlog, fix when convenient | When capacity allows |
133
-
134
- **Justification:** [Explain why this priority level is appropriate given the risk assessment and business impact above. e.g., "Risk is High (likely + revenue-impacting), but a workaround exists (users can switch payment method), so P1 rather than P0."]
135
-
136
- ---
137
-
138
- ## Impact
139
-
140
- ### User Impact
141
-
142
- > [How does this affect the end user? Can they complete their task? What is the degraded experience?]
143
-
144
- ### Business Impact Summary
145
-
146
- > [One-sentence summary referencing the detailed Business Impact section above. e.g., "Blocks ~30% of revenue through guest checkout with no workaround."]
147
-
148
- ---
40
+ - **Revenue impact:** [prevents purchases, incorrect charges, blocked upgrades, reduced conversion?]
41
+ - **Trust impact:** [would a user lose confidence, contact support, churn?]
42
+ - **Regulatory risk:** [SOX, PCI DSS, GDPR, CCPA, HIPAA, WCAG 2.1 AA, KYC/AML, Regulation E — name the regulation, or "none identified"]
43
+ - **Data risk:** [corrupted, lost, or exposed data?]
44
+ - **Scale:** [how many users, under what conditions]
149
45
 
150
46
  ## Evidence
151
47
 
152
- ### Screenshots
153
-
154
- > _Path: `output/sessions/[session-id]/screenshots/[filename].png`_
155
-
156
- ### Console Errors
157
-
158
- ```
159
- [Paste any relevant console errors here]
160
- ```
48
+ - Screenshot: `screenshots/BUG-NNN.png`
49
+ - Video: `videos/BUG-NNN.webm` (if recorded; `.mp4` on mobile)
50
+ - Log: `logs/BUG-NNN.log` (mobile) / `evidence/<file>` (backend), or "none"
51
+ - Console errors: [verbatim, redacted, or "none"]
52
+ - Network failures: [method, URL, status, response excerpt — redacted — or "none"]
161
53
 
162
- ### Network Failures
163
-
164
- ```
165
- [Paste any failed requests -- method, URL, status code, response body excerpt]
166
- ```
167
-
168
- ### Additional Logs
169
-
170
- ```
171
- [Any other diagnostic output -- application logs, state dumps, etc.]
172
- ```
173
-
174
- ---
175
-
176
- ## Additional Context
54
+ ## Recommended Fix Priority
177
55
 
178
- > [Any other relevant information: related bugs, recent deploys, feature flags, user reports, etc.]
56
+ [Why this priority relative to the other findings: risk (likelihood × impact, see `severity-guide.md`), workaround, blast radius, and anything related — recent deploys, feature flags, user reports.]
@@ -1,10 +1,13 @@
1
+ > CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
2
+ > and application details. Do not share outside your organization without review.
3
+
1
4
  # Coverage Map
2
5
 
3
6
  ## Session Reference
4
7
 
5
8
  | Property | Value |
6
9
  |------------|--------------------------------|
7
- | Session ID | [e.g., SESSION-2026-03-28-001] |
10
+ | Session ID | [session directory, e.g. 2026-09-08-1420-explore-parabank] |
8
11
  | Target | [application or feature name] |
9
12
  | Date | [YYYY-MM-DD] |
10
13
 
@@ -12,17 +15,31 @@
12
15
 
13
16
  ## Overall Coverage
14
17
 
15
- | Area | Status | Heuristics Applied | Bugs Found | Notes |
16
- |------|--------|--------------------|------------|-------|
17
- | [Area 1] | tested / partial / not-tested | [list] | 0 | [notes] |
18
- | [Area 2] | tested / partial / not-tested | [list] | 0 | [notes] |
19
- | [Area 3] | tested / partial / not-tested | [list] | 0 | [notes] |
18
+ | Area | Risk | Status | Bugs | Notes |
19
+ |---|---|---|---|---|
20
+ | [Area 1] | P0 | tested | 0 | [heuristics applied, notes] |
21
+ | [Area 2] | P1 | partial | 0 | [what was not reached and why] |
22
+ | [Area 3] | P2 | not-tested | — | [reason] |
20
23
 
21
24
  ### Status Legend
22
25
 
23
26
  - **tested** -- Area was fully explored with the planned heuristics.
24
27
  - **partial** -- Area was visited but not all heuristics or paths were exercised.
25
28
  - **not-tested** -- Area was in scope but was not reached during the session.
29
+ - **code-verified-only** -- Believed correct from reading the source, never observed
30
+ running. This is `UNVERIFIABLE`, **not** a pass -- keep it visually distinct so it
31
+ cannot be skimmed as green.
32
+
33
+ Two distinctions that routinely produce a false **tested**:
34
+
35
+ - **Consistent is not causal.** A setting whose current value happens to match the output
36
+ proves nothing until the setting is changed and the output is watched to follow.
37
+ - **The data has to reach the case.** An area whose behaviour depends on values no
38
+ reachable record has -- a negative, a zero, an empty collection -- is **not-tested** for
39
+ that behaviour. Find such a record, create one, or say plainly what data would settle it.
40
+
41
+ Name the single **not-tested** item that carries the most risk. In a map full of green rows
42
+ that is the line the reader acts on.
26
43
 
27
44
  ---
28
45
 
@@ -0,0 +1,94 @@
1
+ > CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
2
+ > and application details. Do not share outside your organization without review.
3
+
4
+ # Expected Behaviour Specification
5
+
6
+ Use this when a verification session finds behaviour nobody wrote an acceptance criterion
7
+ for — which is most of what an API probe turns up. A bug report says *this is wrong*. This
8
+ says *here is what right looks like*, in enough detail that someone can implement it and
9
+ someone else can verify it. It is frequently the most valuable artifact a session produces,
10
+ because it converts a pile of findings into one reviewable decision.
11
+
12
+ Write it **per area**, not per case. Ten failing inputs that share a cause are one section.
13
+
14
+ ## Header
15
+
16
+ | Property | Value |
17
+ |----------|-------|
18
+ | Date | [YYYY-MM-DD] |
19
+ | Endpoint / surface | [`POST /api/v1/…` — note the verb] |
20
+ | Environment(s) observed | [name, and the build each one was running] |
21
+ | Source findings | [link to the session that produced the observations] |
22
+ | Status | [proposed / agreed / implemented] |
23
+
24
+ ---
25
+
26
+ ## [N]. [Area — e.g. Minimum input length]
27
+
28
+ **Observed**
29
+
30
+ > What the system does today, as evidence. Exact request, exact status, exact response.
31
+ > Note which environments it holds in — behaviour frequently differs, and that difference
32
+ > is part of the finding.
33
+
34
+ ```
35
+ {"query":"L"} → 200 / <the entire collection> [alpha]
36
+ {"query":"L"} → 200 / 0 [beta]
37
+ ```
38
+
39
+ **Expected**
40
+
41
+ > What it should do. Numbered, falsifiable statements — each one a testable assertion, not
42
+ > a sentiment. Include the boundary values themselves.
43
+
44
+ - An input whose tokens are all below the minimum → **`400`**, with the minimum named in
45
+ the message.
46
+ - "No usable tokens after filtering" and "no query supplied" are **different branches**.
47
+ Neither falls through to a match-all.
48
+ - The match-all path is not reachable from this parameter at any input. If "browse
49
+ everything" is wanted, it is an explicit endpoint with a hard server-side cap.
50
+ - The minimum is **N**, matching the existing rule, so that [known-good short inputs] keep
51
+ working.
52
+ - Client: no request fires below N; the user sees a hint rather than an empty result.
53
+
54
+ ### Decisions this section forces
55
+
56
+ State them explicitly — leaving them implicit is how a spec gets implemented three
57
+ different ways:
58
+
59
+ - **Reject, or clamp?** *Reject.* Silently serving a different page size than was asked for
60
+ breaks every caller that trusts the value back.
61
+ - **`400`, or a silent empty result?** *`400`.* A silent zero is indistinguishable from a
62
+ genuine no-match, and it will be read as data.
63
+ - **Where does validation live?** At the layer that covers **every** implementation of the
64
+ feature — not in one of two code paths, and never only in the client.
65
+ - **What must not change?** List the valid inputs and their current answers, so the fix is
66
+ verifiable as non-breaking.
67
+
68
+ ---
69
+
70
+ ## Priority
71
+
72
+ Rank the areas by what is reachable by real users **today**, not by how bad each one reads.
73
+ Name explicitly which finding is live in production right now, and which only appears in an
74
+ environment nobody ships from — the two need very different urgency, and the difference is
75
+ invisible unless someone says it.
76
+
77
+ ---
78
+
79
+ ## Reply — plain English
80
+
81
+ > Ready to paste into a ticket comment, a pull request or a chat reply. Same content, no
82
+ > jargon, no status codes unless they carry meaning to the reader. Written for the person
83
+ > who has to decide whether to fund the fix.
84
+
85
+ For each area, in a short paragraph:
86
+
87
+ - **What happens**, in a sentence a non-engineer can picture.
88
+ - **Whether it is live for users**, and where.
89
+ - **What the fix is**, in outline — enough to size it, not enough to design it.
90
+ - **Where you agree with their reading**, if they proposed one, and precisely where you do
91
+ not, and why.
92
+
93
+ Close with the one thing you would put ahead of everything else in the list, and the reason
94
+ — usually that it is the only one users are hitting today.
@@ -1,3 +1,6 @@
1
+ > CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
2
+ > and application details. Do not share outside your organization without review.
3
+
1
4
  # Session Charter
2
5
 
3
6
  ```
@@ -1,112 +1,89 @@
1
- # Session Report
1
+ > CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
2
+ > and application details. Do not share outside your organization without review.
2
3
 
3
- ## Session Metadata
4
-
5
- | Property | Value |
6
- |------------|--------------------------------|
7
- | Session ID | [e.g., SESSION-2026-03-28-001] |
8
- | Target | [application or feature name] |
9
- | URL | [target URL] |
10
- | Date | [YYYY-MM-DD] |
11
- | Duration | [e.g., 45 minutes] |
12
- | Tester | Claude AI |
13
- | Charter | [link to charter file] |
4
+ # Session Report — [target]
14
5
 
15
- ---
6
+ ## Session Metadata
16
7
 
17
- ## Charter Summary
8
+ | Field | Value |
9
+ |---|---|
10
+ | Session ID | [session directory, e.g. 2026-09-08-1420-explore-parabank] |
11
+ | Kind | explore / quick / mobile / backend |
12
+ | Target | [target id or application name] |
13
+ | URL | [base URL] |
14
+ | Date | [YYYY-MM-DD] |
15
+ | Duration | [N] min |
16
+ | Charter | charter.md |
18
17
 
19
- > [One-paragraph summary of the session charter: what was explored, with what tools/heuristics, and what we hoped to discover.]
18
+ ## Executive Summary
20
19
 
21
- ---
20
+ [Three sentences: what was tested, what was found, and the biggest risk.]
22
21
 
23
22
  ## Summary Stats
24
23
 
25
- | Metric | Count |
26
- |------------------------|-------|
27
- | Pages Explored | 0 |
28
- | Actions Performed | 0 |
29
- | Bugs Found -- Critical | 0 |
30
- | Bugs Found -- High | 0 |
31
- | Bugs Found -- Medium | 0 |
32
- | Bugs Found -- Low | 0 |
33
- | **Total Bugs** | **0** |
34
-
35
- ---
24
+ | Metric | Count |
25
+ |---|---|
26
+ | Pages explored | 0 |
27
+ | Actions performed | 0 |
28
+ | Bugs — Critical | 0 |
29
+ | Bugs — High | 0 |
30
+ | Bugs — Medium | 0 |
31
+ | Bugs — Low | 0 |
32
+ | **Total bugs** | **0** |
36
33
 
37
34
  ## Coverage Map
38
35
 
39
- _High-level view of what was tested. For full details see the dedicated coverage map file._
36
+ | Area | Risk | Status | Bugs | Notes |
37
+ |---|---|---|---|---|
38
+ | [Area 1] | P0 | tested | 0 | [notes] |
39
+ | [Area 2] | P1 | partial | 0 | [what was not reached and why] |
40
+ | [Area 3] | P2 | not-tested | — | [reason] |
40
41
 
41
- | Area | Status | Heuristics Applied | Bugs Found | Notes |
42
- |------|--------|--------------------|------------|-------|
43
- | [Area 1] | tested / partial / not-tested | [list] | 0 | [notes] |
44
- | [Area 2] | tested / partial / not-tested | [list] | 0 | [notes] |
45
- | [Area 3] | tested / partial / not-tested | [list] | 0 | [notes] |
46
-
47
- ---
42
+ _Status: `tested` (fully explored with the planned heuristics) · `partial` (visited, not all heuristics or paths exercised) · `not-tested` (in scope, not reached) · `code-verified-only` (believed correct from reading source, never observed running: this is UNVERIFIABLE, not a pass). Name the single not-tested item that carries the most risk._
48
43
 
49
44
  ## Bugs Found
50
45
 
51
- _Each bug links to its individual report in `output/bugs/`._
52
-
53
46
  | # | ID | Title | Severity | Report |
54
- |---|-----|-------|----------|--------|
55
- | 1 | [BUG-XXX] | [Component] fails [Condition] causing [Impact] | Critical/High/Medium/Low | [link] |
47
+ |---|---|---|---|---|
48
+ | 1 | BUG-001 | [Component] fails [Condition] causing [Impact] | Critical/High/Medium/Low | bugs/BUG-001.md |
56
49
 
57
- > _If no bugs were found, state: "No bugs were found during this session."_
58
-
59
- ---
50
+ _If no bugs were found, state: "No bugs were found during this session."_
60
51
 
61
52
  ## Observations
62
53
 
63
- _Things noticed during testing that are not bugs but may be worth discussing._
64
-
65
- - [Observation 1 -- e.g., page load time felt slow on dashboard but did not exceed threshold]
66
- - [Observation 2 -- e.g., tooltip text seems inconsistent with other similar features]
67
- - [Observation 3]
54
+ _Things noticed that are not bugs but may be worth discussing: what's MISSING against the domain checklist, data-integrity results, questions for the team._
68
55
 
69
- ---
70
-
71
- ## Questions
72
-
73
- _Unclear behaviors that need human review to determine if they are bugs or intended._
74
-
75
- - [ ] [Question 1 -- e.g., Is the empty state message on /reports intentional when no data exists?]
76
- - [ ] [Question 2 -- e.g., Should the "Save" button remain enabled when no changes have been made?]
77
- - [ ] [Question 3]
78
-
79
- ---
56
+ - [Observation 1]
57
+ - [ ] [Question 1 — unclear behaviour that needs a human to decide whether it is a bug]
80
58
 
81
59
  ## Areas Not Tested
82
60
 
83
- _Parts of the charter or scope that were not covered, with reasons._
84
-
85
61
  | Area | Reason |
86
- |------|--------|
87
- | [Area 1] | [e.g., Blocked by authentication issue] |
88
- | [Area 2] | [e.g., Time box expired before reaching this area] |
89
- | [Area 3] | [e.g., Feature not yet deployed to test environment] |
90
-
91
- ---
62
+ |---|---|
63
+ | [Area 1] | [e.g. blocked by authentication issue] |
64
+ | [Area 2] | [e.g. time box expired before reaching this area] |
92
65
 
93
66
  ## Recommendations
94
67
 
95
- _What should be tested next, based on findings from this session._
96
-
97
- - [ ] [Recommendation 1 -- e.g., Deep-dive into payment flow edge cases]
98
- - [ ] [Recommendation 2 -- e.g., Regression test the login form after fix for BUG-001]
99
- - [ ] [Recommendation 3 -- e.g., Cross-browser testing on Safari for the dashboard]
68
+ - [ ] [Recommendation 1 — e.g. deep-dive into payment-flow edge cases]
69
+ - [ ] [Recommendation 2 — e.g. regression-test the login form after the fix for BUG-001]
100
70
 
101
- ---
71
+ ## Reflection
102
72
 
103
- ## Session Quality Metrics
73
+ 1. **Biggest risk found:** [one sentence]
74
+ 2. **What I would tell the CEO:** [one sentence]
75
+ 3. **What I did not test that still worries me:** [one sentence]
76
+ 4. **Most useful heuristic / least useful:** [one line]
77
+ 5. **Next session should focus on:** [one line]
104
78
 
105
- _How time was spent during this session._
79
+ ## Session Stats
106
80
 
107
- | Activity | % of Time | Notes |
108
- |---------------------|-----------|--------------------------------|
109
- | Exploration | 0% | [Navigating and testing] |
110
- | Bug Investigation | 0% | [Reproducing and documenting] |
111
- | Setup | 0% | [Environment, auth, tooling] |
112
- | **Total** | **100%** | |
81
+ | Metric | Value |
82
+ |---|---|
83
+ | Duration | [N] min |
84
+ | Phases completed | [N]/8 |
85
+ | Bugs found | [N] (Critical: [N], High: [N], Medium: [N], Low: [N]) |
86
+ | Pages explored | [N] |
87
+ | Screenshots taken | [N] |
88
+ | Console errors found | [N] |
89
+ | Data integrity checks | [passed]/[total] |