qualiow-exploratory-testing 1.2.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +22 -0
- package/.env.example +27 -0
- package/CHANGELOG.md +194 -0
- package/README.md +428 -179
- package/agents/qa-gather-agent.md +24 -0
- package/{scripts → bin}/doctor-mobile.sh +57 -18
- package/bin/mcli +15 -9
- package/bin/mobile-cli.mjs +574 -286
- package/{scripts → bin}/setup-mobile.sh +56 -19
- package/bin/wadb +10 -4
- package/bin/wk-ios +29 -17
- package/bin/wkeval.mjs +7 -1
- package/data/domains/_default.yml +1 -1
- package/data/domains/identity.yml +138 -0
- package/data/knowledge/changelog.yml +104 -0
- package/data/knowledge/learned-patterns.md +172 -86
- package/data/knowledge/manifest.yml +481 -70
- package/data/knowledge/releases/v0.1.0/entries/technique-business-logic-race-conditions.yml +1 -1
- package/data/knowledge/releases/v0.1.0/release.yml +8 -1
- package/data/knowledge/releases/v0.2.0/entries/technique-contract-narrowing.yml +150 -0
- package/data/knowledge/releases/v0.2.0/entries/technique-test-suite-audit.yml +116 -0
- package/data/knowledge/releases/v0.2.0/release.yml +20 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-functional-diff-analysis.yml +119 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-llm-output-verification.yml +102 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-verification-mode-selection.yml +153 -0
- package/data/knowledge/releases/v0.3.0/release.yml +25 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-authenticated-api-probing.yml +137 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-environment-fingerprinting.yml +113 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-silent-failure-audit.yml +86 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-ui-api-differential.yml +97 -0
- package/data/knowledge/releases/v0.4.0/release.yml +28 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-config-surface-verification.yml +77 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-derived-value-verification.yml +81 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-expected-behaviour-specification.yml +97 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-presentation-integrity.yml +78 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-release-readiness-verification.yml +110 -0
- package/data/knowledge/releases/v0.5.0/release.yml +33 -0
- package/data/knowledge/releases/v0.6.0/entries/technique-async-callback-contracts.yml +98 -0
- package/data/knowledge/releases/v0.6.0/entries/technique-exactly-once-verification.yml +116 -0
- package/data/knowledge/releases/v0.6.0/release.yml +30 -0
- package/data/security/SECURITY-POLICY.md +273 -0
- package/data/targets/_default.yml +1 -1
- package/data/targets/_example-api-only.yml +80 -0
- package/data/targets/_example-backend.yml +123 -0
- package/data/targets/_example-mobile-emulation.yml +2 -2
- package/data/targets/_example-native-mobile.yml +2 -2
- package/data/targets/_example-sim-android-chrome.yml +1 -1
- package/data/targets/_example-sim-ios-safari.yml +1 -1
- package/data/templates/api-probe-matrix.md +99 -0
- package/data/templates/bug-report.md +29 -151
- package/data/templates/coverage-map.md +23 -6
- package/data/templates/expected-behaviour.md +94 -0
- package/data/templates/session-charter.md +3 -0
- package/data/templates/session-report.md +57 -80
- package/dist/cli/index.js +1500 -739
- package/dist/index.d.ts +673 -75
- package/dist/index.js +1664 -270
- package/docs/BACKEND-VERIFICATION.md +307 -0
- package/docs/GETTING-STARTED.md +188 -128
- package/docs/MOBILE-SETUP.md +149 -18
- package/package.json +32 -16
- package/skills/qa-explore/SKILL.md +31 -25
- package/skills/qa-explore/phases/00-setup.md +67 -41
- package/skills/qa-explore/phases/01-auth.md +32 -15
- package/skills/qa-explore/phases/02-charter.md +16 -5
- package/skills/qa-explore/phases/03-discovery.md +50 -26
- package/skills/qa-explore/phases/04-journeys.md +2 -2
- package/skills/qa-explore/phases/05-features.md +39 -24
- package/skills/qa-explore/phases/06-edge-cases.md +24 -13
- package/skills/qa-explore/phases/07-reporting.md +79 -140
- package/skills/qa-explore/references/context-integration.md +1 -1
- package/skills/qa-explore/references/output-contract.md +187 -0
- package/skills/qa-explore/references/paths.md +84 -0
- package/skills/qa-explore/references/playwright-agents-integration.md +61 -84
- package/skills/qa-explore/references/security-rules.md +66 -25
- package/skills/qa-explore/references/severity-guide.md +31 -4
- package/skills/qa-explore-cleanup/SKILL.md +17 -6
- package/skills/qa-explore-feedback/SKILL.md +18 -5
- package/skills/qa-explore-mobile/SKILL.md +86 -64
- package/skills/qa-explore-mobile/phases/00-setup.md +71 -62
- package/skills/qa-explore-mobile/phases/01-auth.md +28 -31
- package/skills/qa-explore-mobile/phases/02-charter.md +8 -4
- package/skills/qa-explore-mobile/phases/03-discovery.md +17 -11
- package/skills/qa-explore-mobile/phases/04-journeys.md +6 -6
- package/skills/qa-explore-mobile/phases/05-features.md +9 -9
- package/skills/qa-explore-mobile/phases/06-edge-cases.md +35 -40
- package/skills/qa-explore-mobile/phases/07-reporting.md +82 -56
- package/skills/qa-explore-mobile/references/mobile-edges.md +26 -26
- package/skills/qa-explore-quick/SKILL.md +60 -25
- package/skills/qa-explore-report/SKILL.md +40 -34
- package/skills/qa-gather/SKILL.md +17 -2
- package/skills/qa-knowledge-add/SKILL.md +20 -9
- package/skills/qa-knowledge-list/SKILL.md +12 -3
- package/skills/qa-target-setup/SKILL.md +63 -41
- package/skills/qa-verify-backend/SKILL.md +172 -0
- package/skills/qa-verify-backend/phases/00-setup.md +142 -0
- package/skills/qa-verify-backend/phases/01-ac-decomposition.md +58 -0
- package/skills/qa-verify-backend/phases/02-static-review.md +71 -0
- package/skills/qa-verify-backend/phases/03-live-verification.md +62 -0
- package/skills/qa-verify-backend/phases/03b-api-verification.md +170 -0
- package/skills/qa-verify-backend/phases/04-e2e-trigger.md +82 -0
- package/skills/qa-verify-backend/phases/05-reporting.md +163 -0
- package/skills/qa-verify-backend/references/api-probes.md +168 -0
- package/skills/qa-verify-backend/references/aws-readonly-probes.md +156 -0
- package/skills/qa-verify-backend/references/environment-fingerprinting.md +104 -0
- package/skills/qa-verify-backend/references/payload-verification.md +112 -0
- package/skills/qa-verify-backend/references/release-readiness.md +97 -0
- package/skills/qa-verify-backend/references/safety-rules.md +95 -0
- package/dist/cli/index.d.ts +0 -1
- package/dist/cli/index.js.map +0 -1
- package/dist/index.js.map +0 -1
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# Template target — WEB-MODE mobile app in the REAL Android Emulator Chrome.
|
|
2
2
|
# Copy this file, rename it, and point web.base_url at your mobile web app.
|
|
3
3
|
# For NATIVE mode (an installed app under test), see _example-native-mobile.yml.
|
|
4
|
-
id:
|
|
4
|
+
id: _example-sim-android-chrome
|
|
5
5
|
name: "Example Mobile Web App — Android Emulator Chrome"
|
|
6
6
|
platform: android
|
|
7
7
|
domain: _default
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# Template target — WEB-MODE mobile app in the REAL iOS Simulator Safari.
|
|
2
2
|
# Copy this file, rename it, and point web.base_url at your mobile web app.
|
|
3
3
|
# For NATIVE mode (an installed app under test), see _example-native-mobile.yml.
|
|
4
|
-
id:
|
|
4
|
+
id: _example-sim-ios-safari
|
|
5
5
|
name: "Example Mobile Web App — iOS Simulator Safari"
|
|
6
6
|
platform: ios
|
|
7
7
|
domain: _default
|
|
@@ -0,0 +1,99 @@
|
|
|
1
|
+
> CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
|
|
2
|
+
> and application details. Do not share outside your organization without review.
|
|
3
|
+
|
|
4
|
+
# API Probe Matrix
|
|
5
|
+
|
|
6
|
+
## Session Reference
|
|
7
|
+
|
|
8
|
+
| Property | Value |
|
|
9
|
+
|-----------------|-----------------------------------------|
|
|
10
|
+
| Session ID | [e.g., SESSION-2026-03-28-001] |
|
|
11
|
+
| Ticket | [ticket id, or "none"] |
|
|
12
|
+
| Endpoint | [`POST /api/v1/…` — note the verb] |
|
|
13
|
+
| Date | [YYYY-MM-DD] |
|
|
14
|
+
| Observation mode| [in-page fetch / out-of-browser request / unauthenticated] |
|
|
15
|
+
| Identity | [role or persona used — never a credential] |
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## Environment Fingerprint
|
|
20
|
+
|
|
21
|
+
Fill this in **before** the matrix. Without it the results below are uninterpretable.
|
|
22
|
+
|
|
23
|
+
| Component | Environment A `[name]` | Environment B `[name]` |
|
|
24
|
+
|------------------|------------------------|------------------------|
|
|
25
|
+
| Frontend build | [build id / date] | [build id / date] |
|
|
26
|
+
| API build | [build id / commit] | [build id / commit] |
|
|
27
|
+
| Service under test | [build id / commit] | [build id / commit] |
|
|
28
|
+
| Changed path selected? | [yes / no — name the flag or config that decides it] | [yes / no] |
|
|
29
|
+
| Commit under test deployed? | [ancestor / not an ancestor] | [ancestor / not an ancestor] |
|
|
30
|
+
|
|
31
|
+
> **Same build, different behaviour ⇒ configuration, not deploy lag.**
|
|
32
|
+
> If the changed path is not selected in an environment, every AC that depends on it is
|
|
33
|
+
> `NOT-REACHABLE` there — not `PASS`, not `FAIL`.
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## Cases
|
|
38
|
+
|
|
39
|
+
One row per case. The **label** is the stable identifier the report, the bug and the
|
|
40
|
+
re-run all refer to. Record passes as well as failures — a matrix of only failures
|
|
41
|
+
cannot show a regression later.
|
|
42
|
+
|
|
43
|
+
| # | Label | Request | Expected | Env A: status / count / rows | Env B: status / count / rows | Verdict |
|
|
44
|
+
|---|-------|---------|----------|------------------------------|------------------------------|---------|
|
|
45
|
+
| 1 | identity | `{"query":"<known value>"}` | 200, stable non-zero | 200 / 77 / 20 | 200 / 91 / 20 | ✅ |
|
|
46
|
+
| 2 | empty string | `{"query":""}` | 400 + error body | | | |
|
|
47
|
+
| 3 | body absent | `{}` | 400 + error body | | | |
|
|
48
|
+
| 4 | metachar — lone `(` | `{"query":"("}` | 400 or escaped 200 | | | |
|
|
49
|
+
| 5 | limit above cap | `?limit=<cap+1>` | 400, or capped | | | |
|
|
50
|
+
| 6 | invalid enum | `{"filter":["<nonsense>"]}` | 400 + error body | | | |
|
|
51
|
+
| 7 | second identity | same request, other user | scoped result | | | |
|
|
52
|
+
|
|
53
|
+
> **Compare status codes and shapes. Never compare absolute counts** — different
|
|
54
|
+
> environments hold different data and it drifts between runs.
|
|
55
|
+
|
|
56
|
+
### Case families to cover
|
|
57
|
+
|
|
58
|
+
Boundary lengths · tokenisation and whitespace · metacharacters of the underlying query
|
|
59
|
+
language · empty / null / absent / whitespace-only · enum values valid, invalid, empty,
|
|
60
|
+
wildcard, injected · pagination bounds and types · type confusion · casing and
|
|
61
|
+
normalisation · second identity and second scope.
|
|
62
|
+
|
|
63
|
+
---
|
|
64
|
+
|
|
65
|
+
## Findings
|
|
66
|
+
|
|
67
|
+
One row per finding, not one per failing case — a family of inputs failing on the same
|
|
68
|
+
character is **one** bug with several examples.
|
|
69
|
+
|
|
70
|
+
| ID | Cases | Finding | Surface | Severity | Bug report |
|
|
71
|
+
|----|-------|---------|---------|----------|------------|
|
|
72
|
+
| F1 | 4, 9–14 | [what is wrong, in one line] | API / UI / both | High | `bugs/BUG-0xx-….md` |
|
|
73
|
+
|
|
74
|
+
**Surface** classifies where the defect lives, from the differential pass:
|
|
75
|
+
|
|
76
|
+
| Surface | Meaning | Owner |
|
|
77
|
+
|---------|---------|-------|
|
|
78
|
+
| **Both** | The UI faithfully reflects the endpoint | Service |
|
|
79
|
+
| **API only** | Real defect the client's guard is hiding — reachable by every other client | Service |
|
|
80
|
+
| **UI only** | The client invents or masks behaviour the service does not have | Client |
|
|
81
|
+
| **Neither** | Environment or access noise, not a product defect | — |
|
|
82
|
+
|
|
83
|
+
---
|
|
84
|
+
|
|
85
|
+
## Evidence
|
|
86
|
+
|
|
87
|
+
| File | Contents |
|
|
88
|
+
|------|----------|
|
|
89
|
+
| `evidence/fingerprint.md` | Build ids, flag values, commit ancestry |
|
|
90
|
+
| `evidence/<case-set>-raw.json` | Raw per-case output, redacted |
|
|
91
|
+
| `probes/<probe>.js` | The probe script, so this is repeatable |
|
|
92
|
+
|
|
93
|
+
---
|
|
94
|
+
|
|
95
|
+
## Notes and Open Questions
|
|
96
|
+
|
|
97
|
+
- [What could not be checked, and exactly what access would unblock it]
|
|
98
|
+
- [Which environment would answer a `NOT-REACHABLE` verdict]
|
|
99
|
+
- [Counts recorded here are valid for the date above only]
|
|
@@ -1,178 +1,56 @@
|
|
|
1
|
-
|
|
1
|
+
> CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
|
|
2
|
+
> and application details. Do not share outside your organization without review.
|
|
2
3
|
|
|
3
|
-
|
|
4
|
+
# BUG-NNN: [Component] fails [Condition] causing [Impact]
|
|
4
5
|
|
|
5
|
-
`[
|
|
6
|
+
> _Title example: `[Login Form] fails [when password contains special characters] causing [authentication error and user lockout]`. Number bugs from BUG-001 per session; one file per bug at `bugs/BUG-NNN.md`._
|
|
6
7
|
|
|
7
|
-
|
|
8
|
+
**Severity:** Critical | High | Medium | Low
|
|
9
|
+
**Priority:** P0 | P1 | P2 | P3
|
|
10
|
+
**Component:** [Component]
|
|
11
|
+
**URL:** [exact URL where the bug was found — mobile: screen name or deep link — backend: endpoint or resource]
|
|
12
|
+
**Environment:** [e.g. Playwright CLI, Chromium, 1280x720, logged in as X — mobile: mode / platform / device / OS — backend: env kind + build]
|
|
13
|
+
**Reproduction rate:** Always | Intermittent (~X%) | Once
|
|
8
14
|
|
|
9
|
-
|
|
15
|
+
> _Severity per `severity-guide.md`: Critical = data loss, security breach, financial loss, system unusable; High = major feature broken, no reasonable workaround; Medium = partial break, workaround exists, or an important missing feature; Low = cosmetic, rare edge case. **When in doubt, go one level LOWER.** Priority = severity adjusted by workaround and blast radius (P0 hours · P1 this sprint · P2 next sprint · P3 backlog)._
|
|
10
16
|
|
|
11
|
-
##
|
|
17
|
+
## Summary
|
|
12
18
|
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
- [ ] **Critical**
|
|
16
|
-
- [ ] **High**
|
|
17
|
-
- [ ] **Medium**
|
|
18
|
-
- [ ] **Low**
|
|
19
|
-
|
|
20
|
-
### Severity Level Definitions
|
|
21
|
-
|
|
22
|
-
| Level | Definition | Examples |
|
|
23
|
-
|----------|----------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------|
|
|
24
|
-
| Critical | System down, data loss, security breach. Blocks all users immediately. | Payment gateway crashes on all transactions; user passwords exposed in API response; database corruption losing customer orders. |
|
|
25
|
-
| High | Major feature broken, many users affected. No reasonable workaround. | Checkout fails for all credit card payments; search returns zero results for every query; login broken on mobile devices. |
|
|
26
|
-
| Medium | Partial break, workaround exists. Some users affected under certain paths. | Discount code applies twice if user clicks fast; filter resets after pagination; profile photo upload fails on Safari only. |
|
|
27
|
-
| Low | Cosmetic issue, rare edge case. Minimal user or business impact. | Footer alignment off by 2px on wide screens; tooltip text truncated in German locale; hover state missing on disabled button. |
|
|
28
|
-
|
|
29
|
-
> **Note: When in doubt, go one level LOWER. Severity inflation destroys credibility.** A bug report rated "Critical" that turns out to be cosmetic undermines every future report you file. Reserve Critical for genuine emergencies. If you hesitate between two levels, pick the lower one and let the evidence speak for itself.
|
|
30
|
-
|
|
31
|
-
---
|
|
32
|
-
|
|
33
|
-
## URL
|
|
34
|
-
|
|
35
|
-
`[URL where the bug was found]`
|
|
36
|
-
|
|
37
|
-
## Environment
|
|
38
|
-
|
|
39
|
-
| Property | Value |
|
|
40
|
-
|-----------|---------------------------|
|
|
41
|
-
| Browser | [e.g., Chrome 125.0] |
|
|
42
|
-
| Viewport | [e.g., 1920x1080] |
|
|
43
|
-
| OS | [e.g., macOS 15.x] |
|
|
44
|
-
| Auth | [e.g., logged in as X] |
|
|
45
|
-
| Device | [e.g., Desktop / Mobile] |
|
|
46
|
-
|
|
47
|
-
---
|
|
19
|
+
[Two sentences: what breaks, and why it matters to the business.]
|
|
48
20
|
|
|
49
21
|
## Expected Behavior
|
|
50
22
|
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
> [Describe the expected outcome clearly.]
|
|
23
|
+
[What should happen according to requirements, conventions, or reasonable user expectations.]
|
|
54
24
|
|
|
55
25
|
## Actual Behavior
|
|
56
26
|
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
> [Describe the actual outcome. Be precise -- include error messages verbatim if applicable.]
|
|
60
|
-
|
|
61
|
-
---
|
|
27
|
+
[What actually happens instead. Be precise; include error messages verbatim.]
|
|
62
28
|
|
|
63
29
|
## Steps to Reproduce
|
|
64
30
|
|
|
65
31
|
1. Navigate to `[URL]`
|
|
66
32
|
2. [Action 2]
|
|
67
33
|
3. [Action 3]
|
|
68
|
-
4. [
|
|
69
|
-
5. Observe: [what goes wrong]
|
|
70
|
-
|
|
71
|
-
> _Reproduction rate: [Always / Intermittent (~X%) / Once]_
|
|
72
|
-
|
|
73
|
-
---
|
|
34
|
+
4. Observe: [what goes wrong]
|
|
74
35
|
|
|
75
36
|
## Business Impact
|
|
76
37
|
|
|
77
|
-
>
|
|
78
|
-
|
|
79
|
-
### Revenue Impact
|
|
80
|
-
|
|
81
|
-
> [Does this bug prevent purchases, cause incorrect charges, block upgrades, or reduce conversion? Quantify if possible -- e.g., "Blocks all guest checkout, which represents ~30% of orders."]
|
|
82
|
-
|
|
83
|
-
### Trust Impact
|
|
84
|
-
|
|
85
|
-
> [Would a user lose confidence in the product? Would they contact support, leave a negative review, or churn? e.g., "User sees a $0.00 balance briefly before correct amount loads -- creates panic."]
|
|
86
|
-
|
|
87
|
-
### Regulatory Risk
|
|
88
|
-
|
|
89
|
-
> [Does this violate any compliance requirements? Specify the regulation. e.g., "Credit card number visible in URL query string -- PCI DSS violation." If none, write "No known regulatory risk."]
|
|
90
|
-
>
|
|
91
|
-
> Common frameworks to consider: SOX, PCI DSS, GDPR, CCPA, HIPAA, WCAG 2.1 AA, CAN-SPAM, KYC/AML, Regulation E.
|
|
92
|
-
|
|
93
|
-
### Data Risk
|
|
94
|
-
|
|
95
|
-
> [Is data corrupted, lost, or exposed? e.g., "Order total saved as $0 in database when coupon equals cart value -- data integrity failure." If none, write "No data risk identified."]
|
|
96
|
-
|
|
97
|
-
### Scale
|
|
98
|
-
|
|
99
|
-
> [How many users are affected and under what conditions? e.g., "All users on iOS Safari 17+ (~18% of mobile traffic)" or "Only affects accounts created before the migration on 2025-01-15 (~200 accounts)."]
|
|
100
|
-
|
|
101
|
-
---
|
|
38
|
+
> _Mandatory. Answer at least one bullet; write "none identified" for the rest. Quantify where you can ("blocks all guest checkout, ~30% of orders")._
|
|
102
39
|
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|-------------|------------------|----------------------------------------------------|
|
|
109
|
-
| Likelihood | High / Med / Low | [How likely is a user to hit this? Every time? Rare edge case?] |
|
|
110
|
-
| Impact | High / Med / Low | [How bad is it when they do? Data loss? Mild annoyance?] |
|
|
111
|
-
| **Risk** | **[Result]** | **Risk = Likelihood x Impact** (see matrix below) |
|
|
112
|
-
|
|
113
|
-
### Risk Matrix
|
|
114
|
-
|
|
115
|
-
| | **Impact: Low** | **Impact: Med** | **Impact: High** |
|
|
116
|
-
|--------------------|-----------------|-----------------|-------------------|
|
|
117
|
-
| **Likelihood: High** | Medium | High | Critical |
|
|
118
|
-
| **Likelihood: Med** | Low | Medium | High |
|
|
119
|
-
| **Likelihood: Low** | Low | Low | Medium |
|
|
120
|
-
|
|
121
|
-
---
|
|
122
|
-
|
|
123
|
-
## Recommended Fix Priority
|
|
124
|
-
|
|
125
|
-
> **Priority: [P0 / P1 / P2 / P3]**
|
|
126
|
-
|
|
127
|
-
| Priority | Meaning | Typical SLA |
|
|
128
|
-
|----------|--------------------------------------|----------------------|
|
|
129
|
-
| P0 | Fix immediately, interrupt sprint | Hours |
|
|
130
|
-
| P1 | Fix this sprint | Days |
|
|
131
|
-
| P2 | Fix next sprint | 1-2 weeks |
|
|
132
|
-
| P3 | Backlog, fix when convenient | When capacity allows |
|
|
133
|
-
|
|
134
|
-
**Justification:** [Explain why this priority level is appropriate given the risk assessment and business impact above. e.g., "Risk is High (likely + revenue-impacting), but a workaround exists (users can switch payment method), so P1 rather than P0."]
|
|
135
|
-
|
|
136
|
-
---
|
|
137
|
-
|
|
138
|
-
## Impact
|
|
139
|
-
|
|
140
|
-
### User Impact
|
|
141
|
-
|
|
142
|
-
> [How does this affect the end user? Can they complete their task? What is the degraded experience?]
|
|
143
|
-
|
|
144
|
-
### Business Impact Summary
|
|
145
|
-
|
|
146
|
-
> [One-sentence summary referencing the detailed Business Impact section above. e.g., "Blocks ~30% of revenue through guest checkout with no workaround."]
|
|
147
|
-
|
|
148
|
-
---
|
|
40
|
+
- **Revenue impact:** [prevents purchases, incorrect charges, blocked upgrades, reduced conversion?]
|
|
41
|
+
- **Trust impact:** [would a user lose confidence, contact support, churn?]
|
|
42
|
+
- **Regulatory risk:** [SOX, PCI DSS, GDPR, CCPA, HIPAA, WCAG 2.1 AA, KYC/AML, Regulation E — name the regulation, or "none identified"]
|
|
43
|
+
- **Data risk:** [corrupted, lost, or exposed data?]
|
|
44
|
+
- **Scale:** [how many users, under what conditions]
|
|
149
45
|
|
|
150
46
|
## Evidence
|
|
151
47
|
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
```
|
|
159
|
-
[Paste any relevant console errors here]
|
|
160
|
-
```
|
|
48
|
+
- Screenshot: `screenshots/BUG-NNN.png`
|
|
49
|
+
- Video: `videos/BUG-NNN.webm` (if recorded; `.mp4` on mobile)
|
|
50
|
+
- Log: `logs/BUG-NNN.log` (mobile) / `evidence/<file>` (backend), or "none"
|
|
51
|
+
- Console errors: [verbatim, redacted, or "none"]
|
|
52
|
+
- Network failures: [method, URL, status, response excerpt — redacted — or "none"]
|
|
161
53
|
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
```
|
|
165
|
-
[Paste any failed requests -- method, URL, status code, response body excerpt]
|
|
166
|
-
```
|
|
167
|
-
|
|
168
|
-
### Additional Logs
|
|
169
|
-
|
|
170
|
-
```
|
|
171
|
-
[Any other diagnostic output -- application logs, state dumps, etc.]
|
|
172
|
-
```
|
|
173
|
-
|
|
174
|
-
---
|
|
175
|
-
|
|
176
|
-
## Additional Context
|
|
54
|
+
## Recommended Fix Priority
|
|
177
55
|
|
|
178
|
-
|
|
56
|
+
[Why this priority relative to the other findings: risk (likelihood × impact, see `severity-guide.md`), workaround, blast radius, and anything related — recent deploys, feature flags, user reports.]
|
|
@@ -1,10 +1,13 @@
|
|
|
1
|
+
> CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
|
|
2
|
+
> and application details. Do not share outside your organization without review.
|
|
3
|
+
|
|
1
4
|
# Coverage Map
|
|
2
5
|
|
|
3
6
|
## Session Reference
|
|
4
7
|
|
|
5
8
|
| Property | Value |
|
|
6
9
|
|------------|--------------------------------|
|
|
7
|
-
| Session ID | [e.g
|
|
10
|
+
| Session ID | [session directory, e.g. 2026-09-08-1420-explore-parabank] |
|
|
8
11
|
| Target | [application or feature name] |
|
|
9
12
|
| Date | [YYYY-MM-DD] |
|
|
10
13
|
|
|
@@ -12,17 +15,31 @@
|
|
|
12
15
|
|
|
13
16
|
## Overall Coverage
|
|
14
17
|
|
|
15
|
-
| Area |
|
|
16
|
-
|
|
17
|
-
| [Area 1] |
|
|
18
|
-
| [Area 2] |
|
|
19
|
-
| [Area 3] |
|
|
18
|
+
| Area | Risk | Status | Bugs | Notes |
|
|
19
|
+
|---|---|---|---|---|
|
|
20
|
+
| [Area 1] | P0 | tested | 0 | [heuristics applied, notes] |
|
|
21
|
+
| [Area 2] | P1 | partial | 0 | [what was not reached and why] |
|
|
22
|
+
| [Area 3] | P2 | not-tested | — | [reason] |
|
|
20
23
|
|
|
21
24
|
### Status Legend
|
|
22
25
|
|
|
23
26
|
- **tested** -- Area was fully explored with the planned heuristics.
|
|
24
27
|
- **partial** -- Area was visited but not all heuristics or paths were exercised.
|
|
25
28
|
- **not-tested** -- Area was in scope but was not reached during the session.
|
|
29
|
+
- **code-verified-only** -- Believed correct from reading the source, never observed
|
|
30
|
+
running. This is `UNVERIFIABLE`, **not** a pass -- keep it visually distinct so it
|
|
31
|
+
cannot be skimmed as green.
|
|
32
|
+
|
|
33
|
+
Two distinctions that routinely produce a false **tested**:
|
|
34
|
+
|
|
35
|
+
- **Consistent is not causal.** A setting whose current value happens to match the output
|
|
36
|
+
proves nothing until the setting is changed and the output is watched to follow.
|
|
37
|
+
- **The data has to reach the case.** An area whose behaviour depends on values no
|
|
38
|
+
reachable record has -- a negative, a zero, an empty collection -- is **not-tested** for
|
|
39
|
+
that behaviour. Find such a record, create one, or say plainly what data would settle it.
|
|
40
|
+
|
|
41
|
+
Name the single **not-tested** item that carries the most risk. In a map full of green rows
|
|
42
|
+
that is the line the reader acts on.
|
|
26
43
|
|
|
27
44
|
---
|
|
28
45
|
|
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
> CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
|
|
2
|
+
> and application details. Do not share outside your organization without review.
|
|
3
|
+
|
|
4
|
+
# Expected Behaviour Specification
|
|
5
|
+
|
|
6
|
+
Use this when a verification session finds behaviour nobody wrote an acceptance criterion
|
|
7
|
+
for — which is most of what an API probe turns up. A bug report says *this is wrong*. This
|
|
8
|
+
says *here is what right looks like*, in enough detail that someone can implement it and
|
|
9
|
+
someone else can verify it. It is frequently the most valuable artifact a session produces,
|
|
10
|
+
because it converts a pile of findings into one reviewable decision.
|
|
11
|
+
|
|
12
|
+
Write it **per area**, not per case. Ten failing inputs that share a cause are one section.
|
|
13
|
+
|
|
14
|
+
## Header
|
|
15
|
+
|
|
16
|
+
| Property | Value |
|
|
17
|
+
|----------|-------|
|
|
18
|
+
| Date | [YYYY-MM-DD] |
|
|
19
|
+
| Endpoint / surface | [`POST /api/v1/…` — note the verb] |
|
|
20
|
+
| Environment(s) observed | [name, and the build each one was running] |
|
|
21
|
+
| Source findings | [link to the session that produced the observations] |
|
|
22
|
+
| Status | [proposed / agreed / implemented] |
|
|
23
|
+
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## [N]. [Area — e.g. Minimum input length]
|
|
27
|
+
|
|
28
|
+
**Observed**
|
|
29
|
+
|
|
30
|
+
> What the system does today, as evidence. Exact request, exact status, exact response.
|
|
31
|
+
> Note which environments it holds in — behaviour frequently differs, and that difference
|
|
32
|
+
> is part of the finding.
|
|
33
|
+
|
|
34
|
+
```
|
|
35
|
+
{"query":"L"} → 200 / <the entire collection> [alpha]
|
|
36
|
+
{"query":"L"} → 200 / 0 [beta]
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
**Expected**
|
|
40
|
+
|
|
41
|
+
> What it should do. Numbered, falsifiable statements — each one a testable assertion, not
|
|
42
|
+
> a sentiment. Include the boundary values themselves.
|
|
43
|
+
|
|
44
|
+
- An input whose tokens are all below the minimum → **`400`**, with the minimum named in
|
|
45
|
+
the message.
|
|
46
|
+
- "No usable tokens after filtering" and "no query supplied" are **different branches**.
|
|
47
|
+
Neither falls through to a match-all.
|
|
48
|
+
- The match-all path is not reachable from this parameter at any input. If "browse
|
|
49
|
+
everything" is wanted, it is an explicit endpoint with a hard server-side cap.
|
|
50
|
+
- The minimum is **N**, matching the existing rule, so that [known-good short inputs] keep
|
|
51
|
+
working.
|
|
52
|
+
- Client: no request fires below N; the user sees a hint rather than an empty result.
|
|
53
|
+
|
|
54
|
+
### Decisions this section forces
|
|
55
|
+
|
|
56
|
+
State them explicitly — leaving them implicit is how a spec gets implemented three
|
|
57
|
+
different ways:
|
|
58
|
+
|
|
59
|
+
- **Reject, or clamp?** *Reject.* Silently serving a different page size than was asked for
|
|
60
|
+
breaks every caller that trusts the value back.
|
|
61
|
+
- **`400`, or a silent empty result?** *`400`.* A silent zero is indistinguishable from a
|
|
62
|
+
genuine no-match, and it will be read as data.
|
|
63
|
+
- **Where does validation live?** At the layer that covers **every** implementation of the
|
|
64
|
+
feature — not in one of two code paths, and never only in the client.
|
|
65
|
+
- **What must not change?** List the valid inputs and their current answers, so the fix is
|
|
66
|
+
verifiable as non-breaking.
|
|
67
|
+
|
|
68
|
+
---
|
|
69
|
+
|
|
70
|
+
## Priority
|
|
71
|
+
|
|
72
|
+
Rank the areas by what is reachable by real users **today**, not by how bad each one reads.
|
|
73
|
+
Name explicitly which finding is live in production right now, and which only appears in an
|
|
74
|
+
environment nobody ships from — the two need very different urgency, and the difference is
|
|
75
|
+
invisible unless someone says it.
|
|
76
|
+
|
|
77
|
+
---
|
|
78
|
+
|
|
79
|
+
## Reply — plain English
|
|
80
|
+
|
|
81
|
+
> Ready to paste into a ticket comment, a pull request or a chat reply. Same content, no
|
|
82
|
+
> jargon, no status codes unless they carry meaning to the reader. Written for the person
|
|
83
|
+
> who has to decide whether to fund the fix.
|
|
84
|
+
|
|
85
|
+
For each area, in a short paragraph:
|
|
86
|
+
|
|
87
|
+
- **What happens**, in a sentence a non-engineer can picture.
|
|
88
|
+
- **Whether it is live for users**, and where.
|
|
89
|
+
- **What the fix is**, in outline — enough to size it, not enough to design it.
|
|
90
|
+
- **Where you agree with their reading**, if they proposed one, and precisely where you do
|
|
91
|
+
not, and why.
|
|
92
|
+
|
|
93
|
+
Close with the one thing you would put ahead of everything else in the list, and the reason
|
|
94
|
+
— usually that it is the only one users are hitting today.
|
|
@@ -1,112 +1,89 @@
|
|
|
1
|
-
|
|
1
|
+
> CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
|
|
2
|
+
> and application details. Do not share outside your organization without review.
|
|
2
3
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
| Property | Value |
|
|
6
|
-
|------------|--------------------------------|
|
|
7
|
-
| Session ID | [e.g., SESSION-2026-03-28-001] |
|
|
8
|
-
| Target | [application or feature name] |
|
|
9
|
-
| URL | [target URL] |
|
|
10
|
-
| Date | [YYYY-MM-DD] |
|
|
11
|
-
| Duration | [e.g., 45 minutes] |
|
|
12
|
-
| Tester | Claude AI |
|
|
13
|
-
| Charter | [link to charter file] |
|
|
4
|
+
# Session Report — [target]
|
|
14
5
|
|
|
15
|
-
|
|
6
|
+
## Session Metadata
|
|
16
7
|
|
|
17
|
-
|
|
8
|
+
| Field | Value |
|
|
9
|
+
|---|---|
|
|
10
|
+
| Session ID | [session directory, e.g. 2026-09-08-1420-explore-parabank] |
|
|
11
|
+
| Kind | explore / quick / mobile / backend |
|
|
12
|
+
| Target | [target id or application name] |
|
|
13
|
+
| URL | [base URL] |
|
|
14
|
+
| Date | [YYYY-MM-DD] |
|
|
15
|
+
| Duration | [N] min |
|
|
16
|
+
| Charter | charter.md |
|
|
18
17
|
|
|
19
|
-
|
|
18
|
+
## Executive Summary
|
|
20
19
|
|
|
21
|
-
|
|
20
|
+
[Three sentences: what was tested, what was found, and the biggest risk.]
|
|
22
21
|
|
|
23
22
|
## Summary Stats
|
|
24
23
|
|
|
25
|
-
| Metric
|
|
26
|
-
|
|
27
|
-
| Pages
|
|
28
|
-
| Actions
|
|
29
|
-
| Bugs
|
|
30
|
-
| Bugs
|
|
31
|
-
| Bugs
|
|
32
|
-
| Bugs
|
|
33
|
-
| **Total
|
|
34
|
-
|
|
35
|
-
---
|
|
24
|
+
| Metric | Count |
|
|
25
|
+
|---|---|
|
|
26
|
+
| Pages explored | 0 |
|
|
27
|
+
| Actions performed | 0 |
|
|
28
|
+
| Bugs — Critical | 0 |
|
|
29
|
+
| Bugs — High | 0 |
|
|
30
|
+
| Bugs — Medium | 0 |
|
|
31
|
+
| Bugs — Low | 0 |
|
|
32
|
+
| **Total bugs** | **0** |
|
|
36
33
|
|
|
37
34
|
## Coverage Map
|
|
38
35
|
|
|
39
|
-
|
|
36
|
+
| Area | Risk | Status | Bugs | Notes |
|
|
37
|
+
|---|---|---|---|---|
|
|
38
|
+
| [Area 1] | P0 | tested | 0 | [notes] |
|
|
39
|
+
| [Area 2] | P1 | partial | 0 | [what was not reached and why] |
|
|
40
|
+
| [Area 3] | P2 | not-tested | — | [reason] |
|
|
40
41
|
|
|
41
|
-
|
|
42
|
-
|------|--------|--------------------|------------|-------|
|
|
43
|
-
| [Area 1] | tested / partial / not-tested | [list] | 0 | [notes] |
|
|
44
|
-
| [Area 2] | tested / partial / not-tested | [list] | 0 | [notes] |
|
|
45
|
-
| [Area 3] | tested / partial / not-tested | [list] | 0 | [notes] |
|
|
46
|
-
|
|
47
|
-
---
|
|
42
|
+
_Status: `tested` (fully explored with the planned heuristics) · `partial` (visited, not all heuristics or paths exercised) · `not-tested` (in scope, not reached) · `code-verified-only` (believed correct from reading source, never observed running: this is UNVERIFIABLE, not a pass). Name the single not-tested item that carries the most risk._
|
|
48
43
|
|
|
49
44
|
## Bugs Found
|
|
50
45
|
|
|
51
|
-
_Each bug links to its individual report in `output/bugs/`._
|
|
52
|
-
|
|
53
46
|
| # | ID | Title | Severity | Report |
|
|
54
|
-
|
|
55
|
-
| 1 |
|
|
47
|
+
|---|---|---|---|---|
|
|
48
|
+
| 1 | BUG-001 | [Component] fails [Condition] causing [Impact] | Critical/High/Medium/Low | bugs/BUG-001.md |
|
|
56
49
|
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
---
|
|
50
|
+
_If no bugs were found, state: "No bugs were found during this session."_
|
|
60
51
|
|
|
61
52
|
## Observations
|
|
62
53
|
|
|
63
|
-
_Things noticed
|
|
64
|
-
|
|
65
|
-
- [Observation 1 -- e.g., page load time felt slow on dashboard but did not exceed threshold]
|
|
66
|
-
- [Observation 2 -- e.g., tooltip text seems inconsistent with other similar features]
|
|
67
|
-
- [Observation 3]
|
|
54
|
+
_Things noticed that are not bugs but may be worth discussing: what's MISSING against the domain checklist, data-integrity results, questions for the team._
|
|
68
55
|
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
## Questions
|
|
72
|
-
|
|
73
|
-
_Unclear behaviors that need human review to determine if they are bugs or intended._
|
|
74
|
-
|
|
75
|
-
- [ ] [Question 1 -- e.g., Is the empty state message on /reports intentional when no data exists?]
|
|
76
|
-
- [ ] [Question 2 -- e.g., Should the "Save" button remain enabled when no changes have been made?]
|
|
77
|
-
- [ ] [Question 3]
|
|
78
|
-
|
|
79
|
-
---
|
|
56
|
+
- [Observation 1]
|
|
57
|
+
- [ ] [Question 1 — unclear behaviour that needs a human to decide whether it is a bug]
|
|
80
58
|
|
|
81
59
|
## Areas Not Tested
|
|
82
60
|
|
|
83
|
-
_Parts of the charter or scope that were not covered, with reasons._
|
|
84
|
-
|
|
85
61
|
| Area | Reason |
|
|
86
|
-
|
|
87
|
-
| [Area 1] | [e.g
|
|
88
|
-
| [Area 2] | [e.g
|
|
89
|
-
| [Area 3] | [e.g., Feature not yet deployed to test environment] |
|
|
90
|
-
|
|
91
|
-
---
|
|
62
|
+
|---|---|
|
|
63
|
+
| [Area 1] | [e.g. blocked by authentication issue] |
|
|
64
|
+
| [Area 2] | [e.g. time box expired before reaching this area] |
|
|
92
65
|
|
|
93
66
|
## Recommendations
|
|
94
67
|
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
- [ ] [Recommendation 1 -- e.g., Deep-dive into payment flow edge cases]
|
|
98
|
-
- [ ] [Recommendation 2 -- e.g., Regression test the login form after fix for BUG-001]
|
|
99
|
-
- [ ] [Recommendation 3 -- e.g., Cross-browser testing on Safari for the dashboard]
|
|
68
|
+
- [ ] [Recommendation 1 — e.g. deep-dive into payment-flow edge cases]
|
|
69
|
+
- [ ] [Recommendation 2 — e.g. regression-test the login form after the fix for BUG-001]
|
|
100
70
|
|
|
101
|
-
|
|
71
|
+
## Reflection
|
|
102
72
|
|
|
103
|
-
|
|
73
|
+
1. **Biggest risk found:** [one sentence]
|
|
74
|
+
2. **What I would tell the CEO:** [one sentence]
|
|
75
|
+
3. **What I did not test that still worries me:** [one sentence]
|
|
76
|
+
4. **Most useful heuristic / least useful:** [one line]
|
|
77
|
+
5. **Next session should focus on:** [one line]
|
|
104
78
|
|
|
105
|
-
|
|
79
|
+
## Session Stats
|
|
106
80
|
|
|
107
|
-
|
|
|
108
|
-
|
|
109
|
-
|
|
|
110
|
-
|
|
|
111
|
-
|
|
|
112
|
-
|
|
|
81
|
+
| Metric | Value |
|
|
82
|
+
|---|---|
|
|
83
|
+
| Duration | [N] min |
|
|
84
|
+
| Phases completed | [N]/8 |
|
|
85
|
+
| Bugs found | [N] (Critical: [N], High: [N], Medium: [N], Low: [N]) |
|
|
86
|
+
| Pages explored | [N] |
|
|
87
|
+
| Screenshots taken | [N] |
|
|
88
|
+
| Console errors found | [N] |
|
|
89
|
+
| Data integrity checks | [passed]/[total] |
|