qualiow-exploratory-testing 1.2.0 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +92 -0
- package/data/domains/_default.md +181 -0
- package/data/domains/ecommerce.md +192 -0
- package/data/domains/fintech.md +200 -0
- package/data/domains/identity.md +88 -0
- package/data/domains/identity.yml +138 -0
- package/data/domains/marketing.md +194 -0
- package/data/domains/saas.md +197 -0
- package/data/knowledge/changelog.yml +86 -0
- package/data/knowledge/learned-patterns.md +172 -86
- package/data/knowledge/manifest.yml +211 -6
- package/data/knowledge/releases/v0.2.0/entries/technique-contract-narrowing.yml +150 -0
- package/data/knowledge/releases/v0.2.0/entries/technique-test-suite-audit.yml +116 -0
- package/data/knowledge/releases/v0.2.0/release.yml +20 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-functional-diff-analysis.yml +119 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-llm-output-verification.yml +102 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-verification-mode-selection.yml +153 -0
- package/data/knowledge/releases/v0.3.0/release.yml +25 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-authenticated-api-probing.yml +137 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-environment-fingerprinting.yml +113 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-silent-failure-audit.yml +86 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-ui-api-differential.yml +97 -0
- package/data/knowledge/releases/v0.4.0/release.yml +28 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-config-surface-verification.yml +77 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-derived-value-verification.yml +81 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-expected-behaviour-specification.yml +97 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-presentation-integrity.yml +78 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-release-readiness-verification.yml +110 -0
- package/data/knowledge/releases/v0.5.0/release.yml +33 -0
- package/data/targets/_example-backend.yml +123 -0
- package/data/templates/api-probe-matrix.md +99 -0
- package/data/templates/coverage-map.md +14 -0
- package/data/templates/expected-behaviour.md +93 -0
- package/dist/cli/index.js +79 -70
- package/dist/cli/index.js.map +1 -1
- package/dist/index.d.ts +90 -0
- package/dist/index.js +58 -49
- package/dist/index.js.map +1 -1
- package/docs/BACKEND-VERIFICATION.md +277 -0
- package/package.json +5 -3
- package/skills/qa-verify-backend/SKILL.md +163 -0
- package/skills/qa-verify-backend/phases/00-setup.md +97 -0
- package/skills/qa-verify-backend/phases/01-ac-decomposition.md +58 -0
- package/skills/qa-verify-backend/phases/02-static-review.md +71 -0
- package/skills/qa-verify-backend/phases/03-live-verification.md +62 -0
- package/skills/qa-verify-backend/phases/03b-api-verification.md +157 -0
- package/skills/qa-verify-backend/phases/04-e2e-trigger.md +66 -0
- package/skills/qa-verify-backend/phases/05-reporting.md +135 -0
- package/skills/qa-verify-backend/references/api-probes.md +141 -0
- package/skills/qa-verify-backend/references/aws-readonly-probes.md +156 -0
- package/skills/qa-verify-backend/references/environment-fingerprinting.md +104 -0
- package/skills/qa-verify-backend/references/payload-verification.md +112 -0
- package/skills/qa-verify-backend/references/release-readiness.md +97 -0
- package/skills/qa-verify-backend/references/safety-rules.md +90 -0
|
@@ -0,0 +1,197 @@
|
|
|
1
|
+
# Domain: SaaS (Software as a Service)
|
|
2
|
+
|
|
3
|
+
## Priority Areas
|
|
4
|
+
|
|
5
|
+
1. **Authentication & Authorization** - Login flows, MFA, session security
|
|
6
|
+
2. **Role-Based Access Control** - Permissions enforced at every level
|
|
7
|
+
3. **Multi-Tenancy** - Strict data isolation between organizations
|
|
8
|
+
4. **Billing & Subscriptions** - Plan management, upgrades, downgrades, cancellations
|
|
9
|
+
5. **Data Integrity** - Export, import, backup, and consistency across views
|
|
10
|
+
6. **Onboarding** - First-time user experience, setup wizards, guided tours
|
|
11
|
+
7. **API Integrations** - Third-party connections, webhooks, OAuth flows
|
|
12
|
+
|
|
13
|
+
## Feature Risk Ranking
|
|
14
|
+
|
|
15
|
+
| Tier | Features | Rationale |
|
|
16
|
+
|------|-----------------------------------------------------------|------------------------------------------------------------------|
|
|
17
|
+
| P0 | Authentication, permissions/RBAC, data CRUD, billing | Security breaches, unauthorized access, data loss, or billing errors are existential risks. |
|
|
18
|
+
| P1 | Dashboard, search, data export | Core daily workflows. Broken dashboards or export = users cannot do their job. |
|
|
19
|
+
| P2 | Settings, notifications, integrations | Supporting features. Failure is annoying but does not block primary tasks. |
|
|
20
|
+
| P3 | Help center, about page, changelog | Informational. Rarely impacts user workflows or business outcomes. |
|
|
21
|
+
|
|
22
|
+
## Completeness Checklist
|
|
23
|
+
|
|
24
|
+
> Minimum features a SaaS application should have. Verify these exist and function before deep-testing.
|
|
25
|
+
|
|
26
|
+
- [ ] Role-based access control enforced (admin vs member vs viewer at minimum)
|
|
27
|
+
- [ ] Multi-tenancy isolation -- user cannot see another organization's data
|
|
28
|
+
- [ ] Data export available (CSV, JSON, or PDF for key data)
|
|
29
|
+
- [ ] Audit log present for sensitive actions (who did what, when)
|
|
30
|
+
- [ ] Session management -- timeout, concurrent session handling, logout
|
|
31
|
+
- [ ] Account deletion or deactivation flow exists
|
|
32
|
+
- [ ] Empty states for all list views (new account with no data shows guidance, not a blank page)
|
|
33
|
+
- [ ] Onboarding flow or first-use guidance for new users
|
|
34
|
+
- [ ] Billing/subscription page shows current plan, usage, and upgrade path
|
|
35
|
+
- [ ] Settings page allows profile, notification, and security configuration
|
|
36
|
+
|
|
37
|
+
## Data Integrity Checks
|
|
38
|
+
|
|
39
|
+
> After every state-changing action in SaaS, verify these.
|
|
40
|
+
|
|
41
|
+
1. **Created item appears in list** -- After creating any entity (project, record, user, etc.), it should immediately appear in the relevant list/table view with correct data.
|
|
42
|
+
2. **Edited fields persist after reload** -- Edit an item, save, reload the page. Every changed field should retain its new value. Check all field types (text, dates, dropdowns, toggles).
|
|
43
|
+
3. **Deleted items do not appear** -- After deletion, the item should vanish from all list views, search results, dashboards, and reports. Check that direct URL access to the deleted item returns a proper 404 or "not found" message.
|
|
44
|
+
4. **Data isolated between tenants/users** -- Switch to a different user account or organization. The created/edited/deleted item should NOT be visible. Verify in list views, search, API responses, and exports.
|
|
45
|
+
5. **Aggregations update** -- Dashboard counts, totals, and charts should reflect the change immediately (or after a documented refresh interval).
|
|
46
|
+
6. **Audit trail records the action** -- If an audit log exists, verify the create/edit/delete action is logged with correct timestamp, user, and details.
|
|
47
|
+
|
|
48
|
+
## Cross-Feature Journeys
|
|
49
|
+
|
|
50
|
+
> End-to-end flows that cross feature boundaries. Bugs often hide at the seams.
|
|
51
|
+
|
|
52
|
+
### Journey 1: Full Data Lifecycle
|
|
53
|
+
`Register -> Complete Onboarding -> Create Item -> Edit Item -> Delete Item -> Verify Empty State`
|
|
54
|
+
- Verify: onboarding completes and does not reappear, created item is visible everywhere, edits persist, deletion is clean with no orphaned references, empty state is shown.
|
|
55
|
+
|
|
56
|
+
### Journey 2: Role-Based Access Verification
|
|
57
|
+
`Login as Admin -> Change User Role to Viewer -> Logout -> Login as Viewer -> Verify Read-Only Access -> Logout -> Login as Admin -> Restore Role`
|
|
58
|
+
- Verify: role change takes effect immediately, viewer cannot create/edit/delete, UI hides write controls AND API rejects write attempts, role restoration works cleanly.
|
|
59
|
+
|
|
60
|
+
### Journey 3: Billing and Feature Gating
|
|
61
|
+
`Login -> View Current Plan -> Upgrade Plan -> Verify New Features Unlocked -> Downgrade Plan -> Verify Features Locked`
|
|
62
|
+
- Verify: plan change reflects in billing page, new features are immediately accessible after upgrade, downgrade restricts access without data loss.
|
|
63
|
+
|
|
64
|
+
## Must-Test Patterns
|
|
65
|
+
|
|
66
|
+
### Authentication
|
|
67
|
+
- Email and password login
|
|
68
|
+
- Social login (Google, GitHub, SSO providers)
|
|
69
|
+
- Multi-factor authentication setup and verification
|
|
70
|
+
- Magic link login flow
|
|
71
|
+
- Password reset with email verification
|
|
72
|
+
- Session timeout and re-authentication prompts
|
|
73
|
+
- Concurrent session limits (same user, multiple devices)
|
|
74
|
+
- Account lockout after failed attempts
|
|
75
|
+
- SSO/SAML integration for enterprise accounts
|
|
76
|
+
- Token refresh and expiration handling
|
|
77
|
+
|
|
78
|
+
### Authorization & Roles
|
|
79
|
+
- Admin can access all features
|
|
80
|
+
- Member restricted from admin-only pages
|
|
81
|
+
- Viewer cannot modify data (read-only enforcement)
|
|
82
|
+
- Custom role creation and assignment
|
|
83
|
+
- Permission changes take effect immediately (no stale cache)
|
|
84
|
+
- API endpoints enforce same permissions as UI
|
|
85
|
+
- Shared links respect permission boundaries
|
|
86
|
+
- Invitation flow assigns correct role
|
|
87
|
+
- Role downgrade removes access to restricted features
|
|
88
|
+
- Owner transfer flow
|
|
89
|
+
|
|
90
|
+
### Multi-Tenancy
|
|
91
|
+
- User A cannot see User B's data (different organizations)
|
|
92
|
+
- Switching between organizations loads correct data
|
|
93
|
+
- Search results scoped to current organization only
|
|
94
|
+
- API responses never leak cross-tenant data
|
|
95
|
+
- Shared resources (templates, integrations) scoped correctly
|
|
96
|
+
- Organization deletion removes all associated data
|
|
97
|
+
- Subdomain or workspace slug routing to correct tenant
|
|
98
|
+
- Invitation to wrong organization prevented
|
|
99
|
+
- Audit logs scoped to correct organization
|
|
100
|
+
|
|
101
|
+
### Dashboards & Data Views
|
|
102
|
+
- Dashboard loads with correct data for the current user
|
|
103
|
+
- Widgets display accurate numbers (totals, averages, counts)
|
|
104
|
+
- Date range filters update all widgets consistently
|
|
105
|
+
- Real-time data updates (if applicable) without full reload
|
|
106
|
+
- Empty states for new accounts with no data
|
|
107
|
+
- Large dataset performance (1000+ records in tables)
|
|
108
|
+
- Sort, filter, and pagination on data tables
|
|
109
|
+
- Export matches what is displayed on screen
|
|
110
|
+
- Chart tooltips show correct values
|
|
111
|
+
- Dashboard customization persists across sessions
|
|
112
|
+
|
|
113
|
+
### Data Export & Import
|
|
114
|
+
- CSV export contains all visible columns
|
|
115
|
+
- Export with filters applied exports only filtered data
|
|
116
|
+
- Large export does not time out (background job with download link)
|
|
117
|
+
- CSV import with valid data
|
|
118
|
+
- Import with missing required fields (error handling)
|
|
119
|
+
- Import with duplicate records (merge or skip strategy)
|
|
120
|
+
- Import encoding issues (UTF-8, special characters)
|
|
121
|
+
- PDF report generation accuracy
|
|
122
|
+
- API data export matches UI data
|
|
123
|
+
|
|
124
|
+
### Settings & Configuration
|
|
125
|
+
- Organization settings saved and applied
|
|
126
|
+
- User profile settings (name, avatar, timezone, language)
|
|
127
|
+
- Notification preferences (email, in-app, push)
|
|
128
|
+
- Integration settings (connect, disconnect, reconfigure)
|
|
129
|
+
- Billing settings accessible only to billing admins
|
|
130
|
+
- Danger zone actions (delete account, reset data) require confirmation
|
|
131
|
+
- Settings changes propagate to all active sessions
|
|
132
|
+
|
|
133
|
+
### Billing & Subscriptions
|
|
134
|
+
- Plan selection and upgrade flow
|
|
135
|
+
- Downgrade with feature restriction warnings
|
|
136
|
+
- Trial expiration behavior (grace period, feature lockout)
|
|
137
|
+
- Payment method add, update, remove
|
|
138
|
+
- Invoice history and download
|
|
139
|
+
- Proration calculation on mid-cycle plan change
|
|
140
|
+
- Cancellation flow with retention offer
|
|
141
|
+
- Reactivation after cancellation
|
|
142
|
+
- Failed payment retry and dunning notifications
|
|
143
|
+
- Seat-based pricing (add/remove seats updates billing)
|
|
144
|
+
- Annual vs monthly billing toggle
|
|
145
|
+
|
|
146
|
+
### API & Integrations
|
|
147
|
+
- OAuth connection flow (connect, authorize, callback)
|
|
148
|
+
- API key generation and revocation
|
|
149
|
+
- Webhook delivery and retry on failure
|
|
150
|
+
- Rate limiting behavior (correct error codes and headers)
|
|
151
|
+
- API pagination consistency
|
|
152
|
+
- API error responses follow documented format
|
|
153
|
+
- Third-party integration data sync accuracy
|
|
154
|
+
- Disconnecting integration cleans up related data
|
|
155
|
+
- API versioning (deprecated endpoints still work or return clear error)
|
|
156
|
+
|
|
157
|
+
### Onboarding
|
|
158
|
+
- First login guided tour or setup wizard
|
|
159
|
+
- Skip onboarding option
|
|
160
|
+
- Onboarding checklist progress tracking
|
|
161
|
+
- Sample data population for new accounts
|
|
162
|
+
- Invite team members during onboarding
|
|
163
|
+
- Onboarding completion state persists
|
|
164
|
+
- Re-trigger onboarding from settings
|
|
165
|
+
|
|
166
|
+
### Notification System
|
|
167
|
+
- Email notifications delivered for configured events
|
|
168
|
+
- In-app notification bell shows unread count
|
|
169
|
+
- Mark as read and mark all as read
|
|
170
|
+
- Notification links navigate to correct resource
|
|
171
|
+
- Notification preferences respected (no unwanted emails)
|
|
172
|
+
- Real-time notifications (WebSocket or polling)
|
|
173
|
+
- Notification for mentions and assignments
|
|
174
|
+
- Digest email frequency settings
|
|
175
|
+
|
|
176
|
+
## Common Bugs
|
|
177
|
+
|
|
178
|
+
- **Permission escalation** - User discovers admin endpoint via URL manipulation, API does not check permissions on update/delete, role change cached and stale
|
|
179
|
+
- **Data leakage between tenants** - Search index not scoped, autocomplete suggests other tenant's data, export includes cross-tenant records, error messages reveal other tenant info
|
|
180
|
+
- **Broken role-based access** - New feature added without permission check, UI hides button but API allows action, permission matrix incomplete for custom roles
|
|
181
|
+
- **Billing edge cases** - Double charge on retry, downgrade does not reduce invoice, cancelled account still charged, proration math incorrect
|
|
182
|
+
- **Session management** - Token not invalidated on password change, session persists after role removal, refresh token reuse after revocation
|
|
183
|
+
- **Onboarding state bugs** - Wizard reappears after completion, skip does not properly set all defaults, onboarding blocks access to needed features
|
|
184
|
+
- **Data consistency** - Dashboard totals do not match detail view, export numbers differ from UI, cached aggregations stale after data change
|
|
185
|
+
- **Integration failures** - OAuth token expired silently, webhook payload format changed, rate limit not communicated to user, disconnected integration still sends data
|
|
186
|
+
- **Multi-device conflicts** - Same record edited on two devices, last write wins without warning, real-time sync delays cause confusion
|
|
187
|
+
|
|
188
|
+
## Compliance Requirements
|
|
189
|
+
|
|
190
|
+
- **SOC 2 Type II** - Access controls, audit logging, data encryption at rest and in transit
|
|
191
|
+
- **GDPR** - Data portability (export), right to erasure, consent management, DPA availability
|
|
192
|
+
- **CCPA** - California consumer data rights, opt-out of data sale
|
|
193
|
+
- **HIPAA** (if health data) - BAA availability, PHI encryption, access logging
|
|
194
|
+
- **ISO 27001** - Information security management practices
|
|
195
|
+
- **Accessibility (WCAG 2.1 AA)** - All core workflows usable by keyboard and screen reader
|
|
196
|
+
- **Audit Trail** - All data modifications logged with timestamp, user, and action
|
|
197
|
+
- **Data Residency** - Data stored in region matching customer requirements
|
|
@@ -2,6 +2,92 @@
|
|
|
2
2
|
# Tracks what changed in each release so /qa-knowledge-list can show history.
|
|
3
3
|
|
|
4
4
|
releases:
|
|
5
|
+
- version: "0.5.0"
|
|
6
|
+
date: "2026-09-03"
|
|
7
|
+
summary: "Payload correctness and release-level verification — what to do with the answer once you have it, and how to report a set of tickets rather than one."
|
|
8
|
+
entries_added:
|
|
9
|
+
- id: technique-derived-value-verification
|
|
10
|
+
type: technique
|
|
11
|
+
name: "Derived Value Verification - Recomputing the Number and Naming What It Does Not Prove"
|
|
12
|
+
description: "Recompute every percentage, total, ratio and aggregate from the raw figures in the same response, using the formula from the spec rather than the code. Choose cases that stress sign, zero, scale and cardinality, add the structural invariants, then state explicitly that internal consistency is not correctness and name the independent oracle that would close the gap."
|
|
13
|
+
- id: technique-presentation-integrity
|
|
14
|
+
type: technique
|
|
15
|
+
name: "Presentation Integrity - Does the Screen Show What the Service Sent?"
|
|
16
|
+
description: "A correct payload can still reach the user as a wrong number. Formatting helpers that guess what a value is, units applied twice, rounding across a threshold, truncation shown as a total. Invisible from either surface alone, and routinely attributed to the wrong change."
|
|
17
|
+
- id: technique-expected-behaviour-specification
|
|
18
|
+
type: technique
|
|
19
|
+
name: "Expected Behaviour Specification - Writing the Spec That Should Have Existed"
|
|
20
|
+
description: "Observed against expected, per area rather than per case, with the reject-or-clamp and error-or-empty decisions made explicit, ranked by what users can reach today, and closed with a plain-English reply for whoever decides to fund the fix."
|
|
21
|
+
- id: technique-config-surface-verification
|
|
22
|
+
type: technique
|
|
23
|
+
name: "Config & Feature-Flag Surface Verification"
|
|
24
|
+
description: "When the configuration mechanism is itself the change: values matching the deployed config rather than the source default, nothing leaking alongside them, the runtime genuinely reading through the surface rather than around it, and the route existing in every environment the change will be promoted to."
|
|
25
|
+
- id: technique-release-readiness-verification
|
|
26
|
+
type: technique
|
|
27
|
+
name: "Release Readiness Verification - Many Tickets, One Build"
|
|
28
|
+
description: "Establish what is deployed before testing anything, give every ticket a result from a fixed vocabulary that keeps not-testable and not-tested visible, mark code-verified-only as UNVERIFIABLE rather than green, separate carry-overs, and end with a disposition and the condition that would reverse it."
|
|
29
|
+
entries_modified: []
|
|
30
|
+
entries_removed: []
|
|
31
|
+
origin: "Sessions that verified calculated values, multi-ticket releases and a feature-flag surface exposed the gaps: derived values signed off from a single payload, screens signed off without the payload beside them, findings with no acceptance criterion to file them against, and release reports where a code reading looked the same colour as an observation."
|
|
32
|
+
|
|
33
|
+
- version: "0.4.0"
|
|
34
|
+
date: "2026-09-03"
|
|
35
|
+
summary: "The HTTP/API verification lane — how to probe a service's own endpoints with real credentials, and how to know the environment is running the code you are judging."
|
|
36
|
+
entries_added:
|
|
37
|
+
- id: technique-environment-fingerprinting
|
|
38
|
+
type: technique
|
|
39
|
+
name: "Environment Fingerprinting - Proving What Is Actually Running"
|
|
40
|
+
description: "Before the first probe: which build is deployed in every component of the path, whether the changed code path is SELECTED here, and whether the commit under test is an ancestor of what is running. Same build plus different behaviour means configuration, not deploy lag. Adds NOT-REACHABLE as the honest verdict when the change is not live in the environment being measured."
|
|
41
|
+
- id: technique-authenticated-api-probing
|
|
42
|
+
type: technique
|
|
43
|
+
name: "Authenticated API Probing - Calling the Endpoint Behind the Screen"
|
|
44
|
+
description: "Get a real authenticated request context without handling a token by running fetch inside the already-logged-in page, then drive a written case matrix through it. Ten case families, how to read each signal, and the evidence and safety rules that keep credentials and response bodies off disk."
|
|
45
|
+
- id: technique-ui-api-differential
|
|
46
|
+
type: technique
|
|
47
|
+
name: "UI-vs-API Differential - Running the Same Matrix at Both Surfaces"
|
|
48
|
+
description: "Sort every finding into reproducible-at-both, API-only, UI-only and neither. Corrects the most common false PASS in exploratory testing: a client-side guard recorded as the endpoint's behaviour."
|
|
49
|
+
- id: technique-silent-failure-audit
|
|
50
|
+
type: technique
|
|
51
|
+
name: "Silent Failure Audit - Failures That Render as Ordinary Results"
|
|
52
|
+
description: "Seven shapes of failure that look like a legitimate empty or zero answer, how to force each one, and why they recur as a class rather than as individual bugs."
|
|
53
|
+
entries_modified: []
|
|
54
|
+
entries_removed: []
|
|
55
|
+
origin: "Paired UI and API sessions on the same features showed three gaps: the endpoint was never called directly, so client-side guards were being recorded as endpoint behaviour; environments were compared without checking which implementation each one ran; and the same silent-failure defect kept being filed as unrelated one-off bugs."
|
|
56
|
+
|
|
57
|
+
- version: "0.3.0"
|
|
58
|
+
date: "2026-08-21"
|
|
59
|
+
summary: "The BE/API verification layer — how to actually observe a backend, API or LLM change, and how to say so when you cannot."
|
|
60
|
+
entries_added:
|
|
61
|
+
- id: technique-verification-mode-selection
|
|
62
|
+
type: technique
|
|
63
|
+
name: "Verification Mode Selection - Choosing How to Observe a Backend Change"
|
|
64
|
+
description: "Route each AC to the observation channel that can falsify it: API-behind-the-screen, direct request to a no-UI endpoint, LLM output judgement, or needs-a-human. Makes UNVERIFIABLE a first-class verdict instead of a PASS inferred from source."
|
|
65
|
+
- id: technique-functional-diff-analysis
|
|
66
|
+
type: technique
|
|
67
|
+
name: "Functional Diff Analysis - Reading a Backend Change for Behaviour, Not Style"
|
|
68
|
+
description: "Eight passes over a BE/API/LLM diff that produce falsifiable hypotheses rather than code comments: spec drift, value tracing to the service boundary, failure paths, identity propagation, producer/consumer contract, scope, and the tests that shipped with it."
|
|
69
|
+
- id: technique-llm-output-verification
|
|
70
|
+
type: technique
|
|
71
|
+
name: "LLM & Agent Output Verification - Running It and Judging the Result"
|
|
72
|
+
description: "Fixed input set, written rubric, before-and-after on the same inputs, N runs to expose variance, assertions on properties and tool trajectory rather than on generated text. Includes what not to claim from a passing run."
|
|
73
|
+
entries_modified: []
|
|
74
|
+
entries_removed: []
|
|
75
|
+
origin: "Built after two backend ticket verifications showed the gap: deep judgement and real test execution existed for the UI, but backend, API and LLM changes were being signed off by reading the diff."
|
|
76
|
+
|
|
77
|
+
- version: "0.2.0"
|
|
78
|
+
date: "2026-08-20"
|
|
79
|
+
summary: "Backend AC-verification techniques derived from real /qa-verify-backend sessions — both aimed at changes that fail silently by construction."
|
|
80
|
+
entries_added:
|
|
81
|
+
- id: technique-contract-narrowing
|
|
82
|
+
type: technique
|
|
83
|
+
name: "Contract Narrowing - Verifying a Swapped Data Source"
|
|
84
|
+
description: "When a full-record read is replaced by a projection, the request's field-selection list becomes the payload specification. Trace every consumed field to a selection token; unrequested fields vanish with no error, no DLQ, and a green suite."
|
|
85
|
+
- id: technique-test-suite-audit
|
|
86
|
+
type: technique
|
|
87
|
+
name: "Auditing the Branch's Own Tests"
|
|
88
|
+
description: "When a change rewrites the tests that are meant to prove it works, those tests become part of the change. Check that each assertion still falsifies what its name claims, and that artefacts named by an AC actually exist."
|
|
89
|
+
origin: "A read-path swap on an event-processing service — a full-record read replaced by a projection read. The contract-narrowing audit found a dropped webhook field that 175 green unit tests could not see; the test audit found an e2e negative scenario that could only pass when the system was broken."
|
|
90
|
+
|
|
5
91
|
- version: "0.1.0"
|
|
6
92
|
date: "2026-03-28"
|
|
7
93
|
summary: "Initial knowledge base — core heuristics, techniques, checklists, and references for exploratory testing."
|
|
@@ -4,7 +4,59 @@ _Seeded from community exploratory testing sessions. Add your own via /qa-explor
|
|
|
4
4
|
|
|
5
5
|
## False Positive Patterns (skip these)
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
Findings that looked real on a first read and did not survive tracing. Check the
|
|
8
|
+
disqualifier before filing any of these.
|
|
9
|
+
|
|
10
|
+
- **"Errors are swallowed instead of retried"** (backend queue handlers): before filing, read the
|
|
11
|
+
wrapper that invokes the handler. A `return false` and a thrown exception often converge on the
|
|
12
|
+
same partial-batch-failure path, so retry and DLQ semantics are identical. *Disqualifier: the
|
|
13
|
+
wrapper's catch block reports the same unprocessed records the `false` path does.*
|
|
14
|
+
- **"Alerting severity regressed from fatal to error"**: check the logger's metric config first.
|
|
15
|
+
If `pushOnError` and `pushOnFatal` are both true, nothing changes operationally. *Disqualifier:
|
|
16
|
+
both flags set.*
|
|
17
|
+
- **"This cleanup/deletion script is dangerous"**: read it to the end before calling it a safety
|
|
18
|
+
defect. Dry-run defaults, cross-checks against another source of truth, and fail-closed error
|
|
19
|
+
handling are common and often present. *Disqualifier: it dry-runs by default and aborts rather
|
|
20
|
+
than proceeding when its safety check fails.* Scope creep may still be a valid, separate finding.
|
|
21
|
+
- **"The branch's tests fail"** — rule out your own harness first. A mis-resolved workspace
|
|
22
|
+
dependency, stale build, or wrong Node version produces errors that look exactly like product
|
|
23
|
+
defects. *Disqualifier: it passes once the harness is corrected.* File nothing until you have
|
|
24
|
+
re-run clean.
|
|
25
|
+
- **"A value never arrives / is never written"**: trace it to the persistence or serialization
|
|
26
|
+
layer before claiming absence. *Disqualifier: some earlier step already stamps it.*
|
|
27
|
+
- **"No evidence record exists, so the work was not done"**: absence of an evidence record is not
|
|
28
|
+
evidence of absence. *Disqualifier: measuring the data shows it is already in the desired state.*
|
|
29
|
+
- **"The result count changed, so it is a regression"**: counts drift with the data and differ per
|
|
30
|
+
environment. *Disqualifier: the two numbers come from different environments, or from runs days
|
|
31
|
+
apart on a collection that changes.* Compare status codes and response shapes instead.
|
|
32
|
+
- **"It behaves differently in staging, so the deploy is broken"**: check the configuration before
|
|
33
|
+
the deployment. *Disqualifier: both environments report the same build, and a flag or config
|
|
34
|
+
value selects a different code path in each.* That is still a finding — just a different one.
|
|
35
|
+
- **"It only reproduces after editing client-side storage / typing a URL the product never
|
|
36
|
+
surfaces"**: no user vector, so not a defect on its own. *Disqualifier: no affordance anywhere in
|
|
37
|
+
the product reaches that state.* **Then go looking for one** — see the triage patterns below,
|
|
38
|
+
because this disqualifier is wrong more often than any other on this list.
|
|
39
|
+
- **"The behaviour is inconsistent across surfaces"** where the requirement was never written:
|
|
40
|
+
route it to product as a requirements gap and a negative-space observation, not as a bug.
|
|
41
|
+
*Disqualifier: no acceptance criterion, spec or design covers it, and both behaviours are
|
|
42
|
+
defensible.*
|
|
43
|
+
|
|
44
|
+
## Triage and Re-Verification Patterns
|
|
45
|
+
|
|
46
|
+
- **"Closed: no legitimate user vector" is a hypothesis to falsify, not a fact.** When a bug is
|
|
47
|
+
closed because "users cannot reach this", the next session on that surface must actively hunt
|
|
48
|
+
for vectors the triage did not consider — alternative menu entries, help and onboarding
|
|
49
|
+
affordances, query parameters, deep links, other clients. A defect closed on this rationale and
|
|
50
|
+
left broken for a release is a recurring, expensive pattern.
|
|
51
|
+
- **"Partial fix" is the most common re-test outcome.** A fix typically addresses the reported
|
|
52
|
+
symptom, not the defect class. When re-verifying anything marked fixed, test the class: the same
|
|
53
|
+
code path with a different trigger, and structurally equivalent surfaces built from the same
|
|
54
|
+
component or pattern. Re-running only the original repro will pass and prove very little.
|
|
55
|
+
- **Re-verify against a build, not against a date.** Record the build id or commit of every
|
|
56
|
+
component when the bug was found and again when it is re-tested. A re-run against the same build
|
|
57
|
+
cannot surface a fix, and "it was merged" is not "it is deployed".
|
|
58
|
+
- **A fix behind a flag that is off is not a fix.** Re-testing in the one environment where the
|
|
59
|
+
flag is on says nothing about the environments users are in. Say both halves.
|
|
8
60
|
|
|
9
61
|
## Missed Bug Patterns (always check these)
|
|
10
62
|
|
|
@@ -30,91 +82,6 @@ _None yet — patterns added here will be skipped in future sessions._
|
|
|
30
82
|
- Price calculations: test with discounts, tax, multiple quantities, $0 totals
|
|
31
83
|
- Test ALL user personas if multiple exist — each may have different bugs
|
|
32
84
|
|
|
33
|
-
### Checkout Interaction Patterns (from production monitoring + 100+ historical bugs)
|
|
34
|
-
|
|
35
|
-
**Session & Basket State:**
|
|
36
|
-
- **Modal dismiss + navigate back**: Reserve seat → dismiss login modal (X) → navigate back. Check: "failed to delete basket" console errors? Seats released?
|
|
37
|
-
- **Multiple tabs**: Open checkout in 2 tabs → ticket/site branding mismatches, theme changes, session conflicts - **Session timer off-focus**: Minimize browser → timer may slow down or stop counting
|
|
38
|
-
- **Session expiry modal**: Let timer expire → "Cancel Order" button should NOT navigate to 404
|
|
39
|
-
- **Session TTL**: Orders may be deleted before user completes if session TTL is too short
|
|
40
|
-
- **Basket refetch on focus**: After switching tabs and coming back, basket view may not update
|
|
41
|
-
|
|
42
|
-
**Payment Patterns:**
|
|
43
|
-
- **Duplicate payment callbacks**: PayPal or 3DS can trigger multiple callbacks → duplicate payments
|
|
44
|
-
- **3DS with no paymentMethodId**: 3DS callback called without payment method → fails
|
|
45
|
-
- **3DS retry after failure**: Cancel 3DS → retry → check "payment method not found" errors
|
|
46
|
-
- **Apple Pay failures**: Indefinite loading when Apple Pay fails
|
|
47
|
-
- **Google Pay**: May not work on certain environments
|
|
48
|
-
- **Zero value basket**: Free tickets → billing should show "Confirm your order" and actually work
|
|
49
|
-
- **Voucher + retry**: Selecting voucher on retry page may incorrectly show PayPal button
|
|
50
|
-
- **Payment form issues**: Amex CVV placeholder may be wrong
|
|
51
|
-
- **Payment options accessibility**: Payment selector may not be keyboard accessible
|
|
52
|
-
|
|
53
|
-
**Navigation & Back Button:**
|
|
54
|
-
- **Back button Safari/Firefox**: Browser back may fail to trigger seat release
|
|
55
|
-
- **Back from payment**: May cause endless loop
|
|
56
|
-
- **Back from contacts**: Never loads seatmap, gets stuck
|
|
57
|
-
- **Back from calendar**: May redirect to homepage or 404
|
|
58
|
-
- **Auto scroll persists**: Page scroll position carries over from previous page
|
|
59
|
-
|
|
60
|
-
**Login & Auth:**
|
|
61
|
-
- **Page refresh after login**: Form loses user details, profile icon shows logged-out, CPS 400 errors
|
|
62
|
-
- **New accounts**: May get stuck in endless loop during checkout
|
|
63
|
-
- **Required login flow**: Can trigger "Unexpected Application Error!" on contacts page intermittently
|
|
64
|
-
- **Guest → login transition**: On voucher flow, refresh after login breaks the form
|
|
65
|
-
|
|
66
|
-
**Delivery & Contacts:**
|
|
67
|
-
- **Skip contacts page**: Reaching payment without delivery method selection → "delivery method ID missing" errors
|
|
68
|
-
- **Customer country data**: May create customer record with wrong country
|
|
69
|
-
- **Delivery method misalignment**: Methods may not match what's expected
|
|
70
|
-
|
|
71
|
-
**Upsells & Extras:**
|
|
72
|
-
- **Upsell removal race condition**: If basket refetches while upsell removal is in-flight, removed upsell reappears and can't be removed
|
|
73
|
-
- **Lounge toggle quantity**: Toggle should add packages × ticket count, not just 1
|
|
74
|
-
- **Unavailable upsells**: Should notify user, not silently fail
|
|
75
|
-
- **Slow lounge addition**: Large ticket count → upsell reservation times out
|
|
76
|
-
- **Upsell ordering**: Should match CMS configuration, not random
|
|
77
|
-
- **Last upsell removal**: Removing the last non-ticket upsell may delete the entire AV session
|
|
78
|
-
|
|
79
|
-
**Ticket Protection:**
|
|
80
|
-
- **TP links wrong region**: UK shows US terms, US shows UK terms
|
|
81
|
-
- **TP purchased multiple times**: Should only be purchasable once
|
|
82
|
-
- **TP skip/redirect**: May skip TP page due to timer expiry logic
|
|
83
|
-
- **TP endless loading**: Can get stuck between TP and payment
|
|
84
|
-
- **TP error handling**: TP service errors should let user continue to payment
|
|
85
|
-
- **TP blocking checkout**: Basket update during TP should block progression until complete
|
|
86
|
-
|
|
87
|
-
**Fees & Pricing:**
|
|
88
|
-
- **Fee copy when no fees**: Shows fee text even when there are no fees
|
|
89
|
-
- **Fees display incorrect**: Basket summary may show wrong fee amounts
|
|
90
|
-
- **Decimal price rejection**: Some tickets rejected because basket API doesn't accept decimal totals
|
|
91
|
-
- **Currency symbol wrong region**: £ symbol showing in US totals
|
|
92
|
-
- **Upsell prices misaligned**: Price formatting inconsistent
|
|
93
|
-
|
|
94
|
-
**Themes & Branding:**
|
|
95
|
-
- **Theme changes post-seatmap**: Theme may reset after navigating back
|
|
96
|
-
- **Theme loading delay**: Visible flash of wrong theme
|
|
97
|
-
- **US domains redirecting to UK**: Links in footer, confirmation, error pages go to UK
|
|
98
|
-
- **Favicons**: Should show new ATG Entertainment logo
|
|
99
|
-
|
|
100
|
-
**Confirmation & Post-Purchase:**
|
|
101
|
-
- **Confirmation page guard**: Should NOT be accessible without completing payment
|
|
102
|
-
- **Confirmation header not clickable**: Users get locked in
|
|
103
|
-
- **Continue to checkout button on confirmation**: Should not appear
|
|
104
|
-
- **Order confirmation CTA**: Should go to correct region MyAccount
|
|
105
|
-
|
|
106
|
-
**Copy & Legal:**
|
|
107
|
-
- **Basket expired modal copy**: Should be correct for all shows/themes
|
|
108
|
-
- **Payment retry phone numbers**: Must be correct per US region
|
|
109
|
-
- **Legal links with www**: Should not have www prefix
|
|
110
|
-
- **Error page phone numbers**: Must be region-appropriate
|
|
111
|
-
- **Voucher copy**: Gift card terminology for US
|
|
112
|
-
|
|
113
|
-
**Security:**
|
|
114
|
-
- **OWASP A01 Broken Access Control**: Basket summary had access control issue
|
|
115
|
-
- **XSS in calendar service**: ReDoS vulnerability
|
|
116
|
-
- **Comp ticket exposure**: Internal pricing visible to regular users
|
|
117
|
-
|
|
118
85
|
### Fintech
|
|
119
86
|
- Session IDs should never appear in URLs (security risk)
|
|
120
87
|
- Admin pages should require authentication
|
|
@@ -123,3 +90,122 @@ _None yet — patterns added here will be skipped in future sessions._
|
|
|
123
90
|
- Verify transactions appear in BOTH sender and receiver account histories
|
|
124
91
|
- Check for MFA/2FA on sensitive operations
|
|
125
92
|
- Loan approval should enforce documented requirements (down payment thresholds, terms disclosure)
|
|
93
|
+
|
|
94
|
+
### Backend / API / event-driven (no UI surface)
|
|
95
|
+
|
|
96
|
+
- **A swapped read path is the whole risk.** "Replace X with Y" tickets on a read path are rarely
|
|
97
|
+
mechanical, however small the diff. See `technique-contract-narrowing`.
|
|
98
|
+
- **`SELECT *` -> projection drops fields silently.** Trace every field the downstream mapper reads
|
|
99
|
+
to the selection token that requests it. The gap table written in the ticket is a claim to
|
|
100
|
+
falsify, not a specification to trust.
|
|
101
|
+
- **Assert on the serialized payload, not the object.** `JSON.stringify` omits `undefined` keys, so
|
|
102
|
+
a mapper that always assigns the key still publishes a payload without it.
|
|
103
|
+
- **"Structurally identical" needs a differential test.** One record, both paths, diff the output.
|
|
104
|
+
If the branch does not contain that test, the AC is unverified no matter how green the suite is.
|
|
105
|
+
- **Optional-everywhere schemas hide missing data.** Zod `.optional()` / TS optional means the
|
|
106
|
+
response parses fine with the field gone. Tests asserting "parses" and "does not throw" cannot
|
|
107
|
+
falsify equivalence.
|
|
108
|
+
- **Check what an AC names before scoring it.** ACs cite Cucumber scenarios, suites, or dashboards
|
|
109
|
+
that sometimes do not exist. That is an AC defect to report, not something to reinterpret.
|
|
110
|
+
- **Producers cannot observe downstream filtering.** A test claiming to verify routing must read
|
|
111
|
+
the router or the consumer (rule metrics, target state, DLQ depth) — never the producer's logs.
|
|
112
|
+
- **Log-scraping oracles hide a LOG_LEVEL dependency.** If the asserted line is `logger.debug`, the
|
|
113
|
+
suite silently requires debug logging, and "broken flow" is indistinguishable from "quiet logs".
|
|
114
|
+
- **Read the infra value, do not infer it.** A committed `.tf` value is not proof of a deployed
|
|
115
|
+
attribute. Without account access the verdict is `BLOCKED` — write the probe commands out ready
|
|
116
|
+
to run rather than inferring `PASS` from source.
|
|
117
|
+
- **Scan the diff for what the ticket did not ask for.** Lockfile regenerations, CI jobs, steering
|
|
118
|
+
docs, and validation tweaks ride along and consume the review attention the real change needed.
|
|
119
|
+
- **Grep committed files for real identifiers.** UIDs, customer emails, and staff contacts turn up
|
|
120
|
+
in incident notes and fixtures. In any system handling personal data that is a compliance
|
|
121
|
+
finding, not just untidiness.
|
|
122
|
+
- **Ask whether anyone consumes the field.** A dropped field nobody reads is a documentation fix,
|
|
123
|
+
not a code fix. Say so explicitly — it is usually answerable only by the consuming team, and
|
|
124
|
+
naming that unknown is more useful than assuming severity.
|
|
125
|
+
|
|
126
|
+
### HTTP endpoints — what a browser can never show you
|
|
127
|
+
|
|
128
|
+
- **Fingerprint before you probe.** Which build is deployed in every component of the path, and
|
|
129
|
+
whether the changed path is even selected here. Same build plus different behaviour means
|
|
130
|
+
configuration, not deploy lag. See `technique-environment-fingerprinting`.
|
|
131
|
+
- **A client-side guard is not the endpoint's contract.** "The button is disabled until you type"
|
|
132
|
+
describes the browser. Call the endpoint the way the guard is preventing before recording a
|
|
133
|
+
`PASS` — an empty body, a filter with no query, a pagination value the control cannot produce.
|
|
134
|
+
These are the cases that reach every non-browser client. See `technique-ui-api-differential`.
|
|
135
|
+
- **`200` is not a pass.** A login page returns `200`. So does an error body, a zero-row page with a
|
|
136
|
+
non-zero count, and a silently rejected filter value. Read the body.
|
|
137
|
+
- **Group the crash family.** Nine inputs that `500` on the same unescaped character are one bug
|
|
138
|
+
with nine examples. Find the shared character before writing anything up.
|
|
139
|
+
- **Metacharacters are ordinary when the data contains them.** If real values carry brackets,
|
|
140
|
+
colons or asterisks, copying a value out of a result and pasting it back in is a normal user
|
|
141
|
+
action — and the fastest route from an "edge case" to a customer report.
|
|
142
|
+
- **Probe the four kinds of nothing separately.** Empty string, `null`, whitespace-only and the
|
|
143
|
+
field absent are four different requests, and they routinely behave four different ways. One of
|
|
144
|
+
them is often a match-all that returns the entire collection.
|
|
145
|
+
- **Check the cap is code, not documentation.** A `limit` above the documented maximum that is
|
|
146
|
+
honoured anyway is both an availability risk and a bulk-extraction path.
|
|
147
|
+
- **An invalid enum answered with `200` and an empty list will be read as data.** Expect a `4xx`
|
|
148
|
+
with an error body; silent rejection is a finding. See `technique-silent-failure-audit`.
|
|
149
|
+
- **Repeat one read as a second identity and in a second scope.** A read that ignores tenant,
|
|
150
|
+
market or ownership is a data-exposure finding no UI session will surface.
|
|
151
|
+
- **Compare shapes across environments, never counts.** Different data, drifting under you.
|
|
152
|
+
- **Run the request from inside the authenticated page.** `fetch` with the session's own cookies
|
|
153
|
+
needs no token plumbing, survives SSO and MFA, and keeps the credential in the browser where it
|
|
154
|
+
belongs. See `technique-authenticated-api-probing`.
|
|
155
|
+
|
|
156
|
+
### Calculated values and what reaches the screen
|
|
157
|
+
|
|
158
|
+
- **Recompute every derived value from the raw figures in the same response**, using the formula
|
|
159
|
+
from the spec — not from the code under test. Recomputing the implementation's own arithmetic
|
|
160
|
+
proves only that arithmetic is deterministic.
|
|
161
|
+
- **Then say what that does not prove.** Both sides came from one payload, so you verified the
|
|
162
|
+
derivation and not the inputs. Name the independent oracle that would close it and whether you
|
|
163
|
+
ran it. See `technique-derived-value-verification`.
|
|
164
|
+
- **Pick a negative, a zero, a very large and a very small value, and a single-item case.** Sign
|
|
165
|
+
handling is where derivation breaks; a single-item aggregate is the cleanest comparison against
|
|
166
|
+
any second system, with no aggregation ambiguity.
|
|
167
|
+
- **Add the structural invariants** — a total equal to the sum of its parts, one metric bounded by
|
|
168
|
+
another, a derived cost positive. Cheap, and they generalise past the cases you picked.
|
|
169
|
+
- **Hold the raw response next to the screen.** A correct payload still reaches the user wrong when
|
|
170
|
+
a formatter guesses what a value is (`value > 1 ? value : value * 100`), a unit is applied twice,
|
|
171
|
+
rounding crosses a threshold, or a truncated figure is presented as a total.
|
|
172
|
+
- **A self-contradicting tile is the tell** — a percentage printed above the fraction it came from,
|
|
173
|
+
and the two disagree. One of them is a lie.
|
|
174
|
+
- **Attribute a rendering defect to the change that owns it.** It is usually not the backend ticket
|
|
175
|
+
that surfaced it, and it is frequently deployed on one environment and not another. Say plainly
|
|
176
|
+
when the change under test is not at fault. See `technique-presentation-integrity`.
|
|
177
|
+
- **Consistent is not causal.** A locale, currency or format setting whose current value happens to
|
|
178
|
+
match the output proves nothing until you change the setting and watch the output follow.
|
|
179
|
+
- **Measure timing in the page, cold and warm.** Wrap the request in `performance.now()` and record
|
|
180
|
+
the first, second and third call. A cold path an order of magnitude slower is the number every
|
|
181
|
+
user gets first, and no averaged dashboard shows it.
|
|
182
|
+
|
|
183
|
+
### Reporting a set of tickets
|
|
184
|
+
|
|
185
|
+
- **Establish what is deployed before testing anything.** A ticket whose backend is not in the
|
|
186
|
+
build is *not testable here* — testing its UI produces a convincing, meaningless result that
|
|
187
|
+
looks exactly like a data bug and gets filed as one.
|
|
188
|
+
- **Keep "not tested" visible.** It disappears between "passed" and "failed" unless the vocabulary
|
|
189
|
+
has a slot for it, and so does "not testable here".
|
|
190
|
+
- **Mark a code reading differently from an observation.** 🔍 code-verified-only is `UNVERIFIABLE`,
|
|
191
|
+
not green.
|
|
192
|
+
- **Carry-overs get their own section.** Pre-existing defects mixed into a release's results
|
|
193
|
+
inflate the apparent risk of shipping it and bury the findings that belong to it.
|
|
194
|
+
- **A deploy landing mid-session invalidates everything measured before it.** Re-fingerprint, keep
|
|
195
|
+
the pre-deploy findings in their own file, and never merge the two into one table.
|
|
196
|
+
- **State the tolerance on any baseline comparison.** A few per cent on a large live collection is
|
|
197
|
+
churn. And when a system-wide figure and a per-user figure disagree, name which one describes the
|
|
198
|
+
behaviour the change actually altered.
|
|
199
|
+
- **Close with a disposition and its reversal condition**, plus the scope of what you checked when
|
|
200
|
+
it is narrower than the question being asked. See `technique-release-readiness-verification`.
|
|
201
|
+
|
|
202
|
+
### Findings with no acceptance criterion
|
|
203
|
+
|
|
204
|
+
- **Write the spec instead of filing ten bugs.** Group by cause, put observed against expected, and
|
|
205
|
+
make the decisions explicit: reject or clamp (reject — silent correction breaks callers that
|
|
206
|
+
trust the value back); an error or an empty result (an error — a silent zero is read as data);
|
|
207
|
+
where validation lives (the layer covering every implementation, never only the client).
|
|
208
|
+
- **Rank by what users can reach today**, not by how bad each finding reads. Name which one is live
|
|
209
|
+
in production and which only exists in an environment nobody ships from.
|
|
210
|
+
- **Finish with a plain-English reply.** The section that gets the work scheduled is the one the
|
|
211
|
+
person funding the fix can read. See `technique-expected-behaviour-specification`.
|