qualiow-exploratory-testing 1.1.0 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +177 -0
- package/bin/mcli +16 -0
- package/bin/mobile-cli.mjs +573 -0
- package/bin/wadb +6 -0
- package/bin/wk-ios +58 -0
- package/bin/wkeval.mjs +44 -0
- package/data/domains/_default.md +181 -0
- package/data/domains/ecommerce.md +192 -0
- package/data/domains/fintech.md +200 -0
- package/data/domains/identity.md +88 -0
- package/data/domains/identity.yml +138 -0
- package/data/domains/marketing.md +194 -0
- package/data/domains/saas.md +197 -0
- package/data/knowledge/changelog.yml +86 -0
- package/data/knowledge/learned-patterns.md +172 -1
- package/data/knowledge/manifest.yml +210 -4
- package/data/knowledge/releases/v0.2.0/entries/technique-contract-narrowing.yml +150 -0
- package/data/knowledge/releases/v0.2.0/entries/technique-test-suite-audit.yml +116 -0
- package/data/knowledge/releases/v0.2.0/release.yml +20 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-functional-diff-analysis.yml +119 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-llm-output-verification.yml +102 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-verification-mode-selection.yml +153 -0
- package/data/knowledge/releases/v0.3.0/release.yml +25 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-authenticated-api-probing.yml +137 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-environment-fingerprinting.yml +113 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-silent-failure-audit.yml +86 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-ui-api-differential.yml +97 -0
- package/data/knowledge/releases/v0.4.0/release.yml +28 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-config-surface-verification.yml +77 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-derived-value-verification.yml +81 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-expected-behaviour-specification.yml +97 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-presentation-integrity.yml +78 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-release-readiness-verification.yml +110 -0
- package/data/knowledge/releases/v0.5.0/release.yml +33 -0
- package/data/targets/_example-backend.yml +123 -0
- package/data/targets/_example-mobile-emulation.yml +52 -0
- package/data/targets/_example-native-mobile.yml +61 -0
- package/data/targets/_example-sim-android-chrome.yml +81 -0
- package/data/targets/_example-sim-ios-safari.yml +80 -0
- package/data/templates/api-probe-matrix.md +99 -0
- package/data/templates/coverage-map.md +14 -0
- package/data/templates/expected-behaviour.md +93 -0
- package/dist/cli/index.js +137 -5
- package/dist/cli/index.js.map +1 -1
- package/dist/index.d.ts +350 -527
- package/dist/index.js +144 -5
- package/dist/index.js.map +1 -1
- package/docs/BACKEND-VERIFICATION.md +277 -0
- package/docs/MOBILE-SETUP.md +92 -0
- package/package.json +26 -12
- package/scripts/doctor-mobile.sh +143 -0
- package/scripts/setup-mobile.sh +213 -0
- package/skills/qa-explore/phases/00-setup.md +8 -1
- package/skills/qa-explore/phases/01-auth.md +4 -0
- package/skills/qa-explore-mobile/SKILL.md +146 -0
- package/skills/qa-explore-mobile/phases/00-setup.md +203 -0
- package/skills/qa-explore-mobile/phases/01-auth.md +127 -0
- package/skills/qa-explore-mobile/phases/02-charter.md +70 -0
- package/skills/qa-explore-mobile/phases/03-discovery.md +79 -0
- package/skills/qa-explore-mobile/phases/04-journeys.md +49 -0
- package/skills/qa-explore-mobile/phases/05-features.md +72 -0
- package/skills/qa-explore-mobile/phases/06-edge-cases.md +157 -0
- package/skills/qa-explore-mobile/phases/07-reporting.md +123 -0
- package/skills/qa-explore-mobile/references/mobile-edges.md +117 -0
- package/skills/qa-explore-quick/SKILL.md +6 -0
- package/skills/qa-verify-backend/SKILL.md +163 -0
- package/skills/qa-verify-backend/phases/00-setup.md +97 -0
- package/skills/qa-verify-backend/phases/01-ac-decomposition.md +58 -0
- package/skills/qa-verify-backend/phases/02-static-review.md +71 -0
- package/skills/qa-verify-backend/phases/03-live-verification.md +62 -0
- package/skills/qa-verify-backend/phases/03b-api-verification.md +157 -0
- package/skills/qa-verify-backend/phases/04-e2e-trigger.md +66 -0
- package/skills/qa-verify-backend/phases/05-reporting.md +135 -0
- package/skills/qa-verify-backend/references/api-probes.md +141 -0
- package/skills/qa-verify-backend/references/aws-readonly-probes.md +156 -0
- package/skills/qa-verify-backend/references/environment-fingerprinting.md +104 -0
- package/skills/qa-verify-backend/references/payload-verification.md +112 -0
- package/skills/qa-verify-backend/references/release-readiness.md +97 -0
- package/skills/qa-verify-backend/references/safety-rules.md +90 -0
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
# Phase 3: Live Verification
|
|
2
|
+
|
|
3
|
+
Skip if the live lane is BLOCKED — but still write every probe command into
|
|
4
|
+
`evidence/live-probes.md` so a human with credentials can run them unchanged.
|
|
5
|
+
|
|
6
|
+
Read `references/aws-readonly-probes.md` for the command catalogue. Read
|
|
7
|
+
`references/safety-rules.md` before the first probe, and confirm the fingerprint from
|
|
8
|
+
phase 0 step 3c is on file — a probe of an environment that does not run the change
|
|
9
|
+
measures something else.
|
|
10
|
+
|
|
11
|
+
This phase covers the **cloud resources**. The service's own HTTP endpoints are phase 3b
|
|
12
|
+
(`phases/03b-api-verification.md`); most tickets need both.
|
|
13
|
+
|
|
14
|
+
## Rules
|
|
15
|
+
|
|
16
|
+
- **Read-only only.** `describe-*`, `get-*`, `list-*`, `query`, `scan`,
|
|
17
|
+
`simulate-principal-policy`, `filter-log-events`. Nothing that mutates.
|
|
18
|
+
- **One probe per AC.** Paste the raw output into `evidence/` before interpreting it.
|
|
19
|
+
Interpretation without the raw output is an opinion.
|
|
20
|
+
- **Confirm resource names first.** Names in a target config are asserted, not proven.
|
|
21
|
+
`list-tables` / `list-functions` before you trust them. A probe against a
|
|
22
|
+
non-existent resource returns an error that reads like a failure but is a typo.
|
|
23
|
+
- **Redact before writing to disk.** Real user emails, tokens, consumer PII.
|
|
24
|
+
|
|
25
|
+
## Per-AC Probes
|
|
26
|
+
|
|
27
|
+
Work through the matrix. For each AC whose evidence source includes `live`:
|
|
28
|
+
|
|
29
|
+
1. Run the probe.
|
|
30
|
+
2. Save raw output to `evidence/<ac-id>-<resource>.json`.
|
|
31
|
+
3. State the verdict and quote the specific field that decides it.
|
|
32
|
+
|
|
33
|
+
Example write-up:
|
|
34
|
+
|
|
35
|
+
```
|
|
36
|
+
AC4 — stream view type
|
|
37
|
+
probe: aws dynamodb describe-table --table-name <config> --query 'Table.StreamSpecification'
|
|
38
|
+
raw: evidence/AC4-config-table.json
|
|
39
|
+
field: StreamViewType = "NEW_AND_OLD_IMAGES"
|
|
40
|
+
verdict: FAIL — AC requires NEW_IMAGE
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
## Probe the Failure Path, Not Just the Resource
|
|
44
|
+
|
|
45
|
+
The resource existing proves almost nothing. Add these regardless of what the ACs say:
|
|
46
|
+
|
|
47
|
+
- **Dead letter queue depth** — a non-zero `ApproximateNumberOfMessages` means events
|
|
48
|
+
are already being lost. Check it before and after the e2e trigger.
|
|
49
|
+
- **Lambda error and throttle metrics** over the life of the environment.
|
|
50
|
+
- **Event source mapping state** — `Enabled`, `LastProcessingResult`, and whether
|
|
51
|
+
partial-batch reporting is actually configured on the mapping (not just returned by
|
|
52
|
+
the handler code — the handler's return value is ignored without it).
|
|
53
|
+
- **Log level in the deployed function's environment** — a debug/trace level on a
|
|
54
|
+
component that logs whole event payloads is a data-exposure finding in any system
|
|
55
|
+
handling personal data.
|
|
56
|
+
- **Effective permissions** — `iam simulate-principal-policy` for both the actions the
|
|
57
|
+
component needs and the actions it must **not** have. Proving a denial is as
|
|
58
|
+
important as proving an allow.
|
|
59
|
+
|
|
60
|
+
## Record
|
|
61
|
+
|
|
62
|
+
Update `ac-matrix.md` verdicts as you go. Append findings to `evidence/live-review.md`.
|
|
@@ -0,0 +1,157 @@
|
|
|
1
|
+
# Phase 3b: API Verification
|
|
2
|
+
|
|
3
|
+
Phase 3 asks *"is the resource configured the way the AC says?"*. This phase asks
|
|
4
|
+
*"does the endpoint actually behave the way the AC says, for inputs the UI will never
|
|
5
|
+
send?"* — the service's own HTTP surface, between the browser and the cloud resources.
|
|
6
|
+
|
|
7
|
+
Run it whenever the ticket touches a request/response contract: a search or list
|
|
8
|
+
endpoint, validation rules, pagination, filter values, an error contract, a payload
|
|
9
|
+
shape a consumer reads. Skip it when the change has no HTTP surface at all.
|
|
10
|
+
|
|
11
|
+
Read `references/api-probes.md` for the mechanics and the case catalogue,
|
|
12
|
+
`references/environment-fingerprinting.md` before you believe any result, and
|
|
13
|
+
`references/payload-verification.md` once you have one — a well-shaped `200` says nothing
|
|
14
|
+
about whether the numbers in it are right.
|
|
15
|
+
|
|
16
|
+
## Step 1: Prove Which Implementation You Are Probing
|
|
17
|
+
|
|
18
|
+
**Do this before the first probe, and record it in `evidence/fingerprint.md`.**
|
|
19
|
+
|
|
20
|
+
A green result against an environment that does not run the changed code proves
|
|
21
|
+
nothing, and a red result there is not the ticket's bug. Establish, with evidence:
|
|
22
|
+
|
|
23
|
+
- the deployed build of every component in the request path (version endpoint, or a
|
|
24
|
+
behavioural fingerprint when there is none),
|
|
25
|
+
- whether the changed code path is **selected** here — a feature flag, a config
|
|
26
|
+
toggle, a routing rule can pick between two implementations of the same feature
|
|
27
|
+
inside one identical build,
|
|
28
|
+
- whether the commit carrying the change is actually an ancestor of what is deployed.
|
|
29
|
+
|
|
30
|
+
If the environment does not run the changed path, say so and stop this lane. The
|
|
31
|
+
verdict is `NOT-REACHABLE` for that environment (phase 5), not `PASS` and not `FAIL`.
|
|
32
|
+
|
|
33
|
+
## Step 2: Get an Authenticated Request Context
|
|
34
|
+
|
|
35
|
+
Preferred: **run the request inside the already-authenticated page**. `fetch` from the
|
|
36
|
+
page context carries the session cookie, the CSRF token, and any client interceptors —
|
|
37
|
+
the same credentials the UI has — so there is no token to obtain and nothing to store.
|
|
38
|
+
|
|
39
|
+
```bash
|
|
40
|
+
playwright-cli -s=<session> open <base_url> --headed --profile .auth/<profile> # once
|
|
41
|
+
playwright-cli -s=<session> eval "$(cat probes/<probe>.js)" --raw
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
`--headed` matters when login goes through an interactive identity provider: that round
|
|
45
|
+
cannot complete headless. `--profile` is what makes it stick — a persistent profile
|
|
46
|
+
directory outlives a `storage_state` JSON by a wide margin.
|
|
47
|
+
|
|
48
|
+
Fall back to an out-of-browser runner (a script driving `curl`) only when you need
|
|
49
|
+
volume or a machine-readable matrix. Take the credential from the authenticated profile
|
|
50
|
+
(`playwright-cli cookie-list`), keep it in a file outside the repo, and reference it by
|
|
51
|
+
path — never inline it into a script, a report, or this repo.
|
|
52
|
+
|
|
53
|
+
Record which mode produced each result. An in-page probe and a bare `curl` do not prove
|
|
54
|
+
the same thing: the first proves the endpoint behaves this way *for a real logged-in
|
|
55
|
+
session*, the second proves it behaves this way *for whoever holds that credential*.
|
|
56
|
+
|
|
57
|
+
## Step 3: Build the Case Matrix Before You Run Anything
|
|
58
|
+
|
|
59
|
+
Write the cases down first, as data, so the run is repeatable and the evidence is a
|
|
60
|
+
table rather than a scroll of output. Derive them from the ACs, then add the standard
|
|
61
|
+
families from `references/api-probes.md` — the ACs are a floor:
|
|
62
|
+
|
|
63
|
+
boundary lengths · tokenisation and whitespace · metacharacters of whatever query
|
|
64
|
+
language sits underneath · empty, null, absent and whitespace-only · enum values that
|
|
65
|
+
are valid, invalid, empty, wildcard and injected · pagination bounds and types · casing
|
|
66
|
+
and normalisation · type confusion on every field.
|
|
67
|
+
|
|
68
|
+
Each case gets a stable label. That label is what the report, the bug and the re-run all
|
|
69
|
+
refer to.
|
|
70
|
+
|
|
71
|
+
## Step 4: Run, Record Raw, Then Interpret
|
|
72
|
+
|
|
73
|
+
Capture per case: **HTTP status, the count or shape the caller would read, how many rows
|
|
74
|
+
actually came back, elapsed time, and the first line of any error body.** Save the raw
|
|
75
|
+
result to `evidence/` — `eval --filename` writes it straight to disk without paying for
|
|
76
|
+
it in context — and interpret afterwards.
|
|
77
|
+
|
|
78
|
+
Read the results for these, in this order:
|
|
79
|
+
|
|
80
|
+
1. **Status vs shape.** A `200` whose body is an error, or a `200` with `count > 0` and
|
|
81
|
+
zero rows, is worse than a `500` — the caller cannot tell it went wrong.
|
|
82
|
+
2. **The silent zero.** An invalid enum, an unknown filter value or an unparseable term
|
|
83
|
+
that returns `200` with an empty result is indistinguishable from a genuine no-match.
|
|
84
|
+
Someone will read it as data.
|
|
85
|
+
3. **Unbounded responses.** An empty or absent query that returns the whole collection;
|
|
86
|
+
a `limit` above the documented cap that is honoured anyway.
|
|
87
|
+
4. **The crash family.** Group every input that `500`s and find the one character or
|
|
88
|
+
shape they share — that is one bug, not nine.
|
|
89
|
+
5. **Inputs the UI cannot produce.** These are not reassurance. The browser guard is the
|
|
90
|
+
only thing preventing them, and mobile clients, integrations and scripts do not have
|
|
91
|
+
it. Judge them on the consequence of the call succeeding, not on how it was reached.
|
|
92
|
+
|
|
93
|
+
## Step 5: Parity Across Environments
|
|
94
|
+
|
|
95
|
+
Run the identical matrix in every environment the ticket claims to affect. Then obey the
|
|
96
|
+
rule that makes the comparison meaningful:
|
|
97
|
+
|
|
98
|
+
> **Compare status codes and shapes. Never compare absolute counts.** Different
|
|
99
|
+
> environments hold different data, and the data drifts under you.
|
|
100
|
+
|
|
101
|
+
A divergence in *shape* between two environments running the same build is a
|
|
102
|
+
configuration finding — go back to step 1 and name the switch. A divergence in *counts*
|
|
103
|
+
is usually nothing.
|
|
104
|
+
|
|
105
|
+
## Step 5b: Check the Values, Not Just the Shape
|
|
106
|
+
|
|
107
|
+
A `200` with a well-formed body has proven the contract, not the content. For every derived
|
|
108
|
+
value the ACs mention — a percentage, total, ratio, delta or aggregate — recompute it from
|
|
109
|
+
the raw figures in the same response, using the formula from the **specification** rather
|
|
110
|
+
than from the code under test. Pick cases that stress sign, zero, scale and cardinality,
|
|
111
|
+
and add the structural invariants the values must satisfy regardless of magnitude.
|
|
112
|
+
|
|
113
|
+
Then write down what the check does not prove: when both sides come from one payload you
|
|
114
|
+
have verified the derivation, not the inputs. Name the independent oracle that would close
|
|
115
|
+
it, and say whether you ran it. Full guidance in `references/payload-verification.md`.
|
|
116
|
+
|
|
117
|
+
## Step 6: The Differential Pass — the Same Matrix at Both Surfaces
|
|
118
|
+
|
|
119
|
+
If the ticket also has a UI, run the overlapping cases through the screen too, and sort
|
|
120
|
+
every finding into four buckets:
|
|
121
|
+
|
|
122
|
+
| Bucket | Meaning |
|
|
123
|
+
|--------|---------|
|
|
124
|
+
| **Both** | The UI faithfully reflects the endpoint. Fix it once, in the service. |
|
|
125
|
+
| **API only** | A real defect the UI's own guard is hiding. Reachable by every non-browser client. |
|
|
126
|
+
| **UI only** | The client invents or masks a behaviour the service does not have — an error rendered as an empty result, a value re-formatted, a field dropped. |
|
|
127
|
+
| **Neither** | Environment or access noise, not a product defect. |
|
|
128
|
+
|
|
129
|
+
Hold the raw response next to every number on screen while you do this. A correct payload
|
|
130
|
+
can still reach the user as a wrong value — a formatter that guesses what a value is, a
|
|
131
|
+
unit applied twice, rounding that crosses a threshold, truncation shown as a total. When it
|
|
132
|
+
happens, work out which change owns the corruption before filing it; it is frequently not
|
|
133
|
+
the change under test, and it is frequently deployed on one environment and not another.
|
|
134
|
+
|
|
135
|
+
Two corrections this pass reliably produces, both worth stating explicitly in the
|
|
136
|
+
report:
|
|
137
|
+
|
|
138
|
+
- **A client-side guard recorded as a pass is not a pass.** "The button is disabled
|
|
139
|
+
until you type" describes the client, not the endpoint. Call the endpoint the way the
|
|
140
|
+
guard is preventing before you write `PASS`.
|
|
141
|
+
- **A discrepancy between what the endpoint returns and what the screen shows is its own
|
|
142
|
+
defect**, independent of the endpoint's behaviour, and it belongs to the client team.
|
|
143
|
+
|
|
144
|
+
## Read-Only Discipline
|
|
145
|
+
|
|
146
|
+
The API lane calls **read** endpoints only. Reads that the ticket's own ACs describe,
|
|
147
|
+
plus the standard families above, against endpoints the target's `api.probe_allowlist`
|
|
148
|
+
declares. Anything that creates, mutates or deletes belongs to phase 4, goes through the
|
|
149
|
+
real write path, and never runs against production. See `references/safety-rules.md`.
|
|
150
|
+
|
|
151
|
+
Response bodies are real records. Redact before they touch the disk.
|
|
152
|
+
|
|
153
|
+
## Record
|
|
154
|
+
|
|
155
|
+
Update `ac-matrix.md` as you go, one row per AC, each verdict naming the environment it
|
|
156
|
+
holds in. Write the case table to `evidence/api-probe-matrix.md` using
|
|
157
|
+
`data/templates/api-probe-matrix.md`.
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# Phase 4: End-to-End Trigger
|
|
2
|
+
|
|
3
|
+
Covers the "verified in environment" style of AC: a real change through the real write
|
|
4
|
+
path, then re-probe the data layer.
|
|
5
|
+
|
|
6
|
+
Skip if `--no-e2e`, if the environment is production, or if the live lane is BLOCKED.
|
|
7
|
+
|
|
8
|
+
## Step 1: Capture the Before State
|
|
9
|
+
|
|
10
|
+
Query the target store and record what exists **before** you touch anything. Without a
|
|
11
|
+
before, "a record appeared" cannot be distinguished from "a record was already there".
|
|
12
|
+
|
|
13
|
+
Also record DLQ depth and the Lambda's error count now.
|
|
14
|
+
|
|
15
|
+
## Step 2: Trigger Through the Real Path
|
|
16
|
+
|
|
17
|
+
Prefer the path a real operator uses — the UI. Hand off to `/qa-explore` or drive
|
|
18
|
+
`playwright-cli` directly against the target's `base_url`. Make the change
|
|
19
|
+
**identifiable**: a value you can search for, containing no real data.
|
|
20
|
+
|
|
21
|
+
Note the exact wall-clock time of the save and the identity you were logged in as.
|
|
22
|
+
Both are needed to judge the resulting record.
|
|
23
|
+
|
|
24
|
+
If the UI is unavailable, call the write API directly with the same auth — but say so
|
|
25
|
+
in the report, because an API-only trigger does not prove the UI's identity
|
|
26
|
+
propagation works.
|
|
27
|
+
|
|
28
|
+
Conversely, if the UI trigger succeeds, that is not evidence the endpoint accepts what
|
|
29
|
+
other clients will send. The client's own guards are still in the path. Phase 3b covers
|
|
30
|
+
what happens without them.
|
|
31
|
+
|
|
32
|
+
## Step 3: Re-Probe and Compare
|
|
33
|
+
|
|
34
|
+
Give the asynchronous path a moment, then query again and compare against the before
|
|
35
|
+
state. Check, in this order:
|
|
36
|
+
|
|
37
|
+
1. **Count** — exactly the expected number of new records. Zero is a failure. More than
|
|
38
|
+
expected means duplication or fan-out nobody designed for.
|
|
39
|
+
2. **Attribution** — the actor equals the user you logged in as. Not a role, not a
|
|
40
|
+
placeholder, not the previous editor.
|
|
41
|
+
3. **Fidelity** — the stored snapshot matches what the source store now holds, field for
|
|
42
|
+
field. Compare the actual documents; do not eyeball two summaries. Where the record
|
|
43
|
+
carries derived values, recompute them from the raw figures beside them
|
|
44
|
+
(`references/payload-verification.md`) rather than trusting that they were computed
|
|
45
|
+
once and correctly.
|
|
46
|
+
4. **Key shape** — the sort key format matches what the consumer will query and sort by.
|
|
47
|
+
5. **Completeness** — every attribute the ACs require is present, with the required type.
|
|
48
|
+
A missing attribute here is a `FAIL` on that AC no matter how well the rest works.
|
|
49
|
+
|
|
50
|
+
## Step 4: The Adversarial Passes
|
|
51
|
+
|
|
52
|
+
The happy path passing is the beginning, not the end. Run these — they are where audit
|
|
53
|
+
and event pipelines actually break:
|
|
54
|
+
|
|
55
|
+
- **Same-tick writes.** Save twice as fast as the UI allows. Then count records. If the
|
|
56
|
+
key is time-based at coarse granularity, one write silently overwrote the other.
|
|
57
|
+
- **Delete.** Delete an entity and check who the record attributes it to.
|
|
58
|
+
- **Bulk save.** A save that writes N items — do you get N attributable records?
|
|
59
|
+
- **Second identity.** Repeat as a different user; confirm attribution follows.
|
|
60
|
+
- **Isolation.** Change one scope (market/tenant/brand) and confirm no record touched
|
|
61
|
+
another.
|
|
62
|
+
- **DLQ after.** Re-check depth. Anything new means the pipeline dropped something
|
|
63
|
+
while you watched.
|
|
64
|
+
|
|
65
|
+
Record each with its own evidence file. Any of these that fails is a bug report,
|
|
66
|
+
whether or not an AC covers it — the ACs are a floor, not a ceiling.
|
|
@@ -0,0 +1,135 @@
|
|
|
1
|
+
# Phase 5: Reporting
|
|
2
|
+
|
|
3
|
+
## Step 1: The AC Traceability Matrix
|
|
4
|
+
|
|
5
|
+
`ac-matrix.md` is the headline deliverable. One row per AC:
|
|
6
|
+
|
|
7
|
+
| AC | Claim | Verdict | Evidence | Note |
|
|
8
|
+
|----|-------|---------|----------|------|
|
|
9
|
+
|
|
10
|
+
Verdicts:
|
|
11
|
+
|
|
12
|
+
| Verdict | Means |
|
|
13
|
+
|---------|-------|
|
|
14
|
+
| **PASS** | Verified with cited evidence |
|
|
15
|
+
| **PARTIAL** | The intent is met but something material deviates from the stated spec |
|
|
16
|
+
| **FAIL** | The AC as written is not true |
|
|
17
|
+
| **BLOCKED** | Not verifiable with available access — probe command attached |
|
|
18
|
+
| **NOT-REACHABLE** | This environment does not run the changed code path — a flag, a config value or an undeployed commit. Not a pass and not a failure |
|
|
19
|
+
| **UNVERIFIABLE** | The AC has no falsifier as written — a DoR defect |
|
|
20
|
+
|
|
21
|
+
Never leave a row at `PENDING`. If you did not check it, the verdict is `BLOCKED` and
|
|
22
|
+
you say why.
|
|
23
|
+
|
|
24
|
+
**Scope every verdict to an environment**, in the row itself: `PASS (dev)` and `PASS` are
|
|
25
|
+
different claims, and only one of them is honest when the change is behind a flag that is
|
|
26
|
+
off everywhere else. Where the ticket claims several environments, give the matrix a
|
|
27
|
+
column per environment rather than one merged verdict.
|
|
28
|
+
|
|
29
|
+
Lead the matrix with the fingerprint from `evidence/fingerprint.md` — build id or commit
|
|
30
|
+
per component, and which implementation each environment selected. A reader who does not
|
|
31
|
+
know what was running cannot use the rest of the table.
|
|
32
|
+
|
|
33
|
+
## Step 2: Bug Reports
|
|
34
|
+
|
|
35
|
+
One bug per report in `bugs/`, using `data/templates/bug-report.md`. Backend findings
|
|
36
|
+
need translation — nobody funds a fix for "StreamViewType is NEW_AND_OLD_IMAGES".
|
|
37
|
+
|
|
38
|
+
Write the **Business Impact** in terms of consequence:
|
|
39
|
+
- What can go wrong for a customer, an operator, or an auditor?
|
|
40
|
+
- What does it cost to fix now versus after the dependent work ships?
|
|
41
|
+
- For audit, identity and privacy findings: what happens during an incident review or a
|
|
42
|
+
regulatory request when this gap is discovered by someone else?
|
|
43
|
+
|
|
44
|
+
Give every reproducible finding a copy-pasteable reproduction — the probe command or
|
|
45
|
+
the trigger sequence — so the developer can confirm in under a minute.
|
|
46
|
+
|
|
47
|
+
Two shapes deserve their own reports rather than a line in someone else's:
|
|
48
|
+
|
|
49
|
+
- **A fix that cannot reach users.** If the changed path is off in the environments that
|
|
50
|
+
matter, the ticket is not done, however green the dev environment is. Say what has to
|
|
51
|
+
happen for it to reach production, and that the switch carries its own risk.
|
|
52
|
+
- **A discrepancy between the endpoint and the screen.** When the API is honest and the
|
|
53
|
+
UI is not — an error rendered as an empty result, a value re-scaled, a field dropped —
|
|
54
|
+
that is a client defect with its own owner, independent of the backend verdict.
|
|
55
|
+
|
|
56
|
+
**Collapse a crash family into one report.** Nine inputs that fail on the same unescaped
|
|
57
|
+
character are one bug with nine examples, not nine bugs.
|
|
58
|
+
|
|
59
|
+
## Step 2b: Carry-Overs
|
|
60
|
+
|
|
61
|
+
Defects you found that pre-date the change under test go in their own section:
|
|
62
|
+
**pre-existing, not caused by this change, tracked separately.** Mixing them into the
|
|
63
|
+
results inflates the apparent risk of shipping this change and buries the findings that
|
|
64
|
+
belong to it.
|
|
65
|
+
|
|
66
|
+
The reverse holds too — a defect that exists only because two changes met in this build
|
|
67
|
+
belongs to the report even though neither change owns it alone. Say which two.
|
|
68
|
+
|
|
69
|
+
## Step 3: DoD and DoR Gaps
|
|
70
|
+
|
|
71
|
+
Separate from bugs, list where the ticket's own Definition of Done is not met:
|
|
72
|
+
tests missing for an AC, documentation still describing the removed design,
|
|
73
|
+
monitoring absent for a new failure mode, manual work described in a runbook but never
|
|
74
|
+
executed and never evidenced.
|
|
75
|
+
|
|
76
|
+
List DoR gaps too — untestable ACs, missing acceptance criteria for behaviour the
|
|
77
|
+
change clearly has. These improve the next ticket rather than this one.
|
|
78
|
+
|
|
79
|
+
## Step 3b: Coverage, Honestly
|
|
80
|
+
|
|
81
|
+
Where the session covered a set of ACs rather than one, keep a coverage map with four
|
|
82
|
+
states — not two:
|
|
83
|
+
|
|
84
|
+
| State | Means |
|
|
85
|
+
|-------|-------|
|
|
86
|
+
| ✅ tested and passing | Observed, with evidence |
|
|
87
|
+
| ⚠️ partially tested | Exercised, but a named part of the AC was not |
|
|
88
|
+
| ❌ not tested | In scope, never reached |
|
|
89
|
+
| 🔍 code-verified only | Read in the source and believed. **This is `UNVERIFIABLE`, not a pass** |
|
|
90
|
+
|
|
91
|
+
Two distinctions worth being pedantic about, because both routinely produce a false pass:
|
|
92
|
+
|
|
93
|
+
- **Consistent is not causal.** A setting whose current value happens to match the output
|
|
94
|
+
proves nothing until you change the setting and watch the output follow.
|
|
95
|
+
- **The data has to reach the case.** An AC about negative values cannot be verified
|
|
96
|
+
against records that are all zero. If no reachable record exercises it, that AC is
|
|
97
|
+
untested — find one, create one, or say plainly it is unproven and name what data would
|
|
98
|
+
settle it.
|
|
99
|
+
|
|
100
|
+
Then name, in one sentence, **which untested item carries the most risk**. In a report full
|
|
101
|
+
of green rows that is the line the reader acts on.
|
|
102
|
+
|
|
103
|
+
## Step 3c: Write the Spec Where None Exists
|
|
104
|
+
|
|
105
|
+
Most of what an API probe finds has no acceptance criterion behind it, so there is nothing
|
|
106
|
+
to file the finding against and it turns into an argument instead of a fix. Produce an
|
|
107
|
+
expected-behaviour specification using `data/templates/expected-behaviour.md`: observed
|
|
108
|
+
against expected, grouped by cause rather than by case, with the decisions the fix forces
|
|
109
|
+
made explicit (reject or clamp; an error or an empty result; where validation lives), and
|
|
110
|
+
ranked by what real users can reach **today** rather than by how bad each one reads.
|
|
111
|
+
|
|
112
|
+
Close it with the same content in plain English, ready to paste into a ticket comment. That
|
|
113
|
+
section is what gets the work scheduled.
|
|
114
|
+
|
|
115
|
+
## Step 4: Session Report
|
|
116
|
+
|
|
117
|
+
Use `data/templates/session-report.md`. Lead with the matrix summary
|
|
118
|
+
(`n PASS / n PARTIAL / n FAIL / n BLOCKED`), then the findings ranked by severity, then
|
|
119
|
+
what you could not check and exactly what access would unblock it.
|
|
120
|
+
|
|
121
|
+
Recommend a disposition in one line: ship, ship-with-follow-ups, or hold — and say what
|
|
122
|
+
would change your mind. Where the scope of what you checked is narrower than the question
|
|
123
|
+
being asked, say so in the same breath; a recommendation with an unstated scope is one
|
|
124
|
+
someone will over-read.
|
|
125
|
+
|
|
126
|
+
For a release-level session covering many tickets against one build, follow
|
|
127
|
+
`references/release-readiness.md` instead of this phase's per-AC shape: the deployment
|
|
128
|
+
table first, then a fixed result vocabulary per ticket that keeps *not testable here* and
|
|
129
|
+
*not tested* visible rather than letting them vanish between passed and failed.
|
|
130
|
+
|
|
131
|
+
## Step 5: Confidentiality
|
|
132
|
+
|
|
133
|
+
Every output file gets the confidentiality header. Scan the whole session directory for
|
|
134
|
+
tokens, keys, real user emails and consumer PII before finishing; replace with
|
|
135
|
+
`[REDACTED]`. Session output stays local — never transmit it anywhere.
|
|
@@ -0,0 +1,141 @@
|
|
|
1
|
+
# API Probes — Reaching the Endpoint Behind the Screen
|
|
2
|
+
|
|
3
|
+
How to call a service's own HTTP surface with real credentials, what to ask it, and how
|
|
4
|
+
to read the answer. Companion to `phases/03b-api-verification.md`.
|
|
5
|
+
|
|
6
|
+
## 1. Getting an Authenticated Request Context
|
|
7
|
+
|
|
8
|
+
### In-page `fetch` — the default
|
|
9
|
+
|
|
10
|
+
Run the request inside the page that is already logged in. It inherits the session
|
|
11
|
+
cookie, the CSRF token and every client interceptor, so there is no token to obtain, no
|
|
12
|
+
credential to store, and no impersonation to arrange.
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
# once per environment — interactive login the first time, the profile persists it
|
|
16
|
+
playwright-cli -s=alpha open https://service-alpha.example.com/ \
|
|
17
|
+
--headed --profile .auth/alpha-profile
|
|
18
|
+
|
|
19
|
+
# then, as many times as you like
|
|
20
|
+
playwright-cli -s=alpha eval "$(cat probes/search-cases.js)" --raw
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
The probe is an async function returning a string or a JSON-serialisable value:
|
|
24
|
+
|
|
25
|
+
```js
|
|
26
|
+
async () => {
|
|
27
|
+
const cases = [
|
|
28
|
+
['identity', { query: 'known-value' }, {}],
|
|
29
|
+
['empty string', { query: '' }, {}],
|
|
30
|
+
['absent field', {}, {}],
|
|
31
|
+
['limit above cap', { query: 'known-value' }, { limit: 1001 }],
|
|
32
|
+
];
|
|
33
|
+
const out = [];
|
|
34
|
+
for (const [label, body, qs] of cases) {
|
|
35
|
+
const p = new URLSearchParams(Object.entries(qs).map(([k, v]) => [k, String(v)]));
|
|
36
|
+
const url = '/api/v1/search' + (p.toString() ? '?' + p : '');
|
|
37
|
+
try {
|
|
38
|
+
const t0 = performance.now();
|
|
39
|
+
const r = await fetch(url, {
|
|
40
|
+
method: 'POST',
|
|
41
|
+
credentials: 'include',
|
|
42
|
+
headers: { 'Content-Type': 'application/json' },
|
|
43
|
+
body: JSON.stringify(body),
|
|
44
|
+
});
|
|
45
|
+
const txt = await r.text();
|
|
46
|
+
let count = null, rows = null;
|
|
47
|
+
try {
|
|
48
|
+
const j = JSON.parse(txt);
|
|
49
|
+
count = j.totalCount ?? j.total ?? null;
|
|
50
|
+
rows = (j.items ?? j.data ?? []).length;
|
|
51
|
+
} catch { /* not JSON — keep the body prefix instead */ }
|
|
52
|
+
out.push([
|
|
53
|
+
label, r.status,
|
|
54
|
+
count === null ? '-' : count,
|
|
55
|
+
rows === null ? '-' : rows,
|
|
56
|
+
Math.round(performance.now() - t0) + 'ms',
|
|
57
|
+
count === null ? txt.replace(/\s+/g, ' ').slice(0, 100) : '',
|
|
58
|
+
].join(' | '));
|
|
59
|
+
} catch (e) {
|
|
60
|
+
out.push(label + ' | NETERR | ' + String(e).slice(0, 60));
|
|
61
|
+
}
|
|
62
|
+
}
|
|
63
|
+
return out.join('\n');
|
|
64
|
+
}
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
Two flags earn their place:
|
|
68
|
+
|
|
69
|
+
- **`--headed`** — an interactive identity provider (SSO, MFA, a device posture check)
|
|
70
|
+
cannot complete headless. Log in once with a head; the profile keeps it.
|
|
71
|
+
- **`--filename`** — writes the result straight to a file. Use it for anything large:
|
|
72
|
+
the evidence lands on disk without being paid for in context.
|
|
73
|
+
|
|
74
|
+
### Out-of-browser runner — when you need volume
|
|
75
|
+
|
|
76
|
+
For a long matrix, or one you want to re-run identically in several environments, drive
|
|
77
|
+
`curl` from a small script and emit a machine-readable result per case: label, status,
|
|
78
|
+
count, rows, elapsed, error prefix.
|
|
79
|
+
|
|
80
|
+
Take the credential from the authenticated profile (`playwright-cli cookie-list`), write
|
|
81
|
+
it to a file **outside the repository**, and pass the path in by environment variable.
|
|
82
|
+
Never inline a session credential into a script, a committed file, a report, or a
|
|
83
|
+
message. It is a live credential for a real account.
|
|
84
|
+
|
|
85
|
+
### Which mode proves what
|
|
86
|
+
|
|
87
|
+
| Mode | Proves |
|
|
88
|
+
|------|--------|
|
|
89
|
+
| In-page `fetch` | The endpoint behaves this way **for a real logged-in session**, through the same client stack the UI uses |
|
|
90
|
+
| Out-of-browser request | The endpoint behaves this way **for anyone holding that credential** — the client-side stack is out of the picture |
|
|
91
|
+
| Public request, no credential | The endpoint is reachable unauthenticated — a finding in itself if it should not be |
|
|
92
|
+
|
|
93
|
+
Name the mode next to every verdict. They are different claims.
|
|
94
|
+
|
|
95
|
+
## 2. The Case Families
|
|
96
|
+
|
|
97
|
+
The ACs are the floor. Add these regardless of what the ticket asks for — every one of
|
|
98
|
+
them has produced a real defect.
|
|
99
|
+
|
|
100
|
+
| Family | Cases |
|
|
101
|
+
|--------|-------|
|
|
102
|
+
| **Identity** | A known-good input whose expected answer you can state in advance. Without it, nothing else is interpretable. |
|
|
103
|
+
| **Length boundaries** | 0, 1, minimum−1, minimum, minimum+1, documented maximum, maximum+1, and something far past it |
|
|
104
|
+
| **Tokenisation** | Multiple words; leading, trailing and repeated inner whitespace; one long token vs the same characters split; a term where every token is below the minimum length |
|
|
105
|
+
| **Metacharacters** | The operators of whatever query language is underneath — the search index, the SQL dialect, the regex engine, the template. Probe them **individually and unbalanced**: a lone `(`, a lone `)`, `*`, `{`, `}`, `:`, `@`, `|`, `\`, `"`, and a realistic value that happens to contain one |
|
|
106
|
+
| **Emptiness** | `""`, `null`, whitespace-only, the field absent, the whole body `{}`. These are four different requests and they routinely behave four different ways |
|
|
107
|
+
| **Enums and filters** | A valid value; an invalid value; an empty string; an empty array; `null`; a wildcard; two values; a value carrying a metacharacter |
|
|
108
|
+
| **Pagination** | `limit` 0, 1, the cap, cap+1, huge, negative, non-numeric, fractional; `offset` 0, negative, non-numeric, past the end |
|
|
109
|
+
| **Type confusion** | A string where a number is expected, a number where a string is, an array where a scalar is, an object where an array is |
|
|
110
|
+
| **Casing and normalisation** | Upper, lower, mixed; accented and full-width characters; a trailing space |
|
|
111
|
+
| **Identity and scope** | The same request as a second user, and as a user in a different tenant/market/scope. A read that ignores scope is a data-exposure finding |
|
|
112
|
+
|
|
113
|
+
**Metacharacters are not an edge case when real data contains them.** If values in the
|
|
114
|
+
collection carry brackets, colons or asterisks, then copying a value out of a result and
|
|
115
|
+
pasting it back in is an ordinary user action — and it is the most common way this class
|
|
116
|
+
of bug reaches a customer.
|
|
117
|
+
|
|
118
|
+
## 3. Reading the Results
|
|
119
|
+
|
|
120
|
+
| Signal | What it usually means |
|
|
121
|
+
|--------|----------------------|
|
|
122
|
+
| `500` on a family of inputs | One shared character or shape is reaching a query builder unescaped. One bug, not one per input |
|
|
123
|
+
| `200` with an error body | The error contract is broken; every caller will treat the failure as success |
|
|
124
|
+
| `200`, `count > 0`, zero rows | Count and page come from different code paths, or the cap silently clamped to nothing |
|
|
125
|
+
| `200` and empty for an invalid enum | Silent rejection. Indistinguishable from a genuine no-match, and it will be read as data |
|
|
126
|
+
| `200` and the whole collection for an empty query | A match-all fallback. The severity is the size of the collection and who can call it |
|
|
127
|
+
| `limit` above the cap honoured | The cap is documentation, not code. A denial-of-service and a bulk-extraction path |
|
|
128
|
+
| Non-numeric `limit` → `400` | Correct. Record the passes too — a matrix of only failures cannot show a regression later |
|
|
129
|
+
| Identical request, different answer per environment | Configuration, not code. Fingerprint before concluding anything |
|
|
130
|
+
|
|
131
|
+
## 4. Evidence Hygiene
|
|
132
|
+
|
|
133
|
+
- **Raw first, interpretation second.** Save the unmodified result, then write the
|
|
134
|
+
verdict beside it. An interpretation with no raw output is an opinion.
|
|
135
|
+
- **Redact on the way in.** Response bodies contain real records — names, addresses,
|
|
136
|
+
identifiers, internal hostnames in error strings. Replace with `[REDACTED]` before the
|
|
137
|
+
file is written, not in a later pass.
|
|
138
|
+
- **Never commit a credential.** Cookie jars, bearer tokens and profile directories stay
|
|
139
|
+
out of the repository. `.auth/` and `data/targets/local-*.yml` are already ignored.
|
|
140
|
+
- **Counts are perishable.** Record the date and the environment beside every number, and
|
|
141
|
+
never compare a count taken today against one taken last week as if it were a result.
|