qualiow-exploratory-testing 1.2.0 → 1.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (54) hide show
  1. package/README.md +92 -0
  2. package/data/domains/_default.md +181 -0
  3. package/data/domains/ecommerce.md +192 -0
  4. package/data/domains/fintech.md +200 -0
  5. package/data/domains/identity.md +88 -0
  6. package/data/domains/identity.yml +138 -0
  7. package/data/domains/marketing.md +194 -0
  8. package/data/domains/saas.md +197 -0
  9. package/data/knowledge/changelog.yml +86 -0
  10. package/data/knowledge/learned-patterns.md +172 -86
  11. package/data/knowledge/manifest.yml +211 -6
  12. package/data/knowledge/releases/v0.2.0/entries/technique-contract-narrowing.yml +150 -0
  13. package/data/knowledge/releases/v0.2.0/entries/technique-test-suite-audit.yml +116 -0
  14. package/data/knowledge/releases/v0.2.0/release.yml +20 -0
  15. package/data/knowledge/releases/v0.3.0/entries/technique-functional-diff-analysis.yml +119 -0
  16. package/data/knowledge/releases/v0.3.0/entries/technique-llm-output-verification.yml +102 -0
  17. package/data/knowledge/releases/v0.3.0/entries/technique-verification-mode-selection.yml +153 -0
  18. package/data/knowledge/releases/v0.3.0/release.yml +25 -0
  19. package/data/knowledge/releases/v0.4.0/entries/technique-authenticated-api-probing.yml +137 -0
  20. package/data/knowledge/releases/v0.4.0/entries/technique-environment-fingerprinting.yml +113 -0
  21. package/data/knowledge/releases/v0.4.0/entries/technique-silent-failure-audit.yml +86 -0
  22. package/data/knowledge/releases/v0.4.0/entries/technique-ui-api-differential.yml +97 -0
  23. package/data/knowledge/releases/v0.4.0/release.yml +28 -0
  24. package/data/knowledge/releases/v0.5.0/entries/technique-config-surface-verification.yml +77 -0
  25. package/data/knowledge/releases/v0.5.0/entries/technique-derived-value-verification.yml +81 -0
  26. package/data/knowledge/releases/v0.5.0/entries/technique-expected-behaviour-specification.yml +97 -0
  27. package/data/knowledge/releases/v0.5.0/entries/technique-presentation-integrity.yml +78 -0
  28. package/data/knowledge/releases/v0.5.0/entries/technique-release-readiness-verification.yml +110 -0
  29. package/data/knowledge/releases/v0.5.0/release.yml +33 -0
  30. package/data/targets/_example-backend.yml +123 -0
  31. package/data/templates/api-probe-matrix.md +99 -0
  32. package/data/templates/coverage-map.md +14 -0
  33. package/data/templates/expected-behaviour.md +93 -0
  34. package/dist/cli/index.js +79 -70
  35. package/dist/cli/index.js.map +1 -1
  36. package/dist/index.d.ts +90 -0
  37. package/dist/index.js +58 -49
  38. package/dist/index.js.map +1 -1
  39. package/docs/BACKEND-VERIFICATION.md +277 -0
  40. package/package.json +5 -3
  41. package/skills/qa-verify-backend/SKILL.md +163 -0
  42. package/skills/qa-verify-backend/phases/00-setup.md +97 -0
  43. package/skills/qa-verify-backend/phases/01-ac-decomposition.md +58 -0
  44. package/skills/qa-verify-backend/phases/02-static-review.md +71 -0
  45. package/skills/qa-verify-backend/phases/03-live-verification.md +62 -0
  46. package/skills/qa-verify-backend/phases/03b-api-verification.md +157 -0
  47. package/skills/qa-verify-backend/phases/04-e2e-trigger.md +66 -0
  48. package/skills/qa-verify-backend/phases/05-reporting.md +135 -0
  49. package/skills/qa-verify-backend/references/api-probes.md +141 -0
  50. package/skills/qa-verify-backend/references/aws-readonly-probes.md +156 -0
  51. package/skills/qa-verify-backend/references/environment-fingerprinting.md +104 -0
  52. package/skills/qa-verify-backend/references/payload-verification.md +112 -0
  53. package/skills/qa-verify-backend/references/release-readiness.md +97 -0
  54. package/skills/qa-verify-backend/references/safety-rules.md +90 -0
@@ -0,0 +1,71 @@
1
+ # Phase 2: Static Review
2
+
3
+ Read the branch against every AC. Use `git show <branch>:<path>` and
4
+ `git diff <base>...<branch> -- <path>` — never check out.
5
+
6
+ ## Step 1: Map the Diff to the ACs
7
+
8
+ ```bash
9
+ git -C <repo> diff --stat <base>...<branch>
10
+ ```
11
+
12
+ Group changed files by AC. Any file group that maps to **no** AC is either scope creep
13
+ or an undocumented dependency — both worth a line in the report. Any AC that maps to
14
+ **no** file is a candidate `FAIL` before you have read a single line.
15
+
16
+ ## Step 2: Read the Implementation Against Each Claim
17
+
18
+ Work AC by AC. For each, open the actual resource or handler and compare it to the
19
+ claim you wrote in phase 1. Record file and line for evidence.
20
+
21
+ Read with these lenses:
22
+
23
+ **Spec drift** — names, types, key shapes, enum values. The ticket says one thing, the
24
+ code says another. Even a benign-looking rename (`newImage` → `newConfig`, `PK` →
25
+ `entityId`, Map → JSON string) is a finding, because the consumer ticket was written
26
+ against the ticket's names.
27
+
28
+ **Scope** — is anything here that the ticket put explicitly **out of scope**? Storing
29
+ data the design said was not needed is not a bonus; it is unreviewed data collection
30
+ with privacy and size consequences.
31
+
32
+ **Least privilege** — enumerate the actions each policy grants and compare to what the
33
+ component provably needs. Note permissions created but unattached, and permissions
34
+ attached but unused.
35
+
36
+ **Failure path** — for every asynchronous component: what happens on retry exhaustion,
37
+ partial batch failure, oversized payload, missing field, or a downstream throttle? Find
38
+ where the error surfaces. If it surfaces nowhere, that is your highest-value finding.
39
+
40
+ **Uniqueness and ordering** — for anything keyed by time: what is the actual timestamp
41
+ granularity, and what happens to two events inside one tick? Collisions in a
42
+ write-once store mean silent overwrites.
43
+
44
+ **Identity propagation** — trace the actor from the token, through the handler, into
45
+ the persisted item, into the derived record. Find every place it can silently default
46
+ to a placeholder. Check create, update **and delete** separately — deletes routinely
47
+ attribute to the wrong person because the only image available is the pre-delete one.
48
+
49
+ **Documentation** — does any README, diagram or runbook still describe the architecture
50
+ this ticket removed? Stale docs fail the DoD line "documentation is created/updated"
51
+ and mislead the next engineer.
52
+
53
+ ## Step 3: Verify the Consumer Contract
54
+
55
+ If the ticket names a downstream ticket, write out the shape that consumer will need
56
+ and diff it against what this branch produces. Attribute names, types, key format,
57
+ sort semantics. Producer/consumer drift discovered here costs a rename; discovered in
58
+ the consumer's sprint it costs a migration of live data.
59
+
60
+ ## Step 4: Check the Tests
61
+
62
+ ```bash
63
+ git -C <repo> diff --stat <base>...<branch> -- '*.spec.*' '*.test.*'
64
+ ```
65
+
66
+ For each AC, is there a test that would fail if the AC broke? Note tests that were
67
+ **deleted** by this branch and not replaced — especially tests whose names described
68
+ behaviour that still ought to hold. A deleted test is a removed guardrail.
69
+
70
+ Record every finding with file, line, AC reference and severity. Write findings to
71
+ `evidence/static-review.md` as you go — do not carry them all in context.
@@ -0,0 +1,62 @@
1
+ # Phase 3: Live Verification
2
+
3
+ Skip if the live lane is BLOCKED — but still write every probe command into
4
+ `evidence/live-probes.md` so a human with credentials can run them unchanged.
5
+
6
+ Read `references/aws-readonly-probes.md` for the command catalogue. Read
7
+ `references/safety-rules.md` before the first probe, and confirm the fingerprint from
8
+ phase 0 step 3c is on file — a probe of an environment that does not run the change
9
+ measures something else.
10
+
11
+ This phase covers the **cloud resources**. The service's own HTTP endpoints are phase 3b
12
+ (`phases/03b-api-verification.md`); most tickets need both.
13
+
14
+ ## Rules
15
+
16
+ - **Read-only only.** `describe-*`, `get-*`, `list-*`, `query`, `scan`,
17
+ `simulate-principal-policy`, `filter-log-events`. Nothing that mutates.
18
+ - **One probe per AC.** Paste the raw output into `evidence/` before interpreting it.
19
+ Interpretation without the raw output is an opinion.
20
+ - **Confirm resource names first.** Names in a target config are asserted, not proven.
21
+ `list-tables` / `list-functions` before you trust them. A probe against a
22
+ non-existent resource returns an error that reads like a failure but is a typo.
23
+ - **Redact before writing to disk.** Real user emails, tokens, consumer PII.
24
+
25
+ ## Per-AC Probes
26
+
27
+ Work through the matrix. For each AC whose evidence source includes `live`:
28
+
29
+ 1. Run the probe.
30
+ 2. Save raw output to `evidence/<ac-id>-<resource>.json`.
31
+ 3. State the verdict and quote the specific field that decides it.
32
+
33
+ Example write-up:
34
+
35
+ ```
36
+ AC4 — stream view type
37
+ probe: aws dynamodb describe-table --table-name <config> --query 'Table.StreamSpecification'
38
+ raw: evidence/AC4-config-table.json
39
+ field: StreamViewType = "NEW_AND_OLD_IMAGES"
40
+ verdict: FAIL — AC requires NEW_IMAGE
41
+ ```
42
+
43
+ ## Probe the Failure Path, Not Just the Resource
44
+
45
+ The resource existing proves almost nothing. Add these regardless of what the ACs say:
46
+
47
+ - **Dead letter queue depth** — a non-zero `ApproximateNumberOfMessages` means events
48
+ are already being lost. Check it before and after the e2e trigger.
49
+ - **Lambda error and throttle metrics** over the life of the environment.
50
+ - **Event source mapping state** — `Enabled`, `LastProcessingResult`, and whether
51
+ partial-batch reporting is actually configured on the mapping (not just returned by
52
+ the handler code — the handler's return value is ignored without it).
53
+ - **Log level in the deployed function's environment** — a debug/trace level on a
54
+ component that logs whole event payloads is a data-exposure finding in any system
55
+ handling personal data.
56
+ - **Effective permissions** — `iam simulate-principal-policy` for both the actions the
57
+ component needs and the actions it must **not** have. Proving a denial is as
58
+ important as proving an allow.
59
+
60
+ ## Record
61
+
62
+ Update `ac-matrix.md` verdicts as you go. Append findings to `evidence/live-review.md`.
@@ -0,0 +1,157 @@
1
+ # Phase 3b: API Verification
2
+
3
+ Phase 3 asks *"is the resource configured the way the AC says?"*. This phase asks
4
+ *"does the endpoint actually behave the way the AC says, for inputs the UI will never
5
+ send?"* — the service's own HTTP surface, between the browser and the cloud resources.
6
+
7
+ Run it whenever the ticket touches a request/response contract: a search or list
8
+ endpoint, validation rules, pagination, filter values, an error contract, a payload
9
+ shape a consumer reads. Skip it when the change has no HTTP surface at all.
10
+
11
+ Read `references/api-probes.md` for the mechanics and the case catalogue,
12
+ `references/environment-fingerprinting.md` before you believe any result, and
13
+ `references/payload-verification.md` once you have one — a well-shaped `200` says nothing
14
+ about whether the numbers in it are right.
15
+
16
+ ## Step 1: Prove Which Implementation You Are Probing
17
+
18
+ **Do this before the first probe, and record it in `evidence/fingerprint.md`.**
19
+
20
+ A green result against an environment that does not run the changed code proves
21
+ nothing, and a red result there is not the ticket's bug. Establish, with evidence:
22
+
23
+ - the deployed build of every component in the request path (version endpoint, or a
24
+ behavioural fingerprint when there is none),
25
+ - whether the changed code path is **selected** here — a feature flag, a config
26
+ toggle, a routing rule can pick between two implementations of the same feature
27
+ inside one identical build,
28
+ - whether the commit carrying the change is actually an ancestor of what is deployed.
29
+
30
+ If the environment does not run the changed path, say so and stop this lane. The
31
+ verdict is `NOT-REACHABLE` for that environment (phase 5), not `PASS` and not `FAIL`.
32
+
33
+ ## Step 2: Get an Authenticated Request Context
34
+
35
+ Preferred: **run the request inside the already-authenticated page**. `fetch` from the
36
+ page context carries the session cookie, the CSRF token, and any client interceptors —
37
+ the same credentials the UI has — so there is no token to obtain and nothing to store.
38
+
39
+ ```bash
40
+ playwright-cli -s=<session> open <base_url> --headed --profile .auth/<profile> # once
41
+ playwright-cli -s=<session> eval "$(cat probes/<probe>.js)" --raw
42
+ ```
43
+
44
+ `--headed` matters when login goes through an interactive identity provider: that round
45
+ cannot complete headless. `--profile` is what makes it stick — a persistent profile
46
+ directory outlives a `storage_state` JSON by a wide margin.
47
+
48
+ Fall back to an out-of-browser runner (a script driving `curl`) only when you need
49
+ volume or a machine-readable matrix. Take the credential from the authenticated profile
50
+ (`playwright-cli cookie-list`), keep it in a file outside the repo, and reference it by
51
+ path — never inline it into a script, a report, or this repo.
52
+
53
+ Record which mode produced each result. An in-page probe and a bare `curl` do not prove
54
+ the same thing: the first proves the endpoint behaves this way *for a real logged-in
55
+ session*, the second proves it behaves this way *for whoever holds that credential*.
56
+
57
+ ## Step 3: Build the Case Matrix Before You Run Anything
58
+
59
+ Write the cases down first, as data, so the run is repeatable and the evidence is a
60
+ table rather than a scroll of output. Derive them from the ACs, then add the standard
61
+ families from `references/api-probes.md` — the ACs are a floor:
62
+
63
+ boundary lengths · tokenisation and whitespace · metacharacters of whatever query
64
+ language sits underneath · empty, null, absent and whitespace-only · enum values that
65
+ are valid, invalid, empty, wildcard and injected · pagination bounds and types · casing
66
+ and normalisation · type confusion on every field.
67
+
68
+ Each case gets a stable label. That label is what the report, the bug and the re-run all
69
+ refer to.
70
+
71
+ ## Step 4: Run, Record Raw, Then Interpret
72
+
73
+ Capture per case: **HTTP status, the count or shape the caller would read, how many rows
74
+ actually came back, elapsed time, and the first line of any error body.** Save the raw
75
+ result to `evidence/` — `eval --filename` writes it straight to disk without paying for
76
+ it in context — and interpret afterwards.
77
+
78
+ Read the results for these, in this order:
79
+
80
+ 1. **Status vs shape.** A `200` whose body is an error, or a `200` with `count > 0` and
81
+ zero rows, is worse than a `500` — the caller cannot tell it went wrong.
82
+ 2. **The silent zero.** An invalid enum, an unknown filter value or an unparseable term
83
+ that returns `200` with an empty result is indistinguishable from a genuine no-match.
84
+ Someone will read it as data.
85
+ 3. **Unbounded responses.** An empty or absent query that returns the whole collection;
86
+ a `limit` above the documented cap that is honoured anyway.
87
+ 4. **The crash family.** Group every input that `500`s and find the one character or
88
+ shape they share — that is one bug, not nine.
89
+ 5. **Inputs the UI cannot produce.** These are not reassurance. The browser guard is the
90
+ only thing preventing them, and mobile clients, integrations and scripts do not have
91
+ it. Judge them on the consequence of the call succeeding, not on how it was reached.
92
+
93
+ ## Step 5: Parity Across Environments
94
+
95
+ Run the identical matrix in every environment the ticket claims to affect. Then obey the
96
+ rule that makes the comparison meaningful:
97
+
98
+ > **Compare status codes and shapes. Never compare absolute counts.** Different
99
+ > environments hold different data, and the data drifts under you.
100
+
101
+ A divergence in *shape* between two environments running the same build is a
102
+ configuration finding — go back to step 1 and name the switch. A divergence in *counts*
103
+ is usually nothing.
104
+
105
+ ## Step 5b: Check the Values, Not Just the Shape
106
+
107
+ A `200` with a well-formed body has proven the contract, not the content. For every derived
108
+ value the ACs mention — a percentage, total, ratio, delta or aggregate — recompute it from
109
+ the raw figures in the same response, using the formula from the **specification** rather
110
+ than from the code under test. Pick cases that stress sign, zero, scale and cardinality,
111
+ and add the structural invariants the values must satisfy regardless of magnitude.
112
+
113
+ Then write down what the check does not prove: when both sides come from one payload you
114
+ have verified the derivation, not the inputs. Name the independent oracle that would close
115
+ it, and say whether you ran it. Full guidance in `references/payload-verification.md`.
116
+
117
+ ## Step 6: The Differential Pass — the Same Matrix at Both Surfaces
118
+
119
+ If the ticket also has a UI, run the overlapping cases through the screen too, and sort
120
+ every finding into four buckets:
121
+
122
+ | Bucket | Meaning |
123
+ |--------|---------|
124
+ | **Both** | The UI faithfully reflects the endpoint. Fix it once, in the service. |
125
+ | **API only** | A real defect the UI's own guard is hiding. Reachable by every non-browser client. |
126
+ | **UI only** | The client invents or masks a behaviour the service does not have — an error rendered as an empty result, a value re-formatted, a field dropped. |
127
+ | **Neither** | Environment or access noise, not a product defect. |
128
+
129
+ Hold the raw response next to every number on screen while you do this. A correct payload
130
+ can still reach the user as a wrong value — a formatter that guesses what a value is, a
131
+ unit applied twice, rounding that crosses a threshold, truncation shown as a total. When it
132
+ happens, work out which change owns the corruption before filing it; it is frequently not
133
+ the change under test, and it is frequently deployed on one environment and not another.
134
+
135
+ Two corrections this pass reliably produces, both worth stating explicitly in the
136
+ report:
137
+
138
+ - **A client-side guard recorded as a pass is not a pass.** "The button is disabled
139
+ until you type" describes the client, not the endpoint. Call the endpoint the way the
140
+ guard is preventing before you write `PASS`.
141
+ - **A discrepancy between what the endpoint returns and what the screen shows is its own
142
+ defect**, independent of the endpoint's behaviour, and it belongs to the client team.
143
+
144
+ ## Read-Only Discipline
145
+
146
+ The API lane calls **read** endpoints only. Reads that the ticket's own ACs describe,
147
+ plus the standard families above, against endpoints the target's `api.probe_allowlist`
148
+ declares. Anything that creates, mutates or deletes belongs to phase 4, goes through the
149
+ real write path, and never runs against production. See `references/safety-rules.md`.
150
+
151
+ Response bodies are real records. Redact before they touch the disk.
152
+
153
+ ## Record
154
+
155
+ Update `ac-matrix.md` as you go, one row per AC, each verdict naming the environment it
156
+ holds in. Write the case table to `evidence/api-probe-matrix.md` using
157
+ `data/templates/api-probe-matrix.md`.
@@ -0,0 +1,66 @@
1
+ # Phase 4: End-to-End Trigger
2
+
3
+ Covers the "verified in environment" style of AC: a real change through the real write
4
+ path, then re-probe the data layer.
5
+
6
+ Skip if `--no-e2e`, if the environment is production, or if the live lane is BLOCKED.
7
+
8
+ ## Step 1: Capture the Before State
9
+
10
+ Query the target store and record what exists **before** you touch anything. Without a
11
+ before, "a record appeared" cannot be distinguished from "a record was already there".
12
+
13
+ Also record DLQ depth and the Lambda's error count now.
14
+
15
+ ## Step 2: Trigger Through the Real Path
16
+
17
+ Prefer the path a real operator uses — the UI. Hand off to `/qa-explore` or drive
18
+ `playwright-cli` directly against the target's `base_url`. Make the change
19
+ **identifiable**: a value you can search for, containing no real data.
20
+
21
+ Note the exact wall-clock time of the save and the identity you were logged in as.
22
+ Both are needed to judge the resulting record.
23
+
24
+ If the UI is unavailable, call the write API directly with the same auth — but say so
25
+ in the report, because an API-only trigger does not prove the UI's identity
26
+ propagation works.
27
+
28
+ Conversely, if the UI trigger succeeds, that is not evidence the endpoint accepts what
29
+ other clients will send. The client's own guards are still in the path. Phase 3b covers
30
+ what happens without them.
31
+
32
+ ## Step 3: Re-Probe and Compare
33
+
34
+ Give the asynchronous path a moment, then query again and compare against the before
35
+ state. Check, in this order:
36
+
37
+ 1. **Count** — exactly the expected number of new records. Zero is a failure. More than
38
+ expected means duplication or fan-out nobody designed for.
39
+ 2. **Attribution** — the actor equals the user you logged in as. Not a role, not a
40
+ placeholder, not the previous editor.
41
+ 3. **Fidelity** — the stored snapshot matches what the source store now holds, field for
42
+ field. Compare the actual documents; do not eyeball two summaries. Where the record
43
+ carries derived values, recompute them from the raw figures beside them
44
+ (`references/payload-verification.md`) rather than trusting that they were computed
45
+ once and correctly.
46
+ 4. **Key shape** — the sort key format matches what the consumer will query and sort by.
47
+ 5. **Completeness** — every attribute the ACs require is present, with the required type.
48
+ A missing attribute here is a `FAIL` on that AC no matter how well the rest works.
49
+
50
+ ## Step 4: The Adversarial Passes
51
+
52
+ The happy path passing is the beginning, not the end. Run these — they are where audit
53
+ and event pipelines actually break:
54
+
55
+ - **Same-tick writes.** Save twice as fast as the UI allows. Then count records. If the
56
+ key is time-based at coarse granularity, one write silently overwrote the other.
57
+ - **Delete.** Delete an entity and check who the record attributes it to.
58
+ - **Bulk save.** A save that writes N items — do you get N attributable records?
59
+ - **Second identity.** Repeat as a different user; confirm attribution follows.
60
+ - **Isolation.** Change one scope (market/tenant/brand) and confirm no record touched
61
+ another.
62
+ - **DLQ after.** Re-check depth. Anything new means the pipeline dropped something
63
+ while you watched.
64
+
65
+ Record each with its own evidence file. Any of these that fails is a bug report,
66
+ whether or not an AC covers it — the ACs are a floor, not a ceiling.
@@ -0,0 +1,135 @@
1
+ # Phase 5: Reporting
2
+
3
+ ## Step 1: The AC Traceability Matrix
4
+
5
+ `ac-matrix.md` is the headline deliverable. One row per AC:
6
+
7
+ | AC | Claim | Verdict | Evidence | Note |
8
+ |----|-------|---------|----------|------|
9
+
10
+ Verdicts:
11
+
12
+ | Verdict | Means |
13
+ |---------|-------|
14
+ | **PASS** | Verified with cited evidence |
15
+ | **PARTIAL** | The intent is met but something material deviates from the stated spec |
16
+ | **FAIL** | The AC as written is not true |
17
+ | **BLOCKED** | Not verifiable with available access — probe command attached |
18
+ | **NOT-REACHABLE** | This environment does not run the changed code path — a flag, a config value or an undeployed commit. Not a pass and not a failure |
19
+ | **UNVERIFIABLE** | The AC has no falsifier as written — a DoR defect |
20
+
21
+ Never leave a row at `PENDING`. If you did not check it, the verdict is `BLOCKED` and
22
+ you say why.
23
+
24
+ **Scope every verdict to an environment**, in the row itself: `PASS (dev)` and `PASS` are
25
+ different claims, and only one of them is honest when the change is behind a flag that is
26
+ off everywhere else. Where the ticket claims several environments, give the matrix a
27
+ column per environment rather than one merged verdict.
28
+
29
+ Lead the matrix with the fingerprint from `evidence/fingerprint.md` — build id or commit
30
+ per component, and which implementation each environment selected. A reader who does not
31
+ know what was running cannot use the rest of the table.
32
+
33
+ ## Step 2: Bug Reports
34
+
35
+ One bug per report in `bugs/`, using `data/templates/bug-report.md`. Backend findings
36
+ need translation — nobody funds a fix for "StreamViewType is NEW_AND_OLD_IMAGES".
37
+
38
+ Write the **Business Impact** in terms of consequence:
39
+ - What can go wrong for a customer, an operator, or an auditor?
40
+ - What does it cost to fix now versus after the dependent work ships?
41
+ - For audit, identity and privacy findings: what happens during an incident review or a
42
+ regulatory request when this gap is discovered by someone else?
43
+
44
+ Give every reproducible finding a copy-pasteable reproduction — the probe command or
45
+ the trigger sequence — so the developer can confirm in under a minute.
46
+
47
+ Two shapes deserve their own reports rather than a line in someone else's:
48
+
49
+ - **A fix that cannot reach users.** If the changed path is off in the environments that
50
+ matter, the ticket is not done, however green the dev environment is. Say what has to
51
+ happen for it to reach production, and that the switch carries its own risk.
52
+ - **A discrepancy between the endpoint and the screen.** When the API is honest and the
53
+ UI is not — an error rendered as an empty result, a value re-scaled, a field dropped —
54
+ that is a client defect with its own owner, independent of the backend verdict.
55
+
56
+ **Collapse a crash family into one report.** Nine inputs that fail on the same unescaped
57
+ character are one bug with nine examples, not nine bugs.
58
+
59
+ ## Step 2b: Carry-Overs
60
+
61
+ Defects you found that pre-date the change under test go in their own section:
62
+ **pre-existing, not caused by this change, tracked separately.** Mixing them into the
63
+ results inflates the apparent risk of shipping this change and buries the findings that
64
+ belong to it.
65
+
66
+ The reverse holds too — a defect that exists only because two changes met in this build
67
+ belongs to the report even though neither change owns it alone. Say which two.
68
+
69
+ ## Step 3: DoD and DoR Gaps
70
+
71
+ Separate from bugs, list where the ticket's own Definition of Done is not met:
72
+ tests missing for an AC, documentation still describing the removed design,
73
+ monitoring absent for a new failure mode, manual work described in a runbook but never
74
+ executed and never evidenced.
75
+
76
+ List DoR gaps too — untestable ACs, missing acceptance criteria for behaviour the
77
+ change clearly has. These improve the next ticket rather than this one.
78
+
79
+ ## Step 3b: Coverage, Honestly
80
+
81
+ Where the session covered a set of ACs rather than one, keep a coverage map with four
82
+ states — not two:
83
+
84
+ | State | Means |
85
+ |-------|-------|
86
+ | ✅ tested and passing | Observed, with evidence |
87
+ | ⚠️ partially tested | Exercised, but a named part of the AC was not |
88
+ | ❌ not tested | In scope, never reached |
89
+ | 🔍 code-verified only | Read in the source and believed. **This is `UNVERIFIABLE`, not a pass** |
90
+
91
+ Two distinctions worth being pedantic about, because both routinely produce a false pass:
92
+
93
+ - **Consistent is not causal.** A setting whose current value happens to match the output
94
+ proves nothing until you change the setting and watch the output follow.
95
+ - **The data has to reach the case.** An AC about negative values cannot be verified
96
+ against records that are all zero. If no reachable record exercises it, that AC is
97
+ untested — find one, create one, or say plainly it is unproven and name what data would
98
+ settle it.
99
+
100
+ Then name, in one sentence, **which untested item carries the most risk**. In a report full
101
+ of green rows that is the line the reader acts on.
102
+
103
+ ## Step 3c: Write the Spec Where None Exists
104
+
105
+ Most of what an API probe finds has no acceptance criterion behind it, so there is nothing
106
+ to file the finding against and it turns into an argument instead of a fix. Produce an
107
+ expected-behaviour specification using `data/templates/expected-behaviour.md`: observed
108
+ against expected, grouped by cause rather than by case, with the decisions the fix forces
109
+ made explicit (reject or clamp; an error or an empty result; where validation lives), and
110
+ ranked by what real users can reach **today** rather than by how bad each one reads.
111
+
112
+ Close it with the same content in plain English, ready to paste into a ticket comment. That
113
+ section is what gets the work scheduled.
114
+
115
+ ## Step 4: Session Report
116
+
117
+ Use `data/templates/session-report.md`. Lead with the matrix summary
118
+ (`n PASS / n PARTIAL / n FAIL / n BLOCKED`), then the findings ranked by severity, then
119
+ what you could not check and exactly what access would unblock it.
120
+
121
+ Recommend a disposition in one line: ship, ship-with-follow-ups, or hold — and say what
122
+ would change your mind. Where the scope of what you checked is narrower than the question
123
+ being asked, say so in the same breath; a recommendation with an unstated scope is one
124
+ someone will over-read.
125
+
126
+ For a release-level session covering many tickets against one build, follow
127
+ `references/release-readiness.md` instead of this phase's per-AC shape: the deployment
128
+ table first, then a fixed result vocabulary per ticket that keeps *not testable here* and
129
+ *not tested* visible rather than letting them vanish between passed and failed.
130
+
131
+ ## Step 5: Confidentiality
132
+
133
+ Every output file gets the confidentiality header. Scan the whole session directory for
134
+ tokens, keys, real user emails and consumer PII before finishing; replace with
135
+ `[REDACTED]`. Session output stays local — never transmit it anywhere.
@@ -0,0 +1,141 @@
1
+ # API Probes — Reaching the Endpoint Behind the Screen
2
+
3
+ How to call a service's own HTTP surface with real credentials, what to ask it, and how
4
+ to read the answer. Companion to `phases/03b-api-verification.md`.
5
+
6
+ ## 1. Getting an Authenticated Request Context
7
+
8
+ ### In-page `fetch` — the default
9
+
10
+ Run the request inside the page that is already logged in. It inherits the session
11
+ cookie, the CSRF token and every client interceptor, so there is no token to obtain, no
12
+ credential to store, and no impersonation to arrange.
13
+
14
+ ```bash
15
+ # once per environment — interactive login the first time, the profile persists it
16
+ playwright-cli -s=alpha open https://service-alpha.example.com/ \
17
+ --headed --profile .auth/alpha-profile
18
+
19
+ # then, as many times as you like
20
+ playwright-cli -s=alpha eval "$(cat probes/search-cases.js)" --raw
21
+ ```
22
+
23
+ The probe is an async function returning a string or a JSON-serialisable value:
24
+
25
+ ```js
26
+ async () => {
27
+ const cases = [
28
+ ['identity', { query: 'known-value' }, {}],
29
+ ['empty string', { query: '' }, {}],
30
+ ['absent field', {}, {}],
31
+ ['limit above cap', { query: 'known-value' }, { limit: 1001 }],
32
+ ];
33
+ const out = [];
34
+ for (const [label, body, qs] of cases) {
35
+ const p = new URLSearchParams(Object.entries(qs).map(([k, v]) => [k, String(v)]));
36
+ const url = '/api/v1/search' + (p.toString() ? '?' + p : '');
37
+ try {
38
+ const t0 = performance.now();
39
+ const r = await fetch(url, {
40
+ method: 'POST',
41
+ credentials: 'include',
42
+ headers: { 'Content-Type': 'application/json' },
43
+ body: JSON.stringify(body),
44
+ });
45
+ const txt = await r.text();
46
+ let count = null, rows = null;
47
+ try {
48
+ const j = JSON.parse(txt);
49
+ count = j.totalCount ?? j.total ?? null;
50
+ rows = (j.items ?? j.data ?? []).length;
51
+ } catch { /* not JSON — keep the body prefix instead */ }
52
+ out.push([
53
+ label, r.status,
54
+ count === null ? '-' : count,
55
+ rows === null ? '-' : rows,
56
+ Math.round(performance.now() - t0) + 'ms',
57
+ count === null ? txt.replace(/\s+/g, ' ').slice(0, 100) : '',
58
+ ].join(' | '));
59
+ } catch (e) {
60
+ out.push(label + ' | NETERR | ' + String(e).slice(0, 60));
61
+ }
62
+ }
63
+ return out.join('\n');
64
+ }
65
+ ```
66
+
67
+ Two flags earn their place:
68
+
69
+ - **`--headed`** — an interactive identity provider (SSO, MFA, a device posture check)
70
+ cannot complete headless. Log in once with a head; the profile keeps it.
71
+ - **`--filename`** — writes the result straight to a file. Use it for anything large:
72
+ the evidence lands on disk without being paid for in context.
73
+
74
+ ### Out-of-browser runner — when you need volume
75
+
76
+ For a long matrix, or one you want to re-run identically in several environments, drive
77
+ `curl` from a small script and emit a machine-readable result per case: label, status,
78
+ count, rows, elapsed, error prefix.
79
+
80
+ Take the credential from the authenticated profile (`playwright-cli cookie-list`), write
81
+ it to a file **outside the repository**, and pass the path in by environment variable.
82
+ Never inline a session credential into a script, a committed file, a report, or a
83
+ message. It is a live credential for a real account.
84
+
85
+ ### Which mode proves what
86
+
87
+ | Mode | Proves |
88
+ |------|--------|
89
+ | In-page `fetch` | The endpoint behaves this way **for a real logged-in session**, through the same client stack the UI uses |
90
+ | Out-of-browser request | The endpoint behaves this way **for anyone holding that credential** — the client-side stack is out of the picture |
91
+ | Public request, no credential | The endpoint is reachable unauthenticated — a finding in itself if it should not be |
92
+
93
+ Name the mode next to every verdict. They are different claims.
94
+
95
+ ## 2. The Case Families
96
+
97
+ The ACs are the floor. Add these regardless of what the ticket asks for — every one of
98
+ them has produced a real defect.
99
+
100
+ | Family | Cases |
101
+ |--------|-------|
102
+ | **Identity** | A known-good input whose expected answer you can state in advance. Without it, nothing else is interpretable. |
103
+ | **Length boundaries** | 0, 1, minimum−1, minimum, minimum+1, documented maximum, maximum+1, and something far past it |
104
+ | **Tokenisation** | Multiple words; leading, trailing and repeated inner whitespace; one long token vs the same characters split; a term where every token is below the minimum length |
105
+ | **Metacharacters** | The operators of whatever query language is underneath — the search index, the SQL dialect, the regex engine, the template. Probe them **individually and unbalanced**: a lone `(`, a lone `)`, `*`, `{`, `}`, `:`, `@`, `|`, `\`, `"`, and a realistic value that happens to contain one |
106
+ | **Emptiness** | `""`, `null`, whitespace-only, the field absent, the whole body `{}`. These are four different requests and they routinely behave four different ways |
107
+ | **Enums and filters** | A valid value; an invalid value; an empty string; an empty array; `null`; a wildcard; two values; a value carrying a metacharacter |
108
+ | **Pagination** | `limit` 0, 1, the cap, cap+1, huge, negative, non-numeric, fractional; `offset` 0, negative, non-numeric, past the end |
109
+ | **Type confusion** | A string where a number is expected, a number where a string is, an array where a scalar is, an object where an array is |
110
+ | **Casing and normalisation** | Upper, lower, mixed; accented and full-width characters; a trailing space |
111
+ | **Identity and scope** | The same request as a second user, and as a user in a different tenant/market/scope. A read that ignores scope is a data-exposure finding |
112
+
113
+ **Metacharacters are not an edge case when real data contains them.** If values in the
114
+ collection carry brackets, colons or asterisks, then copying a value out of a result and
115
+ pasting it back in is an ordinary user action — and it is the most common way this class
116
+ of bug reaches a customer.
117
+
118
+ ## 3. Reading the Results
119
+
120
+ | Signal | What it usually means |
121
+ |--------|----------------------|
122
+ | `500` on a family of inputs | One shared character or shape is reaching a query builder unescaped. One bug, not one per input |
123
+ | `200` with an error body | The error contract is broken; every caller will treat the failure as success |
124
+ | `200`, `count > 0`, zero rows | Count and page come from different code paths, or the cap silently clamped to nothing |
125
+ | `200` and empty for an invalid enum | Silent rejection. Indistinguishable from a genuine no-match, and it will be read as data |
126
+ | `200` and the whole collection for an empty query | A match-all fallback. The severity is the size of the collection and who can call it |
127
+ | `limit` above the cap honoured | The cap is documentation, not code. A denial-of-service and a bulk-extraction path |
128
+ | Non-numeric `limit` → `400` | Correct. Record the passes too — a matrix of only failures cannot show a regression later |
129
+ | Identical request, different answer per environment | Configuration, not code. Fingerprint before concluding anything |
130
+
131
+ ## 4. Evidence Hygiene
132
+
133
+ - **Raw first, interpretation second.** Save the unmodified result, then write the
134
+ verdict beside it. An interpretation with no raw output is an opinion.
135
+ - **Redact on the way in.** Response bodies contain real records — names, addresses,
136
+ identifiers, internal hostnames in error strings. Replace with `[REDACTED]` before the
137
+ file is written, not in a later pass.
138
+ - **Never commit a credential.** Cookie jars, bearer tokens and profile directories stay
139
+ out of the repository. `.auth/` and `data/targets/local-*.yml` are already ignored.
140
+ - **Counts are perishable.** Record the date and the environment beside every number, and
141
+ never compare a count taken today against one taken last week as if it were a result.