qualiow-exploratory-testing 1.2.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +22 -0
- package/.env.example +27 -0
- package/CHANGELOG.md +194 -0
- package/README.md +428 -179
- package/agents/qa-gather-agent.md +24 -0
- package/{scripts → bin}/doctor-mobile.sh +57 -18
- package/bin/mcli +15 -9
- package/bin/mobile-cli.mjs +574 -286
- package/{scripts → bin}/setup-mobile.sh +56 -19
- package/bin/wadb +10 -4
- package/bin/wk-ios +29 -17
- package/bin/wkeval.mjs +7 -1
- package/data/domains/_default.yml +1 -1
- package/data/domains/identity.yml +138 -0
- package/data/knowledge/changelog.yml +104 -0
- package/data/knowledge/learned-patterns.md +172 -86
- package/data/knowledge/manifest.yml +481 -70
- package/data/knowledge/releases/v0.1.0/entries/technique-business-logic-race-conditions.yml +1 -1
- package/data/knowledge/releases/v0.1.0/release.yml +8 -1
- package/data/knowledge/releases/v0.2.0/entries/technique-contract-narrowing.yml +150 -0
- package/data/knowledge/releases/v0.2.0/entries/technique-test-suite-audit.yml +116 -0
- package/data/knowledge/releases/v0.2.0/release.yml +20 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-functional-diff-analysis.yml +119 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-llm-output-verification.yml +102 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-verification-mode-selection.yml +153 -0
- package/data/knowledge/releases/v0.3.0/release.yml +25 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-authenticated-api-probing.yml +137 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-environment-fingerprinting.yml +113 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-silent-failure-audit.yml +86 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-ui-api-differential.yml +97 -0
- package/data/knowledge/releases/v0.4.0/release.yml +28 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-config-surface-verification.yml +77 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-derived-value-verification.yml +81 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-expected-behaviour-specification.yml +97 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-presentation-integrity.yml +78 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-release-readiness-verification.yml +110 -0
- package/data/knowledge/releases/v0.5.0/release.yml +33 -0
- package/data/knowledge/releases/v0.6.0/entries/technique-async-callback-contracts.yml +98 -0
- package/data/knowledge/releases/v0.6.0/entries/technique-exactly-once-verification.yml +116 -0
- package/data/knowledge/releases/v0.6.0/release.yml +30 -0
- package/data/security/SECURITY-POLICY.md +273 -0
- package/data/targets/_default.yml +1 -1
- package/data/targets/_example-api-only.yml +80 -0
- package/data/targets/_example-backend.yml +123 -0
- package/data/targets/_example-mobile-emulation.yml +2 -2
- package/data/targets/_example-native-mobile.yml +2 -2
- package/data/targets/_example-sim-android-chrome.yml +1 -1
- package/data/targets/_example-sim-ios-safari.yml +1 -1
- package/data/templates/api-probe-matrix.md +99 -0
- package/data/templates/bug-report.md +29 -151
- package/data/templates/coverage-map.md +23 -6
- package/data/templates/expected-behaviour.md +94 -0
- package/data/templates/session-charter.md +3 -0
- package/data/templates/session-report.md +57 -80
- package/dist/cli/index.js +1500 -739
- package/dist/index.d.ts +673 -75
- package/dist/index.js +1664 -270
- package/docs/BACKEND-VERIFICATION.md +307 -0
- package/docs/GETTING-STARTED.md +188 -128
- package/docs/MOBILE-SETUP.md +149 -18
- package/package.json +32 -16
- package/skills/qa-explore/SKILL.md +31 -25
- package/skills/qa-explore/phases/00-setup.md +67 -41
- package/skills/qa-explore/phases/01-auth.md +32 -15
- package/skills/qa-explore/phases/02-charter.md +16 -5
- package/skills/qa-explore/phases/03-discovery.md +50 -26
- package/skills/qa-explore/phases/04-journeys.md +2 -2
- package/skills/qa-explore/phases/05-features.md +39 -24
- package/skills/qa-explore/phases/06-edge-cases.md +24 -13
- package/skills/qa-explore/phases/07-reporting.md +79 -140
- package/skills/qa-explore/references/context-integration.md +1 -1
- package/skills/qa-explore/references/output-contract.md +187 -0
- package/skills/qa-explore/references/paths.md +84 -0
- package/skills/qa-explore/references/playwright-agents-integration.md +61 -84
- package/skills/qa-explore/references/security-rules.md +66 -25
- package/skills/qa-explore/references/severity-guide.md +31 -4
- package/skills/qa-explore-cleanup/SKILL.md +17 -6
- package/skills/qa-explore-feedback/SKILL.md +18 -5
- package/skills/qa-explore-mobile/SKILL.md +86 -64
- package/skills/qa-explore-mobile/phases/00-setup.md +71 -62
- package/skills/qa-explore-mobile/phases/01-auth.md +28 -31
- package/skills/qa-explore-mobile/phases/02-charter.md +8 -4
- package/skills/qa-explore-mobile/phases/03-discovery.md +17 -11
- package/skills/qa-explore-mobile/phases/04-journeys.md +6 -6
- package/skills/qa-explore-mobile/phases/05-features.md +9 -9
- package/skills/qa-explore-mobile/phases/06-edge-cases.md +35 -40
- package/skills/qa-explore-mobile/phases/07-reporting.md +82 -56
- package/skills/qa-explore-mobile/references/mobile-edges.md +26 -26
- package/skills/qa-explore-quick/SKILL.md +60 -25
- package/skills/qa-explore-report/SKILL.md +40 -34
- package/skills/qa-gather/SKILL.md +17 -2
- package/skills/qa-knowledge-add/SKILL.md +20 -9
- package/skills/qa-knowledge-list/SKILL.md +12 -3
- package/skills/qa-target-setup/SKILL.md +63 -41
- package/skills/qa-verify-backend/SKILL.md +172 -0
- package/skills/qa-verify-backend/phases/00-setup.md +142 -0
- package/skills/qa-verify-backend/phases/01-ac-decomposition.md +58 -0
- package/skills/qa-verify-backend/phases/02-static-review.md +71 -0
- package/skills/qa-verify-backend/phases/03-live-verification.md +62 -0
- package/skills/qa-verify-backend/phases/03b-api-verification.md +170 -0
- package/skills/qa-verify-backend/phases/04-e2e-trigger.md +82 -0
- package/skills/qa-verify-backend/phases/05-reporting.md +163 -0
- package/skills/qa-verify-backend/references/api-probes.md +168 -0
- package/skills/qa-verify-backend/references/aws-readonly-probes.md +156 -0
- package/skills/qa-verify-backend/references/environment-fingerprinting.md +104 -0
- package/skills/qa-verify-backend/references/payload-verification.md +112 -0
- package/skills/qa-verify-backend/references/release-readiness.md +97 -0
- package/skills/qa-verify-backend/references/safety-rules.md +95 -0
- package/dist/cli/index.d.ts +0 -1
- package/dist/cli/index.js.map +0 -1
- package/dist/index.js.map +0 -1
|
@@ -0,0 +1,150 @@
|
|
|
1
|
+
id: technique-contract-narrowing
|
|
2
|
+
version: "0.2.0"
|
|
3
|
+
type: technique
|
|
4
|
+
name: "Contract Narrowing — Verifying a Swapped Data Source"
|
|
5
|
+
description: >
|
|
6
|
+
When a ticket replaces a data source that returns EVERYTHING with one that
|
|
7
|
+
returns ONLY WHAT YOU ASK FOR, the request's field-selection list silently
|
|
8
|
+
becomes the complete specification of the output. Every field any downstream
|
|
9
|
+
consumer reads must be traced to a selection token. Fields nobody remembered
|
|
10
|
+
to request vanish without an error, a log line, or a failed test.
|
|
11
|
+
author: "Derived from a backend AC-verification session — an event-processing service whose full-record read was replaced by a projection read"
|
|
12
|
+
source: "Qualiow backend AC verification session, 2026-08-20"
|
|
13
|
+
tags: [backend, api-contract, integration, data-integrity, refactor, silent-failure, regression]
|
|
14
|
+
domains: [all]
|
|
15
|
+
priority: high
|
|
16
|
+
added: "2026-08-20"
|
|
17
|
+
updated: "2026-08-20"
|
|
18
|
+
|
|
19
|
+
content:
|
|
20
|
+
summary: >
|
|
21
|
+
A whole class of high-impact, silent regressions comes from one kind of
|
|
22
|
+
change: swapping a "returns the full record" read for a "returns a
|
|
23
|
+
projection" read. The two look interchangeable in a diff — often a single
|
|
24
|
+
line — and every test can stay green while fields quietly stop being
|
|
25
|
+
published. The technique is a two-column audit: list every field the
|
|
26
|
+
downstream consumers read, and next to each one write the selection token
|
|
27
|
+
that requests it. Any row with a blank second column is a dropped field.
|
|
28
|
+
|
|
29
|
+
core_principle: >
|
|
30
|
+
"SELECT * has no contract. A projection IS the contract."
|
|
31
|
+
Before the change, the payload's shape was an emergent property of the
|
|
32
|
+
source record. After the change, the payload's shape is whatever the
|
|
33
|
+
request asked for. That moves the specification from the database into the
|
|
34
|
+
call site — and nobody reviews a call site as if it were a schema.
|
|
35
|
+
|
|
36
|
+
why_it_hides:
|
|
37
|
+
- name: "Optional-everywhere schemas"
|
|
38
|
+
detail: >
|
|
39
|
+
Validation layers (Zod, JSON Schema, TypeScript interfaces) usually mark
|
|
40
|
+
these fields optional, because the source genuinely omits them for some
|
|
41
|
+
records. So the response still parses, and a "does it conform to the
|
|
42
|
+
type?" test passes with the field missing.
|
|
43
|
+
- name: "Serializers drop undefined"
|
|
44
|
+
detail: >
|
|
45
|
+
A mapper that unconditionally assigns `field: source.thing` still
|
|
46
|
+
produces the key in memory when `source.thing` is undefined — but
|
|
47
|
+
JSON.stringify omits undefined keys. The object looks right in a
|
|
48
|
+
debugger and the field is absent on the wire. Always assert on the
|
|
49
|
+
SERIALIZED payload, not the object.
|
|
50
|
+
- name: "The wrong assertions are already written"
|
|
51
|
+
detail: >
|
|
52
|
+
Typical new tests assert (a) the output parses as the shared type and
|
|
53
|
+
(b) the downstream mapper does not throw. Both remain true when an
|
|
54
|
+
optional field goes missing. Neither can ever falsify "the payload is
|
|
55
|
+
unchanged."
|
|
56
|
+
- name: "No error surface"
|
|
57
|
+
detail: >
|
|
58
|
+
Nothing throws, no message reaches a DLQ, no alarm fires, error rates
|
|
59
|
+
are flat. The first signal is a downstream team noticing a column has
|
|
60
|
+
gone blank — typically weeks later, with the events unreplayable.
|
|
61
|
+
|
|
62
|
+
where_this_shape_appears:
|
|
63
|
+
- "SQL: `SELECT *` replaced with an explicit column list"
|
|
64
|
+
- "GraphQL: a query's selection set (adding a field to the schema does not add it to existing queries)"
|
|
65
|
+
- "Identity and account platforms: a full-record `search` call vs a `get account` call that returns only the sections named in an `include=` token list"
|
|
66
|
+
- "REST sparse fieldsets: `?fields=`, JSON:API `fields[type]=`"
|
|
67
|
+
- "gRPC / protobuf: `FieldMask` on read requests"
|
|
68
|
+
- "DynamoDB: adding a `ProjectionExpression`, or a GSI with KEYS_ONLY/INCLUDE projection"
|
|
69
|
+
- "Elasticsearch/OpenSearch: `_source` filtering or `stored_fields`"
|
|
70
|
+
- "MongoDB: the projection argument to `find()`"
|
|
71
|
+
- "Any cache or read-model rebuild that materializes 'the fields we currently need'"
|
|
72
|
+
|
|
73
|
+
procedure:
|
|
74
|
+
- step: "1. Find the true consumer, not the immediate caller."
|
|
75
|
+
detail: >
|
|
76
|
+
Follow the value from the swapped call all the way to what leaves the
|
|
77
|
+
service — the published event, the API response, the written row. The
|
|
78
|
+
mapper at the end of that chain is the real specification.
|
|
79
|
+
- step: "2. Enumerate every field the consumer reads."
|
|
80
|
+
detail: >
|
|
81
|
+
Read the mapper line by line. Include fields read from nested objects
|
|
82
|
+
and fields only used in a conditional branch — a branch that rarely
|
|
83
|
+
fires is still a payload difference when it does.
|
|
84
|
+
- step: "3. Build the two-column table: field -> selection token required."
|
|
85
|
+
detail: >
|
|
86
|
+
For each field, determine from the API's documentation which token
|
|
87
|
+
causes it to be returned. Distinguish three cases: returned in the base
|
|
88
|
+
response, returned only if requested, and requested via a DIFFERENT
|
|
89
|
+
parameter than the main selection list.
|
|
90
|
+
- step: "4. Diff the table against the actual request."
|
|
91
|
+
detail: >
|
|
92
|
+
Every consumed field must map to a token that is actually sent. Blanks
|
|
93
|
+
are the findings. Do this against the code's real constant, not the
|
|
94
|
+
list written in the ticket — they drift.
|
|
95
|
+
- step: "5. Write the differential test."
|
|
96
|
+
detail: >
|
|
97
|
+
One input record, both code paths, compare the SERIALIZED output.
|
|
98
|
+
This is the only assertion that can falsify 'structurally identical'.
|
|
99
|
+
It is usually ~20 lines and it is the deliverable the ticket was
|
|
100
|
+
missing.
|
|
101
|
+
- step: "6. Confirm the API's behaviour from its documentation, not by inference."
|
|
102
|
+
detail: >
|
|
103
|
+
'This token exists in the selection list' is strong evidence a field is
|
|
104
|
+
not returned by default — an option you must request is by definition
|
|
105
|
+
not free. But quote the vendor doc; do not assert from intuition.
|
|
106
|
+
- step: "7. Verify against a real response if you can reach one."
|
|
107
|
+
detail: >
|
|
108
|
+
A single live call, or one log line from a real published event, either
|
|
109
|
+
confirms the finding under production conditions or falsifies your
|
|
110
|
+
premise. Cheap, and it converts a static finding into a measured one.
|
|
111
|
+
|
|
112
|
+
red_flags:
|
|
113
|
+
- "The ticket contains a hand-written 'Gap Analysis' table asserting which fields differ. That table is a claim, not a measurement — it is exactly what to falsify."
|
|
114
|
+
- "The ticket says 'the mapper works unchanged' or 'no downstream changes needed'. That is the hypothesis under test."
|
|
115
|
+
- "The story is small (1-3 points) and labelled 'Low risk' while touching a read path that fans out to other teams."
|
|
116
|
+
- "The selection list in the code was copy-pasted from the ticket description — including any mistakes in it."
|
|
117
|
+
- "A token in the selection list is not a valid value for that parameter (it may belong to a different parameter). It will be ignored silently."
|
|
118
|
+
- "The new tests are property-based over random valid data. Excellent for shape, blind to missing optional fields."
|
|
119
|
+
- "The diff's headline line count is dominated by a lockfile or generated file, so the four lines that matter are invisible in review."
|
|
120
|
+
|
|
121
|
+
test_approach:
|
|
122
|
+
- "Construct one representative record with EVERY consumed field populated — including the rarely-set ones."
|
|
123
|
+
- "Run it through both the old and the new normalization/mapping chain, using the real production mapper, not a stub."
|
|
124
|
+
- "Compare JSON.stringify of both results with strict equality. Assert equality, then let it fail and read the diff."
|
|
125
|
+
- "Repeat for each branch of the mapper (e.g. per-market or per-country variants) — they may read different fields."
|
|
126
|
+
- "Add the test permanently. The same defect recurs the next time the mapper grows a field, and only this test will catch it."
|
|
127
|
+
|
|
128
|
+
business_impact_framing: >
|
|
129
|
+
Explain the finding by its silence, not its size. "One field is missing" reads
|
|
130
|
+
as trivial. What makes it severe is: it affects every event from the moment of
|
|
131
|
+
deploy, nothing errors so nobody is paged, the events are not replayable, and
|
|
132
|
+
the discovery path is a downstream team noticing blank data during a much later
|
|
133
|
+
reconciliation. Also state honestly what would DOWNGRADE it — if no consumer
|
|
134
|
+
actually reads the field, it is a documentation fix, not a code fix. That
|
|
135
|
+
question is usually answerable only by asking the consuming team.
|
|
136
|
+
|
|
137
|
+
when_to_use:
|
|
138
|
+
- "Any ticket whose title contains 'replace X with Y', 'migrate to', or 'switch to' on a read path."
|
|
139
|
+
- "Any change from an implicit full fetch to an explicit projection."
|
|
140
|
+
- "Any AC that says the output is 'identical', 'unchanged', 'backward compatible', or 'transparent to consumers'."
|
|
141
|
+
- "Adding an index, cache, read model, or materialized view that will serve reads previously served by the source of truth."
|
|
142
|
+
- "Whenever a shared library gains a second way to produce the same normalized type — the two ways will drift."
|
|
143
|
+
|
|
144
|
+
gotchas:
|
|
145
|
+
- "The base response of a projection API is not empty — some fields always come back. Do not report those as missing; check the doc for which are free."
|
|
146
|
+
- "A field may be requested through a DIFFERENT parameter than the main selection list (e.g. an 'extra fields' parameter). Finding it absent from the main list is not sufficient to call it missing."
|
|
147
|
+
- "Test fixtures are built from the same wrong assumption as the code. A green suite built on a fixture that omits the field proves nothing."
|
|
148
|
+
- "Do not claim a value 'never arrives' from static reading alone. Trace it to the serializer, and measure it if you can — over-claiming burns credibility on the findings that are real."
|
|
149
|
+
- "Conversely, do not accept 'the tests pass' as evidence of equivalence. Ask what assertion could possibly have failed."
|
|
150
|
+
- "The reverse direction is a finding too: a projection that returns MORE than the old path (e.g. 'all identities' vs 'active identities') adds fields, which can also break strict downstream consumers."
|
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
id: technique-test-suite-audit
|
|
2
|
+
version: "0.2.0"
|
|
3
|
+
type: technique
|
|
4
|
+
name: "Auditing the Branch's Own Tests"
|
|
5
|
+
description: >
|
|
6
|
+
When a change rewrites the tests that are supposed to prove it works, those
|
|
7
|
+
tests stop being evidence and become part of the change under review. Read
|
|
8
|
+
them as production code: does each assertion still falsify what its name
|
|
9
|
+
claims? A suite can go green because the system works, or because the
|
|
10
|
+
assertion quietly stopped meaning anything.
|
|
11
|
+
author: "Derived from a backend AC-verification session — a branch that rewrote the end-to-end suite cited by its own acceptance criteria"
|
|
12
|
+
source: "Qualiow backend AC verification session, 2026-08-20"
|
|
13
|
+
tags: [test-quality, verification, ci, acceptance-criteria, backend, regression]
|
|
14
|
+
domains: [all]
|
|
15
|
+
priority: high
|
|
16
|
+
added: "2026-08-20"
|
|
17
|
+
updated: "2026-08-20"
|
|
18
|
+
|
|
19
|
+
content:
|
|
20
|
+
summary: >
|
|
21
|
+
An acceptance criterion of the form "the integration tests pass" is only as
|
|
22
|
+
good as what those tests assert. Whenever a branch touches its own test
|
|
23
|
+
suite — especially the suite cited by an AC — audit the assertions before
|
|
24
|
+
accepting a green run as proof. The highest-value checks are cheap: does
|
|
25
|
+
the artefact the AC names actually exist, and does each test's return value
|
|
26
|
+
still mean what its caller thinks it means?
|
|
27
|
+
|
|
28
|
+
core_principle: >
|
|
29
|
+
"A passing test proves something. Find out what."
|
|
30
|
+
Tests are the only part of a codebase people trust without reading. That
|
|
31
|
+
makes a weakened assertion the most durable kind of defect: it removes
|
|
32
|
+
protection permanently and reports success while doing it.
|
|
33
|
+
|
|
34
|
+
checks:
|
|
35
|
+
- name: "Do the cited artefacts exist?"
|
|
36
|
+
detail: >
|
|
37
|
+
An AC naming 'the existing Cucumber scenarios' or 'the integration
|
|
38
|
+
suite' is a falsifiable claim. Search for the files. If they do not
|
|
39
|
+
exist, that is an AC defect to report against the ticket — not a code
|
|
40
|
+
defect, and not something to quietly reinterpret as satisfied by
|
|
41
|
+
whatever tests do exist.
|
|
42
|
+
- name: "Did the meaning of a shared return value drift?"
|
|
43
|
+
detail: >
|
|
44
|
+
Look for a helper whose boolean/result is interpreted by several
|
|
45
|
+
callers. If the helper's definition changed but the callers' comments
|
|
46
|
+
still describe the old meaning, at least one caller is now wrong —
|
|
47
|
+
and an inverted caller (`const ok = !result`) will be wrong in the
|
|
48
|
+
direction that fails on a HEALTHY system.
|
|
49
|
+
- name: "Is the assertion at the right layer?"
|
|
50
|
+
detail: >
|
|
51
|
+
A producer cannot observe whether a downstream filter matched its
|
|
52
|
+
event. If a test claims to verify routing, filtering, or delivery, the
|
|
53
|
+
assertion must read the router/consumer (rule metrics, target state,
|
|
54
|
+
DLQ) — not the producer's own logs, which look identical either way.
|
|
55
|
+
- name: "Does pass/fail depend on observability config?"
|
|
56
|
+
detail: >
|
|
57
|
+
Assertions that grep logs inherit a hidden dependency on log level. If
|
|
58
|
+
the line asserted on is emitted at debug, the suite silently requires
|
|
59
|
+
debug logging in the target environment, and 'the flow is broken' and
|
|
60
|
+
'logging is turned down' become indistinguishable failures.
|
|
61
|
+
- name: "Were real assertions demoted to logging?"
|
|
62
|
+
detail: >
|
|
63
|
+
Watch for checks that used to decide pass/fail being reduced to console
|
|
64
|
+
output — often marked 'informational only'. The suite keeps its name
|
|
65
|
+
and its runtime and loses its teeth.
|
|
66
|
+
- name: "Can this test fail?"
|
|
67
|
+
detail: >
|
|
68
|
+
For each test, describe the concrete broken system that would turn it
|
|
69
|
+
red. If you cannot, it is decoration. Apply this hardest to negative
|
|
70
|
+
tests, which are the ones that silently stop discriminating.
|
|
71
|
+
- name: "Do the fixtures encode the same assumption as the code?"
|
|
72
|
+
detail: >
|
|
73
|
+
A fixture built by the author of the change shares the author's mental
|
|
74
|
+
model. If the code forgot a field, the fixture almost certainly omits
|
|
75
|
+
it too, and the test cannot see the gap.
|
|
76
|
+
|
|
77
|
+
failure_patterns:
|
|
78
|
+
- name: "Meaning drift across files"
|
|
79
|
+
example: >
|
|
80
|
+
A runner is changed to return 'the producer logged a publish line',
|
|
81
|
+
while its callers still treat the value as 'the event matched the
|
|
82
|
+
routing rule'. The negative scenario computes `!passed` and therefore
|
|
83
|
+
fails whenever the system behaves correctly.
|
|
84
|
+
- name: "Assertion inversion that hides in review"
|
|
85
|
+
example: >
|
|
86
|
+
Because the inverted case now fails on a healthy system, the team
|
|
87
|
+
learns to ignore that test — which also disables it for the real
|
|
88
|
+
regression it was written to catch.
|
|
89
|
+
- name: "Unconditional green"
|
|
90
|
+
example: >
|
|
91
|
+
A scenario asserts on behaviour that happens for every input, so it
|
|
92
|
+
passes regardless of the condition it claims to test.
|
|
93
|
+
- name: "Log-scraping as an oracle"
|
|
94
|
+
example: >
|
|
95
|
+
Pass/fail is decided by a debug log line, coupling the suite to a
|
|
96
|
+
LOG_LEVEL set in a build file nobody reads during review.
|
|
97
|
+
|
|
98
|
+
test_approach:
|
|
99
|
+
- "Diff the test files as carefully as the source files — read them last, when you already know what the code does."
|
|
100
|
+
- "For every rewritten helper, list its callers and check each caller's interpretation against the new definition."
|
|
101
|
+
- "For each negative/failure-path test, state the broken system that makes it red. No answer means no test."
|
|
102
|
+
- "Grep the AC's named artefacts (feature files, suite names) and confirm they exist before scoring the AC."
|
|
103
|
+
- "Run the suite. Then break the code deliberately in the way the test claims to catch, and confirm it goes red."
|
|
104
|
+
|
|
105
|
+
when_to_use:
|
|
106
|
+
- "Any AC of the form 'tests pass', 'existing scenarios still work', or 'covered by integration tests'."
|
|
107
|
+
- "Any branch whose diff includes non-trivial changes to its own test files."
|
|
108
|
+
- "Any refactor that rewrites a test harness or test runner rather than just updating expectations."
|
|
109
|
+
- "When coverage numbers are offered as evidence of behavioural equivalence."
|
|
110
|
+
|
|
111
|
+
gotchas:
|
|
112
|
+
- "High coverage and a weakened assertion coexist happily — coverage measures execution, not verification."
|
|
113
|
+
- "Property-based tests over random valid inputs are strong on shape and blind to missing optional fields."
|
|
114
|
+
- "A test failing on a correct system is not merely noise; it trains the team to ignore the file it lives in."
|
|
115
|
+
- "Do not report a test failure as a product defect until you have ruled out your own harness — a mis-resolved dependency or stale build produces errors that look exactly like real ones. Fix the harness, re-run, and only then file."
|
|
116
|
+
- "Separate AC defects from code defects in the write-up. 'The AC names something that does not exist' is feedback for the ticket author; 'the test is inverted' is feedback for the implementer."
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
# Release metadata for v0.2.0
|
|
2
|
+
version: "0.2.0"
|
|
3
|
+
date: "2026-08-20"
|
|
4
|
+
author: "qualiow"
|
|
5
|
+
status: active
|
|
6
|
+
|
|
7
|
+
notes: >
|
|
8
|
+
Backend and AC-verification techniques, derived from real /qa-verify-backend
|
|
9
|
+
sessions rather than from published literature.
|
|
10
|
+
Both entries target the same blind spot: changes that are silent by
|
|
11
|
+
construction. Contract Narrowing covers read-path swaps where a projection
|
|
12
|
+
replaces a full fetch and unrequested fields vanish without an error.
|
|
13
|
+
Auditing the Branch's Own Tests covers the case where the evidence for an
|
|
14
|
+
AC is itself part of the change under review.
|
|
15
|
+
|
|
16
|
+
entry_count: 2
|
|
17
|
+
|
|
18
|
+
entries:
|
|
19
|
+
- technique-contract-narrowing
|
|
20
|
+
- technique-test-suite-audit
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
id: technique-functional-diff-analysis
|
|
2
|
+
version: "0.3.0"
|
|
3
|
+
type: technique
|
|
4
|
+
name: "Functional Diff Analysis — Reading a Backend Change for Behaviour, Not Style"
|
|
5
|
+
description: >
|
|
6
|
+
Standard PR review asks "is this code good?". Verification asks a different
|
|
7
|
+
question: "what does the system do now that it did not do before, and who
|
|
8
|
+
feels it?". This technique reads a backend, API or LLM diff functionally —
|
|
9
|
+
tracing changed values to their consumers, hunting the failure paths the diff
|
|
10
|
+
does not show, and separating what the ticket asked for from what actually
|
|
11
|
+
shipped. It produces the hypotheses that the verification modes then go and
|
|
12
|
+
test.
|
|
13
|
+
author: "Qualiow — BE/API verification layer"
|
|
14
|
+
source: "Built out of /qa-verify-backend sessions, 2026-08"
|
|
15
|
+
tags: [backend, api, code-review, static-analysis, acceptance-criteria, llm, scope-creep, regression]
|
|
16
|
+
domains: [all]
|
|
17
|
+
priority: high
|
|
18
|
+
added: "2026-08-21"
|
|
19
|
+
updated: "2026-08-21"
|
|
20
|
+
|
|
21
|
+
content:
|
|
22
|
+
summary: >
|
|
23
|
+
Read the diff twice. The first pass is orientation: what changed, where, and
|
|
24
|
+
what the author says it does. The second pass is functional: for each changed
|
|
25
|
+
value, follow it to the edge of the service and ask what a consumer sees.
|
|
26
|
+
The output is not a list of code comments — it is a list of falsifiable
|
|
27
|
+
claims, each attached to an AC, each with a named way to observe it.
|
|
28
|
+
|
|
29
|
+
core_principle: >
|
|
30
|
+
A diff shows you the lines that changed. It does not show you the behaviour
|
|
31
|
+
that changed — that lives at the boundary of the service, several files
|
|
32
|
+
away, in the payload someone else consumes. Verification happens at the
|
|
33
|
+
boundary, so read from the boundary backwards.
|
|
34
|
+
|
|
35
|
+
what_this_is_not: >
|
|
36
|
+
This is not a code quality review. Naming, structure, duplication and style
|
|
37
|
+
belong to the PR reviewer, and mixing them in dilutes the findings that
|
|
38
|
+
affect behaviour. If a style issue causes a behavioural risk, report the
|
|
39
|
+
behaviour, and mention the style as the cause.
|
|
40
|
+
|
|
41
|
+
the_passes:
|
|
42
|
+
- pass: "1. Orientation"
|
|
43
|
+
questions:
|
|
44
|
+
- "What is the smallest set of files that could implement this ticket? Is the diff bigger than that, and why?"
|
|
45
|
+
- "Which files are generated, vendored, or lockfiles? Exclude them from the line count before judging size."
|
|
46
|
+
- "What does the commit message claim? Hold it as a hypothesis, not a summary."
|
|
47
|
+
- pass: "2. Spec drift — ticket vs code"
|
|
48
|
+
questions:
|
|
49
|
+
- "Take each AC literally. Does the code do that exact thing, or a nearby thing?"
|
|
50
|
+
- "Where the code differs from the ticket, which one is right? Say so — a better implementation than the AC asked for is still a spec that needs updating."
|
|
51
|
+
- "Does any AC name an artefact — a test suite, a scenario, a dashboard — that does not exist in the repo? That AC cannot pass as written."
|
|
52
|
+
- pass: "3. Value tracing"
|
|
53
|
+
questions:
|
|
54
|
+
- "For each value the diff changes, where does it leave the service? Published event, API response, written row, queue message, log."
|
|
55
|
+
- "Who reads it there? Another team, another service, a report, a downstream ticket?"
|
|
56
|
+
- "Does the changed value keep its old name, type, nullability, precision, and time zone? Type-compatible is not semantics-compatible."
|
|
57
|
+
- "Was a field REMOVED from a payload, or made conditional? Both are breaking for a strict consumer, and neither shows up as an error here."
|
|
58
|
+
- pass: "4. The failure paths"
|
|
59
|
+
questions:
|
|
60
|
+
- "The happy path is in the diff. What happens on timeout, on a 4xx from the dependency, on an empty result, on a partial batch failure?"
|
|
61
|
+
- "Is an error swallowed, retried, or propagated — and does the caller's wrapper turn the return value into the same outcome as a throw?"
|
|
62
|
+
- "Does a failure here poison a batch, drop a message silently, or produce a half-written record?"
|
|
63
|
+
- "What is the behaviour on replay or duplicate delivery? Most queue consumers get both."
|
|
64
|
+
- pass: "5. Identity, authorization, and tenancy"
|
|
65
|
+
questions:
|
|
66
|
+
- "If the change moves who calls what, is the caller's identity still propagated to the audit trail and the downstream authorization check?"
|
|
67
|
+
- "Does a broadened permission, a new role, or a new service principal appear in the diff without an AC asking for it?"
|
|
68
|
+
- "Can data cross a tenant, market, or environment boundary through the new path?"
|
|
69
|
+
- pass: "6. Contract with the other side"
|
|
70
|
+
questions:
|
|
71
|
+
- "If this is a producer, does the payload match what the consumer ticket expects — field names, key shape, ordering guarantees?"
|
|
72
|
+
- "If this is a consumer, does it tolerate fields it does not know, and does it fail loudly on fields it requires?"
|
|
73
|
+
- "Is there a version, a schema, or a compatibility mode? If not, the deploy order matters — say so."
|
|
74
|
+
- pass: "7. Scope"
|
|
75
|
+
questions:
|
|
76
|
+
- "Which hunks have nothing to do with any AC? List them explicitly."
|
|
77
|
+
- "Does an unrelated change touch a live path — validation rules, IAM, retention, a deletion script, CI that deletes things?"
|
|
78
|
+
- "Was a dependency lockfile regenerated? That is a behaviour change for every service in the repo, riding along unreviewed."
|
|
79
|
+
- pass: "8. The tests that came with it"
|
|
80
|
+
questions:
|
|
81
|
+
- "Do the new tests assert the thing the AC claims, or only that nothing threw?"
|
|
82
|
+
- "Could any of them fail if the change were wrong? Mentally break the code and see which test goes red."
|
|
83
|
+
- "See `technique-test-suite-audit` when the change rewrites its own evidence."
|
|
84
|
+
|
|
85
|
+
llm_and_agent_specifics:
|
|
86
|
+
note: >
|
|
87
|
+
Prompt, tool, model and retrieval changes are backend changes with an
|
|
88
|
+
unusually wide blast radius and unusually weak type checking. Read them with
|
|
89
|
+
the same passes, plus these.
|
|
90
|
+
questions:
|
|
91
|
+
- "A prompt edit is a spec edit. Which previously-specified behaviour did those words carry, and is it now unstated?"
|
|
92
|
+
- "Did an instruction move from the system prompt into a tool description, or vice versa? Precedence and persistence differ."
|
|
93
|
+
- "Did a tool's schema change — a field added, a description reworded, an enum widened? The model's behaviour is downstream of that text."
|
|
94
|
+
- "Did the model version, temperature, or context window change? That invalidates any prior output-based evidence, including green evals."
|
|
95
|
+
- "Are there few-shot examples that now contradict the edited instruction? The examples usually win."
|
|
96
|
+
- "Does the change affect what the agent is ALLOWED to do — new tool, broader permission, fewer confirmations? That is an authorization change, not a prompt change."
|
|
97
|
+
- "Is there an eval set, and does it contain cases that would fail if this edit were wrong? An eval that only covers the happy prompt proves nothing."
|
|
98
|
+
|
|
99
|
+
output_format: >
|
|
100
|
+
For each hypothesis produced: the AC it belongs to, a one-line statement of
|
|
101
|
+
what would be observed if it were true, the verification mode that could
|
|
102
|
+
observe it, and the file and line that prompted it. Hypotheses with no
|
|
103
|
+
observation channel go straight to the report as UNVERIFIABLE with a note —
|
|
104
|
+
they are still the most useful thing you can hand a reviewer.
|
|
105
|
+
|
|
106
|
+
red_flags:
|
|
107
|
+
- "A one-line diff on a read path that fans out to other teams, labelled low risk."
|
|
108
|
+
- "The ticket includes its own analysis table asserting what does and does not change. That is the claim to falsify, not the summary to trust."
|
|
109
|
+
- "Large generated files inflating the diff so the four lines that matter are invisible in review."
|
|
110
|
+
- "A change described as a refactor that also alters a default value, a timeout, a retry count, or a validation rule."
|
|
111
|
+
- "New code paths guarded by a feature flag whose default nobody states."
|
|
112
|
+
- "Test fixtures updated in the same commit as the behaviour they cover — the fixture may have been changed to match the new bug."
|
|
113
|
+
|
|
114
|
+
gotchas:
|
|
115
|
+
- "Do not review the merge-base you assume; compute it. A stale base makes unrelated main-branch changes look like part of this ticket."
|
|
116
|
+
- "Read branches with `git show <ref>:<path>` rather than checking out — someone may have uncommitted work in the tree."
|
|
117
|
+
- "Reading more diff is not the same as understanding more behaviour. Ten minutes tracing one value to its consumer beats an hour skimming every hunk."
|
|
118
|
+
- "Every hypothesis you cannot observe should still be written down. Silent omission looks identical to 'checked and fine'."
|
|
119
|
+
- "Kill your own findings before filing them. A report where every item survives scrutiny gets acted on; one with plausible-but-wrong items gets argued with."
|
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
id: technique-llm-output-verification
|
|
2
|
+
version: "0.3.0"
|
|
3
|
+
type: technique
|
|
4
|
+
name: "LLM & Agent Output Verification — Running It and Judging the Result"
|
|
5
|
+
description: >
|
|
6
|
+
Prompt, tool, model and retrieval changes cannot be verified by reading the
|
|
7
|
+
diff, and cannot be verified by exact-match assertions either. This technique
|
|
8
|
+
covers the third way: run the changed behaviour on a fixed input set and judge
|
|
9
|
+
the output against a written rubric, with non-determinism handled explicitly
|
|
10
|
+
rather than hoped away. It also covers what NOT to claim — a passing run on
|
|
11
|
+
five inputs is evidence, not proof.
|
|
12
|
+
author: "Qualiow — BE/API verification layer"
|
|
13
|
+
source: "Built out of the /qa-verify-backend verification-mode work, 2026-08"
|
|
14
|
+
tags: [llm, agent, api, verification, evidence, non-determinism, prompt, evaluation]
|
|
15
|
+
domains: [all]
|
|
16
|
+
priority: high
|
|
17
|
+
added: "2026-08-21"
|
|
18
|
+
updated: "2026-08-21"
|
|
19
|
+
|
|
20
|
+
content:
|
|
21
|
+
summary: >
|
|
22
|
+
Treat the LLM step as a component with a contract that happens to be written
|
|
23
|
+
in prose. Fix the inputs, run it enough times to see the variance, and score
|
|
24
|
+
the outputs on properties the AC actually names — not on wording. Report the
|
|
25
|
+
variance alongside the verdict.
|
|
26
|
+
|
|
27
|
+
core_principle: >
|
|
28
|
+
You are not testing whether the model is good. You are testing whether THIS
|
|
29
|
+
CHANGE moved the behaviour in the direction the AC claims, without breaking
|
|
30
|
+
what already worked. That is a comparison, so you need a before and an after,
|
|
31
|
+
on the same inputs.
|
|
32
|
+
|
|
33
|
+
procedure:
|
|
34
|
+
- step: "1. Write the rubric before running anything."
|
|
35
|
+
detail: >
|
|
36
|
+
Turn the AC into properties that can be scored yes/no on a single output:
|
|
37
|
+
'names a source', 'refuses when the record is missing', 'returns valid
|
|
38
|
+
JSON matching the schema', 'does not invent an order ID', 'calls the
|
|
39
|
+
lookup tool before answering'. Vague rubrics ('is helpful') produce
|
|
40
|
+
vague verdicts.
|
|
41
|
+
- step: "2. Fix the input set, and include the hostile cases."
|
|
42
|
+
detail: >
|
|
43
|
+
A handful of representative inputs, plus the ones the change is most
|
|
44
|
+
likely to break - empty input, very long input, ambiguous request,
|
|
45
|
+
missing data, contradictory instructions, and content designed to
|
|
46
|
+
override the system prompt. Reuse the SAME set before and after.
|
|
47
|
+
- step: "3. Run before and after on the same inputs."
|
|
48
|
+
detail: >
|
|
49
|
+
Without a baseline you cannot tell an improvement from a regression, and
|
|
50
|
+
you will attribute pre-existing behaviour to this diff. If the old
|
|
51
|
+
version is no longer runnable, say that — it caps what the run can prove.
|
|
52
|
+
- step: "4. Run each input more than once."
|
|
53
|
+
detail: >
|
|
54
|
+
Same input, several runs. A property that holds 3 times out of 5 is not
|
|
55
|
+
a PASS; it is a documented flake rate, and it is usually the most
|
|
56
|
+
valuable number in the report. State N in the evidence.
|
|
57
|
+
- step: "5. Score against the rubric, on the properties, never on the text."
|
|
58
|
+
detail: >
|
|
59
|
+
Exact-string assertions on generated prose fail on rewording and pass on
|
|
60
|
+
nonsense. Assert on structure (schema validity, required fields), on
|
|
61
|
+
facts (does the cited ID exist), and on behaviour (which tools were
|
|
62
|
+
called, in what order).
|
|
63
|
+
- step: "6. Check the tool calls, not only the final answer."
|
|
64
|
+
detail: >
|
|
65
|
+
For agent changes, the trajectory IS the behaviour. Capture which tools
|
|
66
|
+
were called with which arguments. A correct answer reached by a
|
|
67
|
+
forbidden or unnecessary call is a finding.
|
|
68
|
+
- step: "7. Save the raw outputs as evidence."
|
|
69
|
+
detail: >
|
|
70
|
+
Inputs, outputs, model version, temperature, prompt version, timestamp.
|
|
71
|
+
Without the model version the run is unreproducible and the evidence
|
|
72
|
+
expires the next time the vendor updates anything.
|
|
73
|
+
|
|
74
|
+
what_to_assert:
|
|
75
|
+
- "Schema and format: does the output parse, and does it match the contract the caller expects?"
|
|
76
|
+
- "Groundedness: is every factual claim traceable to something the input actually contained?"
|
|
77
|
+
- "Refusal and fallback: on missing or bad data, does it decline cleanly instead of inventing?"
|
|
78
|
+
- "Tool trajectory: the right tools, the right arguments, the right order, and nothing extra."
|
|
79
|
+
- "Boundary behaviour: empty, oversized, and ambiguous inputs."
|
|
80
|
+
- "Injection resistance: content in the DATA that instructs the model. It must be treated as data — this is the same rule the session security policy applies to web content."
|
|
81
|
+
- "Cost and latency, when the AC mentions them — a prompt that doubled in length is a change to both."
|
|
82
|
+
|
|
83
|
+
what_not_to_claim:
|
|
84
|
+
- "'The output is correct' from one run. Say 'correct in N of M runs' and give N and M."
|
|
85
|
+
- "'The prompt change is safe' from happy-path inputs only."
|
|
86
|
+
- "'No regression' without a before-run on the same inputs."
|
|
87
|
+
- "A verdict that outlives the model version it was measured on. Record the version; treat a model bump as invalidating prior evidence."
|
|
88
|
+
|
|
89
|
+
red_flags:
|
|
90
|
+
- "An eval suite that only contains inputs the current prompt already handles."
|
|
91
|
+
- "Assertions that compare generated text with `toEqual` on a fixed string."
|
|
92
|
+
- "A prompt edit shipped with no eval change at all — the spec moved and the evidence did not."
|
|
93
|
+
- "Temperature or model version changed in the same diff as behaviour, so neither can be attributed."
|
|
94
|
+
- "Few-shot examples that contradict the edited instruction. In practice the examples usually win."
|
|
95
|
+
- "The rubric was written after reading the outputs. That is scoring the target around the arrow."
|
|
96
|
+
|
|
97
|
+
gotchas:
|
|
98
|
+
- "A judge model scoring the outputs is itself a component with variance and bias. State that it was used, and spot-check its verdicts by hand."
|
|
99
|
+
- "Green evals after a model bump can be stale — confirm the runs actually executed against the new version."
|
|
100
|
+
- "Non-determinism hides regressions AND manufactures them. Re-run a failure before filing it."
|
|
101
|
+
- "Retrieval changes look like prompt changes but fail differently: check what was retrieved, not only what was said about it."
|
|
102
|
+
- "If the change is only in wording with no behavioural claim attached, this may be an UNVERIFIABLE — see `technique-verification-mode-selection`. Do not invent a rubric to manufacture a PASS."
|