criteria-review 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +299 -0
- package/agents/README.md +123 -0
- package/agents/skills/claude-code/criteria-lookup/SKILL.md +236 -0
- package/bin/criteria-review.js +870 -0
- package/docs/decisions.md +310 -0
- package/docs/help/01-getting-started.md +170 -0
- package/docs/standard/00-roles-and-ownership.md +353 -0
- package/docs/standard/01-qa-approach.md +273 -0
- package/docs/standard/02-writing-acceptance-criteria.md +187 -0
- package/docs/standard/03-example-acceptance-document.md +244 -0
- package/docs/standard/04-playwright-journeys.md +329 -0
- package/docs/standard/05-example-journey.spec.ts +130 -0
- package/docs/standard/06-engineering-obligations.md +170 -0
- package/docs/standard/07-manual-qa-and-defects.md +138 -0
- package/docs/standard/08-definition-of-done.md +128 -0
- package/docs/standard/09-recording-journeys.md +644 -0
- package/docs/standard/README.md +70 -0
- package/docs/standard/adopting-the-standard.md +104 -0
- package/docs/standard/emit-contract.md +175 -0
- package/package.json +41 -0
- package/public/app.js +791 -0
- package/public/doc-name.js +35 -0
- package/public/index.html +100 -0
- package/public/markdown.js +0 -0
- package/public/queue-order.js +78 -0
- package/public/styles.css +414 -0
- package/src/config.js +104 -0
- package/src/emit.js +183 -0
- package/src/media.js +126 -0
- package/src/parse.js +298 -0
- package/src/scan.js +158 -0
- package/src/server.js +433 -0
- package/src/standard.js +174 -0
- package/src/version.js +19 -0
- package/src/videos.js +185 -0
- package/src/write.js +307 -0
|
@@ -0,0 +1,138 @@
|
|
|
1
|
+
# Manual QA and defect reporting
|
|
2
|
+
|
|
3
|
+
The target is that automation covers the tier 1 scenarios and manual execution shrinks to the
|
|
4
|
+
residue automation genuinely cannot reach. Until then, and permanently for that residue,
|
|
5
|
+
manual QA is a written artefact rather than a session of clicking around.
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## What stays manual, permanently
|
|
10
|
+
|
|
11
|
+
Naming these is what stops the manual checklist quietly re-growing to cover everything.
|
|
12
|
+
|
|
13
|
+
- **Physical devices and their quirks.** A real phone in a real hand: on-screen keyboards
|
|
14
|
+
covering the action button, safe area insets, an installed web app versus the same page in a
|
|
15
|
+
browser tab.
|
|
16
|
+
- **Peripherals.** Card terminals, receipt printers, cash drawer kick-outs, scanners.
|
|
17
|
+
- **Anything requiring a real external account.** A live payment gateway, an app store
|
|
18
|
+
install, a real identity provider consent screen.
|
|
19
|
+
- **Look and feel judgement.** Whether a screen is usable at arm's length on a counter, under
|
|
20
|
+
a shop's lighting, by someone in a hurry.
|
|
21
|
+
- **Genuinely destructive or one-way operations** in a shared environment.
|
|
22
|
+
|
|
23
|
+
Everything else that is still manual is a backlog item, not a category.
|
|
24
|
+
|
|
25
|
+
## The manual scenario format
|
|
26
|
+
|
|
27
|
+
One file per area, mirroring the acceptance areas where they overlap.
|
|
28
|
+
|
|
29
|
+
```markdown
|
|
30
|
+
# Scenario: <name>
|
|
31
|
+
Owner: <who maintains this>
|
|
32
|
+
Last updated: YYYY-MM-DD
|
|
33
|
+
Routes: /a, /b
|
|
34
|
+
Prereqs: <seeded data, role required, environment notes>
|
|
35
|
+
|
|
36
|
+
## TC-<AREA>-NN: <case name>
|
|
37
|
+
| Step | Action | Expected |
|
|
38
|
+
|---|---|---|
|
|
39
|
+
| 1 | ... | ... |
|
|
40
|
+
| 2 | ... | ... |
|
|
41
|
+
|
|
42
|
+
## Edge cases
|
|
43
|
+
- ...
|
|
44
|
+
|
|
45
|
+
## Known limitations
|
|
46
|
+
- ...
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Rules:
|
|
50
|
+
|
|
51
|
+
- **`Expected` is a specific observable outcome**, not "works correctly". A tester who has
|
|
52
|
+
never seen the feature has to be able to tell pass from fail without asking anyone.
|
|
53
|
+
- **Prereqs are complete.** The commonest reason a manual case gets skipped is that nobody can
|
|
54
|
+
work out how to get into the starting state.
|
|
55
|
+
- **Known limitations are stated, including where a case cannot currently be run at all.** A
|
|
56
|
+
case that cannot be exercised because the product does not yet support the situation is
|
|
57
|
+
recorded as exactly that, with a pointer to what would unblock it. Silently dropping it is
|
|
58
|
+
how a coverage gap becomes invisible.
|
|
59
|
+
|
|
60
|
+
## The regression checklist
|
|
61
|
+
|
|
62
|
+
One page, ticked as you go, in two parts.
|
|
63
|
+
|
|
64
|
+
**Part 1: smoke, five minutes, all must pass.** Reach the application, sign in, land on the
|
|
65
|
+
home screen, walk the two or three navigation paths that prove the shell works, sign out. **On
|
|
66
|
+
the first failure here, stop and raise a blocker.** The rest of the checklist is not worth a
|
|
67
|
+
tester's afternoon if the shell is broken.
|
|
68
|
+
|
|
69
|
+
**Part 2: the scenario walk.** One line per scenario file, ticked when every case inside it
|
|
70
|
+
passes. Each line names the cases in a phrase, so a tester reading the checklist knows what
|
|
71
|
+
they are about to cover without opening the file.
|
|
72
|
+
|
|
73
|
+
Then a short cross-cutting section, the things no single scenario owns: keyboard navigation
|
|
74
|
+
and focus order, the mobile viewport, offline and reconnect behaviour, and installability
|
|
75
|
+
where it applies.
|
|
76
|
+
|
|
77
|
+
Then sign off: every scenario green or a defect filed, and the release notes drafted.
|
|
78
|
+
|
|
79
|
+
**A new scenario file is not finished until it has a line in the checklist.** A scenario nobody
|
|
80
|
+
walks is documentation, not QA.
|
|
81
|
+
|
|
82
|
+
## The defect report bar
|
|
83
|
+
|
|
84
|
+
Replacing every placeholder is the bar. Partial reports get handed back, because a defect
|
|
85
|
+
report that cannot be reproduced costs more than it saves.
|
|
86
|
+
|
|
87
|
+
```markdown
|
|
88
|
+
**Title:** [area] short summary in active voice
|
|
89
|
+
e.g. "[till] cart total ignores modifier price"
|
|
90
|
+
|
|
91
|
+
**Severity:** P0 / P1 / P2 / P3
|
|
92
|
+
- P0 = release blocker; data loss or full feature break
|
|
93
|
+
- P1 = major flow broken with no workaround
|
|
94
|
+
- P2 = noticeable issue with a workaround
|
|
95
|
+
- P3 = polish, copy, minor usability
|
|
96
|
+
|
|
97
|
+
**Affects build:** commit hash, change number, or environment URL
|
|
98
|
+
**Browser / device:** e.g. Chrome 122 on macOS 14.4, or "iPhone 13, Safari, iOS 17"
|
|
99
|
+
**Tester:** name
|
|
100
|
+
|
|
101
|
+
## Repro steps
|
|
102
|
+
1. Sign in as ___ to ___
|
|
103
|
+
2. Navigate to ___
|
|
104
|
+
3. ___
|
|
105
|
+
|
|
106
|
+
## Expected
|
|
107
|
+
What should happen, AND where that expectation comes from: a scenario id, a criteria
|
|
108
|
+
document, prior behaviour, or a specification.
|
|
109
|
+
|
|
110
|
+
## Actual
|
|
111
|
+
What happened. Error text verbatim.
|
|
112
|
+
|
|
113
|
+
## Evidence
|
|
114
|
+
- Screenshots or video
|
|
115
|
+
- Console log excerpt
|
|
116
|
+
- Network log excerpt if it looks server shaped
|
|
117
|
+
- Automation trace if it reproduces from a journey
|
|
118
|
+
|
|
119
|
+
## Reproducibility
|
|
120
|
+
- [ ] Always
|
|
121
|
+
- [ ] Sometimes (estimate %): ___
|
|
122
|
+
- [ ] Once, could not reproduce after
|
|
123
|
+
|
|
124
|
+
## Notes
|
|
125
|
+
Recent related changes, account state, network conditions.
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
The field that does the real work is **"where that expectation comes from"**. A report whose
|
|
129
|
+
expected behaviour cites a scenario id is a disagreement between the software and an agreed
|
|
130
|
+
requirement, and it is not arguable. A report whose expected behaviour is the reporter's
|
|
131
|
+
opinion is a conversation, and it is worth knowing which one you have before the triage
|
|
132
|
+
meeting starts.
|
|
133
|
+
|
|
134
|
+
## Turning a manual case into a journey
|
|
135
|
+
|
|
136
|
+
When a manual case is automated, the manual case does not get deleted quietly. It is replaced
|
|
137
|
+
by a line naming the scenario id that now covers it, so the checklist keeps showing the full
|
|
138
|
+
surface and shrinking coverage is visible as a change rather than as an omission.
|
|
@@ -0,0 +1,128 @@
|
|
|
1
|
+
# Definition of done
|
|
2
|
+
|
|
3
|
+
The other files describe the artefacts. This one sequences them: the order a piece of work
|
|
4
|
+
walks, what each stage produces, and what blocks passage.
|
|
5
|
+
|
|
6
|
+
Every stage has an **artefact** and a **gate that cannot be satisfied by assertion**.
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## The stages
|
|
11
|
+
|
|
12
|
+
| # | Stage | Artefact | Gate on leaving |
|
|
13
|
+
|---|---|---|---|
|
|
14
|
+
| 0 | Pickup | Scope read; the shortlist of engineering obligations named | The shortlist is named, not "considered" |
|
|
15
|
+
| 1 | Plan | Tier 1 scenarios, tagged `derived`, and the tier 2 obligations from the shortlist | **Both tiers exist before implementation**, and have been shown to whoever owns the intent |
|
|
16
|
+
| 2 | Build | The implementation | Contracts and callers updated together, no compatibility shims left behind |
|
|
17
|
+
| 3 | Service tests | Tests citing scenario ids for tier 1 and obligation ids for tier 2 | Every test names one or the other |
|
|
18
|
+
| 4 | Gate | Completeness sweep over the surfaces that emerged while building | Answers are lists, test names, or specific sentences. Never "yes" |
|
|
19
|
+
| 5 | Journeys | One journey per tier 1 scenario, its clauses generated from the criteria rather than retyped | Every scenario is cited by a journey, or carries a recorded reason it cannot be |
|
|
20
|
+
| 6 | Evidence | Citations resolve; the videos are indexed by scenario id | The bundle exists and was produced by a command, not assembled by hand |
|
|
21
|
+
| 7 | Sign off | Status promotions, with date and commit | Watching promotes to `verified`. Confirmation of intent promotes to `accepted` |
|
|
22
|
+
|
|
23
|
+
**Stage 1 is the load bearing one.** Both tiers get enumerated before anyone has an interest
|
|
24
|
+
in the list being short, and tier 2 comes from a catalogue rather than from imagination.
|
|
25
|
+
|
|
26
|
+
**Stage 4 exists because planning cannot enumerate what only emerges while building.**
|
|
27
|
+
Anything the sweep finds that planning missed is fed back into how stage 1 is done, so the
|
|
28
|
+
next item plans better. Without that loop the sweep is a patch rather than a correction.
|
|
29
|
+
|
|
30
|
+
**Nothing here proves adequacy.** A clean walk means well formed, and the citations resolve.
|
|
31
|
+
It never means the coverage is sufficient. Say so wherever a stage is reported as passed, or
|
|
32
|
+
the report becomes exactly the false assurance it was built to prevent.
|
|
33
|
+
|
|
34
|
+
---
|
|
35
|
+
|
|
36
|
+
## The one command test
|
|
37
|
+
|
|
38
|
+
The acceptance test for this whole system:
|
|
39
|
+
|
|
40
|
+
> **Can one command, at the end of a piece of work, produce the evidence bundle?**
|
|
41
|
+
|
|
42
|
+
If assembling it is manual work at delivery time, it gets skipped exactly when the delivery is
|
|
43
|
+
most under pressure, which is when it matters most. It has to be **generated** from artefacts
|
|
44
|
+
that were structured for it from the start, which is what actually justifies every constraint
|
|
45
|
+
in the other files: the stable ids, the citations, the status tags, the video naming.
|
|
46
|
+
|
|
47
|
+
The bundle is: the user stories, the acceptance criteria, and a video per scenario id. It has
|
|
48
|
+
to be readable by someone who will never clone the repository.
|
|
49
|
+
|
|
50
|
+
---
|
|
51
|
+
|
|
52
|
+
## Confirming before building, not only after
|
|
53
|
+
|
|
54
|
+
The same export, minus the videos, is what gets shared **before** implementation.
|
|
55
|
+
|
|
56
|
+
A document carries force in a disagreement only if it was **agreed**. Sharing it beforehand is
|
|
57
|
+
what converts it from "what the developer assumed" into "what we agreed", and I have had that
|
|
58
|
+
distinction settle a dispute that would otherwise have been engineering opinion against a
|
|
59
|
+
customer's recollection.
|
|
60
|
+
|
|
61
|
+
It is also where intent is corrected most cheaply, because no code exists yet.
|
|
62
|
+
|
|
63
|
+
---
|
|
64
|
+
|
|
65
|
+
## Enforcement checks output, not input
|
|
66
|
+
|
|
67
|
+
Reminders have failed me repeatedly, and the failures were not ignorance. The rules were
|
|
68
|
+
written down, in front of me, and read. They were then not applied, because a general reminder
|
|
69
|
+
is agreed with and moved past.
|
|
70
|
+
|
|
71
|
+
So enforcement fires at the moment work is declared done, not at the moment it begins, and it
|
|
72
|
+
asks a **specific question derived from the change**, never a general one:
|
|
73
|
+
|
|
74
|
+
> "Remember: do not ship an interface that outruns the behaviour behind it."
|
|
75
|
+
>
|
|
76
|
+
> Agreed with, and moved past.
|
|
77
|
+
|
|
78
|
+
> "This change added a guard refusing the operation when the record is retired, and touched no
|
|
79
|
+
> client file. Name the interface gate, or state why none is needed."
|
|
80
|
+
>
|
|
81
|
+
> Cannot be waved away.
|
|
82
|
+
|
|
83
|
+
That class generalises to any rule enforced on one side of a boundary while the other side is
|
|
84
|
+
untouched: a new endpoint no test names, a new enum value nothing renders, a changed contract
|
|
85
|
+
without regenerated models, a new background job nothing exercises. **Five questions derived
|
|
86
|
+
from the diff beat fifty rules**, because each one is answerable and none can be satisfied by
|
|
87
|
+
nodding along.
|
|
88
|
+
|
|
89
|
+
---
|
|
90
|
+
|
|
91
|
+
## The self review, run as if someone else wrote it
|
|
92
|
+
|
|
93
|
+
Before anything is called done, the change is read back against these. The bar is "would an
|
|
94
|
+
unbiased reviewer call this out?", and grading your own work more gently because you wrote it
|
|
95
|
+
is the whole failure mode this is written to defeat.
|
|
96
|
+
|
|
97
|
+
1. **Order of operations under partial failure.** If the persistent write succeeds and the
|
|
98
|
+
audit, queue enqueue or publish fails after it, what state is left, and can the caller
|
|
99
|
+
retry safely?
|
|
100
|
+
2. **Idempotency under retry.** Redelivery, a client retry, a double tap. Is the deduplication
|
|
101
|
+
keyed off something deterministic and persisted, or off a flag that will not be set yet?
|
|
102
|
+
3. **Silent partial success.** Is there any path returning success while having skipped work?
|
|
103
|
+
4. **Error shape.** Does a structured server error round trip to the user with its intent
|
|
104
|
+
intact, or arrive as an unreadable object?
|
|
105
|
+
5. **Drift between the tests and the code.** If a test asserts behaviour the code does not
|
|
106
|
+
deliver, or the reverse, one of them is wrong. Do not ship until they agree.
|
|
107
|
+
6. **Audit trail intact.** Every state change is recorded, or is itself an immutable creation
|
|
108
|
+
that serves as its own record.
|
|
109
|
+
7. **Best effort boundaries are best effort.** Messaging and realtime failures do not surface
|
|
110
|
+
as server errors on the user's operation.
|
|
111
|
+
8. **Everything green.** Build, type check, unit and integration suites. State the counts. If
|
|
112
|
+
a suite was skipped, say which and why.
|
|
113
|
+
9. **The reproduction is followable.** Someone who has never seen the change can reproduce it
|
|
114
|
+
end to end from the write up: routes, inputs, expected outcomes, edge cases.
|
|
115
|
+
|
|
116
|
+
Answering "not applicable" is fine when it is honestly not applicable. The point of writing
|
|
117
|
+
the answers down is to force the question to be asked at all.
|
|
118
|
+
|
|
119
|
+
---
|
|
120
|
+
|
|
121
|
+
## What "done" is not
|
|
122
|
+
|
|
123
|
+
- Not "the build is green". Green is the floor, not the ceiling.
|
|
124
|
+
- Not "the tests pass". A test never observed failing is not evidence.
|
|
125
|
+
- Not "it is behind a flag". A half feature behind a flag is a half feature, and it looks
|
|
126
|
+
identical to a defect to whoever finds it.
|
|
127
|
+
- Not "I will finish it next time". If it cannot ship whole, the scope gets expanded or the
|
|
128
|
+
feature gets dropped, and that is a decision to raise rather than absorb.
|