criteria-review 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. package/LICENSE +21 -0
  2. package/README.md +299 -0
  3. package/agents/README.md +123 -0
  4. package/agents/skills/claude-code/criteria-lookup/SKILL.md +236 -0
  5. package/bin/criteria-review.js +870 -0
  6. package/docs/decisions.md +310 -0
  7. package/docs/help/01-getting-started.md +170 -0
  8. package/docs/standard/00-roles-and-ownership.md +353 -0
  9. package/docs/standard/01-qa-approach.md +273 -0
  10. package/docs/standard/02-writing-acceptance-criteria.md +187 -0
  11. package/docs/standard/03-example-acceptance-document.md +244 -0
  12. package/docs/standard/04-playwright-journeys.md +329 -0
  13. package/docs/standard/05-example-journey.spec.ts +130 -0
  14. package/docs/standard/06-engineering-obligations.md +170 -0
  15. package/docs/standard/07-manual-qa-and-defects.md +138 -0
  16. package/docs/standard/08-definition-of-done.md +128 -0
  17. package/docs/standard/09-recording-journeys.md +644 -0
  18. package/docs/standard/README.md +70 -0
  19. package/docs/standard/adopting-the-standard.md +104 -0
  20. package/docs/standard/emit-contract.md +175 -0
  21. package/package.json +41 -0
  22. package/public/app.js +791 -0
  23. package/public/doc-name.js +35 -0
  24. package/public/index.html +100 -0
  25. package/public/markdown.js +0 -0
  26. package/public/queue-order.js +78 -0
  27. package/public/styles.css +414 -0
  28. package/src/config.js +104 -0
  29. package/src/emit.js +183 -0
  30. package/src/media.js +126 -0
  31. package/src/parse.js +298 -0
  32. package/src/scan.js +158 -0
  33. package/src/server.js +433 -0
  34. package/src/standard.js +174 -0
  35. package/src/version.js +19 -0
  36. package/src/videos.js +185 -0
  37. package/src/write.js +307 -0
@@ -0,0 +1,138 @@
1
+ # Manual QA and defect reporting
2
+
3
+ The target is that automation covers the tier 1 scenarios and manual execution shrinks to the
4
+ residue automation genuinely cannot reach. Until then, and permanently for that residue,
5
+ manual QA is a written artefact rather than a session of clicking around.
6
+
7
+ ---
8
+
9
+ ## What stays manual, permanently
10
+
11
+ Naming these is what stops the manual checklist quietly re-growing to cover everything.
12
+
13
+ - **Physical devices and their quirks.** A real phone in a real hand: on-screen keyboards
14
+ covering the action button, safe area insets, an installed web app versus the same page in a
15
+ browser tab.
16
+ - **Peripherals.** Card terminals, receipt printers, cash drawer kick-outs, scanners.
17
+ - **Anything requiring a real external account.** A live payment gateway, an app store
18
+ install, a real identity provider consent screen.
19
+ - **Look and feel judgement.** Whether a screen is usable at arm's length on a counter, under
20
+ a shop's lighting, by someone in a hurry.
21
+ - **Genuinely destructive or one-way operations** in a shared environment.
22
+
23
+ Everything else that is still manual is a backlog item, not a category.
24
+
25
+ ## The manual scenario format
26
+
27
+ One file per area, mirroring the acceptance areas where they overlap.
28
+
29
+ ```markdown
30
+ # Scenario: <name>
31
+ Owner: <who maintains this>
32
+ Last updated: YYYY-MM-DD
33
+ Routes: /a, /b
34
+ Prereqs: <seeded data, role required, environment notes>
35
+
36
+ ## TC-<AREA>-NN: <case name>
37
+ | Step | Action | Expected |
38
+ |---|---|---|
39
+ | 1 | ... | ... |
40
+ | 2 | ... | ... |
41
+
42
+ ## Edge cases
43
+ - ...
44
+
45
+ ## Known limitations
46
+ - ...
47
+ ```
48
+
49
+ Rules:
50
+
51
+ - **`Expected` is a specific observable outcome**, not "works correctly". A tester who has
52
+ never seen the feature has to be able to tell pass from fail without asking anyone.
53
+ - **Prereqs are complete.** The commonest reason a manual case gets skipped is that nobody can
54
+ work out how to get into the starting state.
55
+ - **Known limitations are stated, including where a case cannot currently be run at all.** A
56
+ case that cannot be exercised because the product does not yet support the situation is
57
+ recorded as exactly that, with a pointer to what would unblock it. Silently dropping it is
58
+ how a coverage gap becomes invisible.
59
+
60
+ ## The regression checklist
61
+
62
+ One page, ticked as you go, in two parts.
63
+
64
+ **Part 1: smoke, five minutes, all must pass.** Reach the application, sign in, land on the
65
+ home screen, walk the two or three navigation paths that prove the shell works, sign out. **On
66
+ the first failure here, stop and raise a blocker.** The rest of the checklist is not worth a
67
+ tester's afternoon if the shell is broken.
68
+
69
+ **Part 2: the scenario walk.** One line per scenario file, ticked when every case inside it
70
+ passes. Each line names the cases in a phrase, so a tester reading the checklist knows what
71
+ they are about to cover without opening the file.
72
+
73
+ Then a short cross-cutting section, the things no single scenario owns: keyboard navigation
74
+ and focus order, the mobile viewport, offline and reconnect behaviour, and installability
75
+ where it applies.
76
+
77
+ Then sign off: every scenario green or a defect filed, and the release notes drafted.
78
+
79
+ **A new scenario file is not finished until it has a line in the checklist.** A scenario nobody
80
+ walks is documentation, not QA.
81
+
82
+ ## The defect report bar
83
+
84
+ Replacing every placeholder is the bar. Partial reports get handed back, because a defect
85
+ report that cannot be reproduced costs more than it saves.
86
+
87
+ ```markdown
88
+ **Title:** [area] short summary in active voice
89
+ e.g. "[till] cart total ignores modifier price"
90
+
91
+ **Severity:** P0 / P1 / P2 / P3
92
+ - P0 = release blocker; data loss or full feature break
93
+ - P1 = major flow broken with no workaround
94
+ - P2 = noticeable issue with a workaround
95
+ - P3 = polish, copy, minor usability
96
+
97
+ **Affects build:** commit hash, change number, or environment URL
98
+ **Browser / device:** e.g. Chrome 122 on macOS 14.4, or "iPhone 13, Safari, iOS 17"
99
+ **Tester:** name
100
+
101
+ ## Repro steps
102
+ 1. Sign in as ___ to ___
103
+ 2. Navigate to ___
104
+ 3. ___
105
+
106
+ ## Expected
107
+ What should happen, AND where that expectation comes from: a scenario id, a criteria
108
+ document, prior behaviour, or a specification.
109
+
110
+ ## Actual
111
+ What happened. Error text verbatim.
112
+
113
+ ## Evidence
114
+ - Screenshots or video
115
+ - Console log excerpt
116
+ - Network log excerpt if it looks server shaped
117
+ - Automation trace if it reproduces from a journey
118
+
119
+ ## Reproducibility
120
+ - [ ] Always
121
+ - [ ] Sometimes (estimate %): ___
122
+ - [ ] Once, could not reproduce after
123
+
124
+ ## Notes
125
+ Recent related changes, account state, network conditions.
126
+ ```
127
+
128
+ The field that does the real work is **"where that expectation comes from"**. A report whose
129
+ expected behaviour cites a scenario id is a disagreement between the software and an agreed
130
+ requirement, and it is not arguable. A report whose expected behaviour is the reporter's
131
+ opinion is a conversation, and it is worth knowing which one you have before the triage
132
+ meeting starts.
133
+
134
+ ## Turning a manual case into a journey
135
+
136
+ When a manual case is automated, the manual case does not get deleted quietly. It is replaced
137
+ by a line naming the scenario id that now covers it, so the checklist keeps showing the full
138
+ surface and shrinking coverage is visible as a change rather than as an omission.
@@ -0,0 +1,128 @@
1
+ # Definition of done
2
+
3
+ The other files describe the artefacts. This one sequences them: the order a piece of work
4
+ walks, what each stage produces, and what blocks passage.
5
+
6
+ Every stage has an **artefact** and a **gate that cannot be satisfied by assertion**.
7
+
8
+ ---
9
+
10
+ ## The stages
11
+
12
+ | # | Stage | Artefact | Gate on leaving |
13
+ |---|---|---|---|
14
+ | 0 | Pickup | Scope read; the shortlist of engineering obligations named | The shortlist is named, not "considered" |
15
+ | 1 | Plan | Tier 1 scenarios, tagged `derived`, and the tier 2 obligations from the shortlist | **Both tiers exist before implementation**, and have been shown to whoever owns the intent |
16
+ | 2 | Build | The implementation | Contracts and callers updated together, no compatibility shims left behind |
17
+ | 3 | Service tests | Tests citing scenario ids for tier 1 and obligation ids for tier 2 | Every test names one or the other |
18
+ | 4 | Gate | Completeness sweep over the surfaces that emerged while building | Answers are lists, test names, or specific sentences. Never "yes" |
19
+ | 5 | Journeys | One journey per tier 1 scenario, its clauses generated from the criteria rather than retyped | Every scenario is cited by a journey, or carries a recorded reason it cannot be |
20
+ | 6 | Evidence | Citations resolve; the videos are indexed by scenario id | The bundle exists and was produced by a command, not assembled by hand |
21
+ | 7 | Sign off | Status promotions, with date and commit | Watching promotes to `verified`. Confirmation of intent promotes to `accepted` |
22
+
23
+ **Stage 1 is the load bearing one.** Both tiers get enumerated before anyone has an interest
24
+ in the list being short, and tier 2 comes from a catalogue rather than from imagination.
25
+
26
+ **Stage 4 exists because planning cannot enumerate what only emerges while building.**
27
+ Anything the sweep finds that planning missed is fed back into how stage 1 is done, so the
28
+ next item plans better. Without that loop the sweep is a patch rather than a correction.
29
+
30
+ **Nothing here proves adequacy.** A clean walk means well formed, and the citations resolve.
31
+ It never means the coverage is sufficient. Say so wherever a stage is reported as passed, or
32
+ the report becomes exactly the false assurance it was built to prevent.
33
+
34
+ ---
35
+
36
+ ## The one command test
37
+
38
+ The acceptance test for this whole system:
39
+
40
+ > **Can one command, at the end of a piece of work, produce the evidence bundle?**
41
+
42
+ If assembling it is manual work at delivery time, it gets skipped exactly when the delivery is
43
+ most under pressure, which is when it matters most. It has to be **generated** from artefacts
44
+ that were structured for it from the start, which is what actually justifies every constraint
45
+ in the other files: the stable ids, the citations, the status tags, the video naming.
46
+
47
+ The bundle is: the user stories, the acceptance criteria, and a video per scenario id. It has
48
+ to be readable by someone who will never clone the repository.
49
+
50
+ ---
51
+
52
+ ## Confirming before building, not only after
53
+
54
+ The same export, minus the videos, is what gets shared **before** implementation.
55
+
56
+ A document carries force in a disagreement only if it was **agreed**. Sharing it beforehand is
57
+ what converts it from "what the developer assumed" into "what we agreed", and I have had that
58
+ distinction settle a dispute that would otherwise have been engineering opinion against a
59
+ customer's recollection.
60
+
61
+ It is also where intent is corrected most cheaply, because no code exists yet.
62
+
63
+ ---
64
+
65
+ ## Enforcement checks output, not input
66
+
67
+ Reminders have failed me repeatedly, and the failures were not ignorance. The rules were
68
+ written down, in front of me, and read. They were then not applied, because a general reminder
69
+ is agreed with and moved past.
70
+
71
+ So enforcement fires at the moment work is declared done, not at the moment it begins, and it
72
+ asks a **specific question derived from the change**, never a general one:
73
+
74
+ > "Remember: do not ship an interface that outruns the behaviour behind it."
75
+ >
76
+ > Agreed with, and moved past.
77
+
78
+ > "This change added a guard refusing the operation when the record is retired, and touched no
79
+ > client file. Name the interface gate, or state why none is needed."
80
+ >
81
+ > Cannot be waved away.
82
+
83
+ That class generalises to any rule enforced on one side of a boundary while the other side is
84
+ untouched: a new endpoint no test names, a new enum value nothing renders, a changed contract
85
+ without regenerated models, a new background job nothing exercises. **Five questions derived
86
+ from the diff beat fifty rules**, because each one is answerable and none can be satisfied by
87
+ nodding along.
88
+
89
+ ---
90
+
91
+ ## The self review, run as if someone else wrote it
92
+
93
+ Before anything is called done, the change is read back against these. The bar is "would an
94
+ unbiased reviewer call this out?", and grading your own work more gently because you wrote it
95
+ is the whole failure mode this is written to defeat.
96
+
97
+ 1. **Order of operations under partial failure.** If the persistent write succeeds and the
98
+ audit, queue enqueue or publish fails after it, what state is left, and can the caller
99
+ retry safely?
100
+ 2. **Idempotency under retry.** Redelivery, a client retry, a double tap. Is the deduplication
101
+ keyed off something deterministic and persisted, or off a flag that will not be set yet?
102
+ 3. **Silent partial success.** Is there any path returning success while having skipped work?
103
+ 4. **Error shape.** Does a structured server error round trip to the user with its intent
104
+ intact, or arrive as an unreadable object?
105
+ 5. **Drift between the tests and the code.** If a test asserts behaviour the code does not
106
+ deliver, or the reverse, one of them is wrong. Do not ship until they agree.
107
+ 6. **Audit trail intact.** Every state change is recorded, or is itself an immutable creation
108
+ that serves as its own record.
109
+ 7. **Best effort boundaries are best effort.** Messaging and realtime failures do not surface
110
+ as server errors on the user's operation.
111
+ 8. **Everything green.** Build, type check, unit and integration suites. State the counts. If
112
+ a suite was skipped, say which and why.
113
+ 9. **The reproduction is followable.** Someone who has never seen the change can reproduce it
114
+ end to end from the write up: routes, inputs, expected outcomes, edge cases.
115
+
116
+ Answering "not applicable" is fine when it is honestly not applicable. The point of writing
117
+ the answers down is to force the question to be asked at all.
118
+
119
+ ---
120
+
121
+ ## What "done" is not
122
+
123
+ - Not "the build is green". Green is the floor, not the ceiling.
124
+ - Not "the tests pass". A test never observed failing is not evidence.
125
+ - Not "it is behind a flag". A half feature behind a flag is a half feature, and it looks
126
+ identical to a defect to whoever finds it.
127
+ - Not "I will finish it next time". If it cannot ship whole, the scope gets expanded or the
128
+ feature gets dropped, and that is a decision to raise rather than absorb.