criteria-review 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +299 -0
- package/agents/README.md +123 -0
- package/agents/skills/claude-code/criteria-lookup/SKILL.md +236 -0
- package/bin/criteria-review.js +870 -0
- package/docs/decisions.md +310 -0
- package/docs/help/01-getting-started.md +170 -0
- package/docs/standard/00-roles-and-ownership.md +353 -0
- package/docs/standard/01-qa-approach.md +273 -0
- package/docs/standard/02-writing-acceptance-criteria.md +187 -0
- package/docs/standard/03-example-acceptance-document.md +244 -0
- package/docs/standard/04-playwright-journeys.md +329 -0
- package/docs/standard/05-example-journey.spec.ts +130 -0
- package/docs/standard/06-engineering-obligations.md +170 -0
- package/docs/standard/07-manual-qa-and-defects.md +138 -0
- package/docs/standard/08-definition-of-done.md +128 -0
- package/docs/standard/09-recording-journeys.md +644 -0
- package/docs/standard/README.md +70 -0
- package/docs/standard/adopting-the-standard.md +104 -0
- package/docs/standard/emit-contract.md +175 -0
- package/package.json +41 -0
- package/public/app.js +791 -0
- package/public/doc-name.js +35 -0
- package/public/index.html +100 -0
- package/public/markdown.js +0 -0
- package/public/queue-order.js +78 -0
- package/public/styles.css +414 -0
- package/src/config.js +104 -0
- package/src/emit.js +183 -0
- package/src/media.js +126 -0
- package/src/parse.js +298 -0
- package/src/scan.js +158 -0
- package/src/server.js +433 -0
- package/src/standard.js +174 -0
- package/src/version.js +19 -0
- package/src/videos.js +185 -0
- package/src/write.js +307 -0
|
@@ -0,0 +1,310 @@
|
|
|
1
|
+
# Decisions
|
|
2
|
+
|
|
3
|
+
Append-only, id-ordered. Each entry records what was decided, what was weighed against it, and
|
|
4
|
+
why the alternatives lost. This is the surface to check before asking "was this already
|
|
5
|
+
decided?", and the surface to check drift against.
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## D-001. This tool owns the acceptance-criteria standard
|
|
10
|
+
|
|
11
|
+
**Date:** 2026-08-11
|
|
12
|
+
**Status:** decided
|
|
13
|
+
|
|
14
|
+
Until now the standard was owned by the first project that needed it, and this tool described
|
|
15
|
+
itself as "a reader over that schema, not a second authority". Two consumer projects then
|
|
16
|
+
existed, one of which kept a local mirror of the format documentation that described itself as
|
|
17
|
+
a mirror. A mirror cannot know its original changed, so it rots into a contradiction, and the
|
|
18
|
+
project that owned the standard had no way to tell that it had.
|
|
19
|
+
|
|
20
|
+
The standard now lives here, in `docs/standard/`. Consumer projects hold pointers and their own
|
|
21
|
+
instance data, never a restatement.
|
|
22
|
+
|
|
23
|
+
**What decides where a sentence lives.** Does it stay true if the product is swapped for a
|
|
24
|
+
different one? If yes it belongs to the standard; if no it stays in the project. By that test
|
|
25
|
+
the tier model, scenario ids, the status ladder, evidence rules, intent sourcing, the
|
|
26
|
+
verification-cost rule, the journey binding and the lifecycle gates are all the standard's. A
|
|
27
|
+
project's own catalogue of engineering obligations instantiated for its domain, its module
|
|
28
|
+
coverage map, and whatever sign-off arrangement it has with a customer stay with the project.
|
|
29
|
+
|
|
30
|
+
**Alternatives weighed.**
|
|
31
|
+
|
|
32
|
+
- *Leave ownership with the originating project.* Rejected: the second consumer had already
|
|
33
|
+
written a mirror and labelled it as one, which is the failure this exists to remove. It also
|
|
34
|
+
makes a project's internal reorganisation a breaking change for everyone else.
|
|
35
|
+
- *A separate standards repository, with the tool as one consumer.* Rejected for now: the
|
|
36
|
+
parser, the review queue and the enforcement all live here, and a standard whose enforcement
|
|
37
|
+
lives somewhere else is a document rather than a standard. Revisit if a second tool ever
|
|
38
|
+
implements it.
|
|
39
|
+
- *Duplicate the standard into each project and check the copies match.* Rejected: that is a
|
|
40
|
+
drift detector for a problem that does not need to exist. Generate what can be generated,
|
|
41
|
+
point at the rest.
|
|
42
|
+
|
|
43
|
+
**Consequences.**
|
|
44
|
+
|
|
45
|
+
- The tool's own text stops disclaiming authority, and consumer projects stop naming each other
|
|
46
|
+
as the source.
|
|
47
|
+
- The standard needs a version, because a change to the tag vocabulary is now a breaking change
|
|
48
|
+
for every consumer. See D-002.
|
|
49
|
+
- Enforcement has to land with the ownership move rather than after it. Authority without
|
|
50
|
+
enforcement is a source of truth nothing validates against, which is worse than the previous
|
|
51
|
+
state where at least the owning project's document was the one people read.
|
|
52
|
+
|
|
53
|
+
---
|
|
54
|
+
|
|
55
|
+
## D-002. The standard is versioned, and the version is stamped into what it emits
|
|
56
|
+
|
|
57
|
+
**Date:** 2026-08-11
|
|
58
|
+
**Status:** decided
|
|
59
|
+
|
|
60
|
+
Once two repositories read from one source, a change to the vocabulary breaks builds the author
|
|
61
|
+
of the change cannot see. Consumers need to be able to say which version they last conformed
|
|
62
|
+
to, without inspecting the tool.
|
|
63
|
+
|
|
64
|
+
`STANDARD_VERSION` lives in one place in this repository and is stamped into every generated
|
|
65
|
+
artefact. Semantic: a change to the tag vocabulary, the status set, or the emitted shape is a
|
|
66
|
+
major bump; adding an optional field is a minor one; wording and documentation are a patch.
|
|
67
|
+
|
|
68
|
+
**Consequence.** A consumer that regenerates and sees the major digit move knows to read the
|
|
69
|
+
changelog before assuming the build failure is theirs.
|
|
70
|
+
|
|
71
|
+
---
|
|
72
|
+
|
|
73
|
+
## D-003. Scenario clauses reach the tests by generation, never by retyping
|
|
74
|
+
|
|
75
|
+
**Date:** 2026-08-11
|
|
76
|
+
**Status:** decided
|
|
77
|
+
|
|
78
|
+
The standard requires that a journey's step names be the scenario's own Given/When/Then
|
|
79
|
+
wording, so that what a reviewer reads and what the test proves are the same sentence. The
|
|
80
|
+
obvious implementation is to type those clauses into the test as step names. That was in use,
|
|
81
|
+
and it is wrong: it makes two copies of every clause with nothing detecting an edit to either.
|
|
82
|
+
|
|
83
|
+
Measured on the consumer project that has journeys: 18 spec files, 31 tests citing a scenario
|
|
84
|
+
id, 132 narrated steps. Every one of them was in exact sync with its document at the time of
|
|
85
|
+
the decision, which is the point - the sync was held by care alone, on the one artefact where
|
|
86
|
+
care is what already failed. A journey that dropped one of a scenario's five clauses would have
|
|
87
|
+
reported green.
|
|
88
|
+
|
|
89
|
+
Instead the clauses are parsed out of the acceptance documents and emitted as a typed module
|
|
90
|
+
that the journeys reference by key. The consumer supplies a body per clause; the type system
|
|
91
|
+
requires exactly the document's clauses, in the document's order.
|
|
92
|
+
|
|
93
|
+
**What that makes structural rather than conventional:** a reworded clause, a skipped clause, a
|
|
94
|
+
step the document does not contain, and a renamed or deleted scenario id all become compile
|
|
95
|
+
errors. The coverage map becomes the set of referenced keys. The video index stops being
|
|
96
|
+
retyped.
|
|
97
|
+
|
|
98
|
+
**Alternatives weighed.**
|
|
99
|
+
|
|
100
|
+
- *A text checker comparing the document to the spec source.* Rejected: it has to understand
|
|
101
|
+
the consumer's language and formatting. The first such check written by hand during this
|
|
102
|
+
investigation reported a false divergence, because one call was formatted on one line rather
|
|
103
|
+
than three. A checker that is wrong about formatting is worse than no checker.
|
|
104
|
+
- *Adopt a behaviour-driven runner with step definitions bound by expression matching.*
|
|
105
|
+
Rejected: shared reusable steps get written vaguely, and vague steps destroy the near 1:1
|
|
106
|
+
clause-to-assertion mapping that makes review affordable. This design keeps steps specific to
|
|
107
|
+
one scenario and cannot reach that failure mode.
|
|
108
|
+
- *Generate the whole test file.* Rejected: the bodies are the engineering, and generating them
|
|
109
|
+
would mean either a template language or a stub nobody fills in.
|
|
110
|
+
|
|
111
|
+
**Known limit, stated rather than glossed.** A required clause can still be satisfied by a body
|
|
112
|
+
that asserts nothing. That is a separate and cheaper check: a finding on a `Then` or trailing
|
|
113
|
+
`And` body containing no assertion.
|
|
114
|
+
|
|
115
|
+
**Cost accepted.** Editing an acceptance document now breaks the build of any journey citing it.
|
|
116
|
+
That is the feature - the alternative is an amended requirement and a test still proving the old
|
|
117
|
+
one, with nothing on screen to say so - but it does mean a reviewer's wording change is no
|
|
118
|
+
longer free.
|
|
119
|
+
|
|
120
|
+
---
|
|
121
|
+
|
|
122
|
+
## D-004. The emitted artefact is language-neutral, with renderers over it
|
|
123
|
+
|
|
124
|
+
**Date:** 2026-08-11
|
|
125
|
+
**Status:** decided
|
|
126
|
+
|
|
127
|
+
The first consumer is TypeScript, so the obvious move is to emit a TypeScript module. A
|
|
128
|
+
standard whose only output binds to one language is not a standard.
|
|
129
|
+
|
|
130
|
+
The core emit is JSON: the parsed model, filtered and ordered. A typed TypeScript module is one
|
|
131
|
+
renderer over that JSON. Other renderers can follow without touching the core.
|
|
132
|
+
|
|
133
|
+
**Consequence.** Adding the JSON layer now costs almost nothing. Retrofitting it later would
|
|
134
|
+
mean changing every consumer, which is the situation this whole entry exists to avoid.
|
|
135
|
+
|
|
136
|
+
---
|
|
137
|
+
|
|
138
|
+
## D-005. Generation is strict where the viewer is tolerant
|
|
139
|
+
|
|
140
|
+
**Date:** 2026-08-11
|
|
141
|
+
**Status:** decided
|
|
142
|
+
|
|
143
|
+
The parser is deliberately permissive: it accepts unfamiliar statuses, and it reports scenarios
|
|
144
|
+
with no id and no status as `untracked` rather than rejecting them. That is correct for a review
|
|
145
|
+
queue, because those scenarios are exactly the backlog worth surfacing.
|
|
146
|
+
|
|
147
|
+
It is wrong for generation. A consumer's build depends on the emitted artefact, so an unknown
|
|
148
|
+
status or a duplicate id is a defect in the consumer's documents, not a curiosity to render.
|
|
149
|
+
|
|
150
|
+
`generate` therefore refuses to emit on: an unrecognised status value, a duplicate scenario id
|
|
151
|
+
across the documents it was given, or a scenario carrying `@status:verified` without both
|
|
152
|
+
`@verified:` and `@commit:`. Untracked scenarios are excluded from the emitted set, because they
|
|
153
|
+
have no id to key on, but their count is reported so the exclusion is never silent.
|
|
154
|
+
|
|
155
|
+
**Alternative weighed.** *Emit what parses and warn about the rest.* Rejected: a warning in a
|
|
156
|
+
build log is not read. The failure this standard exists to prevent is silence, and a tool that
|
|
157
|
+
degrades quietly reintroduces it at the one point where a machine could have been definite.
|
|
158
|
+
|
|
159
|
+
---
|
|
160
|
+
|
|
161
|
+
## D-006. `proposed` joins the status ladder, and versioning splits additive from breaking
|
|
162
|
+
|
|
163
|
+
**Date:** 2026-08-11
|
|
164
|
+
**Status:** decided
|
|
165
|
+
**Standard version:** 1.1.0
|
|
166
|
+
|
|
167
|
+
Found by running the new generator against a real consumer branch rather than by reading: 22
|
|
168
|
+
scenarios there carry `@status:proposed`, and generation refused all of them. The status was
|
|
169
|
+
introduced by that project before the standard moved here, and this tool's own review UI already
|
|
170
|
+
ranks it, between untracked and `derived`.
|
|
171
|
+
|
|
172
|
+
It is adopted rather than migrated away, for a reason stronger than "it is already in use": the
|
|
173
|
+
standard's own lifecycle says tier 1 scenarios are authored at planning, before implementation,
|
|
174
|
+
**as proposals**. Without a status for that state the ladder cannot express its own stage 1, and
|
|
175
|
+
a scenario written before any code exists has to masquerade as `derived`, which claims it
|
|
176
|
+
describes delivered software.
|
|
177
|
+
|
|
178
|
+
| Status | Meaning |
|
|
179
|
+
|---|---|
|
|
180
|
+
| `proposed` | Written at planning, before the software exists. A proposal, not a description. |
|
|
181
|
+
| `derived` | Written up from the delivered software. Describes what it appears to do. |
|
|
182
|
+
| `verified` | A human watched the software do it, with date and commit. |
|
|
183
|
+
| `accepted` | Confirmed as what the software should do. |
|
|
184
|
+
|
|
185
|
+
`proposed` ranks below `derived` because a description of delivered software at least matches
|
|
186
|
+
something, while a proposal has not met reality yet.
|
|
187
|
+
|
|
188
|
+
**The versioning rule is refined at the same time**, because D-002 as written made this a major
|
|
189
|
+
bump. That was wrong, and worth fixing rather than working around: what breaks a consumer is
|
|
190
|
+
something being **removed, renamed or narrowed**, not something being added. A status nobody
|
|
191
|
+
uses cannot break them. So major covers removal, rename and narrowing; minor covers additions.
|
|
192
|
+
This change is 1.1.0.
|
|
193
|
+
|
|
194
|
+
**Not weakened to avoid a bump.** The distinction is what semantic versioning actually means,
|
|
195
|
+
and the previous wording conflated "the vocabulary changed" with "consumers must act".
|
|
196
|
+
|
|
197
|
+
---
|
|
198
|
+
|
|
199
|
+
## D-007. Paths in the emitted artefact are POSIX, whatever platform produced them
|
|
200
|
+
|
|
201
|
+
**Date:** 2026-08-11
|
|
202
|
+
**Status:** decided
|
|
203
|
+
**Standard version:** 1.1.2
|
|
204
|
+
|
|
205
|
+
`scanProject` recorded each scenario's `source` as `relative(root, file)`, which returns the
|
|
206
|
+
host's own separator. That string does not stay inside the process: it is written into the
|
|
207
|
+
generated artefact that consumers commit, and `generate --check` compares that artefact byte for
|
|
208
|
+
byte.
|
|
209
|
+
|
|
210
|
+
Left native, the same repository with the same documents emits `documentation\ui-qa\x.md` on
|
|
211
|
+
Windows and `documentation/ui-qa/x.md` on macOS. Each platform then fails the other's gate, and
|
|
212
|
+
the failure reports drift that does not exist. That is worse than a missed defect: a gate which
|
|
213
|
+
cries wolf is a gate people learn to skip, and this one is the mechanism holding the criteria and
|
|
214
|
+
the tests together.
|
|
215
|
+
|
|
216
|
+
Normalised at the single point where a path stops being a filesystem handle and becomes recorded
|
|
217
|
+
data. `isCriteriaPath` uses the same splitter, so a mixed-separator path cannot match in one
|
|
218
|
+
place and fail in the other. The video path shown in the review page is normalised too, for
|
|
219
|
+
consistency rather than correctness - it is display only.
|
|
220
|
+
|
|
221
|
+
**Splits on both separators rather than on the platform's `sep`.** Windows accepts forward
|
|
222
|
+
slashes as well, so a mixed path is reachable there, and a `sep`-only fix would also be untestable
|
|
223
|
+
on a POSIX machine: the test would pass whether or not the normalisation existed. The three tests
|
|
224
|
+
guarding this assert against a literal backslash and were proved red by making `toPosixPath` the
|
|
225
|
+
identity function.
|
|
226
|
+
|
|
227
|
+
**Versioning.** The rule had no category for a defect fix that changes emitted content without
|
|
228
|
+
changing its shape, so patch now says so explicitly rather than leaving it to be inferred.
|
|
229
|
+
Consumers regenerate and commit; nothing else changes. On a POSIX machine the emitted bytes are
|
|
230
|
+
identical, confirmed against both consumers.
|
|
231
|
+
|
|
232
|
+
---
|
|
233
|
+
|
|
234
|
+
## D-008. Releases are tagged, so a consumer can pin one
|
|
235
|
+
|
|
236
|
+
**Date:** 2026-08-11
|
|
237
|
+
**Status:** decided
|
|
238
|
+
|
|
239
|
+
Two repositories now depend on this tool, and one of them wires it into a verification gate. A
|
|
240
|
+
consumer that tracks whatever is on the default branch inherits every change the moment it lands,
|
|
241
|
+
including a change to the emitted shape, and discovers it as a broken build with no version to name
|
|
242
|
+
in the report.
|
|
243
|
+
|
|
244
|
+
Every release is therefore an annotated tag, `v<package version>`, pushed with the commit. The tag
|
|
245
|
+
message records the **standard version** that release ships, because those are two different numbers
|
|
246
|
+
answering two different questions: the package version is what you pin and download, and the
|
|
247
|
+
standard version is what your criteria conform to.
|
|
248
|
+
|
|
249
|
+
```
|
|
250
|
+
git tag -a v0.2.0 -m "criteria-review 0.2.0 - standard 1.1.2 ..."
|
|
251
|
+
git push origin main --tags
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
**Alternative weighed.** *Tag only the standard version.* Rejected: a consumer pins an artefact, not
|
|
255
|
+
a specification, and two releases can ship the same standard. The tag has to name the thing being
|
|
256
|
+
downloaded.
|
|
257
|
+
|
|
258
|
+
**Consequence.** Bumping the package version is part of cutting a release rather than an
|
|
259
|
+
afterthought, and a release note that does not name its standard version is incomplete.
|
|
260
|
+
|
|
261
|
+
---
|
|
262
|
+
|
|
263
|
+
## D-009. Project settings and developer settings are different files
|
|
264
|
+
|
|
265
|
+
**Date:** 2026-08-12
|
|
266
|
+
**Status:** decided
|
|
267
|
+
|
|
268
|
+
`criteria.json` is committed and describes the project. `criteria.local.json` is gitignored and
|
|
269
|
+
describes one machine. Precedence is flags, local, project.
|
|
270
|
+
|
|
271
|
+
The property that matters: **a pipeline sets nothing**, so its behaviour is whatever is committed
|
|
272
|
+
and is therefore reviewable in a pull request rather than configured somewhere nobody reads.
|
|
273
|
+
|
|
274
|
+
**A local file may not weaken a gate.** It may change where a developer's own output lands
|
|
275
|
+
(`videoDir`, `publish.target`) and what subset they walk (`since`, `limit`). It may not redirect
|
|
276
|
+
`emit.out`, because the artefact is committed and `--check` compares that exact path, so an
|
|
277
|
+
override would have one machine checking a file nobody else has - passing locally while the build
|
|
278
|
+
fails. It may not set `standard`, because two people on one project reading different rules is a
|
|
279
|
+
fork rather than a preference.
|
|
280
|
+
|
|
281
|
+
Keys outside the allowlist are **refused by name**, not ignored. A setting that silently does
|
|
282
|
+
nothing leaves its author believing it took effect, which is the worse of the two failures.
|
|
283
|
+
|
|
284
|
+
**Discovery stays out of configuration.** Criteria are still found by convention, because a
|
|
285
|
+
missing config file would mean silent zero coverage. Everything settable here fails loudly at the
|
|
286
|
+
moment it is asked for, which is what makes it safe to declare.
|
|
287
|
+
|
|
288
|
+
---
|
|
289
|
+
|
|
290
|
+
## D-010. A team can own the standard; they cannot own the contract
|
|
291
|
+
|
|
292
|
+
**Date:** 2026-08-12
|
|
293
|
+
**Status:** decided
|
|
294
|
+
|
|
295
|
+
The shipped standard is one opinion, and this tool is now public. A team adopting it must be able
|
|
296
|
+
to make the rules theirs: `criteria-review standard eject <dir>` copies the documents into the
|
|
297
|
+
project, and `criteria.json` points at them. Their copy is what the reading tab shows.
|
|
298
|
+
|
|
299
|
+
The shipped standard stays visible beside it as a labelled reference unless they turn it off.
|
|
300
|
+
That is the answer to the mirror problem D-001 exists to prevent: a fork that hides its original
|
|
301
|
+
cannot tell when the original moved, and keeping it on screen makes divergence an observation
|
|
302
|
+
rather than a discovery on some later upgrade.
|
|
303
|
+
|
|
304
|
+
**What cannot be forked by editing prose:** the status vocabulary, the tag grammar, and the
|
|
305
|
+
emitted shape. Those are enforced in code and versioned. A copy that disagrees with them is wrong
|
|
306
|
+
rather than authoritative, and the eject says so, because otherwise the first thing a new adopter
|
|
307
|
+
does is edit the status list and then file a bug when nothing changes.
|
|
308
|
+
|
|
309
|
+
A team that genuinely needs different statuses needs a change to the tool, which should come back
|
|
310
|
+
as a pull request rather than diverge silently in a copy.
|
|
@@ -0,0 +1,170 @@
|
|
|
1
|
+
# Getting started
|
|
2
|
+
|
|
3
|
+
A walkthrough of what this tool is for and how to work it. Ten minutes end to end. The rules it
|
|
4
|
+
enforces are in **The standard**, in the list beside this; this page is how the machinery works.
|
|
5
|
+
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## What problem it solves
|
|
9
|
+
|
|
10
|
+
Acceptance criteria written up from software that already exists describe what the software
|
|
11
|
+
**appears** to do. Until a person confirms that is what it **should** do, a test citing them
|
|
12
|
+
proves the test runs, not that the requirement is right.
|
|
13
|
+
|
|
14
|
+
That gap is where defects hide. This tool makes it visible and cheap to close: every scenario
|
|
15
|
+
carries a status, the queue shows you only the unsettled ones, and moving one along is a
|
|
16
|
+
keystroke.
|
|
17
|
+
|
|
18
|
+
## The loop, in one picture
|
|
19
|
+
|
|
20
|
+
```
|
|
21
|
+
someone writes criteria -> @status:derived
|
|
22
|
+
you read one here -> press v when you have watched it happen
|
|
23
|
+
you confirm the intent -> press a
|
|
24
|
+
it drops out of the queue for good
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
Everything else in the tool exists to make one of those three steps cheaper.
|
|
28
|
+
|
|
29
|
+
---
|
|
30
|
+
|
|
31
|
+
## 1. Point it at a project
|
|
32
|
+
|
|
33
|
+
From inside any working tree:
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
criteria-review here
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
That registers the tree, starts the server if it is down, and opens the page on this project.
|
|
40
|
+
One verb rather than three, and safe to run again, so it is the command to type on entering a
|
|
41
|
+
repository without checking anything first.
|
|
42
|
+
|
|
43
|
+
Give it a name when the directory name is not distinctive:
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
criteria-review here . playwright-worktree
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Several trees can be registered at once, **including two checkouts of the same repository**. A
|
|
50
|
+
worktree and its main clone appear separately with separate counts, which is how you see what a
|
|
51
|
+
branch adds.
|
|
52
|
+
|
|
53
|
+
Nothing found? Criteria live in fenced ` ```gherkin ` blocks in any `.md` under an
|
|
54
|
+
`acceptance/` directory (`ui-qa/` also works, as a legacy alias).
|
|
55
|
+
|
|
56
|
+
## 2. Work the queue
|
|
57
|
+
|
|
58
|
+
The keyboard drives it. The bar along the bottom of the page is the whole cheat sheet:
|
|
59
|
+
|
|
60
|
+
| Key | Does |
|
|
61
|
+
|---|---|
|
|
62
|
+
| `j` / `k` | next / previous scenario |
|
|
63
|
+
| `a` | accept: this is what the software should do |
|
|
64
|
+
| `v` | verified: you watched it happen |
|
|
65
|
+
| `n` | write a note on it |
|
|
66
|
+
| `f` | fullscreen the recording |
|
|
67
|
+
| `r` | reload from disk |
|
|
68
|
+
| `/` | jump to the filter box |
|
|
69
|
+
| `s` | switch between the queue and this reading tab |
|
|
70
|
+
|
|
71
|
+
**Every action writes straight into the source markdown**, next to the scenario it is about.
|
|
72
|
+
There is no database. Your notes are sourced intent, which is the one thing that cannot be
|
|
73
|
+
reconstructed from the code later, so they belong in git beside what they annotate.
|
|
74
|
+
|
|
75
|
+
Statuses go up the ladder as you confirm more: `proposed` (written before the software existed),
|
|
76
|
+
`derived` (written up from what was built), `verified` (you watched it), `accepted` (confirmed as
|
|
77
|
+
what it should do). Accepted items leave the queue, so what stays on screen is exactly the
|
|
78
|
+
unsettled surface.
|
|
79
|
+
|
|
80
|
+
## 3. Watch, do not just read
|
|
81
|
+
|
|
82
|
+
Reading a scenario tells you the test says what the scenario says. Only watching the software
|
|
83
|
+
tells you the scenario describes something that can actually happen - and that is the failure a
|
|
84
|
+
side-by-side comparison cannot catch, because a scenario and its test can agree perfectly and
|
|
85
|
+
both be wrong.
|
|
86
|
+
|
|
87
|
+
Recordings are found by name:
|
|
88
|
+
|
|
89
|
+
```
|
|
90
|
+
<project>/qa/videos/<SCENARIO-ID>.webm
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
If no file is named for the scenario, the tool falls back to searching test output for a path
|
|
94
|
+
containing the id, and labels the result **matched, not named**. Treat that as a guess: accepting
|
|
95
|
+
a scenario on the strength of a video means trusting the video shows *that* scenario.
|
|
96
|
+
|
|
97
|
+
Filter by `has recording` / `no recording` to find what is unevidenced.
|
|
98
|
+
|
|
99
|
+
---
|
|
100
|
+
|
|
101
|
+
## 4. Talking to whoever owns the intent
|
|
102
|
+
|
|
103
|
+
Two flags, and they are not interchangeable. One flag could not say **whose** attention is owed,
|
|
104
|
+
and both sides would keep picking up the other's items.
|
|
105
|
+
|
|
106
|
+
| Flag | Raised by | Means |
|
|
107
|
+
|---|---|---|
|
|
108
|
+
| `@looknow` | an agent or a working session | the architect must look at this |
|
|
109
|
+
| `@review` | the architect saving a note | an agent must act on what they wrote |
|
|
110
|
+
|
|
111
|
+
The loop closes like this, and each step hands the item to exactly one side:
|
|
112
|
+
|
|
113
|
+
1. A session asks a question beside the scenario it is about:
|
|
114
|
+
```bash
|
|
115
|
+
criteria-review ask CO-TYPES-006 myproject --message "Blank cell, or the word none?" --as claude
|
|
116
|
+
```
|
|
117
|
+
That writes the question as a note and raises `@looknow`.
|
|
118
|
+
2. You answer in the same pane by writing a note. That raises `@review` and clears `@looknow`.
|
|
119
|
+
3. The session reads it with `criteria-review notes`, acts, and runs
|
|
120
|
+
`criteria-review handled CO-TYPES-006`, which removes the discussion and raises `@looknow` so
|
|
121
|
+
the item comes back to you for re-review.
|
|
122
|
+
|
|
123
|
+
**Acting on a flagged scenario clears the flag.** Accepting it, marking it verified, sending it
|
|
124
|
+
back, writing a note - each one is you looking, so none of them leaves a badge behind demanding
|
|
125
|
+
attention you have already given.
|
|
126
|
+
|
|
127
|
+
There is also `criteria-review push`, which changes what an open page is showing and never
|
|
128
|
+
touches disk. Use it when someone is at the screen; use a flag when the finding should wait for
|
|
129
|
+
the next review pass.
|
|
130
|
+
|
|
131
|
+
## 5. Generating the scenarios your tests cite
|
|
132
|
+
|
|
133
|
+
The last step, and the one that stops the criteria and the tests drifting apart:
|
|
134
|
+
|
|
135
|
+
```bash
|
|
136
|
+
criteria-review generate . --out tests/e2e/support/scenarios.generated.ts
|
|
137
|
+
criteria-review generate . --out tests/e2e/support/scenarios.generated.ts --check
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
A journey's step names are supposed to be the scenario's own Given/When/Then wording. Typing
|
|
141
|
+
those clauses into the test achieves that on the day it is written and leaves two copies with
|
|
142
|
+
nothing detecting an edit to either. Generating them instead means a reworded clause, a skipped
|
|
143
|
+
clause, an invented step and a renamed id are all **compile errors**.
|
|
144
|
+
|
|
145
|
+
Write and `--check` are separate commands on purpose: a gate that silently regenerated would hide
|
|
146
|
+
exactly the drift it exists to catch.
|
|
147
|
+
|
|
148
|
+
Full detail in **The emit contract**, and the step-by-step in **Adopting the standard**.
|
|
149
|
+
|
|
150
|
+
---
|
|
151
|
+
|
|
152
|
+
## Housekeeping
|
|
153
|
+
|
|
154
|
+
```bash
|
|
155
|
+
criteria-review status # exits 1 when down, 0 when up
|
|
156
|
+
criteria-review list # a summary in the terminal, touches nothing
|
|
157
|
+
criteria-review projects # what is registered, and where
|
|
158
|
+
criteria-review stop | restart
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
The server exits on its own after two hours with no page open, so there is nothing to remember to
|
|
162
|
+
shut down. Starting it again is one command and costs nothing.
|
|
163
|
+
|
|
164
|
+
## Where to go next
|
|
165
|
+
|
|
166
|
+
- **The standard**, in the list beside this, if you are writing criteria rather than reviewing
|
|
167
|
+
them. Start with its README.
|
|
168
|
+
- **Writing acceptance criteria** for the format itself: ids, tags, provenance, and what does not
|
|
169
|
+
belong in the business tier.
|
|
170
|
+
- **Playwright journeys** for how a scenario becomes an automated journey that cites it.
|