criteria-review 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +299 -0
- package/agents/README.md +123 -0
- package/agents/skills/claude-code/criteria-lookup/SKILL.md +236 -0
- package/bin/criteria-review.js +870 -0
- package/docs/decisions.md +310 -0
- package/docs/help/01-getting-started.md +170 -0
- package/docs/standard/00-roles-and-ownership.md +353 -0
- package/docs/standard/01-qa-approach.md +273 -0
- package/docs/standard/02-writing-acceptance-criteria.md +187 -0
- package/docs/standard/03-example-acceptance-document.md +244 -0
- package/docs/standard/04-playwright-journeys.md +329 -0
- package/docs/standard/05-example-journey.spec.ts +130 -0
- package/docs/standard/06-engineering-obligations.md +170 -0
- package/docs/standard/07-manual-qa-and-defects.md +138 -0
- package/docs/standard/08-definition-of-done.md +128 -0
- package/docs/standard/09-recording-journeys.md +644 -0
- package/docs/standard/README.md +70 -0
- package/docs/standard/adopting-the-standard.md +104 -0
- package/docs/standard/emit-contract.md +175 -0
- package/package.json +41 -0
- package/public/app.js +791 -0
- package/public/doc-name.js +35 -0
- package/public/index.html +100 -0
- package/public/markdown.js +0 -0
- package/public/queue-order.js +78 -0
- package/public/styles.css +414 -0
- package/src/config.js +104 -0
- package/src/emit.js +183 -0
- package/src/media.js +126 -0
- package/src/parse.js +298 -0
- package/src/scan.js +158 -0
- package/src/server.js +433 -0
- package/src/standard.js +174 -0
- package/src/version.js +19 -0
- package/src/videos.js +185 -0
- package/src/write.js +307 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Kyley Harris
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,299 @@
|
|
|
1
|
+
# criteria-review
|
|
2
|
+
|
|
3
|
+
Review acceptance criteria across projects. Surfaces the scenarios that still need a human
|
|
4
|
+
decision, lets you annotate them, and writes the outcome back into the source documents.
|
|
5
|
+
|
|
6
|
+
## Why
|
|
7
|
+
|
|
8
|
+
Acceptance criteria written up from delivered software describe what the software *appears* to
|
|
9
|
+
do. Until someone confirms that is what it *should* do, a test citing them proves the test
|
|
10
|
+
runs, not that the requirement is right. The gap between those two states is where defects
|
|
11
|
+
hide, and making it visible is most of the value.
|
|
12
|
+
|
|
13
|
+
That only pays off if reviewing is cheap. This tool exists so the review is a queue you can
|
|
14
|
+
work through rather than a repo you have to go reading.
|
|
15
|
+
|
|
16
|
+
## Two roles
|
|
17
|
+
|
|
18
|
+
The tool assumes two participants and names them throughout:
|
|
19
|
+
|
|
20
|
+
- **The architect** is the person who decides what the software *should* do. Only they can
|
|
21
|
+
accept a scenario, and only their own actions clear a request for their attention.
|
|
22
|
+
- **An agent** is a working session, human or AI, that writes criteria up from the software,
|
|
23
|
+
runs the journeys, and acts on what the architect decides.
|
|
24
|
+
|
|
25
|
+
The distinction is load-bearing rather than decorative. The two attention flags route an item
|
|
26
|
+
by which side raised it, so it always sits with exactly one of them and neither keeps picking
|
|
27
|
+
up the other's work.
|
|
28
|
+
|
|
29
|
+
## The standard
|
|
30
|
+
|
|
31
|
+
**This repository owns the acceptance-criteria standard.** It lives in
|
|
32
|
+
[`docs/standard/`](docs/standard/) and defines how criteria are written, confirmed, cited by
|
|
33
|
+
tests and gated. Consumer projects conform to it and hold pointers rather than restatements; a
|
|
34
|
+
mirror cannot know its original changed, so it rots into a contradiction. See
|
|
35
|
+
[`docs/decisions.md`](docs/decisions.md) D-001.
|
|
36
|
+
|
|
37
|
+
The test for where a sentence lives: **does it stay true if the product is swapped for a
|
|
38
|
+
different one?** If yes it belongs to the standard; if no it stays with the project.
|
|
39
|
+
|
|
40
|
+
## The format
|
|
41
|
+
|
|
42
|
+
````markdown
|
|
43
|
+
```gherkin
|
|
44
|
+
@ONB-ADMIN-001 @status:verified @verified:2026-08-01 @commit:a1b2c3d4 @persona:Administrator
|
|
45
|
+
Scenario: The first administrator must set their own password before using the system
|
|
46
|
+
Given a freshly installed system with no data beyond the bootstrap administrator
|
|
47
|
+
When the administrator signs in with the password the system shipped with
|
|
48
|
+
Then they are required to set a new password before reaching the application
|
|
49
|
+
```
|
|
50
|
+
````
|
|
51
|
+
|
|
52
|
+
| Tag | Meaning |
|
|
53
|
+
| --- | --- |
|
|
54
|
+
| `@<AREA>-<THING>-<NNN>` | Stable ID. The join between the document, the journey that proves it, and the API tests that cover it. |
|
|
55
|
+
| `@status:proposed` | Written at planning, before the software exists. A proposal, not a description. |
|
|
56
|
+
| `@status:derived` | Written up from delivered software. Describes what it appears to do. |
|
|
57
|
+
| `@status:verified` | A human watched the software do it. Carries `@verified:` and `@commit:`. |
|
|
58
|
+
| `@status:accepted` | Confirmed as what the software *should* do. |
|
|
59
|
+
| `@persona:<actor>` | Whose point of view the scenario is written from. |
|
|
60
|
+
|
|
61
|
+
An optional provenance comment records where the intent came from, which is what makes a review
|
|
62
|
+
pass affordable - attention goes to the inferred ones instead of spreading evenly:
|
|
63
|
+
|
|
64
|
+
```markdown
|
|
65
|
+
<!-- intent: docs/pos-architecture.md:214 -->
|
|
66
|
+
<!-- intent: INFERRED from implementation - needs confirmation -->
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Scenarios with **no ID or no status still parse** and are reported as `untracked`. They are the
|
|
70
|
+
backlog, and dropping them would hide it.
|
|
71
|
+
|
|
72
|
+
## Recordings
|
|
73
|
+
|
|
74
|
+
Reading the criteria tells you the test says what the scenario says. Only watching the software
|
|
75
|
+
tells you the scenario describes something that can actually happen - and that is the failure a
|
|
76
|
+
side-by-side comparison cannot catch, because a scenario and its test can agree perfectly and
|
|
77
|
+
both be wrong.
|
|
78
|
+
|
|
79
|
+
**The convention: one file per scenario, named for it, in a fixed directory. A new run replaces
|
|
80
|
+
the old file.**
|
|
81
|
+
|
|
82
|
+
```
|
|
83
|
+
<project>/qa/videos/<SCENARIO-ID>.webm
|
|
84
|
+
<project>/qa/videos/LOCK-UNLOCK-001.webm
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
The producing side owns this. Playwright's own output is date- and hash-shaped
|
|
88
|
+
(`test-results/<sanitised-title>/video.webm`), which is fine for a CI artefact and wrong for
|
|
89
|
+
something a person opens repeatedly: the path changes every run, old recordings pile up, and
|
|
90
|
+
"the video for this scenario" stops being a question with one answer. A reel step should move
|
|
91
|
+
its recording to the fixed name when it finishes.
|
|
92
|
+
|
|
93
|
+
The directory sits **inside the project**, which is what ties a recording to its workspace: a
|
|
94
|
+
worktree filming its own branch writes into its own tree, and two worktrees cannot overwrite
|
|
95
|
+
each other's footage. Recordings are large binaries and belong in `.gitignore`.
|
|
96
|
+
|
|
97
|
+
Override the location per project in the config:
|
|
98
|
+
|
|
99
|
+
```json
|
|
100
|
+
{ "projects": [{ "name": "storefront", "path": "...", "videoDir": "ui/qa-videos" }] }
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
If no named file exists, the tool falls back to scanning `test-results/` and friends for a path
|
|
104
|
+
containing the scenario ID, and labels the result **matched, not named**. That keeps it useful
|
|
105
|
+
against suites that have not adopted the convention, while making clear that a fuzzy match is a
|
|
106
|
+
guess rather than evidence.
|
|
107
|
+
|
|
108
|
+
Filter the queue by `has recording`, `recording (named)`, or `no recording`. The player supports
|
|
109
|
+
seeking and fullscreen (`f`).
|
|
110
|
+
|
|
111
|
+
## Install
|
|
112
|
+
|
|
113
|
+
No dependencies and no build step. Node 20+.
|
|
114
|
+
|
|
115
|
+
```bash
|
|
116
|
+
git clone https://github.com/KyleyHarris/criteria-review.git
|
|
117
|
+
cd criteria-review
|
|
118
|
+
npm link # puts `criteria-review` on your PATH
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
## Use
|
|
122
|
+
|
|
123
|
+
From whatever tree you are working in:
|
|
124
|
+
|
|
125
|
+
```bash
|
|
126
|
+
criteria-review here
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
Registers this tree, starts the server if it is down, and shows this project. One
|
|
130
|
+
verb rather than three, and idempotent, so it is safe to run on entering a repo.
|
|
131
|
+
|
|
132
|
+
```bash
|
|
133
|
+
criteria-review here . playwright-worktree # name it when the directory is not distinctive
|
|
134
|
+
criteria-review status # exits 1 when down
|
|
135
|
+
criteria-review stop | restart
|
|
136
|
+
criteria-review list # terminal summary, touches nothing
|
|
137
|
+
criteria-review projects | add | remove
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
Several trees can be registered at once, **including two checkouts of the same
|
|
141
|
+
repo**. A worktree and its main clone are separate entries with separate counts,
|
|
142
|
+
which is how you see what a branch adds. They share one master video library,
|
|
143
|
+
correctly, because the recordings are of the same product.
|
|
144
|
+
|
|
145
|
+
### Lifetime
|
|
146
|
+
|
|
147
|
+
The server exits on its own after `--idle` minutes with no page open (default 120,
|
|
148
|
+
`0` disables). Tying it to an editor or agent session would be worse: several
|
|
149
|
+
sessions and tabs can be open at once, so the first to end would kill a server
|
|
150
|
+
another is using, and reference-counting sessions leaks a count on every abnormal
|
|
151
|
+
exit. Use is the honest signal, and restarting is cheap.
|
|
152
|
+
|
|
153
|
+
### Driving it from a session
|
|
154
|
+
|
|
155
|
+
```bash
|
|
156
|
+
criteria-review push --message "..." --filter status=proposed \
|
|
157
|
+
--highlight A,B --focus A --in <project>
|
|
158
|
+
criteria-review focus <ID> [project]
|
|
159
|
+
criteria-review refresh
|
|
160
|
+
criteria-review flag <ID> [project] | unflag <ID> [project]
|
|
161
|
+
criteria-review ask <ID> [project] --message "..." [--as name]
|
|
162
|
+
criteria-review notes # what the architect has written back
|
|
163
|
+
criteria-review handled <ID> [project] # acted on it; hand the item back
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
`ask` writes the question as a note beside the scenario and raises `@looknow`, so the
|
|
167
|
+
architect answers where the thing being asked about is, rather than in a chat window
|
|
168
|
+
somewhere else. `flag` says "look at this" and carries no words; `push` carries words
|
|
169
|
+
and evaporates on reload. Neither can ask a question that is still there an hour later.
|
|
170
|
+
|
|
171
|
+
Push is transient - it changes what a page shows, never disk. `@looknow` is the
|
|
172
|
+
durable channel, because it lives in the document and survives a reload.
|
|
173
|
+
|
|
174
|
+
`notes` is the only channel that runs server to session, and it is a pull: push
|
|
175
|
+
reaches a browser, not an agent, so without it a session could discover the
|
|
176
|
+
architect's feedback only by accident. Run it when you enter a repo. It prints every
|
|
177
|
+
outstanding note and every scenario tagged `@review`.
|
|
178
|
+
|
|
179
|
+
Projects are registered once in `~/.config/criteria-review/config.json`, so the command works
|
|
180
|
+
from any directory. With nothing registered it falls back to the current directory.
|
|
181
|
+
|
|
182
|
+
Criteria are found by convention: any `.md` file under an `acceptance/` directory (or `ui-qa/`,
|
|
183
|
+
a recognised legacy alias). `README.md` files in those directories are skipped, because they
|
|
184
|
+
document the format and their examples are not real criteria.
|
|
185
|
+
|
|
186
|
+
### Generating scenarios for a test suite to cite
|
|
187
|
+
|
|
188
|
+
```bash
|
|
189
|
+
criteria-review generate . --out tests/e2e/support/scenarios.generated.ts
|
|
190
|
+
criteria-review generate . --out tests/e2e/support/scenarios.generated.ts --check
|
|
191
|
+
criteria-review generate . --out build/scenarios.json --format json
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
A journey's step names must be the scenario's own Given/When/Then wording. Typing those clauses
|
|
195
|
+
into the test achieves that on the day it is written and makes two copies with nothing detecting
|
|
196
|
+
an edit to either. Generating them instead means a reworded clause, a skipped clause, an invented
|
|
197
|
+
step and a renamed id are all **compile errors** on the consumer's side rather than silent
|
|
198
|
+
divergence.
|
|
199
|
+
|
|
200
|
+
Write and `--check` are separate on purpose: a gate that silently regenerated would hide exactly
|
|
201
|
+
the drift it exists to catch. The generated file is committed, so a consumer builds and tests
|
|
202
|
+
without this tool present.
|
|
203
|
+
|
|
204
|
+
Full contract: [`docs/standard/emit-contract.md`](docs/standard/emit-contract.md).
|
|
205
|
+
|
|
206
|
+
### In the UI
|
|
207
|
+
|
|
208
|
+
`j`/`k` move through the queue, `a` accepts, `v` marks verified, `n` adds a note, `f` fullscreens
|
|
209
|
+
the recording, `r` reloads, `/` filters. Every action writes straight to the source markdown.
|
|
210
|
+
|
|
211
|
+
Notes open a multi-line editor rather than a one-line box, because the architect is discussing an
|
|
212
|
+
item, not labelling it. They are rendered back in the detail pane with their author and date, and
|
|
213
|
+
"Mark handled" retires them once the discussion is closed. A note that is written and never shown
|
|
214
|
+
again is indistinguishable from no note.
|
|
215
|
+
|
|
216
|
+
The page updates itself when the documents change on disk. That is not for its own writes, which
|
|
217
|
+
already refresh the page that made them: it is for a DIFFERENT session editing the same files. An
|
|
218
|
+
agent working in a repo can write `@looknow` on a scenario and it appears in the open queue,
|
|
219
|
+
without the reviewer knowing to press reload. A flag nobody sees is the same as no flag.
|
|
220
|
+
|
|
221
|
+
Server-sent events, watching only the directories that hold criteria. The selection is preserved
|
|
222
|
+
by scenario identity rather than position, because an external edit can reorder the queue and
|
|
223
|
+
restoring by position would move the reviewer onto a different scenario than the one they were
|
|
224
|
+
reading.
|
|
225
|
+
|
|
226
|
+
### Attention, in both directions
|
|
227
|
+
|
|
228
|
+
Two flags, because one cannot say *whose* attention is owed and each side would keep picking up
|
|
229
|
+
the other's items:
|
|
230
|
+
|
|
231
|
+
| Flag | Raised by | Means |
|
|
232
|
+
| --- | --- | --- |
|
|
233
|
+
| `@looknow` | an agent: the tag by hand, `criteria-review flag`, or asking a question | the architect must look at this |
|
|
234
|
+
| `@review` | the architect saving a note | an agent must act on what they wrote |
|
|
235
|
+
|
|
236
|
+
**A note is a message, and the author decides which way it travels.** The architect's note raises
|
|
237
|
+
`@review` and hands the item to an agent; an agent's note is a question and raises `@looknow`,
|
|
238
|
+
handing it back. Routing on the author rather than on the act of writing is what stops the loop
|
|
239
|
+
stalling: a flag that pointed at the side that wrote it would leave both believing the item was
|
|
240
|
+
with the other.
|
|
241
|
+
|
|
242
|
+
So the loop closes: an agent asks with `criteria-review ask`, which writes the question and raises
|
|
243
|
+
`@looknow`; the architect answers in the same pane, which raises `@review` and clears `@looknow`;
|
|
244
|
+
the agent reads it with `criteria-review notes`, acts, and runs `criteria-review handled <ID>`,
|
|
245
|
+
which removes the discussion and raises `@looknow` so the item returns for re-review. Each step
|
|
246
|
+
hands the item to exactly one side, and the tag line always says which.
|
|
247
|
+
|
|
248
|
+
Add a bare `@looknow` to a scenario's tag line:
|
|
249
|
+
|
|
250
|
+
```gherkin
|
|
251
|
+
@ONB-MEMBER-001 @status:derived @persona:Administrator @looknow
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
That is the whole mechanism. An agent already editing these documents needs no API. Flagged
|
|
255
|
+
scenarios jump the document grouping and show a LOOK NOW badge, because grouping optimises a
|
|
256
|
+
steady review pass and a flag means someone wants eyes on it before that pass arrives. Filter to
|
|
257
|
+
them with the status dropdown.
|
|
258
|
+
|
|
259
|
+
**Acting on a flagged scenario clears the flag.** Accepting it, marking it verified, sending it
|
|
260
|
+
back to derived, writing a note or clearing notes all answer the request, because each one is the
|
|
261
|
+
architect looking. A flag that had to be dismissed separately would survive the action that
|
|
262
|
+
answered it, and a queue that keeps presenting items already dealt with teaches you to ignore the
|
|
263
|
+
one badge meant to jump it. The button is still there for a scenario you want to clear without
|
|
264
|
+
otherwise touching. Only the architect's own actions clear it: an agent writing to the same
|
|
265
|
+
scenario cannot retire a flag it raised, which is what lets `handled` hand an item back.
|
|
266
|
+
|
|
267
|
+
## Writes
|
|
268
|
+
|
|
269
|
+
Review state lives in the criteria files, not in a sidecar database. Your notes are sourced
|
|
270
|
+
intent, which is the one thing that cannot be reverse-engineered later, so it belongs in git
|
|
271
|
+
next to the scenario it annotates.
|
|
272
|
+
|
|
273
|
+
Writes are surgical: one tag line, or one inserted comment, and every other byte left alone.
|
|
274
|
+
After writing, the file is re-parsed and the intended change asserted; a write that cannot
|
|
275
|
+
prove it did what it claimed restores the file and reports an error.
|
|
276
|
+
|
|
277
|
+
The server binds to loopback only. It reads and writes your documents and is not something to
|
|
278
|
+
expose on a network interface.
|
|
279
|
+
|
|
280
|
+
## Tests
|
|
281
|
+
|
|
282
|
+
```bash
|
|
283
|
+
npm test
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
45 tests over the parser, the write path and the clause generator, with no test dependencies
|
|
287
|
+
beyond Node's own runner.
|
|
288
|
+
|
|
289
|
+
The write path is the risky part, so its tests were proved able to fail rather than merely
|
|
290
|
+
written: mutating the scenario resolver to return the wrong scenario turns two of them red, and
|
|
291
|
+
restoring it returns the suite to green. A test never observed failing is an untested test.
|
|
292
|
+
|
|
293
|
+
## Licence
|
|
294
|
+
|
|
295
|
+
MIT. See [`LICENSE`](LICENSE).
|
|
296
|
+
|
|
297
|
+
The standard in [`docs/standard/`](docs/standard/) is covered by the same licence: use it,
|
|
298
|
+
adapt it, instantiate it for your own domain. What it deliberately does not carry is any
|
|
299
|
+
project's own instance data, which stays with that project.
|
package/agents/README.md
ADDED
|
@@ -0,0 +1,123 @@
|
|
|
1
|
+
# Agent instructions
|
|
2
|
+
|
|
3
|
+
Instruction sets for an AI assistant working with this tool, and ready-made packaging for the
|
|
4
|
+
harnesses that can load them directly.
|
|
5
|
+
|
|
6
|
+
**Take these into your own system.** They are written to be copied, not depended on: nothing here
|
|
7
|
+
imports anything, and an assistant that has read the protocol below can drive the tool from any
|
|
8
|
+
harness that can run a command.
|
|
9
|
+
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
## What an assistant needs to know
|
|
13
|
+
|
|
14
|
+
The whole protocol is four ideas. Everything in `skills/` is these, spelled out for one harness.
|
|
15
|
+
|
|
16
|
+
### 1. The queue is ordered, and the order is not yours to change
|
|
17
|
+
|
|
18
|
+
`criteria-review queue` returns what still needs a human decision, most important first: flagged
|
|
19
|
+
items, then grouped by document with the riskiest documents leading. The page uses the same
|
|
20
|
+
ordering from the same module. Reordering it in conversation leaves the architect looking at two
|
|
21
|
+
different "most important" and unable to tell which is right.
|
|
22
|
+
|
|
23
|
+
### 2. Show the scenario, do not summarise it
|
|
24
|
+
|
|
25
|
+
`criteria-review show <ID>` prints the title, status, persona, flags, intent, the Given/When/Then
|
|
26
|
+
clauses, and every note. Present all of it, including **an intent line that says it is missing**.
|
|
27
|
+
A scenario whose intent was never sourced is the most dangerous kind, because a test citing it can
|
|
28
|
+
never disagree with the software, and omitting the line makes it look fine.
|
|
29
|
+
|
|
30
|
+
A paraphrase is the reviewer confirming your rewrite rather than the criterion. The wording is the
|
|
31
|
+
requirement.
|
|
32
|
+
|
|
33
|
+
### 3. Their words are the note; their decision is the status
|
|
34
|
+
|
|
35
|
+
Prose in reply to a presented scenario is a note, written verbatim:
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
criteria-review note <ID> --message "<their words>"
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Status moves are separate, named, and never taken unasked:
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
criteria-review accept <ID> # what the software SHOULD do
|
|
45
|
+
criteria-review verify <ID> --commit <sha> # a person watched it happen
|
|
46
|
+
criteria-review reject <ID> # back to derived
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
`verify` resolves the commit from the project under review and reports which one it recorded, so a
|
|
50
|
+
verification is a word rather than a hash hunt. Never take it because a scenario looks plausible:
|
|
51
|
+
it asserts something about a person's own actions.
|
|
52
|
+
|
|
53
|
+
**When a scenario and the software disagree, the reviewer says which side is wrong.** That is the
|
|
54
|
+
case the standard exists to surface, and neither answer is a rejection. Put it to them as the WORK rather than as a verdict: **write the code to match this criterion**, or
|
|
55
|
+
**write the test to match the existing code**. The first keeps the criterion and `accept`s it -
|
|
56
|
+
`accepted` is a claim about intent, not implementation, so a journey citing it then goes red and
|
|
57
|
+
the defect surfaces in CI rather than in a list somebody has to remember. The second `reject`s it
|
|
58
|
+
back to `derived`, which is precisely what that status means: written up from the delivered
|
|
59
|
+
software. Both carry a note naming the work. Never pick between those two on their behalf; both artefacts are
|
|
60
|
+
self-consistent, so nothing you can read settles it.
|
|
61
|
+
|
|
62
|
+
**Checking a scenario against the source is evidence, not a verdict.** Report each `Then` clause as
|
|
63
|
+
supported, contradicted, or untraceable, with `file:line`, and say "read, not run" in those words -
|
|
64
|
+
citing a line proves a line was read, and only running the software proves the system does it.
|
|
65
|
+
Never promote a status off a code read: not `verified`, which claims a person watched it, and not
|
|
66
|
+
`accepted`, which is a claim about intent that the implementation has no standing to settle. A
|
|
67
|
+
finding worth keeping goes in as the agent's, via `ask`, which raises `@looknow` - it is a question
|
|
68
|
+
for the architect, not an instruction for an agent.
|
|
69
|
+
|
|
70
|
+
Write each answer **immediately**, not in a batch at the end. Review state lives in the documents,
|
|
71
|
+
not in a session that can end.
|
|
72
|
+
|
|
73
|
+
### 4. Two flags, and each hands the item to exactly one side
|
|
74
|
+
|
|
75
|
+
| Flag | Raised by | Means |
|
|
76
|
+
|---|---|---|
|
|
77
|
+
| `@looknow` | an agent, via `ask` or `flag` | the architect must look at this |
|
|
78
|
+
| `@review` | the architect writing a note | an agent must act on what they wrote |
|
|
79
|
+
|
|
80
|
+
An agent asks with `criteria-review ask <ID> --message "..."`, which puts the question beside the
|
|
81
|
+
scenario rather than in a transcript that scrolls away. It reads replies with
|
|
82
|
+
`criteria-review notes`, acts, then runs `criteria-review handled <ID>`, which clears the
|
|
83
|
+
discussion and hands the item back for re-review.
|
|
84
|
+
|
|
85
|
+
**Never clear `@review` by hand.** It is the architect's request, and `handled` is the only thing
|
|
86
|
+
entitled to retire it.
|
|
87
|
+
|
|
88
|
+
### And one standing rule
|
|
89
|
+
|
|
90
|
+
**A session that edits an acceptance document regenerates before running tests.** Otherwise the
|
|
91
|
+
suite is proving the previous requirement. See `docs/standard/emit-contract.md`.
|
|
92
|
+
|
|
93
|
+
---
|
|
94
|
+
|
|
95
|
+
## Absorbing the packaged skills
|
|
96
|
+
|
|
97
|
+
`skills/claude-code/` holds ready-made skills. Two ways in, and the difference matters:
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
# Copy: portable, and yours to edit. Updates to this repo do not reach it.
|
|
101
|
+
cp -R agents/skills/claude-code/criteria-lookup ~/.claude/skills/
|
|
102
|
+
|
|
103
|
+
# Link: stays current with the tool. Machine-specific, so it is not something to commit
|
|
104
|
+
# into a config repo that has to work on another machine.
|
|
105
|
+
ln -s "$PWD/agents/skills/claude-code/criteria-lookup" ~/.claude/skills/criteria-lookup
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
On a different harness, read the skill as prose and express it in whatever form that harness takes
|
|
109
|
+
- a system prompt, a tool description, a rule file. The commands are the contract; the packaging is
|
|
110
|
+
not.
|
|
111
|
+
|
|
112
|
+
## What is here
|
|
113
|
+
|
|
114
|
+
| Skill | For |
|
|
115
|
+
|---|---|
|
|
116
|
+
| [`skills/claude-code/criteria-lookup`](skills/claude-code/criteria-lookup/SKILL.md) | Looking a scenario up and answering it in conversation, and walking the queue one at a time |
|
|
117
|
+
|
|
118
|
+
## What deliberately is not here
|
|
119
|
+
|
|
120
|
+
**Anything about how a particular person likes to work.** When to interrupt them, how large a
|
|
121
|
+
review batch they want, which projects they have registered, where their repositories live. That
|
|
122
|
+
belongs in their own configuration, not in a tool other people use, or it reads as normative for
|
|
123
|
+
everyone who adopts this.
|
|
@@ -0,0 +1,236 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: criteria-lookup
|
|
3
|
+
description: Look up an acceptance scenario and answer it in conversation, without opening the review tool's page. Presents the scenario, its intent and any notes, takes your reply as the note, and offers accept / verify / send back. Also walks the whole queue one at a time, most important first, with continue or stop between each. Use when the architect says "look up <ID>", "what do the criteria say about X", "show me that scenario", "let's do a review pass", "walk the queue", "what needs my decision", or replies to a presented scenario with prose rather than a command.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Look up and answer criteria in conversation
|
|
7
|
+
|
|
8
|
+
The visual tool is the right surface for a long review pass beside the recordings. This is the
|
|
9
|
+
other one: a single question mid-task, or a walk through the queue by voice, where opening a page
|
|
10
|
+
and finding the scenario costs more than the answer is worth.
|
|
11
|
+
|
|
12
|
+
**Everything here writes straight into the source document.** No browser, no running server. An
|
|
13
|
+
open page updates itself, because the server watches the criteria directories.
|
|
14
|
+
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
## Looking one thing up
|
|
18
|
+
|
|
19
|
+
```bash
|
|
20
|
+
criteria-review show <ID> [project]
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
Present what it prints, in full, as prose the architect can read without decoding:
|
|
24
|
+
|
|
25
|
+
- the id, title and where it lives
|
|
26
|
+
- the status, persona and any flags
|
|
27
|
+
- **the intent line, always, including when it is missing** - print "not sourced" rather than
|
|
28
|
+
omitting it. An unsourced scenario is the most dangerous kind, and silence reads as fine
|
|
29
|
+
- the Given / When / Then clauses verbatim
|
|
30
|
+
- every note, with its author and date
|
|
31
|
+
|
|
32
|
+
Then offer the choices as a **numbered menu**, always in the same order, so answering is a
|
|
33
|
+
keypress rather than a sentence:
|
|
34
|
+
|
|
35
|
+
```
|
|
36
|
+
1 accept - this is what the software should do
|
|
37
|
+
2 verify - you watched it happen
|
|
38
|
+
3 reject - send it back to derived
|
|
39
|
+
4 code to match - write the code to match this criterion
|
|
40
|
+
5 test to match - write the test to match the existing code
|
|
41
|
+
6 check drift - read the implementation and report whether it matches
|
|
42
|
+
7 skip - leave it, move on
|
|
43
|
+
8 stop - end the pass
|
|
44
|
+
|
|
45
|
+
...or just type what is wrong, and that becomes the note.
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Keep the numbers stable between items. A menu whose options move is a menu that has to be read
|
|
49
|
+
every time, and the whole point is that it does not.
|
|
50
|
+
|
|
51
|
+
`1` may be combined with prose - "1, but the wording is loose" accepts it and writes the note.
|
|
52
|
+
|
|
53
|
+
## Their reply IS the note
|
|
54
|
+
|
|
55
|
+
When the architect answers with prose rather than a command, that prose is the note. Write it:
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
criteria-review note <ID> [project] --message "<their words>"
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
**Use their words, not a summary.** The note is sourced intent, which is the one thing that cannot
|
|
62
|
+
be reconstructed from the code later. Tidy obvious dictation artefacts and nothing else; if their
|
|
63
|
+
meaning is unclear, ask rather than paraphrase a guess into the document.
|
|
64
|
+
|
|
65
|
+
A note written this way raises `@review`, which says an agent must act on it. That is correct even
|
|
66
|
+
when you are about to act immediately: the flag is what survives the session ending.
|
|
67
|
+
|
|
68
|
+
## The three status moves
|
|
69
|
+
|
|
70
|
+
Offer them by name, and never take one without being told to.
|
|
71
|
+
|
|
72
|
+
| They say | Command | What it claims |
|
|
73
|
+
|---|---|---|
|
|
74
|
+
| `1`, accept, that is right | `criteria-review accept <ID>` | This is what the software SHOULD do |
|
|
75
|
+
| `2`, verify, I watched that | `criteria-review verify <ID>` | A person watched the software do it |
|
|
76
|
+
| `3`, reject, not right | `criteria-review reject <ID>` | Back to derived |
|
|
77
|
+
|
|
78
|
+
**Verify claims a person watched the software do this, at a named build.** The commit is resolved
|
|
79
|
+
automatically from the project being reviewed, so nobody has to go and find a hash - **say which
|
|
80
|
+
commit was recorded** when you report it back, and say so plainly if the tree was dirty, because
|
|
81
|
+
then the commit does not contain what was watched.
|
|
82
|
+
|
|
83
|
+
Never take verify because a scenario looks plausible. It is the one status that asserts something
|
|
84
|
+
about a person's own actions, and only they can say it happened.
|
|
85
|
+
|
|
86
|
+
Accepting and noting are independent: `--message` may be passed alongside a status move, and both
|
|
87
|
+
are recorded.
|
|
88
|
+
|
|
89
|
+
## 6, checking for drift against the source
|
|
90
|
+
|
|
91
|
+
Reading the implementation before deciding. Cheap, often decisive, and **evidence rather than a
|
|
92
|
+
verdict** - it is the one option here that produces a finding instead of a record.
|
|
93
|
+
|
|
94
|
+
Work clause by clause, not scenario by scenario. Each `Then` is an assertion, and the useful answer
|
|
95
|
+
is which of them the code supports:
|
|
96
|
+
|
|
97
|
+
1. Find the code that would produce the behaviour. Start from the actor's entry point, not from a
|
|
98
|
+
symbol whose name matches a word in the scenario.
|
|
99
|
+
2. For every `Then` and trailing `And`, report one of three things, with `file:line`:
|
|
100
|
+
- **supported** - the code does this, and here is where
|
|
101
|
+
- **contradicted** - the code does something else, and here is what
|
|
102
|
+
- **cannot tell from reading** - the path is dynamic, config-driven, or spread across services
|
|
103
|
+
3. Say plainly which it is. Do not average three supported and one contradicted into "mostly fine";
|
|
104
|
+
the contradicted clause is the finding.
|
|
105
|
+
|
|
106
|
+
Then re-present the menu, because the drift check informs the decision and does not make it.
|
|
107
|
+
|
|
108
|
+
**State what the evidence is worth, every time.** Citing `file:line` proves a line was read. Only
|
|
109
|
+
running the software proves the system does it. A clause that looks supported can still fail on a
|
|
110
|
+
guard three layers up, and a clause that looks contradicted can be handled somewhere you did not
|
|
111
|
+
read. Say "read, not run" in those words, so nobody mistakes it for verification.
|
|
112
|
+
|
|
113
|
+
**Never promote a status off a code read.** Not `verified`, which claims a person watched it, and
|
|
114
|
+
not `accepted`, which is a claim about intent that the implementation has no standing to settle.
|
|
115
|
+
The implementation is not a source of intent: it contains what the system does, never why anyone
|
|
116
|
+
wanted it.
|
|
117
|
+
|
|
118
|
+
If the finding is worth keeping - a real contradiction, or a clause nobody can trace - offer to
|
|
119
|
+
record it. It is the agent's finding, so it goes in as the agent:
|
|
120
|
+
|
|
121
|
+
```bash
|
|
122
|
+
criteria-review ask <ID> [project] --message "DRIFT: <clause> is contradicted by <file:line>: <what the code does>" --as <who>
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
That raises `@looknow` rather than `@review`, which is correct: the finding is a question for the
|
|
126
|
+
architect, not an instruction for an agent.
|
|
127
|
+
|
|
128
|
+
## When the scenario and the software disagree
|
|
129
|
+
|
|
130
|
+
This is the case the whole standard exists to surface, and the reviewer has to say **which side is
|
|
131
|
+
wrong**. Neither answer is "reject": one of them keeps the criterion exactly as written.
|
|
132
|
+
|
|
133
|
+
Say it as the WORK, not as a verdict. "Which artefact is wrong" is an argument; "write the code to
|
|
134
|
+
match this" is an instruction somebody can pick up.
|
|
135
|
+
|
|
136
|
+
**4, write the code to match this criterion.** The criterion stands, and the build has not caught
|
|
137
|
+
up. Accepting it is correct even though the software fails it, because `accepted` is a claim about
|
|
138
|
+
intent, not about the implementation:
|
|
139
|
+
|
|
140
|
+
```bash
|
|
141
|
+
criteria-review accept <ID> --message "WRITE THE CODE TO MATCH: <what the software does instead, in their words>"
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
Then say plainly how the gap stays visible: a journey citing this scenario now goes **red**, which
|
|
145
|
+
is the defect surfacing in CI rather than in a list somebody has to remember. If no journey cites
|
|
146
|
+
it yet, say so - the note and `@review` are the only thing carrying it, and writing the journey is
|
|
147
|
+
the work that makes it self-reporting.
|
|
148
|
+
|
|
149
|
+
**5, write the test to match the existing code.** The software is the truth here: the criterion was
|
|
150
|
+
inferred, or it describes an intention nobody holds. Send it back to `derived`, which is exactly
|
|
151
|
+
what that status means - written up from the delivered software, describing what it appears to do:
|
|
152
|
+
|
|
153
|
+
```bash
|
|
154
|
+
criteria-review reject <ID> --message "WRITE THE TEST TO MATCH THE CODE: <what the software actually does, in their words>"
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
Both raise `@review`, which hands the item to an agent. Report which one was recorded and what
|
|
158
|
+
happens next, in one line.
|
|
159
|
+
|
|
160
|
+
A drift check (6) is the natural thing to run first when you cannot tell which way it should go.
|
|
161
|
+
It settles what the code does; it never settles what the code should do.
|
|
162
|
+
|
|
163
|
+
**Do not decide this for them.** Which way a disagreement resolves is the only question in the
|
|
164
|
+
system that cannot be answered by reading either artefact, because both are self-consistent. The
|
|
165
|
+
code cannot tell you it was meant to be different, and the document cannot tell you it was
|
|
166
|
+
aspirational. Present the disagreement and ask.
|
|
167
|
+
|
|
168
|
+
**Neither option does the work now.** Both record it as a work order and hand the item on. Doing it
|
|
169
|
+
mid-pass is how a review of seventy-five scenarios becomes an afternoon on the first one.
|
|
170
|
+
|
|
171
|
+
---
|
|
172
|
+
|
|
173
|
+
## Walking the queue
|
|
174
|
+
|
|
175
|
+
```bash
|
|
176
|
+
criteria-review queue --limit 10
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
Ordered exactly as the page orders it: flagged items first, then grouped by document with the
|
|
180
|
+
riskiest documents leading. Do not reorder it yourself - the architect sees the same order on
|
|
181
|
+
screen, and two orderings would leave them unable to tell which is right.
|
|
182
|
+
|
|
183
|
+
Then, **one at a time**:
|
|
184
|
+
|
|
185
|
+
1. Present the scenario in full, as above.
|
|
186
|
+
2. Take their reply. Prose becomes the note; a status word becomes the status move.
|
|
187
|
+
3. **Apply it immediately**, before presenting the next one.
|
|
188
|
+
4. Say in one line what was recorded, then present the next.
|
|
189
|
+
|
|
190
|
+
Keep going until they stop you. Between items, "next", "continue" or a bare reply means carry on;
|
|
191
|
+
"stop", "that's enough" or "cancel" ends the pass. Offer that explicitly at the start, not after
|
|
192
|
+
they have already answered six.
|
|
193
|
+
|
|
194
|
+
**Write as you go, do not batch.** A pass that collected ten answers and wrote them at the end
|
|
195
|
+
loses all ten if the session ends, and the whole design puts review state in the documents rather
|
|
196
|
+
than in a session that can evaporate. Writing each one also means an open page updates as they go.
|
|
197
|
+
|
|
198
|
+
At the end, summarise: how many were answered, what each became, and how many remain in the queue.
|
|
199
|
+
|
|
200
|
+
## While walking, do not
|
|
201
|
+
|
|
202
|
+
- **Do not accept in bulk.** Every promotion is a person deciding one scenario. "Accept the rest"
|
|
203
|
+
is a request to stop and confirm what "the rest" contains, not an instruction to loop.
|
|
204
|
+
- **Do not skip an item silently.** If something cannot be presented, say which and why.
|
|
205
|
+
- **Do not summarise a scenario instead of showing it.** The wording is the requirement; a
|
|
206
|
+
paraphrase is the reviewer confirming your rewrite rather than the criterion.
|
|
207
|
+
- **Do not fix the software mid-pass.** A note recording what is wrong is the output. Acting on it
|
|
208
|
+
is separate work, after the pass, with the note in front of you.
|
|
209
|
+
|
|
210
|
+
---
|
|
211
|
+
|
|
212
|
+
## The other direction
|
|
213
|
+
|
|
214
|
+
If **you** need the architect's ruling on something and they are not in the conversation, the
|
|
215
|
+
question belongs beside the scenario rather than in a transcript that scrolls away:
|
|
216
|
+
|
|
217
|
+
```bash
|
|
218
|
+
criteria-review ask <ID> [project] --message "the question" --as <who>
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
That raises `@looknow` and puts the question in their queue. Read what came back with
|
|
222
|
+
`criteria-review notes`, act, then `criteria-review handled <ID>` to hand the item back for
|
|
223
|
+
re-review.
|
|
224
|
+
|
|
225
|
+
## Ambiguity
|
|
226
|
+
|
|
227
|
+
An id that exists in two registered projects is refused rather than guessed, naming both. Ask which
|
|
228
|
+
one. Writing an answer against a scenario the architect did not read is the failure that refusal
|
|
229
|
+
exists to prevent.
|
|
230
|
+
|
|
231
|
+
## Related
|
|
232
|
+
|
|
233
|
+
- `criteria-review guide` and the standard in `docs/standard/` for what the statuses mean and how
|
|
234
|
+
criteria are written.
|
|
235
|
+
- The visual tool (`criteria-review here`) for a long pass beside the recordings, which is still
|
|
236
|
+
the better surface when there is video to watch.
|