@assuredloop/orca-bot-kit 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +545 -3
- package/SETUP.md +270 -0
- package/package.json +32 -13
- package/rules/evidence.md +15 -0
- package/rules/finishing.md +20 -0
- package/rules/limits.md +18 -0
- package/rules/mail.md +17 -0
- package/rules/profiles.md +9 -0
- package/rules/review.md +14 -0
- package/rules/scope.md +20 -0
- package/rules/simple.md +14 -0
- package/rules/talk.md +13 -0
- package/rules/tests-first.md +15 -0
- package/rules/the-ask.md +19 -0
- package/skills/obk-arch/LICENSES.md +45 -0
- package/skills/obk-arch/NOTICE.md +134 -0
- package/skills/obk-arch/SKILL.md +387 -0
- package/skills/obk-bot-building/LICENSES.md +21 -0
- package/skills/obk-bot-building/NOTICE.md +113 -0
- package/skills/obk-bot-building/SKILL.md +235 -0
- package/skills/obk-debugging/LICENSES.md +67 -0
- package/skills/obk-debugging/NOTICE.md +115 -0
- package/skills/obk-debugging/SKILL.md +419 -0
- package/skills/obk-decision-memo/LICENSES.md +49 -0
- package/skills/obk-decision-memo/NOTICE.md +181 -0
- package/skills/obk-decision-memo/SKILL.md +241 -0
- package/skills/obk-finops/LICENSES.md +19 -0
- package/skills/obk-finops/NOTICE.md +77 -0
- package/skills/obk-finops/SKILL.md +146 -0
- package/skills/obk-fleet-review/LICENSES.md +52 -0
- package/skills/obk-fleet-review/NOTICE.md +104 -0
- package/skills/obk-fleet-review/SKILL.md +219 -0
- package/skills/obk-grilling/LICENSES.md +42 -0
- package/skills/obk-grilling/NOTICE.md +103 -0
- package/skills/obk-grilling/SKILL.md +217 -0
- package/skills/obk-grooming/LICENSES.md +52 -0
- package/skills/obk-grooming/NOTICE.md +111 -0
- package/skills/obk-grooming/SKILL.md +219 -0
- package/skills/obk-handoff/LICENSES.md +45 -0
- package/skills/obk-handoff/NOTICE.md +51 -0
- package/skills/obk-handoff/SKILL.md +119 -0
- package/skills/obk-personal-facilitation/LICENSES.md +53 -0
- package/skills/obk-personal-facilitation/NOTICE.md +62 -0
- package/skills/obk-personal-facilitation/SKILL.md +86 -0
- package/skills/obk-recall/LICENSES.md +42 -0
- package/skills/obk-recall/NOTICE.md +53 -0
- package/skills/obk-recall/SKILL.md +129 -0
- package/skills/obk-researching/LICENSES.md +67 -0
- package/skills/obk-researching/NOTICE.md +170 -0
- package/skills/obk-researching/SKILL.md +259 -0
- package/skills/obk-reviewing/LICENSES.md +66 -0
- package/skills/obk-reviewing/NOTICE.md +58 -0
- package/skills/obk-reviewing/SKILL.md +408 -0
- package/skills/obk-tdd/LICENSES.md +68 -0
- package/skills/obk-tdd/NOTICE.md +54 -0
- package/skills/obk-tdd/SKILL.md +516 -0
- package/skills/obk-tdd/mutation-tools.md +80 -0
- package/skills/obk-teach/LICENSES.md +42 -0
- package/skills/obk-teach/NOTICE.md +65 -0
- package/skills/obk-teach/SKILL.md +106 -0
- package/skills/obk-why/LICENSES.md +42 -0
- package/skills/obk-why/NOTICE.md +74 -0
- package/skills/obk-why/SKILL.md +169 -0
- package/skills/obk-writing/LICENSES.md +272 -0
- package/skills/obk-writing/NOTICE.md +161 -0
- package/skills/obk-writing/SKILL.md +254 -0
- package/src/book.js +268 -0
- package/src/bot.js +451 -0
- package/src/cli.js +1130 -0
- package/src/conversations.js +150 -0
- package/src/groom.js +157 -0
- package/src/health.js +309 -0
- package/src/hooks.js +202 -0
- package/src/init.js +314 -0
- package/src/launch.js +276 -0
- package/src/message.js +380 -0
- package/src/orca.js +291 -0
- package/src/pause.js +58 -0
- package/src/record.js +209 -0
- package/src/restart.js +192 -0
- package/src/retire.js +92 -0
- package/src/roster.js +71 -0
- package/src/rules.js +408 -0
- package/src/skills.js +629 -0
- package/src/sources.js +370 -0
- package/src/up.js +475 -0
- package/src/usage.js +401 -0
- package/bin/okb.js +0 -4
- package/src/index.js +0 -11
|
@@ -0,0 +1,516 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: obk-tdd
|
|
3
|
+
description: >-
|
|
4
|
+
Test-driven work: a failing test before the code, one vertical slice at a
|
|
5
|
+
time, through the public interface; a separate author for the tests, whose
|
|
6
|
+
tests the implementer does not edit; the shapes of tests that catch nothing;
|
|
7
|
+
and the hand mutation check that asks whether the tests would catch a real
|
|
8
|
+
mistake. Use whenever you are about to change what code does (a feature, a
|
|
9
|
+
fix, a one-line change, a bug someone reported) and whenever you are
|
|
10
|
+
writing, changing or judging tests, deciding what to test, or asking whether
|
|
11
|
+
a suite would catch anything. Reach for it before writing the code, not
|
|
12
|
+
after: a small change is exactly where the order gets skipped.
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
# Test-driven work
|
|
16
|
+
|
|
17
|
+
The order is the point: the check exists, and has been seen to fail, before the
|
|
18
|
+
code that satisfies it. What that buys is a test written against the
|
|
19
|
+
requirement rather than against the implementation you already had in mind.
|
|
20
|
+
|
|
21
|
+
Three things hold the tests honest, in this order of value: the author is
|
|
22
|
+
someone other than the implementer, a reviewer reads the tests, and the
|
|
23
|
+
implementer breaks its own logic on purpose to see whether a test notices.
|
|
24
|
+
|
|
25
|
+
These are defaults for work where nothing says otherwise. A user who asks for
|
|
26
|
+
something else gets what they asked for; say which of these you left and why.
|
|
27
|
+
|
|
28
|
+
## Find out how this project tests
|
|
29
|
+
|
|
30
|
+
The loop is the same everywhere; the commands are not. Before the first test,
|
|
31
|
+
learn how *this* project tests: the build file and whether it has a checked-in
|
|
32
|
+
wrapper to use instead of a globally installed tool, the framework and its
|
|
33
|
+
configuration, how it runs one focused test as against the whole suite, where
|
|
34
|
+
tests live and how they are named, what the tests next door already do, and
|
|
35
|
+
which commands the project's own checks actually run. Never assume a default
|
|
36
|
+
command because it is the usual one for the language.
|
|
37
|
+
|
|
38
|
+
Use the focused command through the loop and the whole-suite command before you
|
|
39
|
+
finish.
|
|
40
|
+
|
|
41
|
+
## Start from the requirement, not the code
|
|
42
|
+
|
|
43
|
+
Write the acceptance criteria before any test: one line each, each one
|
|
44
|
+
checkable on its own, each phrased as behaviour someone could observe. A
|
|
45
|
+
criterion you cannot check is a criterion nobody can hold you to.
|
|
46
|
+
|
|
47
|
+
A requirement's silence is not permission. What a reasonable reader would
|
|
48
|
+
expect is part of it: the invalid input, the denied permission, the empty
|
|
49
|
+
collection. List the input classes the requirement implies but no test
|
|
50
|
+
exercises, and give the few most likely to bite someone using it a test each.
|
|
51
|
+
|
|
52
|
+
You cannot test everything, and the point of choosing is that the effort lands
|
|
53
|
+
on the paths people depend on and the logic that is easy to get wrong, not on
|
|
54
|
+
every edge case there is.
|
|
55
|
+
|
|
56
|
+
Expected values come from the requirement, a worked example or a known-good
|
|
57
|
+
literal, never from the code under test or its helpers. An expectation the
|
|
58
|
+
code computes passes whatever the code does.
|
|
59
|
+
|
|
60
|
+
Say which requirement a test covers, in its name or in a line beside it. A test
|
|
61
|
+
that names a requirement but does not assert its behaviour does not cover it.
|
|
62
|
+
|
|
63
|
+
Test your own contract, not the library's. What the framework does is its
|
|
64
|
+
maintainers' test to write; yours is the route you register, the query you
|
|
65
|
+
emit, the payload you produce. A constructor, an accessor, a constant or a
|
|
66
|
+
plain hand-off earns a test only where it validates, normalises, defaults,
|
|
67
|
+
derives, enforces or causes an effect; otherwise assert the first result a
|
|
68
|
+
caller can see that depends on it. When upstream behaviour genuinely surprised
|
|
69
|
+
you, one narrow test that names the assumption is worth keeping.
|
|
70
|
+
|
|
71
|
+
Behaviour, not text. Asserting that a script, a configuration or a document
|
|
72
|
+
contains a particular line proves only that the source is the source. Run the
|
|
73
|
+
thing against a controlled input and assert what comes out: the output, the
|
|
74
|
+
effect, the exit status. Prose written for people earns no test at all.
|
|
75
|
+
|
|
76
|
+
## One vertical slice at a time
|
|
77
|
+
|
|
78
|
+
One test, then the smallest implementation that passes it, then the next. Not
|
|
79
|
+
all the tests first and then all the code: tests written in bulk describe
|
|
80
|
+
behaviour you imagined, and they all go green in one step that nobody watched.
|
|
81
|
+
|
|
82
|
+
A separate author writing the acceptance tests up front is not that, although
|
|
83
|
+
it looks like it: those are a few tests at the top seam, taken from the
|
|
84
|
+
requirement rather than from a plan. You still work through them one at a time,
|
|
85
|
+
watching each one fail and then pass. When a slice turns out to need a narrower
|
|
86
|
+
test (a boundary the top seam does not reach), that one goes to the author
|
|
87
|
+
too; the brief is smaller, not different.
|
|
88
|
+
|
|
89
|
+
Test at a seam: the public boundary where you can observe behaviour without
|
|
90
|
+
reaching inside. Prefer a seam that already exists, and the highest one that
|
|
91
|
+
still shows the behaviour; the fewer seams a codebase has, the better. State
|
|
92
|
+
the seam you are testing at. Ask about it only when the interface itself is
|
|
93
|
+
what is in question.
|
|
94
|
+
|
|
95
|
+
Do not anticipate the next test. Code that no failing test asked for is
|
|
96
|
+
untested code, whatever it does.
|
|
97
|
+
|
|
98
|
+
Before you settle a test, run the behaviour past the kinds of mistake that get
|
|
99
|
+
made (a boundary off by one, a condition the wrong way round, an equality that
|
|
100
|
+
should be an inequality, an arithmetic identity, an empty collection, a side
|
|
101
|
+
effect that never happens) and add the case you find missing there and then.
|
|
102
|
+
The inputs that tell those apart: just below, at and just above a boundary; one
|
|
103
|
+
condition true and the other false; and values that are not identities, since
|
|
104
|
+
0, 1 and empty hide a swapped operator.
|
|
105
|
+
It is the same question the mutation check asks at the end, asked while it is
|
|
106
|
+
still cheap to answer.
|
|
107
|
+
|
|
108
|
+
Use the least substitute that does the job, because the more of the real thing
|
|
109
|
+
a test runs, the more it tells you. In order: the real implementation; a
|
|
110
|
+
working in-memory version of it; something that returns fixed answers; and last
|
|
111
|
+
a stand-in you assert the calls of. Reach past the real thing only when it is
|
|
112
|
+
slow, unpredictable, or has effects you cannot control: an external service,
|
|
113
|
+
the clock, randomness, a database sometimes, the filesystem sometimes. Do not
|
|
114
|
+
stand in for your own modules or internal collaborators.
|
|
115
|
+
|
|
116
|
+
What settles this for a system outside your code is not who owns it but
|
|
117
|
+
whether you can have one to yourself. A database you start for the test, a
|
|
118
|
+
repository you create in a temporary directory, a server you run and throw
|
|
119
|
+
away: those are the real thing, isolated, and they are what the order above
|
|
120
|
+
asks for; a stand-in there would hide exactly the incompatibility you want to
|
|
121
|
+
find. What rules the real one out is sharing it: other people's work in it,
|
|
122
|
+
state you cannot reset, or simply not having permission to touch it. Then an
|
|
123
|
+
author works against a stand-in, and the project's rules usually name which
|
|
124
|
+
systems those are.
|
|
125
|
+
|
|
126
|
+
When a test is hard to write, that is information about the design, not about
|
|
127
|
+
testing. Hard to set up usually means hard to use. Having to stand in for
|
|
128
|
+
everything means the pieces are too tightly bound. Pass what a function
|
|
129
|
+
depends on in rather than letting it reach out and build its own. A single
|
|
130
|
+
function per external operation is easier to substitute than one general one
|
|
131
|
+
with a switch inside it, because each substitute then answers one question
|
|
132
|
+
instead of branching.
|
|
133
|
+
|
|
134
|
+
Four things about a stand-in, when you do need one. Learn what the real thing
|
|
135
|
+
does before you replace it, and stand in at the slow or external level below
|
|
136
|
+
what the test depends on, so you do not swallow an effect the test was there to
|
|
137
|
+
see. Mirror the real shape completely, not only the fields this test reads: a
|
|
138
|
+
partial one passes while the real thing breaks on a field it left out. Assert
|
|
139
|
+
the real behaviour, never the stand-in itself. An assertion that the stand-in
|
|
140
|
+
is present passes because it is present and says nothing about your code. And
|
|
141
|
+
when the setup grows past the test it serves, or you cannot say why it is
|
|
142
|
+
there, drop it and test against the real components.
|
|
143
|
+
|
|
144
|
+
Cleanup that only tests need belongs in the test helpers, not as a method on
|
|
145
|
+
the thing being tested.
|
|
146
|
+
|
|
147
|
+
## The test author is someone else
|
|
148
|
+
|
|
149
|
+
An agent that writes the code and the tests shares one set of blind spots. It
|
|
150
|
+
writes tests it already knows will pass, and when one goes red the cheapest
|
|
151
|
+
road is to soften it. What separation buys is a context that holds the
|
|
152
|
+
requirement and not the implementation. The isolated context is the point, not
|
|
153
|
+
a role or a persona.
|
|
154
|
+
|
|
155
|
+
Who counts: a fresh subagent, another session, another bot, or a person. In
|
|
156
|
+
Claude Code it must be a subagent with its own context, not a fork that
|
|
157
|
+
inherits the conversation. Codex spawns a subagent only when asked in so many
|
|
158
|
+
words, so ask in so many words.
|
|
159
|
+
|
|
160
|
+
What the author gets: the requirement, the acceptance criteria, and the public
|
|
161
|
+
interface (names, signatures, types, the shape of inputs and outputs). Not your
|
|
162
|
+
plan, not your diff, not the implementation. What the author returns: the test
|
|
163
|
+
files, the command, the failing output, and one line per test naming the
|
|
164
|
+
requirement it covers.
|
|
165
|
+
|
|
166
|
+
A brief that works, adjusted to the job:
|
|
167
|
+
|
|
168
|
+
> Write tests for this requirement. You are not writing the implementation and
|
|
169
|
+
> you will not see it.
|
|
170
|
+
> Requirement: <what the behaviour is, and the acceptance criteria>.
|
|
171
|
+
> Out of scope: <what this change does not cover, if anything>.
|
|
172
|
+
> Public interface: <signatures and data shapes, no bodies>.
|
|
173
|
+
> Test at <seam>, in <the repo's test style and framework>; follow the
|
|
174
|
+
> repo's existing tests for shape. Write them the way obk-tdd's parts on
|
|
175
|
+
> writing tests say, if you can load it.
|
|
176
|
+
> Test observable behaviour through that interface. Take expected values from
|
|
177
|
+
> the requirement or a worked example, never by calling the code under test.
|
|
178
|
+
> Cover the unhappy inputs the requirement implies as well as the happy one.
|
|
179
|
+
> Run them. They are all expected to fail, and to fail because the behaviour is
|
|
180
|
+
> missing rather than because of a typo or an import error.
|
|
181
|
+
> Return: the files you wrote, the command, its failing output, and one line
|
|
182
|
+
> per test naming the requirement it covers.
|
|
183
|
+
|
|
184
|
+
Say what state the code is in, and be exact about it. The author cannot see it
|
|
185
|
+
and will believe what the brief says. When you send an author at code you have
|
|
186
|
+
deliberately broken (killing a surviving mutant, see below), say so in the
|
|
187
|
+
brief: *the code has a deliberate break in it and the test is expected to
|
|
188
|
+
fail*. Left out, the author reads your own break as a defect and spends a round
|
|
189
|
+
reporting a bug that does not exist.
|
|
190
|
+
|
|
191
|
+
Then, as the implementer: every test that ships is the author's, the narrow
|
|
192
|
+
ones as much as the first ones. A scratch check you run to see what the code is
|
|
193
|
+
doing is yours and is thrown away; it is not part of the suite, and it is not
|
|
194
|
+
the red you report. You do not edit, weaken, skip or delete the author's tests
|
|
195
|
+
to get to green. A test you believe
|
|
196
|
+
is wrong goes back to its author with what you think is wrong with it. That is
|
|
197
|
+
a real and expected outcome, not a defeat. The tests verify the requirement;
|
|
198
|
+
they do not define the solution, so do not write code that works only for the
|
|
199
|
+
inputs a test happens to use.
|
|
200
|
+
|
|
201
|
+
The author's tests are part of the change, not a stage before it. Commit them
|
|
202
|
+
as soon as they are red, before the implementation: that way a later diff over
|
|
203
|
+
the test paths shows any change to them, and (the part that actually bites)
|
|
204
|
+
they cannot be left behind. Where the repo allows it, keeping those paths out
|
|
205
|
+
of the implementer's write scope makes tampering visible rather than merely
|
|
206
|
+
forbidden.
|
|
207
|
+
|
|
208
|
+
An author you have dispatched and not heard back from is outstanding work. The
|
|
209
|
+
piece is not done while it is still writing, and nothing goes out on the
|
|
210
|
+
strength of tests that have not arrived. When the hand-back comes, read it
|
|
211
|
+
against what is actually there: every file it names, present in the working
|
|
212
|
+
tree, and present in the change you are about to hand on. A finished test file
|
|
213
|
+
still sitting in a working tree is the ordinary way this goes wrong, and it
|
|
214
|
+
looks exactly like success until someone else clones the branch.
|
|
215
|
+
|
|
216
|
+
When you can neither start an author nor reach one, that is where the work
|
|
217
|
+
stops: say so and ask what to do. The separation is the check, and an
|
|
218
|
+
implementer writing its own tests is not a weaker version of it, it is the
|
|
219
|
+
absence of it.
|
|
220
|
+
|
|
221
|
+
## Red is evidence, not a formality
|
|
222
|
+
|
|
223
|
+
Run the test and read the output before you write any code:
|
|
224
|
+
|
|
225
|
+
- It fails, rather than errors.
|
|
226
|
+
- The failure message is the one you expected.
|
|
227
|
+
- It fails because the behaviour is missing, not a typo, a bad import, a
|
|
228
|
+
missing fixture.
|
|
229
|
+
|
|
230
|
+
If it passes, you are testing behaviour that already exists; that is a finding
|
|
231
|
+
about the test, not a step you may skip past. If it errors, fix the error and
|
|
232
|
+
run again until it fails cleanly.
|
|
233
|
+
|
|
234
|
+
Behaviour that already exists is the case this is easiest to fudge, because
|
|
235
|
+
there is no fix to revert and nothing obvious to run the test against. The red
|
|
236
|
+
has to come from somewhere real: an archive of the code as it stood before this
|
|
237
|
+
work (a commit, a tag, a copy) with the test run against that. Name which one
|
|
238
|
+
in the hand-over, because "it would have failed before" is not a run.
|
|
239
|
+
|
|
240
|
+
Sometimes there is no such baseline to be had: the code never existed in that
|
|
241
|
+
shape, or what you would have to reconstruct is not worth the day. Then say so
|
|
242
|
+
in the hand-over, and in a line beside the test marking it as a description of
|
|
243
|
+
what the code does today rather than a guard proven to catch a change. The two
|
|
244
|
+
look identical afterwards, and only one of them has been shown to fail.
|
|
245
|
+
|
|
246
|
+
Reaching a clean red is one step at a time, and none of them is implementation:
|
|
247
|
+
the test cannot find the symbol, so add an empty stub; the call does not match
|
|
248
|
+
the signature, so fix the signature and leave the body empty; now it fails on
|
|
249
|
+
the assertion, and only now does logic get written.
|
|
250
|
+
|
|
251
|
+
Keep the failing output as it was printed. Commit order proves nothing about
|
|
252
|
+
what ran first. Without a captured failing run, say the chronology is unproven
|
|
253
|
+
rather than implying it.
|
|
254
|
+
|
|
255
|
+
Things that look like green and are not: a test that passed on its first run; a
|
|
256
|
+
suite reported as passing that nobody ran; a default test command guessed
|
|
257
|
+
instead of the one this repo actually uses. And your test passing is not the
|
|
258
|
+
suite passing. Any failure the run showed, including one you did not cause,
|
|
259
|
+
goes in your report by name. Green means the output is clean too: warnings and
|
|
260
|
+
errors that scroll past on a passing run are results, not decoration.
|
|
261
|
+
|
|
262
|
+
Where the change reaches further than the test you wrote, run what is next to
|
|
263
|
+
it as well (the neighbouring tests, the type check, whatever the project uses
|
|
264
|
+
to say a thing still holds) and say in your report what you ran.
|
|
265
|
+
|
|
266
|
+
An assertion that already existed is not yours to loosen. It weakens only when
|
|
267
|
+
the behaviour it describes genuinely changed, and then the reason is stated out
|
|
268
|
+
loud, not left to be inferred from the diff.
|
|
269
|
+
|
|
270
|
+
## Green, then refactor outside the loop
|
|
271
|
+
|
|
272
|
+
Write the smallest code that passes the failing test, and stop there.
|
|
273
|
+
|
|
274
|
+
Faking it is allowed and often right. If returning the answer the test asks for
|
|
275
|
+
passes, return it; then write the second case that the fake cannot satisfy, and
|
|
276
|
+
let that one force the real implementation. Generalise when a test makes you,
|
|
277
|
+
not before.
|
|
278
|
+
|
|
279
|
+
Refactoring is not a phase of the red/green loop. A small refactor is part of
|
|
280
|
+
the change and keeps the same contract, so the tests that were green stay
|
|
281
|
+
green; do it as its own step after green rather than while you are trying to
|
|
282
|
+
get there, and keep the tests green on both sides of it. Pulling out something
|
|
283
|
+
whose behaviour already exists elsewhere is refactoring; a function whose
|
|
284
|
+
behaviour appears nowhere else is new, and it starts with a failing test like
|
|
285
|
+
anything else.
|
|
286
|
+
|
|
287
|
+
Do not manufacture a red. A failing test written so a step looks like it
|
|
288
|
+
started with one, a break invented so a restructuring can be called a change of
|
|
289
|
+
behaviour, commits reshaped afterwards to tell a tidier story: all of that
|
|
290
|
+
costs the time of doing it and buys nothing, because none of it was ever a
|
|
291
|
+
check on anything. A step that did not need a red is reported as what it was.
|
|
292
|
+
|
|
293
|
+
When a change is large enough that the old tests cannot hold, that is still not
|
|
294
|
+
a licence to edit them. The author writes them again from the new requirement,
|
|
295
|
+
usually deleting the old ones first. A large refactor is planned as a piece of
|
|
296
|
+
work in its own right, not carried inside another change.
|
|
297
|
+
|
|
298
|
+
Tests against an inner module become waste once the same behaviour is covered
|
|
299
|
+
at the interface above it. Deleting them is the author's call, and deliberate.
|
|
300
|
+
|
|
301
|
+
## Bugs
|
|
302
|
+
|
|
303
|
+
A bug fix starts with a test that reproduces it, written before the fix, at a
|
|
304
|
+
seam where the test exercises the bug as it actually happens at the call site.
|
|
305
|
+
A seam below that gives a green test and no protection. If no correct seam
|
|
306
|
+
exists, that is itself the finding: report it, because the shape of the code is
|
|
307
|
+
what is stopping the bug from being pinned down.
|
|
308
|
+
|
|
309
|
+
Minimise the reproduction until every part of it is load-bearing, turn it into
|
|
310
|
+
a failing test, watch it fail, fix the cause, watch it pass, then run the
|
|
311
|
+
original unminimised scenario again.
|
|
312
|
+
|
|
313
|
+
Two tests are often better than one here: one at the interface, saying what a
|
|
314
|
+
caller should have got, and the smallest one that shows the fault where it
|
|
315
|
+
lives. Get both to pass. The first is what stops the bug coming back in a way
|
|
316
|
+
anyone would notice; the second says where it was.
|
|
317
|
+
|
|
318
|
+
A regression test written after the fix has proved nothing yet. Prove it:
|
|
319
|
+
revert the fix, run it and watch it fail, restore the fix, run it again.
|
|
320
|
+
|
|
321
|
+
When the bug is intermittent, make the test deterministic if you can, and say
|
|
322
|
+
which signal you pinned down. When it is one of a class, land the test for the
|
|
323
|
+
one in front of you first, then say what else has the same shape.
|
|
324
|
+
|
|
325
|
+
When a failing test really is impractical, say so out loud, say why, and name
|
|
326
|
+
the closest executable check you used instead. Quiet omission is the part that
|
|
327
|
+
does damage: a reviewer treats a missing red receipt as a finding.
|
|
328
|
+
|
|
329
|
+
## Tests that catch nothing
|
|
330
|
+
|
|
331
|
+
Before writing the body, answer one question: what change to the production
|
|
332
|
+
code should make this test fail, and is that change a bug or a decision? A test
|
|
333
|
+
earns its place by catching a wrong branch, a missing side effect, a wrong
|
|
334
|
+
argument, a boundary or a broken contract. A test that only fails on a
|
|
335
|
+
deliberate decision fires on every redesign and sleeps through every bug.
|
|
336
|
+
|
|
337
|
+
The cheap check on a test you already have: would it still pass if every
|
|
338
|
+
function it calls returned nothing? If yes, it observes no behaviour.
|
|
339
|
+
|
|
340
|
+
Five shapes that still pass that way, and what to do with each:
|
|
341
|
+
|
|
342
|
+
- **No real assertion.** Nothing asserted, or only "is defined", "is truthy",
|
|
343
|
+
"did not throw". Call the subject with one concrete input and assert the
|
|
344
|
+
literal output or the observable effect.
|
|
345
|
+
- **An absence with nothing to contrast it against.** Only "was not called",
|
|
346
|
+
"is undefined", "is empty", and no case in the same test where something
|
|
347
|
+
*is* produced. Assert the presence on the other input beside it, so the pair
|
|
348
|
+
can tell an empty result from a subject that never ran. For a stand-in,
|
|
349
|
+
assert the payload it received or the state after the call, not only that it
|
|
350
|
+
was called.
|
|
351
|
+
- **Self-referential.** The expected value is produced by the code under test.
|
|
352
|
+
Replace it with a literal you worked out by hand.
|
|
353
|
+
- **Constant pinned.** The assertion restates a constant, a config default or
|
|
354
|
+
the exact wording of a message. Test the mechanism that reads the constant
|
|
355
|
+
with one input instead.
|
|
356
|
+
- **Fixture asserts fixture.** The assertion reads data the test itself built,
|
|
357
|
+
and the subject never runs. Run the subject inside the test body.
|
|
358
|
+
|
|
359
|
+
An empty or absent result is not by itself a weak assertion. "This user sees
|
|
360
|
+
none of the private records" and "this call is never made" are contracts, and
|
|
361
|
+
a test that pins one catches a real defect. So are the arguments a call
|
|
362
|
+
carries, how many times it happens and in what order, where those are the
|
|
363
|
+
contract rather than an accident of how the code is written today. What makes
|
|
364
|
+
the shape weak is an absence with no positive case beside it, or a stub
|
|
365
|
+
asserted instead of the behaviour. Judge by the question that governs the
|
|
366
|
+
whole list: would this catch a defect someone could ship?
|
|
367
|
+
|
|
368
|
+
When no honest assertion exists, delete the test. Prefer no new test to a bad
|
|
369
|
+
one: one that mostly tests stubs, that encodes today's implementation, that
|
|
370
|
+
depends on timing or global state, or that you would delete the moment the fix
|
|
371
|
+
is proven.
|
|
372
|
+
|
|
373
|
+
Two that the five shapes do not condemn: a test that checks a relation holds
|
|
374
|
+
across rows of data (a key present in two tables, a parent that exists), and a
|
|
375
|
+
check the language itself makes at build time.
|
|
376
|
+
|
|
377
|
+
Other tells: the test breaks when you refactor although the behaviour did not
|
|
378
|
+
change; it reaches around the interface to check the result (reading the
|
|
379
|
+
database directly instead of asking for the record back), which couples it to
|
|
380
|
+
the inside just as surely as a stub does; the stub setup is more than half of
|
|
381
|
+
it; the expected values are hidden behind loops or builders; the only way it
|
|
382
|
+
can fail is a crash; it exists to raise a coverage number. Coverage is a
|
|
383
|
+
diagnostic and not a target: a high number says the lines ran, not that
|
|
384
|
+
anything would have noticed them being wrong.
|
|
385
|
+
|
|
386
|
+
One behaviour per test, so that a failure names one thing. That is not a limit
|
|
387
|
+
on how many assertions you write: a table of cases for one behaviour is still
|
|
388
|
+
one test, and the several assertions that together pin one outcome are one
|
|
389
|
+
claim. It is a limit on how many unrelated claims share a name.
|
|
390
|
+
|
|
391
|
+
Repetition between tests is not a fault. A test that reads on its own, with its
|
|
392
|
+
input and its expected value in front of you, is worth more than one that sends
|
|
393
|
+
you through three helpers to find out what it claims. Share setup when it makes
|
|
394
|
+
the test clearer, not to avoid typing.
|
|
395
|
+
|
|
396
|
+
What is missing is usually on the unhappy side: the denied case of a permission
|
|
397
|
+
check, untested guard clauses and early returns, error branches, boundaries
|
|
398
|
+
(zero, empty, negative, one element, the maximum), and the second half of any
|
|
399
|
+
"if". And tests that pass together but not alone are already broken: shared
|
|
400
|
+
mutable state, order dependence, the clock, the timezone, the real network, a
|
|
401
|
+
sleep, an assertion on the order of something unordered.
|
|
402
|
+
|
|
403
|
+
## The mutation check before you call it done
|
|
404
|
+
|
|
405
|
+
The question is the whole point: if a real mistake were made in this code,
|
|
406
|
+
would a test fail? There is no score to reach.
|
|
407
|
+
|
|
408
|
+
**Everyday: your own hand check.** Once per piece of work, at the end, only
|
|
409
|
+
where it earns its place. Five to eight deliberate breaks in the risky logic
|
|
410
|
+
the work changed (money, permissions, eligibility, safety, data that could be
|
|
411
|
+
lost, and whatever else the requirement cares most about), chosen before you
|
|
412
|
+
look at the tests, each one run, each one expected to make a test fail, each
|
|
413
|
+
one put back. Reported in three lines.
|
|
414
|
+
|
|
415
|
+
One break at a time, never two at once: note the original, apply the break, run
|
|
416
|
+
the tests, record killed or survived, restore the code immediately, then the
|
|
417
|
+
next one.
|
|
418
|
+
|
|
419
|
+
What to break, in this order: boundaries (`<` for `<=`, one off), boolean logic
|
|
420
|
+
(`&&` for `||`, a dropped `not`), returned values (an empty or default return,
|
|
421
|
+
a removed early return), a removed statement or side effect, a wrong constant
|
|
422
|
+
or argument. Avoid a change that cannot show: adding zero, multiplying by one, or a value
|
|
423
|
+
identical to the original.
|
|
424
|
+
|
|
425
|
+
Others worth reaching for when the code has them: an arithmetic operator
|
|
426
|
+
swapped; equality turned into its opposite; a string emptied or altered; a
|
|
427
|
+
collection emptied, or a filter or a sort taken out; a safe-navigation step
|
|
428
|
+
removed so an absent value reaches further than it should; a call to a
|
|
429
|
+
collaborator deleted.
|
|
430
|
+
|
|
431
|
+
Skip it, and say that you skipped it, for work with no real logic in it: docs,
|
|
432
|
+
config, wiring, renames, small fixes, prototypes, and code with no runnable
|
|
433
|
+
tests.
|
|
434
|
+
|
|
435
|
+
**A survivor**, a break no test objected to, gets the equivalence question
|
|
436
|
+
first: is there any input for which the changed code returns a different value,
|
|
437
|
+
stores different state, or emits a different message? If there is none it is
|
|
438
|
+
equivalent, and no test can catch it; write the one-line argument for why.
|
|
439
|
+
"No test covers this" is not the same as "no input can reach this". Every
|
|
440
|
+
survivor ends as exactly one of three things: killed by a new test, equivalent
|
|
441
|
+
with that argument, or a judgment call you name and leave. A list of survivors
|
|
442
|
+
with nothing said about them is not a report, and neither is treating each one
|
|
443
|
+
as a defect.
|
|
444
|
+
|
|
445
|
+
When you cannot tell whether a survivor matters (the behaviour is genuinely
|
|
446
|
+
unspecified, the test would be expensive or brittle, or you are not sure it is
|
|
447
|
+
equivalent), name it as that in the report, with the question whoever decides
|
|
448
|
+
the behaviour would have to answer, rather than deciding quietly in either
|
|
449
|
+
direction.
|
|
450
|
+
And if a lot of the breaks turn out to be equivalent, that is a finding about
|
|
451
|
+
the code, not about the tests: code with that much slack in it can usually be
|
|
452
|
+
made simpler.
|
|
453
|
+
|
|
454
|
+
Kill a survivor only where it shows a gap in behaviour the requirement cares
|
|
455
|
+
about. Wording of messages, logging and changes with no visible difference are
|
|
456
|
+
not worth a test; say so in a line and move on. To kill one: keep the break in
|
|
457
|
+
place, write the smallest behaviour test that fails against it for the right
|
|
458
|
+
reason, restore the code, check the new test still passes.
|
|
459
|
+
|
|
460
|
+
The implementer runs the check. A test that needs strengthening goes to the
|
|
461
|
+
separate author, like any other test.
|
|
462
|
+
|
|
463
|
+
**A mutation tool is an audit, not everyday work.** It is never a requirement,
|
|
464
|
+
and it is not worth a user's time on a suite it does not fit. Where a project
|
|
465
|
+
has one, run it over the whole suite occasionally (at a milestone, or when the
|
|
466
|
+
user asks) in the background at a quiet time, in its own tab or session. Read
|
|
467
|
+
the result once; turn the few real gaps into test work; ignore the rest. Not
|
|
468
|
+
once per change, and never twice for the same change. For a piece of work at
|
|
469
|
+
the core of a product, one run narrowed to the logic that changed, in the
|
|
470
|
+
background, around twenty minutes; if it will not fit in that, do the hand
|
|
471
|
+
check instead. Then stop. A run heading towards hours is stopped.
|
|
472
|
+
|
|
473
|
+
When you have decided to run a tool, read [mutation-tools.md](mutation-tools.md)
|
|
474
|
+
beside this file: the scoped run for each language, what older advice gets
|
|
475
|
+
wrong, how to run one and read what it prints, and how far each line there has
|
|
476
|
+
been verified. There is nothing in it you need before that point.
|
|
477
|
+
|
|
478
|
+
Keep it in proportion. This check is the third safety net, after the separate
|
|
479
|
+
author and the reviewer's reading of the tests, and it stays the cheapest of
|
|
480
|
+
the three.
|
|
481
|
+
|
|
482
|
+
## Green that was bought
|
|
483
|
+
|
|
484
|
+
Tightening the bar should be silent; loosening it should be loud. In your own
|
|
485
|
+
diff and in anyone else's, these are the cheap roads to green, and each one is
|
|
486
|
+
worth a sentence when it happens: a threshold moved, a test marked skipped, a
|
|
487
|
+
test file deleted, assertions removed from a test that stayed, a check disabled
|
|
488
|
+
by a comment, a value hard-coded to match one test's input and still there when
|
|
489
|
+
the work is handed on.
|
|
490
|
+
|
|
491
|
+
## What you report
|
|
492
|
+
|
|
493
|
+
- Red: the command, the failing output, why that failure was expected.
|
|
494
|
+
- Green: the command, the passing output, and any other failure the run showed.
|
|
495
|
+
- Who wrote the tests, whether they were changed after they were written, and
|
|
496
|
+
that every file the author handed back is in the change.
|
|
497
|
+
- For a test of behaviour that already existed: which baseline its red came
|
|
498
|
+
from, or that there was none and it stands as a description rather than a
|
|
499
|
+
proven guard.
|
|
500
|
+
- The mutation check in three lines: what you broke, what survived, what you
|
|
501
|
+
did about it, or the line that says you skipped it and why.
|
|
502
|
+
- What you did not check, and the command that would check it.
|
|
503
|
+
|
|
504
|
+
## Away from code
|
|
505
|
+
|
|
506
|
+
The same three moves work on documentation, configuration, prompts and data,
|
|
507
|
+
where no tool will help: state the check before you make the change, see the
|
|
508
|
+
check fail first, and at the end break the artefact on purpose to find out
|
|
509
|
+
whether the check notices.
|
|
510
|
+
|
|
511
|
+
They apply where there is something that can be wrong: a configuration that
|
|
512
|
+
decides who gets what, a prompt whose branches behave differently, a
|
|
513
|
+
transformation over data. The skip rule is the same as for code, and prose
|
|
514
|
+
written for people earns no test and no mutant.
|
|
515
|
+
|
|
516
|
+
Sources and licences: [NOTICE.md](NOTICE.md).
|
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
# Mutation tools, by stack
|
|
2
|
+
|
|
3
|
+
For the occasional audit described in the skill. No project needs one of these,
|
|
4
|
+
and the hand check is the everyday method; this page exists so that a run, when
|
|
5
|
+
there is a reason for one, is scoped to what changed instead of the whole repo.
|
|
6
|
+
|
|
7
|
+
How far to trust each line. The StrykerJS flags below (`--incremental`,
|
|
8
|
+
`--force` and `-m`/`--mutate`) were checked against `stryker run --help` on
|
|
9
|
+
version 10.0.0 on 2026-09-20. Everything else on this page, the line-range form
|
|
10
|
+
included, was read from each tool's own documentation on 2026-09-19 and has not
|
|
11
|
+
been run. Flags change: check the version the project actually has before
|
|
12
|
+
trusting a line here, and prefer a command the project already ships over one
|
|
13
|
+
from this page.
|
|
14
|
+
|
|
15
|
+
**JavaScript, TypeScript: StrykerJS** (Apache-2.0)
|
|
16
|
+
|
|
17
|
+
```sh
|
|
18
|
+
npx stryker run --incremental
|
|
19
|
+
npx stryker run --mutate src/x.ts # one file
|
|
20
|
+
npx stryker run --mutate src/x.ts:10-40 # one range
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
**C#: Stryker.NET** (Apache-2.0): `dotnet stryker --since:main`
|
|
24
|
+
|
|
25
|
+
**Rust: cargo-mutants** (MIT)
|
|
26
|
+
|
|
27
|
+
```sh
|
|
28
|
+
git diff origin/main.. > git.diff && cargo mutants --in-diff git.diff
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
**Go: gremlins** (Apache-2.0): `gremlins unleash --diff origin/main`, with the
|
|
32
|
+
full history fetched.
|
|
33
|
+
|
|
34
|
+
**Python: mutmut 3** (BSD-3-Clause): `mutmut run "module.function*"`;
|
|
35
|
+
incremental by default; `mutmut browse` to work through the survivors.
|
|
36
|
+
|
|
37
|
+
**Python: Cosmic Ray** (MIT): `cosmic-ray init`, then `cr-filter-git`, then
|
|
38
|
+
`cosmic-ray exec`, then `cr-report`.
|
|
39
|
+
|
|
40
|
+
**Java: PIT** (Apache-2.0):
|
|
41
|
+
`mvn -DwithHistory test-compile org.pitest:pitest-maven:mutationCoverage`, with
|
|
42
|
+
`targetClasses` and `targetTests` globs.
|
|
43
|
+
|
|
44
|
+
**Anything else**, shell, SQL, configuration, prompts, a language with no such
|
|
45
|
+
tool: the hand check in the skill is the method, and it is the same method,
|
|
46
|
+
including the rule about when to skip it. Something with no logic in it does
|
|
47
|
+
not get broken on purpose just because no tool would have run anyway.
|
|
48
|
+
|
|
49
|
+
## Four things that older advice gets wrong
|
|
50
|
+
|
|
51
|
+
- PIT's `scmMutationCoverage` goal was deprecated in 1.17.1 and removed in
|
|
52
|
+
1.18.0. Diff scoping in PIT now comes from a commercial plugin; with the free
|
|
53
|
+
one, scope with the globs above and `-DwithHistory`.
|
|
54
|
+
- mutmut 3 has no `--paths-to-mutate`. Scope it in the config (`only_mutate`,
|
|
55
|
+
`do_not_mutate`, `mutate_only_covered_lines = true`) or with a positional
|
|
56
|
+
glob.
|
|
57
|
+
- `cargo mutants --in-diff` takes a diff file, not a git ref: write the diff
|
|
58
|
+
out first.
|
|
59
|
+
- StrykerJS has no git-diff flag; feed it the changed files yourself.
|
|
60
|
+
|
|
61
|
+
## Three caveats that apply to all of them
|
|
62
|
+
|
|
63
|
+
- A diff-scoped run matches the diff against the code under test, not the test
|
|
64
|
+
code. A change that only weakens a test produces no mutants at all and comes
|
|
65
|
+
back clean. When test files change, mutate the code those tests cover.
|
|
66
|
+
- A run that picks its targets from a committed diff sees only committed work,
|
|
67
|
+
so on a dirty tree it reports on the wrong code: commit first, or name the
|
|
68
|
+
files yourself instead. Naming them is the better answer when the point is to
|
|
69
|
+
check work that is not committed yet. Do not make someone stash the code
|
|
70
|
+
they asked about.
|
|
71
|
+
- Whatever selects the targets, a run that dies in the middle can leave a
|
|
72
|
+
mutant in the working copy. Look at the tree afterwards.
|
|
73
|
+
|
|
74
|
+
## Running one and reading it
|
|
75
|
+
|
|
76
|
+
Prove the setup with one small scoped run before spending a long one. Capture
|
|
77
|
+
the output once and read it from the copy rather than re-running to re-read it.
|
|
78
|
+
Set no failing threshold before a measured baseline exists. Paste what the tool
|
|
79
|
+
printed: a mutation result reported from memory has been wrong. A compile error
|
|
80
|
+
is not a kill, and a timeout is inconclusive rather than evidence.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
<!-- The licences of the sources this skill was consolidated from. They travel
|
|
2
|
+
with the skill directory because the licences require it of anyone who
|
|
3
|
+
distributes the material. NOTICE.md says what was taken from each source;
|
|
4
|
+
this file carries the permission notices themselves. -->
|
|
5
|
+
|
|
6
|
+
# Licences of the sources
|
|
7
|
+
|
|
8
|
+
`obk-teach` consolidates material from the sources named in
|
|
9
|
+
[NOTICE.md](NOTICE.md). Their licences are reproduced below in full.
|
|
10
|
+
|
|
11
|
+
## MIT
|
|
12
|
+
|
|
13
|
+
The MIT permission notice below applies to the work of each of these copyright
|
|
14
|
+
holders, and is reproduced with each of their notices. The path after each one
|
|
15
|
+
is the licence file it was read from, in the research pack's source clones.
|
|
16
|
+
|
|
17
|
+
- **Cursor `plugins`, the pstack plugin**, Copyright (c) 2026 Lauren Tan
|
|
18
|
+
(`cursor_plugins/pstack/LICENSE`)
|
|
19
|
+
|
|
20
|
+
```
|
|
21
|
+
MIT License
|
|
22
|
+
|
|
23
|
+
Copyright (c) 2026 Lauren Tan
|
|
24
|
+
|
|
25
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
26
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
27
|
+
in the Software without restriction, including without limitation the rights
|
|
28
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
29
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
30
|
+
furnished to do so, subject to the following conditions:
|
|
31
|
+
|
|
32
|
+
The above copyright notice and this permission notice shall be included in all
|
|
33
|
+
copies or substantial portions of the Software.
|
|
34
|
+
|
|
35
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
36
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
37
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
38
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
39
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
40
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
41
|
+
SOFTWARE.
|
|
42
|
+
```
|