@codesema/speccov 0.0.0-stage → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +607 -2
- package/bin/speccov.mjs +80 -0
- package/package.json +63 -5
- package/src/adapters/typeorm.ts +90 -0
- package/src/answers.ts +126 -0
- package/src/cli.ts +417 -0
- package/src/complexity.ts +75 -0
- package/src/counter.ts +16 -0
- package/src/domain.ts +70 -0
- package/src/draft/domain.ts +138 -0
- package/src/draft/extract.ts +550 -0
- package/src/draft/harvest.ts +184 -0
- package/src/draft/index.ts +39 -0
- package/src/draft/render-forbidden.ts +84 -0
- package/src/draft/render.ts +530 -0
- package/src/draft/sweep.ts +81 -0
- package/src/draft/trace.ts +219 -0
- package/src/forbidden.ts +77 -0
- package/src/gauge.ts +48 -0
- package/src/growth.ts +64 -0
- package/src/index.ts +21 -0
- package/src/json.ts +121 -0
- package/src/metamorphic.ts +68 -0
- package/src/mutate/mutators.ts +166 -0
- package/src/mutate/report.ts +52 -0
- package/src/mutate/run.ts +79 -0
- package/src/report.ts +143 -0
- package/src/resource.ts +76 -0
- package/src/rule.ts +129 -0
- package/src/run.ts +70 -0
- package/src/scan/answers.ts +152 -0
- package/src/scan/climb.ts +26 -0
- package/src/scan/flow.ts +65 -0
- package/src/scan/guarantees.ts +234 -0
- package/src/scan/ignored.ts +40 -0
- package/src/scan/index.ts +213 -0
- package/src/scan/program.ts +137 -0
- package/src/scan/report.ts +267 -0
- package/src/scan/shapes.ts +1139 -0
- package/src/scan/speccovignore.ts +65 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Hasan TASKIN
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,608 @@
|
|
|
1
|
-
#
|
|
1
|
+
# speccov
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
**You have 90% code coverage. What about your spec coverage?**
|
|
4
|
+
|
|
5
|
+
`speccov` looks for the behaviours no rule guarantees, and puts them to a human as closed
|
|
6
|
+
questions.
|
|
7
|
+
|
|
8
|
+
Vocabulary: the tool measures **spec coverage**, and the unit it reports is a **gap**.
|
|
9
|
+
"6 gaps left" is the sentence meant to circulate in an MR.
|
|
10
|
+
|
|
11
|
+
## Install and run
|
|
12
|
+
|
|
13
|
+
```sh
|
|
14
|
+
npx @codesema/speccov scan src
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
That is the static pass: it reads the code, executes nothing, and prints the sites worth a
|
|
18
|
+
rule. It needs Node 22.15 or later, and nothing else.
|
|
19
|
+
|
|
20
|
+
To write rules and run them, add speccov to the project:
|
|
21
|
+
|
|
22
|
+
```sh
|
|
23
|
+
npm install --save-dev @codesema/speccov
|
|
24
|
+
|
|
25
|
+
npx speccov scan src # what is proven, and what deserves a rule
|
|
26
|
+
npx speccov draft src/users.ts:41 # drafts the rule file for a site the scan printed
|
|
27
|
+
npx speccov src/users.rules.ts # runs the rules, reports the gaps
|
|
28
|
+
npx speccov mutate src --run "npm test" # alters the code, lists what nothing caught
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
A rules file is TypeScript and imports the project's own code. Under Node, speccov compiles
|
|
32
|
+
it on load with the options of the nearest `tsconfig.json`, so path aliases, enums and
|
|
33
|
+
decorators work as they do in the project. On Bun 1.3 or later, `bunx --bun speccov` runs
|
|
34
|
+
the same commands.
|
|
35
|
+
|
|
36
|
+
## The problem
|
|
37
|
+
|
|
38
|
+
Gates prove that the code does what the tests say. Nothing proves that the tests say the
|
|
39
|
+
right thing. When the AI writes the code _and_ the tests, the loop closes on itself and
|
|
40
|
+
verification is worth nothing.
|
|
41
|
+
|
|
42
|
+
Trust does not come from proof, it comes from **the map of the holes**. A "90% coverage"
|
|
43
|
+
says nothing about the 10%. Six sentences saying "nothing guarantees this" is actionable.
|
|
44
|
+
|
|
45
|
+
## The definition
|
|
46
|
+
|
|
47
|
+
> A **gap** is an input region where the behaviour of the code can be changed without any
|
|
48
|
+
> rule objecting.
|
|
49
|
+
|
|
50
|
+
Concretely: _surviving mutant_ ∩ _fuzz region where no rule fires_.
|
|
51
|
+
It is computable, it is short, and it can be counted.
|
|
52
|
+
|
|
53
|
+
## Who the report speaks to
|
|
54
|
+
|
|
55
|
+
To an agent, as much as to a human. That is what decides the shape of the output.
|
|
56
|
+
|
|
57
|
+
The loop we are after: an agent receives a plan, writes the code, `speccov` measures, and
|
|
58
|
+
the agent fixes **by referring to the plan** — not to its own judgement, which produced the
|
|
59
|
+
code in the first place. speccov is the only link the agent did not write.
|
|
60
|
+
|
|
61
|
+
Two consequences, and they are not cosmetic:
|
|
62
|
+
|
|
63
|
+
- **The false positive changes severity.** A human ignores a bad question. An agent acts on
|
|
64
|
+
it, and degrades code that was fine. A doubtful question now costs a commit, not a second
|
|
65
|
+
of attention.
|
|
66
|
+
- **Every hole must be named, never implicit.** Hence the `TODO`s that throw, the list of
|
|
67
|
+
decisions left to a human, and the "Not measured" zone. An agent does not guess what it
|
|
68
|
+
was not told — it invents.
|
|
69
|
+
|
|
70
|
+
## The four rule types
|
|
71
|
+
|
|
72
|
+
| Type | What it asserts | The real-world defect it catches |
|
|
73
|
+
| --------------- | ----------------------------------------------------------- | ------------------------------------------------------------ |
|
|
74
|
+
| `complexity()` | The cost does not grow with the input size | the N+1 of `list-admin-users.ts:41` |
|
|
75
|
+
| `resource()` | The observed peak stays under a ceiling | the 27 unbounded `Promise.all` |
|
|
76
|
+
| `metamorphic()` | An observation stays invariant from one variant to the next | the `total: items.length` of `list-admin-users.ts:36` |
|
|
77
|
+
| `forbidden()` | No input ever produces that | the "no `adminAreaId` = global access" of `AreaScopeService` |
|
|
78
|
+
|
|
79
|
+
```sh
|
|
80
|
+
bun src/cli.ts examples/real-world-findings.rules.ts
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
All four defects are found. None is caught by the typed lint, the 90% coverage, the
|
|
84
|
+
mutation tests or the repo's e2e: **all of them pass 100% of the existing gates.**
|
|
85
|
+
|
|
86
|
+
### `allOf()` — the domain, not a list of cases
|
|
87
|
+
|
|
88
|
+
A forbidden rule tried against a hand-written list finds only what its author already
|
|
89
|
+
suspected. `allOf()` takes the **domain** and produces every case:
|
|
90
|
+
|
|
91
|
+
```ts
|
|
92
|
+
cases: allOf<Actor>({
|
|
93
|
+
role: ['viewer', 'org_admin', 'platform_viewer', 'platform_admin'],
|
|
94
|
+
adminAreaId: ['area-north', null],
|
|
95
|
+
});
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
Four roles times two area states: eight actors, all tried. The predicate is pure, the loop
|
|
99
|
+
runs in a millisecond — there is no reason to sample a share of it.
|
|
100
|
+
|
|
101
|
+
**Cases are produced from the simplest to the most twisted**, by distance from the first
|
|
102
|
+
value of each field. The first counterexample met is therefore **the smallest**, with no
|
|
103
|
+
shrinking to write: `forbidden()` already keeps the first one, and it turns out to be the
|
|
104
|
+
right one.
|
|
105
|
+
|
|
106
|
+
On that monorepo's scope defect, the hand-written list found one counterexample. The domain
|
|
107
|
+
finds **two**, and puts `viewer without an area` first — a role even less privileged than
|
|
108
|
+
`org_admin`, so a worse hole, that nobody had thought to write.
|
|
109
|
+
|
|
110
|
+
An absurd domain is **refused**, never silently sampled: twenty values across eight fields
|
|
111
|
+
make twenty-five billion cases — that is a typo, not a domain.
|
|
112
|
+
|
|
113
|
+
### Declared or default?
|
|
114
|
+
|
|
115
|
+
`origin: 'default'` **if and only if** you had no threshold to choose.
|
|
116
|
+
`complexity()` is default — "the cost does not grow" is a law, not a setting.
|
|
117
|
+
`resource()` makes you choose a ceiling, `forbidden()` and `metamorphic()` make you write a
|
|
118
|
+
business sentence: those are declared rules, and breaking one is a violation.
|
|
119
|
+
|
|
120
|
+
Four types are enough to cover the four real defects found in a 153k-line repository that
|
|
121
|
+
already passed 100% of its gates (typed lint, 90% coverage, mutation, e2e, API fuzzing).
|
|
122
|
+
|
|
123
|
+
## Spec format: no format, a rendering
|
|
124
|
+
|
|
125
|
+
- **Source of truth**: a `.rules.ts` next to the use case. Typed, imports the business
|
|
126
|
+
types, executable, completion for free, zero parser to write.
|
|
127
|
+
- **What the human reads**: the rendered page — one sentence per rule.
|
|
128
|
+
- The `intention` field is **mandatory**. A rule with no readable sentence is an error.
|
|
129
|
+
|
|
130
|
+
Writing format ≠ reading format.
|
|
131
|
+
|
|
132
|
+
## The Trojan horse
|
|
133
|
+
|
|
134
|
+
Some rules are universal and **need not be written**:
|
|
135
|
+
|
|
136
|
+
- the number of queries does not grow with the input size → **N+1**
|
|
137
|
+
- outgoing concurrency is bounded → **unbounded fan-out**
|
|
138
|
+
- `total` is invariant under `(limit, offset)` → **broken pagination**
|
|
139
|
+
|
|
140
|
+
Immediate value, zero spec to write. And it teaches the user what a rule is.
|
|
141
|
+
|
|
142
|
+
## Status: working MVP
|
|
143
|
+
|
|
144
|
+
One default rule — **complexity** — and its counting probe.
|
|
145
|
+
|
|
146
|
+
```sh
|
|
147
|
+
bun src/cli.ts examples/list-admin-users.rules.ts
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
```
|
|
151
|
+
✓ 1 rule held
|
|
152
|
+
⚠ 1 gap
|
|
153
|
+
|
|
154
|
+
gap 1/1 list-admin-users · current
|
|
155
|
+
“The number of SQL queries grows by ≈ 1 per item (1→1, 2→2, 4→4, 8→8, 16→16). No written
|
|
156
|
+
rule forbids it. Is this intended?”
|
|
157
|
+
|
|
158
|
+
[y] this is a bug [n] this is intended
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
The example faithfully reproduces the N+1 of a real-world monorepo
|
|
162
|
+
(`list-admin-users.ts:41`) and its fix. **speccov finds the first and validates the second,
|
|
163
|
+
without a single spec being written.** That is the success criterion we set ourselves: a
|
|
164
|
+
tool earns trust by first finding again what we already knew.
|
|
165
|
+
|
|
166
|
+
### What exists
|
|
167
|
+
|
|
168
|
+
- `complexity()`, `resource()`, `metamorphic()`, `forbidden()` — the four rule types
|
|
169
|
+
- `Counter` — the probe, ignorant of the domain
|
|
170
|
+
- `countingLogger()` — TypeORM adapter (`Logger` interface declared structurally, no
|
|
171
|
+
dependency on the ORM)
|
|
172
|
+
- `run()` / `report()` / `exitCode()` — execution, rendering, exit code
|
|
173
|
+
- CLI: `speccov <file.rules.ts>`
|
|
174
|
+
- `loadAnswers()` / `applyAnswers()` / `accept()` — the memory of the answers
|
|
175
|
+
- `Gauge` — the concurrency probe, counterpart of `Counter`
|
|
176
|
+
- `allOf()` — the cartesian product of a finite domain, from the simplest to the most twisted
|
|
177
|
+
- `speccov draft <file> --forbidden <Type>` — the domain written from the type
|
|
178
|
+
- 118 tests, strict typecheck, zero production dependency
|
|
179
|
+
|
|
180
|
+
## `speccov scan` — the static pass
|
|
181
|
+
|
|
182
|
+
No execution, no wiring, no rule to write. It reads and parses, that is all.
|
|
183
|
+
|
|
184
|
+
```sh
|
|
185
|
+
speccov scan apps/api/src --tsconfig apps/api/tsconfig.json
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
The report has **four zones**, and the fourth is the one that makes it honest:
|
|
189
|
+
|
|
190
|
+
| Zone | Where the guarantee comes from |
|
|
191
|
+
| -------------------------- | -------------------------------------------------------------- |
|
|
192
|
+
| **Proven by construction** | compiler flags, fixed-arity `Promise.all` |
|
|
193
|
+
| **Guaranteed by nobody** | flags switched off, and the candidate sites for each rule type |
|
|
194
|
+
| **Not measured** | what the pass did **not** look at, and why |
|
|
195
|
+
|
|
196
|
+
A report that shows only what it managed to verify, without saying what it could not, lies
|
|
197
|
+
by omission. That is the reproach made to a 90% coverage.
|
|
198
|
+
|
|
199
|
+
### What it detects
|
|
200
|
+
|
|
201
|
+
| Shape | Rule proposed |
|
|
202
|
+
| ------------------------------------------------------ | --------------------------------------------------------- |
|
|
203
|
+
| `Promise.all(collection.map(…))` | `resource()` — what concurrency ceiling? |
|
|
204
|
+
| a read (`find*`, `get*`, `load*`…) inside a `.map()` | `complexity()` — should the cost grow? |
|
|
205
|
+
| a `{ items, total }` return | `metamorphic()` — does `total` survive a change of page? |
|
|
206
|
+
| `Promise.all([a, b, c])` | **none** — bounded in writing, that is a guarantee |
|
|
207
|
+
| `const TABLES = […]` then `TABLES.map(…)` | **none** — the collection is fixed in the file |
|
|
208
|
+
| `const batch = xs.slice(i, i + n)` then `batch.map(…)` | **none** — the batch is bounded by its width |
|
|
209
|
+
| `X.map((i) => <button onClick={() => go(i)} />)` | **none** — the call goes out on click, not per item |
|
|
210
|
+
| `(await listDir(root)).map((e) => e.path)` | **none** — the promise is on the receiver: a batched read |
|
|
211
|
+
| `for (const x of xs) { await go(x) }` | `complexity()` — should the wait grow with the list? |
|
|
212
|
+
| `Promise.all([...inflight.values()])` | **none** — a drain: that work was started elsewhere |
|
|
213
|
+
| `import { KEYS }` then `KEYS.map(…)` | **none** — the width is written, one file over |
|
|
214
|
+
| `export function f(id: string) { await load(id) }` | `forbidden()`: what bounds that `string`? |
|
|
215
|
+
| `export function f(id: string) { return id.trim() }` | **none**: the value never leaves the process |
|
|
216
|
+
|
|
217
|
+
The last question bears on a type, not on a behaviour, and a codebase holds many more
|
|
218
|
+
signatures than fan-outs. Two things keep it from burying the rest. It is only asked of a
|
|
219
|
+
parameter that goes on to an asynchronous call, read one hop away: a value that merely
|
|
220
|
+
feeds a computation stays where the types still hold. And the report gives it a block of
|
|
221
|
+
its own, `Typed, not bounded`, with a total and the files where it concentrates. The JSON
|
|
222
|
+
still carries every site.
|
|
223
|
+
|
|
224
|
+
### The shape it used to be blind to
|
|
225
|
+
|
|
226
|
+
Every question above reads **concurrency**: how many calls leave at once, under what ceiling. A
|
|
227
|
+
serial `await` has a concurrency of one, so it triggered nothing at all:
|
|
228
|
+
|
|
229
|
+
```ts
|
|
230
|
+
for (const container of containers) {
|
|
231
|
+
await exec(runtime, ['rm', '-f', container], { timeoutMs: 30_000 }).catch(() => null);
|
|
232
|
+
}
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
The cost grows with the collection just as surely as a fan-out's, in time instead of sockets.
|
|
236
|
+
Measured on codesema and codesema-tools: **78 loops of this shape against 17 concurrency
|
|
237
|
+
sites** — the pass was silent on the more common of the two. That very loop sits two lines
|
|
238
|
+
under a `Promise.all` it was asking about wrongly.
|
|
239
|
+
|
|
240
|
+
Three filters keep it from being noise, because a serial loop is very often deliberate: the
|
|
241
|
+
collection must be unbounded, the awaited call must leave the process, and `Promise.all` is
|
|
242
|
+
left out since it carries its own question at its own address. Against 78 raw loops, 28 sites
|
|
243
|
+
are reported.
|
|
244
|
+
|
|
245
|
+
### It waits on the types it builds
|
|
246
|
+
|
|
247
|
+
`scan` builds a real `ts.Program` per scope, with module resolution: what a site means is read
|
|
248
|
+
from the types, not guessed from the shape. Two questions cannot be answered any other way.
|
|
249
|
+
|
|
250
|
+
**Is this promise being launched, or merely awaited?** `Promise.all(users.map((u) => sync(u)))`
|
|
251
|
+
opens one outgoing call per item. `Promise.all([...inflight.values()])` opens nothing at all:
|
|
252
|
+
the work is already running, and a ceiling here would bound nothing. Both hold N promises and
|
|
253
|
+
read the same by width, so speccov walks back to where the collection came from. Three shutdown
|
|
254
|
+
drains in codesema-tools used to be reported as fan-outs, and a ceiling on any of them would
|
|
255
|
+
have been a fix for a defect that was not there.
|
|
256
|
+
|
|
257
|
+
**How wide is a collection declared next door?** A name imported from another file is followed
|
|
258
|
+
to its declaration, and to the literal that gives it a width. The declaring file is read too:
|
|
259
|
+
`export const rows = []` filled by a `rows.push(…)` over there is an accumulator, and importing
|
|
260
|
+
it must not launder it.
|
|
261
|
+
|
|
262
|
+
What stays outside is what the program cannot show. A **parameter** keeps its question: the
|
|
263
|
+
width belongs to the callers, and an exported function has callers this scope may not even
|
|
264
|
+
contain. `runProbes(probes)` in codesema-tools is called twice, both times with a written list,
|
|
265
|
+
and it is still asked about — because nothing stops the third caller from passing a thousand.
|
|
266
|
+
|
|
267
|
+
The cost of that program is paid once per scope, and only for scopes that have something to
|
|
268
|
+
read: a generated scope declaring nothing but `.d.ts` files is skipped before the program is
|
|
269
|
+
built. Six of them were eating 15s of a 35s scan on codesema to type four files between them.
|
|
270
|
+
|
|
271
|
+
Test files are excluded: a fixture is not a production defect. But the exclusion is
|
|
272
|
+
**counted and displayed** — like everything the pass does not read (`.vue`, `.js`, `.mjs`).
|
|
273
|
+
Skipping in silence would be exactly the lie by omission that coverage is blamed for.
|
|
274
|
+
|
|
275
|
+
The `tsconfig.json` files are discovered by walking: a monorepo has one per app, of
|
|
276
|
+
differing strictness. The report says what is proven **in every** scope read, and names
|
|
277
|
+
those where a flag is missing.
|
|
278
|
+
|
|
279
|
+
And it never concludes beyond what it read: on that monorepo, 15 of the 19 `tsconfig.json`
|
|
280
|
+
have an unresolvable `extends`, so the report announces
|
|
281
|
+
`(reach: 4 of 19 scope(s) — see “Not measured”)` instead of saying "off everywhere".
|
|
282
|
+
|
|
283
|
+
Measured on codesema, 346 files typed across five scopes with their `node_modules` resolved:
|
|
284
|
+
**27s**, everything included, of which about 25 goes into building the programs. An earlier
|
|
285
|
+
figure of 0.78s on 1,566 files was measured where 15 of 19 `tsconfig.json` had an unresolvable
|
|
286
|
+
`extends`, so almost no program was ever built: it timed the refusal, not the work. Being
|
|
287
|
+
slower on a well-configured repository than on a broken one is the wrong way round, and it is
|
|
288
|
+
still true — a pre-commit hook is not the right home for this pass today.
|
|
289
|
+
|
|
290
|
+
### It does not conclude when it does not know
|
|
291
|
+
|
|
292
|
+
On a repository whose `node_modules` are not installed, an `extends` pointing at a workspace
|
|
293
|
+
package does not resolve — and **TypeScript then falls back to its defaults in silence**.
|
|
294
|
+
Concluding on that would produce ten false alarms on a project that is in fact fully strict.
|
|
295
|
+
|
|
296
|
+
`speccov` refuses: it says it could not resolve, and does not pronounce. The only diagnostic
|
|
297
|
+
it ignores is TS18003 ("no inputs were found"), by name, because it bears on the files to
|
|
298
|
+
compile and never on the `compilerOptions`.
|
|
299
|
+
|
|
300
|
+
### Setting a directory aside
|
|
301
|
+
|
|
302
|
+
A test harness or a browser-side client is real code, but not where a value enters the
|
|
303
|
+
system, and its sites drown the ones that matter. Nothing tells that from the code alone
|
|
304
|
+
without guessing by name. So the project says it, in a `.speccovignore` at its root:
|
|
305
|
+
|
|
306
|
+
```
|
|
307
|
+
# not where a value enters the system
|
|
308
|
+
tools/
|
|
309
|
+
apps/*/src/generated/**
|
|
310
|
+
*.actions.ts
|
|
311
|
+
```
|
|
312
|
+
|
|
313
|
+
The syntax is the subset of `.gitignore` a perimeter needs: `*`, `?`, `**`, comments, a
|
|
314
|
+
leading `/` to anchor, and a line without a slash matching at any depth. A negation (`!`)
|
|
315
|
+
is refused rather than misread. The file is found from the scanned directory upwards, and
|
|
316
|
+
its patterns are relative to where it sits, so the same line holds for `scan .` and for
|
|
317
|
+
`scan apps/api`.
|
|
318
|
+
|
|
319
|
+
What is set aside is counted, never silenced: the report ends with
|
|
320
|
+
`42 file(s) set aside by .speccovignore, 334 site(s) with them.`, and the JSON carries the
|
|
321
|
+
same two numbers under `notMeasured.ignored`.
|
|
322
|
+
|
|
323
|
+
## Accepting a gap promotes it to a rule
|
|
324
|
+
|
|
325
|
+
This is the heart of the product: **the human answer gets mechanised**.
|
|
326
|
+
|
|
327
|
+
```sh
|
|
328
|
+
speccov my.rules.ts --accept 'list-admin-users · current' --why "known N+1, ticket #4"
|
|
329
|
+
```
|
|
330
|
+
|
|
331
|
+
The threshold measured that day becomes a rule written into `.speccov/answers.json` — a file
|
|
332
|
+
**versioned and read in review**, not a cache:
|
|
333
|
+
|
|
334
|
+
```json
|
|
335
|
+
{
|
|
336
|
+
"version": 1,
|
|
337
|
+
"answers": [
|
|
338
|
+
{
|
|
339
|
+
"rule": "list-admin-users · current",
|
|
340
|
+
"magnitude": 1,
|
|
341
|
+
"detail": "The number of SQL queries grows by ≈ 1 per item (1→1, 2→2, …)",
|
|
342
|
+
"at": "2026-08-22",
|
|
343
|
+
"why": "known N+1, ticket #4"
|
|
344
|
+
}
|
|
345
|
+
]
|
|
346
|
+
}
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
Then:
|
|
350
|
+
|
|
351
|
+
| Situation | Report | Exit |
|
|
352
|
+
| --------------------------- | --------------------------------------------------------------------------------------- | ------- |
|
|
353
|
+
| The behaviour has not moved | `· 1 gap already accepted` — silenced but **counted** | `0` |
|
|
354
|
+
| It got better | same | `0` |
|
|
355
|
+
| **It got worse** | `✗ 1 violation — Accepted on 2026-08-22 at a magnitude of 1; the threshold is exceeded` | **`1`** |
|
|
356
|
+
|
|
357
|
+
Accepting is therefore not going quiet: it is **setting a threshold the build will defend**.
|
|
358
|
+
|
|
359
|
+
### Two deliberate refusals
|
|
360
|
+
|
|
361
|
+
- **An acceptance never hides.** It is counted in the report. A forgotten acceptance would
|
|
362
|
+
be exactly the blind spot the tool claims to fight.
|
|
363
|
+
- **An unreadable answers file throws.** Reading it as empty would make everything noisy;
|
|
364
|
+
reading it as "all accepted" would make it blind. Both are worse than a halt.
|
|
365
|
+
|
|
366
|
+
### Answering a site works the same way
|
|
367
|
+
|
|
368
|
+
`scan` is the cheapest entry point, so it is the one that ends up in CI and in an agent loop —
|
|
369
|
+
and it had nowhere to write an answer down. A site a human had ruled out came back identical at
|
|
370
|
+
the next pass, carrying the same weight as a real N+1.
|
|
371
|
+
|
|
372
|
+
```sh
|
|
373
|
+
speccov scan . --accept 'packages/cli/src/probes.ts:73' --why "called with a written list only"
|
|
374
|
+
```
|
|
375
|
+
|
|
376
|
+
```json
|
|
377
|
+
{
|
|
378
|
+
"version": 1,
|
|
379
|
+
"sites": [
|
|
380
|
+
{
|
|
381
|
+
"file": "packages/cli/src/probes.ts",
|
|
382
|
+
"kind": "unbounded-fanout",
|
|
383
|
+
"shape": "Promise.all(running)",
|
|
384
|
+
"at": "2026-08-26",
|
|
385
|
+
"why": "called with a written list only"
|
|
386
|
+
}
|
|
387
|
+
]
|
|
388
|
+
}
|
|
389
|
+
```
|
|
390
|
+
|
|
391
|
+
What is remembered is **the shape of the code, never the line**. A site moves down a file every
|
|
392
|
+
time something is inserted above it, and an answer keyed on a line number would expire on the
|
|
393
|
+
next commit, which is a way of remembering nothing. Reformatting is absorbed too, trailing
|
|
394
|
+
comma included.
|
|
395
|
+
|
|
396
|
+
The answer expires when that code changes, because it is then no longer the question anyone
|
|
397
|
+
answered — the same refusal as a threshold overrun above. And an answered site is **counted in
|
|
398
|
+
the report**, for the same reason an accepted gap is: a forgotten acceptance is the blind spot
|
|
399
|
+
this tool exists to refuse.
|
|
400
|
+
|
|
401
|
+
## On a real TypeORM
|
|
402
|
+
|
|
403
|
+
The adapter filters out the infrastructure, because TypeORM passes **everything** to the
|
|
404
|
+
logger:
|
|
405
|
+
|
|
406
|
+
```
|
|
407
|
+
SELECT version() handshake
|
|
408
|
+
SELECT current_schema() AS "currentSchema" same, other shape
|
|
409
|
+
START TRANSACTION / COMMIT transaction boundaries
|
|
410
|
+
SELECT … FROM "pg_tables" synchronize introspection
|
|
411
|
+
```
|
|
412
|
+
|
|
413
|
+
A 2-line `save()` emits **4 events for 1 logical write**. `isDataQuery()` sets them aside —
|
|
414
|
+
and stays **deliberately conservative**: under-counting would make N+1s _slip through_, and
|
|
415
|
+
that is the dangerous direction. When in doubt, a query counts.
|
|
416
|
+
|
|
417
|
+
The integration test runs against a real Postgres, and skips itself without one:
|
|
418
|
+
|
|
419
|
+
```sh
|
|
420
|
+
docker run -d --rm -e POSTGRES_PASSWORD=speccov -e POSTGRES_DB=speccov \
|
|
421
|
+
-p 55432:5432 docker.io/library/postgres:17-alpine
|
|
422
|
+
SPECCOV_PG_URL=postgres://postgres:speccov@localhost:55432/speccov bun test
|
|
423
|
+
```
|
|
424
|
+
|
|
425
|
+
It checks that one `findOneBy` per item is seen as an N+1 at ≈ 1 query per item, and that a
|
|
426
|
+
de-duplicated `findBy(In(…))` stays constant.
|
|
427
|
+
|
|
428
|
+
## `speccov draft` — the bridge
|
|
429
|
+
|
|
430
|
+
```sh
|
|
431
|
+
speccov draft apps/api/.../list-admin-users.ts:41
|
|
432
|
+
```
|
|
433
|
+
|
|
434
|
+
**The bridge is not generation, it is harvesting.** The wiring already exists: in the use
|
|
435
|
+
case (constructor, dependencies) and in the neighbouring spec (doubles, factories, argument
|
|
436
|
+
order). `draft` goes and fetches it, and writes by hand only what is left.
|
|
437
|
+
|
|
438
|
+
| What the subject needs | Where it comes from | Derived? |
|
|
439
|
+
| ------------------------------------ | --------------------------------------------- | -------- |
|
|
440
|
+
| The class and the constructor order | AST of the use case | ✅ |
|
|
441
|
+
| The double factories and their paths | `buildMock*` imports of the neighbouring spec | ✅ |
|
|
442
|
+
| The call to count | the `this.x.find…()` spotted in the `.map()` | ✅ |
|
|
443
|
+
| The public entry point | walked up from the private method | ✅ |
|
|
444
|
+
| The collection of size `size` | — | ❌ human |
|
|
445
|
+
| A valid command | — | ❌ human |
|
|
446
|
+
|
|
447
|
+
On the real site in that monorepo, **2 decisions** are left instead of a blank page.
|
|
448
|
+
|
|
449
|
+
### Two guarantees
|
|
450
|
+
|
|
451
|
+
**It refuses rather than guesses.** Two reads in the same `.map()`, a private method called
|
|
452
|
+
by two public methods, a class with no injected constructor: it names the ambiguity and
|
|
453
|
+
stops. A subject that measures the wrong thing would produce a reassuring, false green —
|
|
454
|
+
worse than no subject at all.
|
|
455
|
+
|
|
456
|
+
**An unfinished draft throws.** Every hole is a function that `throw`s. The subject cannot
|
|
457
|
+
run while it is incomplete, so it cannot lie.
|
|
458
|
+
|
|
459
|
+
### The public entry point
|
|
460
|
+
|
|
461
|
+
The `.map()` is often inside a private method. Calling it would be impossible and would
|
|
462
|
+
measure an internal cog instead of the use case: `draft` walks up to the public method that
|
|
463
|
+
calls it. If there are zero or several, it refuses.
|
|
464
|
+
|
|
465
|
+
### `draft --forbidden` — the domain written from the type
|
|
466
|
+
|
|
467
|
+
```sh
|
|
468
|
+
speccov draft src/identity/actor.ts --forbidden Actor
|
|
469
|
+
```
|
|
470
|
+
|
|
471
|
+
A forbidden rule cannot be spotted statically: it is a business sentence. But the **input
|
|
472
|
+
space** is in the type — and the compiler is there when the draft is written, whereas it
|
|
473
|
+
will be gone at run time.
|
|
474
|
+
|
|
475
|
+
```ts
|
|
476
|
+
cases: allOf<Actor>({
|
|
477
|
+
role: ["viewer", "org_admin", "platform_viewer", "platform_admin"],
|
|
478
|
+
adminAreaId: [missing('a value for adminAreaId: string | null'), null],
|
|
479
|
+
suspended: [false, true],
|
|
480
|
+
}),
|
|
481
|
+
```
|
|
482
|
+
|
|
483
|
+
| What the type knows | Written? |
|
|
484
|
+
| -------------------------------------------- | -------- |
|
|
485
|
+
| The members of a literal union | ✅ |
|
|
486
|
+
| `null` and `undefined` of an optional type | ✅ |
|
|
487
|
+
| Both faces of a boolean | ✅ |
|
|
488
|
+
| A value for a free `string` or `number` | ❌ human |
|
|
489
|
+
| The business sentence, and what is forbidden | ❌ human |
|
|
490
|
+
|
|
491
|
+
**The list is written into the file, not read on the fly.** A type read dynamically would
|
|
492
|
+
drop a case in silence the day a member disappears, and the rule would stay green. Written
|
|
493
|
+
down, the disappearance shows up in review. It is the same refusal of silence as everywhere
|
|
494
|
+
else.
|
|
495
|
+
|
|
496
|
+
The share the type does not bound becomes a `missing(…)` — a call that throws when the
|
|
497
|
+
module loads. The draft cannot run while it is incomplete.
|
|
498
|
+
|
|
499
|
+
## `speccov mutate` — the second engine
|
|
500
|
+
|
|
501
|
+
Rules catch the properties we know how to name. Mutation catches the rest: it alters the
|
|
502
|
+
code and watches whether **anything at all** objects.
|
|
503
|
+
|
|
504
|
+
```sh
|
|
505
|
+
speccov mutate src/scan --run "bun test" --limit 25
|
|
506
|
+
```
|
|
507
|
+
|
|
508
|
+
```
|
|
509
|
+
25 mutants tried, 11 killed.
|
|
510
|
+
⚠ 14 survivor(s) — that many behaviours nobody guarantees.
|
|
511
|
+
|
|
512
|
+
survivor 3/14 src/scan/guarantees.ts:18 (string emptied)
|
|
513
|
+
“I replaced 'useUnknownInCatchVariables' with "" and nothing
|
|
514
|
+
objected. Should this behaviour be guaranteed?”
|
|
515
|
+
```
|
|
516
|
+
|
|
517
|
+
**A surviving mutant is a gap in the strong sense**: the behaviour changed and neither rule
|
|
518
|
+
nor test flinched.
|
|
519
|
+
|
|
520
|
+
### Why not Stryker
|
|
521
|
+
|
|
522
|
+
The catalogue of operators comes from the state of the art (Stryker, PIT), but it is
|
|
523
|
+
reimplemented on the TypeScript AST that `scan` already parses. Three reasons:
|
|
524
|
+
|
|
525
|
+
- **The budget.** Stryker mutates everything and re-runs the whole suite. Here you choose
|
|
526
|
+
the scope and the limit: 25 mutants in 8.8s, compatible with a pre-push.
|
|
527
|
+
- **The granularity.** Stryker asks "is this mutant killed by a _test_?". speccov asks
|
|
528
|
+
"by _anything at all_ — rule or test?". That is not the same question.
|
|
529
|
+
- **The independence.** No inherited runner, config or plugin.
|
|
530
|
+
|
|
531
|
+
### Safety
|
|
532
|
+
|
|
533
|
+
Mutation writes into your files. Three protections:
|
|
534
|
+
|
|
535
|
+
- the restore is in a `finally`, with the original kept in memory — even a command that
|
|
536
|
+
crashes leaves the source intact;
|
|
537
|
+
- `speccov mutate` **refuses to run** outside a git repository, or on a scope that has
|
|
538
|
+
uncommitted changes: `git checkout` must stay a way back;
|
|
539
|
+
- the mutants generated beyond `--limit` are **counted and declared**, never silenced.
|
|
540
|
+
|
|
541
|
+
### It applied itself to itself
|
|
542
|
+
|
|
543
|
+
First pass on `src/scan`: **14 survivors out of 25**, all on the `FLAGS` table of
|
|
544
|
+
`guarantees.ts`. You could empty a flag's name or invert its `impliedByStrict` on 8 entries
|
|
545
|
+
out of 10 without a single test flinching.
|
|
546
|
+
|
|
547
|
+
The answer to a survivor is to write the missing rule — a complete truth table. Second pass
|
|
548
|
+
on the same scope: **25 mutants, 25 killed, no survivor.**
|
|
549
|
+
|
|
550
|
+
## Existing bricks — do not rewrite
|
|
551
|
+
|
|
552
|
+
- `fast-check` — generation and _shrinking_ (the minimal counterexample is what makes the report readable)
|
|
553
|
+
- `@stryker-mutator/core` — `--incremental` for the pre-push delta, JSON report in the
|
|
554
|
+
`mutation-testing-report-schema` schema
|
|
555
|
+
- No Zod 4 → fast-check bridge maintained to this day (`zod-fast-check` stopped in 2023):
|
|
556
|
+
go through `z.toJSONSchema()` for the width, hand-written arbitraries for the depth
|
|
557
|
+
|
|
558
|
+
## Decisions taken
|
|
559
|
+
|
|
560
|
+
| Date | Decision | Why |
|
|
561
|
+
| ---------- | ---------------------------------------------------------- | --------------------------------------------------------------------- |
|
|
562
|
+
| 2026-08-22 | A dependency of the codesema CLI, not a standalone product | YAGNI. Extraction kept free: separate repo + `bin` entry |
|
|
563
|
+
| 2026-08-22 | TypeScript / Bun, not Rust | The core has to **execute user TypeScript** (predicates, arbitraries) |
|
|
564
|
+
| 2026-08-22 | One language, TS | Premature multi-language has killed a hundred quality tools |
|
|
565
|
+
| 2026-08-22 | No dedicated domain name | No value until there is a standalone product |
|
|
566
|
+
| 2026-11-11 | Runs on Node as well as on Bun | A TypeScript project already has Node: `npx` is enough |
|
|
567
|
+
|
|
568
|
+
## Identified risks
|
|
569
|
+
|
|
570
|
+
1. **The slowness of mutation is adoption killer no. 1.** Prove < 10s in pre-push,
|
|
571
|
+
otherwise the hook gets disabled in the second week.
|
|
572
|
+
2. **The false positive is lethal** for a tool of trust. From the start it needs: equivalent
|
|
573
|
+
mutant, judged once by a human, remembered forever.
|
|
574
|
+
3. "Intent coverage" is **not a budget line**. It sells inside the budget that exists: "can
|
|
575
|
+
I trust the agent's PR?"
|
|
576
|
+
4. **The N+1 is the worst example to lead with.** It is the only one of the four defects
|
|
577
|
+
with mature competition — QueryGuard, Scout, and Prisma's built-in dataloader. The scope
|
|
578
|
+
defect of `AreaScopeService` has none: there, the code is correct, typed, tested, and
|
|
579
|
+
does the wrong thing. That is where nobody stands opposite.
|
|
580
|
+
5. **`draft` still refuses 62% of third-party sites**, and for named reasons. The biggest —
|
|
581
|
+
a `Promise.all(promises)` whose array is built elsewhere — is worth 58 sites on its own:
|
|
582
|
+
the collection is not at the site, there is nothing to wire without guessing.
|
|
583
|
+
6. **`lobehub` remains unscannable** past 12 GB. Consumption does not follow the number of
|
|
584
|
+
files, so it is not a problem of scale but of shape.
|
|
585
|
+
|
|
586
|
+
## Contributing
|
|
587
|
+
|
|
588
|
+
```bash
|
|
589
|
+
bun install # installs the dependencies and arms the git hooks
|
|
590
|
+
bun run verify # typecheck + lint + the whole suite, the same thing CI runs
|
|
591
|
+
```
|
|
592
|
+
|
|
593
|
+
**One prerequisite that `bun install` cannot cover: [gitleaks](https://github.com/gitleaks/gitleaks).**
|
|
594
|
+
It is a Go binary, not an npm package — the package named `gitleaks` on npm is unrelated to
|
|
595
|
+
the project. Install it from the official release, Homebrew, or your distribution. Without
|
|
596
|
+
it the `pre-commit` and `pre-push` hooks fail instead of passing silently: a secret scanner
|
|
597
|
+
that is absent must not read as a repository that is clean.
|
|
598
|
+
|
|
599
|
+
What the hooks do, and where:
|
|
600
|
+
|
|
601
|
+
| Hook | What runs |
|
|
602
|
+
| ------------ | --------------------------------------------------------------------- |
|
|
603
|
+
| `pre-commit` | `gitleaks protect --staged`, then prettier and oxlint on staged files |
|
|
604
|
+
| `commit-msg` | commitlint — Conventional Commits |
|
|
605
|
+
| `pre-push` | `gitleaks` over the unpushed commits, typecheck, and the test suite |
|
|
606
|
+
|
|
607
|
+
CI runs the same checks on every pull request, plus `publint` and a full-history secret
|
|
608
|
+
scan. The hooks are the fast loop; CI is the one that cannot be skipped with `--no-verify`.
|