@descent-vtt/spec-guard 0.16.0 → 0.17.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +68 -4
- package/README.md +503 -512
- package/dist/cites.d.ts.map +1 -1
- package/dist/cites.js +5 -1
- package/dist/cites.js.map +1 -1
- package/dist/cli.d.ts +3 -2
- package/dist/cli.d.ts.map +1 -1
- package/dist/cli.js +8 -4
- package/dist/cli.js.map +1 -1
- package/dist/glob.d.ts +25 -18
- package/dist/glob.d.ts.map +1 -1
- package/dist/glob.js +127 -136
- package/dist/glob.js.map +1 -1
- package/dist/impact.d.ts.map +1 -1
- package/dist/impact.js +12 -2
- package/dist/impact.js.map +1 -1
- package/dist/mcp.d.ts.map +1 -1
- package/dist/mcp.js +5 -3
- package/dist/mcp.js.map +1 -1
- package/dist/parser.d.ts +11 -9
- package/dist/parser.d.ts.map +1 -1
- package/dist/parser.js +250 -43
- package/dist/parser.js.map +1 -1
- package/dist/prove.d.ts.map +1 -1
- package/dist/prove.js +2 -1
- package/dist/prove.js.map +1 -1
- package/dist/reporter.d.ts +12 -4
- package/dist/reporter.d.ts.map +1 -1
- package/dist/reporter.js +200 -48
- package/dist/reporter.js.map +1 -1
- package/dist/runner.d.ts +12 -2
- package/dist/runner.d.ts.map +1 -1
- package/dist/runner.js +13 -4
- package/dist/runner.js.map +1 -1
- package/dist/types.d.ts +24 -2
- package/dist/types.d.ts.map +1 -1
- package/dist/vendor/spec-core/markdown/frontmatter.d.ts +11 -1
- package/dist/vendor/spec-core/markdown/frontmatter.d.ts.map +1 -1
- package/dist/vendor/spec-core/markdown/frontmatter.js +20 -3
- package/dist/vendor/spec-core/markdown/frontmatter.js.map +1 -1
- package/dist/vendor/spec-core/pattern/glob.d.ts +85 -0
- package/dist/vendor/spec-core/pattern/glob.d.ts.map +1 -1
- package/dist/vendor/spec-core/pattern/glob.js +207 -17
- package/dist/vendor/spec-core/pattern/glob.js.map +1 -1
- package/dist/vendor/spec-core/pattern/index.d.ts +2 -2
- package/dist/vendor/spec-core/pattern/index.d.ts.map +1 -1
- package/dist/vendor/spec-core/pattern/index.js +1 -1
- package/dist/vendor/spec-core/pattern/index.js.map +1 -1
- package/dist/vendor/spec-core/text/index.d.ts +1 -0
- package/dist/vendor/spec-core/text/index.d.ts.map +1 -1
- package/dist/vendor/spec-core/text/index.js +1 -0
- package/dist/vendor/spec-core/text/index.js.map +1 -1
- package/dist/vendor/spec-core/text/width.d.ts +31 -0
- package/dist/vendor/spec-core/text/width.d.ts.map +1 -0
- package/dist/vendor/spec-core/text/width.js +113 -0
- package/dist/vendor/spec-core/text/width.js.map +1 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -14,6 +14,12 @@ The retired payment gateway is gone from the service layer.
|
|
|
14
14
|
If someone reintroduces `LegacyPaymentGateway`, CI fails and points at the line
|
|
15
15
|
of the ADR that promised it was gone.
|
|
16
16
|
|
|
17
|
+
The words the spec-* tools share are defined in
|
|
18
|
+
[concepts](https://github.com/DescentVTT/spec-core/blob/main/docs/concepts.md),
|
|
19
|
+
and the [tutorial](https://github.com/DescentVTT/spec-core/blob/main/docs/tutorial.md)
|
|
20
|
+
runs spec-guard beside its siblings through one round of work on a small
|
|
21
|
+
repository.
|
|
22
|
+
|
|
17
23
|
---
|
|
18
24
|
|
|
19
25
|
## Why this exists
|
|
@@ -111,6 +117,47 @@ may use double or single quotes, and a bare attribute means `="true"`. Inside a
|
|
|
111
117
|
value, `\"`, `\'` and `\\` are escapes and any other backslash is kept, so a
|
|
112
118
|
regular expression is written as it reads: `symbol="\bTODO\b" regex="true"`.
|
|
113
119
|
|
|
120
|
+
There are eight kinds, and each makes one claim about the tree:
|
|
121
|
+
|
|
122
|
+
| Directive | Claim |
|
|
123
|
+
| --- | --- |
|
|
124
|
+
| `@assert-absence` | this text is gone, or occurs at most N times |
|
|
125
|
+
| `@assert-count` | this text occurs exactly, at least or at most N times |
|
|
126
|
+
| `@assert-import-absence` | no file in scope depends on this module |
|
|
127
|
+
| `@assert-import-count` | exactly, at least or at most N files depend on it |
|
|
128
|
+
| `@assert-layers` | dependencies point one way through these layers |
|
|
129
|
+
| `@assert-import-cycle` | no file depends on itself |
|
|
130
|
+
| `@assert-structure` | files are named by a pattern, directories hold their entries, files have their partners |
|
|
131
|
+
| `@assert-present` | these paths exist |
|
|
132
|
+
|
|
133
|
+
A directive is read only where a renderer shows Markdown. In code - fenced and
|
|
134
|
+
indented blocks, code spans, `<script>`, `<pre>`, `<style>` and `<textarea>` -
|
|
135
|
+
and in front matter it is text, so documentation that shows the syntax, like
|
|
136
|
+
this README, never executes it.
|
|
137
|
+
|
|
138
|
+
- **What is code is what CommonMark says it is**, decided by the Markdown
|
|
139
|
+
scanner every spec-* tool shares. Comments and code spans are resolved left
|
|
140
|
+
to right in one pass, so a backtick inside a comment is a character and
|
|
141
|
+
`<!--` inside a code span is code. A code span ends with its paragraph, so a
|
|
142
|
+
stray backtick cannot hide the directives after it, and a fence may be
|
|
143
|
+
indented with the list item it sits in.
|
|
144
|
+
[ADR-0002](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0002-directive-format.md)'s
|
|
145
|
+
amendment lists where this differs from 0.11.0.
|
|
146
|
+
- **What is masked is still counted.** A comment shaped like a directive -
|
|
147
|
+
`<!-- @assert-...` - in code, raw HTML or front matter is not run, and the
|
|
148
|
+
report says how many there were and where, on one line (every one under
|
|
149
|
+
`--verbose`, and in `maskedDirectives` in JSON), so text read as code by
|
|
150
|
+
accident is not a rule gone quiet without a word.
|
|
151
|
+
- **A block that never closes is a warning** on its opening line: a code fence
|
|
152
|
+
or raw-text block, which makes the rest of its document code, and front
|
|
153
|
+
matter that opens on the first line and never closes. None of that front
|
|
154
|
+
matter is read as front matter, its status is unrecognised and the document
|
|
155
|
+
stays in force, and the warning, on line 1, says so and says to close it
|
|
156
|
+
with `---`, or `+++` for TOML, on a line of its own. Neither fails a run,
|
|
157
|
+
under `--strict` either.
|
|
158
|
+
- **Line numbers stay exact.** Masking preserves every offset and line
|
|
159
|
+
terminator.
|
|
160
|
+
|
|
114
161
|
### `@assert-absence` - this symbol is gone
|
|
115
162
|
|
|
116
163
|
```md
|
|
@@ -120,179 +167,9 @@ regular expression is written as it reads: `symbol="\bTODO\b" regex="true"`.
|
|
|
120
167
|
```
|
|
121
168
|
|
|
122
169
|
Fails when the symbol occurs more than `expected` times (default `0`).
|
|
123
|
-
`max="..."` is accepted as a synonym for `expected`.
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
Most real rules are "nowhere except one place", not "not here":
|
|
128
|
-
|
|
129
|
-
```md
|
|
130
|
-
<!-- @assert-absence target="src" symbol="process.env" exclude="src/config/**" -->
|
|
131
|
-
```
|
|
132
|
-
|
|
133
|
-
That is the rule this README opens with - *no secret is ever read from
|
|
134
|
-
`process.env` outside `src/config`* - written as one assertion instead of a
|
|
135
|
-
hand-maintained list of every directory that is not `src/config`.
|
|
136
|
-
|
|
137
|
-
`exclude` follows gitignore rules, which are **not** the same as `glob`'s:
|
|
138
|
-
|
|
139
|
-
| Pattern | Excludes |
|
|
140
|
-
| --- | --- |
|
|
141
|
-
| `tests` | any directory or file named `tests`, at any depth, and everything under it |
|
|
142
|
-
| `/tests` | only the `tests` at the root, and everything under it |
|
|
143
|
-
| `src/config` | that directory and everything under it |
|
|
144
|
-
| `src/config/**` | the same, written explicitly |
|
|
145
|
-
| `*.test.ts` | any file with that name shape, at any depth |
|
|
146
|
-
|
|
147
|
-
The difference is deliberate. `glob="*.ts"` filters files, so basename matching
|
|
148
|
-
is what you want; `exclude="tests"` means the directory, because that is what
|
|
149
|
-
people mean when they write it - and it is what `rg -g '!tests'` does. Both
|
|
150
|
-
engines implement the same rule, and a parity matrix asserts they agree on it.
|
|
151
|
-
|
|
152
|
-
A leading `./` and a trailing `/` are dropped, so `./tests` and `tests/` are
|
|
153
|
-
`tests`, and a backslash is a separator. Four shapes are refused, because they
|
|
154
|
-
can never exclude anything:
|
|
155
|
-
- **`!`**, which in `.gitignore` re-includes a path. Pasted from one, it was
|
|
156
|
-
silently dropped and the exclusion left wider than it read.
|
|
157
|
-
- **`..`**, since nothing outside the root is searched.
|
|
158
|
-
- **a drive path** such as `C:/repo/dist`.
|
|
159
|
-
- **`.` or `/`**, the root itself.
|
|
160
|
-
|
|
161
|
-
So is any pattern the glob engine cannot read - an unclosed `[` or `{`, an
|
|
162
|
-
extended glob such as `+(a|b)`, a range that runs backwards, `**` inside a
|
|
163
|
-
name, one too large to compile - rather than being read as a literal that
|
|
164
|
-
excludes nothing. The same goes for `glob`, `module`, `order` and the
|
|
165
|
-
structure rules' patterns, and for a spec pattern.
|
|
166
|
-
|
|
167
|
-
`**` means any number of directories only as a whole segment. `docs/**.md` is
|
|
168
|
-
refused, with the two ways to say what was meant, written from the pattern:
|
|
169
|
-
`docs/**/*.md` for any depth, `docs/*.md` for one level. For any depth the
|
|
170
|
-
stars become a segment of their own, with a star kept on each side where the
|
|
171
|
-
name went on; for one level they become one star: `src/a**` is told
|
|
172
|
-
`src/a*/**` or `src/a*`. The tools it could have come from read it three ways,
|
|
173
|
-
and a spec pattern read the narrow way drops every nested document without a
|
|
174
|
-
word.
|
|
175
|
-
|
|
176
|
-
In a directive a refused pattern is an invalid directive, whether or not its
|
|
177
|
-
document is in force; in the configuration's `specs` or `exclude`, a spec
|
|
178
|
-
pattern on the command line or `--spec`, or `--exclude`, it is exit 2.
|
|
179
|
-
|
|
180
|
-
List attributes accept commas or whitespace, so both of these work:
|
|
181
|
-
|
|
182
|
-
```md
|
|
183
|
-
<!-- @assert-absence target="src" symbol="TODO" exclude="src/legacy/** tests" -->
|
|
184
|
-
<!-- @assert-absence target="src" symbol="TODO" exclude="src/legacy/**,tests" -->
|
|
185
|
-
```
|
|
186
|
-
|
|
187
|
-
(A path containing a space therefore cannot be written; there is no quoting
|
|
188
|
-
inside an attribute value. Nor can a brace group with a comma in it:
|
|
189
|
-
`glob="*.{ts,tsx}"` is the two patterns `*.{ts` and `tsx}`, and is refused with a
|
|
190
|
-
message that says so. Write `glob="*.ts, *.tsx"`. A configuration's `exclude` and
|
|
191
|
-
`specs` are JSON arrays, where a brace group keeps its commas, and a trailing
|
|
192
|
-
`/` on an alternative means what it means on a whole pattern: `{build/,dist}`
|
|
193
|
-
in `exclude` is both directories and what they hold, and `{docs/,README.md}`
|
|
194
|
-
in `specs` is everything under `docs`, and the README. A leading `/` on an
|
|
195
|
-
alternative anchors it at the root, as it anchors a whole pattern:
|
|
196
|
-
`{/build,dist}` in `exclude` is the `build` at the root, and every `dist`.
|
|
197
|
-
An alternative that names no path is refused, as it is written alone:
|
|
198
|
-
`{./,docs}` in `specs` is exit 2,
|
|
199
|
-
`the braces expand to "./", which names no path`.)
|
|
200
|
-
|
|
201
|
-
### `comments` - the note about a deletion is not the deletion
|
|
202
|
-
|
|
203
|
-
You delete a symbol and leave the explanation where the next person will look:
|
|
204
|
-
|
|
205
|
-
```ts
|
|
206
|
-
// LegacyThing was removed in ADR-398; do not reintroduce it.
|
|
207
|
-
```
|
|
208
|
-
|
|
209
|
-
A plain text search reads that comment as an occurrence, so the assertion that
|
|
210
|
-
keeps the symbol deleted fails on the sentence proving it was deleted. Both
|
|
211
|
-
fixes are bad: delete the note and lose the reason, or delete the rule.
|
|
212
|
-
|
|
213
|
-
So matches inside comments do not count:
|
|
214
|
-
|
|
215
|
-
```md
|
|
216
|
-
<!-- comments are ignored by default -->
|
|
217
|
-
<!-- @assert-absence target="src" symbol="LegacyThing" -->
|
|
218
|
-
|
|
219
|
-
<!-- ...unless you ask for them -->
|
|
220
|
-
<!-- @assert-absence target="src" symbol="Copyright" comments="include" -->
|
|
221
|
-
```
|
|
222
|
-
|
|
223
|
-
`comments="include"` is right for assertions that really are about text — a
|
|
224
|
-
licence header, or a name that must appear nowhere in the repository at all.
|
|
225
|
-
|
|
226
|
-
Because this is the one thing that can turn a failing run green without anyone
|
|
227
|
-
touching code, a run that passed this way says so:
|
|
228
|
-
|
|
229
|
-
```
|
|
230
|
-
⚠ 1 match inside comments was not counted; add comments="include" to count it
|
|
231
|
-
|
|
232
|
-
1 passed · 7ms
|
|
233
|
-
✔ every spec assertion holds
|
|
234
|
-
```
|
|
235
|
-
|
|
236
|
-
Comment syntax is known for around 68 extensions across 11 families (JS/TS, C,
|
|
237
|
-
C#, Rust, Go, Python-style `#`, shell `#`, YAML, SQL-style `--`, markup -
|
|
238
|
-
MSBuild project files included - and formats with no comments at all). Strings
|
|
239
|
-
are tracked too, because `//` inside a URL is not a comment and reading it as
|
|
240
|
-
one would hide real code. So are JavaScript's regular expressions, because the
|
|
241
|
-
quote in `/["']/` is not a quote. And so is what only looks like either: a Rust
|
|
242
|
-
lifetime (`&'static str`), a C++ digit separator (`100'000`), a `#` inside a
|
|
243
|
-
shell word (`${#items[@]}`), an apostrophe in a YAML value (`name: it's fine`)
|
|
244
|
-
and a `/*` in JSX text (`<div>/*</div>`) all open nothing. Where spec-guard is
|
|
245
|
-
unsure —
|
|
246
|
-
an unknown extension, an unterminated literal — the text counts as code, and the
|
|
247
|
-
report says which files it could not classify. A match wrongly kept is a visible
|
|
248
|
-
failure you can argue with; a match wrongly dropped is a lie.
|
|
249
|
-
|
|
250
|
-
## What gets searched
|
|
251
|
-
|
|
252
|
-
An assertion is worth exactly as much as the set of files behind it, so
|
|
253
|
-
spec-guard is explicit about that set and never quietly narrows it.
|
|
254
|
-
|
|
255
|
-
**Four directory names are skipped**, and nothing else:
|
|
256
|
-
|
|
257
|
-
| Skipped | Why |
|
|
258
|
-
| --- | --- |
|
|
259
|
-
| `.git`, `.hg`, `.svn` | version-control stores hold compressed copies of code you deleted on purpose |
|
|
260
|
-
| `node_modules` | code you did not write, which your architecture rules are not about |
|
|
261
|
-
|
|
262
|
-
Everything else is searched. That includes **hidden directories** - `.github`,
|
|
263
|
-
`.husky`, `.claude-rules`, `.agents` - because that is where CI, hooks and agent
|
|
264
|
-
rules live, and a rule that cannot see your workflow files is not enforcing much.
|
|
265
|
-
It also includes `dist`, `build`, `out` and `coverage`, because spec-guard
|
|
266
|
-
cannot tell build output from a directory of build scripts, and guessing wrong
|
|
267
|
-
means a rule silently stops covering anything.
|
|
268
|
-
|
|
269
|
-
`.gitignore` is not consulted. It describes what git should carry, not what a
|
|
270
|
-
rule covers - and ripgrep applies it only inside a git repository, so honouring
|
|
271
|
-
it made the same tree answer differently depending on whether a `.git` directory
|
|
272
|
-
happened to exist above it.
|
|
273
|
-
|
|
274
|
-
To narrow scope, say so in the assertion:
|
|
275
|
-
|
|
276
|
-
```md
|
|
277
|
-
<!-- @assert-absence target="src" symbol="TODO" exclude="dist coverage" -->
|
|
278
|
-
```
|
|
279
|
-
|
|
280
|
-
`--no-default-skips` removes even those four, for a run that has to be certain.
|
|
281
|
-
|
|
282
|
-
### Nothing is skipped quietly
|
|
283
|
-
|
|
284
|
-
A file spec-guard could not read, or one whose bytes are not text but which
|
|
285
|
-
contained the symbol anyway, is a gap in the answer rather than a detail of it.
|
|
286
|
-
Those are reported, and `--strict` fails on them:
|
|
287
|
-
|
|
288
|
-
```text
|
|
289
|
-
✖ docs/adr.md:3 @assert-absence
|
|
290
|
-
"ApiKey" must not appear in .
|
|
291
|
-
expected no matches, found 0, and 1 file could not be inspected
|
|
292
|
-
⚠ 1 match in 1 binary file not counted: build/app.bin
|
|
293
|
-
```
|
|
294
|
-
|
|
295
|
-
The counts are in `--json` too, as `skipped` on each result.
|
|
170
|
+
`max="..."` is accepted as a synonym for `expected`. A bound turns the rule into
|
|
171
|
+
a budget: "at most 5 TODOs" is an absence claim, and burning a budget down is a
|
|
172
|
+
real workflow. A lower bound, `min`, is `@assert-count`'s.
|
|
296
173
|
|
|
297
174
|
### `@assert-count` - this symbol occurs exactly / at least / at most N times
|
|
298
175
|
|
|
@@ -388,12 +265,15 @@ are reported rather than counted as clean:
|
|
|
388
265
|
```
|
|
389
266
|
|
|
390
267
|
The count is still true of everything that could be seen; the warning is what
|
|
391
|
-
stops it being mistaken for a complete answer.
|
|
392
|
-
`--verbose` too, since a green run is
|
|
393
|
-
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
268
|
+
stops it being mistaken for a complete answer.
|
|
269
|
+
- **A passing rule prints it** without `--verbose` too, since a green run is
|
|
270
|
+
when it would otherwise go unread.
|
|
271
|
+
- **`--strict` turns those warnings into failures.**
|
|
272
|
+
- **Files in scope in a language whose imports spec-guard cannot read are
|
|
273
|
+
counted and reported**, so a rule pointed at the wrong tree says "analysed 2
|
|
274
|
+
of 3 files; 1 is in a language whose imports spec-guard cannot read (.razor)"
|
|
275
|
+
rather than quietly passing - and if *none* of them can be read, the
|
|
276
|
+
assertion fails rather than passing on an empty analysis.
|
|
397
277
|
|
|
398
278
|
[ADR-0005](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0005-import-assertions.md) has the measurements and the
|
|
399
279
|
reasoning behind each boundary.
|
|
@@ -694,7 +574,140 @@ Passes when every listed path exists relative to `--root`. Directories count.
|
|
|
694
574
|
| `reason` | all | Human-readable justification, printed on failure |
|
|
695
575
|
|
|
696
576
|
Unknown attributes are an error, not a shrug: `expct="1"` fails the run instead
|
|
697
|
-
of silently asserting nothing.
|
|
577
|
+
of silently asserting nothing. A spec tool whose typos assert nothing is worse
|
|
578
|
+
than no spec tool.
|
|
579
|
+
|
|
580
|
+
### `exclude` - everywhere except
|
|
581
|
+
|
|
582
|
+
Most real rules are "nowhere except one place", not "not here":
|
|
583
|
+
|
|
584
|
+
```md
|
|
585
|
+
<!-- @assert-absence target="src" symbol="process.env" exclude="src/config/**" -->
|
|
586
|
+
```
|
|
587
|
+
|
|
588
|
+
That is the rule this README opens with - *no secret is ever read from
|
|
589
|
+
`process.env` outside `src/config`* - written as one assertion instead of a
|
|
590
|
+
hand-maintained list of every directory that is not `src/config`.
|
|
591
|
+
|
|
592
|
+
`exclude` follows gitignore rules, which are **not** the same as `glob`'s:
|
|
593
|
+
|
|
594
|
+
| Pattern | Excludes |
|
|
595
|
+
| --- | --- |
|
|
596
|
+
| `tests` | any directory or file named `tests`, at any depth, and everything under it |
|
|
597
|
+
| `/tests` | only the `tests` at the root, and everything under it |
|
|
598
|
+
| `src/config` | that directory and everything under it |
|
|
599
|
+
| `src/config/**` | the same, written explicitly |
|
|
600
|
+
| `*.test.ts` | any file with that name shape, at any depth |
|
|
601
|
+
|
|
602
|
+
The difference is deliberate. `glob="*.ts"` filters files, so basename matching
|
|
603
|
+
is what you want; `exclude="tests"` means the directory, because that is what
|
|
604
|
+
people mean when they write it - and it is what `rg -g '!tests'` does. Both
|
|
605
|
+
engines implement the same rule, and a parity matrix asserts they agree on it.
|
|
606
|
+
|
|
607
|
+
A leading `./`, with the slashes after it, and a trailing `/` are dropped, so
|
|
608
|
+
`./tests`, `.//tests` and `tests/` are `tests`, and a backslash is a separator. Four shapes are refused, because they
|
|
609
|
+
can never exclude anything:
|
|
610
|
+
- **`!`**, which in `.gitignore` re-includes a path. Pasted from one, it was
|
|
611
|
+
silently dropped and the exclusion left wider than it read.
|
|
612
|
+
- **`..`**, since nothing outside the root is searched.
|
|
613
|
+
- **a drive path** such as `C:/repo/dist`.
|
|
614
|
+
- **`.` or `/`**, the root itself.
|
|
615
|
+
|
|
616
|
+
So is any pattern the glob engine cannot read - an unclosed `[` or `{`, an
|
|
617
|
+
extended glob such as `+(a|b)`, a range that runs backwards, `**` inside a
|
|
618
|
+
name, one too large to compile - rather than being read as a literal that
|
|
619
|
+
excludes nothing. The same goes for `glob`, `module`, `order` and the
|
|
620
|
+
structure rules' patterns, and for a spec pattern.
|
|
621
|
+
|
|
622
|
+
`**` means any number of directories only as a whole segment. `docs/**.md` is
|
|
623
|
+
refused, with the two ways to say what was meant, written from the pattern:
|
|
624
|
+
`docs/**/*.md` for any depth, `docs/*.md` for one level. For any depth the
|
|
625
|
+
stars become a segment of their own, with a star kept on each side where the
|
|
626
|
+
name went on; for one level they become one star: `src/a**` is told
|
|
627
|
+
`src/a*/**` or `src/a*`. The tools it could have come from read it three ways,
|
|
628
|
+
and a spec pattern read the narrow way drops every nested document without a
|
|
629
|
+
word.
|
|
630
|
+
|
|
631
|
+
In a directive a refused pattern is an invalid directive, whether or not its
|
|
632
|
+
document is in force; in the configuration's `specs` or `exclude`, a spec
|
|
633
|
+
pattern on the command line or `--spec`, or `--exclude`, it is exit 2.
|
|
634
|
+
|
|
635
|
+
List attributes accept commas or whitespace, so both of these work:
|
|
636
|
+
|
|
637
|
+
```md
|
|
638
|
+
<!-- @assert-absence target="src" symbol="TODO" exclude="src/legacy/** tests" -->
|
|
639
|
+
<!-- @assert-absence target="src" symbol="TODO" exclude="src/legacy/**,tests" -->
|
|
640
|
+
```
|
|
641
|
+
|
|
642
|
+
Two things therefore cannot be written in an attribute:
|
|
643
|
+
- **A path containing a space**: there is no quoting inside an attribute value.
|
|
644
|
+
- **A brace group with a comma in it**: `glob="*.{ts,tsx}"` is the two patterns
|
|
645
|
+
`*.{ts` and `tsx}`, and is refused with a message that says so. Write
|
|
646
|
+
`glob="*.ts, *.tsx"`.
|
|
647
|
+
|
|
648
|
+
A configuration's `exclude` and `specs` are JSON arrays, where a brace group
|
|
649
|
+
keeps its commas. Inside one, an alternative reads as it would written alone:
|
|
650
|
+
- **A trailing `/`** means what it means on a whole pattern: `{build/,dist}` in
|
|
651
|
+
`exclude` is both directories and what they hold, and `{docs/,README.md}` in
|
|
652
|
+
`specs` is everything under `docs`, and the README.
|
|
653
|
+
- **A leading `/`** anchors it at the root, as it anchors a whole pattern:
|
|
654
|
+
`{/build,dist}` in `exclude` is the `build` at the root, and every `dist`.
|
|
655
|
+
After a `./` it anchors nothing, as in `.//build`: `./{/build,dist}` is
|
|
656
|
+
every `build`.
|
|
657
|
+
- **An alternative that names no path is refused**: `{./,docs}` in `specs` is
|
|
658
|
+
exit 2, `the braces expand to "./", which names no path`.
|
|
659
|
+
|
|
660
|
+
### `comments` - the note about a deletion is not the deletion
|
|
661
|
+
|
|
662
|
+
You delete a symbol and leave the explanation where the next person will look:
|
|
663
|
+
|
|
664
|
+
```ts
|
|
665
|
+
// LegacyThing was removed in ADR-398; do not reintroduce it.
|
|
666
|
+
```
|
|
667
|
+
|
|
668
|
+
A plain text search reads that comment as an occurrence, so the assertion that
|
|
669
|
+
keeps the symbol deleted fails on the sentence proving it was deleted. Both
|
|
670
|
+
fixes are bad: delete the note and lose the reason, or delete the rule.
|
|
671
|
+
|
|
672
|
+
So matches inside comments do not count:
|
|
673
|
+
|
|
674
|
+
```md
|
|
675
|
+
<!-- comments are ignored by default -->
|
|
676
|
+
<!-- @assert-absence target="src" symbol="LegacyThing" -->
|
|
677
|
+
|
|
678
|
+
<!-- ...unless you ask for them -->
|
|
679
|
+
<!-- @assert-absence target="src" symbol="Copyright" comments="include" -->
|
|
680
|
+
```
|
|
681
|
+
|
|
682
|
+
`comments="include"` is right for assertions that really are about text — a
|
|
683
|
+
licence header, or a name that must appear nowhere in the repository at all.
|
|
684
|
+
|
|
685
|
+
Because this is the one thing that can turn a failing run green without anyone
|
|
686
|
+
touching code, a run that passed this way says so:
|
|
687
|
+
|
|
688
|
+
```
|
|
689
|
+
⚠ 1 match inside comments was not counted; add comments="include" to count it
|
|
690
|
+
|
|
691
|
+
1 passed · 7ms
|
|
692
|
+
✔ every spec assertion holds
|
|
693
|
+
```
|
|
694
|
+
|
|
695
|
+
Comment syntax is known for around 68 extensions across 11 families (JS/TS, C,
|
|
696
|
+
C#, Rust, Go, Python-style `#`, shell `#`, YAML, SQL-style `--`, markup -
|
|
697
|
+
MSBuild project files included - and formats with no comments at all). What
|
|
698
|
+
would hide real code if misread is tracked too:
|
|
699
|
+
- **Strings**, because `//` inside a URL is not a comment.
|
|
700
|
+
- **JavaScript's regular expressions**, because the quote in `/["']/` is not a
|
|
701
|
+
quote.
|
|
702
|
+
- **What only looks like either**: a Rust lifetime (`&'static str`), a C++ digit
|
|
703
|
+
separator (`100'000`), a `#` inside a shell word (`${#items[@]}`), an
|
|
704
|
+
apostrophe in a YAML value (`name: it's fine`) and a `/*` in JSX text
|
|
705
|
+
(`<div>/*</div>`) all open nothing.
|
|
706
|
+
|
|
707
|
+
Where spec-guard is unsure — an unknown extension, an unterminated literal — the
|
|
708
|
+
text counts as code, and the report says which files it could not classify. A
|
|
709
|
+
match wrongly kept is a visible failure you can argue with; a match wrongly
|
|
710
|
+
dropped is a lie.
|
|
698
711
|
|
|
699
712
|
### `allow-empty` - an assertion that covers nothing is a failure
|
|
700
713
|
|
|
@@ -719,6 +732,11 @@ nothing yet is the honest state of the world - a rule written before the code
|
|
|
719
732
|
it guards - `allow-empty="true"` says so, and `--allow-empty-scope` says it for
|
|
720
733
|
a whole run.
|
|
721
734
|
|
|
735
|
+
A `target` that does not exist fails the run too: an assertion pointed at a
|
|
736
|
+
renamed directory would otherwise search nothing, find nothing and report
|
|
737
|
+
success. `--allow-missing-targets` makes it a warning, for a repository where a
|
|
738
|
+
path is legitimately optional.
|
|
739
|
+
|
|
722
740
|
### `baseline` - adopting a rule the codebase already breaks
|
|
723
741
|
|
|
724
742
|
A strict rule introduced into a mature codebase lands on violations that already
|
|
@@ -759,6 +777,66 @@ paste in. It prints; it does not edit. [ADR-0009](https://github.com/DescentVTT/
|
|
|
759
777
|
explains why that distinction is the whole design, and why there is no `--fix`
|
|
760
778
|
for architecture rules.
|
|
761
779
|
|
|
780
|
+
## What gets searched
|
|
781
|
+
|
|
782
|
+
An assertion is worth exactly as much as the set of files behind it, so
|
|
783
|
+
spec-guard is explicit about that set and never quietly narrows it.
|
|
784
|
+
|
|
785
|
+
**Four directory names are skipped**, and nothing else:
|
|
786
|
+
|
|
787
|
+
| Skipped | Why |
|
|
788
|
+
| --- | --- |
|
|
789
|
+
| `.git`, `.hg`, `.svn` | version-control stores hold compressed copies of code you deleted on purpose |
|
|
790
|
+
| `node_modules` | code you did not write, which your architecture rules are not about |
|
|
791
|
+
|
|
792
|
+
Everything else is searched. That includes **hidden directories** - `.github`,
|
|
793
|
+
`.husky`, `.claude-rules`, `.agents` - because that is where CI, hooks and agent
|
|
794
|
+
rules live, and a rule that cannot see your workflow files is not enforcing much.
|
|
795
|
+
It also includes `dist`, `build`, `out` and `coverage`, because spec-guard
|
|
796
|
+
cannot tell build output from a directory of build scripts, and guessing wrong
|
|
797
|
+
means a rule silently stops covering anything.
|
|
798
|
+
|
|
799
|
+
`.gitignore` is not consulted. It describes what git should carry, not what a
|
|
800
|
+
rule covers - and ripgrep applies it only inside a git repository, so honouring
|
|
801
|
+
it made the same tree answer differently depending on whether a `.git` directory
|
|
802
|
+
happened to exist above it.
|
|
803
|
+
|
|
804
|
+
To narrow scope, say so in the assertion:
|
|
805
|
+
|
|
806
|
+
```md
|
|
807
|
+
<!-- @assert-absence target="src" symbol="TODO" exclude="dist coverage" -->
|
|
808
|
+
```
|
|
809
|
+
|
|
810
|
+
`--no-default-skips` removes even those four, for a run that has to be certain.
|
|
811
|
+
|
|
812
|
+
### Spec files are left out of their own searches
|
|
813
|
+
|
|
814
|
+
An ADR that says "`LegacyGateway` must not appear" contains the string
|
|
815
|
+
`LegacyGateway`, so without this every absence rule would fail on the document
|
|
816
|
+
asserting it - the single most confusing possible first-run experience.
|
|
817
|
+
- **`--include-specs`** restores the naive behaviour, and counts the prose of
|
|
818
|
+
your specs.
|
|
819
|
+
- **The directives themselves stay uncounted** even then: a directive is an
|
|
820
|
+
HTML comment, and a rule whose own text trips it can never be satisfied.
|
|
821
|
+
`comments="include"` counts even those.
|
|
822
|
+
- **`@assert-structure`**, which reads names rather than text, is the one rule
|
|
823
|
+
that counts the spec files.
|
|
824
|
+
|
|
825
|
+
### Nothing is skipped quietly
|
|
826
|
+
|
|
827
|
+
A file spec-guard could not read, or one whose bytes are not text but which
|
|
828
|
+
contained the symbol anyway, is a gap in the answer rather than a detail of it.
|
|
829
|
+
Those are reported, and `--strict` fails on them:
|
|
830
|
+
|
|
831
|
+
```text
|
|
832
|
+
✖ docs/adr.md:3 @assert-absence
|
|
833
|
+
"ApiKey" must not appear in .
|
|
834
|
+
expected no matches, found 0, and 1 file could not be inspected
|
|
835
|
+
⚠ 1 match in 1 binary file not counted: build/app.bin
|
|
836
|
+
```
|
|
837
|
+
|
|
838
|
+
The counts are in `--json` too, as `skipped` on each result.
|
|
839
|
+
|
|
762
840
|
## Document status - drafts and superseded ADRs
|
|
763
841
|
|
|
764
842
|
An ADR has a life. It is `Proposed` before anyone agrees to it, and `Superseded`
|
|
@@ -772,7 +850,7 @@ status is `draft`, `proposed`, `rejected`, `deprecated`, `superseded` or
|
|
|
772
850
|
the record, and its directives, which stated the round's premises and goals,
|
|
773
851
|
are history.
|
|
774
852
|
|
|
775
|
-
|
|
853
|
+
Four spellings are recognised, because four are in use:
|
|
776
854
|
|
|
777
855
|
<!-- Shown fenced on purpose: spec-guard masks code before reading a status. -->
|
|
778
856
|
|
|
@@ -788,30 +866,82 @@ status: proposed
|
|
|
788
866
|
Accepted (0.3.0).
|
|
789
867
|
```
|
|
790
868
|
|
|
869
|
+
```md
|
|
870
|
+
| Status | Accepted |
|
|
871
|
+
| ------ | -------- |
|
|
872
|
+
```
|
|
873
|
+
|
|
791
874
|
```md
|
|
792
875
|
**Status:** accepted
|
|
793
876
|
```
|
|
794
877
|
|
|
795
|
-
Front-matter wins, then the `## Status` section, then
|
|
796
|
-
|
|
797
|
-
|
|
798
|
-
|
|
799
|
-
|
|
800
|
-
|
|
801
|
-
|
|
802
|
-
|
|
803
|
-
|
|
804
|
-
|
|
805
|
-
|
|
806
|
-
and the label are
|
|
807
|
-
|
|
808
|
-
|
|
809
|
-
|
|
810
|
-
|
|
811
|
-
|
|
812
|
-
|
|
813
|
-
|
|
814
|
-
|
|
878
|
+
Front-matter wins, then the `## Status` section, then a table, then the bold
|
|
879
|
+
label, and the first of them that is there decides:
|
|
880
|
+
|
|
881
|
+
- **Front matter** is YAML between `---` lines or TOML between `+++` lines. Of
|
|
882
|
+
TOML, only the top-level `status` key is read, and only as a string on one
|
|
883
|
+
line: `status = "accepted"` or `status = 'accepted'`. Front matter without a
|
|
884
|
+
`status` key, YAML or TOML, hands over to the section, the table and the
|
|
885
|
+
label.
|
|
886
|
+
- **A section's value** is its first line of prose before the next heading.
|
|
887
|
+
Headings are read as CommonMark reads them, so an underlined `Status` is the
|
|
888
|
+
section and a `## Status` kept in a comment is not.
|
|
889
|
+
- **The table and the label** are only read above the first section heading.
|
|
890
|
+
A table is read when it has exactly two columns and one of its rows, the
|
|
891
|
+
header row or another, names the status in its left cell - `Status` or
|
|
892
|
+
`State`, in any case, bold or not - and gives it in the right. A table of
|
|
893
|
+
more columns, or one further down, is a register or a legend of other
|
|
894
|
+
documents and is never read for this one's status.
|
|
895
|
+
- **A status that cannot be read still decides**: the document stays in force,
|
|
896
|
+
the report warns on its line with the reason, and nothing ranked below it is
|
|
897
|
+
read in its place. In front matter that is a value such as
|
|
898
|
+
`status: "accepted" (2024-05-01)`, text after a closing quote, or in TOML a
|
|
899
|
+
multi-line string, an array, a table or a value that is not a string; in a
|
|
900
|
+
section, a table or a label, an empty section or a line that begins with no
|
|
901
|
+
word.
|
|
902
|
+
- **Front matter that opens on the first line and never closes** is read the
|
|
903
|
+
same way: none of it can be read, so the status is unrecognised, the document
|
|
904
|
+
stays in force, the section and the label are not read in its place, and the
|
|
905
|
+
report warns on line 1 and says how to close it.
|
|
906
|
+
- **Anything else** - `Provisional`, `In review`, a misspelled `Supersedded`, or
|
|
907
|
+
no status at all - keeps enforcing.
|
|
908
|
+
|
|
909
|
+
That asymmetry is deliberate: an unanticipated word that keeps enforcing is a
|
|
910
|
+
visible failure with an obvious fix, while one that stops enforcing is a green
|
|
911
|
+
build over a rule nobody is checking.
|
|
912
|
+
|
|
913
|
+
A status may be written in Chinese, Traditional or Simplified, and the key as
|
|
914
|
+
`狀態` or `状态` - a heading, a table's cell, a label with `:` or a full-width
|
|
915
|
+
`:`, or a front-matter key with YAML's colon. YAML ends a key at an ASCII colon
|
|
916
|
+
and at nothing else, so `状态:草稿` in front matter is a status that cannot be
|
|
917
|
+
read: the document stays in force, and the warning says to write `状态:`. A
|
|
918
|
+
Chinese word is read as the English word it translates, and the English then
|
|
919
|
+
does what it does here: only a word for one of the six above withholds a
|
|
920
|
+
document, and the report shows the line as written. These are translations of
|
|
921
|
+
the list, not words added to it:
|
|
922
|
+
|
|
923
|
+
| Read as | Traditional | Simplified |
|
|
924
|
+
| --- | --- | --- |
|
|
925
|
+
| superseded | 已被取代, 被取代, 已取代, and 被 ... 取代, 替代 or 取而代之 within 30 characters | the same |
|
|
926
|
+
| deprecated | 已棄用, 棄用, 已廢棄, 廢棄, 已停用, 已過時 | 已弃用, 弃用, 已废弃, 废弃, 已停用, 已过时 |
|
|
927
|
+
| rejected | 已否決, 否決, 已拒絕, 不採納 | 已否决, 否决, 已拒绝, 不采纳 |
|
|
928
|
+
| archived | 封存, 已封存, 歸檔, 已歸檔 | 封存, 已封存, 归档, 已归档 |
|
|
929
|
+
| draft | 草稿, 草案 | 草稿, 草案 |
|
|
930
|
+
| proposed | 提議, 提案, 審查中, 審核中, 討論中, 待審, 待審核 | 提议, 提案, 审查中, 审核中, 讨论中, 待审, 待审核 |
|
|
931
|
+
| withdrawn, in force | 已撤回, 撤回, 已作廢, 作廢 | 已撤回, 撤回, 已作废, 作废 |
|
|
932
|
+
| deferred, in force | 延後, 暫緩, 擱置 | 延后, 暂缓, 搁置 |
|
|
933
|
+
| final, in force | 已定案, 定案, 已凍結 | 已定案, 定案, 已冻结 |
|
|
934
|
+
| provisionally (accepted), in force | 暫定 | 暂定 |
|
|
935
|
+
| accepted, in force | 已接受, 接受, 已採納, 採納, 已核准, 核准, 已批准, 批准, 已生效, 生效 | 已接受, 接受, 已采纳, 采纳, 已核准, 核准, 已批准, 批准, 已生效, 生效 |
|
|
936
|
+
| implemented, in force | 已實施, 已完成 | 已实施, 已完成 |
|
|
937
|
+
|
|
938
|
+
A word is read at the start of the value, and only when a space, punctuation or
|
|
939
|
+
the end follows it: `草稿已核准` and `暫定接受` are no word listed. Nor is a word
|
|
940
|
+
after a negation - `未接受`, `尚未核准`, `不再生效` - or `已取代` before a
|
|
941
|
+
document reference, `已取代 ADR-0002`, which usually names the document this
|
|
942
|
+
one supersedes; `已接受(取代 ADR-0002)` is accepted. A Chinese value that is
|
|
943
|
+
no word listed keeps its document in force, with a warning, as any value that
|
|
944
|
+
cannot be read does.
|
|
815
945
|
|
|
816
946
|
Withholding is never quiet:
|
|
817
947
|
|
|
@@ -828,6 +958,15 @@ found on the day it is written rather than on the day it is accepted.
|
|
|
828
958
|
`--ignore-status` runs everything, which is how you ask whether a draft would
|
|
829
959
|
pass if you accepted it today.
|
|
830
960
|
|
|
961
|
+
A run in which nothing is in force executes no assertion, and says so rather
|
|
962
|
+
than that every assertion holds: `no assertion was executed, so nothing was
|
|
963
|
+
verified`. Under `--strict`, or `"strict": true`, that run fails, exit 1, as
|
|
964
|
+
the family contract has a check that measured nothing do: JSON says
|
|
965
|
+
`"ok": false` and `"nothingVerified": true`, and SARIF, GitHub and GitLab carry
|
|
966
|
+
a `nothing-verified` finding. One rule in force is enough to pass, whatever else
|
|
967
|
+
is withheld; and the MCP server's check of paths that no rule governs verified
|
|
968
|
+
what it was asked to, and passes.
|
|
969
|
+
|
|
831
970
|
[ADR-0010](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0010-spec-status.md) has the full reasoning, including why
|
|
832
971
|
there is no per-directive `if-status` attribute.
|
|
833
972
|
|
|
@@ -1047,11 +1186,17 @@ in the file name:
|
|
|
1047
1186
|
from the document's status line - `Superseded by ADR-0014` - and followed to
|
|
1048
1187
|
the one in force.
|
|
1049
1188
|
|
|
1050
|
-
|
|
1051
|
-
|
|
1052
|
-
|
|
1053
|
-
|
|
1054
|
-
|
|
1189
|
+
| Found | Exit |
|
|
1190
|
+
| --- | --- |
|
|
1191
|
+
| a ghost citation | 1 |
|
|
1192
|
+
| a stale citation | 0, as a warning; 1 under `--strict` |
|
|
1193
|
+
| a file read in part | 0; 1 under `--strict` |
|
|
1194
|
+
| a check that looked for nothing or read no source file | 0; 1 under `--strict`, and every format then places it as a `nothing-read` finding, an error, with what to do |
|
|
1195
|
+
| a family whose files match no document | 2 |
|
|
1196
|
+
|
|
1197
|
+
`--json` (versioned by `formatVersion`), `--format sarif`, `--format github`
|
|
1198
|
+
and `--format gitlab` carry the same findings; `spec-guard cites src lib/a.ts`
|
|
1199
|
+
reads only those paths.
|
|
1055
1200
|
[ADR-0017](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0017-citations-in-comments.md) has the design, and what is
|
|
1056
1201
|
out of scope: links to symbols, which need a parser the family does not ship.
|
|
1057
1202
|
|
|
@@ -1099,14 +1244,17 @@ src/db/client.ts
|
|
|
1099
1244
|
- **Listed, never guessed**: an import naming no file, an alias such as `@/db`,
|
|
1100
1245
|
a dynamic `import(name)`. Any of them may depend on the path.
|
|
1101
1246
|
|
|
1102
|
-
A directory asks about every file under it.
|
|
1103
|
-
|
|
1104
|
-
|
|
1105
|
-
|
|
1106
|
-
|
|
1107
|
-
|
|
1108
|
-
|
|
1109
|
-
|
|
1247
|
+
- **A directory** asks about every file under it.
|
|
1248
|
+
- **`--depth <n>`** stops `n` imports away; cycles are safe.
|
|
1249
|
+
- **`--ignore-status`** lists rules from documents not in force, which are
|
|
1250
|
+
otherwise counted, as `query` does.
|
|
1251
|
+
- **`--json`** is versioned by `formatVersion`.
|
|
1252
|
+
- **Exit 2** for a path that does not exist, and for a run of `impact` in which
|
|
1253
|
+
no spec matched, as `query` does: the dependents are still written, but "no
|
|
1254
|
+
rules" over no specs is nothing measured. `--allow-empty` asks for the
|
|
1255
|
+
dependents alone, and exits 0.
|
|
1256
|
+
- **The MCP server** answers the same question as `get_dependents`, with the
|
|
1257
|
+
same document.
|
|
1110
1258
|
[ADR-0018](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0018-impact.md) has the table and what it leaves out.
|
|
1111
1259
|
|
|
1112
1260
|
## CLI
|
|
@@ -1131,7 +1279,7 @@ spec-guard impact <paths...> # the files that depend on each path, and
|
|
|
1131
1279
|
| `--json` | Machine-readable report on stdout (same as `--format json`), versioned by `formatVersion` |
|
|
1132
1280
|
| `--format <human\|json\|sarif\|github\|gitlab>` | Output format. `sarif` uploads to GitHub code scanning, `github` annotates a pull request from the job's log, `gitlab` is a GitLab Code Quality report |
|
|
1133
1281
|
| `--engine <auto\|rg\|js>` | Search engine (default `auto`: scanner for small trees, ripgrep for big ones) |
|
|
1134
|
-
| `--strict` / `--no-strict` | Treat analysis that could not be completed as a failure |
|
|
1282
|
+
| `--strict` / `--no-strict` | Treat analysis that could not be completed as a failure, and a run that executed no assertion: its specs matched and state no rule in force |
|
|
1135
1283
|
| `--allow-missing-targets` / `--no-allow-missing-targets` | Warn instead of failing when a `target` path does not exist |
|
|
1136
1284
|
| `--allow-empty-scope` / `--no-allow-empty-scope` | Warn instead of failing when an assertion inspects no files |
|
|
1137
1285
|
| `--print-baseline` | Print the `baseline="..."` that would exempt today's violations, and exit |
|
|
@@ -1237,17 +1385,20 @@ updates the status line and nothing else:
|
|
|
1237
1385
|
watching 15 specs · 1 change · 21 of 60 rules re-executed · 21 ms · Enter re-runs everything, Ctrl+C stops
|
|
1238
1386
|
```
|
|
1239
1387
|
|
|
1240
|
-
A session re-executes only the rules whose directive or inputs changed
|
|
1241
|
-
filesystem read goes through one module
|
|
1242
|
-
read what.
|
|
1243
|
-
|
|
1244
|
-
|
|
1245
|
-
|
|
1246
|
-
|
|
1247
|
-
|
|
1248
|
-
|
|
1249
|
-
|
|
1250
|
-
|
|
1388
|
+
A session re-executes only the rules whose directive or inputs changed:
|
|
1389
|
+
- **Every filesystem read goes through one module**, so the session can record
|
|
1390
|
+
which rule read what.
|
|
1391
|
+
- **A watcher's event only evicts what it may have changed**, and the evicted
|
|
1392
|
+
facts are read again and compared, so noise costs one directory read, and a
|
|
1393
|
+
renamed directory's contents need no events of their own.
|
|
1394
|
+
- **Work that is a pure function of a file's bytes** - tokenizing, comment
|
|
1395
|
+
masking, scanning - is remembered by the bytes' hash.
|
|
1396
|
+
|
|
1397
|
+
Each of those is a way to report a tree that no longer exists, so a test
|
|
1398
|
+
changes trees at random and requires every report a session gives to equal a
|
|
1399
|
+
fresh run's, with six deliberately broken sessions that it has to catch.
|
|
1400
|
+
[ADR-0014](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0014-configuration-and-watch.md)
|
|
1401
|
+
has the design, the measurements and the limits.
|
|
1251
1402
|
|
|
1252
1403
|
On this repository, from a change to the report, in the quietest of four runs
|
|
1253
1404
|
(a plain warm run took 69 ms in it):
|
|
@@ -1272,10 +1423,15 @@ symbolic link, and anything outside the root. CI stays the authority.
|
|
|
1272
1423
|
|
|
1273
1424
|
### Exit codes
|
|
1274
1425
|
|
|
1426
|
+
"Your specs failed" and "spec-guard could not run" are different facts, and CI
|
|
1427
|
+
should be able to tell them apart, so they are 1 and 2:
|
|
1428
|
+
|
|
1275
1429
|
| Code | Meaning |
|
|
1276
1430
|
| --- | --- |
|
|
1277
1431
|
| `0` | Every assertion held |
|
|
1278
|
-
| `1` | An assertion failed, or a directive was malformed;
|
|
1432
|
+
| `1` | An assertion failed, or a directive was malformed; under `--strict`, the specs matched state no rule in force, so no assertion was executed |
|
|
1433
|
+
| `1`, `prove` | A rule survived, or under `--strict` none was proved |
|
|
1434
|
+
| `1`, `cites` | A comment cites a document that does not exist, or under `--strict` a citation is stale, a file was read in part, or no source file was read |
|
|
1279
1435
|
| `2` | spec-guard could not run: bad usage, a malformed configuration, no spec files matched, `--engine rg` with no ripgrep, a watch that could not start, a `cites` family whose files match no document |
|
|
1280
1436
|
| `130` | A `--watch` session was stopped |
|
|
1281
1437
|
|
|
@@ -1306,7 +1462,21 @@ repository is large enough that you want ripgrep's speed, install it first:
|
|
|
1306
1462
|
- run: sudo apt-get update && sudo apt-get install -y ripgrep
|
|
1307
1463
|
```
|
|
1308
1464
|
|
|
1309
|
-
|
|
1465
|
+
Elsewhere, install the package your image or machine has, before spec-guard
|
|
1466
|
+
runs:
|
|
1467
|
+
|
|
1468
|
+
| Where | Install |
|
|
1469
|
+
| --- | --- |
|
|
1470
|
+
| GitHub-hosted Ubuntu runner | `sudo apt-get update && sudo apt-get install -y ripgrep` |
|
|
1471
|
+
| GitLab runner, Debian or Ubuntu image (`node:22`) | `apt-get update && apt-get install -y ripgrep` in `before_script` |
|
|
1472
|
+
| GitLab runner, Alpine image (`node:22-alpine`) | `apk add --no-cache ripgrep` in `before_script` |
|
|
1473
|
+
| Windows | `winget install BurntSushi.ripgrep.MSVC` |
|
|
1474
|
+
|
|
1475
|
+
spec-guard looks for `rg` on `PATH`. `SPEC_GUARD_RG=/path/to/rg` names the
|
|
1476
|
+
binary instead, for one that is not on `PATH` or not called `rg`. Under the
|
|
1477
|
+
default `--engine auto`, a binary that cannot be run leaves the built-in
|
|
1478
|
+
scanner to do the work, and the report's engine says `javascript`; under
|
|
1479
|
+
`--engine rg` it is exit 2.
|
|
1310
1480
|
|
|
1311
1481
|
As a pre-commit hook (assuming a local install, so the bare command resolves):
|
|
1312
1482
|
|
|
@@ -1320,26 +1490,38 @@ npx spec-guard "docs/**/*.md" --fail-fast
|
|
|
1320
1490
|
comment on the offending line:
|
|
1321
1491
|
|
|
1322
1492
|
```yaml
|
|
1493
|
+
permissions:
|
|
1494
|
+
security-events: write
|
|
1495
|
+
# only in a private repository
|
|
1496
|
+
actions: read
|
|
1497
|
+
contents: read
|
|
1498
|
+
steps:
|
|
1499
|
+
# ...checkout and setup-node as above
|
|
1323
1500
|
- run: npx @descent-vtt/spec-guard "docs/**/*.md" --format sarif > spec-guard.sarif || true
|
|
1324
|
-
- uses: github/codeql-action/upload-sarif@
|
|
1501
|
+
- uses: github/codeql-action/upload-sarif@v4
|
|
1325
1502
|
with:
|
|
1326
1503
|
sarif_file: spec-guard.sarif
|
|
1327
1504
|
category: spec-guard
|
|
1328
1505
|
```
|
|
1329
1506
|
|
|
1330
|
-
|
|
1331
|
-
|
|
1332
|
-
|
|
1333
|
-
|
|
1334
|
-
|
|
1335
|
-
|
|
1507
|
+
Version 4 of the upload action is the current one; version 3 is deprecated in
|
|
1508
|
+
December 2026. The job needs `security-events: write` to upload, and in a
|
|
1509
|
+
private repository `actions: read` and `contents: read` as well.
|
|
1510
|
+
|
|
1511
|
+
One alert per broken rule rather than per match - the thing that broke is the
|
|
1512
|
+
rule - anchored on the first offending line, with the directive as a related
|
|
1513
|
+
location because that is often where the fix goes. Alerts carry a fingerprint
|
|
1514
|
+
derived from what the assertion is about rather than where its matches landed,
|
|
1515
|
+
so inserting a line above a violation does not close the alert and open a new
|
|
1516
|
+
one; a directive that cannot be read is known by the directive as written, not
|
|
1517
|
+
by the message, which a release may word differently.
|
|
1336
1518
|
|
|
1337
1519
|
Two formats need no upload step. `--format github` writes one GitHub Actions
|
|
1338
1520
|
workflow command per finding, which the job's log turns into an annotation on
|
|
1339
1521
|
the diff:
|
|
1340
1522
|
|
|
1341
1523
|
```text
|
|
1342
|
-
::error file=src/a.ts,line=2,title=assert-absence::"Legacy" must not appear in src: expected no matches, found 2 (docs/adr/0004-legacy.md:12)
|
|
1524
|
+
::error file=src/a.ts,line=2,title=assert-absence::"Legacy" must not appear in src: expected no matches, found 2 (docs/adr/0004-legacy.md:12). fix the code, or the rule if the decision it records has changed
|
|
1343
1525
|
```
|
|
1344
1526
|
|
|
1345
1527
|
`--format gitlab` writes a
|
|
@@ -1363,13 +1545,24 @@ Every format places the same findings, and `prove` and `cites` write all three:
|
|
|
1363
1545
|
| a rule that survived `prove`, on its directive | `error` | `critical` |
|
|
1364
1546
|
| a ghost citation (`cites`), on its comment | `error` | `critical` |
|
|
1365
1547
|
| a directive that could not be read | `error` | `major` |
|
|
1548
|
+
| a run or a proof under `--strict` that verified nothing (`nothing-verified`), on the first spec's first line | `error` | `major` |
|
|
1549
|
+
| a `cites` check under `--strict` that read no source file (`nothing-read`), on the first spec's first line | `error` | `major` |
|
|
1366
1550
|
| a stale citation (`cites`) | `warning`; `error` under `--strict` | `minor`; `critical` under `--strict` |
|
|
1367
1551
|
| a rule `prove` could make no violation for | `notice` | `minor` |
|
|
1552
|
+
| a document read differently from how it was written (`spec-warning`: a status that cannot be read, front matter or a block never closed), on its line | `warning` | `minor` |
|
|
1368
1553
|
| a file `cites` could read only in part | `notice` | `info` |
|
|
1369
1554
|
| a document not in force, on its first line | `notice` | `info` |
|
|
1370
1555
|
|
|
1371
|
-
A
|
|
1372
|
-
|
|
1556
|
+
A finding's GitLab `description`, and its GitHub message, is its message and
|
|
1557
|
+
then its hint, the next action.
|
|
1558
|
+
|
|
1559
|
+
A GitLab issue's `fingerprint` is the SHA-256 of what the finding is about -
|
|
1560
|
+
its rule, its file and its subject, such as the assertion, the id a comment
|
|
1561
|
+
cites or the directive that cannot be read, as written - with a count for a
|
|
1562
|
+
second finding of the same. It is never taken from the message, the hint or the
|
|
1563
|
+
line: GitLab compares it across pipelines to tell a new finding from an old
|
|
1564
|
+
one, and a message that holds a count, or a line moved down a page, would make
|
|
1565
|
+
every open issue look fixed and a new one appear.
|
|
1373
1566
|
|
|
1374
1567
|
**There is no language server, and that is a decision rather than a gap.**
|
|
1375
1568
|
spec-guard's claims are about a whole repository - "this symbol appears nowhere
|
|
@@ -1549,128 +1742,70 @@ documents of the spec set read last. A single run gains nothing from one.
|
|
|
1549
1742
|
|
|
1550
1743
|
## Design decisions
|
|
1551
1744
|
|
|
1552
|
-
This
|
|
1553
|
-
|
|
1554
|
-
|
|
1555
|
-
|
|
1556
|
-
|
|
1557
|
-
|
|
1558
|
-
|
|
1559
|
-
|
|
1560
|
-
|
|
1561
|
-
|
|
1562
|
-
|
|
1563
|
-
|
|
1564
|
-
|
|
1565
|
-
|
|
1566
|
-
[
|
|
1567
|
-
|
|
1568
|
-
|
|
1569
|
-
|
|
1570
|
-
|
|
1571
|
-
|
|
1572
|
-
|
|
1573
|
-
|
|
1574
|
-
|
|
1575
|
-
|
|
1576
|
-
|
|
1577
|
-
|
|
1578
|
-
|
|
1579
|
-
|
|
1580
|
-
|
|
1581
|
-
|
|
1582
|
-
|
|
1583
|
-
|
|
1584
|
-
|
|
1585
|
-
|
|
1586
|
-
|
|
1587
|
-
|
|
1588
|
-
|
|
1589
|
-
|
|
1590
|
-
|
|
1591
|
-
|
|
1592
|
-
|
|
1593
|
-
|
|
1594
|
-
|
|
1595
|
-
one
|
|
1596
|
-
|
|
1597
|
-
|
|
1598
|
-
|
|
1599
|
-
|
|
1600
|
-
|
|
1601
|
-
|
|
1602
|
-
|
|
1603
|
-
|
|
1604
|
-
|
|
1605
|
-
|
|
1606
|
-
|
|
1607
|
-
|
|
1608
|
-
|
|
1609
|
-
|
|
1610
|
-
|
|
1611
|
-
|
|
1612
|
-
|
|
1613
|
-
|
|
1614
|
-
|
|
1615
|
-
|
|
1616
|
-
through containment (`Primary` / `PrimaryButton`) and dovetailing (`abc` / `cd`
|
|
1617
|
-
in `abcd`), so spec-guard checked for both and fell back to separate passes.
|
|
1618
|
-
Since 0.4.0 it does not need the proof, because ripgrep no longer counts
|
|
1619
|
-
anything - it answers only *which files contain this text*, and the scanner
|
|
1620
|
-
counts each pattern separately over the shared file contents. The batching
|
|
1621
|
-
checks were deleted along with the risk they guarded.
|
|
1622
|
-
|
|
1623
|
-
**Globs are matched by an automaton, not a regular expression.** Every
|
|
1624
|
-
pattern - `glob`, `exclude`, `module`, a layer, a spec pattern - is read by the
|
|
1625
|
-
glob engine the spec-* tools share, copied from spec-core and checked by hash.
|
|
1626
|
-
A `RegExp` backtracks: `*-*-*-*-*-*x` took 55 seconds to fail one long file
|
|
1627
|
-
name. The automaton cannot, and ripgrep is handed each pattern spelled as the
|
|
1628
|
-
automaton reads it ([ADR-0015](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0015-globs-from-spec-core.md)).
|
|
1629
|
-
|
|
1630
|
-
**Exit code 2 exists.** "Your specs failed" and "spec-guard could not run" are
|
|
1631
|
-
different facts, and CI should be able to tell them apart.
|
|
1632
|
-
|
|
1633
|
-
**A missing `target` fails the run.** It used to warn and search what was left,
|
|
1634
|
-
which meant an assertion pointed at a renamed directory searched nothing, found
|
|
1635
|
-
nothing, and reported success — the exact shape of a green check that verified
|
|
1636
|
-
nothing. `--allow-missing-targets` restores the old behaviour for repositories
|
|
1637
|
-
where a path is legitimately optional.
|
|
1638
|
-
|
|
1639
|
-
**Hidden directories are searched, and `.gitignore` is not consulted.** An
|
|
1640
|
-
audit of 0.3.0 found the tool reporting a clean pass on a repository whose
|
|
1641
|
-
forbidden symbol sat in `.github/workflows/ci.yml`; the scanner and ripgrep also
|
|
1642
|
-
disagreed with each other, finding two matches and four on the same tree. Scope
|
|
1643
|
-
is now one policy that both engines are driven by, the skip list is four names
|
|
1644
|
-
long, and anything spec-guard could not inspect is reported rather than assumed
|
|
1645
|
-
clean ([ADR-0007](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0007-search-scope.md)).
|
|
1646
|
-
|
|
1647
|
-
**Comments are excluded by default, and the exclusion is reported.** Counting
|
|
1648
|
-
the note that records a deletion as an occurrence of the thing deleted punishes
|
|
1649
|
-
the documentation this tool exists to keep honest ([ADR-0006](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0006-comment-classification.md)).
|
|
1650
|
-
The reverse risk — a rule that quietly stops checking anything because every
|
|
1651
|
-
match now sits in a comment — is why every run says how many matches it dropped.
|
|
1652
|
-
|
|
1653
|
-
**An assertion that inspected no files fails.** It is the same defect as a
|
|
1654
|
-
missing `target` seen from a different angle: a rule whose scope is empty passes
|
|
1655
|
-
forever and reads exactly like a rule that found nothing. Turning this on found
|
|
1656
|
-
a vacuous assertion inside this repository's own test suite on the first run.
|
|
1657
|
-
`allow-empty="true"` covers the honest case of a rule written before the code it
|
|
1658
|
-
guards.
|
|
1659
|
-
|
|
1660
|
-
**Four more languages, and no parser.** Import assertions read Python, Go, Rust
|
|
1661
|
-
and C# as well as JavaScript. Not by adding four tokenizers - by reusing the
|
|
1662
|
-
comment and string lexer that already existed, masking the source with it, and
|
|
1663
|
-
reading statements off what is left. Tree-sitter would have been the modern
|
|
1664
|
-
answer and costs 94 MB unpacked against this package's 0.33 MB; the four things
|
|
1665
|
-
a real parser would genuinely see that this cannot are listed in
|
|
1666
|
-
[ADR-0008](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0008-polyglot-imports.md) rather than glossed over.
|
|
1667
|
-
|
|
1668
|
-
**There is no `--fix`.** Every edit a machine can make to a failing boundary
|
|
1669
|
-
assertion is an edit that records the rule no longer holding: widen the bound,
|
|
1670
|
-
add an exclusion, append to the baseline, insert an ignore comment. A one-flag
|
|
1671
|
-
path from red to green is a bad button for a person and a much worse one for an
|
|
1672
|
-
agent whose loop terminates on a green build. `--print-baseline` prints what you
|
|
1673
|
-
could paste; it does not paste it ([ADR-0009](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0009-debt-baselines.md)).
|
|
1745
|
+
This README says what spec-guard does. Why it does it that way, what was
|
|
1746
|
+
measured and what was left out are in its ADRs, one decision each:
|
|
1747
|
+
|
|
1748
|
+
- [ADR-0001](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0001-invariants.md) -
|
|
1749
|
+
spec-guard's own invariants are directives, executed against it on every CI
|
|
1750
|
+
run: output through an injected `CliIO`, processes spawned by the engine
|
|
1751
|
+
alone, a parser and a reporter that never touch the filesystem.
|
|
1752
|
+
- [ADR-0002](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0002-directive-format.md) -
|
|
1753
|
+
directives are HTML comments, invisible when rendered; code is masked before
|
|
1754
|
+
they are read, by spec-core's Markdown scanner, and spec files are left out
|
|
1755
|
+
of their own searches.
|
|
1756
|
+
- [ADR-0003](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0003-mutation-testing.md) -
|
|
1757
|
+
mutation testing in CI gates the build, every survivor is replayed rather
|
|
1758
|
+
than excused, and vitest is pinned to 4.x until Stryker's runner reads 5.
|
|
1759
|
+
- [ADR-0004](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0004-adaptive-engine.md) -
|
|
1760
|
+
`--engine auto` chooses per search group, by walking the target with a
|
|
1761
|
+
budget: the scanner under it, ripgrep over it.
|
|
1762
|
+
- [ADR-0005](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0005-import-assertions.md) -
|
|
1763
|
+
import assertions read the dependency rather than the text, and report what
|
|
1764
|
+
they could not resolve rather than counting it clean.
|
|
1765
|
+
- [ADR-0006](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0006-comment-classification.md) -
|
|
1766
|
+
matches inside comments do not count by default, every run says how many it
|
|
1767
|
+
dropped, and where spec-guard is unsure, text is code.
|
|
1768
|
+
- [ADR-0007](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0007-search-scope.md) -
|
|
1769
|
+
one scope policy drives both engines: four directory names are skipped,
|
|
1770
|
+
`.gitignore` is not read, ripgrep only pre-filters, and every other skip is
|
|
1771
|
+
reported.
|
|
1772
|
+
- [ADR-0008](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0008-polyglot-imports.md) -
|
|
1773
|
+
Python, Go, Rust and C# imports are read by masking comments and strings and
|
|
1774
|
+
reading statements off the rest, with no parser.
|
|
1775
|
+
- [ADR-0009](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0009-debt-baselines.md) -
|
|
1776
|
+
a baseline names known violations and ratchets both ways; `--print-baseline`
|
|
1777
|
+
prints one, and there is no `--fix`.
|
|
1778
|
+
- [ADR-0010](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0010-spec-status.md) -
|
|
1779
|
+
a document's lifecycle status decides whether its directives run, withholding
|
|
1780
|
+
is always reported, and there is no per-directive `if-status`.
|
|
1781
|
+
- [ADR-0011](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0011-layers-and-cycles.md) -
|
|
1782
|
+
layers and cycles are read off one import graph; a cycle is counted as a
|
|
1783
|
+
knot, and `tsconfig` paths are not resolved.
|
|
1784
|
+
- [ADR-0012](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0012-query-and-mcp.md) -
|
|
1785
|
+
`query` and the MCP server answer which rules govern a path from the specs
|
|
1786
|
+
alone, held to the walk a real run makes, with no MCP SDK.
|
|
1787
|
+
- [ADR-0013](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0013-structure-assertions.md) -
|
|
1788
|
+
one structure directive, one claim each - names, required entries or
|
|
1789
|
+
partners - with names compared exactly on every platform.
|
|
1790
|
+
- [ADR-0014](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0014-configuration-and-watch.md) -
|
|
1791
|
+
a project's policy lives in `package.json` or `.spec-guard.json`, and a watch
|
|
1792
|
+
session re-executes only what a change can affect.
|
|
1793
|
+
- [ADR-0015](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0015-globs-from-spec-core.md) -
|
|
1794
|
+
globs are spec-core's automata, copied in and held by hash, so no pattern
|
|
1795
|
+
backtracks.
|
|
1796
|
+
- [ADR-0016](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0016-rules-seen-to-fail.md) -
|
|
1797
|
+
a rule is trusted when it has been seen to fail: `prove` shows each one a
|
|
1798
|
+
violation of itself, in memory.
|
|
1799
|
+
- [ADR-0017](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0017-citations-in-comments.md) -
|
|
1800
|
+
a comment that cites a decision is a claim, and `cites` holds it to a
|
|
1801
|
+
document that exists and is in force.
|
|
1802
|
+
- [ADR-0018](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0018-impact.md) -
|
|
1803
|
+
`impact` reads the import graph backwards, with the rules in play, and lists
|
|
1804
|
+
what it could not follow.
|
|
1805
|
+
- [ADR-0019](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0019-releases-are-staged-by-ci.md) -
|
|
1806
|
+
a release is a `v*` tag on main after its full mutation sweep, staged on npm
|
|
1807
|
+
by CI with provenance and published only when the maintainer approves it with
|
|
1808
|
+
a second factor.
|
|
1674
1809
|
|
|
1675
1810
|
## spec-guard checks itself
|
|
1676
1811
|
|
|
@@ -1683,162 +1818,18 @@ build fails if they stop being true. CI proves every one of those rules can
|
|
|
1683
1818
|
fail, and `spec-guard cites` holds every ADR a comment in the code cites to
|
|
1684
1819
|
existing and being in force.
|
|
1685
1820
|
|
|
1821
|
+
Line coverage is 100%, CI's full mutation sweep kills 99.16% of 12,407 mutants
|
|
1822
|
+
(0.17.0), and every survivor a release adds is replayed before it ships
|
|
1823
|
+
([ADR-0003](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0003-mutation-testing.md));
|
|
1824
|
+
[CONTRIBUTING.md](https://github.com/DescentVTT/spec-guard/blob/main/CONTRIBUTING.md)
|
|
1825
|
+
has how to build, test and release, and
|
|
1826
|
+
[SECURITY.md](https://github.com/DescentVTT/spec-guard/blob/main/SECURITY.md)
|
|
1827
|
+
how to report a vulnerability.
|
|
1828
|
+
|
|
1686
1829
|
This README is executable too:
|
|
1687
1830
|
|
|
1688
1831
|
<!-- @assert-present file="src/parser.ts,src/engine.ts,src/reporter.ts,src/runner.ts,src/cli.ts,src/comments.ts" -->
|
|
1689
1832
|
|
|
1690
|
-
## Development
|
|
1691
|
-
|
|
1692
|
-
```bash
|
|
1693
|
-
npm install
|
|
1694
|
-
npm run build # tsc -> dist/
|
|
1695
|
-
npm test # vitest
|
|
1696
|
-
npm run test:coverage
|
|
1697
|
-
npm run test:mutation # stryker (four parallel jobs in CI; hours locally)
|
|
1698
|
-
npm run lint # tsc --noEmit
|
|
1699
|
-
npm run selfcheck # run spec-guard on its own docs
|
|
1700
|
-
```
|
|
1701
|
-
|
|
1702
|
-
The test suite runs every assertion case against **both** engines and asserts
|
|
1703
|
-
they agree, so the fallback cannot quietly drift from ripgrep. `@vscode/ripgrep`
|
|
1704
|
-
is a devDependency purely so that the ripgrep path is exercised on every
|
|
1705
|
-
platform in CI, including machines that have no `rg` on PATH; it ships a
|
|
1706
|
-
prebuilt binary and is never a runtime dependency.
|
|
1707
|
-
|
|
1708
|
-
### Mutation testing
|
|
1709
|
-
|
|
1710
|
-
Coverage says a line ran. It does not say an assertion would notice if the line
|
|
1711
|
-
behaved differently. This repository measures the difference: line coverage is
|
|
1712
|
-
**100%**, and **99.20%** of 11,549 mutants are killed in CI, with 89 survivors and
|
|
1713
|
-
no file below 95%. The second number is the one worth reading, and what was done
|
|
1714
|
-
about the survivors matters more than the score. At 0.5.1 every surviving mutant
|
|
1715
|
-
was checked individually, and the 101 left then produced byte-identical output.
|
|
1716
|
-
Each release since has replayed the survivors its own code added before it
|
|
1717
|
-
shipped: they were killed, deleted as dead code, or shown equivalent and
|
|
1718
|
-
recorded in [ADR-0003](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0003-mutation-testing.md) with the evidence.
|
|
1719
|
-
|
|
1720
|
-
That gap is the point. The first run scored 77.23%, and the weakest file was the
|
|
1721
|
-
reporter at 65.48% - not because it lacked tests, but because its tests were
|
|
1722
|
-
almost all `toContain` against colourless output. A mutant could prepend a junk
|
|
1723
|
-
line, drop a colour, or turn `remaining > 0` into `remaining >= 0` and every
|
|
1724
|
-
test still passed. Exact whole-output comparison took it to 87.30%. The engine
|
|
1725
|
-
and the walker were then rewritten for testability rather than papered over with
|
|
1726
|
-
more tests, which is what moved them from 75%/78% to 83%/93%.
|
|
1727
|
-
|
|
1728
|
-
The most recent pass took every module to its ceiling, and its value was not the
|
|
1729
|
-
number. Writing down four contracts that had only ever been checked through
|
|
1730
|
-
their effect on a match count turned up four wrong answers: two assertions on
|
|
1731
|
-
one symbol answered each other's comment handling, an unreadable directory went
|
|
1732
|
-
unreported on any repository small enough to scan in process, snippets from CRLF
|
|
1733
|
-
files carried a carriage return into the terminal, and an invalid pattern was
|
|
1734
|
-
reported with its error message twice. Seventeen branches turned out to be
|
|
1735
|
-
unable to decide anything and were deleted rather than pinned; thirty-five
|
|
1736
|
-
negative controls - each defect reintroduced one at a time - confirm the suite
|
|
1737
|
-
goes red for every one, and the changes meant to be invisible were checked
|
|
1738
|
-
against 6,976 files of real code, reference for reference, against the published
|
|
1739
|
-
0.5.0 build.
|
|
1740
|
-
|
|
1741
|
-
Then the survivors themselves were checked rather than excused. All 130 were
|
|
1742
|
-
applied one at a time and run through a fingerprint of 4,176 observations, and
|
|
1743
|
-
**29 of them turned out not to be equivalent at all** - a Rust brace counter
|
|
1744
|
-
that only matters when a second statement follows, a type-only test that marks
|
|
1745
|
-
`import A, { B } from 'x'` type-only when loosened, a comment check that opens a
|
|
1746
|
-
block comment on `2*3`. All 29 are now tested and all 29 die. All of it is in
|
|
1747
|
-
[ADR-0003](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0003-mutation-testing.md), including a Stryker limitation
|
|
1748
|
-
found on the way: a mutant that stops a test file *loading* is reported as
|
|
1749
|
-
survived even though the suite is in fact killing it.
|
|
1750
|
-
|
|
1751
|
-
CI runs it in **two tiers**, both gated at 97%. Branches and pull requests run
|
|
1752
|
-
Stryker incrementally, reusing the verdict for any mutant whose source and
|
|
1753
|
-
covering tests are both unchanged. Pushes to `main`, the weekly schedule and
|
|
1754
|
-
manual runs do the full sweep, which is the authoritative number and the one
|
|
1755
|
-
quoted above; a full sweep is also what publishes the cache the branches start
|
|
1756
|
-
from, so an incremental verdict can never be built on another incremental
|
|
1757
|
-
verdict.
|
|
1758
|
-
|
|
1759
|
-
The full sweep was 6m54s at 2,118 mutants, 15m51s at 3,493, and 42m39s at 7,763.
|
|
1760
|
-
That growth is why the tiers exist: a check that gets quietly more expensive
|
|
1761
|
-
every release is a check somebody eventually proposes lowering. It is also why
|
|
1762
|
-
both tiers now run in four parallel shards. One runner
|
|
1763
|
-
could no longer finish the full sweep reliably inside the job's limit, and the
|
|
1764
|
-
answer was to split the sweep rather than raise the limit or drop mutants.
|
|
1765
|
-
Each shard mutates its own files against every test, and a final job merges the
|
|
1766
|
-
reports. The merge refuses anything that is not exactly one sweep, and applies
|
|
1767
|
-
the 97% gate to the merged score with the same library Stryker's gate uses.
|
|
1768
|
-
|
|
1769
|
-
Run it locally with `npm run test:mutation` if you like, but do not calibrate
|
|
1770
|
-
anything on the result: on the Windows machine this was developed on the same
|
|
1771
|
-
suite takes hours against 43 minutes of hosted time, and it scores *higher*,
|
|
1772
|
-
because far more mutants hang there and Stryker counts a hang as a kill. Linux
|
|
1773
|
-
CI is the measurement.
|
|
1774
|
-
|
|
1775
|
-
Eight cautionary tales are in [ADR-0003](https://github.com/DescentVTT/spec-guard/blob/main/docs/adr/0003-mutation-testing.md): a
|
|
1776
|
-
run whose score was pure fiction because the mutants were never activated, a
|
|
1777
|
-
tuning knob that lifted the score six points without adding a test, the platform
|
|
1778
|
-
gap above, the baseline being re-anchored when a tokenizer arrived, and a score
|
|
1779
|
-
that rose partly because code carrying hard-to-kill mutants was deleted rather
|
|
1780
|
-
than because tests improved, and the 0.5.0 sweep that failed the build at 83.77%
|
|
1781
|
-
because a suite can be thorough about the thing it was written to test and
|
|
1782
|
-
silent about the machinery underneath it, and the excuse that let the tokenizer
|
|
1783
|
-
sit at 73% for three releases because "that kind of code carries more equivalent
|
|
1784
|
-
mutants" sounded like judgement rather than an unchecked assumption - along with
|
|
1785
|
-
the real bugs that chasing the gate uncovered, and a class of mutant the runner
|
|
1786
|
-
reports as survived while the suite is in fact killing it.
|
|
1787
|
-
|
|
1788
|
-
### Releasing
|
|
1789
|
-
|
|
1790
|
-
A release is a tag push. Nothing is published from a laptop, and there is no
|
|
1791
|
-
npm token in this repository or in its secrets.
|
|
1792
|
-
|
|
1793
|
-
```bash
|
|
1794
|
-
npm version minor # or patch / major - writes package.json and makes the tag
|
|
1795
|
-
git push origin main --follow-tags
|
|
1796
|
-
```
|
|
1797
|
-
|
|
1798
|
-
[`release.yml`](https://github.com/DescentVTT/spec-guard/blob/main/.github/workflows/release.yml) then:
|
|
1799
|
-
|
|
1800
|
-
1. refuses the release unless the tag, `package.json` and a `## <version>`
|
|
1801
|
-
heading in `CHANGELOG.md` all agree;
|
|
1802
|
-
2. type checks, builds, runs the suite and executes this repository's own ADRs.
|
|
1803
|
-
A tag push does not run CI, so the release job runs those steps itself;
|
|
1804
|
-
3. `npm pack`s the tarball, prints its file list and its SHA-256, and hands that
|
|
1805
|
-
exact file to the publishing job. Nothing is rebuilt in between, so what
|
|
1806
|
-
reaches the registry is the artifact the tests ran against;
|
|
1807
|
-
4. stages it with `npm stage publish --provenance`. The job authenticates with a
|
|
1808
|
-
short-lived OIDC token that GitHub mints for `release.yml` running in the
|
|
1809
|
-
`npm` environment, and npmjs.com accepts it only for that combination - npm's
|
|
1810
|
-
trusted publishing, so there is no credential to leak or to rotate.
|
|
1811
|
-
|
|
1812
|
-
The publishing job installs no dependencies and checks nothing out. Everything
|
|
1813
|
-
that runs third-party code - `npm ci`, a postinstall that downloads a ripgrep
|
|
1814
|
-
binary, the test suite - happens in the earlier job, which has no token.
|
|
1815
|
-
|
|
1816
|
-
Staging is not publishing. The version sits on npmjs.com visible to maintainers
|
|
1817
|
-
and installable by nobody until:
|
|
1818
|
-
|
|
1819
|
-
```bash
|
|
1820
|
-
npm stage list @descent-vtt/spec-guard
|
|
1821
|
-
npm stage view <stage-id>
|
|
1822
|
-
npm stage approve <stage-id> # asks for a second factor
|
|
1823
|
-
```
|
|
1824
|
-
|
|
1825
|
-
`npm stage reject <stage-id>` discards it instead, and the tag can be deleted
|
|
1826
|
-
and remade. Approving needs 2FA and therefore a person: CI can build a release,
|
|
1827
|
-
but it cannot decide to publish one.
|
|
1828
|
-
|
|
1829
|
-
Two repository settings hold the other end of this. The `npm` environment
|
|
1830
|
-
accepts deployments only from `main` and from `v*` tags, so no branch can reach
|
|
1831
|
-
the publishing job. A `Release tags` ruleset lets only repository admins create,
|
|
1832
|
-
move or delete a `v*` tag - under this workflow, pushing one is the release, and
|
|
1833
|
-
an accidental push should not be able to start it.
|
|
1834
|
-
|
|
1835
|
-
`Actions -> Release -> Run workflow` with `dry_run` left on verifies, packs and
|
|
1836
|
-
hands the tarball to npm without spending a version number. It stops at the
|
|
1837
|
-
registry's version check - a dispatch runs from `main`, whose version is already
|
|
1838
|
-
published - and it does not exercise authentication, because the OIDC exchange
|
|
1839
|
-
happens only on a publish that intends to write. The first real tag is what
|
|
1840
|
-
proves that end.
|
|
1841
|
-
|
|
1842
1833
|
## Requirements
|
|
1843
1834
|
|
|
1844
1835
|
- Node.js 22 or newer (native ESM)
|