mjolnir-qa 1.0.9 → 2.0.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +204 -0
- package/README.ar.md +434 -478
- package/README.bn.md +434 -491
- package/README.br.md +434 -520
- package/README.bs.md +431 -500
- package/README.da.md +433 -509
- package/README.de.md +432 -522
- package/README.es.md +428 -517
- package/README.fr.md +426 -520
- package/README.gr.md +433 -517
- package/README.he.md +433 -475
- package/README.it.md +436 -525
- package/README.ja.md +436 -506
- package/README.ko.md +434 -494
- package/README.md +453 -438
- package/README.no.md +435 -509
- package/README.pl.md +433 -511
- package/README.ru.md +434 -515
- package/README.th.md +434 -484
- package/README.tr.md +427 -504
- package/README.uk.md +433 -505
- package/README.vi.md +436 -494
- package/README.zh.md +433 -462
- package/README.zht.md +433 -462
- package/dist/cli.d.mts +716 -110
- package/dist/cli.mjs +9815 -22027
- package/dist/mcp/stdio.mjs +2164 -886
- package/dist/rolldown-runtime-8H4AJuhK.mjs +14 -0
- package/dist/scan-pipeline-C0ka-RmX.mjs +2 -0
- package/dist/scan-pipeline-D3Yk2cef.mjs +15309 -0
- package/package.json +14 -8
package/README.md
CHANGED
|
@@ -1,25 +1,27 @@
|
|
|
1
1
|
<div align="center">
|
|
2
2
|
|
|
3
|
-
<img src="assets/readme/
|
|
3
|
+
<img src="assets/readme/hero.svg" alt="Mjölnir. Prevents release-blocking false-green." width="100%" />
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
<br />
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
suite, the run artifacts and the CI pipeline — and reports a worthiness
|
|
10
|
-
score with the evidence behind every deduction.
|
|
7
|
+
Mjölnir quantifies how much you can trust your CI — and blocks releases
|
|
8
|
+
when verification isn't earning its keep.
|
|
11
9
|
|
|
12
|
-
|
|
13
|
-
|
|
10
|
+
<br />
|
|
11
|
+
|
|
12
|
+
[](https://www.npmjs.com/package/mjolnir-qa)
|
|
13
|
+
[](https://www.npmjs.com/package/mjolnir-qa)
|
|
14
14
|
[](https://github.com/Sergey-Bar/Mjolnir/actions/workflows/ci.yml)
|
|
15
|
-
[](https://codecov.io/gh/Sergey-Bar/Mjolnir)
|
|
16
|
+
[](https://scorecard.dev/viewer/?uri=github.com/Sergey-Bar/Mjolnir)
|
|
17
|
+
[](LICENSE)
|
|
18
|
+
[](https://nodejs.org)
|
|
17
19
|
|
|
18
20
|
```bash
|
|
19
21
|
npx mjolnir-qa@latest
|
|
20
22
|
```
|
|
21
23
|
|
|
22
|
-
[See it work](#see-it-work) · [Quickstart](#quickstart) · [What it finds](#what-mjölnir-finds) · [Score](#the-worthiness-score) · [Evidence
|
|
24
|
+
[See it work](#see-it-work) · [Quickstart](#quickstart) · [What it finds](#what-mjölnir-finds) · [Score](#the-worthiness-score) · [Evidence](#the-evidence-model) · [Forensics](#runtime-forensics) · [CI](#ci-integrity) · [Agents](#ai-agents) · [Security](#trust-and-security) · [Limits](#what-mjölnir-cannot-tell-you) · [Docs](#documentation)
|
|
23
25
|
|
|
24
26
|
<details>
|
|
25
27
|
<summary>Read this in another language — 22 translations</summary>
|
|
@@ -33,110 +35,110 @@ it; `npm run docs:translations` reports how far.
|
|
|
33
35
|
|
|
34
36
|
</div>
|
|
35
37
|
|
|
36
|
-
|
|
38
|
+
<br />
|
|
37
39
|
|
|
38
|
-
##
|
|
40
|
+
## A green check is a claim, not a proof
|
|
39
41
|
|
|
40
|
-
A green
|
|
41
|
-
|
|
42
|
-
|
|
42
|
+
A green check means the pipeline did not fail. It does not mean
|
|
43
|
+
the tests ran, or that they could have failed. Every one of these
|
|
44
|
+
ships green — and every one of them makes your release untrustworthy:
|
|
43
45
|
|
|
44
46
|
- a committed `.only` that ran 3 tests instead of 900
|
|
45
|
-
-
|
|
46
|
-
-
|
|
47
|
-
- a test that asserts nothing, or
|
|
47
|
+
- `continue-on-error: true` on the job that was supposed to gate
|
|
48
|
+
- `|| true` after the test command
|
|
49
|
+
- a test that asserts nothing, or has an empty body
|
|
48
50
|
- a retry wrapper that turns a real failure into a lucky pass
|
|
49
|
-
- a report the workflow uploads but never
|
|
50
|
-
- a hard sleep holding a race condition together
|
|
51
|
-
|
|
52
|
-
None of these are exotic, and none of them turn the pipeline red. They look
|
|
53
|
-
intentional to a reviewer — which is exactly why they survive.
|
|
51
|
+
- a report the workflow uploads but never generated
|
|
52
|
+
- a hard sleep holding a race condition together
|
|
54
53
|
|
|
55
|
-
|
|
54
|
+
None of them turns the pipeline red, and each one looks deliberate
|
|
55
|
+
in review. That is why they survive. **They are the reason releases ship bugs.**
|
|
56
56
|
|
|
57
|
-
|
|
57
|
+
Here is Mjölnir reading a real one:
|
|
58
58
|
|
|
59
|
-
|
|
60
|
-
|
|
59
|
+
<p align="center">
|
|
60
|
+
<img src="assets/readme/scan.svg" alt="The demo repository's CI workflow, read line by line. Mjölnir flags each finding at the line it reported, with its rule, what is wrong, its evidence level and its measured false-positive rate." width="800" />
|
|
61
|
+
</p>
|
|
61
62
|
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
| No baseline / no comparable revision | **UNKNOWN**, with the reason named — never an assumed 0 |
|
|
66
|
-
| Scan truncated (budget, unreadable) | **PARTIAL**, exit `2` — never presented as clean |
|
|
63
|
+
<sub>Every finding the demo scan reported for this workflow, at the line it
|
|
64
|
+
reported. Generated by `npm run docs:readme-brand` from
|
|
65
|
+
[`demo-report.json`](assets/readme/demo-report.json) and drift-locked in CI.</sub>
|
|
67
66
|
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
67
|
+
**Strict mode.** The most aggressive detections — `.only`, `continue-on-error`,
|
|
68
|
+
empty tests, retry abuse — live in the quarantine tier. They run only
|
|
69
|
+
under `--strict` and are capped to `info` severity: they flag, they never
|
|
70
|
+
gate. The default scan (`npx mjolnir-qa@latest` without `--strict`)
|
|
71
|
+
covers core and extended rules only. Add `--strict` when you want the
|
|
72
|
+
advisory layer too.
|
|
71
73
|
|
|
72
|
-
|
|
74
|
+
Mjölnir reads the suite, the CI workflows and, when you have one, the
|
|
75
|
+
report of a real run. It does not run your tests, install your
|
|
76
|
+
dependencies, or execute the code it scans. And when it has no evidence,
|
|
77
|
+
it says so rather than inventing confidence:
|
|
73
78
|
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
79
|
+
| Situation | What Mjölnir reports |
|
|
80
|
+
| ---------------------------------------------- | -------------------------------------------------------- |
|
|
81
|
+
| No test declarations found | Score `null`, shown as **UNKNOWN**. Never a made-up 100. |
|
|
82
|
+
| No baseline or comparable revision | **UNKNOWN**, with the reason named. Never an assumed 0. |
|
|
83
|
+
| Scan cut short (time budget, unreadable files) | **PARTIAL**, exit `2`. Never presented as clean. |
|
|
77
84
|
|
|
78
85
|
<p align="center">
|
|
79
|
-
<img src="assets/readme/
|
|
86
|
+
<img src="assets/readme/how-it-works.svg" alt="How Mjölnir works. It reads the test suite and the CI pipeline statically, and the report of a real run when there is one. It weighs every finding by its evidence level and its trust level, where only a real run can reach L3 to L5, and produces findings, a worthiness score and a CI gate on frozen exit codes. In the agent loop, AI writes the fix and Mjölnir re-scans to prove it." width="880" />
|
|
80
87
|
</p>
|
|
81
88
|
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
[`script.demo.json`](assets/video/script.demo.json), not written by hand.
|
|
89
|
-
Open [`architecture.svg`](assets/readme/architecture.svg) on its own for
|
|
90
|
-
the full-resolution version.</sub>
|
|
89
|
+
<sub>Composed for this page and shown at 1:1. Generated by
|
|
90
|
+
`npm run docs:readme-brand` and drift-locked in CI; the score, counts and
|
|
91
|
+
rule ID come from [`script.demo.json`](assets/video/script.demo.json),
|
|
92
|
+
[`demo-report.json`](assets/readme/demo-report.json) and the rule
|
|
93
|
+
registry, never typed by hand. The same picture as a poster:
|
|
94
|
+
[`architecture.svg`](assets/readme/architecture.svg).</sub>
|
|
91
95
|
|
|
92
|
-
|
|
96
|
+
<br />
|
|
93
97
|
|
|
94
98
|
## See it work
|
|
95
99
|
|
|
96
|
-
|
|
97
|
-
|
|
100
|
+
A real scan of [`examples/demo-repo`](examples/demo-repo), a small
|
|
101
|
+
Playwright suite with a CI workflow. This is where its points went:
|
|
98
102
|
|
|
99
103
|
<p align="center">
|
|
100
|
-
<img src="assets/readme/
|
|
104
|
+
<img src="assets/readme/terminal-hero.svg" alt="Mjölnir's deduction breakdown: WORTHINESS 80/100 WORTHY, the score by category, the per-severity deduction box, and a FIX THIS FIRST list" width="520" />
|
|
101
105
|
</p>
|
|
102
106
|
|
|
107
|
+
<sub>Generated by `npm run docs:hero` from a real scan and drift-locked in
|
|
108
|
+
CI. The full `--verbose` report of the same scan is
|
|
109
|
+
[`demo.svg`](assets/readme/demo.svg) (`npm run docs:demo`).</sub>
|
|
110
|
+
|
|
103
111
|
<details>
|
|
104
|
-
<summary><strong>
|
|
112
|
+
<summary><strong>Watch it</strong> — a scan, the fix it prints, and the re-scan that proves it</summary>
|
|
113
|
+
|
|
114
|
+
<br />
|
|
105
115
|
|
|
106
|
-
<!-- Plays inline on github.com only: <video> is rendered for GitHub's own
|
|
107
|
-
user-content CDN, never for a repo-relative path. The link below is
|
|
108
|
-
the fallback for every other renderer (npm, mirrors, offline clones). -->
|
|
109
116
|
<p align="center">
|
|
110
|
-
<video
|
|
111
|
-
src="
|
|
112
|
-
|
|
113
|
-
controls
|
|
114
|
-
muted
|
|
115
|
-
playsinline
|
|
116
|
-
width="900"></video>
|
|
117
|
+
<a href="assets/video/mjolnir-demo.mp4">
|
|
118
|
+
<img src="assets/video/mjolnir-demo-poster.png" alt="A frame of the demo recording: npx mjolnir-qa@latest scanning the demo repository in a terminal window" width="900" />
|
|
119
|
+
</a>
|
|
117
120
|
</p>
|
|
118
121
|
|
|
119
|
-
<sub>
|
|
120
|
-
|
|
121
|
-
[`
|
|
122
|
-
by `npm run docs:video`. The full `--verbose` report of the same scan is
|
|
123
|
-
[`demo.svg`](assets/readme/demo.svg) (`npm run docs:demo`).</sub>
|
|
122
|
+
<sub>Rendered frame by frame from a real scan by `npm run docs:video`;
|
|
123
|
+
never screen-recorded. Select the frame to open
|
|
124
|
+
[`mjolnir-demo.mp4`](assets/video/mjolnir-demo.mp4).</sub>
|
|
124
125
|
|
|
125
126
|
</details>
|
|
126
127
|
|
|
127
|
-
<sub>Every number above is read from
|
|
128
|
-
[`script.demo.json`](assets/video/script.demo.json) — the same values
|
|
129
|
-
[`video-script.spec.ts`](tests/contract/video-script.spec.ts) checks
|
|
130
|
-
against real CLI output — and the diff quotes the two committed workflows
|
|
131
|
-
verbatim. Regenerate with `npm run docs:flow`; drift-locked in CI.</sub>
|
|
132
|
-
|
|
133
128
|
### One finding, up close
|
|
134
129
|
|
|
135
|
-
|
|
130
|
+
Every finding answers four questions: where it is, how sure Mjölnir is,
|
|
131
|
+
how often the rule is wrong, and how to fix it.
|
|
132
|
+
|
|
133
|
+
<p align="center">
|
|
134
|
+
<img src="assets/readme/finding-anatomy.svg" alt="The first finding of the demo scan, exactly as the terminal prints it, with its four parts marked: where, how sure, how often the rule is wrong, and the fix." width="100%" />
|
|
135
|
+
</p>
|
|
136
|
+
|
|
137
|
+
`mjolnir explain QA-CI-001` prints a rule's whole trust record, including
|
|
136
138
|
its measured false-positive rate and the tier that rate earned it:
|
|
137
139
|
|
|
138
140
|
```text
|
|
139
|
-
|
|
141
|
+
▍ QA-CI-001 — continue-on-error masks a failing verification gate
|
|
140
142
|
|
|
141
143
|
Severity: error
|
|
142
144
|
Confidence: high
|
|
@@ -179,10 +181,9 @@ HOW TO VERIFY THE FIX
|
|
|
179
181
|
Docs: mjolnir rules --md (full catalog, this rule included)
|
|
180
182
|
```
|
|
181
183
|
|
|
182
|
-
That is the unit of value:
|
|
183
|
-
reporting a pass it did not earn.
|
|
184
|
+
That is the unit of value: one place where CI reports a pass it did not earn.
|
|
184
185
|
|
|
185
|
-
|
|
186
|
+
<br />
|
|
186
187
|
|
|
187
188
|
## Quickstart
|
|
188
189
|
|
|
@@ -190,84 +191,99 @@ reporting a pass it did not earn.
|
|
|
190
191
|
npx mjolnir-qa@latest
|
|
191
192
|
```
|
|
192
193
|
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
194
|
+
Three steps to trust your release pipeline:
|
|
195
|
+
|
|
196
|
+
1. **Scan** — `npx mjolnir-qa@latest` — scans the current directory, prints the Trust Report, exits `1` if findings at the gate
|
|
197
|
+
2. **Block in CI** — `npx mjolnir-qa@latest --scope changed` — scans only changed lines, blocks PRs with error findings
|
|
198
|
+
3. **Fix and re-verify** — `mjolnir fix` applies safe auto-fixes, then re-scans to prove each one landed
|
|
199
|
+
|
|
200
|
+
Add it to your pipeline so it gates releases on error findings:
|
|
198
201
|
|
|
199
202
|
```bash
|
|
200
|
-
npx mjolnir-qa@latest --scope changed
|
|
203
|
+
npx mjolnir-qa@latest --scope changed --blocking error
|
|
201
204
|
```
|
|
202
205
|
|
|
203
|
-
`mjolnir ci install` writes
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
|
216
|
-
|
|
|
217
|
-
| `mjolnir
|
|
218
|
-
| `mjolnir --
|
|
219
|
-
| `mjolnir --
|
|
220
|
-
| `mjolnir --
|
|
206
|
+
`mjolnir ci install` writes the GitHub Actions workflow with a blocking gate (error severity), pinned to this version:
|
|
207
|
+
|
|
208
|
+
```bash
|
|
209
|
+
mjolnir ci install --gate error
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
Run advisory mode only when you want to preview findings without blocking:
|
|
213
|
+
|
|
214
|
+
```bash
|
|
215
|
+
mjolnir ci install --gate advisory
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
| Command | What it does |
|
|
219
|
+
| ----------------------------------- | ----------------------------------------------- |
|
|
220
|
+
| `mjolnir` | Trust Report: verdict, confidence, next action |
|
|
221
|
+
| `mjolnir --scope changed` | Only what your branch introduced (CI form) |
|
|
222
|
+
| `mjolnir --blocking error` | Exit 1 on error findings — gate releases |
|
|
223
|
+
| `mjolnir ci install --gate error` | Write the blocking PR workflow |
|
|
224
|
+
| `mjolnir business-case` | ROI estimate: projected savings per finding |
|
|
225
|
+
| `mjolnir explain QA-CI-001` | What, why and fix, plus the measured FP rate |
|
|
226
|
+
| `mjolnir why src/a.spec.ts:42` | Why this exact line was flagged. Never gates. |
|
|
227
|
+
| `mjolnir forensics ./test-results/` | Runtime evidence from a real run |
|
|
228
|
+
| `mjolnir trust-report` | Self-contained Trust Artifact (md + json) |
|
|
229
|
+
| `mjolnir handoff` | Remediation plan for a coding agent |
|
|
230
|
+
| `mjolnir --json` / `--format sarif` | Machine-readable output, GitHub Code Scanning |
|
|
231
|
+
| `mjolnir --format codequality` | GitLab Code Quality report (MR widget artifact) |
|
|
232
|
+
| `mjolnir --strict` | Also run quarantine-tier rules (higher FP risk) |
|
|
221
233
|
|
|
222
234
|
<details>
|
|
223
|
-
<summary><strong>
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
|
228
|
-
|
|
|
229
|
-
| `mjolnir
|
|
230
|
-
| `mjolnir
|
|
231
|
-
| `mjolnir
|
|
232
|
-
| `mjolnir
|
|
233
|
-
| `mjolnir
|
|
234
|
-
| `mjolnir
|
|
235
|
-
| `mjolnir
|
|
236
|
-
| `mjolnir
|
|
237
|
-
| `mjolnir
|
|
238
|
-
| `mjolnir
|
|
239
|
-
| `mjolnir
|
|
240
|
-
| `mjolnir
|
|
241
|
-
| `mjolnir
|
|
242
|
-
| `mjolnir
|
|
243
|
-
| `mjolnir
|
|
244
|
-
| `mjolnir
|
|
245
|
-
| `mjolnir
|
|
246
|
-
| `mjolnir
|
|
247
|
-
| `mjolnir
|
|
248
|
-
| `mjolnir
|
|
235
|
+
<summary><strong>Every other command</strong> — flake triage, reporting, governance</summary>
|
|
236
|
+
|
|
237
|
+
<br />
|
|
238
|
+
|
|
239
|
+
| Command | What it does |
|
|
240
|
+
| ----------------------------------- | ---------------------------------------------------- |
|
|
241
|
+
| `mjolnir --classic` | The pre-Trust-Report score banner render |
|
|
242
|
+
| `mjolnir explain verdict` | Why the saved scan's verdict is what it is |
|
|
243
|
+
| `mjolnir triage ./test-results/` | Guided triage. Every row ends in a next action. |
|
|
244
|
+
| `mjolnir pw-report ./test-results/` | Playwright run summary: retries, flakes, slowest |
|
|
245
|
+
| `mjolnir doctor:playwright` | Playwright-only deep scan plus Selector Health Score |
|
|
246
|
+
| `mjolnir fix --dry-run` / `fix` | Safe auto-fixes, each re-scanned to prove it landed |
|
|
247
|
+
| `mjolnir baseline` / `diff` | Snapshot findings, then report only new or worse |
|
|
248
|
+
| `mjolnir impact --since <ref>` | What a commit introduced and resolved |
|
|
249
|
+
| `mjolnir summary` | CI annotations and a step summary from a report |
|
|
250
|
+
| `mjolnir pr-comment` | A scoped PR comment, as Markdown |
|
|
251
|
+
| `mjolnir debt` | Test-debt register with a cost model |
|
|
252
|
+
| `mjolnir handover` | Onboarding map of the suite for a new QA engineer |
|
|
253
|
+
| `mjolnir init` | Detect frameworks, print a setup checklist |
|
|
254
|
+
| `mjolnir suppressions` | List suppressed findings, for governance |
|
|
255
|
+
| `mjolnir rules --unmeasured` | The rules running on assumption, not measurement |
|
|
256
|
+
| `mjolnir rules --md` | Full rule catalog (JSON or Markdown) |
|
|
257
|
+
| `mjolnir doctor` | Self-audit of Mjölnir's own rule base |
|
|
258
|
+
| `mjolnir create-rule <ID>` | Scaffold a new rule and its fixtures |
|
|
259
|
+
| `mjolnir stats` | Local all-time counters of fixes seen |
|
|
260
|
+
| `mjolnir badge` | shields.io endpoint JSON and snippet |
|
|
261
|
+
| `mjolnir --cache` | Incremental re-scans via a local verdict cache |
|
|
262
|
+
| `mjolnir --format mermaid` | Test-architecture diagram for a PR comment |
|
|
249
263
|
|
|
250
264
|
`mjolnir help <command>` prints usage, examples and the next step for any
|
|
251
265
|
of them.
|
|
252
266
|
|
|
253
267
|
</details>
|
|
254
268
|
|
|
255
|
-
Requires **Node.js ≥ 22.18
|
|
256
|
-
|
|
257
|
-
(
|
|
258
|
-
|
|
259
|
-
have no such requirement.)
|
|
269
|
+
Requires **Node.js ≥ 22.18** on Windows, macOS or Linux. Prefer a global
|
|
270
|
+
install? `npm i -g mjolnir-qa`. The floor comes from the build toolchain
|
|
271
|
+
(tsdown targets it and the release pipeline smoke-tests against it); the
|
|
272
|
+
runtime dependencies need no more than that.
|
|
260
273
|
|
|
261
|
-
|
|
274
|
+
<br />
|
|
262
275
|
|
|
263
276
|
## What Mjölnir finds
|
|
264
277
|
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
278
|
+
<p align="center">
|
|
279
|
+
<img src="assets/readme/stack.svg" alt="Works with your stack: the languages, test frameworks and CI systems its rules cover, from the rule registry." width="100%" />
|
|
280
|
+
</p>
|
|
281
|
+
|
|
282
|
+
**<!-- census:total-rules -->79 rules<!-- /census:total-rules -->** in four families — test hygiene, test quality, Playwright and
|
|
283
|
+
CI integrity — across TypeScript and JavaScript, Python, Java, C# and
|
|
284
|
+
GitHub Actions YAML. They cover Playwright in all four bindings, plus
|
|
285
|
+
pytest, JUnit, TestNG, NUnit, xUnit, MSTest, Jest, Vitest and Mocha, with
|
|
286
|
+
starter coverage for Cypress and Selenium. Nine of them, to show the shape:
|
|
271
287
|
|
|
272
288
|
| ID | Rule | Severity | Tier |
|
|
273
289
|
| ------------ | ----------------------------------------------------------------- | -------- | ---------- |
|
|
@@ -281,242 +297,240 @@ is clear:
|
|
|
281
297
|
| QA-PY-002 | Skipped test (`skip`, non-strict `xfail`) | warning | core |
|
|
282
298
|
| QA-CS-103 | Test method with no assertions | error | core |
|
|
283
299
|
|
|
284
|
-
The full catalog is generated from the registry, never hand
|
|
300
|
+
The full catalog is generated from the registry, never maintained by hand:
|
|
285
301
|
`mjolnir rules --md`, [`docs/rules/`](docs/rules/), or the
|
|
286
302
|
[what-it-checks guide](https://sergey-bar.github.io/Mjolnir/guide/what-it-checks).
|
|
287
303
|
|
|
288
304
|
<details>
|
|
289
|
-
<summary><strong>Every rule named in this README
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
|
297
|
-
|
|
|
298
|
-
| QA-TEST-
|
|
299
|
-
| QA-TEST-
|
|
300
|
-
| QA-TEST-
|
|
301
|
-
| QA-TEST-
|
|
302
|
-
| QA-
|
|
303
|
-
| QA-
|
|
304
|
-
| QA-TQUAL-
|
|
305
|
-
| QA-
|
|
306
|
-
| QA-
|
|
307
|
-
| QA-PW-
|
|
308
|
-
| QA-PW-
|
|
309
|
-
| QA-PW-
|
|
310
|
-
| QA-
|
|
311
|
-
| QA-
|
|
312
|
-
| QA-CI-
|
|
313
|
-
| QA-CI-
|
|
314
|
-
| QA-CI-
|
|
315
|
-
| QA-CI-
|
|
316
|
-
| QA-CI-
|
|
317
|
-
| QA-
|
|
318
|
-
| QA-
|
|
319
|
-
| QA-PY-
|
|
320
|
-
| QA-PY-
|
|
321
|
-
| QA-
|
|
322
|
-
| QA-
|
|
323
|
-
| QA-JV-
|
|
324
|
-
| QA-JV-
|
|
325
|
-
| QA-JV-
|
|
326
|
-
| QA-
|
|
327
|
-
| QA-
|
|
328
|
-
| QA-CS-
|
|
329
|
-
| QA-CS-
|
|
330
|
-
| QA-CS-
|
|
305
|
+
<summary><strong>Every rule named in this README</strong>, in one table</summary>
|
|
306
|
+
|
|
307
|
+
<br />
|
|
308
|
+
|
|
309
|
+
> `quarantine` rules run only under `--strict` and never gate (they are
|
|
310
|
+
> capped to info). The severity shown is the authored severity.
|
|
311
|
+
|
|
312
|
+
| ID | Family | Rule | Severity | Tier |
|
|
313
|
+
| ------------ | ---------- | ------------------------------------------------------------ | -------- | ---------- |
|
|
314
|
+
| QA-TEST-001 | Hygiene | Focused test committed (`.only`, `fit`) | error | quarantine |
|
|
315
|
+
| QA-TEST-002 | Hygiene | Skipped test. Escalates to `error` without a tracked reason. | warning | quarantine |
|
|
316
|
+
| QA-TEST-003 | Hygiene | Test with no assertions | error | quarantine |
|
|
317
|
+
| QA-TEST-004 | Hygiene | Hard sleep (`waitForTimeout`, `sleep()`, `delay()`) | warning | extended |
|
|
318
|
+
| QA-TEST-006 | Hygiene | Retry abuse hiding flakiness | warning | quarantine |
|
|
319
|
+
| QA-TEST-010 | Hygiene | Empty test body | error | quarantine |
|
|
320
|
+
| QA-TQUAL-002 | Quality | Tautological assertion | error | quarantine |
|
|
321
|
+
| QA-TQUAL-009 | Quality | Unawaited promise assertion | error | quarantine |
|
|
322
|
+
| QA-TQUAL-011 | Quality | Commented-out tests | warning | extended |
|
|
323
|
+
| QA-PW-002 | Playwright | Unawaited locator assertion | error | core |
|
|
324
|
+
| QA-PW-003 | Playwright | `page.pause()` / `test.only()` committed | error | core |
|
|
325
|
+
| QA-PW-004 | Playwright | Brittle CSS/XPath selectors | warning | quarantine |
|
|
326
|
+
| QA-PW-123 | Playwright | Hardcoded environment URLs | warning | quarantine |
|
|
327
|
+
| QA-PW-140 | Playwright | Screenshot without `maxDiffPixelRatio` | warning | core |
|
|
328
|
+
| QA-CI-001 | CI | `continue-on-error` masks a failing gate | error | quarantine |
|
|
329
|
+
| QA-CI-002 | CI | `\|\| true` swallows exit codes | error | extended |
|
|
330
|
+
| QA-CI-005 | CI | Report consumed but never generated | error | quarantine |
|
|
331
|
+
| QA-CI-007 | CI | Retry wrappers around tests | warning | extended |
|
|
332
|
+
| QA-CI-008 | CI | Always-success step masks failures | error | quarantine |
|
|
333
|
+
| QA-CI-009 | CI | Exit code not propagated (`\|` without pipefail, `;` chains) | error | extended |
|
|
334
|
+
| QA-CI-010 | CI | Tests skipped where they must block | error | quarantine |
|
|
335
|
+
| QA-PY-002 | Python | Skipped test (`skip`, non-strict `xfail`) | warning | core |
|
|
336
|
+
| QA-PY-003 | Python | Test function with no assertions | error | quarantine |
|
|
337
|
+
| QA-PY-005 | Python | `time.sleep()` in tests | warning | extended |
|
|
338
|
+
| QA-PY-012 | Python | Tautological assertion | error | quarantine |
|
|
339
|
+
| QA-JV-101 | Java | Disabled test (`@Disabled`) | warning | core |
|
|
340
|
+
| QA-JV-102 | Java | Hard sleep (`Thread.sleep()`) | warning | extended |
|
|
341
|
+
| QA-JV-103 | Java | Test method with no assertions | error | extended |
|
|
342
|
+
| QA-JV-105 | Java | Playwright `waitForTimeout()` hard sleep | warning | core |
|
|
343
|
+
| QA-JV-106 | Java | Brittle selector instead of role locator | warning | quarantine |
|
|
344
|
+
| QA-CS-101 | C# | Skipped test (`[Ignore]`, `[Fact(Skip=)]`) | warning | core |
|
|
345
|
+
| QA-CS-102 | C# | Hard sleep (`Thread.Sleep` / `Task.Delay`) | warning | core |
|
|
346
|
+
| QA-CS-103 | C# | Test method with no assertions | error | core |
|
|
347
|
+
| QA-CS-105 | C# | `WaitForTimeoutAsync()` hard sleep | warning | extended |
|
|
348
|
+
| QA-CS-106 | C# | Brittle selector instead of role locator | warning | quarantine |
|
|
331
349
|
|
|
332
350
|
Python also ships QA-PY-001…012 (pytest hygiene) and QA-PY-101…108
|
|
333
|
-
(Playwright
|
|
351
|
+
(Playwright for Python). Cypress and Selenium have starter sets of three
|
|
334
352
|
rules each.
|
|
335
353
|
|
|
336
354
|
</details>
|
|
337
355
|
|
|
338
|
-
Every rule ships with a must-fire **and** a must-not-fire fixture
|
|
339
|
-
that fires on its own negative fixture cannot ship. That is the
|
|
340
|
-
false-positive firewall
|
|
341
|
-
|
|
356
|
+
Every rule ships with a must-fire **and** a must-not-fire fixture, and a
|
|
357
|
+
rule that fires on its own negative fixture cannot ship. That is the
|
|
358
|
+
false-positive firewall; `mjolnir doctor` enforces it in this repository's
|
|
359
|
+
own CI.
|
|
342
360
|
|
|
343
361
|
### Selector Health Score
|
|
344
362
|
|
|
345
363
|
`mjolnir doctor:playwright` grades every locator by how it finds an
|
|
346
|
-
element
|
|
347
|
-
|
|
348
|
-
|
|
364
|
+
element: the way a user would (role, label, text), an explicit contract
|
|
365
|
+
(`data-testid`), or a structural accident (CSS chains, XPath). Each file
|
|
366
|
+
gets a score from 0 to 100:
|
|
349
367
|
|
|
350
368
|
```text
|
|
351
|
-
|
|
369
|
+
▍ SELECTOR HEALTH
|
|
352
370
|
|
|
353
371
|
e2e/login.spec.ts
|
|
354
372
|
[█████████████░░░░░░░] 65 / 100
|
|
355
|
-
role/text: 1 · testid: 0 · css-chains: 1 ⚠ · xpath: 0
|
|
373
|
+
role/text: 1 · testid: 0 · plain-css: 0 · css-chains: 1 ⚠ · xpath: 0
|
|
356
374
|
|
|
357
375
|
e2e/checkout.spec.ts
|
|
358
|
-
[
|
|
359
|
-
role/text:
|
|
376
|
+
[██████████████████░░] 88 / 100
|
|
377
|
+
role/text: 4 · testid: 1 · plain-css: 0 · css-chains: 1 ⚠ · xpath: 0
|
|
360
378
|
```
|
|
361
379
|
|
|
362
|
-
This
|
|
380
|
+
This measures **resilience, not correctness**.
|
|
363
381
|
`.btn.btn-primary > div:nth-child(2)` passes today and keeps passing until
|
|
364
|
-
someone touches the markup. A low score never claims the test is broken
|
|
365
|
-
only that
|
|
382
|
+
someone touches the markup. A low score never claims the test is broken,
|
|
383
|
+
only that it depends on markup nobody promised to keep.
|
|
366
384
|
|
|
367
|
-
|
|
385
|
+
<br />
|
|
368
386
|
|
|
369
387
|
## The Worthiness Score
|
|
370
388
|
|
|
371
|
-
<
|
|
372
|
-
<
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
|
|
386
|
-
|
|
387
|
-
|
|
388
|
-
(`npm run docs:gauge` · `npm run docs:hero`) and drift-locked
|
|
389
|
-
([gauge](tests/contract/score-gauge-asset-reproducibility.spec.ts) ·
|
|
390
|
-
[breakdown](tests/contract/hero-asset-reproducibility.spec.ts)).</sub>
|
|
391
|
-
|
|
392
|
-
| Score | Verdict |
|
|
393
|
-
| --------- | ---------------------------------------- |
|
|
394
|
-
| `0 – 49` | **UNWORTHY** |
|
|
395
|
-
| `50 – 79` | **NEEDS WORK** |
|
|
396
|
-
| `80 – 99` | **WORTHY** |
|
|
397
|
-
| `100` | **FORGED** |
|
|
398
|
-
| `null` | **UNKNOWN** — no test declarations found |
|
|
399
|
-
|
|
400
|
-
**How it is computed.** Severity sets a base deduction — `error −8`,
|
|
401
|
-
`warning −3`, `info −1` — which the evidence level then discounts: E2 pays
|
|
389
|
+
<p align="center">
|
|
390
|
+
<img src="assets/readme/score-gauge.svg" alt="The worthiness scale from 0 to 100, with a marker sweeping every score: UNWORTHY below 50, NEEDS WORK from 50 to 79, WORTHY from 80 to 99, FORGED at 100" width="720" />
|
|
391
|
+
</p>
|
|
392
|
+
|
|
393
|
+
<sub>Every score from 0 to 100, placed by the real `deriveScoreState`.
|
|
394
|
+
Generated by `npm run docs:gauge` and drift-locked in CI.</sub>
|
|
395
|
+
|
|
396
|
+
| Score | Verdict |
|
|
397
|
+
| --------- | --------------------------------------- |
|
|
398
|
+
| `0 – 49` | **UNWORTHY** |
|
|
399
|
+
| `50 – 79` | **NEEDS WORK** |
|
|
400
|
+
| `80 – 99` | **WORTHY** |
|
|
401
|
+
| `100` | **FORGED** |
|
|
402
|
+
| `null` | **UNKNOWN**: no test declarations found |
|
|
403
|
+
|
|
404
|
+
**How it is computed.** Severity sets a base deduction (`error −8`,
|
|
405
|
+
`warning −3`, `info −1`) and the evidence level discounts it: E2 pays in
|
|
402
406
|
full, E1 half (rounded down), E0 nothing. The total is normalized by suite
|
|
403
|
-
exposure
|
|
404
|
-
prints the same discounted numbers the score used
|
|
405
|
-
[docs/SCORING.md](docs/SCORING.md)
|
|
407
|
+
exposure, meaning deductions per test declaration rather than per file.
|
|
408
|
+
The terminal prints the same discounted numbers the score used; there is
|
|
409
|
+
no hidden second model. Details: [docs/SCORING.md](docs/SCORING.md) and the
|
|
406
410
|
[scoring guide](https://sergey-bar.github.io/Mjolnir/guide/scoring).
|
|
407
411
|
|
|
408
|
-
**What 100 does not mean.**
|
|
409
|
-
adequate, or the product free of defects.
|
|
410
|
-
Mjölnir's evaluated rules produced a deduction under this scan
|
|
411
|
-
evidence model.**
|
|
412
|
+
**What 100 does not mean.** It does not mean the software is correct, the
|
|
413
|
+
suite is adequate, or the product is free of defects. It means one thing:
|
|
414
|
+
**none of Mjölnir's evaluated rules produced a deduction under this scan
|
|
415
|
+
and this evidence model.**
|
|
412
416
|
|
|
413
|
-
|
|
417
|
+
<br />
|
|
414
418
|
|
|
415
419
|
## The evidence model
|
|
416
420
|
|
|
417
|
-
Every finding carries
|
|
418
|
-
|
|
419
|
-
release on.
|
|
421
|
+
Every finding carries two labels: how sure Mjölnir is, and how far the
|
|
422
|
+
finding was checked. That is the difference between a tool that reports
|
|
423
|
+
patterns and a tool you can gate a release on.
|
|
420
424
|
|
|
421
|
-
|
|
422
|
-
STATIC SIGNAL → EVIDENCE LEVEL → RUNTIME CORROBORATION → TRUST DECISION
|
|
423
|
-
```
|
|
425
|
+
**How sure — the evidence level.**
|
|
424
426
|
|
|
425
|
-
| Level | Name | Means
|
|
426
|
-
| ------ | ------------------- |
|
|
427
|
-
| **E2** | Deterministic proof | The defect is
|
|
428
|
-
| **E1** | Pattern evidence | A pattern strongly
|
|
429
|
-
| **E0** | Observation | Worth knowing
|
|
427
|
+
| Level | Name | Means | Deduction |
|
|
428
|
+
| ------ | ------------------- | -------------------------------------------------- | --------- |
|
|
429
|
+
| **E2** | Deterministic proof | The defect is present in the code as written | Full |
|
|
430
|
+
| **E1** | Pattern evidence | A pattern strongly tied to the defect matched | Half |
|
|
431
|
+
| **E0** | Observation | Worth knowing. Not a claim that anything is wrong. | Zero |
|
|
430
432
|
|
|
431
|
-
Confidence in a detection is not strength of proof
|
|
432
|
-
it matched what it looked for and still be looking at a heuristic.
|
|
433
|
-
findings are
|
|
433
|
+
Confidence in a detection is not the strength of the proof. A rule can be
|
|
434
|
+
certain it matched what it looked for and still be looking at a heuristic.
|
|
435
|
+
E1 findings are there to be read and judged, never applied blindly, and
|
|
434
436
|
that boundary is stamped on the finding in the terminal, the JSON and the
|
|
435
437
|
agent handoff.
|
|
436
438
|
|
|
437
|
-
|
|
438
|
-
|
|
439
|
-
|
|
440
|
-
|
|
441
|
-
|
|
439
|
+
**How far it was checked — the trust level.** Most findings come from
|
|
440
|
+
reading your code. Give Mjölnir the report of a real test run and it can
|
|
441
|
+
confirm that the code actually ran.
|
|
442
|
+
|
|
443
|
+
<p align="center">
|
|
444
|
+
<img src="assets/readme/trust-ladder.svg" alt="The trust ladder from L0 to L5. L0 to L2 come from reading the code; L3 to L5 need a real run report, marked by a break in the ladder." width="100%" />
|
|
445
|
+
</p>
|
|
446
|
+
|
|
447
|
+
| Level | In plain words | What it takes |
|
|
448
|
+
| ------ | ---------------------- | ---------------------------------------------- |
|
|
449
|
+
| **L0** | Noted | Reading the code |
|
|
450
|
+
| **L1** | Looks like the problem | Reading the code: a pattern matched |
|
|
451
|
+
| **L2** | Proven in the code | Reading the code: the defect is structural |
|
|
452
|
+
| **L3** | The file ran | A run report shows the finding's file executed |
|
|
453
|
+
| **L4** | The test ran | A run report shows the finding's test executed |
|
|
454
|
+
| **L5** | The run agrees | The run's own result confirms the defect class |
|
|
455
|
+
|
|
456
|
+
A static scan stops at L2. Only a real run report (Playwright JSON, Jest or
|
|
457
|
+
Vitest JSON, JUnit XML) can lift a finding to L3 or above, so a finding
|
|
458
|
+
that was never seen running can never claim it was. Definitions:
|
|
459
|
+
[docs/TERMINOLOGY.md](docs/TERMINOLOGY.md).
|
|
442
460
|
|
|
443
461
|
### How much of this is measured
|
|
444
462
|
|
|
445
463
|
**<!-- census:measured-of-total -->74 of 79<!-- /census:measured-of-total --> rules carry a false-positive rate measured against real OSS code**
|
|
446
|
-
(
|
|
447
|
-
The other <!-- census:unmeasured -->5<!-- /census:unmeasured --> ship on the author's estimate and say so,
|
|
448
|
-
`mjolnir explain
|
|
449
|
-
footer reports how many of the rules that actually _fired_ are
|
|
464
|
+
(at least 10 hand-classified findings each; see
|
|
465
|
+
[docs/FP-AUDIT.md](docs/FP-AUDIT.md)). The other <!-- census:unmeasured -->5<!-- /census:unmeasured --> ship on the author's estimate and say so, rule by rule,
|
|
466
|
+
in `mjolnir explain`. `mjolnir rules --unmeasured` lists them, and every
|
|
467
|
+
scan footer reports how many of the rules that actually _fired_ are
|
|
468
|
+
measured.
|
|
450
469
|
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
470
|
+
Rates stay public when they are bad. QA-TEST-001 (a committed `.only`)
|
|
471
|
+
audits poorly on real repositories and sits in quarantine for it. The live
|
|
472
|
+
number for every rule, QA-PW-141 included, is in the audit.
|
|
454
473
|
|
|
455
474
|
### Trust tiers
|
|
456
475
|
|
|
457
|
-
Tiers follow measured false-positive
|
|
476
|
+
Tiers follow the measured false-positive rate, not opinion:
|
|
458
477
|
|
|
459
|
-
| Tier | Measured FP
|
|
460
|
-
| -------------- |
|
|
461
|
-
| **core** | ≤ 10%
|
|
462
|
-
| **extended** | ≤ 30%
|
|
463
|
-
| **quarantine** | > 30%
|
|
464
|
-
| _unmeasured_ | n < 10
|
|
478
|
+
| Tier | Measured FP | Behavior |
|
|
479
|
+
| -------------- | ---------------------------- | -------------------------------------------- |
|
|
480
|
+
| **core** | ≤ 10% | Default report, gates |
|
|
481
|
+
| **extended** | ≤ 30% | Default report, lower confidence |
|
|
482
|
+
| **quarantine** | > 30% or explicitly declared | `--strict` only, capped to info, never gates |
|
|
483
|
+
| _unmeasured_ | n < 10 | Cannot be promoted to core until measured |
|
|
465
484
|
|
|
466
|
-
|
|
467
|
-
|
|
485
|
+
FP bands can only demote a tier — they never promote a rule out of
|
|
486
|
+
`quarantine` if it was explicitly declared there. An explicitly
|
|
487
|
+
quarantined rule stays quarantine regardless of its measured FP rate.
|
|
468
488
|
|
|
469
|
-
|
|
489
|
+
Promotion, demotion and per-language maturity:
|
|
490
|
+
[rule lifecycle](https://sergey-bar.github.io/Mjolnir/reference/rule-lifecycle).
|
|
470
491
|
|
|
471
|
-
|
|
492
|
+
### Why this is not a linter
|
|
472
493
|
|
|
473
494
|
Linters tell you whether code follows rules. Mjölnir tells you whether your
|
|
474
495
|
verification can be trusted.
|
|
475
496
|
|
|
476
497
|
| | Linters (ESLint, SonarQube) | Coverage tools | AI code review | **Mjölnir** |
|
|
477
498
|
| -------------------------------------------------------- | :-------------------------: | :------------: | :------------: | :--------------: |
|
|
478
|
-
| Scores the **verification system**, not the product code |
|
|
479
|
-
| CI workflow integrity (`continue-on-error`, `\|\| true`) |
|
|
480
|
-
| Grades Playwright locator resilience (Selector Health) |
|
|
481
|
-
| Reads real run data for `TRUE-FLAKE` verdicts |
|
|
482
|
-
| Publishes a measured false-positive rate per rule |
|
|
483
|
-
| Flags tests with no assertions |
|
|
484
|
-
| Catches hard sleeps (`waitForTimeout`, `time.sleep`) |
|
|
485
|
-
| Deterministic (same input
|
|
499
|
+
| Scores the **verification system**, not the product code | No | No | No | Yes |
|
|
500
|
+
| CI workflow integrity (`continue-on-error`, `\|\| true`) | No | No | only the diff | Yes |
|
|
501
|
+
| Grades Playwright locator resilience (Selector Health) | No | No | No | Yes |
|
|
502
|
+
| Reads real run data for `TRUE-FLAKE` verdicts | No | No | No | Yes |
|
|
503
|
+
| Publishes a measured false-positive rate per rule | No | No | No | Yes |
|
|
504
|
+
| Flags tests with no assertions | Yes\* | No | sometimes | Yes |
|
|
505
|
+
| Catches hard sleeps (`waitForTimeout`, `time.sleep`) | Yes\* | No | sometimes | Yes |
|
|
506
|
+
| Deterministic (same input, same output) | Yes | Yes | No | Yes |
|
|
486
507
|
| Cost per scan | free | free | tokens | **zero** (local) |
|
|
487
508
|
|
|
488
|
-
<sub>\*Covered by `eslint-plugin-jest`
|
|
509
|
+
<sub>\*Covered by `eslint-plugin-jest` and `eslint-plugin-playwright`
|
|
489
510
|
(`expect-expect`, `no-wait-for-timeout`) and by SonarQube's own assertion
|
|
490
|
-
rules.
|
|
491
|
-
plugins, paid tiers and custom rules change some answers.
|
|
492
|
-
summary, not a benchmark.</sub>
|
|
511
|
+
rules. The columns describe default behavior for test-suite verification;
|
|
512
|
+
plugins, paid tiers and custom rules change some answers. This is a
|
|
513
|
+
positioning summary, not a benchmark.</sub>
|
|
493
514
|
|
|
494
|
-
|
|
495
|
-
can find. Mjölnir catches what AI overlooks because it looks
|
|
496
|
-
a committed `.only`, a swallowed exit code, a
|
|
497
|
-
|
|
498
|
-
scanning.
|
|
515
|
+
Use AI review too. It catches nuance, intent and design flaws no pattern
|
|
516
|
+
can find. Mjölnir catches what AI review overlooks because it looks
|
|
517
|
+
intentional: a committed `.only`, a swallowed exit code, a
|
|
518
|
+
`continue-on-error` on a test job. Those need scanning, not reasoning.
|
|
499
519
|
|
|
500
|
-
|
|
520
|
+
<br />
|
|
501
521
|
|
|
502
522
|
## Runtime forensics
|
|
503
523
|
|
|
504
|
-
Static analysis reasons about code that
|
|
505
|
-
actually happened
|
|
506
|
-
from any runner
|
|
507
|
-
|
|
508
|
-
```text
|
|
509
|
-
Static analysis → what the code appears to do
|
|
510
|
-
Runtime evidence → what the run actually did
|
|
511
|
-
both → a finding that can climb the trust ladder
|
|
512
|
-
```
|
|
524
|
+
Static analysis reasons about code that never ran. Forensics reads what
|
|
525
|
+
actually happened: Playwright JSON, Jest JSON, Vitest JSON, and JUnit XML
|
|
526
|
+
from any runner.
|
|
513
527
|
|
|
514
528
|
```bash
|
|
515
529
|
mjolnir forensics ./test-results/
|
|
516
530
|
```
|
|
517
531
|
|
|
518
532
|
```text
|
|
519
|
-
|
|
533
|
+
▍ FLAKINESS LEADERBOARD
|
|
520
534
|
|
|
521
535
|
3 tests · 1 failed · 1 flaky · 1 retried
|
|
522
536
|
|
|
@@ -526,54 +540,66 @@ FAILING declines an expired card (e2e/checkout.spec.ts)
|
|
|
526
540
|
████░░░░░░░░░░░░░░░░ 1.1s · 1 attempt
|
|
527
541
|
```
|
|
528
542
|
|
|
529
|
-
`TRUE-FLAKE`
|
|
530
|
-
|
|
531
|
-
|
|
532
|
-
|
|
543
|
+
`TRUE-FLAKE` does not mean the test retried. It means the test **failed at
|
|
544
|
+
least one attempt and then finished green**: a lucky pass, flagged whatever
|
|
545
|
+
the final checkmark says. `mjolnir triage` turns that history into a
|
|
546
|
+
quarantine proposal, and `mjolnir pw-report` summarizes a run. The same run
|
|
547
|
+
reports are what lift findings to trust levels L3 and above.
|
|
533
548
|
|
|
534
|
-
|
|
549
|
+
<br />
|
|
535
550
|
|
|
536
551
|
## CI integrity
|
|
537
552
|
|
|
538
|
-
A test can pass while the pipeline around it
|
|
539
|
-
|
|
540
|
-
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
|
|
553
|
+
A test can pass while the pipeline around it cannot fail. Mjölnir reads the
|
|
554
|
+
workflows too: `continue-on-error`, `|| true`, exit codes that never
|
|
555
|
+
propagate, always-success steps, reports consumed but never generated, and
|
|
556
|
+
gates skipped on the events that should block. Each finding names the job,
|
|
557
|
+
the step and the line, and carries its own evidence level.
|
|
558
|
+
|
|
559
|
+
Generate the PR workflow with a blocking gate (error severity) — this is the
|
|
560
|
+
default for a reason: findings that survive your pipeline survive your release.
|
|
561
|
+
|
|
562
|
+
```bash
|
|
563
|
+
mjolnir ci install --gate error
|
|
564
|
+
```
|
|
544
565
|
|
|
545
|
-
|
|
566
|
+
The gate is pinned to this version — a new release must not change your gate
|
|
567
|
+
semantics without a commit of yours. Switch to advisory only when you want
|
|
568
|
+
to preview findings without blocking:
|
|
546
569
|
|
|
547
570
|
```bash
|
|
548
|
-
mjolnir ci install
|
|
571
|
+
mjolnir ci install --gate advisory
|
|
549
572
|
```
|
|
550
573
|
|
|
551
|
-
|
|
574
|
+
In your existing workflow, pin the action and gate on error severity:
|
|
552
575
|
|
|
553
576
|
```yaml
|
|
554
|
-
- uses: Sergey-Bar/Mjolnir@
|
|
577
|
+
- uses: Sergey-Bar/Mjolnir@4a588bc62d517bc85fc44c0eae64c6587d3bf70b0
|
|
555
578
|
with:
|
|
556
579
|
scope: changed
|
|
557
580
|
fail-on: error
|
|
558
581
|
```
|
|
559
582
|
|
|
560
|
-
|
|
561
|
-
reproducible gate — [docs/DISTRIBUTION-KIT.md](docs/DISTRIBUTION-KIT.md)
|
|
562
|
-
covers Marketplace, Smithery and the MCP registries.
|
|
583
|
+
The `--blocking error` flag does the same in any CI pipeline:
|
|
563
584
|
|
|
564
|
-
|
|
585
|
+
```bash
|
|
586
|
+
npx mjolnir-qa@latest --scope changed --blocking error
|
|
587
|
+
```
|
|
588
|
+
|
|
589
|
+
To put findings in GitHub Code Scanning, upload SARIF (requires
|
|
590
|
+
`security-events: write` at workflow or job scope):
|
|
565
591
|
|
|
566
592
|
```yaml
|
|
567
593
|
- run: npx mjolnir-qa@latest --format sarif > mjolnir.sarif
|
|
594
|
+
continue-on-error: true
|
|
568
595
|
- uses: github/codeql-action/upload-sarif@v3
|
|
596
|
+
if: ${{ !cancelled() }}
|
|
569
597
|
with:
|
|
570
598
|
sarif_file: mjolnir.sarif
|
|
571
599
|
```
|
|
572
600
|
|
|
573
|
-
On GitLab, `--format codequality`
|
|
574
|
-
widget and diff annotations
|
|
575
|
-
([docs/GITLAB-CI.md](docs/GITLAB-CI.md)).
|
|
576
|
-
|
|
601
|
+
On GitLab, `--format codequality` writes the Code Quality report that the
|
|
602
|
+
MR widget and diff annotations read ([docs/GITLAB-CI.md](docs/GITLAB-CI.md)).
|
|
577
603
|
Editor and pipeline setup: [docs/SARIF-INTEGRATION.md](docs/SARIF-INTEGRATION.md).
|
|
578
604
|
|
|
579
605
|
### Changed-scope attribution
|
|
@@ -582,20 +608,20 @@ Editor and pipeline setup: [docs/SARIF-INTEGRATION.md](docs/SARIF-INTEGRATION.md
|
|
|
582
608
|
npx mjolnir-qa@latest --scope changed
|
|
583
609
|
```
|
|
584
610
|
|
|
585
|
-
Findings are attributed to the lines your branch added, against
|
|
586
|
-
**merge-base**. The scope is the same file set a full scan discovers
|
|
587
|
-
TS/JS specs and adapter configs, `test_*.py`, `*Test.java`, `*Tests.cs`,
|
|
588
|
-
`.github/workflows/*.yml`
|
|
589
|
-
|
|
590
|
-
`main → master → origin/main → origin/master → origin/HEAD`; override
|
|
591
|
-
`--base <ref>`.
|
|
611
|
+
Findings are attributed to the lines your branch added, measured against
|
|
612
|
+
the **merge-base**. The scope is the same file set a full scan discovers
|
|
613
|
+
(TS/JS specs and adapter configs, `test_*.py`, `*Test.java`, `*Tests.cs`,
|
|
614
|
+
`.github/workflows/*.yml`), plus uncommitted and untracked changes, so it
|
|
615
|
+
works before you commit. The base resolves
|
|
616
|
+
`main → master → origin/main → origin/master → origin/HEAD`; override it
|
|
617
|
+
with `--base <ref>`.
|
|
592
618
|
|
|
593
|
-
When the merge-base cannot be resolved
|
|
594
|
-
|
|
595
|
-
report says
|
|
596
|
-
|
|
619
|
+
When the merge-base cannot be resolved (a shallow clone, a detached HEAD, a
|
|
620
|
+
target outside git), findings fall back to whole-file attribution **and the
|
|
621
|
+
report says so.** A silent fallback would be the same kind of defect this
|
|
622
|
+
tool exists to catch.
|
|
597
623
|
|
|
598
|
-
|
|
624
|
+
<br />
|
|
599
625
|
|
|
600
626
|
## AI agents
|
|
601
627
|
|
|
@@ -605,14 +631,14 @@ Findings are only worth something if something acts on them.
|
|
|
605
631
|
SCAN → EVIDENCE → HANDOFF → AGENT → RE-SCAN → PROOF
|
|
606
632
|
```
|
|
607
633
|
|
|
608
|
-
**AI writes the fix. Mjölnir verifies
|
|
634
|
+
**AI writes the fix. Mjölnir verifies it.** The proof comes from the
|
|
609
635
|
re-scan, never from the agent's own report of success.
|
|
610
636
|
|
|
611
637
|
| Command | What the agent gets |
|
|
612
638
|
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
613
|
-
| `mjolnir mcp` | An [MCP](https://modelcontextprotocol.io) server over stdio
|
|
639
|
+
| `mjolnir mcp` | An [MCP](https://modelcontextprotocol.io) server over stdio. `scan`, `explain` and `diff` become callable tools. |
|
|
614
640
|
| `mjolnir handoff` | A saved `--json` report becomes a deterministic Markdown plan: what was detected, the evidence boundary per finding, what must **not** change, how to verify. |
|
|
615
|
-
| `mjolnir install` | Writes into the agent surfaces your repo already has
|
|
641
|
+
| `mjolnir install` | Writes into the agent surfaces your repo already has (`.claude/`, `.cursor/`, `.kilo/`, `AGENTS.md`) so the agent re-scans before it claims it is done. |
|
|
616
642
|
|
|
617
643
|
Add it to a client that ships its own CLI:
|
|
618
644
|
|
|
@@ -631,130 +657,122 @@ Or to any client that takes an `mcpServers` block:
|
|
|
631
657
|
```
|
|
632
658
|
|
|
633
659
|
**The guardrail matters more than the convenience.** Every finding in a
|
|
634
|
-
handoff carries its boundary
|
|
635
|
-
and apply the fix_
|
|
636
|
-
does not prove the defect_. An agent that fixes E1 blindly, suppresses
|
|
637
|
-
rule, or edits a rule to raise the score is doing
|
|
638
|
-
exists to catch
|
|
639
|
-
finding.
|
|
660
|
+
handoff carries its boundary. **E2** says _deterministic: check the location
|
|
661
|
+
and apply the fix_. **E1** says _REQUIRES CONFIRMATION: the observation
|
|
662
|
+
alone does not prove the defect_. An agent that fixes E1 blindly, suppresses
|
|
663
|
+
a rule, or edits a rule to raise the score is doing exactly what this tool
|
|
664
|
+
exists to catch, so the handoff says so in the prompt, next to the finding.
|
|
640
665
|
|
|
641
|
-
|
|
666
|
+
<br />
|
|
642
667
|
|
|
643
668
|
## Trust and security
|
|
644
669
|
|
|
645
|
-
**Local-first, zero telemetry.** No network-capable API
|
|
646
|
-
`https`, `net`, `dns`, `dgram`, WebSocket
|
|
670
|
+
**Local-first, zero telemetry.** No network-capable API (`fetch`, `http`,
|
|
671
|
+
`https`, `net`, `dns`, `dgram`, WebSocket) exists anywhere in `src/`, and
|
|
647
672
|
[`privacy-network-isolation.spec.ts`](tests/contract/privacy-network-isolation.spec.ts)
|
|
648
|
-
fails the build if one appears
|
|
673
|
+
fails the build if one appears. It also bars `eval` and `new Function`.
|
|
649
674
|
Scanning untrusted code never executes it: static analysis reads source
|
|
650
675
|
text, and forensics parses report files that already exist on disk.
|
|
651
676
|
|
|
652
|
-
Two caveats
|
|
653
|
-
|
|
654
|
-
|
|
655
|
-
**Plugins are not sandboxed
|
|
656
|
-
|
|
657
|
-
|
|
658
|
-
|
|
659
|
-
|
|
660
|
-
|
|
661
|
-
|
|
662
|
-
|
|
663
|
-
|
|
664
|
-
|
|
665
|
-
|
|
666
|
-
|
|
667
|
-
|
|
668
|
-
|
|
669
|
-
a **partial** scan or a **crashed rule**, because a truncated self-scan
|
|
670
|
-
that reports nothing is precisely the false green this project exists to
|
|
677
|
+
Two caveats: `npx` itself fetches the package before anything runs, and the
|
|
678
|
+
guarantee covers `src/`, not third-party plugins.
|
|
679
|
+
|
|
680
|
+
**Plugins are not sandboxed.** JS plugins (`mjolnir-rules/*.mjs`, or npm
|
|
681
|
+
packages listed under `"plugins"`) run with full Node privileges, the same
|
|
682
|
+
trust model as ESLint or Vitest plugins. Loading them is opt-in **per
|
|
683
|
+
scan**: without `--enable-plugins` (or `MJOLNIR_ENABLE_PLUGINS=1`) their
|
|
684
|
+
sources are never loaded, and a notice on stderr lists what was skipped.
|
|
685
|
+
JSON rule manifests execute no code, and core rule-ID prefixes are reserved
|
|
686
|
+
so a plugin cannot impersonate one. Report vulnerabilities through
|
|
687
|
+
[SECURITY.md](SECURITY.md).
|
|
688
|
+
|
|
689
|
+
**It runs on itself.** A verification trust engine has no standing unless
|
|
690
|
+
it is itself verifiable. Every CI run scans this repository with the build
|
|
691
|
+
that same run produced. The gate fails on any error-severity finding, and
|
|
692
|
+
also on a **partial** scan or a **crashed rule**, because a truncated
|
|
693
|
+
self-scan that reports nothing is the false green this project exists to
|
|
671
694
|
catch. `mjolnir doctor` re-audits the rule base in the same run (fixture
|
|
672
|
-
firewall, tier honesty, the core-tier cap),
|
|
673
|
-
|
|
674
|
-
artifacts.
|
|
675
|
-
|
|
676
|
-
---
|
|
695
|
+
firewall, tier honesty, the core-tier cap), and an INCONCLUSIVE check fails
|
|
696
|
+
exactly like a failing one. Both reports are uploaded as build artifacts.
|
|
677
697
|
|
|
678
|
-
|
|
698
|
+
### Exit codes and the machine contract
|
|
679
699
|
|
|
680
|
-
|
|
681
|
-
- **It cannot tell you an assertion is _wrong_.** `expect(total).toBe(41)`
|
|
682
|
-
looks perfectly healthy. Mjölnir finds tests that _cannot fail_ and
|
|
683
|
-
pipelines that _cannot go red_ — not tests that check the wrong thing.
|
|
684
|
-
- **It does not prove business correctness.** Nothing here says your
|
|
685
|
-
product does what the requirement asked for.
|
|
686
|
-
- **A 100 is not proof of a good suite.** Whether your suite covers your
|
|
687
|
-
actual risk is a different question, and this tool does not answer it.
|
|
688
|
-
- **<!-- census:unmeasured-of-total -->5 of 79<!-- /census:unmeasured-of-total --> rules ship on an estimate**, not a measured rate — disclosed
|
|
689
|
-
per rule, not buried here.
|
|
690
|
-
- **E1 is not E2.** Heuristic findings are worth reading, not worth
|
|
691
|
-
applying blindly.
|
|
692
|
-
- **An empty repo scores `null`, never 100.**
|
|
693
|
-
|
|
694
|
-
---
|
|
695
|
-
|
|
696
|
-
## Exit codes and the machine contract
|
|
697
|
-
|
|
698
|
-
Frozen surfaces — safe to build CI logic on:
|
|
700
|
+
Frozen, so you can build CI logic on them:
|
|
699
701
|
|
|
700
702
|
| Exit code | Meaning |
|
|
701
703
|
| --------- | --------------------------------------------------------------- |
|
|
702
|
-
| `0` | Clean
|
|
704
|
+
| `0` | Clean: no findings at or above the gate |
|
|
703
705
|
| `1` | Findings at or above the gate |
|
|
704
|
-
| `2` | Partial scan (time budget hit, unreadable files)
|
|
706
|
+
| `2` | Partial scan (time budget hit, unreadable files). Never blocks. |
|
|
705
707
|
| `10` | Usage error (bad flag, missing target) |
|
|
706
708
|
| `20` | Internal error |
|
|
707
709
|
|
|
708
710
|
`2` is deliberately distinct from `0`: a scan that did not finish has not
|
|
709
|
-
found nothing
|
|
711
|
+
found nothing. It has not finished looking.
|
|
710
712
|
|
|
711
|
-
Everything a machine consumes
|
|
712
|
-
|
|
713
|
-
(`schemaVersion: 1`, `contractVersion: 1`), so no consumer
|
|
714
|
-
|
|
713
|
+
Everything a machine consumes (MCP tool results, `--json`, SARIF 2.1) comes
|
|
714
|
+
from one canonical result under a versioned, **additive-only** schema
|
|
715
|
+
(`schemaVersion: 1`, `contractVersion: 1`), so no consumer has to rebuild
|
|
716
|
+
meaning from rendered text. See [the machine contract](docs/machine-contract.md).
|
|
715
717
|
Rule IDs (`QA-<FAMILY>-NNN`) are immutable once shipped and never reused.
|
|
716
718
|
|
|
717
|
-
|
|
719
|
+
<br />
|
|
720
|
+
|
|
721
|
+
## What Mjölnir cannot tell you
|
|
722
|
+
|
|
723
|
+
- **It does not run your tests.** A clean scan is not a passing suite.
|
|
724
|
+
- **It cannot tell you an assertion is _wrong_.** `expect(total).toBe(41)`
|
|
725
|
+
looks healthy. Mjölnir finds tests that _cannot fail_ and pipelines that
|
|
726
|
+
_cannot go red_, not tests that check the wrong thing.
|
|
727
|
+
- **It does not prove business correctness.** Nothing here says your
|
|
728
|
+
product does what the requirement asked for.
|
|
729
|
+
- **A 100 is not proof of a good suite.** Whether your suite covers your
|
|
730
|
+
real risk is a different question, and this tool does not answer it.
|
|
731
|
+
- **<!-- census:unmeasured-of-total -->5 of 79<!-- /census:unmeasured-of-total --> rules ship on an estimate**, not a measured rate. Each one
|
|
732
|
+
says so on its own finding.
|
|
733
|
+
- **E1 is not E2.** Heuristic findings are worth reading, not worth applying
|
|
734
|
+
blindly.
|
|
735
|
+
- **An empty repo scores `null`, never 100.**
|
|
736
|
+
- **A file named `*.spec.ts` with no test declarations does not count as coverage.** A repo whose only spec files contain imports or types (zero `it`/`test` calls) scores `null`, not 100.
|
|
737
|
+
|
|
738
|
+
<br />
|
|
718
739
|
|
|
719
740
|
## Documentation
|
|
720
741
|
|
|
742
|
+
The full docs site is at <https://sergey-bar.github.io/Mjolnir/>.
|
|
743
|
+
|
|
721
744
|
| Document | What's in it |
|
|
722
745
|
| ------------------------------------------------------ | ------------------------------------------------- |
|
|
723
|
-
| [docs/
|
|
724
|
-
| [docs/
|
|
725
|
-
| [docs/
|
|
746
|
+
| [docs/EXECUTIVE-ROI.md](docs/EXECUTIVE-ROI.md) | Business case for decision-makers |
|
|
747
|
+
| [docs/SCORING.md](docs/SCORING.md) | Score normalization and evidence weighting |
|
|
748
|
+
| [docs/TERMINOLOGY.md](docs/TERMINOLOGY.md) | Canonical vocabulary: one word per concept |
|
|
749
|
+
| [docs/FP-AUDIT.md](docs/FP-AUDIT.md) | Measured false-positive rates and the method |
|
|
726
750
|
| [docs/RULE-LIFECYCLE.md](docs/RULE-LIFECYCLE.md) | Rule states, tiers, suppression, deprecation |
|
|
727
751
|
| [docs/VERSIONING.md](docs/VERSIONING.md) | Semver policy, frozen surfaces, deprecation cycle |
|
|
728
752
|
| [docs/machine-contract.md](docs/machine-contract.md) | The canonical machine-readable result |
|
|
729
|
-
| [docs/SARIF-INTEGRATION.md](docs/SARIF-INTEGRATION.md) | SARIF output
|
|
753
|
+
| [docs/SARIF-INTEGRATION.md](docs/SARIF-INTEGRATION.md) | SARIF output and editor or CI setup |
|
|
730
754
|
| [docs/GITLAB-CI.md](docs/GITLAB-CI.md) | GitLab: Code Quality report, MR recipe, gate |
|
|
731
755
|
| [docs/rules/](docs/rules/) | Generated per-rule catalog |
|
|
732
|
-
| [CONTRIBUTING.md](CONTRIBUTING.md) | Dev setup
|
|
756
|
+
| [CONTRIBUTING.md](CONTRIBUTING.md) | Dev setup and contribution workflow |
|
|
733
757
|
| [SUPPORT.md](SUPPORT.md) | Where to ask, report and get help |
|
|
734
758
|
| [SECURITY.md](SECURITY.md) | Vulnerability reporting |
|
|
735
759
|
| [CHANGELOG.md](CHANGELOG.md) | Release history |
|
|
736
760
|
|
|
737
|
-
|
|
738
|
-
|
|
739
|
-
---
|
|
761
|
+
### Status
|
|
740
762
|
|
|
741
|
-
|
|
742
|
-
|
|
743
|
-
|
|
744
|
-
contracts. TypeScript and Python have the broadest measured coverage; Java
|
|
745
|
-
and C# are newer — read them through the
|
|
763
|
+
**Version 1.** The JSON schema and the exit codes are frozen contracts.
|
|
764
|
+
TypeScript and Python have the broadest measured coverage. Java and C# are
|
|
765
|
+
newer; read them through the
|
|
746
766
|
[maturity table](https://sergey-bar.github.io/Mjolnir/reference/rule-lifecycle).
|
|
747
|
-
|
|
767
|
+
What comes next, with no invented dates:
|
|
748
768
|
[the public roadmap](https://sergey-bar.github.io/Mjolnir/reference/roadmap).
|
|
749
769
|
|
|
750
|
-
|
|
751
|
-
|
|
752
|
-
## Contributing
|
|
770
|
+
### Contributing
|
|
753
771
|
|
|
754
772
|
New rules are the easiest first contribution. One command scaffolds the
|
|
755
|
-
rule
|
|
756
|
-
|
|
757
|
-
|
|
773
|
+
rule with its must-fire **and** must-not-fire fixtures. The generated rule
|
|
774
|
+
fails its own fixtures on purpose until real detection is written, because
|
|
775
|
+
a stub that ships is a rule nobody measured:
|
|
758
776
|
|
|
759
777
|
```bash
|
|
760
778
|
mjolnir create-rule QA-PW-140 --title "Screenshot without diff bound"
|
|
@@ -763,26 +781,23 @@ mjolnir create-rule QA-PW-140 --title "Screenshot without diff bound"
|
|
|
763
781
|
Dev setup, the standing-gate commands, and the anti-creep and
|
|
764
782
|
fixture-firewall laws are in [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
765
783
|
|
|
766
|
-
|
|
767
|
-
|
|
768
|
-
## The Mjölnir Standard
|
|
769
|
-
|
|
770
|
-
Don't ask whether the tests passed.
|
|
771
|
-
|
|
772
|
-
Ask whether the evidence proves they deserve to be trusted.
|
|
784
|
+
<br />
|
|
773
785
|
|
|
774
786
|
<div align="center">
|
|
775
787
|
|
|
776
|
-
|
|
777
|
-
|
|
778
|
-
**Stop shipping tests you can't trust.**
|
|
788
|
+
<img src="assets/readme/closing.svg" alt="Run it on your repo." width="100%" />
|
|
779
789
|
|
|
780
790
|
```bash
|
|
781
791
|
npx mjolnir-qa@latest
|
|
782
792
|
```
|
|
783
793
|
|
|
784
|
-
|
|
794
|
+
[Read the guide](https://sergey-bar.github.io/Mjolnir/guide/getting-started) · [Docs site](https://sergey-bar.github.io/Mjolnir/) · [npm](https://www.npmjs.com/package/mjolnir-qa)
|
|
795
|
+
|
|
796
|
+
<br />
|
|
797
|
+
|
|
798
|
+
Don't ask whether the tests passed.<br />
|
|
799
|
+
Ask whether the evidence proves they deserve to be trusted.
|
|
785
800
|
|
|
786
|
-
Built by [Sergey Bar](https://www.linkedin.com/in/sergeybar/)
|
|
801
|
+
<sub>Built by [Sergey Bar](https://www.linkedin.com/in/sergeybar/) · MIT licensed</sub>
|
|
787
802
|
|
|
788
803
|
</div>
|