mjolnir-qa 4.0.0 → 6.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +3178 -16
- package/README.md +81 -760
- package/dist/cli.d.mts +552 -206
- package/dist/cli.mjs +31409 -13990
- package/dist/mcp/stdio.mjs +1290 -405
- package/package.json +76 -32
- package/README.ar.md +0 -647
- package/README.bn.md +0 -647
- package/README.br.md +0 -647
- package/README.bs.md +0 -647
- package/README.da.md +0 -647
- package/README.de.md +0 -647
- package/README.es.md +0 -647
- package/README.fr.md +0 -647
- package/README.gr.md +0 -647
- package/README.he.md +0 -647
- package/README.it.md +0 -647
- package/README.ja.md +0 -647
- package/README.ko.md +0 -647
- package/README.no.md +0 -647
- package/README.pl.md +0 -647
- package/README.ru.md +0 -647
- package/README.th.md +0 -647
- package/README.tr.md +0 -647
- package/README.uk.md +0 -647
- package/README.vi.md +0 -647
- package/README.zh.md +0 -647
- package/README.zht.md +0 -647
- package/dist/scan-pipeline-Bg-46cLS.mjs +0 -2
- package/dist/scan-pipeline-CCJdEKNa.mjs +0 -16336
package/README.md
CHANGED
|
@@ -1,470 +1,99 @@
|
|
|
1
1
|
<div align="center">
|
|
2
2
|
|
|
3
|
-
<img src="assets/readme/hero.svg" alt="Mjölnir
|
|
3
|
+
<img src="assets/readme/hero.svg" alt="Mjölnir — the verification trust engine" width="100%" />
|
|
4
4
|
|
|
5
5
|
<br />
|
|
6
6
|
|
|
7
|
-
Mjölnir quantifies how much you can trust your CI — and blocks releases
|
|
8
|
-
when verification isn't earning its keep.
|
|
7
|
+
**Mjölnir quantifies how much you can trust your CI — and blocks releases when verification isn't earning its keep.**
|
|
9
8
|
|
|
10
9
|
<br />
|
|
11
10
|
|
|
12
11
|
[](https://www.npmjs.com/package/mjolnir-qa)
|
|
13
|
-
[](https://www.npmjs.com/package/mjolnir-qa)
|
|
14
12
|
[](https://github.com/Sergey-Bar/Mjolnir/actions/workflows/ci.yml)
|
|
15
13
|
[](https://codecov.io/gh/Sergey-Bar/Mjolnir)
|
|
16
|
-
[](https://scorecard.dev/viewer/?uri=github.com/Sergey-Bar/Mjolnir)
|
|
17
14
|
[](LICENSE)
|
|
18
15
|
[](https://nodejs.org)
|
|
19
16
|
|
|
20
17
|
```bash
|
|
21
|
-
npx mjolnir-qa@
|
|
18
|
+
npx mjolnir-qa@6.0.0
|
|
22
19
|
```
|
|
23
20
|
|
|
24
|
-
[
|
|
25
|
-
|
|
26
|
-
<details>
|
|
27
|
-
<summary>Read this in another language — 22 translations</summary>
|
|
28
|
-
|
|
29
|
-
English | [简体中文](README.zh.md) | [繁體中文](README.zht.md) | [한국어](README.ko.md) | [Deutsch](README.de.md) | [Español](README.es.md) | [Français](README.fr.md) | [Italiano](README.it.md) | [Dansk](README.da.md) | [日本語](README.ja.md) | [Polski](README.pl.md) | [Русский](README.ru.md) | [Norsk](README.no.md) | [Português (Brasil)](README.br.md) | [ไทย](README.th.md) | [Türkçe](README.tr.md) | [Українська](README.uk.md) | [বাংলা](README.bn.md) | [Ελληνικά](README.gr.md) | [Tiếng Việt](README.vi.md) | [עברית](README.he.md) | [العربية](README.ar.md) | [Bosanski](README.bs.md)
|
|
30
|
-
|
|
31
|
-
English is canonical. Translations are machine-assisted and may lag behind
|
|
32
|
-
it; `npm run docs:translations` reports how far.
|
|
33
|
-
|
|
34
|
-
</details>
|
|
21
|
+
[**Interactive walkthrough**](https://sergey-bar.github.io/Mjolnir/) · [Docs](https://sergey-bar.github.io/Mjolnir/) · [CHANGELOG](CHANGELOG.md) · [Video](assets/video/mjolnir-demo.mp4)
|
|
35
22
|
|
|
36
23
|
</div>
|
|
37
24
|
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
## A green check is a claim, not a proof
|
|
41
|
-
|
|
42
|
-
A green check means the pipeline did not fail. It does not mean
|
|
43
|
-
the tests ran, or that they could have failed. Every one of these
|
|
44
|
-
ships green — and every one of them makes your release untrustworthy:
|
|
45
|
-
|
|
46
|
-
- a committed `.only` that ran 3 tests instead of 900
|
|
47
|
-
- `continue-on-error: true` on the job that was supposed to gate
|
|
48
|
-
- `|| true` after the test command
|
|
49
|
-
- a test that asserts nothing, or has an empty body
|
|
50
|
-
- a retry wrapper that turns a real failure into a lucky pass
|
|
51
|
-
- a report the workflow uploads but never generated
|
|
52
|
-
- a hard sleep holding a race condition together
|
|
53
|
-
|
|
54
|
-
None of them turns the pipeline red, and each one looks deliberate
|
|
55
|
-
in review. That is why they survive. **They are the reason releases ship bugs.**
|
|
56
|
-
|
|
57
|
-
Here is Mjölnir reading a real one:
|
|
58
|
-
|
|
59
|
-
<p align="center">
|
|
60
|
-
<img src="assets/readme/scan.svg" alt="The demo repository's CI workflow, read line by line. Mjölnir flags each finding at the line it reported, with its rule, what is wrong, its evidence level and its measured false-positive rate." width="800" />
|
|
61
|
-
</p>
|
|
62
|
-
|
|
63
|
-
<sub>Every finding the demo scan reported for this workflow, at the line it
|
|
64
|
-
reported. Generated by `npm run docs:readme-brand` from
|
|
65
|
-
[`demo-report.json`](assets/readme/demo-report.json) and drift-locked in CI.</sub>
|
|
66
|
-
|
|
67
|
-
**Strict mode.** The most aggressive detections — `.only`, `continue-on-error`,
|
|
68
|
-
empty tests, retry abuse — live in the quarantine tier. They run only
|
|
69
|
-
under `--strict` and are capped to `info` severity: they flag, they never
|
|
70
|
-
gate. The default scan (`npx mjolnir-qa@4.0.0` without `--strict`)
|
|
71
|
-
covers core and extended rules only. Add `--strict` when you want the
|
|
72
|
-
advisory layer too.
|
|
73
|
-
|
|
74
|
-
Mjölnir reads the suite, the CI workflows and, when you have one, the
|
|
75
|
-
report of a real run. It does not run your tests, install your
|
|
76
|
-
dependencies, or execute the code it scans. And when it has no evidence,
|
|
77
|
-
it says so rather than inventing confidence:
|
|
78
|
-
|
|
79
|
-
| Situation | What Mjölnir reports |
|
|
80
|
-
| ---------------------------------------------- | -------------------------------------------------------- |
|
|
81
|
-
| No test declarations found | Score `null`, shown as **UNKNOWN**. Never a made-up 100. |
|
|
82
|
-
| No baseline or comparable revision | **UNKNOWN**, with the reason named. Never an assumed 0. |
|
|
83
|
-
| Scan cut short (time budget, unreadable files) | **PARTIAL**, exit `2`. Never presented as clean. |
|
|
84
|
-
|
|
85
|
-
<p align="center">
|
|
86
|
-
<img src="assets/readme/how-it-works.svg" alt="How Mjölnir works. It reads the test suite and the CI pipeline statically, and the report of a real run when there is one. It weighs every finding by its evidence level and its trust level, where only a real run can reach L3 to L5, and produces findings, a worthiness score and a CI gate on frozen exit codes. In the agent loop, AI writes the fix and Mjölnir re-scans to prove it." width="880" />
|
|
87
|
-
</p>
|
|
88
|
-
|
|
89
|
-
<sub>Composed for this page and shown at 1:1. Generated by
|
|
90
|
-
`npm run docs:readme-brand` and drift-locked in CI; the score, counts and
|
|
91
|
-
rule ID come from [`script.demo.json`](assets/video/script.demo.json),
|
|
92
|
-
[`demo-report.json`](assets/readme/demo-report.json) and the rule
|
|
93
|
-
registry, never typed by hand. The same picture as a poster:
|
|
94
|
-
[`architecture.svg`](assets/readme/architecture.svg).</sub>
|
|
95
|
-
|
|
96
|
-
<br />
|
|
97
|
-
|
|
98
|
-
## Release status
|
|
99
|
-
|
|
100
|
-
The published line is `3.0.0`; this working tree is the `4.0.0-rc.1`
|
|
101
|
-
candidate. `3.0.0` must not be republished or retagged. The M26–M50 program is
|
|
102
|
-
tracked in
|
|
103
|
-
[`docs/ROADMAP.yaml`](docs/ROADMAP.yaml); provisional capability contracts
|
|
104
|
-
are not automatically enabled or certified. Repository-owned checks pass, but
|
|
105
|
-
Trust certification remains `NOT_CERTIFIED` until the protected holdout,
|
|
106
|
-
real-world, platform/consumer, remote-workflow, support-matrix, and corpus
|
|
107
|
-
evidence gates pass. Run `npm run m26:readiness` before treating any candidate
|
|
108
|
-
as releasable. This document does not publish a tag or authorize a release.
|
|
109
|
-
|
|
110
|
-
## See it work
|
|
111
|
-
|
|
112
|
-
A real scan of [`examples/demo-repo`](examples/demo-repo), a small
|
|
113
|
-
Playwright suite with a CI workflow. This is where its points went:
|
|
114
|
-
|
|
115
|
-
<p align="center">
|
|
116
|
-
<img src="assets/readme/terminal-hero.svg" alt="Mjölnir's deduction breakdown: WORTHINESS 80/100 WORTHY, the score by category, the per-severity deduction box, and a FIX THIS FIRST list" width="520" />
|
|
117
|
-
</p>
|
|
118
|
-
|
|
119
|
-
<sub>Generated by `npm run docs:hero` from a real scan and drift-locked in
|
|
120
|
-
CI. The full `--verbose` report of the same scan is
|
|
121
|
-
[`demo.svg`](assets/readme/demo.svg) (`npm run docs:demo`).</sub>
|
|
122
|
-
|
|
123
|
-
<details>
|
|
124
|
-
<summary><strong>Watch it</strong> — a scan, the fix it prints, and the re-scan that proves it</summary>
|
|
25
|
+
---
|
|
125
26
|
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
<p align="center">
|
|
129
|
-
<a href="assets/video/mjolnir-demo.mp4">
|
|
130
|
-
<img src="assets/video/mjolnir-demo-poster.png" alt="A frame of the demo recording: npx mjolnir-qa@4.0.0 scanning the demo repository in a terminal window" width="900" />
|
|
131
|
-
</a>
|
|
132
|
-
</p>
|
|
27
|
+
## The problem: a green check is a claim, not proof
|
|
133
28
|
|
|
134
|
-
|
|
135
|
-
never screen-recorded. Select the frame to open
|
|
136
|
-
[`mjolnir-demo.mp4`](assets/video/mjolnir-demo.mp4).</sub>
|
|
29
|
+
A green check means the pipeline did not fail. It does **not** mean the tests ran, or that they could have failed.
|
|
137
30
|
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
31
|
+
```yaml
|
|
32
|
+
# These all ship green. None earned it.
|
|
33
|
+
steps:
|
|
34
|
+
- run: npm test || true # exit code swallowed
|
|
35
|
+
- run: pytest tests/ --maxfail=0 # retry wrapper
|
|
36
|
+
- run: playwright test --workers=1 # serial but no --exit-on-failure
|
|
37
|
+
- uses: some/action@main # mutable ref
|
|
38
|
+
```
|
|
141
39
|
|
|
142
|
-
|
|
143
|
-
how often the rule is wrong, and how to fix it.
|
|
40
|
+
Mjölnir reads your test suite, your CI workflows, and (when available) the report of a real run. It reports every finding with a **trust level** (L0–L5, where L3+ requires a real run) and a **measured false-positive rate** where corpus evidence exists.
|
|
144
41
|
|
|
145
42
|
<p align="center">
|
|
146
|
-
<img src="assets/readme/
|
|
43
|
+
<img src="assets/readme/scan.svg" alt="Mjölnir reads a workflow line by line, flagging each finding with its rule, evidence level, and measured FP rate." width="800" />
|
|
147
44
|
</p>
|
|
148
45
|
|
|
149
|
-
|
|
150
|
-
its measured false-positive rate and the tier that rate earned it:
|
|
151
|
-
|
|
152
|
-
```text
|
|
153
|
-
▍ QA-CI-001 — continue-on-error masks a failing verification gate
|
|
154
|
-
|
|
155
|
-
Severity: error
|
|
156
|
-
Confidence: high
|
|
157
|
-
Tier: quarantine
|
|
158
|
-
Evidence: E2
|
|
159
|
-
QA impact: False-green risk (FALSE-GREEN)
|
|
160
|
-
Measured FP: 11% (19 hand-classified corpus verdicts)
|
|
161
|
-
FP risk: low (author estimate)
|
|
162
|
-
Languages: yaml
|
|
163
|
-
Frameworks: github-actions, azure-pipelines
|
|
164
|
-
|
|
165
|
-
COPY-READY REVIEW COMMENT
|
|
166
|
-
Advisory finding — this does not block merging: Job `security-scan` runs a
|
|
167
|
-
verification gate under `continue-on-error: true`.
|
|
168
|
-
Why it weakens verification: This job can fail every day and CI will still
|
|
169
|
-
show green. The checkmark on this workflow cannot be trusted.
|
|
170
|
-
Confidence: high, evidence E2, tier quarantine; measured FP 11% (19 verdicts).
|
|
171
|
-
Suggested fix: Remove continue-on-error, or scope it to individual
|
|
172
|
-
non-blocking steps only.
|
|
173
|
-
Verify with: mjolnir --scope changed, then mjolnir explain QA-CI-001 if the
|
|
174
|
-
finding still appears.
|
|
175
|
-
|
|
176
|
-
WHAT WAS FOUND (real detector output, not a mockup)
|
|
177
|
-
Job `security-scan` runs a verification gate under `continue-on-error: true`.
|
|
178
|
-
|
|
179
|
-
WHY IT MATTERS
|
|
180
|
-
This job can fail every day and CI will still show green. The checkmark on
|
|
181
|
-
this workflow cannot be trusted.
|
|
182
|
-
|
|
183
|
-
HOW TO FIX
|
|
184
|
-
Remove continue-on-error, or scope it to individual non-blocking steps only.
|
|
185
|
-
|
|
186
|
-
Example from this rule's own must-fire fixture: QA-CI-001/must-fire/masked.yml
|
|
187
|
-
|
|
188
|
-
WHAT WOULD CHANGE THE VERDICT
|
|
189
|
-
- a run report next to the scan target (mjolnir.report.json or test-results/)
|
|
190
|
-
corroborating this file lifts its findings to L3–L5
|
|
191
|
-
- a documented suppression (mjolnir.config.json) lowers the finding count
|
|
192
|
-
without claiming correctness
|
|
193
|
-
- quarantine findings run only under --strict and are advisory (E0) — they can
|
|
194
|
-
never gate CI
|
|
195
|
-
|
|
196
|
-
NEXT ACTION
|
|
197
|
-
Fix the first occurrence, then re-run: `mjolnir --scope changed`. Every
|
|
198
|
-
occurrence of this rule is listed in the scan output.
|
|
199
|
-
|
|
200
|
-
HOW TO VERIFY THE FIX
|
|
201
|
-
Re-run `mjolnir` on the changed file(s) — this finding should no longer
|
|
202
|
-
appear. `mjolnir --scope changed` scopes the check to just what you touched.
|
|
203
|
-
|
|
204
|
-
Docs: mjolnir rules --md (full catalog, this rule included)
|
|
205
|
-
```
|
|
206
|
-
|
|
207
|
-
That is the unit of value: one place where CI reports a pass it did not earn.
|
|
208
|
-
|
|
209
|
-
<br />
|
|
46
|
+
---
|
|
210
47
|
|
|
211
48
|
## Quickstart
|
|
212
49
|
|
|
213
50
|
```bash
|
|
214
|
-
|
|
215
|
-
|
|
51
|
+
# 1. Scan — Trust Report, exits 1 on gate findings
|
|
52
|
+
npx mjolnir-qa@6.0.0
|
|
216
53
|
|
|
217
|
-
|
|
54
|
+
# 2. Block in CI — scan only what your branch changed
|
|
55
|
+
npx mjolnir-qa@6.0.0 --scope changed
|
|
218
56
|
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
3. **Fix and re-verify** — `mjolnir fix` applies safe auto-fixes, then re-scans to prove each one landed
|
|
222
|
-
|
|
223
|
-
Install the advisory PR workflow first, so existing trust debt cannot break
|
|
224
|
-
CI on day one:
|
|
225
|
-
|
|
226
|
-
```bash
|
|
227
|
-
mjolnir ci install
|
|
57
|
+
# 3. Fix and re-verify — safe auto-fixes, each re-scanned to prove it landed
|
|
58
|
+
mjolnir fix
|
|
228
59
|
```
|
|
229
60
|
|
|
230
|
-
|
|
61
|
+
**Prove the fix with a baseline, then delta:**
|
|
231
62
|
|
|
232
63
|
```bash
|
|
233
|
-
mjolnir
|
|
64
|
+
npx mjolnir-qa@6.0.0 --save-baseline # once: writes .mjolnir/baseline.json
|
|
65
|
+
# … make the fix …
|
|
66
|
+
mjolnir ci verify # RESOLVED (gone) / NEW (introduced)
|
|
234
67
|
```
|
|
235
68
|
|
|
236
|
-
|
|
237
|
-
| --------------------------------------------- | ------------------------------------------------------------ |
|
|
238
|
-
| `mjolnir` | Trust Report: verdict, confidence, next action |
|
|
239
|
-
| `mjolnir --scope changed` | Only what your branch introduced (CI form) |
|
|
240
|
-
| `mjolnir --blocking error` | Exit 1 on error findings — gate releases |
|
|
241
|
-
| `mjolnir ci install` | Write the advisory PR workflow (default) |
|
|
242
|
-
| `mjolnir ci install --gate error` | Opt into the blocking PR workflow |
|
|
243
|
-
| `mjolnir business-case` | Measured FP rates; a cost figure only with `--incident-cost` |
|
|
244
|
-
| `mjolnir release-report` | Release readiness: GO, CONDITIONAL GO, or NO-GO |
|
|
245
|
-
| `mjolnir release-trust` | 12-dimension release assurance verdict |
|
|
246
|
-
| `mjolnir report` | Findings in a Playwright-shaped file (no test outcomes) |
|
|
247
|
-
| `mjolnir diff` | What changed against the stored baseline, by fingerprint |
|
|
248
|
-
| `mjolnir verify` | Digest of what a change resolved and what it introduced |
|
|
249
|
-
| `mjolnir trend` | Record, show, or diff local quality snapshots |
|
|
250
|
-
| `mjolnir policy` | Initialize, validate, or check policy gates |
|
|
251
|
-
| `mjolnir quarantine` | Review deterministic proposals (prototype) |
|
|
252
|
-
| `mjolnir analyze --cross-file` | Bounded cross-file analysis |
|
|
253
|
-
| `mjolnir ci-adapter github .` | Generate CI templates for supported providers |
|
|
254
|
-
| `mjolnir dashboard` | Generate a self-contained quality dashboard |
|
|
255
|
-
| `mjolnir exec-report` | Executive KPIs and recommendations (advisory) |
|
|
256
|
-
| `mjolnir enterprise` | Self-hosted templates (prototype) |
|
|
257
|
-
| `mjolnir maturity` | Assess maturity or display maturity levels |
|
|
258
|
-
| `mjolnir mutation tests/mutation-report.json` | Analyze mutation reports; never promotes trust |
|
|
259
|
-
| `mjolnir mcp` | Read-only MCP tools over stdio |
|
|
260
|
-
| `mjolnir explain QA-CI-001` | What, why and fix, plus measured/unmeasured state |
|
|
261
|
-
| `mjolnir why src/a.spec.ts:42` | Why this exact line was flagged. Never gates. |
|
|
262
|
-
| `mjolnir forensics ./test-results/` | Runtime evidence from a real run |
|
|
263
|
-
| `mjolnir trust-report` | Self-contained Trust Artifact (md + json) |
|
|
264
|
-
| `mjolnir handoff` | Remediation plan for a coding agent |
|
|
265
|
-
| `mjolnir --json` / `--format sarif` | Machine-readable output, GitHub Code Scanning |
|
|
266
|
-
| `mjolnir --format codequality` | GitLab Code Quality report (MR widget artifact) |
|
|
267
|
-
| `mjolnir --strict` | Also run quarantine-tier rules (higher FP risk) |
|
|
268
|
-
|
|
269
|
-
<details>
|
|
270
|
-
<summary><strong>Every other command</strong> — flake triage, reporting, governance</summary>
|
|
271
|
-
|
|
272
|
-
<br />
|
|
273
|
-
|
|
274
|
-
| Command | What it does |
|
|
275
|
-
| ----------------------------------- | ----------------------------------------------------- |
|
|
276
|
-
| `mjolnir --classic` | The pre-Trust-Report score banner render |
|
|
277
|
-
| `mjolnir explain verdict` | Why the saved scan's verdict is what it is |
|
|
278
|
-
| `mjolnir triage ./test-results/` | Guided triage. Every row ends in a next action. |
|
|
279
|
-
| `mjolnir pw-report ./test-results/` | Playwright run summary: retries, flakes, slowest |
|
|
280
|
-
| `mjolnir doctor:playwright` | Playwright-only deep scan plus Selector Health Score |
|
|
281
|
-
| `mjolnir fix --dry-run` / `fix` | Safe auto-fixes, each re-scanned to prove it landed |
|
|
282
|
-
| `mjolnir baseline` / `diff` | Snapshot findings, then report only new or worse |
|
|
283
|
-
| `mjolnir impact --since <ref>` | What a commit introduced and resolved |
|
|
284
|
-
| `mjolnir summary` | CI annotations and a step summary from a report |
|
|
285
|
-
| `mjolnir pr-comment` | A scoped PR comment, as Markdown |
|
|
286
|
-
| `mjolnir debt` | Test-debt register with a cost model |
|
|
287
|
-
| `mjolnir handover` | Onboarding map of the suite for a new QA engineer |
|
|
288
|
-
| `mjolnir init` | Detect frameworks, print a setup checklist |
|
|
289
|
-
| `mjolnir suppressions` | List suppressed findings, for governance |
|
|
290
|
-
| `mjolnir ci-integrity` | Verify blocking CI scans and suppression policy |
|
|
291
|
-
| `mjolnir framework-maturity` | Inspect bounded maturity with human calibration |
|
|
292
|
-
| `mjolnir suppression-gate` | Enforce expiry, allowlist, and mass-suppression rules |
|
|
293
|
-
| `mjolnir cross-file` | Analyze duplicate, shared, and circular test signals |
|
|
294
|
-
| `mjolnir contract-verify` | Verify a persisted machine contract artifact |
|
|
295
|
-
| `mjolnir trust-trend` | Persist and compare trust snapshots over time |
|
|
296
|
-
| `mjolnir evidence-graph` | Build or query the verification evidence graph |
|
|
297
|
-
| `mjolnir rules --unmeasured` | The rules running on assumption, not measurement |
|
|
298
|
-
| `mjolnir rules --md` | Full rule catalog (JSON or Markdown) |
|
|
299
|
-
| `mjolnir doctor` | Self-audit of Mjölnir's own rule base |
|
|
300
|
-
| `mjolnir create-rule <ID>` | Scaffold a new rule and its fixtures |
|
|
301
|
-
| `mjolnir stats` | Local all-time counters of fixes seen |
|
|
302
|
-
| `mjolnir badge` | shields.io endpoint JSON and snippet |
|
|
303
|
-
| `mjolnir --cache` | Incremental re-scans via a local verdict cache |
|
|
304
|
-
| `mjolnir --format mermaid` | Test-architecture diagram for a PR comment |
|
|
305
|
-
|
|
306
|
-
`mjolnir help <command>` prints usage, examples and the next step for any
|
|
307
|
-
of them.
|
|
308
|
-
|
|
309
|
-
</details>
|
|
310
|
-
|
|
311
|
-
Requires **Node.js ≥ 22.18** on Windows, macOS or Linux. Prefer a global
|
|
312
|
-
install? `npm i -g mjolnir-qa`. The floor comes from the build toolchain
|
|
313
|
-
(tsdown targets it and the release pipeline smoke-tests against it); the
|
|
314
|
-
runtime dependencies need no more than that.
|
|
315
|
-
|
|
316
|
-
<br />
|
|
317
|
-
|
|
318
|
-
## What Mjölnir finds
|
|
319
|
-
|
|
320
|
-
<p align="center">
|
|
321
|
-
<img src="assets/readme/stack.svg" alt="Works with your stack: the languages, test frameworks and CI systems its rules cover, from the rule registry." width="100%" />
|
|
322
|
-
</p>
|
|
69
|
+
---
|
|
323
70
|
|
|
324
|
-
|
|
325
|
-
CI integrity — across TypeScript and JavaScript, Python, Java, C# and
|
|
326
|
-
GitHub Actions YAML. They cover Playwright in all four bindings, plus
|
|
327
|
-
pytest, JUnit, TestNG, NUnit, xUnit, MSTest, Jest, Vitest and Mocha, with
|
|
328
|
-
starter coverage for Cypress and Selenium. Nine of them, to show the shape:
|
|
329
|
-
|
|
330
|
-
| ID | Rule | Severity | Tier |
|
|
331
|
-
| ------------ | ----------------------------------------------------------------- | -------- | ---------- |
|
|
332
|
-
| QA-CI-001 | `continue-on-error` masks a failing verification gate | error | quarantine |
|
|
333
|
-
| QA-CI-009 | Test exit code not propagated (`\|` without pipefail, `;` chains) | error | extended |
|
|
334
|
-
| QA-TEST-001 | Focused test committed (`.only`, `fit`) | error | quarantine |
|
|
335
|
-
| QA-TEST-003 | Test with no assertions | error | quarantine |
|
|
336
|
-
| QA-TQUAL-009 | Unawaited promise assertion | error | quarantine |
|
|
337
|
-
| QA-PW-002 | Unawaited locator assertion | error | core |
|
|
338
|
-
| QA-PW-004 | Brittle CSS/XPath selectors | warning | quarantine |
|
|
339
|
-
| QA-PY-002 | Skipped test (`skip`, non-strict `xfail`) | warning | core |
|
|
340
|
-
| QA-CS-103 | Test method with no assertions | error | core |
|
|
341
|
-
|
|
342
|
-
The full catalog is generated from the registry, never maintained by hand:
|
|
343
|
-
`mjolnir rules --md`, [`docs/rules/`](docs/rules/), or the
|
|
344
|
-
[what-it-checks guide](https://sergey-bar.github.io/Mjolnir/guide/what-it-checks).
|
|
345
|
-
|
|
346
|
-
<details>
|
|
347
|
-
<summary><strong>Every rule named in this README</strong>, in one table</summary>
|
|
71
|
+
## What it catches by default (79 rules)
|
|
348
72
|
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
|
355
|
-
|
|
|
356
|
-
|
|
|
357
|
-
| QA-TEST-002 | Hygiene | Skipped test. Escalates to `error` without a tracked reason. | warning | quarantine |
|
|
358
|
-
| QA-TEST-003 | Hygiene | Test with no assertions | error | quarantine |
|
|
359
|
-
| QA-TEST-004 | Hygiene | Hard sleep (`waitForTimeout`, `sleep()`, `delay()`) | warning | extended |
|
|
360
|
-
| QA-TEST-006 | Hygiene | Retry abuse hiding flakiness | warning | quarantine |
|
|
361
|
-
| QA-TEST-010 | Hygiene | Empty test body | error | quarantine |
|
|
362
|
-
| QA-TQUAL-002 | Quality | Tautological assertion | error | quarantine |
|
|
363
|
-
| QA-TQUAL-009 | Quality | Unawaited promise assertion | error | quarantine |
|
|
364
|
-
| QA-TQUAL-011 | Quality | Commented-out tests | warning | extended |
|
|
365
|
-
| QA-PW-002 | Playwright | Unawaited locator assertion | error | core |
|
|
366
|
-
| QA-PW-003 | Playwright | `page.pause()` / `test.only()` committed | error | core |
|
|
367
|
-
| QA-PW-004 | Playwright | Brittle CSS/XPath selectors | warning | quarantine |
|
|
368
|
-
| QA-PW-123 | Playwright | Hardcoded environment URLs | warning | quarantine |
|
|
369
|
-
| QA-PW-140 | Playwright | Screenshot without `maxDiffPixelRatio` | warning | core |
|
|
370
|
-
| QA-CI-001 | CI | `continue-on-error` masks a failing gate | error | quarantine |
|
|
371
|
-
| QA-CI-002 | CI | `\|\| true` swallows exit codes | error | extended |
|
|
372
|
-
| QA-CI-005 | CI | Report consumed but never generated | error | quarantine |
|
|
373
|
-
| QA-CI-007 | CI | Retry wrappers around tests | warning | extended |
|
|
374
|
-
| QA-CI-008 | CI | Always-success step masks failures | error | quarantine |
|
|
375
|
-
| QA-CI-009 | CI | Exit code not propagated (`\|` without pipefail, `;` chains) | error | extended |
|
|
376
|
-
| QA-CI-010 | CI | Tests skipped where they must block | error | quarantine |
|
|
377
|
-
| QA-PY-002 | Python | Skipped test (`skip`, non-strict `xfail`) | warning | core |
|
|
378
|
-
| QA-PY-003 | Python | Test function with no assertions | error | quarantine |
|
|
379
|
-
| QA-PY-005 | Python | `time.sleep()` in tests | warning | extended |
|
|
380
|
-
| QA-PY-012 | Python | Tautological assertion | error | quarantine |
|
|
381
|
-
| QA-JV-101 | Java | Disabled test (`@Disabled`) | warning | core |
|
|
382
|
-
| QA-JV-102 | Java | Hard sleep (`Thread.sleep()`) | warning | extended |
|
|
383
|
-
| QA-JV-103 | Java | Test method with no assertions | error | extended |
|
|
384
|
-
| QA-JV-105 | Java | Playwright `waitForTimeout()` hard sleep | warning | core |
|
|
385
|
-
| QA-JV-106 | Java | Brittle selector instead of role locator | warning | quarantine |
|
|
386
|
-
| QA-CS-101 | C# | Skipped test (`[Ignore]`, `[Fact(Skip=)]`) | warning | core |
|
|
387
|
-
| QA-CS-102 | C# | Hard sleep (`Thread.Sleep` / `Task.Delay`) | warning | core |
|
|
388
|
-
| QA-CS-103 | C# | Test method with no assertions | error | core |
|
|
389
|
-
| QA-CS-105 | C# | `WaitForTimeoutAsync()` hard sleep | warning | extended |
|
|
390
|
-
| QA-CS-106 | C# | Brittle selector instead of role locator | warning | quarantine |
|
|
391
|
-
|
|
392
|
-
Python also ships QA-PY-001…012 (pytest hygiene) and QA-PY-101…108
|
|
393
|
-
(Playwright for Python). Cypress and Selenium have starter sets of three
|
|
394
|
-
rules each.
|
|
395
|
-
|
|
396
|
-
</details>
|
|
397
|
-
|
|
398
|
-
Every rule ships with a must-fire **and** a must-not-fire fixture, and a
|
|
399
|
-
rule that fires on its own negative fixture cannot ship. That is the
|
|
400
|
-
false-positive firewall; `mjolnir doctor` enforces it in this repository's
|
|
401
|
-
own CI.
|
|
402
|
-
|
|
403
|
-
### Selector Health Score
|
|
404
|
-
|
|
405
|
-
`mjolnir doctor:playwright` grades every locator by how it finds an
|
|
406
|
-
element: the way a user would (role, label, text), an explicit contract
|
|
407
|
-
(`data-testid`), or a structural accident (CSS chains, XPath). Each file
|
|
408
|
-
gets a score from 0 to 100:
|
|
409
|
-
|
|
410
|
-
```text
|
|
411
|
-
▍ SELECTOR HEALTH
|
|
412
|
-
|
|
413
|
-
e2e/login.spec.ts
|
|
414
|
-
[█████████████░░░░░░░] 65 / 100
|
|
415
|
-
role/text: 1 · testid: 0 · plain-css: 0 · css-chains: 1 ⚠ · xpath: 0
|
|
416
|
-
|
|
417
|
-
e2e/checkout.spec.ts
|
|
418
|
-
[██████████████████░░] 88 / 100
|
|
419
|
-
role/text: 4 · testid: 1 · plain-css: 0 · css-chains: 1 ⚠ · xpath: 0
|
|
420
|
-
```
|
|
73
|
+
| Family | Example rules |
|
|
74
|
+
| ---------------- | ------------------------------------------------------------------------------------------------------------------- |
|
|
75
|
+
| **CI integrity** | ` | | true` swallowing exit codes (`QA-CI-002`), `continue-on-error` on gate jobs (`QA-CI-001`), mutable Action refs (`QA-CI-008`) |
|
|
76
|
+
| **Test hygiene** | Committed `.only`/`fit` (`QA-TEST-001`), tests with no assertions (`QA-TEST-003`), hard sleeps (`QA-TEST-004`) |
|
|
77
|
+
| **Test quality** | Unawaited promise assertions (`QA-TQUAL-009`), tautological assertions (`QA-TQUAL-002`) |
|
|
78
|
+
| **Playwright** | Unawaited locator assertions (`QA-PW-002`), `page.pause()` committed (`QA-PW-003`), brittle selectors (`QA-PW-004`) |
|
|
79
|
+
| **Python** | Skipped tests (`QA-PY-002`), `time.sleep()` (`QA-PY-005`), tautological assertions (`QA-PY-012`) |
|
|
80
|
+
| **Java / C#** | Disabled tests, hard sleeps, no assertions, brittle selectors |
|
|
421
81
|
|
|
422
|
-
|
|
423
|
-
`.btn.btn-primary > div:nth-child(2)` passes today and keeps passing until
|
|
424
|
-
someone touches the markup. A low score never claims the test is broken,
|
|
425
|
-
only that it depends on markup nobody promised to keep.
|
|
82
|
+
Full catalog: `mjolnir explain --list --md` or [what-it-checks](https://sergey-bar.github.io/Mjolnir/guide/what-it-checks). **73 of 79 rules carry a measured false-positive rate.**
|
|
426
83
|
|
|
427
|
-
|
|
84
|
+
---
|
|
428
85
|
|
|
429
86
|
## The Worthiness Score
|
|
430
87
|
|
|
431
88
|
<p align="center">
|
|
432
|
-
<img src="assets/readme/score-gauge.svg" alt="
|
|
89
|
+
<img src="assets/readme/score-gauge.svg" alt="Worthiness scale: UNWORTHY 0–49, NEEDS WORK 50–79, WORTHY 80–99, FORGED 100, UNKNOWN when no tests exist." width="720" />
|
|
433
90
|
</p>
|
|
434
91
|
|
|
435
|
-
|
|
436
|
-
Generated by `npm run docs:gauge` and drift-locked in CI.</sub>
|
|
437
|
-
|
|
438
|
-
| Score | Verdict |
|
|
439
|
-
| --------- | --------------------------------------- |
|
|
440
|
-
| `0 – 49` | **UNWORTHY** |
|
|
441
|
-
| `50 – 79` | **NEEDS WORK** |
|
|
442
|
-
| `80 – 99` | **WORTHY** |
|
|
443
|
-
| `100` | **FORGED** |
|
|
444
|
-
| `null` | **UNKNOWN**: no test declarations found |
|
|
445
|
-
|
|
446
|
-
**How it is computed.** Severity sets a base deduction (`error −8`,
|
|
447
|
-
`warning −3`, `info −1`) and the evidence level discounts it: E2 pays in
|
|
448
|
-
full, E1 half (rounded down), E0 nothing. The total is normalized by suite
|
|
449
|
-
exposure, meaning deductions per test declaration rather than per file.
|
|
450
|
-
The terminal prints the same discounted numbers the score used; there is
|
|
451
|
-
no hidden second model. Details: [docs/SCORING.md](docs/SCORING.md) and the
|
|
452
|
-
[scoring guide](https://sergey-bar.github.io/Mjolnir/guide/scoring).
|
|
453
|
-
|
|
454
|
-
**What 100 does not mean.** It does not mean the software is correct, the
|
|
455
|
-
suite is adequate, or the product is free of defects. It means one thing:
|
|
456
|
-
**none of Mjölnir's evaluated rules produced a deduction under this scan
|
|
457
|
-
and this evidence model.**
|
|
92
|
+
Severity sets a base deduction (`error −8`, `warning −3`, `info −1`), discounted by evidence level: **E2** (deterministic proof) pays in full, **E1** (pattern) half, **E0** (observation) nothing. Normalized by suite exposure — deductions per test declaration, not per file.
|
|
458
93
|
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
## The evidence model
|
|
94
|
+
---
|
|
462
95
|
|
|
463
|
-
|
|
464
|
-
finding was checked. That is the difference between a tool that reports
|
|
465
|
-
patterns and a tool you can gate a release on.
|
|
466
|
-
|
|
467
|
-
**How sure — the evidence level.**
|
|
96
|
+
## Evidence you can gate on
|
|
468
97
|
|
|
469
98
|
| Level | Name | Means | Deduction |
|
|
470
99
|
| ------ | ------------------- | -------------------------------------------------- | --------- |
|
|
@@ -472,382 +101,74 @@ patterns and a tool you can gate a release on.
|
|
|
472
101
|
| **E1** | Pattern evidence | A pattern strongly tied to the defect matched | Half |
|
|
473
102
|
| **E0** | Observation | Worth knowing. Not a claim that anything is wrong. | Zero |
|
|
474
103
|
|
|
475
|
-
|
|
476
|
-
certain it matched what it looked for and still be looking at a heuristic.
|
|
477
|
-
E1 findings are there to be read and judged, never applied blindly, and
|
|
478
|
-
that boundary is stamped on the finding in the terminal, the JSON and the
|
|
479
|
-
agent handoff.
|
|
104
|
+
Static scans stop at **L2** (proven in code). Give Mjölnir a real run report (Playwright JSON, Jest/Vitest JSON, JUnit XML) and findings lift to **L3–L5** (file ran, test ran, run agrees).
|
|
480
105
|
|
|
481
|
-
|
|
482
|
-
reading your code. Give Mjölnir the report of a real test run and it can
|
|
483
|
-
confirm that the code actually ran.
|
|
106
|
+
---
|
|
484
107
|
|
|
485
|
-
|
|
486
|
-
<img src="assets/readme/trust-ladder.svg" alt="The trust ladder from L0 to L5. L0 to L2 come from reading the code; L3 to L5 need a real run report, marked by a break in the ladder." width="100%" />
|
|
487
|
-
</p>
|
|
488
|
-
|
|
489
|
-
| Level | In plain words | What it takes |
|
|
490
|
-
| ------ | ---------------------- | ---------------------------------------------- |
|
|
491
|
-
| **L0** | Noted | Reading the code |
|
|
492
|
-
| **L1** | Looks like the problem | Reading the code: a pattern matched |
|
|
493
|
-
| **L2** | Proven in the code | Reading the code: the defect is structural |
|
|
494
|
-
| **L3** | The file ran | A run report shows the finding's file executed |
|
|
495
|
-
| **L4** | The test ran | A run report shows the finding's test executed |
|
|
496
|
-
| **L5** | The run agrees | The run's own result confirms the defect class |
|
|
497
|
-
|
|
498
|
-
A static scan stops at L2. Only a real run report (Playwright JSON, Jest or
|
|
499
|
-
Vitest JSON, JUnit XML) can lift a finding to L3 or above, so a finding
|
|
500
|
-
that was never seen running can never claim it was. Definitions:
|
|
501
|
-
[docs/TERMINOLOGY.md](docs/TERMINOLOGY.md).
|
|
502
|
-
|
|
503
|
-
### How much of this is measured
|
|
504
|
-
|
|
505
|
-
**<!-- census:measured-of-total -->74 of 79<!-- /census:measured-of-total --> rules carry a false-positive rate measured against real OSS code**
|
|
506
|
-
(at least 10 hand-classified findings each; see
|
|
507
|
-
[docs/FP-AUDIT.md](docs/FP-AUDIT.md)). The other <!-- census:unmeasured -->5<!-- /census:unmeasured --> ship on the author's estimate and say so, rule by rule,
|
|
508
|
-
in `mjolnir explain`. `mjolnir rules --unmeasured` lists them, and every
|
|
509
|
-
scan footer reports how many of the rules that actually _fired_ are
|
|
510
|
-
measured.
|
|
511
|
-
|
|
512
|
-
Rates stay public when they are bad. QA-TEST-001 (a committed `.only`)
|
|
513
|
-
audits poorly on real repositories and sits in quarantine for it. The live
|
|
514
|
-
number for every rule, QA-PW-141 included, is in the audit.
|
|
515
|
-
|
|
516
|
-
### Trust tiers
|
|
517
|
-
|
|
518
|
-
Tiers follow the measured false-positive rate, not opinion:
|
|
519
|
-
|
|
520
|
-
| Tier | Measured FP | Behavior |
|
|
521
|
-
| -------------- | ---------------------------- | -------------------------------------------- |
|
|
522
|
-
| **core** | ≤ 10% | Default report, gates |
|
|
523
|
-
| **extended** | ≤ 30% | Default report, lower confidence |
|
|
524
|
-
| **quarantine** | > 30% or explicitly declared | `--strict` only, capped to info, never gates |
|
|
525
|
-
| _unmeasured_ | n < 10 | Cannot be promoted to core until measured |
|
|
526
|
-
|
|
527
|
-
FP bands can only demote a tier — they never promote a rule out of
|
|
528
|
-
`quarantine` if it was explicitly declared there. An explicitly
|
|
529
|
-
quarantined rule stays quarantine regardless of its measured FP rate.
|
|
530
|
-
|
|
531
|
-
Promotion, demotion and per-language maturity:
|
|
532
|
-
[rule lifecycle](https://sergey-bar.github.io/Mjolnir/reference/rule-lifecycle).
|
|
533
|
-
|
|
534
|
-
### Why this is not a linter
|
|
535
|
-
|
|
536
|
-
Linters tell you whether code follows rules. Mjölnir tells you whether your
|
|
537
|
-
verification can be trusted.
|
|
538
|
-
|
|
539
|
-
| | Linters (ESLint, SonarQube) | Coverage tools | AI code review | **Mjölnir** |
|
|
540
|
-
| -------------------------------------------------------------------- | :-------------------------: | :------------: | :------------: | :--------------: |
|
|
541
|
-
| Scores the **verification system**, not the product code | No | No | No | Yes |
|
|
542
|
-
| CI workflow integrity (`continue-on-error`, `\|\| true`) | No | No | only the diff | Yes |
|
|
543
|
-
| Grades Playwright locator resilience (Selector Health) | No | No | No | Yes |
|
|
544
|
-
| Reads real run data for `TRUE-FLAKE` verdicts | No | No | No | Yes |
|
|
545
|
-
| Publishes measured false-positive rates where corpus evidence exists | No | No | No | Yes |
|
|
546
|
-
| Flags tests with no assertions | Yes\* | No | sometimes | Yes |
|
|
547
|
-
| Catches hard sleeps (`waitForTimeout`, `time.sleep`) | Yes\* | No | sometimes | Yes |
|
|
548
|
-
| Deterministic (same input, same output) | Yes | Yes | No | Yes |
|
|
549
|
-
| Cost per scan | free | free | tokens | **zero** (local) |
|
|
550
|
-
|
|
551
|
-
<sub>\*Covered by `eslint-plugin-jest` and `eslint-plugin-playwright`
|
|
552
|
-
(`expect-expect`, `no-wait-for-timeout`) and by SonarQube's own assertion
|
|
553
|
-
rules. The columns describe default behavior for test-suite verification;
|
|
554
|
-
plugins, paid tiers and custom rules change some answers. This is a
|
|
555
|
-
positioning summary, not a benchmark.</sub>
|
|
556
|
-
|
|
557
|
-
Use AI review too. It catches nuance, intent and design flaws no pattern
|
|
558
|
-
can find. Mjölnir catches what AI review overlooks because it looks
|
|
559
|
-
intentional: a committed `.only`, a swallowed exit code, a
|
|
560
|
-
`continue-on-error` on a test job. Those need scanning, not reasoning.
|
|
561
|
-
|
|
562
|
-
<br />
|
|
563
|
-
|
|
564
|
-
## Runtime forensics
|
|
565
|
-
|
|
566
|
-
Static analysis reasons about code that never ran. Forensics reads what
|
|
567
|
-
actually happened: Playwright JSON, Jest JSON, Vitest JSON, and JUnit XML
|
|
568
|
-
from any runner.
|
|
569
|
-
|
|
570
|
-
```bash
|
|
571
|
-
mjolnir forensics ./test-results/
|
|
572
|
-
```
|
|
573
|
-
|
|
574
|
-
```text
|
|
575
|
-
▍ FLAKINESS LEADERBOARD
|
|
576
|
-
|
|
577
|
-
3 tests · 1 failed · 1 flaky · 1 retried
|
|
578
|
-
|
|
579
|
-
TRUE-FLAKE completes checkout with saved card (e2e/checkout.spec.ts)
|
|
580
|
-
████████████████████ 6.0s · 2 attempts
|
|
581
|
-
FAILING declines an expired card (e2e/checkout.spec.ts)
|
|
582
|
-
████░░░░░░░░░░░░░░░░ 1.1s · 1 attempt
|
|
583
|
-
```
|
|
584
|
-
|
|
585
|
-
`TRUE-FLAKE` does not mean the test retried. It means the test **failed at
|
|
586
|
-
least one attempt and then finished green**: a lucky pass, flagged whatever
|
|
587
|
-
the final checkmark says. `mjolnir triage` turns that history into a
|
|
588
|
-
quarantine proposal, and `mjolnir pw-report` summarizes a run. The same run
|
|
589
|
-
reports are what lift findings to trust levels L3 and above.
|
|
590
|
-
|
|
591
|
-
<br />
|
|
592
|
-
|
|
593
|
-
## CI integrity
|
|
594
|
-
|
|
595
|
-
A test can pass while the pipeline around it cannot fail. Mjölnir reads the
|
|
596
|
-
workflows too: `continue-on-error`, `|| true`, exit codes that never
|
|
597
|
-
propagate, always-success steps, reports consumed but never generated, and
|
|
598
|
-
gates skipped on the events that should block. Each finding names the job,
|
|
599
|
-
the step and the line, and carries its own evidence level.
|
|
600
|
-
|
|
601
|
-
Generate an advisory PR workflow first. It reports findings without blocking
|
|
602
|
-
and is pinned to this version, so adopting it cannot change gate semantics:
|
|
603
|
-
|
|
604
|
-
```bash
|
|
605
|
-
mjolnir ci install
|
|
606
|
-
```
|
|
607
|
-
|
|
608
|
-
The generated workflow is advisory by default. A direct
|
|
609
|
-
`uses: Sergey-Bar/Mjolnir@...` composite Action is blocking by default; set
|
|
610
|
-
`fail-on` explicitly when its policy differs.
|
|
611
|
-
|
|
612
|
-
After reviewing existing debt, opt into an enforcing gate:
|
|
108
|
+
## AI agents: SCAN → EVIDENCE → HANDOFF → AGENT → RE-SCAN → PROOF
|
|
613
109
|
|
|
614
110
|
```bash
|
|
615
|
-
|
|
616
|
-
|
|
617
|
-
|
|
618
|
-
In your existing workflow, pin the action and explicitly opt into an error gate:
|
|
619
|
-
|
|
620
|
-
```yaml
|
|
621
|
-
- uses: Sergey-Bar/Mjolnir@4a588bc62d517bc85fc44c0eae64c6587d3bf70b0
|
|
622
|
-
with:
|
|
623
|
-
scope: changed
|
|
624
|
-
fail-on: error
|
|
625
|
-
```
|
|
111
|
+
# MCP server over stdio (read-only tools: scan, explain, diff, verify, forensics, triage, pw-report)
|
|
112
|
+
claude mcp add mjolnir -- npx -y mjolnir-qa mcp
|
|
626
113
|
|
|
627
|
-
|
|
114
|
+
# Deterministic handoff plan for any agent
|
|
115
|
+
mjolnir handoff
|
|
628
116
|
|
|
629
|
-
|
|
630
|
-
|
|
117
|
+
# Install into your repo's agent surfaces (.claude/, .cursor/, .kilo/, AGENTS.md)
|
|
118
|
+
mjolnir install
|
|
631
119
|
```
|
|
632
120
|
|
|
633
|
-
|
|
634
|
-
`security-events: write` at workflow or job scope):
|
|
121
|
+
Every finding in a handoff carries its boundary: **E2** = deterministic, apply fix; **E1** = requires confirmation, do not suppress blindly.
|
|
635
122
|
|
|
636
|
-
|
|
637
|
-
- run: npx mjolnir-qa@4.0.0 --format sarif > mjolnir.sarif
|
|
638
|
-
continue-on-error: true
|
|
639
|
-
- uses: github/codeql-action/upload-sarif@v3
|
|
640
|
-
if: ${{ !cancelled() }}
|
|
641
|
-
with:
|
|
642
|
-
sarif_file: mjolnir.sarif
|
|
643
|
-
```
|
|
644
|
-
|
|
645
|
-
On GitLab, `--format codequality` writes the Code Quality report that the
|
|
646
|
-
MR widget and diff annotations read ([docs/GITLAB-CI.md](docs/GITLAB-CI.md)).
|
|
647
|
-
Editor and pipeline setup: [docs/SARIF-INTEGRATION.md](docs/SARIF-INTEGRATION.md).
|
|
648
|
-
|
|
649
|
-
### Changed-scope attribution
|
|
650
|
-
|
|
651
|
-
```bash
|
|
652
|
-
npx mjolnir-qa@4.0.0 --scope changed
|
|
653
|
-
```
|
|
654
|
-
|
|
655
|
-
Findings are attributed to the lines your branch added, measured against
|
|
656
|
-
the **merge-base**. The scope is the same file set a full scan discovers
|
|
657
|
-
(TS/JS specs and adapter configs, `test_*.py`, `*Test.java`, `*Tests.cs`,
|
|
658
|
-
`.github/workflows/*.yml`), plus uncommitted and untracked changes, so it
|
|
659
|
-
works before you commit. The base resolves
|
|
660
|
-
`main → master → origin/main → origin/master → origin/HEAD`; override it
|
|
661
|
-
with `--base <ref>`.
|
|
662
|
-
|
|
663
|
-
When the merge-base cannot be resolved (a shallow clone, a detached HEAD, a
|
|
664
|
-
target outside git), findings fall back to whole-file attribution **and the
|
|
665
|
-
report says so.** A silent fallback would be the same kind of defect this
|
|
666
|
-
tool exists to catch.
|
|
667
|
-
|
|
668
|
-
<br />
|
|
669
|
-
|
|
670
|
-
## AI agents
|
|
671
|
-
|
|
672
|
-
Findings are only worth something if something acts on them.
|
|
673
|
-
|
|
674
|
-
```text
|
|
675
|
-
SCAN → EVIDENCE → HANDOFF → AGENT → RE-SCAN → PROOF
|
|
676
|
-
```
|
|
677
|
-
|
|
678
|
-
**AI writes the fix. Mjölnir verifies it.** The proof comes from the
|
|
679
|
-
re-scan, never from the agent's own report of success.
|
|
680
|
-
|
|
681
|
-
| Command | What the agent gets |
|
|
682
|
-
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
683
|
-
| `mjolnir mcp` | An [MCP](https://modelcontextprotocol.io) server over stdio. `scan`, `explain`, `diff`, `verify`, `forensics`, `triage`, and `pw-report` are read-only callable tools. |
|
|
684
|
-
| `mjolnir handoff` | A saved `--json` report becomes a deterministic Markdown plan: what was detected, the evidence boundary per finding, what must **not** change, how to verify. |
|
|
685
|
-
| `mjolnir install` | Writes into the agent surfaces your repo already has (`.claude/`, `.cursor/`, `.kilo/`, `AGENTS.md`) so the agent re-scans before it claims it is done. |
|
|
686
|
-
|
|
687
|
-
Add it to a client that ships its own CLI:
|
|
688
|
-
|
|
689
|
-
```bash
|
|
690
|
-
claude mcp add mjolnir -- npx -y mjolnir-qa@4.0.0 mcp
|
|
691
|
-
```
|
|
692
|
-
|
|
693
|
-
Or to any client that takes an `mcpServers` block:
|
|
694
|
-
|
|
695
|
-
```json
|
|
696
|
-
{
|
|
697
|
-
"mcpServers": {
|
|
698
|
-
"mjolnir": {
|
|
699
|
-
"command": "npx",
|
|
700
|
-
"args": ["-y", "mjolnir-qa@4.0.0", "mcp"]
|
|
701
|
-
}
|
|
702
|
-
}
|
|
703
|
-
}
|
|
704
|
-
```
|
|
705
|
-
|
|
706
|
-
**The guardrail matters more than the convenience.** Every finding in a
|
|
707
|
-
handoff carries its boundary. **E2** says _deterministic: check the location
|
|
708
|
-
and apply the fix_. **E1** says _REQUIRES CONFIRMATION: the observation
|
|
709
|
-
alone does not prove the defect_. An agent that fixes E1 blindly, suppresses
|
|
710
|
-
a rule, or edits a rule to raise the score is doing exactly what this tool
|
|
711
|
-
exists to catch, so the handoff says so in the prompt, next to the finding.
|
|
712
|
-
|
|
713
|
-
<br />
|
|
123
|
+
---
|
|
714
124
|
|
|
715
125
|
## Trust and security
|
|
716
126
|
|
|
717
|
-
**Local-first, zero telemetry.** No network-capable API (`
|
|
718
|
-
|
|
719
|
-
|
|
720
|
-
fails the build if one appears. It also bars `eval` and `new Function`.
|
|
721
|
-
Scanning untrusted code never executes it: static analysis reads source
|
|
722
|
-
text, and forensics parses report files that already exist on disk.
|
|
723
|
-
|
|
724
|
-
Two caveats: `npx` itself fetches the package before anything runs, and the
|
|
725
|
-
guarantee covers `src/`, not third-party plugins.
|
|
726
|
-
|
|
727
|
-
**Plugins are not sandboxed.** JS plugins (`mjolnir-rules/*.mjs`, or npm
|
|
728
|
-
packages listed under `"plugins"`) run with full Node privileges, the same
|
|
729
|
-
trust model as ESLint or Vitest plugins. Loading them is opt-in **per
|
|
730
|
-
scan**: without `--enable-plugins` (or `MJOLNIR_ENABLE_PLUGINS=1`) their
|
|
731
|
-
sources are never loaded, and a notice on stderr lists what was skipped.
|
|
732
|
-
JSON rule manifests execute no code, and core rule-ID prefixes are reserved
|
|
733
|
-
so a plugin cannot impersonate one. Report vulnerabilities through
|
|
734
|
-
[SECURITY.md](SECURITY.md).
|
|
735
|
-
|
|
736
|
-
**It runs on itself.** A verification trust engine has no standing unless
|
|
737
|
-
it is itself verifiable. Every CI run scans this repository with the build
|
|
738
|
-
that same run produced. The gate fails on any error-severity finding, and
|
|
739
|
-
also on a **partial** scan or a **crashed rule**, because a truncated
|
|
740
|
-
self-scan that reports nothing is the false green this project exists to
|
|
741
|
-
catch. `mjolnir doctor` re-audits the rule base in the same run (fixture
|
|
742
|
-
firewall, tier honesty, the core-tier cap), and an INCONCLUSIVE check fails
|
|
743
|
-
exactly like a failing one. Both reports are uploaded as build artifacts.
|
|
744
|
-
|
|
745
|
-
### Exit codes and the machine contract
|
|
746
|
-
|
|
747
|
-
Frozen, so you can build CI logic on them:
|
|
748
|
-
|
|
749
|
-
| Exit code | Meaning |
|
|
750
|
-
| --------- | --------------------------------------------------------------- |
|
|
751
|
-
| `0` | Clean: no findings at or above the gate |
|
|
752
|
-
| `1` | Findings at or above the gate |
|
|
753
|
-
| `2` | Partial scan (time budget hit, unreadable files). Never blocks. |
|
|
754
|
-
| `10` | Usage error (bad flag, missing target) |
|
|
755
|
-
| `20` | Internal error |
|
|
756
|
-
|
|
757
|
-
`2` is deliberately distinct from `0`: a scan that did not finish has not
|
|
758
|
-
found nothing. It has not finished looking.
|
|
759
|
-
|
|
760
|
-
Everything a machine consumes (MCP tool results, `--json`, SARIF 2.1) comes
|
|
761
|
-
from one canonical result under a versioned, **additive-only** schema
|
|
762
|
-
(`schemaVersion: 1`, `contractVersion: 1`), so no consumer has to rebuild
|
|
763
|
-
meaning from rendered text. See [the machine contract](docs/machine-contract.md).
|
|
764
|
-
Rule IDs (`QA-<FAMILY>-NNN`) are immutable once shipped and never reused.
|
|
127
|
+
- **Local-first, zero telemetry.** No network-capable API exists in `src/`. A contract test (`privacy-network-isolation.spec.ts`) fails the build if one appears. Scanning untrusted code never executes it.
|
|
128
|
+
- **Plugins are not sandboxed** — same trust model as ESLint/Vitest plugins. Opt-in per scan with `--enable-plugins`.
|
|
129
|
+
- **Self-verifying.** Every CI run scans this repository with the build it produced. The gate fails on any error finding, a partial scan, or a crashed rule. `mjolnir doctor` re-audits the rule base in the same run.
|
|
765
130
|
|
|
766
|
-
|
|
131
|
+
---
|
|
767
132
|
|
|
768
|
-
##
|
|
133
|
+
## Exit codes (frozen contract)
|
|
769
134
|
|
|
770
|
-
|
|
771
|
-
|
|
772
|
-
|
|
773
|
-
|
|
774
|
-
|
|
775
|
-
|
|
776
|
-
|
|
777
|
-
real risk is a different question, and this tool does not answer it.
|
|
778
|
-
- **<!-- census:unmeasured-of-total -->5 of 79<!-- /census:unmeasured-of-total --> rules ship on an estimate**, not a measured rate. Each one
|
|
779
|
-
says so on its own finding.
|
|
780
|
-
- **E1 is not E2.** Heuristic findings are worth reading, not worth applying
|
|
781
|
-
blindly.
|
|
782
|
-
- **An empty repo scores `null`, never 100.**
|
|
783
|
-
- **A file named `*.spec.ts` with no test declarations does not count as coverage.** A repo whose only spec files contain imports or types (zero `it`/`test` calls) scores `null`, not 100.
|
|
135
|
+
| Code | Meaning |
|
|
136
|
+
| ---- | -------------------------------------------------------------- |
|
|
137
|
+
| `0` | Clean — no findings at or above the gate |
|
|
138
|
+
| `1` | Findings at or above the gate |
|
|
139
|
+
| `2` | Inconclusive — partial scan, unreadable files, or crashed rule |
|
|
140
|
+
| `10` | Usage error |
|
|
141
|
+
| `20` | Internal error |
|
|
784
142
|
|
|
785
|
-
|
|
143
|
+
`2` is distinct from `0`: an incomplete scan has not found nothing — it has not finished looking. No flag turns it into `0`.
|
|
786
144
|
|
|
787
|
-
|
|
788
|
-
|
|
789
|
-
The full docs site is at <https://sergey-bar.github.io/Mjolnir/>.
|
|
790
|
-
|
|
791
|
-
| Document | What's in it |
|
|
792
|
-
| ------------------------------------------------------------------ | --------------------------------------------------------- |
|
|
793
|
-
| [docs/EXECUTIVE-ROI.md](docs/EXECUTIVE-ROI.md) | Business case for decision-makers |
|
|
794
|
-
| [docs/SCORING.md](docs/SCORING.md) | Score normalization and evidence weighting |
|
|
795
|
-
| [docs/TERMINOLOGY.md](docs/TERMINOLOGY.md) | Canonical vocabulary: one word per concept |
|
|
796
|
-
| [docs/FP-AUDIT.md](docs/FP-AUDIT.md) | Measured false-positive rates and the method |
|
|
797
|
-
| [docs/RULE-LIFECYCLE.md](docs/RULE-LIFECYCLE.md) | Rule states, tiers, suppression, deprecation |
|
|
798
|
-
| [docs/VERSIONING.md](docs/VERSIONING.md) | Semver policy, frozen surfaces, deprecation cycle |
|
|
799
|
-
| [docs/machine-contract.md](docs/machine-contract.md) | The canonical machine-readable result |
|
|
800
|
-
| [docs/SARIF-INTEGRATION.md](docs/SARIF-INTEGRATION.md) | SARIF output and editor or CI setup |
|
|
801
|
-
| [docs/GITLAB-CI.md](docs/GITLAB-CI.md) | GitLab: Code Quality report, MR recipe, gate |
|
|
802
|
-
| [docs/rules/](docs/rules/) | Generated per-rule catalog |
|
|
803
|
-
| [CONTRIBUTING.md](CONTRIBUTING.md) | Dev setup and contribution workflow |
|
|
804
|
-
| [SUPPORT.md](SUPPORT.md) | Where to ask, report and get help |
|
|
805
|
-
| [SECURITY.md](SECURITY.md) | Vulnerability reporting |
|
|
806
|
-
| [docs/MIGRATION-3.0.0.md](docs/MIGRATION-3.0.0.md) | 3.0.0 migration checklist |
|
|
807
|
-
| [docs/INSTALLATION-3.0.0.md](docs/INSTALLATION-3.0.0.md) | Reproducible CLI, Action, and MCP installation |
|
|
808
|
-
| [docs/ROLLBACK-3.0.0.md](docs/ROLLBACK-3.0.0.md) | Rollback and corrective-release procedure |
|
|
809
|
-
| [docs/RELEASE-3.0.0-READINESS.md](docs/RELEASE-3.0.0-READINESS.md) | 3.0.0 release gates, migration, QA, and rollout checklist |
|
|
810
|
-
| [CHANGELOG.md](CHANGELOG.md) | Release history |
|
|
811
|
-
|
|
812
|
-
### Status
|
|
813
|
-
|
|
814
|
-
**Version 3.0.0.** The JSON schema and the exit codes are frozen contracts.
|
|
815
|
-
TypeScript and Python have the broadest measured coverage. Java and C# are
|
|
816
|
-
newer; read them through the
|
|
817
|
-
[maturity table](https://sergey-bar.github.io/Mjolnir/reference/rule-lifecycle).
|
|
818
|
-
What comes next, with no invented dates:
|
|
819
|
-
[the public roadmap](https://sergey-bar.github.io/Mjolnir/reference/roadmap).
|
|
820
|
-
|
|
821
|
-
### Contributing
|
|
822
|
-
|
|
823
|
-
New rules are the easiest first contribution. One command scaffolds the
|
|
824
|
-
rule with its must-fire **and** must-not-fire fixtures. The generated rule
|
|
825
|
-
fails its own fixtures on purpose until real detection is written, because
|
|
826
|
-
a stub that ships is a rule nobody measured:
|
|
145
|
+
---
|
|
827
146
|
|
|
828
|
-
|
|
829
|
-
mjolnir create-rule QA-PW-140 --title "Screenshot without diff bound"
|
|
830
|
-
```
|
|
147
|
+
## What Mjölnir cannot tell you
|
|
831
148
|
|
|
832
|
-
|
|
833
|
-
|
|
149
|
+
- It does not run your tests. A clean scan ≠ a passing suite.
|
|
150
|
+
- It cannot tell you an assertion is **wrong** — only that it **cannot fail**.
|
|
151
|
+
- It does not prove business correctness.
|
|
152
|
+
- A score of 100 means: _none of Mjölnir's evaluated rules produced a deduction under this scan and evidence model._
|
|
153
|
+
- 6 of 79 rules ship on an author estimate, not a measured FP rate. Each says so on its finding.
|
|
154
|
+
- E1 is not E2 — heuristic findings are worth reading, not worth applying blindly.
|
|
834
155
|
|
|
835
|
-
|
|
156
|
+
---
|
|
836
157
|
|
|
837
158
|
<div align="center">
|
|
838
159
|
|
|
839
160
|
<img src="assets/readme/closing.svg" alt="Run it on your repo." width="100%" />
|
|
840
161
|
|
|
841
162
|
```bash
|
|
842
|
-
npx mjolnir-qa@
|
|
163
|
+
npx mjolnir-qa@6.0.0
|
|
843
164
|
```
|
|
844
165
|
|
|
845
|
-
[Read the guide](https://sergey-bar.github.io/Mjolnir/guide/getting-started) · [Docs
|
|
166
|
+
[Read the guide](https://sergey-bar.github.io/Mjolnir/guide/getting-started) · [Docs](https://sergey-bar.github.io/Mjolnir/) · [npm](https://www.npmjs.com/package/mjolnir-qa)
|
|
846
167
|
|
|
847
168
|
<br />
|
|
848
169
|
|
|
849
|
-
Don't ask whether the tests passed
|
|
850
|
-
Ask whether the evidence proves they deserve to be trusted
|
|
170
|
+
**Don't ask whether the tests passed.**
|
|
171
|
+
**Ask whether the evidence proves they deserve to be trusted.**
|
|
851
172
|
|
|
852
173
|
<sub>Built by [Sergey Bar](https://www.linkedin.com/in/sergeybar/) · MIT licensed</sub>
|
|
853
174
|
|