slopbouncer 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +37 -0
- data/README.md +170 -40
- data/lib/slopbouncer/commit_hook.rb +66 -0
- data/lib/slopbouncer/commit_log.rb +46 -0
- data/lib/slopbouncer/commit_memory.rb +47 -0
- data/lib/slopbouncer/compare.rb +104 -27
- data/lib/slopbouncer/explained.rb +54 -0
- data/lib/slopbouncer/form.rb +45 -0
- data/lib/slopbouncer/git.rb +63 -7
- data/lib/slopbouncer/install.rb +54 -4
- data/lib/slopbouncer/ruby_tests.rb +73 -13
- data/lib/slopbouncer/terminal_report.rb +29 -6
- data/lib/slopbouncer/version.rb +1 -1
- data/lib/slopbouncer.rb +187 -18
- metadata +17 -6
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: d7ae3e83e53cde3d529fcfe6d61a45569b1437012f0cbc0e6419fa3870773bf5
|
|
4
|
+
data.tar.gz: 2a5e54814bf04382ab07924353fccd2fc4c9cf3124596a2dfeaff3e589b28a5f
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: aab2c9d00992a192944112d05bfc8d38c9482405810f2ee940e00513679e58eeff09ce08fe0a326b7ae1a0c024c57fa65ba892ca4f61eabd7355fafbfeb23908
|
|
7
|
+
data.tar.gz: 85a0b413d120416d88806af0eda0a2c64805dd06cd29158d572890bbf4fd13633822ca95777404cabb667fd1d6e12dacd3892ce5471648645058c2fb4bfc7d63
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
What changed in each version of Slopbouncer.
|
|
4
|
+
|
|
5
|
+
## 0.4.0 - 2026-11-17
|
|
6
|
+
|
|
7
|
+
- Slopbouncer checks commits. `slopbouncer install` sets up git's `commit-msg` hook, wherever git runs hooks from, `.git/hooks` or `core.hooksPath`. A `commit-msg` hook the repository already has gets one line, after its first.
|
|
8
|
+
- The hook reads only the test files as staged. A commit that weakens an existing test is stopped, with a line for each finding and the "Test changes:" lines to add to the message. A message that names each flagged test with a reason lets it through.
|
|
9
|
+
- Once a message has explained a finding, it is remembered in `.git/slopbouncer/explained.json`, so the same change committed again goes through quietly. A change to the assertions declared equivalent is asked about every time.
|
|
10
|
+
- Each commit the hook checks is logged in `.git/slopbouncer/log.jsonl`. `slopbouncer stats` counts the commits checked, stopped, explained, already explained and not checked; `--json` prints the counts as JSON.
|
|
11
|
+
- When the hook cannot run, it says the commit was not checked and lets it through.
|
|
12
|
+
- `slopbouncer install claude-code` sets up the commit hook and adds rules to `.claude/settings.json` that stop Claude Code committing with `--no-verify` or `-n`. Naming Cursor, Gemini CLI or Codex sets up the commit hook.
|
|
13
|
+
- The turn-end hooks are still there, and off unless asked for: `slopbouncer install claude-code --turn-end`.
|
|
14
|
+
- Two tests with the same name in one file are told apart. Before, a change anywhere in such a file reported the first test's assertions as removed.
|
|
15
|
+
- A test that was only reformatted is not flagged. Assertions are compared by what they parse to, so quote style, white space and line breaks, parentheses round arguments, a block's braces against `do` and `end`, and a trailing comma no longer count as a change.
|
|
16
|
+
- At a terminal, more than 30 findings are shown as a count of each rule in each file. `--all` lists every finding. Piped output and `--json` are complete as before.
|
|
17
|
+
- A coding agent is told about the first 20 findings, how many more there are and how to list them all.
|
|
18
|
+
- `--help` and `-h` print how to use the command. `--version` and `-v` print the version.
|
|
19
|
+
- Run outside a git repository, the command says so in one line. Any other git failure is reported as git's first line of error.
|
|
20
|
+
- It runs on Ruby 3.2 and later. Before, it asked for Ruby 3.4.
|
|
21
|
+
- Its reports say tests look weaker, where they said tests got weaker. A finding is something to look at, and the tool cannot tell whether the change was intended.
|
|
22
|
+
- An edited assertion is named by the side that changed. `expected_value_changed` is now only for what an assertion expects. When the thing it checks is edited, that is `subject_changed`. When Slopbouncer cannot tell the sides apart, as with a helper of your own or an assertion's message, that is `assertion_changed`. Before, all three were reported as `expected_value_changed`.
|
|
23
|
+
- A project can declare assertions that make the same check under `equivalents` in `.slopbouncer.yml`. One replaced by an equivalent with the same arguments is not flagged. A change to the declarations is flagged as `equivalents_changed`.
|
|
24
|
+
- A coding agent's hook sends the agent back only for findings it has not been shown for the work in hand, and lets it finish when nothing is new. It forgets a finding once it goes, and forgets everything when HEAD moves. The command run by hand, the pre-push stage and `--json` still report everything.
|
|
25
|
+
- When a change touches nothing but tests, assertions written another way, one for one, are shown at a terminal and to an agent as one line: "Test refactor: 23 assertions reworded in 6 files, no other code changed." `--all`, piped output and `--json` list them. Anything that could be a weaker test is still reported in full, and an agent is not sent back for the line alone.
|
|
26
|
+
- A method defined in the tests that changed, and that tests assert through, is reported once as `helper_changed`, with how many tests depend on it. Slopbouncer cannot judge such a change and asks for it to be looked at.
|
|
27
|
+
|
|
28
|
+
## 0.3.0 - 2026-11-12
|
|
29
|
+
|
|
30
|
+
- At a terminal the report is grouped by file, with each rule in colour and a count at the end. Sent to a file or another program it stays one plain line for each finding.
|
|
31
|
+
- `--color` gives the terminal layout wherever the output goes. `NO_COLOR` keeps the layout and leaves the colour out.
|
|
32
|
+
- At a terminal, the command says when no tests got weaker.
|
|
33
|
+
|
|
34
|
+
## 0.2.0 - 2026-11-12
|
|
35
|
+
|
|
36
|
+
- Reads RSpec: expectations with their matchers, one-line specs with no name, `:skip` and `:pending` beside a name, `should` and `should_not`, groups with two descriptions, and shared examples.
|
|
37
|
+
- A `before`, `after` or `around` hook that makes an assertion is read as a test of its own, and the tests it covers are not reported as asserting nothing.
|
data/README.md
CHANGED
|
@@ -1,20 +1,160 @@
|
|
|
1
1
|
# Slopbouncer
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[](https://rubygems.org/gems/slopbouncer)
|
|
4
|
+
[](https://github.com/mainline-dev/slopbouncer/actions/workflows/test.yml)
|
|
4
5
|
|
|
5
|
-
A
|
|
6
|
+
A commit hook that catches weakened tests, whoever weakened them.
|
|
7
|
+
|
|
8
|
+
A coding agent's job is to make the checks pass. One way is to weaken the check. When a commit weakens an existing test, Slopbouncer stops it until the commit message says which behaviour changed and why. It reads code and never runs it, so it takes a fraction of a second.
|
|
6
9
|
|
|
7
10
|
Status: in development. It reads Ruby tests today: Minitest, as `it` blocks or `test_` methods, and RSpec.
|
|
8
11
|
|
|
12
|
+
## Setting it up
|
|
13
|
+
|
|
14
|
+
It needs Ruby 3.2 or later and git.
|
|
15
|
+
|
|
16
|
+
```shell
|
|
17
|
+
gem install slopbouncer
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
From the root of your repository:
|
|
21
|
+
|
|
22
|
+
```shell
|
|
23
|
+
slopbouncer install
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
This sets up git's `commit-msg` hook, so every commit is checked, whoever or whatever makes it. It goes wherever git runs hooks from, `.git/hooks` or the directory `core.hooksPath` names, as Husky and Lefthook set. If the repository already has a `commit-msg` hook, Slopbouncer adds one line to it, after its first, and the rest of the hook runs as before. Running it again changes nothing.
|
|
27
|
+
|
|
28
|
+
Name your coding agent to set it up for that agent too:
|
|
29
|
+
|
|
30
|
+
| Agent | Command | What it adds |
|
|
31
|
+
|---|---|---|
|
|
32
|
+
| Claude Code | `slopbouncer install claude-code` | The commit hook, and rules in `.claude/settings.json` that stop Claude Code committing with `--no-verify` |
|
|
33
|
+
| Cursor | `slopbouncer install cursor` | The commit hook |
|
|
34
|
+
| Gemini CLI | `slopbouncer install gemini` | The commit hook |
|
|
35
|
+
| Codex | `slopbouncer install codex` | The commit hook |
|
|
36
|
+
|
|
37
|
+
For Claude Code, `--local` writes the rules to `.claude/settings.local.json` instead, which keeps them out of the settings your team shares.
|
|
38
|
+
|
|
39
|
+
The hook needs the `slopbouncer` command on the `PATH` of whatever commits. When it cannot run, because the command is not there, its settings file cannot be read, or git fails, it says so and lets the commit through, with "This commit was not checked." A broken setup never stops anyone committing.
|
|
40
|
+
|
|
41
|
+
## When it stops a commit
|
|
42
|
+
|
|
43
|
+
It reads only what the commit will hold: the test files as staged, whatever else is in the working tree. A commit that changes no tests, or only adds them, goes through without a word. One that weakens an existing test is stopped, with a line for each finding and the lines to add to the commit message:
|
|
44
|
+
|
|
45
|
+
```
|
|
46
|
+
Slopbouncer: this commit looks to weaken 1 existing test.
|
|
47
|
+
|
|
48
|
+
test/basket_test.rb: Basket adds up: assertion_removed: assert_equal 2, size
|
|
49
|
+
|
|
50
|
+
Put each test back, or add this to the commit message, with a reason on each line:
|
|
51
|
+
|
|
52
|
+
Test changes:
|
|
53
|
+
- test/basket_test.rb: Basket adds up:
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Fill in a reason after each test and commit again:
|
|
57
|
+
|
|
58
|
+
```
|
|
59
|
+
Make the basket simpler
|
|
60
|
+
|
|
61
|
+
Test changes:
|
|
62
|
+
- test/basket_test.rb: Basket adds up: size moved to its own test
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
A test named with no reason does not count. Once a message has explained a finding, Slopbouncer remembers it, so the same change committed again, after a reset or an amend, goes through quietly. A change to the assertions declared equivalent is asked about every time, however often it was explained. The memory is `.git/slopbouncer/explained.json`, in the git directory, and is never committed.
|
|
66
|
+
|
|
67
|
+
A change that only rewords tests, with no other code changed, is not stopped.
|
|
68
|
+
|
|
69
|
+
## How often it stops a commit
|
|
70
|
+
|
|
71
|
+
Every commit the hook checks is logged in `.git/slopbouncer/log.jsonl`. After a week or so, see how often it stopped one and why the others went through:
|
|
72
|
+
|
|
73
|
+
```shell
|
|
74
|
+
slopbouncer stats
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
```
|
|
78
|
+
Commits checked: 214
|
|
79
|
+
Stopped: 9
|
|
80
|
+
Findings: 31
|
|
81
|
+
Explained in the commit message: 17
|
|
82
|
+
Already explained (repeats not asked about): 6
|
|
83
|
+
Not checked, because Slopbouncer could not run: 0
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
`--json` prints the same counts as JSON.
|
|
87
|
+
|
|
88
|
+
## Getting round it
|
|
89
|
+
|
|
90
|
+
`git commit --no-verify`, or `-n`, skips every hook, this one included. A person can choose to do that. To stop Claude Code doing it, `slopbouncer install claude-code` adds these rules to `.claude/settings.json`, and you can add them by hand:
|
|
91
|
+
|
|
92
|
+
```json
|
|
93
|
+
{
|
|
94
|
+
"permissions": {
|
|
95
|
+
"deny": [
|
|
96
|
+
"Bash(git commit --no-verify:*)",
|
|
97
|
+
"Bash(git commit -n:*)"
|
|
98
|
+
]
|
|
99
|
+
}
|
|
100
|
+
}
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Keep running the plain command in a pipeline as the backstop (see below).
|
|
104
|
+
|
|
105
|
+
## Turn-end hooks for coding agents
|
|
106
|
+
|
|
107
|
+
Slopbouncer can also check an agent's work each time the agent is about to finish, before anything is committed. This is off unless you set it up:
|
|
108
|
+
|
|
109
|
+
```shell
|
|
110
|
+
slopbouncer install claude-code --turn-end
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
| Agent | Command | What it adds |
|
|
114
|
+
|---|---|---|
|
|
115
|
+
| Claude Code | `slopbouncer install claude-code --turn-end` | A `Stop` hook in `.claude/settings.json` |
|
|
116
|
+
| Cursor | `slopbouncer install cursor --turn-end` | A `stop` hook in `.cursor/hooks.json` |
|
|
117
|
+
| Gemini CLI | `slopbouncer install gemini --turn-end` | An `AfterAgent` hook in `.gemini/settings.json` |
|
|
118
|
+
| Codex | `slopbouncer install codex --turn-end` | A `Stop` hook in `.codex/hooks.json` |
|
|
119
|
+
|
|
120
|
+
It adds one hook and keeps everything else in the file. Running it again changes nothing. For Claude Code, `--local` writes to `.claude/settings.local.json`.
|
|
121
|
+
|
|
122
|
+
When the agent is about to finish and has weakened an existing test, it is sent back once and told which tests (the first 20, and how to list the rest), with two ways to answer: put the test back and make the code pass it, or leave the change and tell the person which behaviour changed. The second time it is let finish, so its reasons are on the record for the person to read.
|
|
123
|
+
|
|
124
|
+
The hook checks the work the agent has not committed. If your agent commits as it goes, edit the hook's command to name the commit its work started from, such as `origin/main`, in place of `HEAD`.
|
|
125
|
+
|
|
126
|
+
Codex skips a new hook until you have trusted it: after installing, open `/hooks` in Codex and review it.
|
|
127
|
+
|
|
128
|
+
Claude Code is the one this has been run under. The Cursor, Gemini CLI and Codex hooks are written to those agents' published hook contracts and tested against them, and have not yet been run inside the agents themselves. If one misbehaves, please open an issue.
|
|
129
|
+
|
|
130
|
+
This is what the turn-end hook printed for an agent that had removed an assertion from one test and edited the value another expects:
|
|
131
|
+
|
|
132
|
+
```
|
|
133
|
+
Slopbouncer: this change looks to weaken 2 existing tests.
|
|
134
|
+
|
|
135
|
+
test/checkout_test.rb: Checkout lets a guest pay: assertion_removed: assert_equal 'paid', order.status
|
|
136
|
+
test/refund_test.rb: Refund refunds part of an order: expected_value_changed: 100 → 90
|
|
137
|
+
|
|
138
|
+
For each one, do one of two things before you finish:
|
|
139
|
+
1. Put the test back as it was, and make the code pass it.
|
|
140
|
+
2. Leave the change, and tell the person which behaviour changed and why the test had to change with it.
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
The turn-end hook remembers what it has shown the agent for the work in hand, and sends it back only for findings it has not been shown. When nothing is new it lets the agent finish and says nothing. A finding that is put right and later brought back is shown again, and once HEAD moves, as it does on a commit, the memory starts empty. It keeps this in `slopbouncer-explained.json` in the git directory, never in the working tree. A change to the assertions declared equivalent is shown on every stop. The command run by hand, the pre-push stage and `--json` never use this memory and always report everything.
|
|
144
|
+
|
|
9
145
|
## What it flags
|
|
10
146
|
|
|
11
147
|
| Rule | When |
|
|
12
148
|
|---|---|
|
|
13
149
|
| `assertion_removed` | An assertion is gone from a test that still exists |
|
|
14
150
|
| `expected_value_changed` | The value an assertion expects was edited |
|
|
151
|
+
| `subject_changed` | The thing an assertion checks was edited, so the test now checks something else |
|
|
152
|
+
| `assertion_changed` | An argument was edited in an assertion Slopbouncer cannot read sides into, such as a helper of your own, or the message an assertion shows when it fails |
|
|
15
153
|
| `test_deleted` | A test, or its whole file, was deleted |
|
|
16
154
|
| `skip_added` | A test was marked to be skipped or pending |
|
|
17
155
|
| `no_assertions` | A test the change touched asserts nothing |
|
|
156
|
+
| `equivalents_changed` | The assertions declared equivalent in `.slopbouncer.yml` changed, naming each group added or removed |
|
|
157
|
+
| `helper_changed` | A method defined in the tests changed, and tests assert through it. Reported once, with how many tests depend on it |
|
|
18
158
|
|
|
19
159
|
## What it leaves alone
|
|
20
160
|
|
|
@@ -23,15 +163,21 @@ Status: in development. It reads Ruby tests today: Minitest, as `it` blocks or `
|
|
|
23
163
|
- A test that was reworded, moved to another file, or both, with its checks unchanged.
|
|
24
164
|
- Files that are not tests.
|
|
25
165
|
|
|
26
|
-
##
|
|
166
|
+
## A change that only rewords tests
|
|
27
167
|
|
|
28
|
-
|
|
168
|
+
When a change touches nothing but tests, their directories and `.slopbouncer.yml`, an assertion removed from a test that gained as many as it lost, and has as many in all as before or more, is taken to be reworded. At a terminal and in an agent's hook those are one line:
|
|
29
169
|
|
|
30
|
-
```shell
|
|
31
|
-
gem install slopbouncer
|
|
32
170
|
```
|
|
171
|
+
Test refactor: 23 assertions reworded in 6 files, no other code changed.
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
`--all` lists them, and piped output and `--json` always do. Everything else is still reported in full, among them a deleted test, a test that asserts nothing, a test with fewer assertions than before, `expected_value_changed`, `subject_changed` and `equivalents_changed`. An agent is not sent back for the line alone. As soon as the same change touches a file outside the tests, every finding is reported as usual.
|
|
33
175
|
|
|
34
|
-
|
|
176
|
+
## Running it yourself and in a pipeline
|
|
177
|
+
|
|
178
|
+
Anyone can get round a hook with `--no-verify`, so keep running the plain command where a person will see it. In a pipeline, or before a push, it is the backstop to the hook.
|
|
179
|
+
|
|
180
|
+
Run it from the root of the repository you want checked:
|
|
35
181
|
|
|
36
182
|
```shell
|
|
37
183
|
slopbouncer main..HEAD
|
|
@@ -52,7 +198,7 @@ test/refund_test.rb
|
|
|
52
198
|
Refund refunds part of an order
|
|
53
199
|
expected_value_changed 100 → 90
|
|
54
200
|
|
|
55
|
-
3 tests
|
|
201
|
+
3 tests look weaker in 3 files
|
|
56
202
|
```
|
|
57
203
|
|
|
58
204
|
When its output goes to a file or another program, it prints one plain line for each finding:
|
|
@@ -67,40 +213,14 @@ test/refund_test.rb: Refund refunds part of an order: expected_value_changed: 10
|
|
|
67
213
|
|---|---|
|
|
68
214
|
| `--json` | Prints the findings as JSON |
|
|
69
215
|
| `--strict` | Exits with 1 when anything is flagged |
|
|
216
|
+
| `--all` | At a terminal, lists every finding. Without it, more than 30 findings are shown as a count of each rule in each file |
|
|
70
217
|
| `--color` | Uses the terminal layout, in colour, wherever the output goes. For a script whose own output a person reads |
|
|
71
|
-
| `--agent=NAME` | Runs as that coding agent's hook (see
|
|
218
|
+
| `--agent=NAME` | Runs as that coding agent's turn-end hook (see above) |
|
|
219
|
+
| `--help`, `-h` | Prints how to use it |
|
|
220
|
+
| `--version`, `-v` | Prints the version |
|
|
72
221
|
|
|
73
222
|
Setting `NO_COLOR` keeps the terminal layout and leaves the colour out. Without `--strict` it exits with 0 whatever it finds: it flags for review and does not block. It exits with 2 when it is given nothing to compare.
|
|
74
223
|
|
|
75
|
-
## Feeding it back to a coding agent
|
|
76
|
-
|
|
77
|
-
Slopbouncer is most use when the agent that made the change hears about it. The agent knows why it changed a test, and it can put the test back.
|
|
78
|
-
|
|
79
|
-
One command sets an agent up, from the root of your project:
|
|
80
|
-
|
|
81
|
-
```shell
|
|
82
|
-
slopbouncer install claude-code
|
|
83
|
-
```
|
|
84
|
-
|
|
85
|
-
| Agent | Command | What it adds |
|
|
86
|
-
|---|---|---|
|
|
87
|
-
| Claude Code | `slopbouncer install claude-code` | A `Stop` hook in `.claude/settings.json` |
|
|
88
|
-
| Cursor | `slopbouncer install cursor` | A `stop` hook in `.cursor/hooks.json` |
|
|
89
|
-
| Gemini CLI | `slopbouncer install gemini` | An `AfterAgent` hook in `.gemini/settings.json` |
|
|
90
|
-
| Codex | `slopbouncer install codex` | A `Stop` hook in `.codex/hooks.json` |
|
|
91
|
-
|
|
92
|
-
It adds one hook and keeps everything else in the file. Running it again changes nothing. For Claude Code, `--local` writes to `.claude/settings.local.json`, which keeps the hook to you and out of the settings your team shares.
|
|
93
|
-
|
|
94
|
-
When the agent is about to finish and has weakened an existing test, it is sent back once and told which tests, with two ways to answer: put the test back and make the code pass it, or leave the change and tell the person which behaviour changed. The second time it is let finish, so its reasons are on the record for the person to read.
|
|
95
|
-
|
|
96
|
-
The hook checks the work the agent has not committed. If your agent commits as it goes, edit the hook's command to name the commit its work started from, such as `origin/main`, in place of `HEAD`.
|
|
97
|
-
|
|
98
|
-
Codex skips a new hook until you have trusted it: after installing, open `/hooks` in Codex and review it.
|
|
99
|
-
|
|
100
|
-
Claude Code is the one this has been run under. The Cursor, Gemini CLI and Codex hooks are written to those agents' published hook contracts and tested against them, and have not yet been run inside the agents themselves. If one misbehaves, please open an issue.
|
|
101
|
-
|
|
102
|
-
An agent set on a passing build can still get round a static check, so keep running the plain command where a person will see it.
|
|
103
|
-
|
|
104
224
|
## RSpec
|
|
105
225
|
|
|
106
226
|
An expectation is read whole: its subject, its direction and its matcher. For a plain matcher such as `eq(5)` an edited value shows as `5 → 6`. For a matcher built from several calls, such as `change { total }.by(1)`, the whole matcher is shown before and after.
|
|
@@ -122,12 +242,22 @@ assertions:
|
|
|
122
242
|
|
|
123
243
|
A test that only calls helpers Slopbouncer has not been told about looks like a test with no assertions.
|
|
124
244
|
|
|
245
|
+
When two assertions make the same check, as a framework's assertion and the helper you wrap round it might, declare them equivalent in the same file:
|
|
246
|
+
|
|
247
|
+
```yaml
|
|
248
|
+
equivalents:
|
|
249
|
+
- [assert_selector, assert_shown]
|
|
250
|
+
- [assert_no_selector, refute_shown]
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
An assertion replaced by an equivalent one with the same arguments is then not flagged. With different arguments it is flagged as an edit, as `assertion_changed`. A change to these declarations is flagged as `equivalents_changed` in the same change, so declaring them cannot quietly hide a weaker test.
|
|
254
|
+
|
|
125
255
|
## Its limits
|
|
126
256
|
|
|
127
257
|
- It catches tests that are visibly weaker. A test that looks thorough and checks nothing needs mutation testing, which is slower and runs the code. Use both: Slopbouncer on every change, mutation testing behind it.
|
|
128
258
|
- It cannot tell a stronger assertion from a weaker one. An assertion swapped for another kind is reported as removed.
|
|
129
|
-
- It compares assertions
|
|
130
|
-
- It reads
|
|
259
|
+
- It compares what assertions parse to, so what a formatter changes is ignored: a string's quote style, white space and line breaks, parentheses round an assertion's arguments, a block's braces against `do` and `end`, a trailing comma. Anything that parses differently is still flagged, including the braces taken from round a trailing hash, as in `eq({ a: 1 })` rewritten as `eq(a: 1)`.
|
|
260
|
+
- It reads each test on its own. When a method defined in the tests changes and tests assert through it, as in `assert paid?(order)` or `assert_predicate order, :paid?`, it says so once as `helper_changed`, with how many tests depend on it, and leaves the judging to you: it cannot tell whether the helper checks more or less than it did. It finds those tests by the helper's name, in the helper's own file when that holds tests and in every test file when it does not. A helper reached only through another helper is not counted, and a check that lives elsewhere, such as an RSpec shared example pulled in with `it_behaves_like`, is still not followed, so a test that relies on one can look as if it asserts nothing.
|
|
131
261
|
- It cannot see intent. A test changed on purpose, because the behaviour changed, is flagged the same way as one weakened to get a build to pass.
|
|
132
262
|
|
|
133
263
|
On its first run, over the 706 commits in [Mainline](https://mainline.dev)'s history that touch a test, it flagged 210 of them. In a sample of those, every finding was a real change to a test and every one looked intended. So read a finding as "this test changed in a way worth a look".
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative 'commit_log'
|
|
4
|
+
require_relative 'commit_memory'
|
|
5
|
+
|
|
6
|
+
module Slopbouncer
|
|
7
|
+
# Run as git's commit-msg hook. It reads the tests the commit changes, as
|
|
8
|
+
# staged, and stops a commit that weakens one until the message says why,
|
|
9
|
+
# in a "Test changes:" section with a line for each such test:
|
|
10
|
+
#
|
|
11
|
+
# Test changes:
|
|
12
|
+
# - test/basket_test.rb: Basket adds up: size moved to its own test
|
|
13
|
+
#
|
|
14
|
+
# What a message has explained is remembered, so the same change committed
|
|
15
|
+
# again goes through quietly. When it cannot check a commit at all, it says
|
|
16
|
+
# so and lets the commit through: a broken setup never stops anyone.
|
|
17
|
+
module CommitHook
|
|
18
|
+
SECTION = 'Test changes:'
|
|
19
|
+
|
|
20
|
+
def self.call(message_path)
|
|
21
|
+
message = File.read(message_path)
|
|
22
|
+
findings = Slopbouncer.check_staged.reject(&:reworded)
|
|
23
|
+
remembered, unknown = findings.partition { |finding| CommitMemory.known?(finding) }
|
|
24
|
+
explained, unexplained = unknown.partition { |finding| explains?(message, finding) }
|
|
25
|
+
CommitMemory.remember(explained)
|
|
26
|
+
CommitLog.record('checked' => true, 'findings' => findings.size, 'explained_in_message' => explained.size,
|
|
27
|
+
'already_explained' => remembered.size, 'stopped' => unexplained.any?)
|
|
28
|
+
return 0 if unexplained.empty?
|
|
29
|
+
|
|
30
|
+
warn stopped(unexplained)
|
|
31
|
+
1
|
|
32
|
+
rescue StandardError => e
|
|
33
|
+
warn "Slopbouncer could not run: #{e.message}. This commit was not checked."
|
|
34
|
+
CommitLog.record('checked' => false)
|
|
35
|
+
0
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
# Whether the message's "Test changes:" section names the finding's file
|
|
39
|
+
# and test, with a reason after them. Lines git would strip as comments
|
|
40
|
+
# are not read.
|
|
41
|
+
def self.explains?(message, finding)
|
|
42
|
+
start = "- #{finding.file}: #{finding.test}:"
|
|
43
|
+
section(message).any? { |line| line.start_with?(start) && !line.delete_prefix(start).strip.empty? }
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
def self.section(message)
|
|
47
|
+
lines = message.lines.map(&:rstrip).reject { |line| line.start_with?('#') }
|
|
48
|
+
after = lines.drop_while { |line| line.strip != SECTION }.drop(1)
|
|
49
|
+
after.take_while { |line| line.start_with?('- ') }
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
def self.stopped(findings)
|
|
53
|
+
<<~MESSAGE
|
|
54
|
+
Slopbouncer: this commit looks to weaken #{findings.size} existing #{findings.size == 1 ? 'test' : 'tests'}.
|
|
55
|
+
|
|
56
|
+
#{Slopbouncer.report(findings)}
|
|
57
|
+
|
|
58
|
+
Put each test back, or add this to the commit message, with a reason on each line:
|
|
59
|
+
|
|
60
|
+
#{SECTION}
|
|
61
|
+
#{findings.map { |finding| "- #{finding.file}: #{finding.test}: " }.join("\n")}
|
|
62
|
+
MESSAGE
|
|
63
|
+
end
|
|
64
|
+
private_class_method :explains?, :section, :stopped
|
|
65
|
+
end
|
|
66
|
+
end
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'fileutils'
|
|
4
|
+
require 'json'
|
|
5
|
+
require 'time'
|
|
6
|
+
require_relative 'git'
|
|
7
|
+
|
|
8
|
+
module Slopbouncer
|
|
9
|
+
# One line for each commit the hook ran on, in the git directory, so that
|
|
10
|
+
# `slopbouncer stats` can say how often it stopped one and why the others
|
|
11
|
+
# went through.
|
|
12
|
+
module CommitLog
|
|
13
|
+
FILE = 'log.jsonl'
|
|
14
|
+
COUNTS = %w[checked stopped findings explained_in_message already_explained not_checked].freeze
|
|
15
|
+
|
|
16
|
+
def self.record(entry)
|
|
17
|
+
FileUtils.mkdir_p(File.dirname(path))
|
|
18
|
+
File.open(path, 'a') { |file| file.puts(JSON.generate({ 'at' => Time.now.utc.iso8601 }.merge(entry))) }
|
|
19
|
+
rescue SystemCallError
|
|
20
|
+
nil
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
def self.totals
|
|
24
|
+
entries = File.readlines(path).filter_map { |line| entry(line) }
|
|
25
|
+
checked = entries.select { |entry| entry['checked'] }
|
|
26
|
+
{ 'checked' => checked.size, 'stopped' => checked.count { |entry| entry['stopped'] },
|
|
27
|
+
'findings' => checked.sum { |entry| entry['findings'].to_i },
|
|
28
|
+
'explained_in_message' => checked.sum { |entry| entry['explained_in_message'].to_i },
|
|
29
|
+
'already_explained' => checked.sum { |entry| entry['already_explained'].to_i },
|
|
30
|
+
'not_checked' => entries.count { |entry| !entry['checked'] } }
|
|
31
|
+
rescue Errno::ENOENT
|
|
32
|
+
COUNTS.to_h { |count| [count, 0] }
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
def self.entry(line)
|
|
36
|
+
JSON.parse(line)
|
|
37
|
+
rescue JSON::ParserError
|
|
38
|
+
nil
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
def self.path
|
|
42
|
+
File.join(Git.directory, 'slopbouncer', FILE)
|
|
43
|
+
end
|
|
44
|
+
private_class_method :entry, :path
|
|
45
|
+
end
|
|
46
|
+
end
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'digest'
|
|
4
|
+
require 'fileutils'
|
|
5
|
+
require 'json'
|
|
6
|
+
require_relative 'git'
|
|
7
|
+
|
|
8
|
+
module Slopbouncer
|
|
9
|
+
# The findings a commit message has explained, so the same change committed
|
|
10
|
+
# again, after a reset or an amend, goes through quietly. It is kept in the
|
|
11
|
+
# git directory, out of the working tree, and lasts across commits. A
|
|
12
|
+
# change to the assertions declared equivalent quiets Slopbouncer, so
|
|
13
|
+
# nothing quiets that change in turn.
|
|
14
|
+
module CommitMemory
|
|
15
|
+
FILE = 'explained.json'
|
|
16
|
+
ALWAYS = %w[equivalents_changed].freeze
|
|
17
|
+
|
|
18
|
+
def self.known?(finding)
|
|
19
|
+
!ALWAYS.include?(finding.rule) && recall.include?(fingerprint(finding))
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def self.remember(findings)
|
|
23
|
+
return if findings.empty?
|
|
24
|
+
|
|
25
|
+
FileUtils.mkdir_p(File.dirname(path))
|
|
26
|
+
File.write(path, JSON.generate('fingerprints' => (recall + findings.map { |finding| fingerprint(finding) }).uniq))
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
# The same finding: the same file, test and kind, changing the same thing
|
|
30
|
+
# into the same thing, however it was spaced.
|
|
31
|
+
def self.fingerprint(finding)
|
|
32
|
+
said = [finding.file, finding.test, finding.rule, finding.before || finding.detail, finding.after]
|
|
33
|
+
Digest::SHA256.hexdigest(JSON.generate(said.map { |part| part.to_s.gsub(/\s+/, ' ').strip }))
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
def self.recall
|
|
37
|
+
JSON.parse(File.read(path)).fetch('fingerprints', [])
|
|
38
|
+
rescue Errno::ENOENT, JSON::ParserError
|
|
39
|
+
[]
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
def self.path
|
|
43
|
+
File.join(Git.directory, 'slopbouncer', FILE)
|
|
44
|
+
end
|
|
45
|
+
private_class_method :fingerprint, :recall, :path
|
|
46
|
+
end
|
|
47
|
+
end
|