scully 0.0.1 → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +33 -0
- data/README.md +272 -9
- data/Rakefile +5 -1
- data/exe/scully +6 -0
- data/lib/scully/analyzer.rb +90 -0
- data/lib/scully/cli.rb +126 -0
- data/lib/scully/discovery.rb +53 -0
- data/lib/scully/frameworks/detector.rb +86 -0
- data/lib/scully/frameworks/helpers.rb +215 -0
- data/lib/scully/frameworks/minitest.rb +305 -0
- data/lib/scully/frameworks/rspec.rb +386 -0
- data/lib/scully/model.rb +48 -0
- data/lib/scully/render/report.rb +79 -0
- data/lib/scully/render/snippet.rb +92 -0
- data/lib/scully/render/terminal.rb +97 -0
- data/lib/scully/render/why.rb +146 -0
- data/lib/scully/rule.rb +85 -0
- data/lib/scully/rules/no_assertion.rb +51 -0
- data/lib/scully/rules/parse.rb +34 -0
- data/lib/scully/rules/tautology.rb +101 -0
- data/lib/scully/source_file.rb +53 -0
- data/lib/scully/version.rb +1 -1
- data/lib/scully.rb +18 -1
- metadata +39 -6
- data/sig/test_scully.rbs +0 -3
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: c42579c421ad026eb06b80cc2e42d679a39d3a5bec2789ef73e26857888b2e69
|
|
4
|
+
data.tar.gz: 8cf9890e7b6f5b3c54cf3a77f22124624f7523f86b0f73bc678ef2b3bf382ddc
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 8dd3ac44b0c9e686943493444f2ad1273a4b5d3cee9da46029cdd6456fc0c8cf4f44cf577d0e0234d7e5014d6a0ea9c237fdb5d3076ae0c91b9aad341cb5013e
|
|
7
|
+
data.tar.gz: 461bbbab137a7fcca5837c9ed250e6fe28d36cf706beeb91d2be87e684500d9ee66ff63f6afa98fd4910b788fd90c8b805efb00caf4c9f51345cf51b61ec2936
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.1.0] - 2026-10-14
|
|
11
|
+
|
|
12
|
+
First usable release: a static checker that reports RSpec and Minitest tests that cannot be
|
|
13
|
+
trusted as evidence of behaviour. It reads tests with Prism and never runs them.
|
|
14
|
+
|
|
15
|
+
### Added
|
|
16
|
+
|
|
17
|
+
- `scully check [PATH ...]` (or bare `scully`) finds `*_spec.rb` and `*_test.rb` files, works
|
|
18
|
+
out whether each is RSpec or Minitest, and prints a report grouped into errors, warnings and
|
|
19
|
+
skipped files, with source snippets, a summary line and one `scully why` command per finding.
|
|
20
|
+
- Rule `no_assertion` (warning): an active test whose body makes no assertion. Counts RSpec
|
|
21
|
+
expectations, Minitest assertions and expectations, mocha and `Minitest::Mock` message
|
|
22
|
+
expectations, Minitest-style `assert`/`refute` calls in RSpec examples (including Capybara's
|
|
23
|
+
`page.assert_selector`), `assert_*`/`expect_*` helpers, and helpers defined in the same file
|
|
24
|
+
that assert.
|
|
25
|
+
- Rule `tautology` (error): an equality assertion that compares a stubbed method with the value
|
|
26
|
+
the test stubbed it to return, so it can never fail. The stub is shown next to the assertion.
|
|
27
|
+
- Syntax errors are reported as `parse` errors. Findings from a file that could not be read
|
|
28
|
+
completely are marked inconclusive, and such a file makes the run exit 1, so a broken suite
|
|
29
|
+
never looks clean.
|
|
30
|
+
- `scully why PATH[:LINE]` explains a finding: why it matters, what scully saw, how to fix it,
|
|
31
|
+
how certain it is, and the matching RuboCop cop, if any. Without a line it lists the file's
|
|
32
|
+
findings.
|
|
33
|
+
- `--color` and `--hyperlinks` options (`auto`, `always`, `never`), honouring `NO_COLOR` and
|
|
34
|
+
`FORCE_HYPERLINK`. Colour and links never change the text.
|
|
35
|
+
- Exit status 0 for a clean run, 1 for findings or incomplete files, 2 when scully cannot run.
|
|
36
|
+
- Output is deterministic: the same files give byte-identical output whatever the argument
|
|
37
|
+
order, locale or checkout location.
|
|
38
|
+
|
|
10
39
|
## [0.0.1] - 2026-10-12
|
|
11
40
|
|
|
12
41
|
- Initial release
|
|
42
|
+
|
|
43
|
+
[Unreleased]: https://github.com/theomelo/scully/compare/v0.1.0...HEAD
|
|
44
|
+
[0.1.0]: https://github.com/theomelo/scully/compare/v0.0.1...v0.1.0
|
|
45
|
+
[0.0.1]: https://github.com/theomelo/scully/releases/tag/v0.0.1
|
data/README.md
CHANGED
|
@@ -1,15 +1,20 @@
|
|
|
1
1
|
# Scully
|
|
2
2
|
|
|
3
|
-
"The truth is out there"
|
|
3
|
+
"The truth is out there" (Use tests to prove it!)
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
Scully reads a Ruby test suite and reports tests that cannot be trusted as evidence of
|
|
6
|
+
behaviour: tests that check nothing, and tests that can never fail. It works on RSpec and
|
|
7
|
+
Minitest files, by static analysis only: it parses your tests with Prism and never runs them.
|
|
8
|
+
The same input always gives the same output, byte for byte.
|
|
6
9
|
|
|
7
10
|
## Installation
|
|
8
11
|
|
|
12
|
+
Scully needs Ruby 4.0.6 or newer.
|
|
13
|
+
|
|
9
14
|
Install the gem and add to the application's Gemfile by executing:
|
|
10
15
|
|
|
11
16
|
```bash
|
|
12
|
-
bundle add scully
|
|
17
|
+
bundle add scully --group=development,test
|
|
13
18
|
```
|
|
14
19
|
|
|
15
20
|
If bundler is not being used to manage dependencies, install the gem by executing:
|
|
@@ -20,22 +25,280 @@ gem install scully
|
|
|
20
25
|
|
|
21
26
|
## Usage
|
|
22
27
|
|
|
23
|
-
|
|
28
|
+
```text
|
|
29
|
+
scully check [options] [PATH ...]
|
|
30
|
+
scully [options] [PATH ...] # same as check
|
|
31
|
+
scully why [options] PATH[:LINE]
|
|
32
|
+
scully --version
|
|
33
|
+
scully --help
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
### `scully check`
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
scully check # the current directory
|
|
40
|
+
scully check spec test # only these directories
|
|
41
|
+
scully check spec/models/order_spec.rb
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
- With no PATH, scully checks `.`. A bare `scully` is the same as `scully check`, and
|
|
45
|
+
`scully spec` is the same as `scully check spec`. To check a directory named `check` or
|
|
46
|
+
`why`, name the command first: `scully check why`.
|
|
47
|
+
- Directories are searched recursively for `*_spec.rb` and `*_test.rb` files, skipping
|
|
48
|
+
`vendor`, `node_modules`, `tmp`, `.git` and any other directory whose name starts with `.`.
|
|
49
|
+
The skip list applies only below the directory you pass, so `scully check tmp` still
|
|
50
|
+
searches `tmp`. A file passed explicitly is checked whatever its name.
|
|
51
|
+
- Each file is classified as RSpec or Minitest by its content (`RSpec.describe`,
|
|
52
|
+
`require "spec_helper"`, a class inheriting from `Minitest::Test` or
|
|
53
|
+
`ActiveSupport::TestCase`, `require "test_helper"`, ...), then by whether it sits under
|
|
54
|
+
`spec/` or `test/`. Files that match neither are listed as skipped.
|
|
55
|
+
- Paths in the report are relative to the current directory. Argument order does not change
|
|
56
|
+
the output, and a file reached through several paths is checked once.
|
|
57
|
+
|
|
58
|
+
### The report
|
|
59
|
+
|
|
60
|
+
Given these two files:
|
|
61
|
+
|
|
62
|
+
```ruby
|
|
63
|
+
# spec/tautology_spec.rb
|
|
64
|
+
RSpec.describe Pricing do
|
|
65
|
+
let(:tax) { double(rate: 0.2) }
|
|
66
|
+
|
|
67
|
+
before { allow(gateway).to receive(:fee).and_return(3) }
|
|
68
|
+
|
|
69
|
+
it "reads the tax rate" do
|
|
70
|
+
expect(tax.rate).to eq(0.2)
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
it "reads the fee" do
|
|
74
|
+
expect(gateway.fee).to eq(3)
|
|
75
|
+
end
|
|
76
|
+
end
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
```ruby
|
|
80
|
+
# spec/no_assertion_spec.rb
|
|
81
|
+
RSpec.describe Order do
|
|
82
|
+
it "sums line items" do
|
|
83
|
+
Order.new.total
|
|
84
|
+
end
|
|
85
|
+
end
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
`scully check spec` prints:
|
|
89
|
+
|
|
90
|
+
```text
|
|
91
|
+
errors (2)
|
|
92
|
+
|
|
93
|
+
error[tautology]: assertion checks a value the test stubbed
|
|
94
|
+
--> spec/tautology_spec.rb:7:5
|
|
95
|
+
|
|
|
96
|
+
2 | let(:tax) { double(rate: 0.2) }
|
|
97
|
+
| ----------------- stubbed in setup to return `0.2`
|
|
98
|
+
...
|
|
99
|
+
7 | expect(tax.rate).to eq(0.2)
|
|
100
|
+
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^ compares the stub with its own return value
|
|
101
|
+
|
|
102
|
+
error[tautology]: assertion checks a value the test stubbed
|
|
103
|
+
--> spec/tautology_spec.rb:11:5
|
|
104
|
+
|
|
|
105
|
+
4 | before { allow(gateway).to receive(:fee).and_return(3) }
|
|
106
|
+
| --------------------------------------------- stubbed in setup to return `3`
|
|
107
|
+
...
|
|
108
|
+
11 | expect(gateway.fee).to eq(3)
|
|
109
|
+
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ compares the stub with its own return value
|
|
110
|
+
|
|
111
|
+
warnings (1)
|
|
112
|
+
|
|
113
|
+
warning[no_assertion]: test makes no assertion
|
|
114
|
+
--> spec/no_assertion_spec.rb:2:3
|
|
115
|
+
|
|
|
116
|
+
2 | it "sums line items" do
|
|
117
|
+
| ^^^^^^^^^^^^^^^^^^^^ this test runs code but never checks a result
|
|
118
|
+
|
|
119
|
+
summary: 2 files (0 incomplete, 0 skipped), 3 tests, 2 errors, 1 warnings
|
|
120
|
+
evidence: static analysis only; coverage and mutation testing not run
|
|
121
|
+
|
|
122
|
+
Use `scully why <location>` for more info about each finding:
|
|
123
|
+
|
|
124
|
+
scully why spec/tautology_spec.rb:7 # tautology
|
|
125
|
+
scully why spec/tautology_spec.rb:11 # tautology
|
|
126
|
+
scully why spec/no_assertion_spec.rb:2 # no_assertion
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
and exits 1.
|
|
130
|
+
|
|
131
|
+
- Findings are grouped as `errors`, then `warnings`, then `skipped` files, and sorted by path,
|
|
132
|
+
line, column and rule. Empty groups are left out.
|
|
133
|
+
- Each finding shows the rule, the message, the `path:line:column` location, and the source
|
|
134
|
+
lines involved. `^` marks the main location; `-` marks related code, such as the stub a
|
|
135
|
+
tautology compares against.
|
|
136
|
+
- A file with syntax errors is analyzed only in part. Its findings are marked
|
|
137
|
+
`(incomplete file: inconclusive)`, and the summary counts it as incomplete and lists the tests
|
|
138
|
+
that could not be analyzed (`5 tests (1 not analyzed)`).
|
|
139
|
+
- The summary line always has the same shape, so it is easy to read from scripts. The
|
|
140
|
+
`evidence:` line is a reminder that scully reads code; it does not measure coverage or run
|
|
141
|
+
mutation testing.
|
|
142
|
+
- The report ends with one `scully why` command per finding.
|
|
143
|
+
|
|
144
|
+
### `scully why`
|
|
145
|
+
|
|
146
|
+
`scully why PATH:LINE` re-checks one file and explains the finding(s) on that line: why it
|
|
147
|
+
matters, what scully saw, how to fix it, how sure it is, and which RuboCop cop covers similar
|
|
148
|
+
ground.
|
|
149
|
+
|
|
150
|
+
```text
|
|
151
|
+
$ scully why spec/tautology_spec.rb:7
|
|
152
|
+
error[tautology]: assertion checks a value the test stubbed
|
|
153
|
+
--> spec/tautology_spec.rb:7:5
|
|
154
|
+
|
|
|
155
|
+
2 | let(:tax) { double(rate: 0.2) }
|
|
156
|
+
| ----------------- stubbed in setup to return `0.2`
|
|
157
|
+
...
|
|
158
|
+
7 | expect(tax.rate).to eq(0.2)
|
|
159
|
+
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^ compares the stub with its own return value
|
|
160
|
+
|
|
161
|
+
because:
|
|
162
|
+
The assertion checks a value the test itself stubbed, so it can never fail and says
|
|
163
|
+
nothing about the code under test.
|
|
164
|
+
|
|
165
|
+
what happened:
|
|
166
|
+
`tax.rate` is stubbed to return `0.2` (line 2), and the expectation on line 7 compares
|
|
167
|
+
`tax.rate` with `0.2`. No code under test runs between the stub and the assertion.
|
|
168
|
+
|
|
169
|
+
help:
|
|
170
|
+
Assert on the result of the code that uses the stubbed object, not on the stub itself. If
|
|
171
|
+
the test only documents the stub, delete it.
|
|
172
|
+
|
|
173
|
+
evidence: static analysis (complete file)
|
|
174
|
+
see also: no matching cop in rubocop-rspec or rubocop-minitest
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
`scully why PATH` without a line lists every finding in the file, with the `scully why`
|
|
178
|
+
command for each. When there is no finding on the given line, scully says so on stderr, lists
|
|
179
|
+
the findings the file does have, and exits 1.
|
|
180
|
+
|
|
181
|
+
## Rules
|
|
182
|
+
|
|
183
|
+
### `no_assertion` (warning)
|
|
184
|
+
|
|
185
|
+
A test that makes no assertion passes whatever the code does, so it gives coverage without
|
|
186
|
+
evidence. Scully reports active tests (not pending or skipped) whose body contains no
|
|
187
|
+
assertion. It counts:
|
|
188
|
+
|
|
189
|
+
- RSpec expectations: `expect(...).to`, `expect { ... }.to`, `is_expected`, `should`, and
|
|
190
|
+
message expectations such as `expect(mailer).to receive(:deliver)`.
|
|
191
|
+
- Minitest assertions and expectations: `assert*`, `refute*`, `flunk`, `must_*`, `wont_*`,
|
|
192
|
+
mocha `expects`, and `Minitest::Mock#expect` / `#verify`.
|
|
193
|
+
- Minitest-style `assert*` and `refute*` calls in RSpec examples, with or without a receiver
|
|
194
|
+
(rspec-rails mixes them in, and Capybara adds `page.assert_selector` and friends).
|
|
195
|
+
- Helpers whose name starts with `assert_` or `expect_`, and helpers defined in the same file
|
|
196
|
+
whose body asserts, directly or through other helpers in that file.
|
|
197
|
+
|
|
198
|
+
Assertions only in `after` or `teardown` do not count. A helper defined in another file cannot
|
|
199
|
+
be followed yet, so a test that asserts only through one is reported; `scully why` lists the
|
|
200
|
+
calls it could not follow.
|
|
201
|
+
|
|
202
|
+
Companion cops:
|
|
203
|
+
|
|
204
|
+
- `RSpec/NoExpectationExample` in
|
|
205
|
+
[rubocop-rspec](https://docs.rubocop.org/rubocop-rspec/latest/cops_rspec.html#rspecnoexpectationexample)
|
|
206
|
+
- `Minitest/NoAssertions` in
|
|
207
|
+
[rubocop-minitest](https://docs.rubocop.org/rubocop-minitest/latest/cops_minitest.html#minitestnoassertions),
|
|
208
|
+
disabled by default
|
|
209
|
+
|
|
210
|
+
### `tautology` (error)
|
|
211
|
+
|
|
212
|
+
An assertion that checks a value the test itself stubbed can never fail and says nothing about
|
|
213
|
+
the code under test. Scully reports an equality assertion whose subject is a stubbed method
|
|
214
|
+
called on the stubbed receiver, when the expected value matches the stubbed return value:
|
|
215
|
+
|
|
216
|
+
```ruby
|
|
217
|
+
let(:tax) { double(rate: 0.2) }
|
|
218
|
+
|
|
219
|
+
it "reads the tax rate" do
|
|
220
|
+
expect(tax.rate).to eq(0.2) # checks the double, not the code
|
|
221
|
+
end
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
Recognised stubs: `allow(...).to receive(...).and_return(...)`, `receive_messages`, doubles
|
|
225
|
+
created with attributes (`double(rate: 0.2)`, `instance_double`, ...), Minitest's
|
|
226
|
+
`obj.stub(:m, value) do ... end` (inside its block only) and mocha's
|
|
227
|
+
`obj.stubs(:m).returns(value)`, in the test body or in `let`, `before` or `setup`. Message
|
|
228
|
+
expectations (`expect(...).to receive(...).and_return(...)`, `Minitest::Mock#expect`) can fail
|
|
229
|
+
on their own and are not treated as stubs.
|
|
230
|
+
|
|
231
|
+
Companion cops: none; no cop in rubocop-rspec or rubocop-minitest matches this rule.
|
|
232
|
+
|
|
233
|
+
### `parse` (error)
|
|
234
|
+
|
|
235
|
+
Scully can only check code it can read. Each line with a syntax error is reported once, with
|
|
236
|
+
the first parser message and a count of the others. `ruby -c PATH` shows Ruby's own message.
|
|
237
|
+
|
|
238
|
+
## Options
|
|
239
|
+
|
|
240
|
+
| Option | Values | Default | Effect |
|
|
241
|
+
|---|---|---|---|
|
|
242
|
+
| `--color=WHEN` | `auto`, `always`, `never` | `auto` | Colour the severity words and markers. `auto` colours only when stdout is a terminal and `NO_COLOR` is unset or empty. |
|
|
243
|
+
| `--hyperlinks=WHEN` | `auto`, `always`, `never` | `auto` | In `scully why`, make cop names clickable links instead of printing their URLs on a separate line. |
|
|
244
|
+
| `--version` | | | Print `scully <version>` and exit. |
|
|
245
|
+
| `--help` | | | Print usage and exit. |
|
|
246
|
+
|
|
247
|
+
Colour and links never change the text: removing the escape sequences gives exactly the
|
|
248
|
+
output you get through a pipe.
|
|
249
|
+
|
|
250
|
+
### Environment variables
|
|
251
|
+
|
|
252
|
+
| Variable | Effect |
|
|
253
|
+
|---|---|
|
|
254
|
+
| `NO_COLOR` | A non-empty value turns colour off in `auto` mode ([no-color.org](https://no-color.org)). |
|
|
255
|
+
| `FORCE_HYPERLINK` | `0` turns links off, any other non-empty value turns them on, in `auto` mode. |
|
|
256
|
+
| `CI`, `TEAMCITY_VERSION` | Turn links off in `auto` mode. |
|
|
257
|
+
| `TERM_PROGRAM`, `TERM_PROGRAM_VERSION`, `VTE_VERSION`, `WT_SESSION`, `TERM` | Used in `auto` mode to detect terminals that render links (for example iTerm2 3.1+, WezTerm, VS Code 1.72+, Ghostty, Windows Terminal, kitty, Alacritty, VTE 0.50.1+). |
|
|
258
|
+
|
|
259
|
+
Without `FORCE_HYPERLINK`, links are on only when stdout is a terminal. No other environment
|
|
260
|
+
variable or locale setting affects the output.
|
|
261
|
+
|
|
262
|
+
## Exit status
|
|
263
|
+
|
|
264
|
+
| Status | `scully check` | `scully why` |
|
|
265
|
+
|---|---|---|
|
|
266
|
+
| 0 | Every file was read completely and there are no findings. | The finding(s) or the file index were printed. |
|
|
267
|
+
| 1 | Any finding, or any file that could not be read completely. | No finding at that location. |
|
|
268
|
+
| 2 | Scully could not run: a path does not exist, or an option or argument is wrong. | The path does not exist, or the arguments are wrong. |
|
|
269
|
+
|
|
270
|
+
A run is clean only when it exits 0: a file with syntax errors or invalid UTF-8 makes the run
|
|
271
|
+
exit 1 even without findings. Errors that exit 2 print `scully: <message>` to stderr.
|
|
24
272
|
|
|
25
273
|
## Development
|
|
26
274
|
|
|
27
|
-
After checking out the repo, run `bin/setup` to install dependencies. Then, run
|
|
275
|
+
After checking out the repo, run `bin/setup` to install dependencies. Then, run
|
|
276
|
+
`bundle exec rake` to run the tests and RuboCop. You can also run `bin/console` for an
|
|
277
|
+
interactive prompt that will allow you to experiment.
|
|
278
|
+
|
|
279
|
+
Fixtures live in `test/fixtures/` and the expected reports in `test/golden/`. After an
|
|
280
|
+
intended output change, regenerate the goldens with
|
|
281
|
+
`SCULLY_UPDATE_GOLDEN=1 bundle exec rake test` and review the diff.
|
|
28
282
|
|
|
29
|
-
To install this gem onto your local machine, run `bundle exec rake install`. To release a new
|
|
283
|
+
To install this gem onto your local machine, run `bundle exec rake install`. To release a new
|
|
284
|
+
version, update the version number in `version.rb`, and then run `bundle exec rake release`,
|
|
285
|
+
which will create a git tag for the version, push git commits and the created tag, and push the
|
|
286
|
+
`.gem` file to [rubygems.org](https://rubygems.org).
|
|
30
287
|
|
|
31
288
|
## Contributing
|
|
32
289
|
|
|
33
|
-
Bug reports and pull requests are welcome on GitHub at https://github.com/theomelo/scully. This
|
|
290
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/theomelo/scully. This
|
|
291
|
+
project is intended to be a safe, welcoming space for collaboration, and contributors are
|
|
292
|
+
expected to adhere to the
|
|
293
|
+
[code of conduct](https://github.com/theomelo/scully/blob/main/CODE_OF_CONDUCT.md).
|
|
34
294
|
|
|
35
295
|
## License
|
|
36
296
|
|
|
37
|
-
The gem is available as open source under the terms of the
|
|
297
|
+
The gem is available as open source under the terms of the
|
|
298
|
+
[MIT License](https://opensource.org/licenses/MIT).
|
|
38
299
|
|
|
39
300
|
## Code of Conduct
|
|
40
301
|
|
|
41
|
-
Everyone interacting in the Scully project's codebases, issue trackers, chat rooms and mailing
|
|
302
|
+
Everyone interacting in the Scully project's codebases, issue trackers, chat rooms and mailing
|
|
303
|
+
lists is expected to follow the
|
|
304
|
+
[code of conduct](https://github.com/theomelo/scully/blob/main/CODE_OF_CONDUCT.md).
|
data/Rakefile
CHANGED
|
@@ -3,7 +3,11 @@
|
|
|
3
3
|
require 'bundler/gem_tasks'
|
|
4
4
|
require 'minitest/test_task'
|
|
5
5
|
|
|
6
|
-
|
|
6
|
+
# Only test_*.rb files are the gem's own tests; *_test.rb files under test/fixtures are sample
|
|
7
|
+
# input for scully and must never run.
|
|
8
|
+
Minitest::TestTask.create do |task|
|
|
9
|
+
task.test_globs = ['test/**/test_*.rb']
|
|
10
|
+
end
|
|
7
11
|
|
|
8
12
|
require 'rubocop/rake_task'
|
|
9
13
|
|
data/exe/scully
ADDED
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Scully
|
|
4
|
+
# Runs discovery, parsing, framework detection, normalization and rules, and collects the
|
|
5
|
+
# results into a RunReport.
|
|
6
|
+
class Analyzer
|
|
7
|
+
# Framework normalizers turn a SourceFile into TestCases.
|
|
8
|
+
NORMALIZERS = { rspec: Frameworks::RSpec, minitest: Frameworks::Minitest }.freeze
|
|
9
|
+
|
|
10
|
+
# SourceFiles read during the last run, by path.
|
|
11
|
+
attr_reader :sources
|
|
12
|
+
|
|
13
|
+
# Built-in rule run on every file report.
|
|
14
|
+
PARSE = Rules::Parse.new
|
|
15
|
+
|
|
16
|
+
def initialize(rules:, normalizers: NORMALIZERS)
|
|
17
|
+
@rules = rules
|
|
18
|
+
@normalizers = normalizers
|
|
19
|
+
@sources = {}
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def run(paths, cwd: Dir.pwd)
|
|
23
|
+
files = Discovery.call(paths, cwd:).map { analyze_file(it, cwd) }
|
|
24
|
+
findings = files.flat_map { check(it) }.sort_by(&:sort_key)
|
|
25
|
+
counts = count(files, findings)
|
|
26
|
+
exit_status = findings.empty? && counts[:incomplete].zero? ? 0 : 1
|
|
27
|
+
RunReport.new(files:, findings:, counts:, exit_status:)
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
private
|
|
31
|
+
|
|
32
|
+
def analyze_file(path, cwd)
|
|
33
|
+
source = @sources[path] = SourceFile.read(path, base: cwd)
|
|
34
|
+
framework = Frameworks::Detector.detect(path, source.result&.value)
|
|
35
|
+
# A broken file is incomplete even without a framework: its errors may hide the tests.
|
|
36
|
+
return skipped(path) if framework.nil? && !source.incomplete?
|
|
37
|
+
|
|
38
|
+
FileReport.new(path:, framework:, status: source.incomplete? ? :incomplete : :complete, skip_reason: nil,
|
|
39
|
+
parse_errors: source.parse_errors, test_cases: test_cases(source, framework))
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
def test_cases(source, framework)
|
|
43
|
+
return [] unless source.result && framework
|
|
44
|
+
|
|
45
|
+
mark_not_analyzed(@normalizers.fetch(framework).call(source), source.parse_errors)
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
def skipped(path)
|
|
49
|
+
FileReport.new(path:, framework: nil, status: :skipped, skip_reason: Frameworks::Detector::SKIP_REASON,
|
|
50
|
+
parse_errors: [], test_cases: [])
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
# A test whose extent contains a parse error is partly invented by the parser, so no rule
|
|
54
|
+
# runs on it.
|
|
55
|
+
def mark_not_analyzed(test_cases, parse_errors)
|
|
56
|
+
test_cases.map do |test_case|
|
|
57
|
+
next test_case unless parse_errors.any? { contains?(test_case.extent, it.span) }
|
|
58
|
+
|
|
59
|
+
test_case.with(status: :not_analyzed, assertions: [], doubles: [], helper_calls: [])
|
|
60
|
+
end
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
# Span ends are exclusive: an error that starts right where a test ends is outside it.
|
|
64
|
+
def contains?(outer, inner)
|
|
65
|
+
start = [inner.start_line, inner.start_column]
|
|
66
|
+
(start <=> [outer.start_line, outer.start_column]) >= 0 &&
|
|
67
|
+
(start <=> [outer.end_line, outer.end_column]).negative?
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
# Parse findings are certain; what the rules found in an incomplete file may not be.
|
|
71
|
+
def check(file)
|
|
72
|
+
findings = file.test_cases.select { it.status == :active }.flat_map do |test_case|
|
|
73
|
+
@rules.flat_map { it.check(test_case) }
|
|
74
|
+
end
|
|
75
|
+
findings = findings.map { it.with(evidence: :inconclusive) } if file.status == :incomplete
|
|
76
|
+
PARSE.findings(file) + findings
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
def count(files, findings)
|
|
80
|
+
tests = files.flat_map(&:test_cases)
|
|
81
|
+
{ files: files.size,
|
|
82
|
+
incomplete: files.count { it.status == :incomplete },
|
|
83
|
+
skipped: files.count { it.status == :skipped },
|
|
84
|
+
tests: tests.size,
|
|
85
|
+
not_analyzed: tests.count { it.status == :not_analyzed },
|
|
86
|
+
errors: findings.count { it.severity == :error },
|
|
87
|
+
warnings: findings.count { it.severity == :warning } }
|
|
88
|
+
end
|
|
89
|
+
end
|
|
90
|
+
end
|
data/lib/scully/cli.rb
ADDED
|
@@ -0,0 +1,126 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'optparse'
|
|
4
|
+
|
|
5
|
+
module Scully
|
|
6
|
+
# Command line entry point: `scully check [PATH ...]` (the default) and `scully why`.
|
|
7
|
+
class CLI
|
|
8
|
+
COMMANDS = %w[check why].freeze
|
|
9
|
+
MODES = %w[auto always never].freeze
|
|
10
|
+
USAGE_HINT = "Run 'scully --help' for usage."
|
|
11
|
+
|
|
12
|
+
# Rules run by default.
|
|
13
|
+
RULES = [Rules::NoAssertion, Rules::Tautology].freeze
|
|
14
|
+
|
|
15
|
+
# Runs the command and returns the process exit status.
|
|
16
|
+
# +env+ is read only to decide colour and hyperlinks.
|
|
17
|
+
def self.start(argv, out: $stdout, err: $stderr, cwd: Dir.pwd, env: ENV)
|
|
18
|
+
new(out:, err:, cwd:, env:).start(argv.map { argument(it) })
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
# Arguments arrive in the locale's encoding: US-ASCII under LANG=C even when they hold UTF-8
|
|
22
|
+
# bytes, which makes any regexp match raise. They are read as UTF-8 when valid, else as bytes.
|
|
23
|
+
def self.argument(text)
|
|
24
|
+
utf8 = text.b.force_encoding(Encoding::UTF_8)
|
|
25
|
+
utf8.valid_encoding? ? utf8 : text.b
|
|
26
|
+
end
|
|
27
|
+
private_class_method :argument
|
|
28
|
+
|
|
29
|
+
def initialize(out:, err:, cwd:, env:)
|
|
30
|
+
@out = out
|
|
31
|
+
@env = env
|
|
32
|
+
@err = err
|
|
33
|
+
@cwd = cwd
|
|
34
|
+
@rules = RULES.map(&:new)
|
|
35
|
+
@options = { color: 'auto', hyperlinks: 'auto' }
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
# The first non-option argument selects `check` or `why`; anything else runs `check`.
|
|
39
|
+
def start(argv)
|
|
40
|
+
catch(:done) do
|
|
41
|
+
arguments = parser.parse!(argv)
|
|
42
|
+
command = COMMANDS.include?(arguments.first) ? arguments.shift : 'check'
|
|
43
|
+
return command == 'why' ? why(arguments) : check(arguments)
|
|
44
|
+
end
|
|
45
|
+
0
|
|
46
|
+
rescue OptionParser::ParseError, UsageError => e
|
|
47
|
+
fail_with(e.message, hint: true)
|
|
48
|
+
rescue Error => e
|
|
49
|
+
fail_with(e.message)
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
# A bad argument: reported with the usage hint.
|
|
53
|
+
class UsageError < Error; end
|
|
54
|
+
|
|
55
|
+
private
|
|
56
|
+
|
|
57
|
+
def check(paths)
|
|
58
|
+
analyzer = Analyzer.new(rules: @rules)
|
|
59
|
+
run = analyzer.run(paths, cwd: @cwd)
|
|
60
|
+
messages = rules_by_id.transform_values { it.class.message }
|
|
61
|
+
@out.print Render::Report.render(run, messages:, sources: analyzer.sources, terminal:)
|
|
62
|
+
run.exit_status
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
# Re-analyzes one file; prints the findings on LINE, or the file's index without a line.
|
|
66
|
+
# A skipped file prints its skip line either way.
|
|
67
|
+
def why(arguments)
|
|
68
|
+
path, line = why_target(arguments)
|
|
69
|
+
view = why_view(path)
|
|
70
|
+
text = view.skipped || (line ? view.location(line) : view.index)
|
|
71
|
+
text ? @out.print(text) : @err.print(view.missing(line))
|
|
72
|
+
text ? 0 : 1
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
def why_view(path)
|
|
76
|
+
analyzer = Analyzer.new(rules: @rules)
|
|
77
|
+
run = analyzer.run([path], cwd: @cwd)
|
|
78
|
+
file = run.files.first
|
|
79
|
+
Render::Why.new(file, findings: run.findings, rules: rules_by_id, source: analyzer.sources.fetch(file.path),
|
|
80
|
+
terminal:)
|
|
81
|
+
end
|
|
82
|
+
|
|
83
|
+
# PATH[:LINE] from the only argument, matched as bytes so a name that is not valid UTF-8
|
|
84
|
+
# still parses.
|
|
85
|
+
def why_target(arguments)
|
|
86
|
+
raise UsageError, 'why needs one file' if arguments.empty?
|
|
87
|
+
|
|
88
|
+
match = arguments.first.b.match(/\A(.+):(\d+)\z/m)
|
|
89
|
+
path, line = match ? [match[1], match[2].to_i] : [arguments.first.b, nil]
|
|
90
|
+
offending = File.directory?(File.expand_path(path, @cwd.b)) ? arguments.first : arguments[1]
|
|
91
|
+
raise UsageError, "why needs one file: #{offending}" if offending
|
|
92
|
+
|
|
93
|
+
[path, line]
|
|
94
|
+
end
|
|
95
|
+
|
|
96
|
+
def rules_by_id = [Analyzer::PARSE, *@rules].to_h { [it.class.id, it] }
|
|
97
|
+
|
|
98
|
+
def terminal
|
|
99
|
+
Render::Terminal.new(env: @env, tty: @out.tty?, color: @options[:color], hyperlinks: @options[:hyperlinks])
|
|
100
|
+
end
|
|
101
|
+
|
|
102
|
+
def parser
|
|
103
|
+
OptionParser.new do |opts|
|
|
104
|
+
opts.banner = "Usage: scully check [options] [PATH ...]\n scully why [options] PATH[:LINE]"
|
|
105
|
+
opts.separator ''
|
|
106
|
+
opts.on('--color=WHEN', MODES, 'Colour output: auto, always, never (default auto)') { @options[:color] = it }
|
|
107
|
+
opts.on('--hyperlinks=WHEN', MODES, 'Link cop names in `scully why`: auto, always, never (default auto)') do
|
|
108
|
+
@options[:hyperlinks] = it
|
|
109
|
+
end
|
|
110
|
+
opts.on('--version', 'Print the version and exit') { finish("scully #{VERSION}") }
|
|
111
|
+
opts.on('--help', 'Print this help and exit') { finish(opts.help) }
|
|
112
|
+
end
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
def finish(text)
|
|
116
|
+
@out.puts text
|
|
117
|
+
throw :done
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
def fail_with(message, hint: false)
|
|
121
|
+
@err.puts "scully: #{message}"
|
|
122
|
+
@err.puts USAGE_HINT if hint
|
|
123
|
+
2
|
|
124
|
+
end
|
|
125
|
+
end
|
|
126
|
+
end
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'find'
|
|
4
|
+
|
|
5
|
+
module Scully
|
|
6
|
+
# Expands PATH arguments into the sorted list of test files to analyze.
|
|
7
|
+
#
|
|
8
|
+
# A file PATH is taken as given. A directory PATH is searched recursively for `*_spec.rb` and
|
|
9
|
+
# `*_test.rb`, skipping vendored, generated and hidden directories below it.
|
|
10
|
+
module Discovery
|
|
11
|
+
TEST_FILE = /_(spec|test)\.rb\z/
|
|
12
|
+
SKIPPED_DIRECTORIES = %w[vendor node_modules tmp .git].freeze
|
|
13
|
+
|
|
14
|
+
module_function
|
|
15
|
+
|
|
16
|
+
# Returns UTF-8 paths relative to +cwd+, deduplicated and sorted by bytes.
|
|
17
|
+
# Raises Scully::Error naming the first PATH that does not exist.
|
|
18
|
+
#
|
|
19
|
+
# PATHs and +cwd+ come in the locale's encoding (BINARY or US-ASCII under LANG=C), so they
|
|
20
|
+
# are handled as bytes, which never raise on names that are not valid UTF-8, and tagged
|
|
21
|
+
# UTF-8 on the way out so they join with source text.
|
|
22
|
+
def call(paths, cwd: Dir.pwd)
|
|
23
|
+
paths = ['.'] if paths.empty?
|
|
24
|
+
root = Pathname.new(File.expand_path(cwd.b))
|
|
25
|
+
paths.flat_map { expand(it.b, root) }.uniq.sort
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
def expand(path, root)
|
|
29
|
+
absolute = root.join(path).cleanpath
|
|
30
|
+
raise Error, "no such file: #{path}" unless absolute.exist?
|
|
31
|
+
return [relative(absolute, root)] unless absolute.directory?
|
|
32
|
+
|
|
33
|
+
search(absolute.to_s).map { relative(Pathname.new(it), root) }
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
# The directory given on the command line is always searched, even if its name would be skipped.
|
|
37
|
+
def search(directory)
|
|
38
|
+
found = []
|
|
39
|
+
Find.find(directory) do |path|
|
|
40
|
+
if File.directory?(path)
|
|
41
|
+
Find.prune if path != directory && skip?(File.basename(path))
|
|
42
|
+
elsif TEST_FILE.match?(File.basename(path)) && File.file?(path)
|
|
43
|
+
found << path
|
|
44
|
+
end
|
|
45
|
+
end
|
|
46
|
+
found
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
def skip?(name) = SKIPPED_DIRECTORIES.include?(name) || name.start_with?('.')
|
|
50
|
+
|
|
51
|
+
def relative(path, root) = path.relative_path_from(root).to_s.force_encoding(Encoding::UTF_8)
|
|
52
|
+
end
|
|
53
|
+
end
|