branchproof 0.9.0 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +19 -0
- data/README.md +77 -18
- data/doc/Branchproof/Evidence.md +3 -0
- data/doc/Branchproof/Project.md +4 -1
- data/doc/Branchproof/RSpecAdapter/ClassRunnerGuard.md +11 -0
- data/doc/Branchproof/RSpecAdapter/ContextLifecycle.md +14 -0
- data/doc/Branchproof/RSpecAdapter/DefaultDiscovery.md +11 -0
- data/doc/Branchproof/RSpecAdapter/ExampleLifecycle.md +11 -0
- data/doc/Branchproof/RSpecAdapter/RunnerGuard.md +11 -0
- data/doc/Branchproof/RSpecAdapter/UnsupportedRunner.md +12 -0
- data/doc/Branchproof/RSpecAdapter.md +47 -0
- data/doc/Branchproof/RailsSupport.md +35 -2
- data/doc/Branchproof/Report.md +3 -0
- data/doc/Branchproof/Runtime.md +3 -0
- data/doc/Branchproof/Worker.md +8 -1
- data/doc/Branchproof.md +8 -1
- data/doc/CHANGELOG.md +19 -0
- data/doc/README.md +77 -18
- data/lib/branchproof/cli.rb +74 -28
- data/lib/branchproof/comparison.rb +52 -6
- data/lib/branchproof/evidence.rb +4 -0
- data/lib/branchproof/focused_report.rb +5 -0
- data/lib/branchproof/project.rb +49 -2
- data/lib/branchproof/rails_support.rb +101 -0
- data/lib/branchproof/report.rb +21 -1
- data/lib/branchproof/rspec_adapter.rb +423 -0
- data/lib/branchproof/runtime.rb +4 -0
- data/lib/branchproof/saved_report.rb +9 -2
- data/lib/branchproof/version.rb +1 -1
- data/lib/branchproof/worker.rb +75 -15
- data/lib/branchproof.rb +1 -0
- data/llms.txt +8 -1
- data/sig/branchproof.rbs +28 -1
- metadata +10 -2
data/doc/README.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# Branchproof
|
|
2
2
|
|
|
3
3
|
Branchproof measures decision, condition, modified condition/decision (MC/DC),
|
|
4
|
-
and decision-table coverage from one serial Minitest run. It discovers Ruby
|
|
4
|
+
and decision-table coverage from one serial Minitest or RSpec run. It discovers Ruby
|
|
5
5
|
decisions through Prism, records their runtime paths, and attributes evidence
|
|
6
6
|
to tests. Boolean decisions receive the coverage ladder; `case`, pattern
|
|
7
7
|
alternatives, safe navigation, and conditional assignments receive alternative
|
|
@@ -22,10 +22,13 @@ Or add it to a bundle:
|
|
|
22
22
|
bundle add branchproof
|
|
23
23
|
```
|
|
24
24
|
|
|
25
|
-
Branchproof targets CRuby 3.3 and 3.4, Minitest 5.x, and Prism 1.x. The
|
|
26
|
-
published core runtime matrix is the CI matrix. Rails
|
|
27
|
-
|
|
28
|
-
|
|
25
|
+
Branchproof targets CRuby 3.3 and 3.4, Minitest 5.x, RSpec 3.13, and Prism 1.x. The
|
|
26
|
+
published core runtime matrix is the CI matrix. Rails/RSpec execution is limited
|
|
27
|
+
to Rails 8.1.x, rspec-rails 8.x, RSpec 3.13.x, and CRuby 3.4.x; passing core
|
|
28
|
+
tests on Ruby 4.0 does not imply Rails/RSpec support. Integration has been checked
|
|
29
|
+
with Rails 8.1.3.1, rspec-rails 8.0.4, RSpec Core 3.13.6, and CRuby 3.4.7.
|
|
30
|
+
Rails and RSpec are optional dependencies supplied by the application.
|
|
31
|
+
Minitest 5.x remains a runtime dependency of this gem.
|
|
29
32
|
Unsupported syntax and incomplete observations remain visible in the report
|
|
30
33
|
instead of being counted as coverage.
|
|
31
34
|
|
|
@@ -33,7 +36,7 @@ instead of being counted as coverage.
|
|
|
33
36
|
|
|
34
37
|
Run `branchproof analyze` with the source files or globs to inspect, followed by
|
|
35
38
|
options. A second `--` separates Branchproof options from arguments passed to
|
|
36
|
-
|
|
39
|
+
the selected test framework:
|
|
37
40
|
|
|
38
41
|
```sh
|
|
39
42
|
branchproof analyze 'lib/**/*.rb' --test 'test/**/*_test.rb' \
|
|
@@ -58,22 +61,60 @@ the application's bundle and Rails version remain in effect:
|
|
|
58
61
|
bundle exec branchproof analyze 'app/**/*.rb' --project rails --test 'test/**/*_test.rb'
|
|
59
62
|
```
|
|
60
63
|
|
|
64
|
+
Choose the test framework independently from project kind:
|
|
65
|
+
|
|
66
|
+
```sh
|
|
67
|
+
bundle exec branchproof analyze 'app/**/*.rb' --project rails --framework rspec
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
`--framework auto|minitest|rspec` is explicit about the adapter. Auto mode
|
|
71
|
+
selects RSpec when `.rspec` or `spec/**/*_spec.rb` markers exist and Minitest
|
|
72
|
+
when `test/**/*_test.rb` or `test/**/test_*.rb` markers exist. If both are
|
|
73
|
+
present, specify the framework. RSpec discovery uses `spec/**/*_spec.rb`; an
|
|
74
|
+
explicit `--test` selection remains authoritative. Project roots are resolved
|
|
75
|
+
explicitly, and report paths are relative to that root so reports from
|
|
76
|
+
different checkouts can be compared.
|
|
77
|
+
|
|
78
|
+
Install `rspec` in the application's test bundle; Rails applications also need
|
|
79
|
+
`rspec-rails`. RSpec reads its usual option files and `SPEC_OPTS`, including
|
|
80
|
+
helper requires, filters, ordering, and file or example-ID selectors. Selectors
|
|
81
|
+
from RSpec configuration replace default discovery. Combining those selectors
|
|
82
|
+
with an explicit Branchproof `--test` is rejected as ambiguous; ordinary filters
|
|
83
|
+
such as `--tag` and `--example` can accompany `--test`.
|
|
84
|
+
|
|
85
|
+
RSpec before hooks, eager `let!`, and around-hook prefixes own setup evidence.
|
|
86
|
+
The example body and helpers evaluated there own body evidence; after hooks,
|
|
87
|
+
mock cleanup, and around-hook suffixes own teardown evidence. Suite and context
|
|
88
|
+
hooks remain unattributed. Pending and skipped examples do not invent execution;
|
|
89
|
+
an unexpectedly passing pending example remains a failure.
|
|
90
|
+
|
|
91
|
+
RSpec support is serial: dry-run, bisect, DRb, custom runners, nested runs, and
|
|
92
|
+
repeated example attempts are rejected. Rails/RSpec supports Rails 8.1.x with
|
|
93
|
+
rspec-rails 8.x on CRuby 3.4.x. Feature and system specs use the in-process
|
|
94
|
+
Capybara `rack_test` driver; browser drivers require execution-context support
|
|
95
|
+
outside this release. Capybara is optional for apps that do not use those specs.
|
|
96
|
+
|
|
61
97
|
For a project rooted at the current directory, `--project auto` is the
|
|
62
98
|
default. It selects Rails only when both `config/application.rb` and
|
|
63
99
|
`config/environment.rb` exist; otherwise it selects a plain Ruby project.
|
|
64
100
|
Use `--project ruby` or `--project rails` to override detection. An explicit
|
|
65
101
|
Rails project without both boot files is a usage error.
|
|
66
102
|
|
|
67
|
-
|
|
103
|
+
Minitest defaults discover the sorted, de-duplicated union of
|
|
68
104
|
`test/**/*_test.rb` and `test/**/test_*.rb`. Helper, support, and fixture files
|
|
69
105
|
are excluded from that default set. `--test` remains authoritative when test
|
|
70
106
|
files are selected explicitly. The worker prepends the project's `lib` and
|
|
71
|
-
`test` directories to its child load path, so application `require` calls
|
|
107
|
+
`test` directories (`lib` and `spec` for RSpec) to its child load path, so application `require` calls
|
|
72
108
|
resolve without changing the parent process.
|
|
73
109
|
|
|
74
|
-
Rails analysis boots
|
|
75
|
-
|
|
76
|
-
|
|
110
|
+
Rails analysis boots the application inside the isolated worker after
|
|
111
|
+
Branchproof's loader and the selected framework hooks are installed. The
|
|
112
|
+
application's `rails_helper` owns requiring and configuring `rspec/rails` after
|
|
113
|
+
the loader; the helper must not require `branchproof` again. Selected specs
|
|
114
|
+
may require only `spec_helper` and run without booting Rails,
|
|
115
|
+
even when project detection selects Rails. Once Rails is loaded, Branchproof
|
|
116
|
+
requires an initialized application and enforces the Rails/RSpec support limits.
|
|
117
|
+
The child receives `RAILS_ENV=test`, `RACK_ENV=test`, `PARALLEL_WORKERS=1`,
|
|
77
118
|
`DISABLE_BOOTSNAP=1`, and `DISABLE_SPRING=1`; the invoking process environment
|
|
78
119
|
is unchanged. The Rails metadata in the report identifies the selected
|
|
79
120
|
project and Rails version.
|
|
@@ -93,6 +134,17 @@ the witness pair or missing counterpart constraints beside each condition;
|
|
|
93
134
|
Level 2 and Level 3 also list named supporting tests. Repeated supporting-set
|
|
94
135
|
rows are collapsed in terminal output only.
|
|
95
136
|
|
|
137
|
+
RSpec owners use the example's `full_description` and positional `example_id`.
|
|
138
|
+
Terminal views include quoted rerun commands, including selectors for shared
|
|
139
|
+
examples. These selectors apply to the recorded spec revision. Comparisons
|
|
140
|
+
require matching spec and declaration digests before matching example owners;
|
|
141
|
+
changed specs and older reports without digests are treated conservatively.
|
|
142
|
+
Runner output streams to stderr while JSON reports remain on stdout.
|
|
143
|
+
Pending, skipped, fixed-pending, and failed examples map to the corresponding
|
|
144
|
+
baseline statuses; suite and context lifecycle events remain visible, and
|
|
145
|
+
observations without a test owner are counted as unattributed. The same levels
|
|
146
|
+
and terminal views are available for both adapters.
|
|
147
|
+
|
|
96
148
|
For example, running the contents of the small `decision.rb` /
|
|
97
149
|
`test_decision_test.rb` fixture from a project `lib/` and `test/` directory
|
|
98
150
|
with the terminal format produces a summary like this:
|
|
@@ -654,12 +706,14 @@ the serial Minitest runner. This is useful for seeds and name filters:
|
|
|
654
706
|
branchproof analyze 'lib/**/*.rb' --level 1 -- --seed 9001 -n /checkout/
|
|
655
707
|
```
|
|
656
708
|
|
|
657
|
-
The
|
|
658
|
-
Rails applications. Rails lazy and eager loading are supported
|
|
659
|
-
application does not enable reloading for the test run.
|
|
660
|
-
|
|
661
|
-
|
|
662
|
-
|
|
709
|
+
The first release supports serial Minitest and RSpec execution in plain Ruby
|
|
710
|
+
projects and Rails applications. Rails lazy and eager loading are supported
|
|
711
|
+
when the application does not enable reloading for the test run. Transactions
|
|
712
|
+
and in-process specs are supported within the serial process policy. Parallel
|
|
713
|
+
or forked runners, remote or threaded browser drivers, mutation execution,
|
|
714
|
+
Rails system/browser tests, custom Rails test commands, generated tests, and
|
|
715
|
+
reloading configurations are explicitly unsupported and produce diagnostics
|
|
716
|
+
rather than a passing analysis.
|
|
663
717
|
|
|
664
718
|
## Library entry points
|
|
665
719
|
|
|
@@ -672,7 +726,8 @@ Branchproof::Records.id(name: "stable identity")
|
|
|
672
726
|
|
|
673
727
|
`require "mcdc"` and `MCDC` are compatibility aliases for the same public
|
|
674
728
|
namespace. The CLI is the supported way to run a complete analysis;
|
|
675
|
-
`Branchproof::Project` exposes project selection and
|
|
729
|
+
`Branchproof::Project` exposes project and framework selection and
|
|
730
|
+
child-environment policy,
|
|
676
731
|
and `Branchproof::RailsSupport` is the optional Rails boot boundary. The library
|
|
677
732
|
classes expose the source, runtime, evidence, analysis, and report contracts
|
|
678
733
|
for adapters and integrations.
|
|
@@ -705,6 +760,10 @@ Run the commands with the Ruby executable you intend to validate. The checked
|
|
|
705
760
|
release environments are CRuby 3.3.6 and 3.4.5. Each runtime
|
|
706
761
|
must provide the declared Minitest 5.x and Prism 1.x dependencies.
|
|
707
762
|
|
|
763
|
+
The repeatable native-versus-instrumented adapter benchmark and its captured
|
|
764
|
+
Ruby 3.4.7 result are in
|
|
765
|
+
[`docs/benchmarks/rspec-adapter.md`](docs/benchmarks/rspec-adapter.md).
|
|
766
|
+
|
|
708
767
|
## License
|
|
709
768
|
|
|
710
769
|
Branchproof is available under the [Apache License, Version 2.0](LICENSE.txt).
|
data/lib/branchproof/cli.rb
CHANGED
|
@@ -33,10 +33,10 @@ module Branchproof
|
|
|
33
33
|
|
|
34
34
|
inventory = build_inventory(options)
|
|
35
35
|
evidence = empty_evidence(inventory, options)
|
|
36
|
-
baseline = if options
|
|
37
|
-
{ status: "INCOMPLETE", executed_tests: 0, failed_tests: 0, skipped_tests: 0, finalized: false }
|
|
38
|
-
else
|
|
36
|
+
baseline = if run_worker?(options)
|
|
39
37
|
run_worker(options, inventory, evidence)
|
|
38
|
+
else
|
|
39
|
+
{ status: "INCOMPLETE", executed_tests: 0, failed_tests: 0, skipped_tests: 0, finalized: false }
|
|
40
40
|
end
|
|
41
41
|
if value(baseline, :evidence)
|
|
42
42
|
merge_status = evidence.merge(snapshot: value(baseline, :evidence))
|
|
@@ -93,7 +93,7 @@ module Branchproof
|
|
|
93
93
|
def help
|
|
94
94
|
@stdout.write(<<~HELP)
|
|
95
95
|
Usage:
|
|
96
|
-
branchproof analyze [SOURCE_GLOB ...] [--test TEST_GLOB] [--project auto|ruby|rails]
|
|
96
|
+
branchproof analyze [SOURCE_GLOB ...] [--test TEST_GLOB] [--project auto|ruby|rails] [--framework auto|minitest|rspec]
|
|
97
97
|
[--view decisions|conditions|tests|decision-tables] [--level 1|2|3] [--missing-only]
|
|
98
98
|
[--format terminal|json] [--output PATH] [--limits PATH] [--no-reachability] [-- RUNNER_ARGS]
|
|
99
99
|
branchproof report SNAPSHOT [--view decisions|conditions|tests|decision-tables]
|
|
@@ -205,12 +205,24 @@ module Branchproof
|
|
|
205
205
|
source = value(test, :source) || {}
|
|
206
206
|
[value(test, :id), { relative_path: relative_path(value(source, :path), root), line: value(source, :line) }]
|
|
207
207
|
end
|
|
208
|
-
|
|
208
|
+
project_metadata = value(baseline, :project) || options[:project]
|
|
209
|
+
selected_test_files = value(baseline, :selected_test_files)
|
|
210
|
+
test_files = selected_test_files || options[:tests]
|
|
211
|
+
metadata = {
|
|
212
|
+
captured_at: Time.now.utc.iso8601, requested_level: options[:level], project_kind: options[:project][:kind],
|
|
209
213
|
project_root: root, source_patterns: options[:source_patterns].map { |path| relative_path(path, root) },
|
|
210
214
|
test_patterns: options[:test_patterns].map { |path| relative_path(path, root) },
|
|
211
|
-
test_files:
|
|
215
|
+
test_files: Array(test_files).map { |path| relative_path(path, root) }, runner_args: options[:runner_args],
|
|
212
216
|
seed: value(baseline, :seed), limits: options[:limits], test_locations: locations,
|
|
213
|
-
reachability: options[:reachability]
|
|
217
|
+
reachability: options[:reachability]
|
|
218
|
+
}
|
|
219
|
+
%i[selected_test_files selected_example_ids].each do |key|
|
|
220
|
+
metadata[key] = value(baseline, key) if value(baseline, key)
|
|
221
|
+
end
|
|
222
|
+
%i[framework framework_version rspec_rails_version rails_version].each do |key|
|
|
223
|
+
metadata[key] = value(project_metadata, key) if value(project_metadata, key)
|
|
224
|
+
end
|
|
225
|
+
metadata
|
|
214
226
|
end
|
|
215
227
|
|
|
216
228
|
def relative_path(path, root)
|
|
@@ -228,7 +240,7 @@ module Branchproof
|
|
|
228
240
|
args = args[0...delimiter] if delimiter
|
|
229
241
|
options = { level: 3, format: :terminal, output: nil, tests: [], source_paths: [], limits: Limits.default,
|
|
230
242
|
runner_args: runner_args, project: nil, missing_only: false, view: :decisions,
|
|
231
|
-
reachability: true }
|
|
243
|
+
reachability: true, project_mode: "auto", framework: "auto", explicit_tests: false }
|
|
232
244
|
until args.empty?
|
|
233
245
|
token = args.shift
|
|
234
246
|
case token
|
|
@@ -255,11 +267,14 @@ module Branchproof
|
|
|
255
267
|
when "--test"
|
|
256
268
|
options[:tests] << args.shift
|
|
257
269
|
raise ArgumentError, "--test requires a glob" if options[:tests].last.nil? || options[:tests].last.empty?
|
|
258
|
-
when "--project"
|
|
259
|
-
mode = args.shift
|
|
260
|
-
raise ArgumentError, "--project requires auto, ruby, or rails" if mode.nil? || mode.empty?
|
|
261
270
|
|
|
262
|
-
options[:
|
|
271
|
+
options[:explicit_tests] = true
|
|
272
|
+
when "--framework"
|
|
273
|
+
options[:framework] = args.shift
|
|
274
|
+
raise ArgumentError, "--framework requires auto, minitest, or rspec" if options[:framework].nil? || options[:framework].empty?
|
|
275
|
+
when "--project"
|
|
276
|
+
options[:project_mode] = args.shift
|
|
277
|
+
raise ArgumentError, "--project requires auto, ruby, or rails" if options[:project_mode].nil? || options[:project_mode].empty?
|
|
263
278
|
when "--limits"
|
|
264
279
|
limits_path = args.shift
|
|
265
280
|
raise ArgumentError, "--limits requires a readable JSON path" unless limits_path && File.file?(limits_path)
|
|
@@ -273,16 +288,16 @@ module Branchproof
|
|
|
273
288
|
options[:source_paths] << token
|
|
274
289
|
end
|
|
275
290
|
end
|
|
291
|
+
options[:project] = Project.new(root: Dir.pwd, mode: options[:project_mode], framework: options[:framework]).to_h
|
|
276
292
|
validate_view!(options)
|
|
277
293
|
if options[:missing_only] && (options[:format] != :terminal || options[:level] == 1)
|
|
278
294
|
raise ArgumentError, "--missing-only requires terminal format and level 2 or 3"
|
|
279
295
|
end
|
|
280
296
|
|
|
281
|
-
options[:project] ||= Project.new(root: Dir.pwd, mode: "auto").to_h
|
|
282
297
|
options[:source_paths] = default_sources if options[:source_paths].empty?
|
|
283
|
-
options[:tests] = default_tests if options[:tests].empty?
|
|
298
|
+
options[:tests] = default_tests(options[:project]) if options[:tests].empty?
|
|
284
299
|
options[:source_patterns] = options[:source_paths].dup
|
|
285
|
-
options[:test_patterns] =
|
|
300
|
+
options[:test_patterns] = options[:explicit_tests] ? options[:tests].dup : options[:project][:test_patterns]
|
|
286
301
|
options[:tests] = expand_paths(options[:tests], root: options[:project][:root])
|
|
287
302
|
options
|
|
288
303
|
end
|
|
@@ -315,18 +330,10 @@ module Branchproof
|
|
|
315
330
|
begin
|
|
316
331
|
directory = Dir.mktmpdir("branchproof-run-")
|
|
317
332
|
config = File.join(directory, "request.json")
|
|
318
|
-
payload =
|
|
319
|
-
inventory: inventory, limits: options[:limits], run_id: evidence.run_id,
|
|
320
|
-
test_files: options[:tests], runner_args: options[:runner_args], project: options[:project],
|
|
321
|
-
result_path: File.join(directory, "result.json"), marker_path: File.join(directory, "complete.marker")
|
|
322
|
-
}
|
|
333
|
+
payload = worker_payload(options, inventory, evidence, directory)
|
|
323
334
|
File.binwrite(config, JSON.generate(normalize(payload)))
|
|
324
335
|
script = "require 'branchproof'; exit(Branchproof::Worker.child_process(ARGV.fetch(0)).to_i)"
|
|
325
|
-
|
|
326
|
-
File.expand_path("..", __dir__), "-e", script, config,
|
|
327
|
-
chdir: options[:project][:root])
|
|
328
|
-
@stderr.write(child_stdout) unless child_stdout.to_s.empty?
|
|
329
|
-
@stderr.write(stderr) unless stderr.to_s.empty?
|
|
336
|
+
stderr, child_status = stream_worker(options, script, config)
|
|
330
337
|
unless File.file?(payload[:marker_path]) && File.file?(payload[:result_path])
|
|
331
338
|
return { status: "ERROR", executed_tests: 0, failed_tests: 0, skipped_tests: 0, finalized: false,
|
|
332
339
|
diagnostics: [{ code: "worker_incomplete", severity: "error", message: stderr.to_s.strip }] }
|
|
@@ -346,6 +353,30 @@ module Branchproof
|
|
|
346
353
|
diagnostics: [{ code: "runner", severity: "error", message: e.message }] }
|
|
347
354
|
end
|
|
348
355
|
|
|
356
|
+
def stream_worker(options, script, config)
|
|
357
|
+
diagnostic_output = +""
|
|
358
|
+
Open3.popen3(options[:project][:environment], RbConfig.ruby, "-I",
|
|
359
|
+
File.expand_path("..", __dir__), "-e", script, config,
|
|
360
|
+
chdir: options[:project][:root]) do |input, output, errors, child|
|
|
361
|
+
input.close
|
|
362
|
+
streams = [output, errors]
|
|
363
|
+
until streams.empty?
|
|
364
|
+
IO.select(streams).first.each do |stream|
|
|
365
|
+
chunk = stream.read_nonblock(16_384, exception: false)
|
|
366
|
+
if chunk.nil?
|
|
367
|
+
streams.delete(stream)
|
|
368
|
+
elsif chunk != :wait_readable
|
|
369
|
+
@stderr.write(chunk)
|
|
370
|
+
@stderr.flush if @stderr.respond_to?(:flush)
|
|
371
|
+
diagnostic_output << chunk
|
|
372
|
+
diagnostic_output = diagnostic_output.byteslice(-16_384, 16_384) if diagnostic_output.bytesize > 16_384
|
|
373
|
+
end
|
|
374
|
+
end
|
|
375
|
+
end
|
|
376
|
+
[diagnostic_output, child.value]
|
|
377
|
+
end
|
|
378
|
+
end
|
|
379
|
+
|
|
349
380
|
def output_report(report, options)
|
|
350
381
|
created = false
|
|
351
382
|
if options[:output]
|
|
@@ -362,6 +393,19 @@ module Branchproof
|
|
|
362
393
|
File.unlink(temporary) if created && temporary && File.file?(temporary)
|
|
363
394
|
end
|
|
364
395
|
|
|
396
|
+
def run_worker?(options)
|
|
397
|
+
!options[:tests].empty? || options[:project][:framework].to_s == "rspec"
|
|
398
|
+
end
|
|
399
|
+
|
|
400
|
+
def worker_payload(options, inventory, evidence, directory)
|
|
401
|
+
{
|
|
402
|
+
inventory: inventory, limits: options[:limits], run_id: evidence.run_id,
|
|
403
|
+
test_files: options[:tests], runner_args: options[:runner_args], project: options[:project],
|
|
404
|
+
test_selection_explicit: options[:explicit_tests],
|
|
405
|
+
result_path: File.join(directory, "result.json"), marker_path: File.join(directory, "complete.marker")
|
|
406
|
+
}
|
|
407
|
+
end
|
|
408
|
+
|
|
365
409
|
def report_string(report, format)
|
|
366
410
|
buffer = StringIO.new
|
|
367
411
|
report.write(io: buffer, format: format)
|
|
@@ -377,8 +421,9 @@ module Branchproof
|
|
|
377
421
|
%w[lib/**/*.rb app/**/*.rb]
|
|
378
422
|
end
|
|
379
423
|
|
|
380
|
-
def default_tests
|
|
381
|
-
|
|
424
|
+
def default_tests(project = nil)
|
|
425
|
+
project ||= Project.new(root: Dir.pwd, mode: "auto", framework: "auto").to_h
|
|
426
|
+
candidates = Array(project[:test_patterns]).flat_map { |pattern| Dir.glob(pattern, base: project[:root]) }
|
|
382
427
|
candidates.map! { |path| File.expand_path(path, Dir.pwd) }
|
|
383
428
|
candidates.reject { |path| default_test_excluded?(path) }.uniq.sort
|
|
384
429
|
end
|
|
@@ -387,7 +432,8 @@ module Branchproof
|
|
|
387
432
|
relative = Pathname.new(path).relative_path_from(Pathname.new(Dir.pwd)).to_s
|
|
388
433
|
segments = relative.split(File::SEPARATOR)
|
|
389
434
|
basename = File.basename(path)
|
|
390
|
-
|
|
435
|
+
%w[test_helper.rb spec_helper.rb rails_helper.rb].include?(basename) ||
|
|
436
|
+
segments.include?("support") || segments.include?("fixtures")
|
|
391
437
|
end
|
|
392
438
|
|
|
393
439
|
def expand_paths(paths, root: Dir.pwd)
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
# rubocop:disable Metrics/ClassLength, Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
|
|
4
4
|
require "json"
|
|
5
|
+
require "pathname"
|
|
5
6
|
|
|
6
7
|
module Branchproof
|
|
7
8
|
# Compares two complete report documents without loading or executing the project.
|
|
@@ -369,9 +370,17 @@ module Branchproof
|
|
|
369
370
|
tests = Array(value(value(document, :observations), :tests)).to_h { |test| [value(test, :id).to_s, test] }
|
|
370
371
|
rows = index.tests.to_h { |test| [test[:id].to_s, test] }
|
|
371
372
|
keys = tests.to_h do |id, test|
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
|
|
373
|
+
adapter = value(test, :adapter).to_s
|
|
374
|
+
path = rows.dig(id, :relative_path)
|
|
375
|
+
key = if adapter.downcase == "rspec"
|
|
376
|
+
example_id = value(test, :example_id).to_s
|
|
377
|
+
# Scoped IDs are positional selectors, valid only within the same spec revision.
|
|
378
|
+
[adapter, rspec_example_path(document, example_id) || path, rspec_scope(example_id),
|
|
379
|
+
value(test, :name), path, value(test, :spec_digest), value(test, :source_digest)]
|
|
380
|
+
else
|
|
381
|
+
[adapter, value(test, :class_name), value(test, :method_name), path]
|
|
382
|
+
end
|
|
383
|
+
[id, key.all? { |part| part.is_a?(String) && !part.empty? } ? key : nil]
|
|
375
384
|
end
|
|
376
385
|
results = Array(value(value(document, :analysis), :decisions)).flat_map do |decision|
|
|
377
386
|
Array(value(decision, :condition_results))
|
|
@@ -447,6 +456,13 @@ module Branchproof
|
|
|
447
456
|
if context(@before)[:test_keys].values.sort_by(&:to_s) != context(@after)[:test_keys].values.sort_by(&:to_s)
|
|
448
457
|
changes << "test population differs"
|
|
449
458
|
end
|
|
459
|
+
if [@before, @after].any? do |document|
|
|
460
|
+
context(document)[:tests].any? do |id, test|
|
|
461
|
+
value(test, :adapter).to_s.downcase == "rspec" && context(document)[:test_keys][id].nil?
|
|
462
|
+
end
|
|
463
|
+
end
|
|
464
|
+
changes << "RSpec example identity is uncertain: missing spec revision or selector evidence"
|
|
465
|
+
end
|
|
450
466
|
changes << "Deltas describe observed runs; nondeterminism and external environment changes are not ruled out."
|
|
451
467
|
changes
|
|
452
468
|
end
|
|
@@ -454,9 +470,10 @@ module Branchproof
|
|
|
454
470
|
def comparability_reasons(changed_paths)
|
|
455
471
|
reasons = []
|
|
456
472
|
reasons << "source files changed: #{changed_paths.sort.join(", ")}" unless changed_paths.empty?
|
|
457
|
-
%i[project_kind runner_args source_patterns test_patterns limits
|
|
458
|
-
|
|
459
|
-
|
|
473
|
+
%i[project_kind framework framework_version rspec_rails_version runner_args source_patterns test_patterns limits
|
|
474
|
+
runtime].each do |key|
|
|
475
|
+
left = key == :framework ? comparison_framework(@before) : metadata(@before, key)
|
|
476
|
+
right = key == :framework ? comparison_framework(@after) : metadata(@after, key)
|
|
460
477
|
if key == :runner_args
|
|
461
478
|
left = comparable_runner_args(left)
|
|
462
479
|
right = comparable_runner_args(right)
|
|
@@ -467,6 +484,35 @@ module Branchproof
|
|
|
467
484
|
reasons
|
|
468
485
|
end
|
|
469
486
|
|
|
487
|
+
def comparison_framework(document)
|
|
488
|
+
explicit = metadata(document, :framework)
|
|
489
|
+
return explicit unless explicit.nil? || explicit.to_s.empty?
|
|
490
|
+
|
|
491
|
+
adapters = Array(value(value(document, :observations), :tests)).filter_map do |test|
|
|
492
|
+
adapter = value(test, :adapter).to_s.downcase
|
|
493
|
+
adapter unless adapter.empty? || adapter == "unknown"
|
|
494
|
+
end.uniq
|
|
495
|
+
adapters.length == 1 ? adapters.first : nil
|
|
496
|
+
end
|
|
497
|
+
|
|
498
|
+
def rspec_scope(example_id)
|
|
499
|
+
match = example_id.to_s.match(/(\[[^\]]+\])\z/)
|
|
500
|
+
match && match[1]
|
|
501
|
+
end
|
|
502
|
+
|
|
503
|
+
def rspec_example_path(document, example_id)
|
|
504
|
+
path = example_id.to_s.sub(/\[[^\]]+\]\z/, "")
|
|
505
|
+
return nil if path.empty?
|
|
506
|
+
return path.sub(%r{\A\./}, "") unless path.start_with?("/")
|
|
507
|
+
|
|
508
|
+
root = value(value(document, :source_inventory), :root) || metadata(document, :project_root)
|
|
509
|
+
return nil unless root
|
|
510
|
+
|
|
511
|
+
Pathname.new(path).relative_path_from(Pathname.new(root.to_s)).to_s
|
|
512
|
+
rescue ArgumentError
|
|
513
|
+
nil
|
|
514
|
+
end
|
|
515
|
+
|
|
470
516
|
def source_selection(document)
|
|
471
517
|
value(value(document, :run_metadata), :source_patterns)
|
|
472
518
|
end
|
data/lib/branchproof/evidence.rb
CHANGED
|
@@ -120,6 +120,10 @@ module Branchproof
|
|
|
120
120
|
nil
|
|
121
121
|
end
|
|
122
122
|
|
|
123
|
+
def test_phase_counts
|
|
124
|
+
@tests.transform_values { |test| test[:phase_counts].dup.freeze }.freeze
|
|
125
|
+
end
|
|
126
|
+
|
|
123
127
|
def snapshot
|
|
124
128
|
deep_freeze(deep_dup({ schema_version: SCHEMA_VERSION, tool_version: TOOL_VERSION,
|
|
125
129
|
criterion_version: CRITERION_VERSION,
|
|
@@ -238,6 +238,8 @@ module Branchproof
|
|
|
238
238
|
lines << "Test: #{row[:name]}"
|
|
239
239
|
lines << "Location: #{location(row[:relative_path], row[:line], unavailable: "location unavailable")}"
|
|
240
240
|
lines << "Status: #{row[:status] || "unknown"}"
|
|
241
|
+
command = @coordinator.rerun_command(row[:id])
|
|
242
|
+
lines << "Rerun: #{command}" if command
|
|
241
243
|
lines << "Phases: #{row[:phases].join(", ")}" unless row[:phases].empty?
|
|
242
244
|
render_test_observations(lines, observations)
|
|
243
245
|
lines << ""
|
|
@@ -341,6 +343,9 @@ module Branchproof
|
|
|
341
343
|
return id.to_s unless test
|
|
342
344
|
|
|
343
345
|
label = "#{test[:name]} (#{location(test[:relative_path], test[:line])})"
|
|
346
|
+
command = @coordinator.rerun_command(id)
|
|
347
|
+
return "#{label} (rerun: #{command})" if command
|
|
348
|
+
|
|
344
349
|
duplicates = @test_name_counts[[test[:name], test[:relative_path], test[:line]]]
|
|
345
350
|
duplicates > 1 ? "#{label} [#{id}]" : label
|
|
346
351
|
end
|
data/lib/branchproof/project.rb
CHANGED
|
@@ -4,6 +4,7 @@ module Branchproof
|
|
|
4
4
|
# Resolves the project policy used by the isolated analysis worker.
|
|
5
5
|
class Project
|
|
6
6
|
MODES = %w[auto ruby rails].freeze
|
|
7
|
+
FRAMEWORKS = %w[auto minitest rspec].freeze
|
|
7
8
|
RAILS_ENVIRONMENT = {
|
|
8
9
|
"RAILS_ENV" => "test",
|
|
9
10
|
"RACK_ENV" => "test",
|
|
@@ -12,19 +13,24 @@ module Branchproof
|
|
|
12
13
|
"DISABLE_SPRING" => "1"
|
|
13
14
|
}.freeze
|
|
14
15
|
|
|
15
|
-
def initialize(root:, mode: "auto")
|
|
16
|
+
def initialize(root:, mode: "auto", framework: "auto")
|
|
16
17
|
@root = File.expand_path(root)
|
|
17
18
|
@mode = mode.to_s
|
|
19
|
+
@framework = framework.to_s
|
|
18
20
|
raise ArgumentError, "project must be auto, ruby, or rails" unless MODES.include?(@mode)
|
|
21
|
+
raise ArgumentError, "framework must be auto, minitest, or rspec" unless FRAMEWORKS.include?(@framework)
|
|
19
22
|
|
|
20
23
|
validate_root!
|
|
24
|
+
resolve_framework!
|
|
21
25
|
end
|
|
22
26
|
|
|
23
27
|
def to_h
|
|
24
28
|
{
|
|
25
29
|
kind: kind,
|
|
26
30
|
root: @root,
|
|
27
|
-
|
|
31
|
+
framework: framework,
|
|
32
|
+
test_patterns: test_patterns,
|
|
33
|
+
load_paths: load_paths,
|
|
28
34
|
environment: kind == "rails" ? RAILS_ENVIRONMENT.dup : {}
|
|
29
35
|
}
|
|
30
36
|
end
|
|
@@ -43,6 +49,47 @@ module Branchproof
|
|
|
43
49
|
File.file?(File.join(@root, "config", "environment.rb"))
|
|
44
50
|
end
|
|
45
51
|
|
|
52
|
+
def framework
|
|
53
|
+
@resolved_framework
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
def resolve_framework!
|
|
57
|
+
rspec = rspec_markers?
|
|
58
|
+
minitest = minitest_markers?
|
|
59
|
+
if @framework == "auto"
|
|
60
|
+
raise_both_frameworks if rspec && minitest
|
|
61
|
+
@resolved_framework = rspec ? "rspec" : "minitest"
|
|
62
|
+
else
|
|
63
|
+
@resolved_framework = @framework
|
|
64
|
+
end
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
def rspec_markers?
|
|
68
|
+
File.file?(File.join(@root, ".rspec")) || !Dir.glob(File.join(@root, "spec", "**", "*_spec.rb")).empty?
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
def raise_both_frameworks
|
|
72
|
+
raise ArgumentError,
|
|
73
|
+
"project contains both RSpec and Minitest markers; specify --framework rspec or --framework minitest"
|
|
74
|
+
end
|
|
75
|
+
|
|
76
|
+
def minitest_markers?
|
|
77
|
+
!Dir.glob(File.join(@root, "test", "**", "*_test.rb")).empty? ||
|
|
78
|
+
!Dir.glob(File.join(@root, "test", "**", "test_*.rb")).empty?
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
def test_patterns
|
|
82
|
+
if framework == "rspec"
|
|
83
|
+
%w[spec/**/*_spec.rb].freeze
|
|
84
|
+
else
|
|
85
|
+
%w[test/**/*_test.rb test/**/test_*.rb].freeze
|
|
86
|
+
end
|
|
87
|
+
end
|
|
88
|
+
|
|
89
|
+
def load_paths
|
|
90
|
+
[File.join(@root, "lib"), File.join(@root, framework == "rspec" ? "spec" : "test")]
|
|
91
|
+
end
|
|
92
|
+
|
|
46
93
|
def validate_root!
|
|
47
94
|
raise ArgumentError, "project root does not exist: #{@root}" unless File.directory?(@root)
|
|
48
95
|
return unless @mode == "rails" && !rails_files?
|