branchproof 0.11.1 → 0.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +82 -0
- data/README.md +440 -29
- data/doc/Branchproof/CLI.md +1 -1
- data/doc/Branchproof/ChangedCoverage.md +19 -0
- data/doc/Branchproof/ChangedCoveragePolicy.md +24 -0
- data/doc/Branchproof/ChangedDecisionMap.md +20 -0
- data/doc/Branchproof/ChangedScope.md +18 -0
- data/doc/Branchproof/Collation.md +23 -0
- data/doc/Branchproof/CollationManifest.md +29 -0
- data/doc/Branchproof/CoverageSummary.md +20 -0
- data/doc/Branchproof/Doctor/TerminalRenderer.md +15 -0
- data/doc/Branchproof/Doctor.md +34 -0
- data/doc/Branchproof/FallbackInstrumentation.md +8 -0
- data/doc/Branchproof/FallbackRuntime.md +12 -0
- data/doc/Branchproof/FallbackSyntax.md +13 -0
- data/doc/Branchproof/GitChanges.md +19 -0
- data/doc/Branchproof/GithubReport.md +25 -0
- data/doc/Branchproof/HtmlReport.md +22 -0
- data/doc/Branchproof/Instrumenter.md +1 -1
- data/doc/Branchproof/MinitestAdapter.md +4 -0
- data/doc/Branchproof/RakeTask.md +121 -0
- data/doc/Branchproof/Report.md +41 -3
- data/doc/Branchproof/ReportSelection.md +10 -1
- data/doc/Branchproof/Runtime.md +4 -1
- data/doc/Branchproof/Source.md +1 -1
- data/doc/Branchproof/SummaryRanking/Decision.md +86 -0
- data/doc/Branchproof/SummaryRanking/SourceFile.md +43 -0
- data/doc/Branchproof/SummaryRanking.md +36 -0
- data/doc/Branchproof/SummaryReport.md +42 -0
- data/doc/Branchproof/Worker.md +1 -1
- data/doc/Branchproof.md +21 -1
- data/doc/CHANGELOG.md +82 -0
- data/doc/README.md +440 -29
- data/lib/branchproof/analyzer.rb +2 -52
- data/lib/branchproof/changed_coverage.rb +177 -0
- data/lib/branchproof/changed_decision_map.rb +359 -0
- data/lib/branchproof/changed_scope.rb +93 -0
- data/lib/branchproof/cli.rb +242 -21
- data/lib/branchproof/collation.rb +525 -0
- data/lib/branchproof/collation_manifest.rb +129 -0
- data/lib/branchproof/comparison.rb +11 -3
- data/lib/branchproof/comparison_report.rb +2 -0
- data/lib/branchproof/configuration.rb +12 -2
- data/lib/branchproof/coverage_policy.rb +122 -0
- data/lib/branchproof/coverage_summary.rb +94 -0
- data/lib/branchproof/doctor.rb +241 -0
- data/lib/branchproof/evidence.rb +5 -1
- data/lib/branchproof/exception_instrumentation.rb +4 -3
- data/lib/branchproof/exception_syntax.rb +15 -1
- data/lib/branchproof/fallback_instrumentation.rb +25 -0
- data/lib/branchproof/fallback_runtime.rb +14 -0
- data/lib/branchproof/fallback_syntax.rb +90 -0
- data/lib/branchproof/focused_report.rb +81 -22
- data/lib/branchproof/git_changes.rb +188 -0
- data/lib/branchproof/github_report.rb +292 -0
- data/lib/branchproof/html_report.rb +412 -0
- data/lib/branchproof/instrumenter.rb +2 -0
- data/lib/branchproof/minimizer.rb +4 -2
- data/lib/branchproof/minitest_adapter.rb +45 -5
- data/lib/branchproof/rails_support.rb +2 -2
- data/lib/branchproof/rake_task.rb +186 -0
- data/lib/branchproof/report.rb +383 -71
- data/lib/branchproof/report_selection.rb +26 -5
- data/lib/branchproof/runtime.rb +2 -0
- data/lib/branchproof/saved_report.rb +323 -5
- data/lib/branchproof/source.rb +15 -3
- data/lib/branchproof/summary_ranking.rb +165 -0
- data/lib/branchproof/summary_report.rb +241 -0
- data/lib/branchproof/version.rb +1 -1
- data/lib/branchproof/worker.rb +14 -9
- data/lib/branchproof.rb +12 -0
- data/llms.txt +21 -1
- data/sig/branchproof.rbs +206 -5
- metadata +39 -23
data/README.md
CHANGED
|
@@ -4,8 +4,8 @@ Branchproof measures decision, condition, modified condition/decision (MC/DC),
|
|
|
4
4
|
and decision-table coverage from one serial Minitest or RSpec run. It discovers Ruby
|
|
5
5
|
decisions through Prism, records their runtime paths, and attributes evidence
|
|
6
6
|
to tests. Boolean decisions receive the coverage ladder; `case`, pattern
|
|
7
|
-
alternatives, safe navigation,
|
|
8
|
-
coverage.
|
|
7
|
+
alternatives, safe navigation, conditional assignments, and value fallback
|
|
8
|
+
chains such as `name || "untitled"` receive alternative coverage.
|
|
9
9
|
|
|
10
10
|
The gem and primary command are named `branchproof`. The `mcdc` command and
|
|
11
11
|
`MCDC` namespace remain compatibility aliases with the same behavior.
|
|
@@ -22,16 +22,77 @@ Or add it to a bundle:
|
|
|
22
22
|
bundle add branchproof
|
|
23
23
|
```
|
|
24
24
|
|
|
25
|
-
Branchproof
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
Minitest 5.x remains a runtime dependency of this gem.
|
|
25
|
+
Branchproof requires CRuby 4.0 or newer and Prism 1.x at runtime.
|
|
26
|
+
CI tests CRuby 4.0, including Rails 8.1.x and rspec-rails 8.x integrations.
|
|
27
|
+
Newer Ruby versions are allowed by the gem requirement but are not yet tested in CI.
|
|
28
|
+
The application supplies the test framework selected for analysis: Minitest
|
|
29
|
+
`>= 5.25.5, < 7` or RSpec 3.13. Branchproof does not install either framework;
|
|
30
|
+
projects that use both can keep both in their test bundle.
|
|
32
31
|
Unsupported syntax and incomplete observations remain visible in the report
|
|
33
32
|
instead of being counted as coverage.
|
|
34
33
|
|
|
34
|
+
For a Minitest application, keep the runner in the application bundle:
|
|
35
|
+
|
|
36
|
+
```ruby
|
|
37
|
+
gem "branchproof"
|
|
38
|
+
gem "minitest", ">= 5.25.5", "< 7"
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
For an RSpec application, use its existing RSpec bundle:
|
|
42
|
+
|
|
43
|
+
```ruby
|
|
44
|
+
gem "branchproof"
|
|
45
|
+
gem "rspec", "~> 3.13.0", require: false
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
`branchproof report`, `branchproof compare`, and `branchproof collate` read saved
|
|
49
|
+
JSON without loading either test framework, so reports can be used in a
|
|
50
|
+
minimal deployment or offline environment.
|
|
51
|
+
|
|
52
|
+
## Check a project's setup
|
|
53
|
+
|
|
54
|
+
Run `branchproof doctor` from the project root to check static setup facts before
|
|
55
|
+
starting an analysis:
|
|
56
|
+
|
|
57
|
+
```sh
|
|
58
|
+
bundle exec branchproof doctor 'lib/**/*.rb' \
|
|
59
|
+
--test 'test/**/*_test.rb' \
|
|
60
|
+
--project ruby --framework minitest
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Doctor accepts source globs followed by `--test GLOB`, `--project
|
|
64
|
+
auto|ruby|rails`, `--framework auto|minitest|rspec`, `--config PATH` or
|
|
65
|
+
`--no-config`, and `--format terminal|json`. It uses the same project detection,
|
|
66
|
+
configuration precedence, framework selection, test discovery, and source
|
|
67
|
+
safety exclusions as `analyze`. Command-line selections override matching
|
|
68
|
+
`.branchproof.json` values. Default discovery excludes helper, support, and
|
|
69
|
+
fixture test files; an explicit `--test` glob selects those files when requested.
|
|
70
|
+
`--help` prints usage without inspecting the project.
|
|
71
|
+
|
|
72
|
+
Doctor exits 0 when its static checks pass, including warning-only results, and
|
|
73
|
+
2 when setup is blocked or the command is invalid. JSON is one document with
|
|
74
|
+
`schema_version: 1`, `command: "doctor"`, a `ready` or `blocked` status,
|
|
75
|
+
`runtime`, resolved `project` and `configuration`, `selection`, selected
|
|
76
|
+
`framework` metadata, `checks`, and `limitations`. Each check has a stable code,
|
|
77
|
+
`pass`, `warning`, or `error` status, and a message. Add `--format json` to get
|
|
78
|
+
this document on both successful and blocked checks, including configuration
|
|
79
|
+
and argument errors.
|
|
80
|
+
|
|
81
|
+
Framework availability comes from RubyGems metadata. An `activated` version is
|
|
82
|
+
already in the current process; a `discoverable` version is visible in the
|
|
83
|
+
current RubyGems environment but has not been activated. Doctor checks only the
|
|
84
|
+
selected framework, using Minitest `>= 5.25.5, < 7` or `rspec-core ~> 3.13.0`.
|
|
85
|
+
Discoverable metadata does not prove that a framework can load successfully.
|
|
86
|
+
|
|
87
|
+
Doctor is a static readiness check. It does not load source, helpers, tests,
|
|
88
|
+
framework code, or Rails; run a test runner; or inspect Ruby syntax. A ready
|
|
89
|
+
result does not verify Rails boot, actual framework test discovery, runner
|
|
90
|
+
compatibility, instrumentation, coverage, or whether a selected file contains
|
|
91
|
+
executable tests. It warns about Spring or Bootsnap only when already visible in
|
|
92
|
+
loaded features. Configured coverage minima are shown but not evaluated. As with
|
|
93
|
+
other Branchproof commands, Branchproof itself must load successfully, including
|
|
94
|
+
its supported Ruby runtime and core dependencies.
|
|
95
|
+
|
|
35
96
|
## Analyze a test run
|
|
36
97
|
|
|
37
98
|
Run `branchproof analyze` with the source files or globs to inspect, followed by
|
|
@@ -90,7 +151,7 @@ an unexpectedly passing pending example remains a failure.
|
|
|
90
151
|
|
|
91
152
|
RSpec support is serial: dry-run, bisect, DRb, custom runners, nested runs, and
|
|
92
153
|
repeated example attempts are rejected. Rails/RSpec supports Rails 8.1.x with
|
|
93
|
-
rspec-rails 8.x on CRuby
|
|
154
|
+
rspec-rails 8.x on CRuby 4.0. Feature and system specs use the in-process
|
|
94
155
|
Capybara `rack_test` driver; browser drivers require execution-context support
|
|
95
156
|
outside this release. Capybara is optional for apps that do not use those specs.
|
|
96
157
|
|
|
@@ -158,6 +219,11 @@ requested policy gates unavailable and exits 2; without a policy, a failed test
|
|
|
158
219
|
run retains the existing exit status 1. With no policy, the existing report
|
|
159
220
|
exit behavior remains unchanged.
|
|
160
221
|
|
|
222
|
+
For a separate policy on changed decisions, use `--changed-since REF` with
|
|
223
|
+
repeatable `--minimum-changed criterion=threshold` options. The configuration
|
|
224
|
+
equivalent is `"minimum_changed": { "mcdc": 80 }`; CLI values override matching
|
|
225
|
+
entries. See [Changed-decision coverage and gates](#changed-decision-coverage-and-gates).
|
|
226
|
+
|
|
161
227
|
For a Minitest project, a focused configuration can select the library and
|
|
162
228
|
test trees directly:
|
|
163
229
|
|
|
@@ -186,8 +252,9 @@ is unchanged. The Rails metadata in the report identifies the selected
|
|
|
186
252
|
project and Rails version.
|
|
187
253
|
|
|
188
254
|
`--output` writes the report through an atomic replacement. Without it, the
|
|
189
|
-
report is written to standard output.
|
|
190
|
-
default)
|
|
255
|
+
report is written to standard output. `analyze` accepts `terminal` (the
|
|
256
|
+
default), `json`, and `github`; `compare` accepts `terminal` and `json`.
|
|
257
|
+
`report` also accepts `html` for a portable offline page. JSON includes source identities, criterion and schema
|
|
191
258
|
versions, baseline status, diagnostics, completeness, observations, and
|
|
192
259
|
analysis fields.
|
|
193
260
|
|
|
@@ -486,7 +553,7 @@ Decision tables fully covered: 13/18 decisions
|
|
|
486
553
|
|
|
487
554
|
### Find missing cases
|
|
488
555
|
|
|
489
|
-
Use `--missing-only` to focus
|
|
556
|
+
Use `--missing-only` to focus terminal and HTML reports on conditions that still
|
|
490
557
|
lack independence evidence:
|
|
491
558
|
|
|
492
559
|
```sh
|
|
@@ -511,11 +578,15 @@ making the decision false. The missing condition is presented as:
|
|
|
511
578
|
|
|
512
579
|
```text
|
|
513
580
|
Condition 1: Instruction.installed?(content)
|
|
514
|
-
NOT_PROVEN —
|
|
515
|
-
|
|
516
|
-
|
|
517
|
-
Instruction.installed?(content)
|
|
518
|
-
|
|
581
|
+
NOT_PROVEN — Missing evidence for: Instruction.installed?(content)
|
|
582
|
+
Observed decision expression: content && Instruction.installed?(content)
|
|
583
|
+
Find two executions where:
|
|
584
|
+
Instruction.installed?(content) changes between true and false
|
|
585
|
+
content remains true
|
|
586
|
+
decision outcome changes between true and false
|
|
587
|
+
Observed: content = true; Instruction.installed?(content) = true; decision = true
|
|
588
|
+
Missing counterpart: content = true; Instruction.installed?(content) = false; decision = false (candidate; not observed)
|
|
589
|
+
Boolean requirement; application-level feasibility unknown
|
|
519
590
|
```
|
|
520
591
|
|
|
521
592
|
An existing file without the instruction block
|
|
@@ -524,6 +595,13 @@ describes required truth values, not application inputs or guaranteed
|
|
|
524
595
|
reachable paths. In Ruby, only `false` and `nil` are falsey; an empty string
|
|
525
596
|
is truthy.
|
|
526
597
|
|
|
598
|
+
This wording is generated from recorded expressions and analyzer candidates.
|
|
599
|
+
Boolean values describe truthiness, not the original Ruby objects. Short-circuited
|
|
600
|
+
conditions are marked as not evaluated. Unary negation remains in the full
|
|
601
|
+
decision expression: for `user.paid? && !user.suspended?`, the atomic condition is
|
|
602
|
+
`user.suspended?`, so its observed value is `false` and the missing counterpart
|
|
603
|
+
requires `true`. No domain wording or application inputs are inferred.
|
|
604
|
+
|
|
527
605
|
`NOT_PROVEN` means analysis ran but did not find the required pair of
|
|
528
606
|
observations. `NOT CALCULATED` means analysis was not available or was not
|
|
529
607
|
requested; check the test status and diagnostics before adding tests.
|
|
@@ -576,9 +654,9 @@ execution. JSON output always contains the complete evidence document, so an
|
|
|
576
654
|
explicit view cannot be combined with `--format json`. The `mcdc` executable
|
|
577
655
|
accepts the same arguments for existing scripts.
|
|
578
656
|
|
|
579
|
-
Terminal reports can be narrowed with `--focus PATH[:LINE]` and bounded with
|
|
657
|
+
Terminal and HTML reports can be narrowed with `--focus PATH[:LINE]` and bounded with
|
|
580
658
|
`--top N`, where `N` is a positive integer. These options are accepted by
|
|
581
|
-
`analyze` and `report
|
|
659
|
+
`analyze` and `report`; for HTML they apply only to offline `report`. Focus matches the source spans
|
|
582
660
|
captured in the report; rendering does not reopen or need the original source
|
|
583
661
|
file. `--focus` and `--top` affect displayed detail only: the coverage summary,
|
|
584
662
|
policy gates, diagnostics, exit status, and global counts remain unchanged.
|
|
@@ -590,7 +668,8 @@ relative path and source line, with deterministic tie-breakers for each view:
|
|
|
590
668
|
decisions use column and stable ID; decision tables use decision ID; conditions
|
|
591
669
|
and alternatives use column, decision ID, and condition or alternative index;
|
|
592
670
|
tests use test name and ID. `--top` follows that source order; it does not rank
|
|
593
|
-
rows by risk or coverage.
|
|
671
|
+
rows by risk or coverage. Only `--view summary` ranks rows (see
|
|
672
|
+
[Summary view](#summary-view-where-to-start)). For example:
|
|
594
673
|
|
|
595
674
|
```sh
|
|
596
675
|
bundle exec branchproof analyze 'lib/**/*.rb' --focus lib/access.rb:12 --top 5
|
|
@@ -604,6 +683,77 @@ recorded selectors and commands. The main finding rows respect the selection;
|
|
|
604
683
|
supplemental unexecuted, unattributed, and unsupported sections retain their
|
|
605
684
|
run-wide counts rather than expanding into all detail rows.
|
|
606
685
|
|
|
686
|
+
### Summary view: where to start
|
|
687
|
+
|
|
688
|
+
Use `--view summary` to see the largest gaps first. It is accepted by
|
|
689
|
+
`analyze` and `report` and is terminal-only.
|
|
690
|
+
|
|
691
|
+
```sh
|
|
692
|
+
bundle exec branchproof analyze 'lib/**/*.rb' --view summary --top 10
|
|
693
|
+
bundle exec branchproof report .branchproof/current.json --view summary
|
|
694
|
+
```
|
|
695
|
+
|
|
696
|
+
The view has two ranked lists:
|
|
697
|
+
|
|
698
|
+
* **Files**: one row per source file, with counts and explicit denominators.
|
|
699
|
+
* **Where to start**: one row per supported decision that still has a gap,
|
|
700
|
+
with its location, the missing cases, and the tests that already reach it.
|
|
701
|
+
|
|
702
|
+
Both lists use the same deterministic order:
|
|
703
|
+
|
|
704
|
+
1. Unexecuted decisions (or files with more of them) come first.
|
|
705
|
+
2. Then more missing obligations: decision-table rules, unproven MC/DC
|
|
706
|
+
conditions, and missing alternatives.
|
|
707
|
+
3. Ties break by project-relative path, line, column, and decision ID.
|
|
708
|
+
|
|
709
|
+
The order counts missing obligations. It is not a risk estimate.
|
|
710
|
+
|
|
711
|
+
```text
|
|
712
|
+
Files:
|
|
713
|
+
1. lib/access.rb: 3/3 decisions with gaps; 1 unexecuted; DT 8/11 rules missing; MC/DC 7/8 conditions unproven
|
|
714
|
+
2. lib/pricing.rb: 2/2 decisions with gaps; DT 2/3 rules missing; MC/DC 2/2 conditions unproven; 2/3 alternatives missing
|
|
715
|
+
|
|
716
|
+
Where to start:
|
|
717
|
+
1. lib/access.rb:15 a && (b || c)
|
|
718
|
+
unexecuted; DT 4/4 rules missing; MC/DC 3/3 conditions unproven
|
|
719
|
+
Cases to test: 4
|
|
720
|
+
R1 [F--]: a falsey; expected decision false
|
|
721
|
+
R2 [TT-]: a truthy, b truthy; expected decision true
|
|
722
|
+
R3 [TFT]: a truthy, b falsey, c truthy; expected decision true
|
|
723
|
+
R4 [TFF]: a truthy, b falsey, c falsey; expected decision false
|
|
724
|
+
Tests reaching this decision: none recorded
|
|
725
|
+
|
|
726
|
+
2. lib/access.rb:7 user.owner?(doc) || (user.editor? && !doc.locked?)
|
|
727
|
+
DT 3/4 rules missing; MC/DC 3/3 conditions unproven
|
|
728
|
+
Cases to test: 3
|
|
729
|
+
...
|
|
730
|
+
Tests reaching this decision:
|
|
731
|
+
AccessTest#test_owner (test/access_test.rb:19)
|
|
732
|
+
```
|
|
733
|
+
|
|
734
|
+
`Cases to test` counts the new observations the decision needs. When the
|
|
735
|
+
decision table was calculated, each missing rule is one case, and one
|
|
736
|
+
observation can cover at most one rule. When the table was not calculated,
|
|
737
|
+
or no rule is missing but a condition is still unproven (for example, when
|
|
738
|
+
its pair needs a statically impossible rule), the view lists the unproven
|
|
739
|
+
conditions instead. For decisions with
|
|
740
|
+
alternative coverage, such as `case`/`when` or `case`/`in`, it lists the
|
|
741
|
+
missing alternatives. A case describes truth values, not application inputs.
|
|
742
|
+
|
|
743
|
+
The level controls detail per row:
|
|
744
|
+
|
|
745
|
+
* Level 1 shows ranked rows and counts only.
|
|
746
|
+
* Level 2 adds the missing cases.
|
|
747
|
+
* Level 3 (the default) also names up to three tests that already reach the
|
|
748
|
+
decision, with RSpec rerun commands when recorded. These tests are good
|
|
749
|
+
places to add the new case.
|
|
750
|
+
|
|
751
|
+
`--top N` keeps the first `N` rows of each list and reports how many were
|
|
752
|
+
hidden. `--focus PATH[:LINE]` ranks only the matching decisions.
|
|
753
|
+
`--missing-only` hides fully covered files. Unsupported decisions are counted
|
|
754
|
+
but not ranked. As with the other views, the coverage ladder, policy gates,
|
|
755
|
+
diagnostics, and exit status are always global.
|
|
756
|
+
|
|
607
757
|
### Saved reports and offline comparison
|
|
608
758
|
|
|
609
759
|
Reports are saved only when requested. Create a local artifact directory and
|
|
@@ -615,6 +765,8 @@ bundle exec branchproof analyze 'lib/**/*.rb' --format json \
|
|
|
615
765
|
--output .branchproof/baseline.json
|
|
616
766
|
bundle exec branchproof report .branchproof/baseline.json --view conditions
|
|
617
767
|
bundle exec branchproof report .branchproof/baseline.json --view tests
|
|
768
|
+
bundle exec branchproof report .branchproof/baseline.json --format html \
|
|
769
|
+
--output .branchproof/coverage.html
|
|
618
770
|
```
|
|
619
771
|
|
|
620
772
|
The output's parent directory must already exist. Replacing a baseline is an
|
|
@@ -630,12 +782,28 @@ saved policy entries for that rendering without changing the snapshot file;
|
|
|
630
782
|
other saved thresholds remain inherited. Offline rendering never consults the
|
|
631
783
|
current `.branchproof.json`.
|
|
632
784
|
|
|
785
|
+
HTML is a single self-contained document with inline CSS, no JavaScript, external
|
|
786
|
+
assets, source reads, or network requests. It can be written to a file with
|
|
787
|
+
`--output` or streamed to stdout. The page keeps whole-run coverage and policy
|
|
788
|
+
global even when `--focus`, `--top`, or `--missing-only` filters displayed
|
|
789
|
+
decisions. Its status and exit code still reflect the saved run and its policy.
|
|
790
|
+
HTML is available from `report` and `collate`; `analyze` and `compare` reject it. The
|
|
791
|
+
integrated HTML layout does not accept `--view`. At level 1 it shows the summary
|
|
792
|
+
and decision inventory; level 2 adds missing scenarios; level 3 also adds
|
|
793
|
+
contributing tests. Legacy snapshots without analysis support only level 1,
|
|
794
|
+
and HTML reports failed or incomplete runs with their existing exit status.
|
|
795
|
+
|
|
633
796
|
New live JSON reports use schema `1.4` and include the required
|
|
634
797
|
`coverage_policy` object with the normalized requested minima and recomputed
|
|
635
798
|
gate results. The report validates those results from its exact coverage
|
|
636
|
-
counts rather than trusting a persisted percentage.
|
|
637
|
-
|
|
638
|
-
|
|
799
|
+
counts rather than trusting a persisted percentage. Reports created with
|
|
800
|
+
`--changed-since` use schema `1.5` and add the captured changed scope and its
|
|
801
|
+
informational coverage. A changed policy adds `changed_coverage_policy` and uses
|
|
802
|
+
schema `1.6`. Collated reports use schema `1.7` and record their contributing
|
|
803
|
+
shards and collection completeness. Readers accept schemas `1.0` through `1.7`.
|
|
804
|
+
A whole-run policy overlay preserves the input schema; adding a changed policy to a `1.5` snapshot
|
|
805
|
+
upgrades the output to `1.6` without changing the input file.
|
|
806
|
+
Policy overlays on collated reports preserve schema `1.7` and its provenance.
|
|
639
807
|
New snapshots retain the ladder at every level. Legacy snapshots without
|
|
640
808
|
analysis can still be rendered at Level 1; levels 2 and 3 require analysis in
|
|
641
809
|
the saved report.
|
|
@@ -644,6 +812,119 @@ policy when a project needs to retain reports.
|
|
|
644
812
|
Saved JSON includes existing raw metadata such as test names and expressions;
|
|
645
813
|
relative terminal labels do not mean every legacy JSON field is sanitized.
|
|
646
814
|
|
|
815
|
+
### Combine serial test shards
|
|
816
|
+
|
|
817
|
+
When CI splits tests into separate jobs, save a JSON report from each serial
|
|
818
|
+
run and combine their observations offline:
|
|
819
|
+
|
|
820
|
+
```sh
|
|
821
|
+
bundle exec branchproof collate unit.json integration.json \
|
|
822
|
+
--manifest shards.json --output combined.json
|
|
823
|
+
bundle exec branchproof report combined.json --format html --output coverage.html
|
|
824
|
+
```
|
|
825
|
+
|
|
826
|
+
Collation recalculates coverage, supporting sets, missing evidence, and gates.
|
|
827
|
+
For example, a true decision in one shard and a matching false decision in
|
|
828
|
+
another can together prove a condition's effect. Shard coverage percentages
|
|
829
|
+
are never averaged. No application code or tests run during collation, and the
|
|
830
|
+
original checkout does not need to exist.
|
|
831
|
+
|
|
832
|
+
Declare the expected artifacts in `shards.json`:
|
|
833
|
+
|
|
834
|
+
```json
|
|
835
|
+
{
|
|
836
|
+
"shards": [
|
|
837
|
+
{ "id": "unit", "report": "unit.json" },
|
|
838
|
+
{ "id": "integration", "report": "integration.json" }
|
|
839
|
+
]
|
|
840
|
+
}
|
|
841
|
+
```
|
|
842
|
+
|
|
843
|
+
Manifest paths are relative to the manifest file. Pass the artifacts actually
|
|
844
|
+
received as command arguments; omitted entries stay missing even if their
|
|
845
|
+
files happen to exist. Unknown inputs and duplicate manifest IDs or paths are
|
|
846
|
+
rejected. One artifact cannot satisfy two different shard IDs.
|
|
847
|
+
|
|
848
|
+
The manifest is optional. Without it, collation displays the supplied evidence
|
|
849
|
+
but marks collection completeness unknown and exits **2**. Missing declared
|
|
850
|
+
shards also produce an incomplete report and exit 2. A complete manifest proves
|
|
851
|
+
only that every declared artifact arrived; it does not prove those artifacts
|
|
852
|
+
selected every test in the application. Failed or incomplete input runs cannot
|
|
853
|
+
become a passing collection.
|
|
854
|
+
|
|
855
|
+
Inputs must be raw schema `1.4`–`1.6` reports from the current Branchproof and
|
|
856
|
+
Ruby runtime, with matching report schema, source inventory/digests, framework,
|
|
857
|
+
limits, reachability settings, analysis level, and coverage thresholds. Captured
|
|
858
|
+
changed scope and changed thresholds must also match. Different test selections,
|
|
859
|
+
seeds, and checkout roots are allowed; per-shard metadata and relative test
|
|
860
|
+
locations are retained. Missing reconstruction metadata is an error. Collated
|
|
861
|
+
reports cannot themselves be collated in this first version.
|
|
862
|
+
|
|
863
|
+
An identical report supplied twice is counted once. Conflicting or partially
|
|
864
|
+
overlapping run IDs are rejected. A test executed in different runs contributes
|
|
865
|
+
each execution to baseline counts and phase counts, while retaining its existing
|
|
866
|
+
test identity; any failure takes precedence over a later pass.
|
|
867
|
+
|
|
868
|
+
Collate defaults to JSON and supports `--format terminal|json|github|html`.
|
|
869
|
+
Use `report` for views, display filters, or threshold overrides on the saved
|
|
870
|
+
result. Output is written atomically and cannot replace input reports, declared
|
|
871
|
+
artifacts, or the manifest. Combined evidence exceeding storage limits is
|
|
872
|
+
rejected; bounded analysis that cannot finish remains incomplete. Schema `1.7`
|
|
873
|
+
requires an updated reader. Collation does not enable threaded or forked runners.
|
|
874
|
+
|
|
875
|
+
### Changed-decision coverage and gates
|
|
876
|
+
|
|
877
|
+
The analyze-only `--changed-since REF` option resolves `REF` to a commit and compares that
|
|
878
|
+
commit directly with the current tracked worktree, including staged and
|
|
879
|
+
unstaged edits. It does not infer a merge base. Untracked files are excluded.
|
|
880
|
+
The option adds informational coverage for current decisions mapped to changed
|
|
881
|
+
source lines; it does not select or skip tests. Additions and renames include
|
|
882
|
+
all current decisions in their files, deletions have no current decision
|
|
883
|
+
obligation, and comment or spacing changes with no semantic decision change
|
|
884
|
+
produce an empty scope. Invalid refs, non-Git projects, and observed source or
|
|
885
|
+
Git drift fail with usage status 2 before tests start. Capture checks for drift
|
|
886
|
+
but does not lock the checkout atomically.
|
|
887
|
+
|
|
888
|
+
The full inventory, analysis, run metrics, and `--minimum` gates remain
|
|
889
|
+
whole-run. A whole-run minimum can fail even when changed coverage is complete.
|
|
890
|
+
Add `--minimum-changed` to enforce an independent changed-decision threshold:
|
|
891
|
+
|
|
892
|
+
```sh
|
|
893
|
+
bundle exec branchproof analyze 'lib/**/*.rb' --changed-since main \
|
|
894
|
+
--minimum-changed mcdc=80
|
|
895
|
+
```
|
|
896
|
+
|
|
897
|
+
The changed gate uses exact counts for the captured changed decisions and the
|
|
898
|
+
same five criteria as `--minimum`. For example, 1 of 2 conditions proven means
|
|
899
|
+
50% MC/DC: an 80% gate fails even when every test passes. Add the missing
|
|
900
|
+
independence evidence and 2 of 2 conditions passes. Both policies must pass when
|
|
901
|
+
both are configured. A changed minimum requires `--changed-since` during
|
|
902
|
+
analysis; missing scope is rejected before tests run.
|
|
903
|
+
|
|
904
|
+
A valid empty changed scope is `not_applicable`, with no coverage percentage.
|
|
905
|
+
It does not fail the run or waive failed tests, incomplete evidence, or whole-run
|
|
906
|
+
gates. A nonempty scope with only unsupported decisions, or a requested criterion
|
|
907
|
+
with no applicable denominator, is unavailable. As with whole-run gates, a failed
|
|
908
|
+
test run makes a requested changed gate unavailable. Below-threshold gates exit
|
|
909
|
+
1; unavailable gates exit 2.
|
|
910
|
+
|
|
911
|
+
Terminal focus and top filters intersect the changed decision IDs for display;
|
|
912
|
+
they do not change either coverage denominator. JSON accepts the scope option,
|
|
913
|
+
while its existing focus and top restrictions remain. Saved reports retain the
|
|
914
|
+
captured scope for offline rendering without Git access. Their validation
|
|
915
|
+
checks the scope's internal consistency, not the historical Git diff or
|
|
916
|
+
cross-revision coverage.
|
|
917
|
+
|
|
918
|
+
```sh
|
|
919
|
+
bundle exec branchproof analyze 'lib/**/*.rb' --changed-since main
|
|
920
|
+
bundle exec branchproof report .branchproof/current.json --minimum-changed mcdc=80
|
|
921
|
+
```
|
|
922
|
+
|
|
923
|
+
Offline reports inherit saved changed thresholds and allow matching overrides.
|
|
924
|
+
They use the captured scope without consulting Git or project configuration.
|
|
925
|
+
Terminal, GitHub, and HTML output show changed gate results separately from
|
|
926
|
+
whole-run gates.
|
|
927
|
+
|
|
647
928
|
To compare two explicitly saved runs:
|
|
648
929
|
|
|
649
930
|
```sh
|
|
@@ -699,6 +980,71 @@ with `if: always()` so failed gates and failed test runs leave an artifact:
|
|
|
699
980
|
if-no-files-found: warn
|
|
700
981
|
```
|
|
701
982
|
|
|
983
|
+
To attach a portable HTML report as well, render it from the saved JSON and
|
|
984
|
+
upload both files. The report step is offline and does not run tests again:
|
|
985
|
+
|
|
986
|
+
```yaml
|
|
987
|
+
- name: Render Branchproof HTML
|
|
988
|
+
if: always() && hashFiles('.branchproof/coverage.json') != ''
|
|
989
|
+
run: >-
|
|
990
|
+
bundle exec branchproof report .branchproof/coverage.json --format html
|
|
991
|
+
--output .branchproof/coverage.html
|
|
992
|
+
|
|
993
|
+
- name: Upload Branchproof reports
|
|
994
|
+
if: always()
|
|
995
|
+
uses: actions/upload-artifact@v4
|
|
996
|
+
with:
|
|
997
|
+
name: branchproof-coverage
|
|
998
|
+
path: .branchproof/coverage.*
|
|
999
|
+
if-no-files-found: warn
|
|
1000
|
+
```
|
|
1001
|
+
|
|
1002
|
+
### GitHub Actions annotations and job summary
|
|
1003
|
+
|
|
1004
|
+
`--format github` prints GitHub Actions workflow commands instead of the
|
|
1005
|
+
terminal report. `analyze` and `report` accept it; `compare` does not.
|
|
1006
|
+
|
|
1007
|
+
* One `::warning` per decision with a gap, in [summary view](#summary-view-where-to-start)
|
|
1008
|
+
order, placed on the decision's file and line. At level 2 or 3 the message
|
|
1009
|
+
lists the cases to test.
|
|
1010
|
+
* An `::error` if the test run did not pass or is incomplete, and one per
|
|
1011
|
+
policy gate that did not pass (failed or unavailable, with its reason).
|
|
1012
|
+
Every non-zero exit status has at least one error.
|
|
1013
|
+
* A final `::notice` with the coverage ladder and the number of decisions with
|
|
1014
|
+
gaps.
|
|
1015
|
+
|
|
1016
|
+
When `GITHUB_STEP_SUMMARY` is set, as it is inside Actions, Branchproof also
|
|
1017
|
+
appends a Markdown summary to that file: the ladder, the policy gates, and a
|
|
1018
|
+
ranked "Where to start" table. The table shows `--top N` rows, or 20 rows
|
|
1019
|
+
when `--top` is not given.
|
|
1020
|
+
|
|
1021
|
+
Render both from the saved report, so the tests run only once:
|
|
1022
|
+
|
|
1023
|
+
```yaml
|
|
1024
|
+
- name: Branchproof coverage
|
|
1025
|
+
run: |
|
|
1026
|
+
mkdir -p .branchproof
|
|
1027
|
+
bundle exec branchproof analyze 'lib/**/*.rb' --test 'test/**/*_test.rb' \
|
|
1028
|
+
--format json --output .branchproof/coverage.json \
|
|
1029
|
+
--minimum mcdc=80 --minimum decision_table=75
|
|
1030
|
+
|
|
1031
|
+
- name: Branchproof annotations
|
|
1032
|
+
if: always() && hashFiles('.branchproof/coverage.json') != ''
|
|
1033
|
+
run: bundle exec branchproof report .branchproof/coverage.json --format github --top 10
|
|
1034
|
+
```
|
|
1035
|
+
|
|
1036
|
+
GitHub shows a limited number of annotations per step (currently 10
|
|
1037
|
+
warnings), so `--top 10` keeps the output to the ones that are displayed.
|
|
1038
|
+
Annotation paths must be relative to the repository. When the project is in
|
|
1039
|
+
a subdirectory (a step with `working-directory: gems/tool`), Branchproof
|
|
1040
|
+
prefixes each path with the current directory's location inside
|
|
1041
|
+
`GITHUB_WORKSPACE`. Run `report` from the same directory as `analyze`.
|
|
1042
|
+
|
|
1043
|
+
`--focus` also applies. `--view` and `--missing-only` are rejected because
|
|
1044
|
+
the output always uses the summary ranking of gaps. The exit status is the
|
|
1045
|
+
same as for the terminal format, so the report step inherits the saved
|
|
1046
|
+
policy and fails with the same gates.
|
|
1047
|
+
|
|
702
1048
|
MC/DC has two related questions. Evaluation asks whether a condition was
|
|
703
1049
|
observed with a value, including short-circuiting. Independent proof asks
|
|
704
1050
|
whether the analyzer found a pair of observations where that condition changes
|
|
@@ -809,8 +1155,8 @@ Ruby-defined custom `!` methods keep their runtime behavior;
|
|
|
809
1155
|
evidence that contradicts Boolean negation is rejected instead of proving
|
|
810
1156
|
coverage with an invalid logical model.
|
|
811
1157
|
|
|
812
|
-
New reports use schema `1.4`;
|
|
813
|
-
reports remain readable. Comparison distinguishes decision-table coverage movement
|
|
1158
|
+
New live reports use schema `1.4`; `--changed-since` reports use schema `1.5`.
|
|
1159
|
+
Saved schema `1.0` through `1.5` reports remain readable. Comparison distinguishes decision-table coverage movement
|
|
814
1160
|
(`rule coverage gained`, `rule coverage lost`) from analysis movement
|
|
815
1161
|
(`rule reachability changed`), and treats a structurally changed decision as a
|
|
816
1162
|
changed decision-table context instead of guessing which old rule a new rule
|
|
@@ -845,6 +1191,17 @@ the serial Minitest runner. This is useful for seeds and name filters:
|
|
|
845
1191
|
branchproof analyze 'lib/**/*.rb' --level 1 -- --seed 9001 -n /checkout/
|
|
846
1192
|
```
|
|
847
1193
|
|
|
1194
|
+
Minitest 5 and 6 both accept `-n`/`--name`. Minitest 6 also accepts
|
|
1195
|
+
`-i`/`--include`. Minitest 6 plugins must be loaded explicitly by the application;
|
|
1196
|
+
Branchproof preserves the framework's plugin-loading behavior. Bisect and server
|
|
1197
|
+
modes are unsupported because they do not execute one serial local test run.
|
|
1198
|
+
|
|
1199
|
+
Plain Ruby compatibility is tested with Minitest 5.26.2, 5.27.0, 6.0.0, and
|
|
1200
|
+
6.0.6. Rails integration continues to be tested with Minitest 5; Minitest 6
|
|
1201
|
+
support does not extend the verified Rails compatibility matrix.
|
|
1202
|
+
Minitest's own Ruby requirements also apply: Minitest 5.25.5 excludes Ruby 4,
|
|
1203
|
+
so it cannot be selected in Branchproof's Ruby 4 bundle.
|
|
1204
|
+
|
|
848
1205
|
The first release supports serial Minitest and RSpec execution in plain Ruby
|
|
849
1206
|
projects and Rails applications. Rails lazy and eager loading are supported
|
|
850
1207
|
when the application does not enable reloading for the test run. Transactions
|
|
@@ -854,6 +1211,54 @@ Rails system/browser tests, custom Rails test commands, generated tests, and
|
|
|
854
1211
|
reloading configurations are explicitly unsupported and produce diagnostics
|
|
855
1212
|
rather than a passing analysis.
|
|
856
1213
|
|
|
1214
|
+
### Rake task
|
|
1215
|
+
|
|
1216
|
+
`require "branchproof/rake_task"` in a Rakefile defines a task that runs
|
|
1217
|
+
`branchproof analyze` in a subprocess. The gem does not depend on Rake at
|
|
1218
|
+
runtime, so the file is not loaded by `require "branchproof"`. Only options
|
|
1219
|
+
set in the Rakefile are passed to the CLI; everything else keeps the CLI and
|
|
1220
|
+
`.branchproof.json` defaults.
|
|
1221
|
+
|
|
1222
|
+
A Minitest project:
|
|
1223
|
+
|
|
1224
|
+
```ruby
|
|
1225
|
+
require "branchproof/rake_task"
|
|
1226
|
+
|
|
1227
|
+
Branchproof::RakeTask.new(:branchproof) do |t|
|
|
1228
|
+
t.sources = ["lib/**/*.rb"]
|
|
1229
|
+
t.tests = ["test/**/*_test.rb"]
|
|
1230
|
+
end
|
|
1231
|
+
```
|
|
1232
|
+
|
|
1233
|
+
An RSpec project in a Rails application:
|
|
1234
|
+
|
|
1235
|
+
```ruby
|
|
1236
|
+
Branchproof::RakeTask.new(:branchproof) do |t|
|
|
1237
|
+
t.sources = ["app/**/*.rb"]
|
|
1238
|
+
t.project = "rails"
|
|
1239
|
+
t.framework = "rspec"
|
|
1240
|
+
t.runner_args = ["--seed", "1234"]
|
|
1241
|
+
end
|
|
1242
|
+
```
|
|
1243
|
+
|
|
1244
|
+
Then run `bundle exec rake branchproof`. The task accepts `sources`, `tests`,
|
|
1245
|
+
`project`, `framework`, `view`, `level`, `missing_only`, `minimum`, `format`,
|
|
1246
|
+
`output`, `runner_args`, and `description`. `RakeTask#argv` returns the CLI
|
|
1247
|
+
arguments the task will pass, which is useful for checking a Rakefile without
|
|
1248
|
+
running the analysis.
|
|
1249
|
+
|
|
1250
|
+
In CI, the task exits with the CLI status, so a failed test run, a coverage
|
|
1251
|
+
gate failure, or a usage error fails the job:
|
|
1252
|
+
|
|
1253
|
+
```ruby
|
|
1254
|
+
Branchproof::RakeTask.new(:branchproof) do |t|
|
|
1255
|
+
t.sources = ["lib/**/*.rb"]
|
|
1256
|
+
t.format = "json"
|
|
1257
|
+
t.output = ".branchproof/coverage.json"
|
|
1258
|
+
t.minimum = ["mcdc=80", "decision_table=75"]
|
|
1259
|
+
end
|
|
1260
|
+
```
|
|
1261
|
+
|
|
857
1262
|
## Library entry points
|
|
858
1263
|
|
|
859
1264
|
```ruby
|
|
@@ -899,9 +1304,15 @@ the core suite does not require Rails. Bootsnap is disabled for the child
|
|
|
899
1304
|
Rails process so its compilation cache cannot own the load path during an
|
|
900
1305
|
analysis.
|
|
901
1306
|
|
|
902
|
-
Run the commands with
|
|
903
|
-
|
|
904
|
-
|
|
1307
|
+
Run the commands with CRuby 4.0 or newer and the declared Prism 1.x dependency.
|
|
1308
|
+
The default development bundle keeps Minitest 5 for the full suite. CI runs eight
|
|
1309
|
+
jobs on CRuby 4.0: the core suite, Minitest 5.26.2/6.0.0/6.0.6 consumer
|
|
1310
|
+
compatibility, RSpec Core 3.13.0/3.13.6 compatibility, Rails 8.1 integration, and
|
|
1311
|
+
rspec-rails integration. The Minitest compatibility jobs select an exact version
|
|
1312
|
+
with `BRANCHPROOF_MINITEST_VERSION` and run adapter, CLI, lifecycle, doctor,
|
|
1313
|
+
collation, and packaged-consumer checks. They do not require the mock library removed from
|
|
1314
|
+
Minitest 6.
|
|
1315
|
+
Lint runs once in the core job. Pull requests and pushes to main trigger CI.
|
|
905
1316
|
|
|
906
1317
|
The repeatable native-versus-instrumented adapter benchmark and its captured
|
|
907
1318
|
Ruby 3.4.7 result are in
|
data/doc/Branchproof/CLI.md
CHANGED
|
@@ -15,5 +15,5 @@ Not documented.
|
|
|
15
15
|
### `call(argv)` <a id="method-i-call"></a> <a id="call-instance_method"></a>
|
|
16
16
|
Not documented.
|
|
17
17
|
|
|
18
|
-
### `initialize(stdout:, stderr:)` <a id="method-i-initialize"></a> <a id="initialize-instance_method"></a>
|
|
18
|
+
### `initialize(stdout:, stderr:, env: = ENV)` <a id="method-i-initialize"></a> <a id="initialize-instance_method"></a>
|
|
19
19
|
- **@return** [CLI] a new instance of CLI
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
# Module Branchproof::ChangedCoverage <a id="module-Branchproof-ChangedCoverage"></a>
|
|
2
|
+
|
|
3
|
+
| | |
|
|
4
|
+
| --- | --- |
|
|
5
|
+
| **Defined in** | lib/branchproof/changed_coverage.rb |
|
|
6
|
+
|
|
7
|
+
Computes informational coverage for the current decisions captured by a
|
|
8
|
+
changed scope.
|
|
9
|
+
|
|
10
|
+
## Constants
|
|
11
|
+
### `COMPLETENESS_FIELDS` <a id="constant-COMPLETENESS_FIELDS"></a> <a id="COMPLETENESS_FIELDS-constant"></a>
|
|
12
|
+
Not documented.
|
|
13
|
+
|
|
14
|
+
### `CRITERIA` <a id="constant-CRITERIA"></a> <a id="CRITERIA-constant"></a>
|
|
15
|
+
Not documented.
|
|
16
|
+
|
|
17
|
+
## Public Class Methods
|
|
18
|
+
### `call(document:, scope:)` <a id="method-c-call"></a> <a id="call-class_method"></a>
|
|
19
|
+
Not documented.
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# Class Branchproof::ChangedCoveragePolicy <a id="class-Branchproof-ChangedCoveragePolicy"></a>
|
|
2
|
+
|
|
3
|
+
| | |
|
|
4
|
+
| --- | --- |
|
|
5
|
+
| **Inherits** | Object |
|
|
6
|
+
| **Defined in** | lib/branchproof/coverage_policy.rb |
|
|
7
|
+
|
|
8
|
+
Evaluates gates against the captured changed-scope summary independently from
|
|
9
|
+
whole-run coverage policy.
|
|
10
|
+
|
|
11
|
+
## Constants
|
|
12
|
+
### `COMPLETENESS_FIELDS` <a id="constant-COMPLETENESS_FIELDS"></a> <a id="COMPLETENESS_FIELDS-constant"></a>
|
|
13
|
+
Not documented.
|
|
14
|
+
|
|
15
|
+
### `CRITERIA` <a id="constant-CRITERIA"></a> <a id="CRITERIA-constant"></a>
|
|
16
|
+
Not documented.
|
|
17
|
+
|
|
18
|
+
## Public Instance Methods
|
|
19
|
+
### `call(document:, scope:)` <a id="method-i-call"></a> <a id="call-instance_method"></a>
|
|
20
|
+
Not documented.
|
|
21
|
+
|
|
22
|
+
### `initialize(minimum_changed:)` <a id="method-i-initialize"></a> <a id="initialize-instance_method"></a>
|
|
23
|
+
- **@raise** [ArgumentError]
|
|
24
|
+
- **@return** [ChangedCoveragePolicy] a new instance of ChangedCoveragePolicy
|