branchproof 0.5.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +20 -0
- data/README.md +129 -31
- data/doc/Branchproof/CoverageIndex.md +3 -0
- data/doc/Branchproof/DecisionSyntax.md +20 -0
- data/doc/Branchproof/FlowInstrumentation.md +7 -0
- data/doc/Branchproof/Instrumenter.md +1 -0
- data/doc/Branchproof/Report.md +10 -0
- data/doc/Branchproof/Runtime.md +19 -0
- data/doc/Branchproof/RuntimeFlow.md +27 -0
- data/doc/Branchproof/SavedReport.md +6 -0
- data/doc/Branchproof/Source.md +1 -0
- data/doc/Branchproof.md +6 -2
- data/doc/CHANGELOG.md +20 -0
- data/doc/README.md +129 -31
- data/lib/branchproof/analyzer.rb +273 -13
- data/lib/branchproof/cli.rb +2 -2
- data/lib/branchproof/comparison.rb +1 -1
- data/lib/branchproof/coverage_index.rb +90 -2
- data/lib/branchproof/decision_syntax.rb +294 -0
- data/lib/branchproof/evidence.rb +54 -9
- data/lib/branchproof/flow_instrumentation.rb +107 -0
- data/lib/branchproof/focused_report.rb +102 -21
- data/lib/branchproof/instrumenter.rb +28 -13
- data/lib/branchproof/report.rb +348 -31
- data/lib/branchproof/runtime.rb +3 -0
- data/lib/branchproof/runtime_flow.rb +58 -0
- data/lib/branchproof/saved_report.rb +245 -6
- data/lib/branchproof/source.rb +191 -35
- data/lib/branchproof/version.rb +1 -1
- data/llms.txt +6 -2
- data/sig/branchproof.rbs +7 -0
- metadata +7 -1
data/doc/README.md
CHANGED
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
# Branchproof
|
|
2
2
|
|
|
3
|
-
Branchproof measures modified condition/decision
|
|
4
|
-
serial Minitest run. It
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
3
|
+
Branchproof measures decision, condition, and modified condition/decision
|
|
4
|
+
coverage (MC/DC) from one serial Minitest run. It discovers Ruby decisions
|
|
5
|
+
through Prism, records their runtime paths, and attributes evidence to tests.
|
|
6
|
+
Boolean decisions receive the coverage ladder; `case`, pattern alternatives,
|
|
7
|
+
safe navigation, and conditional assignments receive alternative coverage.
|
|
8
8
|
|
|
9
9
|
The gem and primary command are named `branchproof`. The `mcdc` command and
|
|
10
10
|
`MCDC` namespace remain compatibility aliases with the same behavior.
|
|
@@ -126,6 +126,50 @@ Additional tests outside this MC/DC evidence set may improve coverage.
|
|
|
126
126
|
The test execution still runs once for the selected level. Use `--format
|
|
127
127
|
json` when a consumer needs the complete identifiers and versioned schema.
|
|
128
128
|
|
|
129
|
+
### Coverage ladder
|
|
130
|
+
|
|
131
|
+
Each successful `analyze` run calculates four criteria from the same completed
|
|
132
|
+
observations. The report shows their status for each supported decision and
|
|
133
|
+
aggregate counts with explicit denominators:
|
|
134
|
+
|
|
135
|
+
| Criterion | Requirement | Aggregate denominator |
|
|
136
|
+
| --- | --- | --- |
|
|
137
|
+
| Decision | The decision produced both true and false | Supported decisions |
|
|
138
|
+
| Condition | Every atomic condition evaluated both true and false | Two required truth values per supported condition |
|
|
139
|
+
| Condition/Decision | Both Decision and Condition Coverage hold for a decision | Supported decisions |
|
|
140
|
+
| MC/DC | Every condition has an independence witness pair | Supported conditions |
|
|
141
|
+
|
|
142
|
+
For `logged_in? && admin?`, observations `[F-] => F` and `[TT] => T` give:
|
|
143
|
+
|
|
144
|
+
```text
|
|
145
|
+
Decision PASS
|
|
146
|
+
Condition FAIL (3/4 values observed)
|
|
147
|
+
Condition/Decision FAIL
|
|
148
|
+
MC/DC FAIL (1/2 conditions proven)
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
The skipped `admin?` in `[F-]` counts as neither true nor false. Conditions
|
|
152
|
+
that did evaluate count toward Condition Coverage even when their value was
|
|
153
|
+
masked by another condition. Here, the two observations prove independence
|
|
154
|
+
for `logged_in?`; `admin?` still needs `[TF] => F`.
|
|
155
|
+
|
|
156
|
+
Decision and Condition Coverage are calculated independently from the captured
|
|
157
|
+
evidence. Condition/Decision requires both; MC/DC adds independence evidence.
|
|
158
|
+
Unsupported decisions are excluded from every denominator, while
|
|
159
|
+
unexecuted supported decisions remain in scope. Empty denominators are N/A.
|
|
160
|
+
|
|
161
|
+
Reports retain the observations and owning tests for each decision outcome
|
|
162
|
+
and condition value. Missing values describe required runtime observations,
|
|
163
|
+
not application inputs or a guarantee that the path is reachable. Unattributed
|
|
164
|
+
observations can provide truth-value evidence without identifying a test;
|
|
165
|
+
the report's completeness and diagnostics still apply.
|
|
166
|
+
|
|
167
|
+
JSON stores aggregate counts under `analysis.coverage`, per-decision results
|
|
168
|
+
under `analysis.decisions[].coverage`, and condition value evidence under
|
|
169
|
+
`analysis.decisions[].condition_results[].coverage`. Existing MC/DC witness
|
|
170
|
+
pairs, counterpart constraints, and raw vectors remain available. Statuses
|
|
171
|
+
distinguish `covered`, `partial`, `unexecuted`, and `unsupported` results.
|
|
172
|
+
|
|
129
173
|
### Find missing cases
|
|
130
174
|
|
|
131
175
|
Use `--missing-only` to focus the terminal report on conditions that still
|
|
@@ -175,15 +219,17 @@ test inputs; they do not by themselves prove a bug in the application.
|
|
|
175
219
|
|
|
176
220
|
The levels select how much of the one-run result is displayed:
|
|
177
221
|
|
|
178
|
-
* Level 1 reports observed vectors grouped by their
|
|
222
|
+
* Level 1 reports the coverage ladder and observed vectors grouped by their
|
|
223
|
+
raw decision outcomes.
|
|
179
224
|
* Level 2 includes the smallest supporting sets found for vectors and tests.
|
|
180
225
|
* Level 3 (the default) includes condition independence witnesses and
|
|
181
226
|
counterpart constraints, as well as the Level 1 and 2 evidence.
|
|
182
227
|
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
228
|
+
All levels calculate the same criteria from one captured run. The level
|
|
229
|
+
controls displayed evidence, not the coverage criterion; JSON retains the
|
|
230
|
+
analysis even at Level 1. A failed, unsupported, or incomplete run is reported
|
|
231
|
+
with its status and diagnostics and cannot become a successful coverage
|
|
232
|
+
result by changing the display level.
|
|
187
233
|
|
|
188
234
|
### Focused condition and test views
|
|
189
235
|
|
|
@@ -232,7 +278,8 @@ artifacts when you need to retain multiple runs.
|
|
|
232
278
|
The `report` command reads the saved document without loading the application
|
|
233
279
|
or running tests. It uses locations and metadata captured in the report, so
|
|
234
280
|
rendering remains useful after the original checkout has moved or been
|
|
235
|
-
removed.
|
|
281
|
+
removed. New snapshots retain the ladder at every level. Legacy snapshots
|
|
282
|
+
without analysis can still be rendered at Level 1; levels 2 and 3 require
|
|
236
283
|
analysis in the saved report. The repository ignores `.branchproof/`; choose a
|
|
237
284
|
different path and CI artifact policy when a project needs to retain reports.
|
|
238
285
|
Saved JSON includes existing raw metadata such as test names and expressions;
|
|
@@ -274,26 +321,77 @@ short-circuited or masked rather than fixed to the same observed values. For exa
|
|
|
274
321
|
short-circuits it once, but does not prove `right`; `[TF]` is also required.
|
|
275
322
|
The condition and test views preserve that distinction.
|
|
276
323
|
|
|
277
|
-
### Supported
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
324
|
+
### Supported decision forms
|
|
325
|
+
|
|
326
|
+
Every decision has a stable `kind` and `context` in JSON. The Boolean ladder
|
|
327
|
+
applies to the following forms:
|
|
328
|
+
|
|
329
|
+
| Construct | Kind | Context |
|
|
330
|
+
| --- | --- | --- |
|
|
331
|
+
| `if`, modifier `if`, `elsif`, `unless`, ternary | `boolean` | `if`, `elsif`, `unless`, `ternary` |
|
|
332
|
+
| `while`, `until`, including modifier and post-test loops | `boolean` | `while`, `until` |
|
|
333
|
+
| Each subjectless `case` candidate | `boolean` | `case_when` |
|
|
334
|
+
| Standalone `value in pattern` | `boolean` | `pattern_in` |
|
|
335
|
+
| Evaluated pattern guard predicate | `boolean` | `pattern_guard` |
|
|
336
|
+
| Standalone `&&`, `||`, `and`, `or` | `boolean` | `short_circuit` |
|
|
337
|
+
|
|
338
|
+
Prism determines precedence. `!` and `not` appear as NOT nodes in the Boolean
|
|
339
|
+
tree; their operands remain the conditions. Short-circuited operands remain
|
|
340
|
+
not evaluated. A Boolean subtree already decomposed in a decision is not
|
|
341
|
+
inventoried again as a standalone decision.
|
|
342
|
+
|
|
343
|
+
Loop outcomes describe the predicate as written: an `until` predicate that
|
|
344
|
+
returns true ends the loop. Every predicate evaluation receives an execution
|
|
345
|
+
ID. Repeated equivalent executions aggregate into a vector's `count`, retaining
|
|
346
|
+
the supporting tests. Ternary outcomes likewise describe the predicate, not
|
|
347
|
+
the value returned by the chosen branch.
|
|
348
|
+
|
|
349
|
+
Other constructs use alternative coverage, separate from MC/DC:
|
|
350
|
+
|
|
351
|
+
| Construct | Kind | Context | Required alternatives |
|
|
352
|
+
| --- | --- | --- | --- |
|
|
353
|
+
| `case subject` | `multiway` | `case` | Each `when` candidate and `else` (or implicit no-match path) |
|
|
354
|
+
| `case/in` | `pattern` | `case_in` | Each pattern clause and explicit `else`, if present |
|
|
355
|
+
| `receiver&.method` | `implicit` | `safe_navigation` | Receiver nil / non-nil |
|
|
356
|
+
| `lhs ||= rhs` | `implicit` | `or_assignment` | RHS skipped / executed |
|
|
357
|
+
| `lhs &&= rhs` | `implicit` | `and_assignment` | RHS skipped / executed |
|
|
358
|
+
|
|
359
|
+
Each safe-navigation operation in a chain is a distinct decision. Assignment
|
|
360
|
+
instrumentation preserves Ruby's native local, instance, class, global,
|
|
361
|
+
constant, method, and indexed assignment operations, including receiver and
|
|
362
|
+
index evaluation order. Safe navigation distinguishes nil from false.
|
|
363
|
+
|
|
364
|
+
For multiway decisions, vector values mean selected (`true`), evaluated but
|
|
365
|
+
not selected (`false`), and skipped (`null`). Later alternatives remain skipped
|
|
366
|
+
when an earlier candidate matches. Implicit vectors record the selected path
|
|
367
|
+
and its unselected complement. Their `outcome` is a selection marker, not the
|
|
368
|
+
truthiness of the application's return value. Reports label these as paths,
|
|
369
|
+
not Boolean outcomes. Each alternative exposes selected, not-selected, and
|
|
370
|
+
skipped evidence with test and vector IDs. The alternative denominator is the
|
|
371
|
+
number of supported selectable alternatives; these decisions do not enter
|
|
372
|
+
Boolean-ladder or MC/DC denominators.
|
|
373
|
+
|
|
374
|
+
A `case/in` without `else` retains Ruby's native no-match exception. An execution
|
|
375
|
+
that fails before choosing a branch is aborted, not counted as a selected
|
|
376
|
+
alternative. Selected branches and assignment paths remain observed even when
|
|
377
|
+
their bodies or right-hand sides subsequently raise or return.
|
|
378
|
+
|
|
379
|
+
Unsupported syntax stays visible and outside coverage denominators. Current
|
|
380
|
+
exclusions include guarded `case/in` (`unsupported_pattern_guard`), dynamic
|
|
381
|
+
`when` splats (`unsupported_case_splat`), safe-navigation compound assignment
|
|
382
|
+
(`unsupported_assignment_target`), and rescue alternatives
|
|
383
|
+
(`unsupported_rescue_control_flow`). A guard predicate can still supply Boolean
|
|
384
|
+
evidence when Ruby evaluates it; an unsupported guarded case does not claim
|
|
385
|
+
pattern-match coverage from that evidence. Flip-flops remain
|
|
386
|
+
`unsupported_flip_flop`. Decisions inside `defined?`, contextual regular
|
|
387
|
+
expressions, heredocs, unsafe predicates, and limit overflows retain explicit
|
|
388
|
+
exclusions. Ruby-defined custom `!` methods keep their runtime behavior;
|
|
389
|
+
evidence that contradicts Boolean negation is rejected instead of proving
|
|
390
|
+
coverage with an invalid logical model.
|
|
391
|
+
|
|
392
|
+
New reports use schema `1.2`; saved schema `1.0` and `1.1` reports remain
|
|
393
|
+
readable. Expanded discovery changes coverage denominators, so compare reports
|
|
394
|
+
with their supported syntax scope in mind.
|
|
297
395
|
|
|
298
396
|
### Limits
|
|
299
397
|
|
data/lib/branchproof/analyzer.rb
CHANGED
|
@@ -30,11 +30,13 @@ module Branchproof
|
|
|
30
30
|
proven_count: proven,
|
|
31
31
|
eligible_count: eligible,
|
|
32
32
|
completeness: completeness(decisions),
|
|
33
|
-
diagnostics: decisions.flat_map { |decision| decision[:diagnostics] } + @invalid_diagnostics
|
|
33
|
+
diagnostics: decisions.flat_map { |decision| decision[:diagnostics] } + @invalid_diagnostics,
|
|
34
|
+
coverage: aggregate_coverage(decisions)
|
|
34
35
|
}.freeze
|
|
35
36
|
end
|
|
36
37
|
|
|
37
38
|
def pair?(decision_id:, condition_index:, left:, right:)
|
|
39
|
+
return false if alternative_decision_for_id?(decision_id)
|
|
38
40
|
return false unless compatible_vectors?(left, right, decision_id)
|
|
39
41
|
return false unless observed?(left, condition_index) && observed?(right, condition_index)
|
|
40
42
|
return false if value(left, condition_index) == value(right, condition_index)
|
|
@@ -51,6 +53,8 @@ module Branchproof
|
|
|
51
53
|
decision = records(@inventory, :decisions).find { id(_1, :id) == decision_id }
|
|
52
54
|
return @missing_cache[cache_key] = nil unless decision
|
|
53
55
|
|
|
56
|
+
return @missing_cache[cache_key] = nil if alternative_decision?(decision)
|
|
57
|
+
|
|
54
58
|
support_status = id(decision, :support_status).to_s
|
|
55
59
|
return @missing_cache[cache_key] = nil unless support_status.empty? || support_status.upcase == "SUPPORTED"
|
|
56
60
|
|
|
@@ -93,11 +97,13 @@ module Branchproof
|
|
|
93
97
|
private
|
|
94
98
|
|
|
95
99
|
def analyze_decision(decision)
|
|
100
|
+
return analyze_alternative_decision(decision) if alternative_decision?(decision)
|
|
101
|
+
|
|
96
102
|
decision_id = id(decision, :id)
|
|
97
103
|
unless id(decision, :support_status).to_s.empty? || id(decision, :support_status).to_s.upcase == "SUPPORTED"
|
|
98
104
|
return {
|
|
99
105
|
decision_id: decision_id, effective_masks_by_vector: {}, condition_results: [], witness_buckets: {},
|
|
100
|
-
conditions: [], unsupported: true,
|
|
106
|
+
conditions: [], unsupported: true, coverage: unsupported_coverage,
|
|
101
107
|
completeness: { observation: true, attribution: true, analysis: true },
|
|
102
108
|
diagnostics: [diagnostic("unsupported_decision", "warning", decision_id, nil)]
|
|
103
109
|
}
|
|
@@ -136,9 +142,13 @@ module Branchproof
|
|
|
136
142
|
nil
|
|
137
143
|
else
|
|
138
144
|
(true_ids.empty? || false_ids.empty? ? "missing_effective_sign" : "no_independent_pair")
|
|
139
|
-
end
|
|
145
|
+
end,
|
|
146
|
+
coverage: condition_coverage(index, vectors)
|
|
140
147
|
}
|
|
141
148
|
end
|
|
149
|
+
decision_coverage = decision_coverage(vectors)
|
|
150
|
+
condition_coverage_summary = condition_summary(results)
|
|
151
|
+
mcdc = mcdc_coverage(results)
|
|
142
152
|
{
|
|
143
153
|
decision_id: decision_id,
|
|
144
154
|
effective_masks_by_vector: masks,
|
|
@@ -146,11 +156,211 @@ module Branchproof
|
|
|
146
156
|
witness_buckets: buckets,
|
|
147
157
|
edge_table: edge_table(decision[:tree]),
|
|
148
158
|
conditions: conditions(decision),
|
|
159
|
+
coverage: { decision: decision_coverage, condition: condition_coverage_summary,
|
|
160
|
+
condition_decision: conjunction_coverage(decision_coverage, condition_coverage_summary),
|
|
161
|
+
mcdc: mcdc },
|
|
149
162
|
completeness: { observation: true, attribution: true, analysis: !@analysis_invalid },
|
|
150
163
|
diagnostics: []
|
|
151
164
|
}
|
|
152
165
|
end
|
|
153
166
|
|
|
167
|
+
def analyze_alternative_decision(decision)
|
|
168
|
+
decision_id = id(decision, :id)
|
|
169
|
+
unless id(decision, :support_status).to_s.empty? || id(decision, :support_status).to_s.upcase == "SUPPORTED"
|
|
170
|
+
return {
|
|
171
|
+
decision_id: decision_id, kind: id(decision, :kind), effective_masks_by_vector: {},
|
|
172
|
+
condition_results: [], witness_buckets: {},
|
|
173
|
+
conditions: [], alternatives: alternatives(decision), unsupported: true,
|
|
174
|
+
coverage: unsupported_alternative_coverage,
|
|
175
|
+
completeness: { observation: true, attribution: true, analysis: true },
|
|
176
|
+
diagnostics: [diagnostic("unsupported_decision", "warning", decision_id, nil)]
|
|
177
|
+
}
|
|
178
|
+
end
|
|
179
|
+
|
|
180
|
+
vectors = @vectors.filter_map { |vector| valid_vector(vector, decision) }
|
|
181
|
+
rows = alternatives(decision).map do |alternative|
|
|
182
|
+
index = id(alternative, :index).to_i
|
|
183
|
+
selected = vectors.select { |vector| value(vector, index) == true }
|
|
184
|
+
not_selected = vectors.select { |vector| value(vector, index) == false }
|
|
185
|
+
skipped = vectors.select { |vector| value(vector, index).nil? }
|
|
186
|
+
{
|
|
187
|
+
alternative_id: id(alternative, :id), index: id(alternative, :index),
|
|
188
|
+
expression: id(alternative, :expression),
|
|
189
|
+
selected: alternative_evidence_bucket(selected),
|
|
190
|
+
not_selected: alternative_evidence_bucket(not_selected),
|
|
191
|
+
skipped: alternative_evidence_bucket(skipped)
|
|
192
|
+
}
|
|
193
|
+
end
|
|
194
|
+
missing = rows.filter_map { |row| row[:alternative_id] unless row[:selected][:observed] }
|
|
195
|
+
covered = rows.count { |row| row[:selected][:observed] }
|
|
196
|
+
{
|
|
197
|
+
decision_id: decision_id, kind: id(decision, :kind), effective_masks_by_vector: {},
|
|
198
|
+
condition_results: [], witness_buckets: {},
|
|
199
|
+
conditions: [], alternatives: alternatives(decision), unsupported: false,
|
|
200
|
+
coverage: { alternative: { status: coverage_status(covered, rows.length), covered_alternatives: covered,
|
|
201
|
+
required_alternatives: rows.length, alternatives: rows,
|
|
202
|
+
missing_alternatives: missing },
|
|
203
|
+
mcdc: { status: "not_applicable" } },
|
|
204
|
+
completeness: { observation: true, attribution: true, analysis: !@analysis_invalid }, diagnostics: []
|
|
205
|
+
}
|
|
206
|
+
end
|
|
207
|
+
|
|
208
|
+
def unsupported_alternative_coverage
|
|
209
|
+
{ alternative: { status: "unsupported", covered_alternatives: 0, required_alternatives: 0,
|
|
210
|
+
alternatives: [], missing_alternatives: [] }, mcdc: { status: "unsupported" } }
|
|
211
|
+
end
|
|
212
|
+
|
|
213
|
+
def alternative_evidence_bucket(vectors)
|
|
214
|
+
evidence = vectors.map { |vector| provenance(vector) }
|
|
215
|
+
{ observed: !vectors.empty?, vector_ids: evidence.map { |item| item[:vector_id] }.uniq.sort,
|
|
216
|
+
test_ids: evidence.flat_map { |item| item[:test_ids] }.uniq.sort,
|
|
217
|
+
unattributed_count: evidence.sum { |item| item[:unattributed_count] } }
|
|
218
|
+
end
|
|
219
|
+
|
|
220
|
+
def alternative_decision?(decision)
|
|
221
|
+
kind = id(decision, :kind).to_s
|
|
222
|
+
!kind.empty? && kind != "boolean"
|
|
223
|
+
end
|
|
224
|
+
|
|
225
|
+
def alternative_decision_for_id?(decision_id)
|
|
226
|
+
decision = records(@inventory, :decisions).find { |item| id(item, :id) == decision_id }
|
|
227
|
+
decision && alternative_decision?(decision)
|
|
228
|
+
end
|
|
229
|
+
|
|
230
|
+
def alternatives(decision)
|
|
231
|
+
records(decision, :alternatives)
|
|
232
|
+
end
|
|
233
|
+
|
|
234
|
+
def unsupported_coverage
|
|
235
|
+
{ decision: { status: "unsupported", true_observed: false, false_observed: false,
|
|
236
|
+
covered_outcomes: 0, required_outcomes: 2, outcomes: [], missing_outcomes: [] },
|
|
237
|
+
condition: { status: "unsupported", covered_values: 0, required_values: 0,
|
|
238
|
+
covered_conditions: 0, condition_count: 0 },
|
|
239
|
+
condition_decision: { status: "unsupported" },
|
|
240
|
+
mcdc: { status: "unsupported", proven_conditions: 0, condition_count: 0 } }
|
|
241
|
+
end
|
|
242
|
+
|
|
243
|
+
def provenance(vector)
|
|
244
|
+
{ vector_id: id(vector, :id).to_s,
|
|
245
|
+
test_ids: Array(id(vector, :test_ids)).map(&:to_s).uniq.sort,
|
|
246
|
+
unattributed_count: id(vector, :unattributed_count).to_i }
|
|
247
|
+
end
|
|
248
|
+
|
|
249
|
+
def evidence_bucket(value, vectors)
|
|
250
|
+
matching = vectors.select do |vector|
|
|
251
|
+
observed?(vector, value[:index]) && self.value(vector, value[:index]) == value[:value]
|
|
252
|
+
end
|
|
253
|
+
evidence = matching.map { |vector| provenance(vector) }
|
|
254
|
+
{ value: value[:value], observed: !matching.empty?,
|
|
255
|
+
vector_ids: evidence.flat_map { |item| item[:vector_id] }.uniq.sort,
|
|
256
|
+
test_ids: evidence.flat_map { |item| item[:test_ids] }.uniq.sort,
|
|
257
|
+
unattributed_count: evidence.sum { |item| item[:unattributed_count] } }
|
|
258
|
+
end
|
|
259
|
+
|
|
260
|
+
def condition_coverage(index, vectors)
|
|
261
|
+
observed_vectors = vectors.select { |vector| observed?(vector, index) }
|
|
262
|
+
values = [true, false].map do |item|
|
|
263
|
+
evidence_bucket({ index: index, value: item }, observed_vectors)
|
|
264
|
+
end
|
|
265
|
+
covered = values.count { |entry| entry[:observed] }
|
|
266
|
+
{ status: coverage_status(covered, 2), true_observed: values[0][:observed],
|
|
267
|
+
false_observed: values[1][:observed], covered_values: covered, required_values: 2,
|
|
268
|
+
values: values, missing_values: values.reject { |entry| entry[:observed] }.map { |entry| entry[:value] } }
|
|
269
|
+
end
|
|
270
|
+
|
|
271
|
+
def decision_coverage(vectors)
|
|
272
|
+
outcomes = [false, true].map do |value|
|
|
273
|
+
matching = vectors.select { |vector| outcome(vector) == value }
|
|
274
|
+
evidence = matching.map { |vector| provenance(vector) }
|
|
275
|
+
{ value: value, observed: !matching.empty?,
|
|
276
|
+
vector_ids: evidence.flat_map { |item| item[:vector_id] }.uniq.sort,
|
|
277
|
+
test_ids: evidence.flat_map { |item| item[:test_ids] }.uniq.sort,
|
|
278
|
+
unattributed_count: evidence.sum { |item| item[:unattributed_count] } }
|
|
279
|
+
end
|
|
280
|
+
covered = outcomes.count { |entry| entry[:observed] }
|
|
281
|
+
{ status: coverage_status(covered, 2), true_observed: outcomes[1][:observed],
|
|
282
|
+
false_observed: outcomes[0][:observed], covered_outcomes: covered, required_outcomes: 2,
|
|
283
|
+
outcomes: outcomes,
|
|
284
|
+
missing_outcomes: outcomes.reject { |entry| entry[:observed] }.map { |entry| entry[:value] } }
|
|
285
|
+
end
|
|
286
|
+
|
|
287
|
+
def condition_summary(results)
|
|
288
|
+
values = results.sum { |result| result[:coverage][:covered_values] }
|
|
289
|
+
count = results.length
|
|
290
|
+
{ status: coverage_status(values, count * 2), covered_values: values,
|
|
291
|
+
required_values: count * 2,
|
|
292
|
+
covered_conditions: results.count { |result| result[:coverage][:status] == "covered" },
|
|
293
|
+
condition_count: count }
|
|
294
|
+
end
|
|
295
|
+
|
|
296
|
+
def mcdc_coverage(results)
|
|
297
|
+
proven = results.count { |result| result[:status] == "PROVEN" }
|
|
298
|
+
observed = results.any? { |result| result[:coverage][:covered_values].positive? }
|
|
299
|
+
status = if !observed
|
|
300
|
+
"unexecuted"
|
|
301
|
+
elsif proven == results.length
|
|
302
|
+
"covered"
|
|
303
|
+
else
|
|
304
|
+
"partial"
|
|
305
|
+
end
|
|
306
|
+
{ status: status, proven_conditions: proven, condition_count: results.length }
|
|
307
|
+
end
|
|
308
|
+
|
|
309
|
+
def conjunction_coverage(decision, condition)
|
|
310
|
+
status = if decision[:status] == "covered" && condition[:status] == "covered"
|
|
311
|
+
"covered"
|
|
312
|
+
elsif decision[:status] == "unexecuted" && condition[:status] == "unexecuted"
|
|
313
|
+
"unexecuted"
|
|
314
|
+
else
|
|
315
|
+
"partial"
|
|
316
|
+
end
|
|
317
|
+
{ status: status }
|
|
318
|
+
end
|
|
319
|
+
|
|
320
|
+
def coverage_status(covered, required)
|
|
321
|
+
return "unexecuted" if covered.zero?
|
|
322
|
+
return "covered" if covered == required
|
|
323
|
+
|
|
324
|
+
"partial"
|
|
325
|
+
end
|
|
326
|
+
|
|
327
|
+
def aggregate_coverage(decisions)
|
|
328
|
+
supported = decisions.reject { |decision| decision[:unsupported] }
|
|
329
|
+
boolean_supported = supported.reject { |decision| alternative_decision?(decision) }
|
|
330
|
+
decision_covered = boolean_supported.count do |decision|
|
|
331
|
+
decision[:coverage][:decision][:status] == "covered"
|
|
332
|
+
end
|
|
333
|
+
condition_decision_covered = boolean_supported.count do |decision|
|
|
334
|
+
decision[:coverage][:condition_decision][:status] == "covered"
|
|
335
|
+
end
|
|
336
|
+
condition_count = boolean_supported.sum { |decision| decision[:coverage][:condition][:condition_count] }
|
|
337
|
+
condition_values = boolean_supported.sum { |decision| decision[:coverage][:condition][:covered_values] }
|
|
338
|
+
covered_conditions = boolean_supported.sum do |decision|
|
|
339
|
+
decision[:coverage][:condition][:covered_conditions]
|
|
340
|
+
end
|
|
341
|
+
proven = boolean_supported.sum { |decision| decision[:coverage][:mcdc][:proven_conditions] }
|
|
342
|
+
percentage = ->(covered, required) { required.zero? ? nil : (covered.to_f / required * 100).round(2) }
|
|
343
|
+
{ decision: { covered_decisions: decision_covered, supported_decisions: boolean_supported.length,
|
|
344
|
+
percentage: percentage.call(decision_covered, boolean_supported.length) },
|
|
345
|
+
condition: { covered_values: condition_values, required_values: condition_count * 2,
|
|
346
|
+
covered_conditions: covered_conditions, condition_count: condition_count,
|
|
347
|
+
percentage: percentage.call(condition_values, condition_count * 2) },
|
|
348
|
+
condition_decision: { covered_decisions: condition_decision_covered,
|
|
349
|
+
supported_decisions: boolean_supported.length,
|
|
350
|
+
percentage: percentage.call(condition_decision_covered, boolean_supported.length) },
|
|
351
|
+
mcdc: { proven_conditions: proven, supported_conditions: condition_count,
|
|
352
|
+
percentage: percentage.call(proven, condition_count) },
|
|
353
|
+
alternative: alternative_aggregate(decisions) }
|
|
354
|
+
end
|
|
355
|
+
|
|
356
|
+
def alternative_aggregate(decisions)
|
|
357
|
+
flow = decisions.select { |decision| !decision[:unsupported] && alternative_decision?(decision) }
|
|
358
|
+
required = flow.sum { |decision| decision.dig(:coverage, :alternative, :required_alternatives).to_i }
|
|
359
|
+
covered = flow.sum { |decision| decision.dig(:coverage, :alternative, :covered_alternatives).to_i }
|
|
360
|
+
{ covered_alternatives: covered, required_alternatives: required,
|
|
361
|
+
supported_decisions: flow.length, percentage: required.zero? ? nil : (covered.to_f / required * 100).round(2) }
|
|
362
|
+
end
|
|
363
|
+
|
|
154
364
|
def valid_vector(vector, decision)
|
|
155
365
|
return nil unless id(vector, :decision_id) == id(decision, :id)
|
|
156
366
|
|
|
@@ -180,7 +390,11 @@ module Branchproof
|
|
|
180
390
|
return nil
|
|
181
391
|
end
|
|
182
392
|
|
|
183
|
-
|
|
393
|
+
if alternative_decision?(decision)
|
|
394
|
+
handle_alternative_vector(vector, decision)
|
|
395
|
+
else
|
|
396
|
+
effective_mask(vector, id(decision, :id), decision[:tree])
|
|
397
|
+
end
|
|
184
398
|
decision_source.nil? ? vector : vector.merge(source_id: decision_source)
|
|
185
399
|
rescue ArgumentError
|
|
186
400
|
@analysis_invalid = true
|
|
@@ -188,6 +402,36 @@ module Branchproof
|
|
|
188
402
|
nil
|
|
189
403
|
end
|
|
190
404
|
|
|
405
|
+
# rubocop:disable-next Naming/PredicateMethod -- raises on malformed flow vectors
|
|
406
|
+
def handle_alternative_vector(vector, decision)
|
|
407
|
+
values = values_for(vector)
|
|
408
|
+
expected = alternatives(decision).length
|
|
409
|
+
raise ArgumentError, "invalid alternative vector shape" unless values.length == expected
|
|
410
|
+
|
|
411
|
+
observations = values.each_with_index.filter_map { |item, index| [index, item] unless item.nil? }
|
|
412
|
+
raise ArgumentError, "invalid alternative trace" unless valid_alternative_trace?(decision, observations,
|
|
413
|
+
outcome(vector))
|
|
414
|
+
|
|
415
|
+
true
|
|
416
|
+
end
|
|
417
|
+
|
|
418
|
+
def valid_alternative_trace?(decision, observations, result)
|
|
419
|
+
return false unless result == true
|
|
420
|
+
|
|
421
|
+
expected = alternatives(decision).length
|
|
422
|
+
return false unless expected.positive?
|
|
423
|
+
if id(decision, :kind).to_s == "implicit"
|
|
424
|
+
return expected == 2 && observations.length == 2 &&
|
|
425
|
+
observations.map(&:first) == [0, 1] && observations.map(&:last).count(true) == 1
|
|
426
|
+
end
|
|
427
|
+
|
|
428
|
+
return false unless observations.length.between?(1, expected)
|
|
429
|
+
|
|
430
|
+
observations.each_with_index.all? do |(index, value), position|
|
|
431
|
+
index == position && value == (position == observations.length - 1)
|
|
432
|
+
end
|
|
433
|
+
end
|
|
434
|
+
|
|
191
435
|
def effective_mask(vector, decision_id, tree = nil)
|
|
192
436
|
decision = records(@inventory, :decisions).find { id(_1, :id) == decision_id } unless tree
|
|
193
437
|
tree ||= decision && decision[:tree]
|
|
@@ -213,6 +457,10 @@ module Branchproof
|
|
|
213
457
|
|
|
214
458
|
return [value, 1 << index, offset + 1]
|
|
215
459
|
end
|
|
460
|
+
if type == :not
|
|
461
|
+
child_result, child_mask, consumed = replay(node.fetch(:child), values, offset)
|
|
462
|
+
return [!child_result, child_mask, consumed]
|
|
463
|
+
end
|
|
216
464
|
left_result, left_mask, consumed = replay(node.fetch(:left), values, offset)
|
|
217
465
|
return [left_result, left_mask, consumed] if (type == :and && !left_result) || (type == :or && left_result)
|
|
218
466
|
|
|
@@ -312,6 +560,8 @@ module Branchproof
|
|
|
312
560
|
return { entry: entry, true_exits: [edges[0]], false_exits: [edges[1]], all_bits: 1 << index, edges: edges }
|
|
313
561
|
end
|
|
314
562
|
type = id(node, :type).to_sym
|
|
563
|
+
return build_graph(node.fetch(:child), false_destination, true_destination) if type == :not
|
|
564
|
+
|
|
315
565
|
right = build_graph(node[:right], true_destination, false_destination)
|
|
316
566
|
if type == :and
|
|
317
567
|
left = build_graph(node[:left], right[:entry], false_destination)
|
|
@@ -327,7 +577,9 @@ module Branchproof
|
|
|
327
577
|
end
|
|
328
578
|
|
|
329
579
|
def impose_path(node, target, target_value, constraints)
|
|
330
|
-
|
|
580
|
+
type = id(node, :type).to_sym
|
|
581
|
+
return true if type == :atom && id(node, :index) == target
|
|
582
|
+
return impose_path(node.fetch(:child), target, target_value, constraints) if type == :not
|
|
331
583
|
|
|
332
584
|
left = node[:left]
|
|
333
585
|
right = node[:right]
|
|
@@ -352,6 +604,8 @@ module Branchproof
|
|
|
352
604
|
|
|
353
605
|
constraints << { condition_index: id(node, :index), value: desired }
|
|
354
606
|
true
|
|
607
|
+
elsif id(node, :type).to_sym == :not
|
|
608
|
+
impose_subtree(node.fetch(:child), !desired, constraints)
|
|
355
609
|
elsif desired == (id(node, :type).to_sym == :and)
|
|
356
610
|
impose_subtree(node[:left], desired, constraints) && impose_subtree(node[:right], desired, constraints)
|
|
357
611
|
else
|
|
@@ -360,7 +614,9 @@ module Branchproof
|
|
|
360
614
|
end
|
|
361
615
|
|
|
362
616
|
def evaluate(node, values)
|
|
363
|
-
|
|
617
|
+
type = id(node, :type).to_sym
|
|
618
|
+
return values[id(node, :index)] if type == :atom
|
|
619
|
+
return !evaluate(node.fetch(:child), values) if type == :not
|
|
364
620
|
|
|
365
621
|
left = evaluate(node[:left], values)
|
|
366
622
|
return left if id(node, :type).to_sym == :and && !left
|
|
@@ -370,11 +626,16 @@ module Branchproof
|
|
|
370
626
|
end
|
|
371
627
|
|
|
372
628
|
def evaluate_with_trace(node, values, trace = [])
|
|
373
|
-
|
|
629
|
+
type = id(node, :type).to_sym
|
|
630
|
+
if type == :atom
|
|
374
631
|
value = !values[id(node, :index)].nil? && values[id(node, :index)] != false
|
|
375
632
|
trace << [id(node, :index), value]
|
|
376
633
|
return [value, trace]
|
|
377
634
|
end
|
|
635
|
+
if type == :not
|
|
636
|
+
child, = evaluate_with_trace(node.fetch(:child), values, trace)
|
|
637
|
+
return [!child, trace]
|
|
638
|
+
end
|
|
378
639
|
left, = evaluate_with_trace(node[:left], values, trace)
|
|
379
640
|
return [left, trace] if id(node, :type).to_sym == :and && !left
|
|
380
641
|
return [left, trace] if id(node, :type).to_sym == :or && left
|
|
@@ -383,13 +644,12 @@ module Branchproof
|
|
|
383
644
|
end
|
|
384
645
|
|
|
385
646
|
def contains?(node, target)
|
|
386
|
-
if id(node,
|
|
387
|
-
|
|
388
|
-
|
|
389
|
-
|
|
647
|
+
if id(node, :type).to_sym == :atom
|
|
648
|
+
id(node, :index) == target
|
|
649
|
+
elsif id(node, :type).to_sym == :not
|
|
650
|
+
contains?(node.fetch(:child), target)
|
|
390
651
|
else
|
|
391
|
-
contains?(node[:left],
|
|
392
|
-
target) || contains?(node[:right], target)
|
|
652
|
+
contains?(node[:left], target) || contains?(node[:right], target)
|
|
393
653
|
end
|
|
394
654
|
end
|
|
395
655
|
|
data/lib/branchproof/cli.rb
CHANGED
|
@@ -49,7 +49,7 @@ module Branchproof
|
|
|
49
49
|
message: merge_status[:reason].to_s }]
|
|
50
50
|
end
|
|
51
51
|
end
|
|
52
|
-
if
|
|
52
|
+
if value(baseline, :status).to_s == "PASSED"
|
|
53
53
|
analysis = Analyzer.new(inventory: inventory, evidence: evidence.snapshot, limits: options[:limits]).call
|
|
54
54
|
baseline[:analysis] = analysis
|
|
55
55
|
if options[:level] >= 2
|
|
@@ -70,7 +70,7 @@ module Branchproof
|
|
|
70
70
|
diagnostics = Array(value(inventory, :diagnostics)) + Array(value(baseline, :diagnostics)) +
|
|
71
71
|
Array(value(evidence.snapshot,
|
|
72
72
|
:diagnostics)) + Array(value(value(baseline, :analysis), :diagnostics))
|
|
73
|
-
analysis =
|
|
73
|
+
analysis = value(baseline, :analysis)
|
|
74
74
|
minima = options[:level] == 1 ? [] : Array(value(baseline, :minima))
|
|
75
75
|
report = Report.new(inventory: inventory, evidence: value(baseline, :evidence) || evidence.snapshot,
|
|
76
76
|
analysis: analysis, minima: minima, baseline: baseline, diagnostics: diagnostics,
|
|
@@ -7,7 +7,7 @@ module Branchproof
|
|
|
7
7
|
# Compares two complete report documents without loading or executing the project.
|
|
8
8
|
class Comparison
|
|
9
9
|
SCHEMA_VERSION = "1.0"
|
|
10
|
-
SUPPORTED_REPORT_SCHEMAS = %w[1.0 1.1].freeze
|
|
10
|
+
SUPPORTED_REPORT_SCHEMAS = %w[1.0 1.1 1.2].freeze
|
|
11
11
|
CRITERION_VERSION = "masking_occurrence_v1"
|
|
12
12
|
|
|
13
13
|
def initialize(before:, after:)
|