pbt 0.1.1 → 0.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +4 -0
- data/README.md +67 -2
- data/lib/pbt/check/runner_iterator.rb +1 -1
- data/lib/pbt/reporter/run_details.rb +1 -0
- data/lib/pbt/reporter/run_details_reporter.rb +74 -13
- data/lib/pbt/reporter/run_execution.rb +32 -1
- data/lib/pbt/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: d02ba2667633586ac6e6582c3f0f181dd089e63d56b160877a0b3e51ebb353e0
|
4
|
+
data.tar.gz: 1e4dc60aa8ddbac429091bd86d3f8d601bb1a34034aa8e7c6c8c914d43e5e9b8
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 9ba49e79546492deec531f0ebef12e86e8458482740d20b3d0b3398a02aae220ea5fed476bee6489909bc2f62990fadcda2fff556fab2df69d30984a9d48435c
|
7
|
+
data.tar.gz: df6c5128d3f2b839068bd736946114a81dabf590b49d1d93d7053cf7368fa65d299515323396864c9a942f2e004a3cab73f36dc215ba5660c3d5213e41ee7cfd
|
data/CHANGELOG.md
CHANGED
data/README.md
CHANGED
@@ -107,7 +107,7 @@ There are many built-in arbitraries in `Pbt`. You can use them to generate rando
|
|
107
107
|
#### Primitives
|
108
108
|
|
109
109
|
```ruby
|
110
|
-
rng = Random.new
|
110
|
+
rng = Random.new
|
111
111
|
|
112
112
|
Pbt.integer.generate(rng) # => 42
|
113
113
|
Pbt.integer(min: -1, max: 8).generate(rng) # => Integer between -1 and 8
|
@@ -139,6 +139,71 @@ Pbt.one_of(:a, 1, 0.1).generate(rng) # => :a or 1 or 0.1
|
|
139
139
|
|
140
140
|
See [ArbitraryMethods](https://github.com/ohbarye/pbt/blob/main/lib/pbt/arbitrary/arbitrary_methods.rb) module for more details.
|
141
141
|
|
142
|
+
## What if property-based tests fail?
|
143
|
+
|
144
|
+
Once a test fails it's time to debug. `Pbt` provides some features to help you debug.
|
145
|
+
|
146
|
+
### How to reproduce
|
147
|
+
|
148
|
+
When a test fails, you'll see a message like below.
|
149
|
+
|
150
|
+
```text
|
151
|
+
Pbt::PropertyFailure:
|
152
|
+
Property failed after 23 test(s)
|
153
|
+
{ seed: 11001296583699917659214176011685741769 }
|
154
|
+
Counterexample: 0
|
155
|
+
Shrunk 3 time(s)
|
156
|
+
Got ZeroDivisionError: divided by 0
|
157
|
+
# and backtraces
|
158
|
+
```
|
159
|
+
|
160
|
+
You can reproduce the failure by passing the seed to `Pbt.assert`.
|
161
|
+
|
162
|
+
```ruby
|
163
|
+
Pbt.assert(seed: 11001296583699917659214176011685741769) do
|
164
|
+
Pbt.property(Pbt.integer) do |number|
|
165
|
+
# your test
|
166
|
+
end
|
167
|
+
end
|
168
|
+
```
|
169
|
+
|
170
|
+
### Verbose mode
|
171
|
+
|
172
|
+
You may want to know which values pass and which values fail. You can enable verbose mode by passing `verbose: true` to `Pbt.assert`.
|
173
|
+
|
174
|
+
```ruby
|
175
|
+
Pbt.assert(verbose: true) do
|
176
|
+
Pbt.property(Pbt.array(Pbt.integer)) do |numbers|
|
177
|
+
# your failed test
|
178
|
+
end
|
179
|
+
end
|
180
|
+
```
|
181
|
+
|
182
|
+
The verbose mode prints the results of each tested values.
|
183
|
+
|
184
|
+
```text
|
185
|
+
Encountered failures were:
|
186
|
+
- [152477, 666997, -531468, -92182, 623948, 425913, 656138, 856463, -64529]
|
187
|
+
- [76239, 666997, -531468, -92182, 623948]
|
188
|
+
- [76239, 666997, -531468]
|
189
|
+
(snipped for README)
|
190
|
+
- [2, 163]
|
191
|
+
- [2, 11]
|
192
|
+
|
193
|
+
Execution summary:
|
194
|
+
. × [152477, 666997, -531468, -92182, 623948, 425913, 656138, 856463, -64529]
|
195
|
+
. . √ [152477, 666997, -531468, -92182, 623948]
|
196
|
+
. . √ [-64529]
|
197
|
+
. . × [76239, 666997, -531468, -92182, 623948, 425913, 656138, 856463, -64529]
|
198
|
+
. . . × [76239, 666997, -531468, -92182, 623948]
|
199
|
+
(snipped for README)
|
200
|
+
. . . . . . . . . . . . . . . . . √ [2, 21]
|
201
|
+
. . . . . . . . . . . . . . . . . × [2, 11]
|
202
|
+
. . . . . . . . . . . . . . . . . . √ []
|
203
|
+
. . . . . . . . . . . . . . . . . . √ [2, 1]
|
204
|
+
. . . . . . . . . . . . . . . . . . √ [2, 0]
|
205
|
+
```
|
206
|
+
|
142
207
|
## Configuration
|
143
208
|
|
144
209
|
You can configure `Pbt` by calling `Pbt.configure` before running tests.
|
@@ -277,7 +342,7 @@ Once this project finishes the following, we will release v1.0.0.
|
|
277
342
|
- [x] Arbitrary usage
|
278
343
|
- [x] Configuration
|
279
344
|
- [x] Benchmark
|
280
|
-
- [
|
345
|
+
- [x] Rich report by verbose mode
|
281
346
|
- [ ] Allow to use expectations and matchers provided by test framework in Ractor if possible.
|
282
347
|
- It'd be so hard to pass assertions like `expect`, `assert` to a Ractor.
|
283
348
|
- [ ] More parallelism or faster execution if possible
|
@@ -16,24 +16,85 @@ module Pbt
|
|
16
16
|
def report_run_details
|
17
17
|
if @run_details.failed
|
18
18
|
message = []
|
19
|
-
|
20
|
-
message <<
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
Got #{@run_details.error_instance.class}: #{@run_details.error_message}
|
26
|
-
EOS
|
27
|
-
|
28
|
-
if @run_details.verbose
|
29
|
-
message << " \n#{@run_details.error_instance.backtrace_locations.join("\n ")}"
|
30
|
-
message << "\nEncountered failures were:"
|
31
|
-
message << @run_details.failures.map { |f| "- #{f.val}" }
|
19
|
+
message << format_error_message
|
20
|
+
message << error_backtrace
|
21
|
+
message << if @run_details.verbose
|
22
|
+
verbose_details
|
23
|
+
else
|
24
|
+
hint
|
32
25
|
end
|
33
26
|
|
34
27
|
raise PropertyFailure, message.join("\n") if message.size > 0
|
35
28
|
end
|
36
29
|
end
|
30
|
+
|
31
|
+
private
|
32
|
+
|
33
|
+
# @return [String]
|
34
|
+
def format_error_message
|
35
|
+
<<~MSG.chomp
|
36
|
+
Property failed after #{@run_details.num_runs} test(s)
|
37
|
+
{ seed: #{@run_details.seed} }
|
38
|
+
Counterexample: #{@run_details.counterexample}
|
39
|
+
Shrunk #{@run_details.num_shrinks} time(s)
|
40
|
+
Got #{@run_details.error_instance.class}: #{@run_details.error_message}
|
41
|
+
MSG
|
42
|
+
end
|
43
|
+
|
44
|
+
def error_backtrace
|
45
|
+
i = @run_details.verbose ? -1 : 10
|
46
|
+
" #{@run_details.error_instance.backtrace_locations[..i].join("\n ")}"
|
47
|
+
end
|
48
|
+
|
49
|
+
# @return [String]
|
50
|
+
def verbose_details
|
51
|
+
[
|
52
|
+
"\nEncountered failures were:",
|
53
|
+
@run_details.failures.map { |f| "- #{f.val}" },
|
54
|
+
format_execution_summary
|
55
|
+
].join("\n")
|
56
|
+
end
|
57
|
+
|
58
|
+
# @return [String]
|
59
|
+
def format_execution_summary
|
60
|
+
summary_lines = []
|
61
|
+
remaining_trees_and_depth = []
|
62
|
+
|
63
|
+
@run_details.execution_summary.reverse_each do |tree|
|
64
|
+
remaining_trees_and_depth << {depth: 1, tree:}
|
65
|
+
end
|
66
|
+
|
67
|
+
until remaining_trees_and_depth.empty?
|
68
|
+
current_tree_and_depth = remaining_trees_and_depth.pop
|
69
|
+
|
70
|
+
# format current tree according to its depth and result
|
71
|
+
current_tree = current_tree_and_depth[:tree]
|
72
|
+
current_depth = current_tree_and_depth[:depth]
|
73
|
+
result_icon = case current_tree.result
|
74
|
+
in :success
|
75
|
+
"\x1b[32m\u221A\x1b[0m" # green "√"
|
76
|
+
in :failure
|
77
|
+
"\x1b[31m\u00D7\x1b[0m" # red "×"
|
78
|
+
end
|
79
|
+
left_padding = ". " * current_depth
|
80
|
+
summary_lines << "#{left_padding}#{result_icon} #{current_tree.value}"
|
81
|
+
|
82
|
+
# push its children to the queue
|
83
|
+
current_tree.children.reverse_each do |tree|
|
84
|
+
remaining_trees_and_depth << {depth: current_depth + 1, tree:}
|
85
|
+
end
|
86
|
+
end
|
87
|
+
|
88
|
+
"\nExecution summary:\n#{summary_lines.join("\n")}\n"
|
89
|
+
end
|
90
|
+
|
91
|
+
# @return [String]
|
92
|
+
def hint
|
93
|
+
[
|
94
|
+
"\nHint: Set `verbose: true` in order to check the list of all failing values encountered during the run.",
|
95
|
+
"Hint: Set `seed: #{@run_details.seed}` in order to reproduce the failed test case with the same values."
|
96
|
+
].join("\n")
|
97
|
+
end
|
37
98
|
end
|
38
99
|
end
|
39
100
|
end
|
@@ -6,6 +6,14 @@ module Pbt
|
|
6
6
|
module Reporter
|
7
7
|
# Represents the result of a single run of a property test.
|
8
8
|
class RunExecution
|
9
|
+
# A tree node of the execution for verbose output.
|
10
|
+
ExecutionTreeNode = Struct.new(
|
11
|
+
:result,
|
12
|
+
:value,
|
13
|
+
:children,
|
14
|
+
keyword_init: true
|
15
|
+
)
|
16
|
+
|
9
17
|
# @param verbose [Boolean] Whether to print verbose output.
|
10
18
|
def initialize(verbose)
|
11
19
|
@verbose = verbose
|
@@ -13,12 +21,18 @@ module Pbt
|
|
13
21
|
@failure = nil
|
14
22
|
@failures = []
|
15
23
|
@num_successes = 0
|
24
|
+
@root_execution_trees = []
|
25
|
+
@current_level_execution_trees = @root_execution_trees
|
16
26
|
end
|
17
27
|
|
18
28
|
# Record a failure in the run.
|
19
29
|
#
|
20
30
|
# @param c [Pbt::Check::Case]
|
21
31
|
def record_failure(c)
|
32
|
+
if @verbose
|
33
|
+
current_tree = append_execution_tree_node(:failure, c.val)
|
34
|
+
@current_level_execution_trees = current_tree.children
|
35
|
+
end
|
22
36
|
@path_to_failure << c.index
|
23
37
|
@failures << c
|
24
38
|
|
@@ -29,8 +43,12 @@ module Pbt
|
|
29
43
|
|
30
44
|
# Record a successful run.
|
31
45
|
#
|
46
|
+
# @param c [Pbt::Check::Case]
|
32
47
|
# @return [void]
|
33
|
-
def record_success
|
48
|
+
def record_success(c)
|
49
|
+
if @verbose
|
50
|
+
append_execution_tree_node(:success, c.val)
|
51
|
+
end
|
34
52
|
@num_successes += 1
|
35
53
|
end
|
36
54
|
|
@@ -58,6 +76,7 @@ module Pbt
|
|
58
76
|
error_instance: nil,
|
59
77
|
failures: @failures,
|
60
78
|
verbose: @verbose,
|
79
|
+
execution_summary: @root_execution_trees,
|
61
80
|
run_configuration: config
|
62
81
|
)
|
63
82
|
else
|
@@ -72,10 +91,22 @@ module Pbt
|
|
72
91
|
error_instance: @failure,
|
73
92
|
failures: @failures,
|
74
93
|
verbose: @verbose,
|
94
|
+
execution_summary: @root_execution_trees,
|
75
95
|
run_configuration: config
|
76
96
|
)
|
77
97
|
end
|
78
98
|
end
|
99
|
+
|
100
|
+
private
|
101
|
+
|
102
|
+
# @param result [Symbol] The result of the current node.
|
103
|
+
# @param value [Object] The value to test.
|
104
|
+
# @return [Hash] The current execution tree.
|
105
|
+
def append_execution_tree_node(result, value)
|
106
|
+
current_tree = ExecutionTreeNode.new(result:, value:, children: [])
|
107
|
+
@current_level_execution_trees << current_tree
|
108
|
+
current_tree
|
109
|
+
end
|
79
110
|
end
|
80
111
|
end
|
81
112
|
end
|
data/lib/pbt/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: pbt
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.2.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- ohbarye
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2024-04-
|
11
|
+
date: 2024-04-16 00:00:00.000000000 Z
|
12
12
|
dependencies: []
|
13
13
|
description:
|
14
14
|
email:
|