pbt 0.4.1 → 0.5.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +9 -0
- data/README.md +67 -103
- data/benchmark/failure_simple.rb +0 -20
- data/benchmark/success_cpu_bound.rb +0 -16
- data/benchmark/success_io_bound.rb +0 -16
- data/benchmark/success_simple.rb +0 -16
- data/lib/pbt/arbitrary/arbitrary_methods.rb +1 -1
- data/lib/pbt/check/configuration.rb +3 -6
- data/lib/pbt/check/runner_methods.rb +1 -85
- data/lib/pbt/version.rb +1 -1
- metadata +3 -6
- data/lib/pbt/check/rspec_adapter/integration.rb +0 -64
- data/lib/pbt/check/rspec_adapter/predicate_block_inspector.rb +0 -47
- data/lib/pbt/check/rspec_adapter/property_extension.rb +0 -74
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 2b93d8a06cbff5946bab644527868f42601ccd7521d4d11e3746fa00c279cd62
|
4
|
+
data.tar.gz: ab19c9a123d4408cb49e7a24225bca930d5bd4134000e98a3efc9c233ce97f9c
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: cd5fd29fb5e5d3afb5817fd24b0c8967037e2158dd3cc57e4d4ab83eb12c7a31c1dfff376a88b96173ca14853cd3d56b89de212c3630bb6cffe4bb4f04b106d6
|
7
|
+
data.tar.gz: b0d37a7198cd0d1c504170f00041b5decab223e73c8896ebd1d357f4482503665ec0ca200cf5557bbed6c04f775cb3c7264eb61b1a42e6b6982f30e90941155e
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,14 @@
|
|
1
1
|
## [Unreleased]
|
2
2
|
|
3
|
+
## [0.5.0] - 2024-12-30
|
4
|
+
|
5
|
+
- [Breaking change] Drop `:process` and `:thread` workers since there are no concrete use cases.
|
6
|
+
- [Breaking change] Drop `:experimental_ractor_rspec_integration` option since there are no concrete use cases.
|
7
|
+
|
8
|
+
## [0.4.2] - 2024-05-23
|
9
|
+
|
10
|
+
- Fix Prism `LoadError` message [#27](https://github.com/ohbarye/pbt/pull/27) by @sambostock
|
11
|
+
|
3
12
|
## [0.4.1] - 2024-05-10
|
4
13
|
|
5
14
|
- Fix a bug for experimental_ractor_rspec_integration mode. When a test file name starts with a number, it can't be a constant name.
|
data/README.md
CHANGED
@@ -8,6 +8,8 @@ A property-based testing tool for Ruby with experimental features that allow you
|
|
8
8
|
|
9
9
|
PBT stands for Property-Based Testing.
|
10
10
|
|
11
|
+
As for the results of the parallelization experiment, please refer the talk at RubyKaigi 2024: [Unlocking Potential of Property Based Testing with Ractor](https://rubykaigi.org/2024/presentations/ohbarye.html).
|
12
|
+
|
11
13
|
## What's Property-Based Testing?
|
12
14
|
|
13
15
|
Property-Based Testing is a testing methodology that focuses on the properties a system should always satisfy, rather than checking individual examples. Instead of writing tests for predefined inputs and outputs, PBT allows you to specify the general characteristics that your code should adhere to and then automatically generates a wide range of inputs to verify these properties.
|
@@ -39,29 +41,34 @@ Off course you can install with `gem intstall pbt`.
|
|
39
41
|
### Simple property
|
40
42
|
|
41
43
|
```ruby
|
42
|
-
# Let's say you have
|
43
|
-
def
|
44
|
-
|
44
|
+
# Let's say you have your own sort method.
|
45
|
+
def sort(array)
|
46
|
+
return array if array.size <= 2 # Here's a bug! It should be 1.
|
47
|
+
pivot, *rest = array
|
48
|
+
left, right = rest.partition { |n| n <= pivot }
|
49
|
+
sort(left) + [pivot] + sort(right)
|
45
50
|
end
|
46
51
|
|
47
52
|
Pbt.assert do
|
48
|
-
# The given block is executed 100 times with different random numbers.
|
49
|
-
# Besides,
|
50
|
-
Pbt.property(Pbt.integer) do |
|
51
|
-
result =
|
52
|
-
|
53
|
+
# The given block is executed 100 times with different arrays with random numbers.
|
54
|
+
# Besides, if you set `worker: :ractor` option to `assert` method, it runs in parallel using Ractor.
|
55
|
+
Pbt.property(Pbt.array(Pbt.integer)) do |numbers|
|
56
|
+
result = sort(numbers)
|
57
|
+
result.each_cons(2) do |x, y|
|
58
|
+
raise "Sort algorithm is wrong." unless x <= y
|
59
|
+
end
|
53
60
|
end
|
54
61
|
end
|
55
62
|
|
56
|
-
# If the
|
57
|
-
# For example, the
|
63
|
+
# If the method has a bug, the test fails and it reports a minimum counterexample.
|
64
|
+
# For example, the sort method doesn't work for [0, -1].
|
58
65
|
#
|
59
66
|
# Pbt::PropertyFailure:
|
60
67
|
# Property failed after 23 test(s)
|
61
|
-
#
|
62
|
-
#
|
63
|
-
# Shrunk
|
64
|
-
# Got
|
68
|
+
# seed: 43738985293126714007411539287084402325
|
69
|
+
# counterexample: [0, -1]
|
70
|
+
# Shrunk 40 time(s)
|
71
|
+
# Got RuntimeError: Sort algorithm is wrong.
|
65
72
|
```
|
66
73
|
|
67
74
|
### Explain The Snippet
|
@@ -92,7 +99,7 @@ Here, we used only one type of arbitrary, `Pbt.integer`. There are many other bu
|
|
92
99
|
In PBT, If a test fails, it attempts to shrink the case that caused the failure into a form that is easier for humans to understand.
|
93
100
|
In other words, instead of stopping the test itself the first time it fails and reporting the failed value, it tries to find the minimal value that causes the error.
|
94
101
|
|
95
|
-
When there is a test that fails when given an even number, a counterexample of `
|
102
|
+
When there is a test that fails when given an even number, a counterexample of `[0, -1]` is simpler and easier to understand than any complex example like `[-897860, -930517, 577817, -16302, 310864, 856411, -304517, 86613, -78231]`.
|
96
103
|
|
97
104
|
### Arbitrary
|
98
105
|
|
@@ -103,16 +110,16 @@ There are many built-in arbitraries in `Pbt`. You can use them to generate rando
|
|
103
110
|
```ruby
|
104
111
|
rng = Random.new
|
105
112
|
|
106
|
-
Pbt.integer.generate(rng)
|
113
|
+
Pbt.integer.generate(rng) # => 42
|
107
114
|
Pbt.integer(min: -1, max: 8).generate(rng) # => Integer between -1 and 8
|
108
115
|
|
109
|
-
Pbt.symbol.generate(rng)
|
116
|
+
Pbt.symbol.generate(rng) # => :atq
|
110
117
|
|
111
|
-
Pbt.ascii_char.generate(rng)
|
112
|
-
Pbt.ascii_string.generate(rng)
|
118
|
+
Pbt.ascii_char.generate(rng) # => "a"
|
119
|
+
Pbt.ascii_string.generate(rng) # => "aagjZfao"
|
113
120
|
|
114
|
-
Pbt.boolean.generate(rng)
|
115
|
-
Pbt.constant(42).generate(rng)
|
121
|
+
Pbt.boolean.generate(rng) # => true or false
|
122
|
+
Pbt.constant(42).generate(rng) # => 42 always
|
116
123
|
```
|
117
124
|
|
118
125
|
#### Composites
|
@@ -120,15 +127,15 @@ Pbt.constant(42).generate(rng) # => 42 always
|
|
120
127
|
```ruby
|
121
128
|
rng = Random.new
|
122
129
|
|
123
|
-
Pbt.array(Pbt.integer).generate(rng)
|
124
|
-
Pbt.array(Pbt.integer, max: 1, empty: true).generate(rng)
|
130
|
+
Pbt.array(Pbt.integer).generate(rng) # => [121, -13141, 9825]
|
131
|
+
Pbt.array(Pbt.integer, max: 1, empty: true).generate(rng) # => [] or [42] etc.
|
125
132
|
|
126
|
-
Pbt.tuple(Pbt.symbol, Pbt.integer).generate(rng)
|
133
|
+
Pbt.tuple(Pbt.symbol, Pbt.integer).generate(rng) # => [:atq, 42]
|
127
134
|
|
128
135
|
Pbt.fixed_hash(x: Pbt.symbol, y: Pbt.integer).generate(rng) # => {x: :atq, y: 42}
|
129
|
-
Pbt.hash(Pbt.symbol, Pbt.integer).generate(rng)
|
136
|
+
Pbt.hash(Pbt.symbol, Pbt.integer).generate(rng) # => {atq: 121, ygab: -1142}
|
130
137
|
|
131
|
-
Pbt.one_of(:a, 1, 0.1).generate(rng)
|
138
|
+
Pbt.one_of(:a, 1, 0.1).generate(rng) # => :a or 1 or 0.1
|
132
139
|
````
|
133
140
|
|
134
141
|
See [ArbitraryMethods](https://github.com/ohbarye/pbt/blob/main/lib/pbt/arbitrary/arbitrary_methods.rb) module for more details.
|
@@ -144,18 +151,18 @@ When a test fails, you'll see a message like below.
|
|
144
151
|
```text
|
145
152
|
Pbt::PropertyFailure:
|
146
153
|
Property failed after 23 test(s)
|
147
|
-
|
148
|
-
|
149
|
-
Shrunk
|
150
|
-
Got
|
154
|
+
seed: 43738985293126714007411539287084402325
|
155
|
+
counterexample: [0, -1]
|
156
|
+
Shrunk 40 time(s)
|
157
|
+
Got RuntimeError: Sort algorithm is wrong.
|
151
158
|
# and backtraces
|
152
159
|
```
|
153
160
|
|
154
161
|
You can reproduce the failure by passing the seed to `Pbt.assert`.
|
155
162
|
|
156
163
|
```ruby
|
157
|
-
Pbt.assert(seed:
|
158
|
-
Pbt.property(Pbt.integer) do |number|
|
164
|
+
Pbt.assert(seed: 43738985293126714007411539287084402325) do
|
165
|
+
Pbt.property(Pbt.array(Pbt.integer)) do |number|
|
159
166
|
# your test
|
160
167
|
end
|
161
168
|
end
|
@@ -177,25 +184,29 @@ The verbose mode prints the results of each tested values.
|
|
177
184
|
|
178
185
|
```text
|
179
186
|
Encountered failures were:
|
180
|
-
- [
|
181
|
-
- [
|
182
|
-
- [
|
187
|
+
- [-897860, -930517, 577817, -16302, 310864, 856411, -304517, 86613, -78231]
|
188
|
+
- [310864, 856411, -304517, 86613, -78231]
|
189
|
+
- [-304517, 86613, -78231]
|
183
190
|
(snipped for README)
|
184
|
-
- [
|
185
|
-
- [
|
191
|
+
- [0, -3]
|
192
|
+
- [0, -2]
|
193
|
+
- [0, -1]
|
186
194
|
|
187
195
|
Execution summary:
|
188
|
-
. × [
|
189
|
-
. . √ [
|
190
|
-
. . √ [-
|
191
|
-
. .
|
192
|
-
. .
|
196
|
+
. × [-897860, -930517, 577817, -16302, 310864, 856411, -304517, 86613, -78231]
|
197
|
+
. . √ [-897860, -930517, 577817, -16302, 310864]
|
198
|
+
. . √ [-930517, 577817, -16302, 310864, 856411]
|
199
|
+
. . √ [577817, -16302, 310864, 856411, -304517]
|
200
|
+
. . √ [-16302, 310864, 856411, -304517, 86613]
|
201
|
+
. . × [310864, 856411, -304517, 86613, -78231]
|
193
202
|
(snipped for README)
|
194
|
-
. . . . . . . . . . . . . . . . . √ [2
|
195
|
-
. . . . . . . . . . . . . . . . .
|
196
|
-
. . . . . . . . . . . . . . . . . .
|
197
|
-
. . . . . . . . . . . . . . . . . . √ [
|
198
|
-
. . . . . . . . . . . . . . . . . . √ [
|
203
|
+
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . √ [-2]
|
204
|
+
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . √ []
|
205
|
+
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . × [0, -1]
|
206
|
+
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . √ [0]
|
207
|
+
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . √ [-1]
|
208
|
+
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . √ []
|
209
|
+
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . √ [0, 0]
|
199
210
|
```
|
200
211
|
|
201
212
|
## Configuration
|
@@ -207,7 +218,7 @@ Pbt.configure do |config|
|
|
207
218
|
# Whether to print verbose output. Default is `false`.
|
208
219
|
config.verbose = false
|
209
220
|
|
210
|
-
# The concurrency method to use. `:ractor
|
221
|
+
# The concurrency method to use. `:ractor` and `:none` are supported. Default is `:none`.
|
211
222
|
config.worker = :none
|
212
223
|
|
213
224
|
# The number of runs to perform. Default is `100`.
|
@@ -220,9 +231,6 @@ Pbt.configure do |config|
|
|
220
231
|
# Whether to report exceptions in threads.
|
221
232
|
# It's useful to suppress error logs on Ractor that reports many errors. Default is `false`.
|
222
233
|
config.thread_report_on_exception = false
|
223
|
-
|
224
|
-
# Whether to allow RSpec expectation and matchers in Ractor. It's quite experimental! Default is `false`.
|
225
|
-
config.experimental_ractor_rspec_integration = false
|
226
234
|
end
|
227
235
|
```
|
228
236
|
|
@@ -238,15 +246,13 @@ end
|
|
238
246
|
|
239
247
|
One of the key features of `Pbt` is its ability to rapidly execute test cases in parallel or concurrently, using a large number of values (by default, `100`) generated by `Arbitrary`.
|
240
248
|
|
241
|
-
For concurrent processing, you can specify
|
242
|
-
|
243
|
-
`Pbt` supports 3 concurrency methods and 1 sequential one. You can choose one of them by setting the `worker` option.
|
249
|
+
For concurrent processing, you can specify `:ractor` using the `worker` option. Alternatively, choose `:none` for serial execution.
|
244
250
|
|
245
251
|
Be aware that the performance of each method depends on the test subject. For example, if the test subject is CPU-bound, `:ractor` may be the best choice. Otherwise, `:none` shall be the best choice for most cases. See [benchmarks](benchmark/README.md).
|
246
252
|
|
247
253
|
### Ractor
|
248
254
|
|
249
|
-
`:ractor` worker is useful for test cases that are CPU-bound. But it's experimental and has some limitations as described below. If you encounter any issues due to those limitations, consider
|
255
|
+
`:ractor` worker is useful for test cases that are CPU-bound. But it's experimental and has some limitations as described below. If you encounter any issues due to those limitations, consider falling back to `:none`.
|
250
256
|
|
251
257
|
```ruby
|
252
258
|
Pbt.assert(worker: :ractor) do
|
@@ -286,53 +292,9 @@ it do
|
|
286
292
|
end
|
287
293
|
```
|
288
294
|
|
289
|
-
If you're a challenger, you can enable the experimental feature to allow using RSpec expectations and matchers in Ractor. It works but it's quite experimental and could cause unexpected behaviors.
|
290
|
-
|
291
|
-
Please note that this feature depends on [prism](https://ruby.github.io/prism/) gem. If you use Ruby 3.2 or prior, you need to install the gem by yourself.
|
292
|
-
|
293
|
-
```ruby
|
294
|
-
it do
|
295
|
-
Pbt.assert(worker: :ractor, experimental_ractor_rspec_integration: true) do
|
296
|
-
Pbt.property(Pbt.integer) do |n|
|
297
|
-
# Some RSpec expectations and matchers are available in Ractor by hack.
|
298
|
-
# Other features like `let`, `subject`, `before`, `after` that access out of block are still not available.
|
299
|
-
expect(n).to be_an(Integer)
|
300
|
-
end
|
301
|
-
end
|
302
|
-
end
|
303
|
-
```
|
304
|
-
|
305
|
-
### Process
|
306
|
-
|
307
|
-
If you'd like to run test cases that are CPU-bound and `:ractor` is not available, `:process` becomes a good choice.
|
308
|
-
|
309
|
-
```ruby
|
310
|
-
Pbt.assert(worker: :process) do
|
311
|
-
Pbt.property(Pbt.integer) do |n|
|
312
|
-
# ...
|
313
|
-
end
|
314
|
-
end
|
315
|
-
```
|
316
|
-
|
317
|
-
If you want to use `:process`, you need to install the [parallel](https://github.com/grosser/parallel) gem.
|
318
|
-
|
319
|
-
### Thread
|
320
|
-
|
321
|
-
You may not need to run test cases with multi-threads.
|
322
|
-
|
323
|
-
```ruby
|
324
|
-
Pbt.assert(worker: :thread) do
|
325
|
-
Pbt.property(Pbt.integer) do |n|
|
326
|
-
# ...
|
327
|
-
end
|
328
|
-
end
|
329
|
-
```
|
330
|
-
|
331
|
-
If you want to use `:thread`, you need to install the [parallel](https://github.com/grosser/parallel) gem.
|
332
|
-
|
333
295
|
### None
|
334
296
|
|
335
|
-
For most cases, `:none` is the best choice. It runs tests sequentially
|
297
|
+
For most cases, `:none` is the best choice. It runs tests sequentially but most test cases finishes within a reasonable time.
|
336
298
|
|
337
299
|
```ruby
|
338
300
|
Pbt.assert(worker: :none) do
|
@@ -351,8 +313,8 @@ Once this project finishes the following, we will release v1.0.0.
|
|
351
313
|
- [x] Support shrinking
|
352
314
|
- [x] Support multiple concurrency methods
|
353
315
|
- [x] Ractor
|
354
|
-
- [x] Process
|
355
|
-
- [x] Thread
|
316
|
+
- [x] Process (dropped)
|
317
|
+
- [x] Thread (dropped)
|
356
318
|
- [x] None (Run tests sequentially)
|
357
319
|
- [x] Documentation
|
358
320
|
- [x] Add better examples
|
@@ -360,11 +322,13 @@ Once this project finishes the following, we will release v1.0.0.
|
|
360
322
|
- [x] Configuration
|
361
323
|
- [x] Benchmark
|
362
324
|
- [x] Rich report by verbose mode
|
363
|
-
- [x] (Partially) Allow to use expectations and matchers provided by test framework in Ractor
|
325
|
+
- [x] (Partially) Allow to use expectations and matchers provided by test framework in Ractor. (dropped)
|
364
326
|
- It'd be so hard to pass assertions like `expect`, `assert` to a Ractor.
|
365
327
|
- [ ] Implement frequency arbitrary
|
366
328
|
- [ ] Statistics feature to aggregate generated values
|
367
329
|
- [ ] Decide DSL
|
330
|
+
- [ ] Try Fiber
|
331
|
+
- [ ] Stateful property-based testing
|
368
332
|
|
369
333
|
## Development
|
370
334
|
|
data/benchmark/failure_simple.rb
CHANGED
@@ -17,26 +17,6 @@ Benchmark.ips do |x|
|
|
17
17
|
# noop
|
18
18
|
end
|
19
19
|
|
20
|
-
x.report("process") do
|
21
|
-
Pbt.assert(worker: :process, seed:, num_runs: 100) do
|
22
|
-
Pbt.property(Pbt.integer) do |x|
|
23
|
-
task(x)
|
24
|
-
end
|
25
|
-
end
|
26
|
-
rescue Pbt::PropertyFailure
|
27
|
-
# noop
|
28
|
-
end
|
29
|
-
|
30
|
-
x.report("thread") do
|
31
|
-
Pbt.assert(worker: :thread, seed:, num_runs: 100) do
|
32
|
-
Pbt.property(Pbt.integer) do |x|
|
33
|
-
task(x)
|
34
|
-
end
|
35
|
-
end
|
36
|
-
rescue Pbt::PropertyFailure
|
37
|
-
# noop
|
38
|
-
end
|
39
|
-
|
40
20
|
x.report("none") do
|
41
21
|
Pbt.assert(worker: :none, seed:, num_runs: 100) do
|
42
22
|
Pbt.property(Pbt.integer) do |x|
|
@@ -25,22 +25,6 @@ Benchmark.ips do |x|
|
|
25
25
|
end
|
26
26
|
end
|
27
27
|
|
28
|
-
x.report("process") do
|
29
|
-
Pbt.assert(worker: :process, num_runs: 100) do
|
30
|
-
Pbt.property(Pbt.constant([a, b, c])) do |x, y, z|
|
31
|
-
task(x, y, z)
|
32
|
-
end
|
33
|
-
end
|
34
|
-
end
|
35
|
-
|
36
|
-
x.report("thread") do
|
37
|
-
Pbt.assert(worker: :thread, num_runs: 100) do
|
38
|
-
Pbt.property(Pbt.constant([a, b, c])) do |x, y, z|
|
39
|
-
task(x, y, z)
|
40
|
-
end
|
41
|
-
end
|
42
|
-
end
|
43
|
-
|
44
28
|
x.report("none") do
|
45
29
|
Pbt.assert(worker: :none, num_runs: 100) do
|
46
30
|
Pbt.property(Pbt.constant([a, b, c])) do |x, y, z|
|
@@ -18,22 +18,6 @@ Benchmark.ips do |x|
|
|
18
18
|
end
|
19
19
|
end
|
20
20
|
|
21
|
-
x.report("process") do
|
22
|
-
Pbt.assert(worker: :process, seed:, num_runs: 100) do
|
23
|
-
Pbt.property(Pbt.ascii_string) do |str|
|
24
|
-
task(str)
|
25
|
-
end
|
26
|
-
end
|
27
|
-
end
|
28
|
-
|
29
|
-
x.report("thread") do
|
30
|
-
Pbt.assert(worker: :thread, seed:, num_runs: 100) do
|
31
|
-
Pbt.property(Pbt.ascii_string) do |str|
|
32
|
-
task(str)
|
33
|
-
end
|
34
|
-
end
|
35
|
-
end
|
36
|
-
|
37
21
|
x.report("none") do
|
38
22
|
Pbt.assert(worker: :none, seed:, num_runs: 100) do
|
39
23
|
Pbt.property(Pbt.ascii_string) do |str|
|
data/benchmark/success_simple.rb
CHANGED
@@ -14,22 +14,6 @@ Benchmark.ips do |x|
|
|
14
14
|
end
|
15
15
|
end
|
16
16
|
|
17
|
-
x.report("process") do
|
18
|
-
Pbt.assert(worker: :process, seed:, num_runs: 100) do
|
19
|
-
Pbt.property(Pbt.integer) do |x|
|
20
|
-
task(x)
|
21
|
-
end
|
22
|
-
end
|
23
|
-
end
|
24
|
-
|
25
|
-
x.report("thread") do
|
26
|
-
Pbt.assert(worker: :thread, seed:, num_runs: 100) do
|
27
|
-
Pbt.property(Pbt.integer) do |x|
|
28
|
-
task(x)
|
29
|
-
end
|
30
|
-
end
|
31
|
-
end
|
32
|
-
|
33
17
|
x.report("none") do
|
34
18
|
Pbt.assert(worker: :none, seed:, num_runs: 100) do
|
35
19
|
Pbt.property(Pbt.integer) do |x|
|
@@ -9,22 +9,19 @@ module Pbt
|
|
9
9
|
:num_runs,
|
10
10
|
:seed,
|
11
11
|
:thread_report_on_exception,
|
12
|
-
:experimental_ractor_rspec_integration,
|
13
12
|
keyword_init: true
|
14
13
|
) do
|
15
14
|
# @param verbose [Boolean] Whether to print verbose output. Default is `false`.
|
16
|
-
# @param worker [Symbol] The concurrency method to use. :ractor
|
15
|
+
# @param worker [Symbol] The concurrency method to use. :ractor` and `:none` are supported. Default is `:none`.
|
17
16
|
# @param num_runs [Integer] The number of runs to perform. Default is `100`.
|
18
17
|
# @param seed [Integer] The seed to use for random number generation. It's useful to reproduce failed test with the seed you'd pick up from failure messages. Default is a random seed.
|
19
18
|
# @param thread_report_on_exception [Boolean] Whether to report exceptions in threads. It's useful to suppress error logs on Ractor that reports many errors. Default is `false`.
|
20
|
-
# @param experimental_ractor_rspec_integration [Boolean] Whether to allow RSpec expectation and matchers in Ractor. It's quite experimental! Default is `false`.
|
21
19
|
def initialize(
|
22
20
|
verbose: false,
|
23
21
|
worker: :none,
|
24
22
|
num_runs: 100,
|
25
|
-
seed: Random.
|
26
|
-
thread_report_on_exception: false
|
27
|
-
experimental_ractor_rspec_integration: false
|
23
|
+
seed: Random.new_seed,
|
24
|
+
thread_report_on_exception: false
|
28
25
|
)
|
29
26
|
super
|
30
27
|
end
|
@@ -54,8 +54,6 @@ module Pbt
|
|
54
54
|
#
|
55
55
|
# - `:thread_report_on_exception`
|
56
56
|
# So many exception reports happen in Ractor and a console gets too messy. Suppress them to avoid that.
|
57
|
-
# - `:experimental_ractor_rspec_integration`
|
58
|
-
# Allow to use Ractor with RSpec. This is an experimental feature and it's not stable.
|
59
57
|
#
|
60
58
|
# @param config [Hash] Configuration parameters.
|
61
59
|
# @param property [Property]
|
@@ -64,21 +62,12 @@ module Pbt
|
|
64
62
|
if config[:worker] == :ractor
|
65
63
|
original_report_on_exception = Thread.report_on_exception
|
66
64
|
Thread.report_on_exception = config[:thread_report_on_exception]
|
67
|
-
|
68
|
-
if config[:experimental_ractor_rspec_integration]
|
69
|
-
require "pbt/check/rspec_adapter/integration"
|
70
|
-
class << property
|
71
|
-
include Pbt::Check::RSpecAdapter::PropertyExtension
|
72
|
-
end
|
73
|
-
property.setup_rspec_integration
|
74
|
-
end
|
75
65
|
end
|
76
66
|
|
77
67
|
yield
|
78
68
|
ensure
|
79
69
|
if config[:worker] == :ractor
|
80
70
|
Thread.report_on_exception = original_report_on_exception
|
81
|
-
property.teardown_rspec_integration if config[:experimental_ractor_rspec_integration]
|
82
71
|
end
|
83
72
|
end
|
84
73
|
|
@@ -96,10 +85,6 @@ module Pbt
|
|
96
85
|
run_it_in_sequential(property, runner)
|
97
86
|
in :ractor
|
98
87
|
run_it_in_ractors(property, runner)
|
99
|
-
in :process
|
100
|
-
run_it_in_processes(property, runner)
|
101
|
-
in :thread
|
102
|
-
run_it_in_threads(property, runner)
|
103
88
|
end
|
104
89
|
end
|
105
90
|
runner.run_execution
|
@@ -133,80 +118,11 @@ module Pbt
|
|
133
118
|
c.ractor.take
|
134
119
|
runner.handle_result(c)
|
135
120
|
rescue => e
|
136
|
-
|
121
|
+
c.exception = e.cause # Ractor error is wrapped in a Ractor::RemoteError. We need to get the cause.
|
137
122
|
runner.handle_result(c)
|
138
123
|
break # Ignore the rest of the cases. Just pick up the first failure.
|
139
124
|
end
|
140
125
|
end
|
141
|
-
|
142
|
-
def handle_ractor_error(cause, c)
|
143
|
-
# Ractor error is wrapped in a Ractor::RemoteError. We need to get the cause.
|
144
|
-
unless defined?(Pbt::Check::RSpecAdapter) && cause.is_a?(Pbt::Check::RSpecAdapter::ExpectationNotMet) # Unknown error.
|
145
|
-
c.exception = cause
|
146
|
-
return
|
147
|
-
end
|
148
|
-
|
149
|
-
# Convert Pbt's custom error to RSpec's error.
|
150
|
-
begin
|
151
|
-
RSpec::Expectations::ExpectationHelper.handle_failure(cause.matcher, cause.custom_message, cause.failure_message_method)
|
152
|
-
rescue RSpec::Expectations::ExpectationNotMetError => e # The class inherits Exception, not StandardError.
|
153
|
-
c.exception = e
|
154
|
-
end
|
155
|
-
end
|
156
|
-
|
157
|
-
# @param property [Property] Property to test.
|
158
|
-
# @param runner [RunnerIterator]
|
159
|
-
# @return [void]
|
160
|
-
def run_it_in_threads(property, runner)
|
161
|
-
require_parallel
|
162
|
-
|
163
|
-
Parallel.map_with_index(runner, in_threads: Parallel.processor_count) do |val, index|
|
164
|
-
Case.new(val:, index:).tap do |c|
|
165
|
-
property.run(val)
|
166
|
-
# Catch all exceptions including RSpec's ExpectationNotMet (It inherits Exception).
|
167
|
-
rescue Exception => e # standard:disable Lint/RescueException:
|
168
|
-
c.exception = e
|
169
|
-
# It's possible to break this loop here by raising `Parallel::Break`.
|
170
|
-
# But if it raises, we cannot fetch all cases' result. So this loop continues until the end.
|
171
|
-
end
|
172
|
-
end.each do |c|
|
173
|
-
runner.handle_result(c)
|
174
|
-
break if c.exception
|
175
|
-
# Ignore the rest of the cases. Just pick up the first failure.
|
176
|
-
end
|
177
|
-
end
|
178
|
-
|
179
|
-
# @param property [Property] Property to test.
|
180
|
-
# @param runner [RunnerIterator]
|
181
|
-
# @return [void]
|
182
|
-
def run_it_in_processes(property, runner)
|
183
|
-
require_parallel
|
184
|
-
|
185
|
-
Parallel.map_with_index(runner, in_processes: Parallel.processor_count) do |val, index|
|
186
|
-
Case.new(val:, index:).tap do |c|
|
187
|
-
property.run(val)
|
188
|
-
# Catch all exceptions including RSpec's ExpectationNotMet (It inherits Exception).
|
189
|
-
rescue Exception => e # standard:disable Lint/RescueException:
|
190
|
-
c.exception = e
|
191
|
-
# It's possible to break this loop here by raising `Parallel::Break`.
|
192
|
-
# But if it raises, we cannot fetch all cases' result. So this loop continues until the end.
|
193
|
-
end
|
194
|
-
end.each do |c|
|
195
|
-
runner.handle_result(c)
|
196
|
-
break if c.exception
|
197
|
-
# Ignore the rest of the cases. Just pick up the first failure.
|
198
|
-
end
|
199
|
-
end
|
200
|
-
|
201
|
-
# Load Parallel gem. If it's not installed, raise an error.
|
202
|
-
# @see https://github.com/grosser/parallel
|
203
|
-
# @raise [InvalidConfiguration]
|
204
|
-
def require_parallel
|
205
|
-
require "parallel"
|
206
|
-
rescue LoadError
|
207
|
-
raise InvalidConfiguration,
|
208
|
-
"Parallel gem (https://github.com/grosser/parallel) is required to use worker `:process` or `:thread`. Please add `gem 'parallel'` to your Gemfile."
|
209
|
-
end
|
210
126
|
end
|
211
127
|
end
|
212
128
|
end
|
data/lib/pbt/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: pbt
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.5.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- ohbarye
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2024-
|
11
|
+
date: 2024-12-30 00:00:00.000000000 Z
|
12
12
|
dependencies: []
|
13
13
|
description:
|
14
14
|
email:
|
@@ -46,9 +46,6 @@ files:
|
|
46
46
|
- lib/pbt/check/case.rb
|
47
47
|
- lib/pbt/check/configuration.rb
|
48
48
|
- lib/pbt/check/property.rb
|
49
|
-
- lib/pbt/check/rspec_adapter/integration.rb
|
50
|
-
- lib/pbt/check/rspec_adapter/predicate_block_inspector.rb
|
51
|
-
- lib/pbt/check/rspec_adapter/property_extension.rb
|
52
49
|
- lib/pbt/check/runner_iterator.rb
|
53
50
|
- lib/pbt/check/runner_methods.rb
|
54
51
|
- lib/pbt/check/tosser.rb
|
@@ -79,7 +76,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
79
76
|
- !ruby/object:Gem::Version
|
80
77
|
version: '0'
|
81
78
|
requirements: []
|
82
|
-
rubygems_version: 3.5.
|
79
|
+
rubygems_version: 3.5.21
|
83
80
|
signing_key:
|
84
81
|
specification_version: 4
|
85
82
|
summary: Property-Based Testing tool for Ruby, utilizing Ractor for parallelizing
|
@@ -1,64 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
unless defined?(RSpec)
|
4
|
-
raise InvalidConfigurationError, "You configured `experimental_ractor_rspec_integration: true` but RSpec is not loaded. Please use RSpec or set the config `false`."
|
5
|
-
end
|
6
|
-
|
7
|
-
require "pbt/check/rspec_adapter/predicate_block_inspector"
|
8
|
-
require "pbt/check/rspec_adapter/property_extension"
|
9
|
-
|
10
|
-
module Pbt
|
11
|
-
module Check
|
12
|
-
# @private
|
13
|
-
module RSpecAdapter
|
14
|
-
# This custom error contains RSpec matcher and message to handle Pbt's runner.
|
15
|
-
# @private
|
16
|
-
class ExpectationNotMet < StandardError
|
17
|
-
attr_reader :matcher, :custom_message, :failure_message_method
|
18
|
-
|
19
|
-
def initialize(msg, matcher, custom_message, failure_message_method)
|
20
|
-
super(msg)
|
21
|
-
@matcher = matcher
|
22
|
-
@custom_message = custom_message
|
23
|
-
@failure_message_method = failure_message_method
|
24
|
-
end
|
25
|
-
end
|
26
|
-
end
|
27
|
-
end
|
28
|
-
end
|
29
|
-
|
30
|
-
# `autoload` is not allowed in Ractor but RSpec uses autoload for matchers.
|
31
|
-
# We need to load them in advance in order to be able to use them in Ractor.
|
32
|
-
#
|
33
|
-
# e.g. Ractor raises... `be_a_kind_of': require by autoload on non-main Ractor is not supported (BeAKindOf) (Ractor::UnsafeError)
|
34
|
-
RSpec::Matchers::BuiltIn.constants.each { |c| Object.const_get("RSpec::Matchers::BuiltIn::#{c}") }
|
35
|
-
|
36
|
-
# TODO: preload more helpers like aggregate_failures.
|
37
|
-
# RSpec::Expectations.constants.each { |c| Object.const_get("RSpec::Expectations::#{c}") }
|
38
|
-
# The code above is not enough. Even if we run this code in advance, Ractor raises...
|
39
|
-
# in `failure_notifier': can not access non-shareable objects in constant RSpec::Support::DEFAULT_FAILURE_NOTIFIER by non-main ractor. (Ractor::IsolationError)
|
40
|
-
|
41
|
-
# CAUTION: This is a dirty hack! We need to override the original method to make it Ractor-safe.
|
42
|
-
RSpec::Expectations::ExpectationHelper.singleton_class.prepend(Module.new do
|
43
|
-
def with_matcher(handler, matcher, message)
|
44
|
-
check_message(message)
|
45
|
-
matcher = modern_matcher_from(matcher)
|
46
|
-
yield matcher
|
47
|
-
ensure
|
48
|
-
# The original method is not Ractor-safe unless stopping assigning these class variables.
|
49
|
-
if Ractor.current == Ractor.main
|
50
|
-
::RSpec::Matchers.last_expectation_handler = handler
|
51
|
-
::RSpec::Matchers.last_matcher = matcher
|
52
|
-
end
|
53
|
-
end
|
54
|
-
|
55
|
-
def handle_failure(matcher, message, failure_message_method)
|
56
|
-
# This method is not Ractor-safe. RSpec::Support::ObjectFormatter.default_instance assigns class variables.
|
57
|
-
# If this method is called in non-main-Ractor, it raises a custom error and let it handle in the main Ractor.
|
58
|
-
if Ractor.current != Ractor.main
|
59
|
-
raise Pbt::Check::RSpecAdapter::ExpectationNotMet.new(nil, matcher, message, failure_message_method)
|
60
|
-
end
|
61
|
-
|
62
|
-
super
|
63
|
-
end
|
64
|
-
end)
|
@@ -1,47 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
begin
|
4
|
-
# Use prism to get user-defined block code.
|
5
|
-
require "prism"
|
6
|
-
rescue LoadError
|
7
|
-
raise InvalidConfiguration,
|
8
|
-
"Prism gem (https://github.com/grosser/parallel) is required to use worker `:rator` and `:experimental_ractor_rspec_integration`. Please add `gem 'parallel'` to your Gemfile."
|
9
|
-
end
|
10
|
-
|
11
|
-
module Pbt
|
12
|
-
module Check
|
13
|
-
module RSpecAdapter
|
14
|
-
# This class is used to get user-defined block code.
|
15
|
-
# If a user defines code like below:
|
16
|
-
#
|
17
|
-
# Pbt.property(Pbt.integer, Pbt.integer) do |x, y|
|
18
|
-
# x > 0 && y > 0
|
19
|
-
# end
|
20
|
-
#
|
21
|
-
# inspector.method_params #=> "x, y"
|
22
|
-
# inspector.method_body #=> "x > 0 && y > 0"
|
23
|
-
#
|
24
|
-
# @private
|
25
|
-
# @!attribute [r] method_body
|
26
|
-
# @!attribute [r] method_params
|
27
|
-
class PredicateBlockInspector < Prism::Visitor
|
28
|
-
attr_reader :method_body, :method_params
|
29
|
-
|
30
|
-
def initialize(line)
|
31
|
-
@line = line
|
32
|
-
@method_body = nil
|
33
|
-
super()
|
34
|
-
end
|
35
|
-
|
36
|
-
def visit_call_node(node)
|
37
|
-
if node.name == :property && node.block.opening_loc.start_line == @line
|
38
|
-
@method_params = node.block.parameters.parameters.slice
|
39
|
-
@method_body = node.block.body.slice
|
40
|
-
end
|
41
|
-
|
42
|
-
super
|
43
|
-
end
|
44
|
-
end
|
45
|
-
end
|
46
|
-
end
|
47
|
-
end
|
@@ -1,74 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
module Pbt
|
4
|
-
module Check
|
5
|
-
module RSpecAdapter
|
6
|
-
# @private
|
7
|
-
module PropertyExtension
|
8
|
-
# Define an original class to be called in Ractor.
|
9
|
-
#
|
10
|
-
# @return [void]
|
11
|
-
def setup_rspec_integration
|
12
|
-
filepath, line = @predicate.source_location
|
13
|
-
basename = File.basename(filepath, ".rb")
|
14
|
-
@class_name = "Test" + basename.split("_").map(&:capitalize).join + line.to_s
|
15
|
-
@method_name = "predicate_#{basename}_#{line}"
|
16
|
-
define_ractor_callable_class
|
17
|
-
end
|
18
|
-
|
19
|
-
# Clean up an original class to be called in Ractor to avoid any persisted namespace pollution.
|
20
|
-
#
|
21
|
-
# @return [void]
|
22
|
-
def teardown_rspec_integration
|
23
|
-
RSpecAdapter.__send__(:remove_const, @class_name) if RSpecAdapter.const_defined?(@class_name)
|
24
|
-
end
|
25
|
-
|
26
|
-
# Run the predicate with the generated `val`.
|
27
|
-
# This overrides the original `Property#run_in_ractor`.
|
28
|
-
#
|
29
|
-
# @param val [Object]
|
30
|
-
# @return [Ractor]
|
31
|
-
def run_in_ractor(val)
|
32
|
-
Ractor.new(@class_name, @method_name, @predicate.parameters.size, val) do |class_name, method_name, param_size, val|
|
33
|
-
klass = RSpecAdapter.const_get(class_name)
|
34
|
-
if val.is_a?(Hash)
|
35
|
-
klass.new.send(method_name, **val)
|
36
|
-
elsif param_size >= 2
|
37
|
-
klass.new.send(method_name, *val)
|
38
|
-
else
|
39
|
-
klass.new.send(method_name, val)
|
40
|
-
end
|
41
|
-
end
|
42
|
-
end
|
43
|
-
|
44
|
-
private
|
45
|
-
|
46
|
-
# @return [void]
|
47
|
-
def define_ractor_callable_class
|
48
|
-
# The @method_name is invisible in the Class.new block, so we need to assign it to a local variable.
|
49
|
-
method_name = @method_name
|
50
|
-
|
51
|
-
inspector = extract_predicate_source_code
|
52
|
-
|
53
|
-
RSpecAdapter.const_set(@class_name, Class.new do
|
54
|
-
include ::RSpec::Matchers
|
55
|
-
class_eval <<-RUBY, __FILE__, __LINE__ + 1
|
56
|
-
def #{method_name}(#{inspector.method_params})
|
57
|
-
#{inspector.method_body}
|
58
|
-
end
|
59
|
-
RUBY
|
60
|
-
end)
|
61
|
-
end
|
62
|
-
|
63
|
-
# @return [PredicateBlockInspector]
|
64
|
-
def extract_predicate_source_code
|
65
|
-
filepath, line = @predicate.source_location
|
66
|
-
PredicateBlockInspector.new(line).tap do |inspector|
|
67
|
-
res = Prism.parse_file(filepath)
|
68
|
-
res.value.statements.accept(inspector)
|
69
|
-
end
|
70
|
-
end
|
71
|
-
end
|
72
|
-
end
|
73
|
-
end
|
74
|
-
end
|