pbt 0.6.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.standard.yml +1 -1
- data/CHANGELOG.md +21 -0
- data/CLAUDE.md +2 -2
- data/README.md +116 -36
- data/benchmark/FIBER.md +97 -0
- data/benchmark/fiber_experiment.rb +39 -0
- data/benchmark/fiber_scheduler/Gemfile +7 -0
- data/benchmark/fiber_scheduler/experiment.rb +131 -0
- data/benchmark/fiber_scheduler/spec/experiment_spec.rb +38 -0
- data/docs/plans/fiber-experiment.md +12 -0
- data/docs/plans/generator-statistics.md +16 -0
- data/docs/plans/stateful-pbt-redesign.md +415 -0
- data/lib/pbt/arbitrary/arbitrary.rb +65 -12
- data/lib/pbt/arbitrary/arbitrary_methods.rb +51 -24
- data/lib/pbt/arbitrary/array_arbitrary.rb +46 -12
- data/lib/pbt/arbitrary/choose_arbitrary.rb +28 -1
- data/lib/pbt/arbitrary/constant.rb +0 -1
- data/lib/pbt/arbitrary/constant_arbitrary.rb +6 -1
- data/lib/pbt/arbitrary/constant_from_arbitrary.rb +39 -0
- data/lib/pbt/arbitrary/filter_arbitrary.rb +41 -3
- data/lib/pbt/arbitrary/fixed_hash_arbitrary.rb +17 -1
- data/lib/pbt/arbitrary/float_arbitrary.rb +31 -0
- data/lib/pbt/arbitrary/frequency_arbitrary.rb +61 -0
- data/lib/pbt/arbitrary/integer_arbitrary.rb +12 -1
- data/lib/pbt/arbitrary/map_arbitrary.rb +41 -3
- data/lib/pbt/arbitrary/one_of_arbitrary.rb +47 -14
- data/lib/pbt/arbitrary/sample.rb +97 -0
- data/lib/pbt/arbitrary/tuple_arbitrary.rb +23 -1
- data/lib/pbt/check/case.rb +1 -1
- data/lib/pbt/check/configuration.rb +5 -3
- data/lib/pbt/check/property.rb +24 -5
- data/lib/pbt/check/runner_iterator.rb +84 -10
- data/lib/pbt/check/runner_methods.rb +47 -17
- data/lib/pbt/check/tosser.rb +1 -1
- data/lib/pbt/reporter/run_details.rb +2 -2
- data/lib/pbt/reporter/run_details_reporter.rb +13 -0
- data/lib/pbt/reporter/run_execution.rb +47 -4
- data/lib/pbt/stateful/property.rb +8 -102
- data/lib/pbt/statistics.rb +28 -0
- data/lib/pbt/version.rb +1 -1
- data/lib/pbt.rb +45 -9
- metadata +16 -3
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: b56a720a974f89209c3237c6b7a57adde668ebccbd77c76a51c882d1b5afd9c3
|
|
4
|
+
data.tar.gz: cae99f6c6d1b0d3b0de2a5ec00ca4544c7899e318c3ccaf8ec42da724d0cdb42
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 0e366c7d57f1428e240ce4faf92c0e0f302b0423d446af18610b81518a4a8f4528c8bdc0a3b8fc50a9e713979278520ba54cefbf430983185431f00e58cfd6ad
|
|
7
|
+
data.tar.gz: 87835150a4af3bcdbab40fb041d07103fbe3deb5a92d4980631f3485ae3aa89753a983908136d18f7224bf8b85db4e1d9fadede51cacc94471cbf8adb18b285f
|
data/.standard.yml
CHANGED
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,26 @@
|
|
|
1
1
|
## [Unreleased]
|
|
2
2
|
|
|
3
|
+
## [0.8.0] - 2026-11-22
|
|
4
|
+
|
|
5
|
+
- [Breaking change] Drop support for Ruby 3.2 and earlier (EOL); Ruby 3.3 or later is now required, and Ruby 4.1.0-preview1 is added to CI [#66](https://github.com/ohbarye/pbt/pull/66)
|
|
6
|
+
- [Breaking change] Split literal constant selection into `Pbt.constant_from(*choices)` and change `Pbt.one_of(*arbitraries)` to select uniformly among arbitraries with contextual shrinking [#70](https://github.com/ohbarye/pbt/pull/70)
|
|
7
|
+
- Add `Pbt.frequency` weighted generators with contextual lazy shrink trees (`Arbitrary::Sample`) [#61](https://github.com/ohbarye/pbt/pull/61)
|
|
8
|
+
- Add `Pbt.statistics` for reproducible generator distribution inspection [#62](https://github.com/ohbarye/pbt/pull/62)
|
|
9
|
+
- Allow `Arbitrary#map` without an `unmapper` and with block syntax (`arb.map { |v| ... }`), shrinking via `Sample` provenance [#71](https://github.com/ohbarye/pbt/pull/71)
|
|
10
|
+
- Add 3-valued `Arbitrary#cover?(value)` (including `FloatArbitrary` for `Pbt.float`) and `RunnerIterator` `same_failure?` / `shrink_case` hooks [#72](https://github.com/ohbarye/pbt/pull/72)
|
|
11
|
+
- Document v1 Property DSL contract and fix positional Hash / strict lambda argument passing in `Check::Property#run` [#73](https://github.com/ohbarye/pbt/pull/73)
|
|
12
|
+
- Re-raise fatal exceptions (`SystemExit`, `SignalException`, `NoMemoryError`) in runners without shrinking [#74](https://github.com/ohbarye/pbt/pull/74)
|
|
13
|
+
- Bound `FilterArbitrary` generation retries with `max_attempts:` (raising `Pbt::Arbitrary::FilterExhaustedError`) to prevent infinite loops [#75](https://github.com/ohbarye/pbt/pull/75)
|
|
14
|
+
- Improve array shrinking (continue element shrinking at `min_length` and support non-contiguous element deletion) and fix `Pbt.nat` default `max` [#76](https://github.com/ohbarye/pbt/pull/76)
|
|
15
|
+
- Add `Pbt.assume` precondition helper with `max_skips` guard (`Pbt::TooManySkipsError`) and `RunDetails#num_skips` [#77](https://github.com/ohbarye/pbt/pull/77)
|
|
16
|
+
- Note: the current `Pbt.stateful(model:, sut:)` API is planned to be replaced with a redesigned DSL in v0.9.0 ([#67](https://github.com/ohbarye/pbt/issues/67)).
|
|
17
|
+
|
|
18
|
+
## [0.7.0] - 2026-04-04
|
|
19
|
+
|
|
20
|
+
- [Breaking change] Simplify stateful command protocol: `arguments` must now accept `state` parameter, and `applicable?` must now accept both `state` and `args` parameters
|
|
21
|
+
- Make `rng` optional in all arbitrary `generate` methods with default `Random.new` [#44](https://github.com/ohbarye/pbt/pull/44)
|
|
22
|
+
- Mark stateful testing API as stable (no longer experimental)
|
|
23
|
+
|
|
3
24
|
## [0.6.0] - 2026-03-15
|
|
4
25
|
|
|
5
26
|
- Add experimental `Pbt.stateful` API for model-based stateful property testing [#38](https://github.com/ohbarye/pbt/pull/38)
|
data/CLAUDE.md
CHANGED
|
@@ -98,7 +98,7 @@ The codebase uses RSpec for testing and follows these patterns:
|
|
|
98
98
|
|
|
99
99
|
### Important Notes
|
|
100
100
|
|
|
101
|
-
- Ruby 3.
|
|
101
|
+
- Ruby 3.3+ is required
|
|
102
102
|
- When using `:ractor` worker, be aware of Ractor limitations (no shared state, limited object sharing)
|
|
103
103
|
- The gem uses Standard Ruby for code style (configured in `.standard.yml`)
|
|
104
|
-
- CI runs tests against Ruby 3.
|
|
104
|
+
- CI runs tests against Ruby 3.3, 3.4, 4.0, and 4.1.0-preview1
|
data/README.md
CHANGED
|
@@ -28,13 +28,15 @@ For a more in-depth understanding of Property-Based Testing, please refer to ext
|
|
|
28
28
|
|
|
29
29
|
## Installation
|
|
30
30
|
|
|
31
|
+
Ruby 3.3 or later is required. CI tests Ruby 3.3, 3.4, 4.0, and 4.1.0-preview1.
|
|
32
|
+
|
|
31
33
|
Add this line to your application's Gemfile and run `bundle install`.
|
|
32
34
|
|
|
33
35
|
```ruby
|
|
34
36
|
gem 'pbt'
|
|
35
37
|
```
|
|
36
38
|
|
|
37
|
-
|
|
39
|
+
Of course you can install with `gem install pbt`.
|
|
38
40
|
|
|
39
41
|
## Basic Usage
|
|
40
42
|
|
|
@@ -77,16 +79,36 @@ The above snippet is very simple but contains the basic components.
|
|
|
77
79
|
|
|
78
80
|
#### Runner
|
|
79
81
|
|
|
80
|
-
`Pbt.assert` is the runner.
|
|
82
|
+
`Pbt.assert` is the primary runner. It executes the property returned by the block multiple times, shrinks any failure, and raises `Pbt::PropertyFailure` with reproduction details.
|
|
83
|
+
|
|
84
|
+
If you want to inspect predicate failures without raising `Pbt::PropertyFailure`, use `Pbt.check`, which returns a `Pbt::Reporter::RunDetails` struct (`failed`, `num_runs`, `num_skips`, `num_shrinks`, `seed`, `counterexample`, `counterexample_path`, `error_message`, `error_instance`, `failures`, `verbose`, `execution_summary`, `run_configuration`). Exceptions raised during property construction or value generation still propagate to the caller.
|
|
85
|
+
|
|
86
|
+
#### Preconditions (`Pbt.assume`)
|
|
87
|
+
|
|
88
|
+
Use `Pbt.assume(condition)` inside a predicate to discard inputs that do not satisfy a precondition instead of returning early (which would count skipped inputs as passed runs):
|
|
89
|
+
|
|
90
|
+
```ruby
|
|
91
|
+
Pbt.assert do
|
|
92
|
+
Pbt.property(Pbt.integer, Pbt.integer) do |x, y|
|
|
93
|
+
Pbt.assume(y != 0)
|
|
94
|
+
raise "unexpected" unless (x / y) * y + (x % y) == x
|
|
95
|
+
end
|
|
96
|
+
end
|
|
97
|
+
```
|
|
81
98
|
|
|
82
|
-
|
|
99
|
+
Discarded cases do not count toward `num_runs` and are skipped during shrinking. If skips exceed `max_skips` (default: `num_runs * 10`), `Pbt.assert` raises `Pbt::TooManySkipsError` (`< Pbt::PropertyFailure`). Prefer narrowing your arbitrary when a precondition rejects most inputs.
|
|
83
100
|
|
|
84
|
-
|
|
101
|
+
#### Property (v1 DSL Contract)
|
|
85
102
|
|
|
86
|
-
|
|
87
|
-
2. How to generate inputs for the predicate — using `Arbitrary`
|
|
103
|
+
`Pbt` intentionally uses explicit runner/property composition (`Pbt.assert` / `Pbt.check` wrapping `Pbt.property` or `Pbt.stateful`):
|
|
88
104
|
|
|
89
|
-
|
|
105
|
+
1. Runner options (`seed`, `num_runs`, `verbose`, `worker`) stay separated from generated inputs.
|
|
106
|
+
2. Predicates fail by **raising** (including RSpec `expect(...)` and Minitest assertions under default `worker: :none`). Returning `false` alone does not fail a property.
|
|
107
|
+
3. Argument passing rules:
|
|
108
|
+
- Single arbitrary: passed directly (`Pbt.property(Pbt.integer) { |n| ... }`)
|
|
109
|
+
- Multiple positional arbitraries: wrapped as a tuple (`Pbt.property(Pbt.integer, Pbt.symbol) { |n, sym| ... }`)
|
|
110
|
+
- Keyword arbitraries: wrapped as a fixed hash and passed as keyword arguments under `worker: :none` (`Pbt.property(left: Pbt.integer, right: Pbt.integer) { |left:, right:| ... }`). Under `worker: :ractor`, accept a single hash parameter (`{ |h| ... }`).
|
|
111
|
+
- Mixing positional and keyword arbitraries in one `Pbt.property` call raises `ArgumentError`.
|
|
90
112
|
|
|
91
113
|
#### Arbitrary
|
|
92
114
|
|
|
@@ -108,44 +130,98 @@ There are many built-in arbitraries in `Pbt`. You can use them to generate rando
|
|
|
108
130
|
#### Primitives
|
|
109
131
|
|
|
110
132
|
```ruby
|
|
111
|
-
|
|
133
|
+
Pbt.integer.generate # => 42
|
|
134
|
+
Pbt.integer(min: -1, max: 8).generate # => Integer between -1 and 8
|
|
112
135
|
|
|
113
|
-
Pbt.
|
|
114
|
-
Pbt.integer(min: -1, max: 8).generate(rng) # => Integer between -1 and 8
|
|
136
|
+
Pbt.symbol.generate # => :atq
|
|
115
137
|
|
|
116
|
-
Pbt.
|
|
138
|
+
Pbt.ascii_char.generate # => "a"
|
|
139
|
+
Pbt.ascii_string.generate # => "aagjZfao"
|
|
140
|
+
|
|
141
|
+
Pbt.boolean.generate # => true or false
|
|
142
|
+
Pbt.constant(42).generate # => 42 always
|
|
143
|
+
```
|
|
117
144
|
|
|
118
|
-
|
|
119
|
-
Pbt.ascii_string.generate(rng) # => "aagjZfao"
|
|
145
|
+
You can also pass a custom random number generator if needed:
|
|
120
146
|
|
|
121
|
-
|
|
122
|
-
|
|
147
|
+
```ruby
|
|
148
|
+
rng = Random.new(42) # with a specific seed for reproducibility
|
|
149
|
+
Pbt.integer.generate(rng)
|
|
123
150
|
```
|
|
124
151
|
|
|
125
152
|
#### Composites
|
|
126
153
|
|
|
127
154
|
```ruby
|
|
128
|
-
|
|
155
|
+
Pbt.array(Pbt.integer).generate # => [121, -13141, 9825]
|
|
156
|
+
Pbt.array(Pbt.integer, max: 1, empty: true).generate # => [] or [42] etc.
|
|
129
157
|
|
|
130
|
-
Pbt.
|
|
131
|
-
Pbt.array(Pbt.integer, max: 1, empty: true).generate(rng) # => [] or [42] etc.
|
|
158
|
+
Pbt.tuple(Pbt.symbol, Pbt.integer).generate # => [:atq, 42]
|
|
132
159
|
|
|
133
|
-
Pbt.
|
|
160
|
+
Pbt.fixed_hash(x: Pbt.symbol, y: Pbt.integer).generate # => {x: :atq, y: 42}
|
|
161
|
+
Pbt.hash(Pbt.symbol, Pbt.integer).generate # => {atq: 121, ygab: -1142}
|
|
134
162
|
|
|
135
|
-
Pbt.
|
|
136
|
-
Pbt.
|
|
163
|
+
Pbt.constant_from(:a, 1, 0.1).generate # => :a or 1 or 0.1
|
|
164
|
+
Pbt.one_of(Pbt.integer, Pbt.constant(nil)).generate # => 42 or nil
|
|
137
165
|
|
|
138
|
-
Pbt.
|
|
166
|
+
Pbt.integer(min: 0, max: 10).map { |n| n * 2 }.generate # => 8
|
|
167
|
+
Pbt.integer(min: 0, max: 10).filter(&:even?).generate # => 4
|
|
139
168
|
````
|
|
140
169
|
|
|
141
170
|
See [ArbitraryMethods](https://github.com/ohbarye/pbt/blob/main/lib/pbt/arbitrary/arbitrary_methods.rb) module for more details.
|
|
142
171
|
|
|
143
|
-
|
|
172
|
+
### Weighted generators
|
|
173
|
+
|
|
174
|
+
Use `Pbt.frequency` to select generators by relative nonnegative integer weights:
|
|
144
175
|
|
|
145
|
-
|
|
176
|
+
```ruby
|
|
177
|
+
Pbt.assert(seed: 42) do
|
|
178
|
+
Pbt.property(Pbt.frequency([3, Pbt.integer], [1, Pbt.constant(nil)])) do |value|
|
|
179
|
+
raise "unexpected value" unless value.nil? || value.is_a?(Integer)
|
|
180
|
+
end
|
|
181
|
+
end
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
At least one weight must be positive. Zero-weight generators are never selected.
|
|
185
|
+
Normal property execution retains the selected generator's shrink tree, including
|
|
186
|
+
inside arrays, tuples, fixed hashes, maps and filters. Shrinking stays within that
|
|
187
|
+
branch. Generated values and reported counterexamples remain ordinary Ruby values.
|
|
188
|
+
|
|
189
|
+
For manual shrinking, retain a sample: `sample = arb.sample(Random.new(42))`,
|
|
190
|
+
then inspect `sample.value` and `sample.shrink.map(&:value)`. Calling
|
|
191
|
+
`arb.shrink(value)` directly on `Pbt.frequency`, `Pbt.one_of`, or `Arbitrary#map`
|
|
192
|
+
without an `unmapper` returns no candidates because a bare value cannot identify its
|
|
193
|
+
source branch or preimage. The current `Pbt.stateful` command arguments use the
|
|
194
|
+
plain-value protocol, so those contextual generators do not shrink command arguments there.
|
|
195
|
+
|
|
196
|
+
### Generator statistics
|
|
197
|
+
|
|
198
|
+
Inspect generated values before running a property:
|
|
199
|
+
|
|
200
|
+
```ruby
|
|
201
|
+
stats = Pbt.statistics(Pbt.integer, num_runs: 1000, seed: 42) do |number|
|
|
202
|
+
number.negative? ? :negative : :nonnegative
|
|
203
|
+
end
|
|
204
|
+
stats.counts # => counts grouped by label
|
|
205
|
+
stats.percentages # => unrounded percentages grouped by label
|
|
206
|
+
stats.seed # => 42; reuse this seed to replay the sample
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
Without a block, actual generated values are counted. Each call generates exactly
|
|
210
|
+
`num_runs` values using its own seeded RNG; it does not execute a property or shrink
|
|
211
|
+
values. The default is 100 runs with a fresh seed, retained in the result. Zero runs
|
|
212
|
+
return empty counts and percentages. Counts are a mutable Hash local to the result;
|
|
213
|
+
values or labels used as its keys must remain stable. For generators that reuse mutable
|
|
214
|
+
objects, return a stable label from the block (for example, a symbol or integer).
|
|
215
|
+
Generator and classifier errors propagate to the caller.
|
|
216
|
+
|
|
217
|
+
## Stateful Testing
|
|
218
|
+
|
|
219
|
+
`Pbt` also provides a stateful property API for model-based / command-based testing.
|
|
146
220
|
It is designed as a property-compatible object (`generate`, `shrink`, `run`) so it works with the existing runner (`Pbt.assert` / `Pbt.check`) without changing the runner API.
|
|
147
221
|
|
|
148
|
-
|
|
222
|
+
> **Note:** The current `Pbt.stateful(model:, sut:)` protocol is planned to be replaced with a cleaner yield-builder DSL and improved shrinking engine in v0.9.0 before v1.0.0. See [#67](https://github.com/ohbarye/pbt/issues/67) and [docs/plans/stateful-pbt-redesign.md](docs/plans/stateful-pbt-redesign.md).
|
|
223
|
+
|
|
224
|
+
Commands currently implement `arguments(state)` and `applicable?(state, args)`; the earlier zero-argument / state-only forms are no longer supported.
|
|
149
225
|
|
|
150
226
|
### Minimal usage
|
|
151
227
|
|
|
@@ -169,11 +245,11 @@ class IncrementCommand
|
|
|
169
245
|
:increment
|
|
170
246
|
end
|
|
171
247
|
|
|
172
|
-
def arguments
|
|
248
|
+
def arguments(_state)
|
|
173
249
|
Pbt.nil
|
|
174
250
|
end
|
|
175
251
|
|
|
176
|
-
def applicable?(_state)
|
|
252
|
+
def applicable?(_state, _args)
|
|
177
253
|
true
|
|
178
254
|
end
|
|
179
255
|
|
|
@@ -220,18 +296,18 @@ end
|
|
|
220
296
|
- `model.initial_state`
|
|
221
297
|
- `model.commands(state)` -> `Array<command>`
|
|
222
298
|
- `command.name`
|
|
223
|
-
- `command.arguments` (a `Pbt` arbitrary)
|
|
224
|
-
- `command.applicable?(state)` -> `true` / `false`
|
|
299
|
+
- `command.arguments(state)` (a `Pbt` arbitrary, may depend on current model state)
|
|
300
|
+
- `command.applicable?(state, args)` -> `true` / `false`
|
|
225
301
|
- `command.next_state(state, args)` -> next model state
|
|
226
302
|
- `command.run!(sut, args)` -> command result
|
|
227
303
|
- `command.verify!(before_state:, after_state:, args:, result:, sut:)`
|
|
228
304
|
|
|
229
|
-
### Current limitations
|
|
305
|
+
### Current limitations
|
|
230
306
|
|
|
231
307
|
- `Pbt.stateful` runs sequentially by default, even if the global worker is `:ractor`.
|
|
232
308
|
- You can still pass `worker: :none` explicitly if you want to make that choice obvious in a test.
|
|
233
309
|
- `worker: :ractor` is currently unsupported and raises `Pbt::InvalidConfiguration`.
|
|
234
|
-
- Shrinking supports shorter prefixes and command-argument shrinking (using `command.arguments.shrink(args)`).
|
|
310
|
+
- Shrinking supports shorter prefixes and command-argument shrinking (using `command.arguments(state).shrink(args)`).
|
|
235
311
|
|
|
236
312
|
## What if property-based tests fail?
|
|
237
313
|
|
|
@@ -324,6 +400,10 @@ Pbt.configure do |config|
|
|
|
324
400
|
# Whether to report exceptions in threads.
|
|
325
401
|
# It's useful to suppress error logs on Ractor that reports many errors. Default is `false`.
|
|
326
402
|
config.thread_report_on_exception = false
|
|
403
|
+
|
|
404
|
+
# Maximum number of skipped cases allowed via `Pbt.assume`.
|
|
405
|
+
# Defaults to `num_runs * 10` when `nil`.
|
|
406
|
+
config.max_skips = nil
|
|
327
407
|
end
|
|
328
408
|
```
|
|
329
409
|
|
|
@@ -417,11 +497,11 @@ Once this project finishes the following, we will release v1.0.0.
|
|
|
417
497
|
- [x] Rich report by verbose mode
|
|
418
498
|
- [x] (Partially) Allow to use expectations and matchers provided by test framework in Ractor. (dropped)
|
|
419
499
|
- It'd be so hard to pass assertions like `expect`, `assert` to a Ractor.
|
|
420
|
-
- [
|
|
421
|
-
- [
|
|
422
|
-
- [
|
|
423
|
-
- [
|
|
424
|
-
- [
|
|
500
|
+
- [x] Implement frequency arbitrary
|
|
501
|
+
- [x] Statistics feature to aggregate generated values
|
|
502
|
+
- [x] Decide DSL (explicit `Pbt.assert` / `Pbt.check` + `Pbt.property` / `Pbt.stateful` composition)
|
|
503
|
+
- [x] Try Fiber ([experiment and v1 decision](benchmark/FIBER.md))
|
|
504
|
+
- [x] Stateful property-based testing (via `Pbt.stateful`)
|
|
425
505
|
|
|
426
506
|
## Development
|
|
427
507
|
|
data/benchmark/FIBER.md
ADDED
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
# Fiber predicate experiment
|
|
2
|
+
|
|
3
|
+
Run from the repository root:
|
|
4
|
+
|
|
5
|
+
```sh
|
|
6
|
+
bundle exec ruby benchmark/fiber_experiment.rb
|
|
7
|
+
NUM_RUNS=1000 bundle exec ruby benchmark/fiber_experiment.rb
|
|
8
|
+
```
|
|
9
|
+
|
|
10
|
+
This compares the normal sequential runner with the same runner wrapping each
|
|
11
|
+
predicate call in a newly created Fiber. Both use seed 42, the same number of
|
|
12
|
+
cases, and the same CPU / blocking-sleep predicates. The script checks successful
|
|
13
|
+
case counts and measures wall time with a monotonic clock. It installs no scheduler.
|
|
14
|
+
|
|
15
|
+
## Decision for v1
|
|
16
|
+
|
|
17
|
+
Keep `worker: :none` as the default and `:ractor` as the existing opt-in worker.
|
|
18
|
+
The bare-Fiber experiment is complete; it does not justify a new public worker.
|
|
19
|
+
A Fiber per case adds allocation/context-switching work, while blocking sleep
|
|
20
|
+
still blocks its thread. This is a baseline experiment, not a benchmark of a
|
|
21
|
+
scheduler-backed asynchronous worker or evidence against all Fiber designs.
|
|
22
|
+
|
|
23
|
+
A local smoke run on Ruby 4.0.9 (arm64 macOS), 100 cases, seed 42 produced:
|
|
24
|
+
|
|
25
|
+
```text
|
|
26
|
+
cpu sequential 0.001758 seconds
|
|
27
|
+
cpu fiber 0.002105 seconds
|
|
28
|
+
blocking_sleep sequential 0.126800 seconds
|
|
29
|
+
blocking_sleep fiber 0.128960 seconds
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
These short timings are illustrative and noisy, not performance guarantees.
|
|
33
|
+
Re-run on the target Ruby/platform with representative workloads before drawing
|
|
34
|
+
performance conclusions. CPU parallelism is not provided by these same-thread Fibers.
|
|
35
|
+
|
|
36
|
+
## Scheduler-backed TCP IO experiment
|
|
37
|
+
|
|
38
|
+
The separate experiment in `benchmark/fiber_scheduler/experiment.rb` uses Async's
|
|
39
|
+
scheduler. Async is an optional benchmark dependency, not a dependency of pbt:
|
|
40
|
+
|
|
41
|
+
```sh
|
|
42
|
+
BUNDLE_GEMFILE=benchmark/fiber_scheduler/Gemfile bundle install
|
|
43
|
+
BUNDLE_GEMFILE=benchmark/fiber_scheduler/Gemfile bundle exec rspec benchmark/fiber_scheduler/spec
|
|
44
|
+
BUNDLE_GEMFILE=benchmark/fiber_scheduler/Gemfile bundle exec ruby benchmark/fiber_scheduler/experiment.rb
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
Parameters are `NUM_RUNS` (100), `CONCURRENCY` (10), `IO_DELAY` (0.01 seconds),
|
|
48
|
+
and `REPEATS` (5; use an odd number). Each mode gets a separate warmup. The timer
|
|
49
|
+
includes client socket creation, connection, reads, property calls and scheduler
|
|
50
|
+
startup/cleanup, but excludes gem loading, input generation and server startup.
|
|
51
|
+
|
|
52
|
+
A dedicated server thread accepts loopback TCP connections and independently delays
|
|
53
|
+
each `ok` response by 10ms. Clients perform a real blocking `Socket#read` through
|
|
54
|
+
`TCPSocket`, which Async makes cooperative. The three modes use the same seed-42
|
|
55
|
+
integer inputs and identical protocol: serial requests, serial requests wrapped in
|
|
56
|
+
bare Fibers, and batches of at most 10 concurrent Async tasks. Batches wait for all
|
|
57
|
+
members before starting the next batch; this is not a continuously filled pool.
|
|
58
|
+
The server records its peak number of pending responses to verify that IO waits
|
|
59
|
+
actually overlap; timing thresholds are deliberately absent from tests.
|
|
60
|
+
|
|
61
|
+
Local measurement: Ruby 4.0.9, Async 2.46.0, arm64 macOS, 100 cases, seed 42,
|
|
62
|
+
10ms per response, maximum concurrency 10, 5 repetitions after warmup:
|
|
63
|
+
|
|
64
|
+
```text
|
|
65
|
+
mode elapsed seconds (all 5 samples) median seconds
|
|
66
|
+
sequential 1.232761 1.246074 1.233465 1.250332 1.226240 1.233465
|
|
67
|
+
bare Fiber 1.232813 1.231938 1.224650 1.241142 1.213324 1.231938
|
|
68
|
+
scheduler 0.124735 0.129789 0.130015 0.130826 0.128622 0.129789
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Every sample completed 100 requests and produced the same input checksum, 47379.
|
|
72
|
+
Server peak pending responses were 1 for serial/bare Fiber and 10 for Scheduler
|
|
73
|
+
in every repetition. The Scheduler median was about 9.5 times faster than serial
|
|
74
|
+
for this intentionally wait-dominated workload. This demonstrates overlapping IO
|
|
75
|
+
waits, rather than CPU parallelism. It is not a general throughput guarantee;
|
|
76
|
+
loopback, synthetic delay, small payloads, batch size and machine load all matter.
|
|
77
|
+
|
|
78
|
+
The optional tests cover identical inputs and counts, bounded pending requests,
|
|
79
|
+
exception propagation, server-thread cleanup and refusing to replace an existing
|
|
80
|
+
caller-owned scheduler. Client and server sockets are closed in ensure blocks;
|
|
81
|
+
remaining batch tasks are stopped after failure. Async may log the expected failure
|
|
82
|
+
in the cleanup test. The benchmark owns its scheduler only when none is installed.
|
|
83
|
+
|
|
84
|
+
### Decision after both experiments
|
|
85
|
+
|
|
86
|
+
Keep the existing public workers for v1. The scheduled experiment calls generated
|
|
87
|
+
properties directly; it does not integrate with `Pbt.check`, its early-failure
|
|
88
|
+
ordering, cancellation policy, shrinking, reports or Ractor worker. Concurrent failure/shrink/report parity is outside this experiment.
|
|
89
|
+
|
|
90
|
+
Scheduler-backed execution can help IO-heavy properties, but adoption needs a
|
|
91
|
+
separate design for in-flight case cleanup, failure ordering, exception propagation,
|
|
92
|
+
shared fixture state, concurrency limits and scheduler ownership. A caller-installed
|
|
93
|
+
scheduler must not be replaced or closed by pbt. No public API or runtime dependency
|
|
94
|
+
is added by this experiment.
|
|
95
|
+
|
|
96
|
+
References: [Ruby Fiber scheduler interface](https://docs.ruby-lang.org/en/master/Fiber/Scheduler.html),
|
|
97
|
+
[Async](https://github.com/socketry/async).
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "pbt"
|
|
4
|
+
require "delegate"
|
|
5
|
+
|
|
6
|
+
# Standalone experiment, not a supported worker implementation.
|
|
7
|
+
module FiberExperiment
|
|
8
|
+
class Property < SimpleDelegator
|
|
9
|
+
def run(value)
|
|
10
|
+
Fiber.new { __getobj__.run(value) }.resume
|
|
11
|
+
end
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
def self.check(fiber:, num_runs:, &predicate)
|
|
15
|
+
property = Pbt.property(Pbt.integer, &predicate)
|
|
16
|
+
property = Property.new(property) if fiber
|
|
17
|
+
Pbt.check(seed: 42, num_runs: num_runs, worker: :none) { property }
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
def self.run(num_runs: 100)
|
|
21
|
+
workloads = {
|
|
22
|
+
cpu: ->(n) { 1000.times { n * n } },
|
|
23
|
+
blocking_sleep: ->(_n) { sleep(0.001) }
|
|
24
|
+
}
|
|
25
|
+
puts "Ruby #{RUBY_VERSION}; #{num_runs} cases; seed 42; no scheduler installed"
|
|
26
|
+
workloads.each do |name, predicate|
|
|
27
|
+
[false, true].each do |fiber|
|
|
28
|
+
started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
29
|
+
result = check(fiber: fiber, num_runs: num_runs, &predicate)
|
|
30
|
+
elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started
|
|
31
|
+
raise "incorrect execution" if result.failed || result.num_runs != num_runs
|
|
32
|
+
|
|
33
|
+
puts format("%-26s %.6f seconds", "#{name} #{fiber ? "fiber" : "sequential"}", elapsed)
|
|
34
|
+
end
|
|
35
|
+
end
|
|
36
|
+
end
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
FiberExperiment.run(num_runs: Integer(ENV.fetch("NUM_RUNS", "100"))) if $PROGRAM_NAME == __FILE__
|
|
@@ -0,0 +1,131 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "pbt"
|
|
4
|
+
require "async"
|
|
5
|
+
require "socket"
|
|
6
|
+
|
|
7
|
+
# Benchmark only: does not execute Pbt.check or implement a public worker.
|
|
8
|
+
module SchedulerExperiment
|
|
9
|
+
# One server thread delays each response independently. It never uses a Fiber
|
|
10
|
+
# scheduler, so all three client modes see the same delayed TCP protocol.
|
|
11
|
+
class Server
|
|
12
|
+
attr_reader :port
|
|
13
|
+
|
|
14
|
+
def initialize(num_runs:, delay:)
|
|
15
|
+
@listener = TCPServer.new("127.0.0.1", 0)
|
|
16
|
+
@port = @listener.addr[1]
|
|
17
|
+
@thread = Thread.new { serve(num_runs, delay) }
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
def serve(num_runs, delay)
|
|
21
|
+
pending = {}
|
|
22
|
+
completed = 0
|
|
23
|
+
peak = 0
|
|
24
|
+
until completed == num_runs
|
|
25
|
+
timeout = pending.empty? ? nil : [pending.values.min - SchedulerExperiment.now, 0].max
|
|
26
|
+
if IO.select([@listener], nil, nil, timeout)
|
|
27
|
+
loop do
|
|
28
|
+
socket = @listener.accept_nonblock(exception: false)
|
|
29
|
+
break if socket == :wait_readable
|
|
30
|
+
pending[socket] = SchedulerExperiment.now + delay
|
|
31
|
+
peak = [peak, pending.size].max
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
pending.keys.each do |socket|
|
|
35
|
+
next if pending.fetch(socket) > SchedulerExperiment.now
|
|
36
|
+
socket.write("ok")
|
|
37
|
+
socket.close
|
|
38
|
+
pending.delete(socket)
|
|
39
|
+
completed += 1
|
|
40
|
+
end
|
|
41
|
+
end
|
|
42
|
+
peak
|
|
43
|
+
ensure
|
|
44
|
+
pending&.each_key { |socket| socket.close unless socket.closed? }
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
def peak
|
|
48
|
+
@thread.value
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
def close
|
|
52
|
+
@thread.kill if @thread.alive?
|
|
53
|
+
@thread.join
|
|
54
|
+
@listener.close
|
|
55
|
+
end
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
def self.now
|
|
59
|
+
Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
def self.measure(mode:, num_runs:, concurrency:, delay:)
|
|
63
|
+
raise ArgumentError, "existing scheduler must not be replaced" if Fiber.scheduler
|
|
64
|
+
raise ArgumentError, "unknown mode" unless [:sequential, :fiber, :scheduler].include?(mode)
|
|
65
|
+
raise ArgumentError, "positive runs and concurrency required" unless num_runs.positive? && concurrency.positive?
|
|
66
|
+
|
|
67
|
+
rng = Random.new(42)
|
|
68
|
+
values = Array.new(num_runs) { Pbt.integer(min: 0, max: 1000).generate(rng) }
|
|
69
|
+
property = Pbt.property(Pbt.integer) { |value| yield(value) if block_given? }
|
|
70
|
+
server = Server.new(num_runs: num_runs, delay: delay)
|
|
71
|
+
observed = Array.new(num_runs)
|
|
72
|
+
request = lambda do |index|
|
|
73
|
+
socket = TCPSocket.new("127.0.0.1", server.port)
|
|
74
|
+
raise "incorrect response" unless socket.read(2) == "ok"
|
|
75
|
+
property.run(values.fetch(index))
|
|
76
|
+
observed[index] = values.fetch(index)
|
|
77
|
+
ensure
|
|
78
|
+
socket&.close
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
started = now
|
|
82
|
+
if mode == :scheduler
|
|
83
|
+
Async do |parent|
|
|
84
|
+
(0...num_runs).each_slice(concurrency) do |batch|
|
|
85
|
+
tasks = []
|
|
86
|
+
begin
|
|
87
|
+
batch.each { |index| tasks << parent.async { request.call(index) } }
|
|
88
|
+
tasks.each(&:wait)
|
|
89
|
+
ensure
|
|
90
|
+
tasks.each(&:stop)
|
|
91
|
+
end
|
|
92
|
+
end
|
|
93
|
+
end.wait
|
|
94
|
+
else
|
|
95
|
+
num_runs.times do |index|
|
|
96
|
+
if mode == :fiber
|
|
97
|
+
Fiber.new { request.call(index) }.resume
|
|
98
|
+
else
|
|
99
|
+
request.call(index)
|
|
100
|
+
end
|
|
101
|
+
end
|
|
102
|
+
end
|
|
103
|
+
elapsed = now - started
|
|
104
|
+
raise "incorrect generated inputs" unless observed == values
|
|
105
|
+
{elapsed: elapsed, values: observed, completed: observed.size, max_pending: server.peak}
|
|
106
|
+
ensure
|
|
107
|
+
server&.close
|
|
108
|
+
end
|
|
109
|
+
|
|
110
|
+
def self.run(num_runs: 100, concurrency: 10, delay: 0.01, repeats: 5)
|
|
111
|
+
raise ArgumentError, "repeats must be a positive odd number" unless repeats.positive? && repeats.odd?
|
|
112
|
+
|
|
113
|
+
puts "Ruby #{RUBY_VERSION}; async #{Async::VERSION}; #{num_runs} cases; seed 42; delay #{delay}s; concurrency #{concurrency}; #{repeats} repeats"
|
|
114
|
+
[:sequential, :fiber, :scheduler].each do |mode|
|
|
115
|
+
# Warm up separately; exclude startup/gem loading from each timed request loop.
|
|
116
|
+
measure(mode: mode, num_runs: concurrency, concurrency: concurrency, delay: delay)
|
|
117
|
+
results = Array.new(repeats) { measure(mode: mode, num_runs: num_runs, concurrency: concurrency, delay: delay) }
|
|
118
|
+
times = results.map { |result| result.fetch(:elapsed) }
|
|
119
|
+
puts "#{mode}: seconds=#{times.map { |time| format("%.6f", time) }.join(",")}; median=#{format("%.6f", times.sort.fetch(repeats / 2))}; completed=#{results.map { |r| r.fetch(:completed) }.uniq.inspect}; max_pending=#{results.map { |r| r.fetch(:max_pending) }.inspect}; checksum=#{results.first.fetch(:values).sum}"
|
|
120
|
+
end
|
|
121
|
+
end
|
|
122
|
+
end
|
|
123
|
+
|
|
124
|
+
if $PROGRAM_NAME == __FILE__
|
|
125
|
+
SchedulerExperiment.run(
|
|
126
|
+
num_runs: Integer(ENV.fetch("NUM_RUNS", "100")),
|
|
127
|
+
concurrency: Integer(ENV.fetch("CONCURRENCY", "10")),
|
|
128
|
+
delay: Float(ENV.fetch("IO_DELAY", "0.01")),
|
|
129
|
+
repeats: Integer(ENV.fetch("REPEATS", "5"))
|
|
130
|
+
)
|
|
131
|
+
end
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "rspec"
|
|
4
|
+
require_relative "../experiment"
|
|
5
|
+
|
|
6
|
+
RSpec.describe SchedulerExperiment do
|
|
7
|
+
it "preserves the seeded inputs and bounds concurrent socket requests" do
|
|
8
|
+
results = [:sequential, :fiber, :scheduler].map do |mode|
|
|
9
|
+
described_class.measure(mode: mode, num_runs: 12, concurrency: 4, delay: 0.01)
|
|
10
|
+
end
|
|
11
|
+
expect(results.map { |result| result.fetch(:values) }.uniq.size).to eq(1)
|
|
12
|
+
expect(results.map { |result| result.fetch(:completed) }).to eq([12, 12, 12])
|
|
13
|
+
expect(results.first(2).map { |result| result.fetch(:max_pending) }).to eq([1, 1])
|
|
14
|
+
expect(results.last.fetch(:max_pending)).to be_between(1, 4)
|
|
15
|
+
expect(Fiber.scheduler).to be_nil
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
it "propagates predicate failures and cleans up the server and scheduler" do
|
|
19
|
+
server_threads = Thread.list
|
|
20
|
+
expect {
|
|
21
|
+
described_class.measure(mode: :scheduler, num_runs: 8, concurrency: 4, delay: 0.001) do
|
|
22
|
+
raise "predicate failure"
|
|
23
|
+
end
|
|
24
|
+
}.to raise_error("predicate failure")
|
|
25
|
+
expect(Fiber.scheduler).to be_nil
|
|
26
|
+
expect(Thread.list - server_threads).to be_empty
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
it "does not replace a caller's scheduler" do
|
|
30
|
+
Async do
|
|
31
|
+
scheduler = Fiber.scheduler
|
|
32
|
+
expect {
|
|
33
|
+
described_class.measure(mode: :scheduler, num_runs: 1, concurrency: 1, delay: 0)
|
|
34
|
+
}.to raise_error(ArgumentError, /existing scheduler/)
|
|
35
|
+
expect(Fiber.scheduler).to equal(scheduler)
|
|
36
|
+
end.wait
|
|
37
|
+
end
|
|
38
|
+
end
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
Fiber experiment: add reproducible benchmark comparing ordinary sequential calls with
|
|
2
|
+
creating/resuming one Fiber per case, same generated inputs and predicates. Include CPU
|
|
3
|
+
and blocking sleep workloads; configurable small runs. No production worker yet.
|
|
4
|
+
Document that bare Fibers do not overlap blocking work without a scheduler; using an
|
|
5
|
+
external scheduler introduces dependency/lifecycle/cancellation decisions. Validate same
|
|
6
|
+
successful count and deterministic failure+shrinking by running each predicate inside a Fiber.
|
|
7
|
+
Decision: keep :none/:ractor for v1; revisit scheduler-backed worker for concrete IO use case.
|
|
8
|
+
|
|
9
|
+
Completed: red specs committed a439561; adapter and benchmark added.
|
|
10
|
+
The existing halving strategy reaches 7 (not global minimum 5) with seed42;
|
|
11
|
+
regression expectation corrected to the serial baseline. Full rake: 183 examples,
|
|
12
|
+
0 failures; Standard passes. Smoke benchmark recorded in benchmark/FIBER.md.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Generator statistics
|
|
2
|
+
|
|
3
|
+
Goal: inspect generated value distributions independently of property execution.
|
|
4
|
+
API: `Pbt.statistics(arb, num_runs: 100, seed: Random.new_seed) { |value| label }`.
|
|
5
|
+
Return seed, num_runs, counts and unrounded percentages; no block counts values.
|
|
6
|
+
Runs must be a nonnegative integer. Zero generates/classifies nothing. Use a local
|
|
7
|
+
seeded RNG, propagate errors and keep each invocation independent. Values or classifier
|
|
8
|
+
labels used as Hash keys must be stable (not mutated by later generation).
|
|
9
|
+
|
|
10
|
+
- [x] Write tests, run red (12 failures, missing API), review contract/tests.
|
|
11
|
+
- [x] Commit tests before implementation.
|
|
12
|
+
- [x] Add statistics class and Pbt entry point, docs and full suite/lint.
|
|
13
|
+
- [x] Independent review, fix/re-review, commit, PR referencing #26.
|
|
14
|
+
|
|
15
|
+
Verification: 13 red tests committed 04fb022; full suite 194 examples, 0 failures;
|
|
16
|
+
Standard passes. Independent review clean, including global RNG isolation regression.
|