pbt 0.6.0 → 0.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. checksums.yaml +4 -4
  2. data/.standard.yml +1 -1
  3. data/CHANGELOG.md +21 -0
  4. data/CLAUDE.md +2 -2
  5. data/README.md +116 -36
  6. data/benchmark/FIBER.md +97 -0
  7. data/benchmark/fiber_experiment.rb +39 -0
  8. data/benchmark/fiber_scheduler/Gemfile +7 -0
  9. data/benchmark/fiber_scheduler/experiment.rb +131 -0
  10. data/benchmark/fiber_scheduler/spec/experiment_spec.rb +38 -0
  11. data/docs/plans/fiber-experiment.md +12 -0
  12. data/docs/plans/generator-statistics.md +16 -0
  13. data/docs/plans/stateful-pbt-redesign.md +415 -0
  14. data/lib/pbt/arbitrary/arbitrary.rb +65 -12
  15. data/lib/pbt/arbitrary/arbitrary_methods.rb +51 -24
  16. data/lib/pbt/arbitrary/array_arbitrary.rb +46 -12
  17. data/lib/pbt/arbitrary/choose_arbitrary.rb +28 -1
  18. data/lib/pbt/arbitrary/constant.rb +0 -1
  19. data/lib/pbt/arbitrary/constant_arbitrary.rb +6 -1
  20. data/lib/pbt/arbitrary/constant_from_arbitrary.rb +39 -0
  21. data/lib/pbt/arbitrary/filter_arbitrary.rb +41 -3
  22. data/lib/pbt/arbitrary/fixed_hash_arbitrary.rb +17 -1
  23. data/lib/pbt/arbitrary/float_arbitrary.rb +31 -0
  24. data/lib/pbt/arbitrary/frequency_arbitrary.rb +61 -0
  25. data/lib/pbt/arbitrary/integer_arbitrary.rb +12 -1
  26. data/lib/pbt/arbitrary/map_arbitrary.rb +41 -3
  27. data/lib/pbt/arbitrary/one_of_arbitrary.rb +47 -14
  28. data/lib/pbt/arbitrary/sample.rb +97 -0
  29. data/lib/pbt/arbitrary/tuple_arbitrary.rb +23 -1
  30. data/lib/pbt/check/case.rb +1 -1
  31. data/lib/pbt/check/configuration.rb +5 -3
  32. data/lib/pbt/check/property.rb +24 -5
  33. data/lib/pbt/check/runner_iterator.rb +84 -10
  34. data/lib/pbt/check/runner_methods.rb +47 -17
  35. data/lib/pbt/check/tosser.rb +1 -1
  36. data/lib/pbt/reporter/run_details.rb +2 -2
  37. data/lib/pbt/reporter/run_details_reporter.rb +13 -0
  38. data/lib/pbt/reporter/run_execution.rb +47 -4
  39. data/lib/pbt/stateful/property.rb +8 -102
  40. data/lib/pbt/statistics.rb +28 -0
  41. data/lib/pbt/version.rb +1 -1
  42. data/lib/pbt.rb +45 -9
  43. metadata +16 -3
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 3f0aceef7e66e5512062e15dc72d8bf97617bf99f8c800b320381ef4e48562ab
4
- data.tar.gz: ff29061f4878170a57e1ce919cee6f065548349c876343c376e22047ef63c5c4
3
+ metadata.gz: b56a720a974f89209c3237c6b7a57adde668ebccbd77c76a51c882d1b5afd9c3
4
+ data.tar.gz: cae99f6c6d1b0d3b0de2a5ec00ca4544c7899e318c3ccaf8ec42da724d0cdb42
5
5
  SHA512:
6
- metadata.gz: 68e5b018aaceebd522d8ef3f7e3837bc469b0a4513fd0e8e4080a01b637ca219694d9605c38752f60eb5cde1ee414c5116c0e153a8077ac1d9f8d28aa8ab3ef5
7
- data.tar.gz: 26e75baa90dfe41b0a8b0bcd625e8b39c69480e0a2343daca161ce0100fd6c72386c62d9c0a2afb5072c4baec39b4ccc41b499e650080093fad187791e883c60
6
+ metadata.gz: 0e366c7d57f1428e240ce4faf92c0e0f302b0423d446af18610b81518a4a8f4528c8bdc0a3b8fc50a9e713979278520ba54cefbf430983185431f00e58cfd6ad
7
+ data.tar.gz: 87835150a4af3bcdbab40fb041d07103fbe3deb5a92d4980631f3485ae3aa89753a983908136d18f7224bf8b85db4e1d9fadede51cacc94471cbf8adb18b285f
data/.standard.yml CHANGED
@@ -1,3 +1,3 @@
1
1
  # For available configuration options, see:
2
2
  # https://github.com/standardrb/standard
3
- ruby_version: 3.1
3
+ ruby_version: 3.3
data/CHANGELOG.md CHANGED
@@ -1,5 +1,26 @@
1
1
  ## [Unreleased]
2
2
 
3
+ ## [0.8.0] - 2026-11-22
4
+
5
+ - [Breaking change] Drop support for Ruby 3.2 and earlier (EOL); Ruby 3.3 or later is now required, and Ruby 4.1.0-preview1 is added to CI [#66](https://github.com/ohbarye/pbt/pull/66)
6
+ - [Breaking change] Split literal constant selection into `Pbt.constant_from(*choices)` and change `Pbt.one_of(*arbitraries)` to select uniformly among arbitraries with contextual shrinking [#70](https://github.com/ohbarye/pbt/pull/70)
7
+ - Add `Pbt.frequency` weighted generators with contextual lazy shrink trees (`Arbitrary::Sample`) [#61](https://github.com/ohbarye/pbt/pull/61)
8
+ - Add `Pbt.statistics` for reproducible generator distribution inspection [#62](https://github.com/ohbarye/pbt/pull/62)
9
+ - Allow `Arbitrary#map` without an `unmapper` and with block syntax (`arb.map { |v| ... }`), shrinking via `Sample` provenance [#71](https://github.com/ohbarye/pbt/pull/71)
10
+ - Add 3-valued `Arbitrary#cover?(value)` (including `FloatArbitrary` for `Pbt.float`) and `RunnerIterator` `same_failure?` / `shrink_case` hooks [#72](https://github.com/ohbarye/pbt/pull/72)
11
+ - Document v1 Property DSL contract and fix positional Hash / strict lambda argument passing in `Check::Property#run` [#73](https://github.com/ohbarye/pbt/pull/73)
12
+ - Re-raise fatal exceptions (`SystemExit`, `SignalException`, `NoMemoryError`) in runners without shrinking [#74](https://github.com/ohbarye/pbt/pull/74)
13
+ - Bound `FilterArbitrary` generation retries with `max_attempts:` (raising `Pbt::Arbitrary::FilterExhaustedError`) to prevent infinite loops [#75](https://github.com/ohbarye/pbt/pull/75)
14
+ - Improve array shrinking (continue element shrinking at `min_length` and support non-contiguous element deletion) and fix `Pbt.nat` default `max` [#76](https://github.com/ohbarye/pbt/pull/76)
15
+ - Add `Pbt.assume` precondition helper with `max_skips` guard (`Pbt::TooManySkipsError`) and `RunDetails#num_skips` [#77](https://github.com/ohbarye/pbt/pull/77)
16
+ - Note: the current `Pbt.stateful(model:, sut:)` API is planned to be replaced with a redesigned DSL in v0.9.0 ([#67](https://github.com/ohbarye/pbt/issues/67)).
17
+
18
+ ## [0.7.0] - 2026-04-04
19
+
20
+ - [Breaking change] Simplify stateful command protocol: `arguments` must now accept `state` parameter, and `applicable?` must now accept both `state` and `args` parameters
21
+ - Make `rng` optional in all arbitrary `generate` methods with default `Random.new` [#44](https://github.com/ohbarye/pbt/pull/44)
22
+ - Mark stateful testing API as stable (no longer experimental)
23
+
3
24
  ## [0.6.0] - 2026-03-15
4
25
 
5
26
  - Add experimental `Pbt.stateful` API for model-based stateful property testing [#38](https://github.com/ohbarye/pbt/pull/38)
data/CLAUDE.md CHANGED
@@ -98,7 +98,7 @@ The codebase uses RSpec for testing and follows these patterns:
98
98
 
99
99
  ### Important Notes
100
100
 
101
- - Ruby 3.1+ is required due to Ractor usage
101
+ - Ruby 3.3+ is required
102
102
  - When using `:ractor` worker, be aware of Ractor limitations (no shared state, limited object sharing)
103
103
  - The gem uses Standard Ruby for code style (configured in `.standard.yml`)
104
- - CI runs tests against Ruby 3.1, 3.2, 3.3, and 3.4
104
+ - CI runs tests against Ruby 3.3, 3.4, 4.0, and 4.1.0-preview1
data/README.md CHANGED
@@ -28,13 +28,15 @@ For a more in-depth understanding of Property-Based Testing, please refer to ext
28
28
 
29
29
  ## Installation
30
30
 
31
+ Ruby 3.3 or later is required. CI tests Ruby 3.3, 3.4, 4.0, and 4.1.0-preview1.
32
+
31
33
  Add this line to your application's Gemfile and run `bundle install`.
32
34
 
33
35
  ```ruby
34
36
  gem 'pbt'
35
37
  ```
36
38
 
37
- Off course you can install with `gem intstall pbt`.
39
+ Of course you can install with `gem install pbt`.
38
40
 
39
41
  ## Basic Usage
40
42
 
@@ -77,16 +79,36 @@ The above snippet is very simple but contains the basic components.
77
79
 
78
80
  #### Runner
79
81
 
80
- `Pbt.assert` is the runner. The runner interprets and executes the given property. `Pbt.assert` takes a property and runs it multiple times. If the property fails, it tries to shrink the input that caused the failure.
82
+ `Pbt.assert` is the primary runner. It executes the property returned by the block multiple times, shrinks any failure, and raises `Pbt::PropertyFailure` with reproduction details.
83
+
84
+ If you want to inspect predicate failures without raising `Pbt::PropertyFailure`, use `Pbt.check`, which returns a `Pbt::Reporter::RunDetails` struct (`failed`, `num_runs`, `num_skips`, `num_shrinks`, `seed`, `counterexample`, `counterexample_path`, `error_message`, `error_instance`, `failures`, `verbose`, `execution_summary`, `run_configuration`). Exceptions raised during property construction or value generation still propagate to the caller.
85
+
86
+ #### Preconditions (`Pbt.assume`)
87
+
88
+ Use `Pbt.assume(condition)` inside a predicate to discard inputs that do not satisfy a precondition instead of returning early (which would count skipped inputs as passed runs):
89
+
90
+ ```ruby
91
+ Pbt.assert do
92
+ Pbt.property(Pbt.integer, Pbt.integer) do |x, y|
93
+ Pbt.assume(y != 0)
94
+ raise "unexpected" unless (x / y) * y + (x % y) == x
95
+ end
96
+ end
97
+ ```
81
98
 
82
- #### Property
99
+ Discarded cases do not count toward `num_runs` and are skipped during shrinking. If skips exceed `max_skips` (default: `num_runs * 10`), `Pbt.assert` raises `Pbt::TooManySkipsError` (`< Pbt::PropertyFailure`). Prefer narrowing your arbitrary when a precondition rejects most inputs.
83
100
 
84
- The snippet above declared a property by calling `Pbt.property`. The property describes the following:
101
+ #### Property (v1 DSL Contract)
85
102
 
86
- 1. What the user wants to evaluate. This corresponds to the block (let's call this `predicate`) enclosed by `do` `end`
87
- 2. How to generate inputs for the predicate — using `Arbitrary`
103
+ `Pbt` intentionally uses explicit runner/property composition (`Pbt.assert` / `Pbt.check` wrapping `Pbt.property` or `Pbt.stateful`):
88
104
 
89
- The `predicate` block is a function that directly asserts, taking values generated by `Arbitrary` as input.
105
+ 1. Runner options (`seed`, `num_runs`, `verbose`, `worker`) stay separated from generated inputs.
106
+ 2. Predicates fail by **raising** (including RSpec `expect(...)` and Minitest assertions under default `worker: :none`). Returning `false` alone does not fail a property.
107
+ 3. Argument passing rules:
108
+ - Single arbitrary: passed directly (`Pbt.property(Pbt.integer) { |n| ... }`)
109
+ - Multiple positional arbitraries: wrapped as a tuple (`Pbt.property(Pbt.integer, Pbt.symbol) { |n, sym| ... }`)
110
+ - Keyword arbitraries: wrapped as a fixed hash and passed as keyword arguments under `worker: :none` (`Pbt.property(left: Pbt.integer, right: Pbt.integer) { |left:, right:| ... }`). Under `worker: :ractor`, accept a single hash parameter (`{ |h| ... }`).
111
+ - Mixing positional and keyword arbitraries in one `Pbt.property` call raises `ArgumentError`.
90
112
 
91
113
  #### Arbitrary
92
114
 
@@ -108,44 +130,98 @@ There are many built-in arbitraries in `Pbt`. You can use them to generate rando
108
130
  #### Primitives
109
131
 
110
132
  ```ruby
111
- rng = Random.new
133
+ Pbt.integer.generate # => 42
134
+ Pbt.integer(min: -1, max: 8).generate # => Integer between -1 and 8
112
135
 
113
- Pbt.integer.generate(rng) # => 42
114
- Pbt.integer(min: -1, max: 8).generate(rng) # => Integer between -1 and 8
136
+ Pbt.symbol.generate # => :atq
115
137
 
116
- Pbt.symbol.generate(rng) # => :atq
138
+ Pbt.ascii_char.generate # => "a"
139
+ Pbt.ascii_string.generate # => "aagjZfao"
140
+
141
+ Pbt.boolean.generate # => true or false
142
+ Pbt.constant(42).generate # => 42 always
143
+ ```
117
144
 
118
- Pbt.ascii_char.generate(rng) # => "a"
119
- Pbt.ascii_string.generate(rng) # => "aagjZfao"
145
+ You can also pass a custom random number generator if needed:
120
146
 
121
- Pbt.boolean.generate(rng) # => true or false
122
- Pbt.constant(42).generate(rng) # => 42 always
147
+ ```ruby
148
+ rng = Random.new(42) # with a specific seed for reproducibility
149
+ Pbt.integer.generate(rng)
123
150
  ```
124
151
 
125
152
  #### Composites
126
153
 
127
154
  ```ruby
128
- rng = Random.new
155
+ Pbt.array(Pbt.integer).generate # => [121, -13141, 9825]
156
+ Pbt.array(Pbt.integer, max: 1, empty: true).generate # => [] or [42] etc.
129
157
 
130
- Pbt.array(Pbt.integer).generate(rng) # => [121, -13141, 9825]
131
- Pbt.array(Pbt.integer, max: 1, empty: true).generate(rng) # => [] or [42] etc.
158
+ Pbt.tuple(Pbt.symbol, Pbt.integer).generate # => [:atq, 42]
132
159
 
133
- Pbt.tuple(Pbt.symbol, Pbt.integer).generate(rng) # => [:atq, 42]
160
+ Pbt.fixed_hash(x: Pbt.symbol, y: Pbt.integer).generate # => {x: :atq, y: 42}
161
+ Pbt.hash(Pbt.symbol, Pbt.integer).generate # => {atq: 121, ygab: -1142}
134
162
 
135
- Pbt.fixed_hash(x: Pbt.symbol, y: Pbt.integer).generate(rng) # => {x: :atq, y: 42}
136
- Pbt.hash(Pbt.symbol, Pbt.integer).generate(rng) # => {atq: 121, ygab: -1142}
163
+ Pbt.constant_from(:a, 1, 0.1).generate # => :a or 1 or 0.1
164
+ Pbt.one_of(Pbt.integer, Pbt.constant(nil)).generate # => 42 or nil
137
165
 
138
- Pbt.one_of(:a, 1, 0.1).generate(rng) # => :a or 1 or 0.1
166
+ Pbt.integer(min: 0, max: 10).map { |n| n * 2 }.generate # => 8
167
+ Pbt.integer(min: 0, max: 10).filter(&:even?).generate # => 4
139
168
  ````
140
169
 
141
170
  See [ArbitraryMethods](https://github.com/ohbarye/pbt/blob/main/lib/pbt/arbitrary/arbitrary_methods.rb) module for more details.
142
171
 
143
- ## Stateful Testing (Experimental)
172
+ ### Weighted generators
173
+
174
+ Use `Pbt.frequency` to select generators by relative nonnegative integer weights:
144
175
 
145
- `Pbt` also provides an experimental stateful property API for model-based / command-based testing.
176
+ ```ruby
177
+ Pbt.assert(seed: 42) do
178
+ Pbt.property(Pbt.frequency([3, Pbt.integer], [1, Pbt.constant(nil)])) do |value|
179
+ raise "unexpected value" unless value.nil? || value.is_a?(Integer)
180
+ end
181
+ end
182
+ ```
183
+
184
+ At least one weight must be positive. Zero-weight generators are never selected.
185
+ Normal property execution retains the selected generator's shrink tree, including
186
+ inside arrays, tuples, fixed hashes, maps and filters. Shrinking stays within that
187
+ branch. Generated values and reported counterexamples remain ordinary Ruby values.
188
+
189
+ For manual shrinking, retain a sample: `sample = arb.sample(Random.new(42))`,
190
+ then inspect `sample.value` and `sample.shrink.map(&:value)`. Calling
191
+ `arb.shrink(value)` directly on `Pbt.frequency`, `Pbt.one_of`, or `Arbitrary#map`
192
+ without an `unmapper` returns no candidates because a bare value cannot identify its
193
+ source branch or preimage. The current `Pbt.stateful` command arguments use the
194
+ plain-value protocol, so those contextual generators do not shrink command arguments there.
195
+
196
+ ### Generator statistics
197
+
198
+ Inspect generated values before running a property:
199
+
200
+ ```ruby
201
+ stats = Pbt.statistics(Pbt.integer, num_runs: 1000, seed: 42) do |number|
202
+ number.negative? ? :negative : :nonnegative
203
+ end
204
+ stats.counts # => counts grouped by label
205
+ stats.percentages # => unrounded percentages grouped by label
206
+ stats.seed # => 42; reuse this seed to replay the sample
207
+ ```
208
+
209
+ Without a block, actual generated values are counted. Each call generates exactly
210
+ `num_runs` values using its own seeded RNG; it does not execute a property or shrink
211
+ values. The default is 100 runs with a fresh seed, retained in the result. Zero runs
212
+ return empty counts and percentages. Counts are a mutable Hash local to the result;
213
+ values or labels used as its keys must remain stable. For generators that reuse mutable
214
+ objects, return a stable label from the block (for example, a symbol or integer).
215
+ Generator and classifier errors propagate to the caller.
216
+
217
+ ## Stateful Testing
218
+
219
+ `Pbt` also provides a stateful property API for model-based / command-based testing.
146
220
  It is designed as a property-compatible object (`generate`, `shrink`, `run`) so it works with the existing runner (`Pbt.assert` / `Pbt.check`) without changing the runner API.
147
221
 
148
- This API is still experimental. Expect interface refinements and behavior changes in future releases.
222
+ > **Note:** The current `Pbt.stateful(model:, sut:)` protocol is planned to be replaced with a cleaner yield-builder DSL and improved shrinking engine in v0.9.0 before v1.0.0. See [#67](https://github.com/ohbarye/pbt/issues/67) and [docs/plans/stateful-pbt-redesign.md](docs/plans/stateful-pbt-redesign.md).
223
+
224
+ Commands currently implement `arguments(state)` and `applicable?(state, args)`; the earlier zero-argument / state-only forms are no longer supported.
149
225
 
150
226
  ### Minimal usage
151
227
 
@@ -169,11 +245,11 @@ class IncrementCommand
169
245
  :increment
170
246
  end
171
247
 
172
- def arguments
248
+ def arguments(_state)
173
249
  Pbt.nil
174
250
  end
175
251
 
176
- def applicable?(_state)
252
+ def applicable?(_state, _args)
177
253
  true
178
254
  end
179
255
 
@@ -220,18 +296,18 @@ end
220
296
  - `model.initial_state`
221
297
  - `model.commands(state)` -> `Array<command>`
222
298
  - `command.name`
223
- - `command.arguments` (a `Pbt` arbitrary)
224
- - `command.applicable?(state)` -> `true` / `false`
299
+ - `command.arguments(state)` (a `Pbt` arbitrary, may depend on current model state)
300
+ - `command.applicable?(state, args)` -> `true` / `false`
225
301
  - `command.next_state(state, args)` -> next model state
226
302
  - `command.run!(sut, args)` -> command result
227
303
  - `command.verify!(before_state:, after_state:, args:, result:, sut:)`
228
304
 
229
- ### Current limitations (MVP)
305
+ ### Current limitations
230
306
 
231
307
  - `Pbt.stateful` runs sequentially by default, even if the global worker is `:ractor`.
232
308
  - You can still pass `worker: :none` explicitly if you want to make that choice obvious in a test.
233
309
  - `worker: :ractor` is currently unsupported and raises `Pbt::InvalidConfiguration`.
234
- - Shrinking supports shorter prefixes and command-argument shrinking (using `command.arguments.shrink(args)`).
310
+ - Shrinking supports shorter prefixes and command-argument shrinking (using `command.arguments(state).shrink(args)`).
235
311
 
236
312
  ## What if property-based tests fail?
237
313
 
@@ -324,6 +400,10 @@ Pbt.configure do |config|
324
400
  # Whether to report exceptions in threads.
325
401
  # It's useful to suppress error logs on Ractor that reports many errors. Default is `false`.
326
402
  config.thread_report_on_exception = false
403
+
404
+ # Maximum number of skipped cases allowed via `Pbt.assume`.
405
+ # Defaults to `num_runs * 10` when `nil`.
406
+ config.max_skips = nil
327
407
  end
328
408
  ```
329
409
 
@@ -417,11 +497,11 @@ Once this project finishes the following, we will release v1.0.0.
417
497
  - [x] Rich report by verbose mode
418
498
  - [x] (Partially) Allow to use expectations and matchers provided by test framework in Ractor. (dropped)
419
499
  - It'd be so hard to pass assertions like `expect`, `assert` to a Ractor.
420
- - [ ] Implement frequency arbitrary
421
- - [ ] Statistics feature to aggregate generated values
422
- - [ ] Decide DSL
423
- - [ ] Try Fiber
424
- - [ ] Stateful property-based testing
500
+ - [x] Implement frequency arbitrary
501
+ - [x] Statistics feature to aggregate generated values
502
+ - [x] Decide DSL (explicit `Pbt.assert` / `Pbt.check` + `Pbt.property` / `Pbt.stateful` composition)
503
+ - [x] Try Fiber ([experiment and v1 decision](benchmark/FIBER.md))
504
+ - [x] Stateful property-based testing (via `Pbt.stateful`)
425
505
 
426
506
  ## Development
427
507
 
@@ -0,0 +1,97 @@
1
+ # Fiber predicate experiment
2
+
3
+ Run from the repository root:
4
+
5
+ ```sh
6
+ bundle exec ruby benchmark/fiber_experiment.rb
7
+ NUM_RUNS=1000 bundle exec ruby benchmark/fiber_experiment.rb
8
+ ```
9
+
10
+ This compares the normal sequential runner with the same runner wrapping each
11
+ predicate call in a newly created Fiber. Both use seed 42, the same number of
12
+ cases, and the same CPU / blocking-sleep predicates. The script checks successful
13
+ case counts and measures wall time with a monotonic clock. It installs no scheduler.
14
+
15
+ ## Decision for v1
16
+
17
+ Keep `worker: :none` as the default and `:ractor` as the existing opt-in worker.
18
+ The bare-Fiber experiment is complete; it does not justify a new public worker.
19
+ A Fiber per case adds allocation/context-switching work, while blocking sleep
20
+ still blocks its thread. This is a baseline experiment, not a benchmark of a
21
+ scheduler-backed asynchronous worker or evidence against all Fiber designs.
22
+
23
+ A local smoke run on Ruby 4.0.9 (arm64 macOS), 100 cases, seed 42 produced:
24
+
25
+ ```text
26
+ cpu sequential 0.001758 seconds
27
+ cpu fiber 0.002105 seconds
28
+ blocking_sleep sequential 0.126800 seconds
29
+ blocking_sleep fiber 0.128960 seconds
30
+ ```
31
+
32
+ These short timings are illustrative and noisy, not performance guarantees.
33
+ Re-run on the target Ruby/platform with representative workloads before drawing
34
+ performance conclusions. CPU parallelism is not provided by these same-thread Fibers.
35
+
36
+ ## Scheduler-backed TCP IO experiment
37
+
38
+ The separate experiment in `benchmark/fiber_scheduler/experiment.rb` uses Async's
39
+ scheduler. Async is an optional benchmark dependency, not a dependency of pbt:
40
+
41
+ ```sh
42
+ BUNDLE_GEMFILE=benchmark/fiber_scheduler/Gemfile bundle install
43
+ BUNDLE_GEMFILE=benchmark/fiber_scheduler/Gemfile bundle exec rspec benchmark/fiber_scheduler/spec
44
+ BUNDLE_GEMFILE=benchmark/fiber_scheduler/Gemfile bundle exec ruby benchmark/fiber_scheduler/experiment.rb
45
+ ```
46
+
47
+ Parameters are `NUM_RUNS` (100), `CONCURRENCY` (10), `IO_DELAY` (0.01 seconds),
48
+ and `REPEATS` (5; use an odd number). Each mode gets a separate warmup. The timer
49
+ includes client socket creation, connection, reads, property calls and scheduler
50
+ startup/cleanup, but excludes gem loading, input generation and server startup.
51
+
52
+ A dedicated server thread accepts loopback TCP connections and independently delays
53
+ each `ok` response by 10ms. Clients perform a real blocking `Socket#read` through
54
+ `TCPSocket`, which Async makes cooperative. The three modes use the same seed-42
55
+ integer inputs and identical protocol: serial requests, serial requests wrapped in
56
+ bare Fibers, and batches of at most 10 concurrent Async tasks. Batches wait for all
57
+ members before starting the next batch; this is not a continuously filled pool.
58
+ The server records its peak number of pending responses to verify that IO waits
59
+ actually overlap; timing thresholds are deliberately absent from tests.
60
+
61
+ Local measurement: Ruby 4.0.9, Async 2.46.0, arm64 macOS, 100 cases, seed 42,
62
+ 10ms per response, maximum concurrency 10, 5 repetitions after warmup:
63
+
64
+ ```text
65
+ mode elapsed seconds (all 5 samples) median seconds
66
+ sequential 1.232761 1.246074 1.233465 1.250332 1.226240 1.233465
67
+ bare Fiber 1.232813 1.231938 1.224650 1.241142 1.213324 1.231938
68
+ scheduler 0.124735 0.129789 0.130015 0.130826 0.128622 0.129789
69
+ ```
70
+
71
+ Every sample completed 100 requests and produced the same input checksum, 47379.
72
+ Server peak pending responses were 1 for serial/bare Fiber and 10 for Scheduler
73
+ in every repetition. The Scheduler median was about 9.5 times faster than serial
74
+ for this intentionally wait-dominated workload. This demonstrates overlapping IO
75
+ waits, rather than CPU parallelism. It is not a general throughput guarantee;
76
+ loopback, synthetic delay, small payloads, batch size and machine load all matter.
77
+
78
+ The optional tests cover identical inputs and counts, bounded pending requests,
79
+ exception propagation, server-thread cleanup and refusing to replace an existing
80
+ caller-owned scheduler. Client and server sockets are closed in ensure blocks;
81
+ remaining batch tasks are stopped after failure. Async may log the expected failure
82
+ in the cleanup test. The benchmark owns its scheduler only when none is installed.
83
+
84
+ ### Decision after both experiments
85
+
86
+ Keep the existing public workers for v1. The scheduled experiment calls generated
87
+ properties directly; it does not integrate with `Pbt.check`, its early-failure
88
+ ordering, cancellation policy, shrinking, reports or Ractor worker. Concurrent failure/shrink/report parity is outside this experiment.
89
+
90
+ Scheduler-backed execution can help IO-heavy properties, but adoption needs a
91
+ separate design for in-flight case cleanup, failure ordering, exception propagation,
92
+ shared fixture state, concurrency limits and scheduler ownership. A caller-installed
93
+ scheduler must not be replaced or closed by pbt. No public API or runtime dependency
94
+ is added by this experiment.
95
+
96
+ References: [Ruby Fiber scheduler interface](https://docs.ruby-lang.org/en/master/Fiber/Scheduler.html),
97
+ [Async](https://github.com/socketry/async).
@@ -0,0 +1,39 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "pbt"
4
+ require "delegate"
5
+
6
+ # Standalone experiment, not a supported worker implementation.
7
+ module FiberExperiment
8
+ class Property < SimpleDelegator
9
+ def run(value)
10
+ Fiber.new { __getobj__.run(value) }.resume
11
+ end
12
+ end
13
+
14
+ def self.check(fiber:, num_runs:, &predicate)
15
+ property = Pbt.property(Pbt.integer, &predicate)
16
+ property = Property.new(property) if fiber
17
+ Pbt.check(seed: 42, num_runs: num_runs, worker: :none) { property }
18
+ end
19
+
20
+ def self.run(num_runs: 100)
21
+ workloads = {
22
+ cpu: ->(n) { 1000.times { n * n } },
23
+ blocking_sleep: ->(_n) { sleep(0.001) }
24
+ }
25
+ puts "Ruby #{RUBY_VERSION}; #{num_runs} cases; seed 42; no scheduler installed"
26
+ workloads.each do |name, predicate|
27
+ [false, true].each do |fiber|
28
+ started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
29
+ result = check(fiber: fiber, num_runs: num_runs, &predicate)
30
+ elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started
31
+ raise "incorrect execution" if result.failed || result.num_runs != num_runs
32
+
33
+ puts format("%-26s %.6f seconds", "#{name} #{fiber ? "fiber" : "sequential"}", elapsed)
34
+ end
35
+ end
36
+ end
37
+ end
38
+
39
+ FiberExperiment.run(num_runs: Integer(ENV.fetch("NUM_RUNS", "100"))) if $PROGRAM_NAME == __FILE__
@@ -0,0 +1,7 @@
1
+ # frozen_string_literal: true
2
+
3
+ source "https://rubygems.org"
4
+
5
+ gem "pbt", path: "../.."
6
+ gem "async", "~> 2.0"
7
+ gem "rspec", "~> 3.13"
@@ -0,0 +1,131 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "pbt"
4
+ require "async"
5
+ require "socket"
6
+
7
+ # Benchmark only: does not execute Pbt.check or implement a public worker.
8
+ module SchedulerExperiment
9
+ # One server thread delays each response independently. It never uses a Fiber
10
+ # scheduler, so all three client modes see the same delayed TCP protocol.
11
+ class Server
12
+ attr_reader :port
13
+
14
+ def initialize(num_runs:, delay:)
15
+ @listener = TCPServer.new("127.0.0.1", 0)
16
+ @port = @listener.addr[1]
17
+ @thread = Thread.new { serve(num_runs, delay) }
18
+ end
19
+
20
+ def serve(num_runs, delay)
21
+ pending = {}
22
+ completed = 0
23
+ peak = 0
24
+ until completed == num_runs
25
+ timeout = pending.empty? ? nil : [pending.values.min - SchedulerExperiment.now, 0].max
26
+ if IO.select([@listener], nil, nil, timeout)
27
+ loop do
28
+ socket = @listener.accept_nonblock(exception: false)
29
+ break if socket == :wait_readable
30
+ pending[socket] = SchedulerExperiment.now + delay
31
+ peak = [peak, pending.size].max
32
+ end
33
+ end
34
+ pending.keys.each do |socket|
35
+ next if pending.fetch(socket) > SchedulerExperiment.now
36
+ socket.write("ok")
37
+ socket.close
38
+ pending.delete(socket)
39
+ completed += 1
40
+ end
41
+ end
42
+ peak
43
+ ensure
44
+ pending&.each_key { |socket| socket.close unless socket.closed? }
45
+ end
46
+
47
+ def peak
48
+ @thread.value
49
+ end
50
+
51
+ def close
52
+ @thread.kill if @thread.alive?
53
+ @thread.join
54
+ @listener.close
55
+ end
56
+ end
57
+
58
+ def self.now
59
+ Process.clock_gettime(Process::CLOCK_MONOTONIC)
60
+ end
61
+
62
+ def self.measure(mode:, num_runs:, concurrency:, delay:)
63
+ raise ArgumentError, "existing scheduler must not be replaced" if Fiber.scheduler
64
+ raise ArgumentError, "unknown mode" unless [:sequential, :fiber, :scheduler].include?(mode)
65
+ raise ArgumentError, "positive runs and concurrency required" unless num_runs.positive? && concurrency.positive?
66
+
67
+ rng = Random.new(42)
68
+ values = Array.new(num_runs) { Pbt.integer(min: 0, max: 1000).generate(rng) }
69
+ property = Pbt.property(Pbt.integer) { |value| yield(value) if block_given? }
70
+ server = Server.new(num_runs: num_runs, delay: delay)
71
+ observed = Array.new(num_runs)
72
+ request = lambda do |index|
73
+ socket = TCPSocket.new("127.0.0.1", server.port)
74
+ raise "incorrect response" unless socket.read(2) == "ok"
75
+ property.run(values.fetch(index))
76
+ observed[index] = values.fetch(index)
77
+ ensure
78
+ socket&.close
79
+ end
80
+
81
+ started = now
82
+ if mode == :scheduler
83
+ Async do |parent|
84
+ (0...num_runs).each_slice(concurrency) do |batch|
85
+ tasks = []
86
+ begin
87
+ batch.each { |index| tasks << parent.async { request.call(index) } }
88
+ tasks.each(&:wait)
89
+ ensure
90
+ tasks.each(&:stop)
91
+ end
92
+ end
93
+ end.wait
94
+ else
95
+ num_runs.times do |index|
96
+ if mode == :fiber
97
+ Fiber.new { request.call(index) }.resume
98
+ else
99
+ request.call(index)
100
+ end
101
+ end
102
+ end
103
+ elapsed = now - started
104
+ raise "incorrect generated inputs" unless observed == values
105
+ {elapsed: elapsed, values: observed, completed: observed.size, max_pending: server.peak}
106
+ ensure
107
+ server&.close
108
+ end
109
+
110
+ def self.run(num_runs: 100, concurrency: 10, delay: 0.01, repeats: 5)
111
+ raise ArgumentError, "repeats must be a positive odd number" unless repeats.positive? && repeats.odd?
112
+
113
+ puts "Ruby #{RUBY_VERSION}; async #{Async::VERSION}; #{num_runs} cases; seed 42; delay #{delay}s; concurrency #{concurrency}; #{repeats} repeats"
114
+ [:sequential, :fiber, :scheduler].each do |mode|
115
+ # Warm up separately; exclude startup/gem loading from each timed request loop.
116
+ measure(mode: mode, num_runs: concurrency, concurrency: concurrency, delay: delay)
117
+ results = Array.new(repeats) { measure(mode: mode, num_runs: num_runs, concurrency: concurrency, delay: delay) }
118
+ times = results.map { |result| result.fetch(:elapsed) }
119
+ puts "#{mode}: seconds=#{times.map { |time| format("%.6f", time) }.join(",")}; median=#{format("%.6f", times.sort.fetch(repeats / 2))}; completed=#{results.map { |r| r.fetch(:completed) }.uniq.inspect}; max_pending=#{results.map { |r| r.fetch(:max_pending) }.inspect}; checksum=#{results.first.fetch(:values).sum}"
120
+ end
121
+ end
122
+ end
123
+
124
+ if $PROGRAM_NAME == __FILE__
125
+ SchedulerExperiment.run(
126
+ num_runs: Integer(ENV.fetch("NUM_RUNS", "100")),
127
+ concurrency: Integer(ENV.fetch("CONCURRENCY", "10")),
128
+ delay: Float(ENV.fetch("IO_DELAY", "0.01")),
129
+ repeats: Integer(ENV.fetch("REPEATS", "5"))
130
+ )
131
+ end
@@ -0,0 +1,38 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "rspec"
4
+ require_relative "../experiment"
5
+
6
+ RSpec.describe SchedulerExperiment do
7
+ it "preserves the seeded inputs and bounds concurrent socket requests" do
8
+ results = [:sequential, :fiber, :scheduler].map do |mode|
9
+ described_class.measure(mode: mode, num_runs: 12, concurrency: 4, delay: 0.01)
10
+ end
11
+ expect(results.map { |result| result.fetch(:values) }.uniq.size).to eq(1)
12
+ expect(results.map { |result| result.fetch(:completed) }).to eq([12, 12, 12])
13
+ expect(results.first(2).map { |result| result.fetch(:max_pending) }).to eq([1, 1])
14
+ expect(results.last.fetch(:max_pending)).to be_between(1, 4)
15
+ expect(Fiber.scheduler).to be_nil
16
+ end
17
+
18
+ it "propagates predicate failures and cleans up the server and scheduler" do
19
+ server_threads = Thread.list
20
+ expect {
21
+ described_class.measure(mode: :scheduler, num_runs: 8, concurrency: 4, delay: 0.001) do
22
+ raise "predicate failure"
23
+ end
24
+ }.to raise_error("predicate failure")
25
+ expect(Fiber.scheduler).to be_nil
26
+ expect(Thread.list - server_threads).to be_empty
27
+ end
28
+
29
+ it "does not replace a caller's scheduler" do
30
+ Async do
31
+ scheduler = Fiber.scheduler
32
+ expect {
33
+ described_class.measure(mode: :scheduler, num_runs: 1, concurrency: 1, delay: 0)
34
+ }.to raise_error(ArgumentError, /existing scheduler/)
35
+ expect(Fiber.scheduler).to equal(scheduler)
36
+ end.wait
37
+ end
38
+ end
@@ -0,0 +1,12 @@
1
+ Fiber experiment: add reproducible benchmark comparing ordinary sequential calls with
2
+ creating/resuming one Fiber per case, same generated inputs and predicates. Include CPU
3
+ and blocking sleep workloads; configurable small runs. No production worker yet.
4
+ Document that bare Fibers do not overlap blocking work without a scheduler; using an
5
+ external scheduler introduces dependency/lifecycle/cancellation decisions. Validate same
6
+ successful count and deterministic failure+shrinking by running each predicate inside a Fiber.
7
+ Decision: keep :none/:ractor for v1; revisit scheduler-backed worker for concrete IO use case.
8
+
9
+ Completed: red specs committed a439561; adapter and benchmark added.
10
+ The existing halving strategy reaches 7 (not global minimum 5) with seed42;
11
+ regression expectation corrected to the serial baseline. Full rake: 183 examples,
12
+ 0 failures; Standard passes. Smoke benchmark recorded in benchmark/FIBER.md.
@@ -0,0 +1,16 @@
1
+ # Generator statistics
2
+
3
+ Goal: inspect generated value distributions independently of property execution.
4
+ API: `Pbt.statistics(arb, num_runs: 100, seed: Random.new_seed) { |value| label }`.
5
+ Return seed, num_runs, counts and unrounded percentages; no block counts values.
6
+ Runs must be a nonnegative integer. Zero generates/classifies nothing. Use a local
7
+ seeded RNG, propagate errors and keep each invocation independent. Values or classifier
8
+ labels used as Hash keys must be stable (not mutated by later generation).
9
+
10
+ - [x] Write tests, run red (12 failures, missing API), review contract/tests.
11
+ - [x] Commit tests before implementation.
12
+ - [x] Add statistics class and Pbt entry point, docs and full suite/lint.
13
+ - [x] Independent review, fix/re-review, commit, PR referencing #26.
14
+
15
+ Verification: 13 red tests committed 04fb022; full suite 194 examples, 0 failures;
16
+ Standard passes. Independent review clean, including global RNG isolation regression.