pbt 0.5.1 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +7 -0
- data/CLAUDE.md +104 -0
- data/README.md +93 -0
- data/lib/pbt/arbitrary/arbitrary.rb +5 -3
- data/lib/pbt/arbitrary/integer_arbitrary.rb +4 -0
- data/lib/pbt/check/runner_methods.rb +27 -2
- data/lib/pbt/stateful/property.rb +455 -0
- data/lib/pbt/version.rb +1 -1
- data/lib/pbt.rb +24 -0
- metadata +4 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 3f0aceef7e66e5512062e15dc72d8bf97617bf99f8c800b320381ef4e48562ab
|
|
4
|
+
data.tar.gz: ff29061f4878170a57e1ce919cee6f065548349c876343c376e22047ef63c5c4
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 68e5b018aaceebd522d8ef3f7e3837bc469b0a4513fd0e8e4080a01b637ca219694d9605c38752f60eb5cde1ee414c5116c0e153a8077ac1d9f8d28aa8ab3ef5
|
|
7
|
+
data.tar.gz: 26e75baa90dfe41b0a8b0bcd625e8b39c69480e0a2343daca161ce0100fd6c72386c62d9c0a2afb5072c4baec39b4ccc41b499e650080093fad187791e883c60
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,12 @@
|
|
|
1
1
|
## [Unreleased]
|
|
2
2
|
|
|
3
|
+
## [0.6.0] - 2026-03-15
|
|
4
|
+
|
|
5
|
+
- Add experimental `Pbt.stateful` API for model-based stateful property testing [#38](https://github.com/ohbarye/pbt/pull/38)
|
|
6
|
+
- Support state-aware and arg-aware stateful command protocols, including argument shrinking and generation edge cases [#40](https://github.com/ohbarye/pbt/pull/40)
|
|
7
|
+
- Validate stateful model/command contracts more consistently and improve diagnostics [#39](https://github.com/ohbarye/pbt/pull/39)
|
|
8
|
+
- Add Ruby 4.0 to CI test matrix
|
|
9
|
+
|
|
3
10
|
## [0.5.1] - 2025-06-29
|
|
4
11
|
|
|
5
12
|
- Fix `IntegerArbitrary#shrink` to respect min/max bounds [#36](https://github.com/ohbarye/pbt/pull/36)
|
data/CLAUDE.md
ADDED
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
# CLAUDE.md
|
|
2
|
+
|
|
3
|
+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
4
|
+
|
|
5
|
+
## Project Overview
|
|
6
|
+
|
|
7
|
+
PBT (Property-Based Testing) is a Ruby gem that provides property-based testing capabilities with experimental features for parallel test execution using Ractor. The gem allows developers to specify properties that code should satisfy and automatically generates test cases to verify these properties.
|
|
8
|
+
|
|
9
|
+
## Development Commands
|
|
10
|
+
|
|
11
|
+
### Setup
|
|
12
|
+
```bash
|
|
13
|
+
bundle install
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
### Testing
|
|
17
|
+
```bash
|
|
18
|
+
# Run all tests
|
|
19
|
+
bundle exec rspec
|
|
20
|
+
|
|
21
|
+
# Run a specific test file
|
|
22
|
+
bundle exec rspec spec/path/to/spec.rb
|
|
23
|
+
|
|
24
|
+
# Run tests matching a pattern
|
|
25
|
+
bundle exec rspec -e "pattern"
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
### Linting
|
|
29
|
+
```bash
|
|
30
|
+
# Check code style
|
|
31
|
+
bundle exec rake standard
|
|
32
|
+
|
|
33
|
+
# Auto-fix code style issues
|
|
34
|
+
bundle exec rake standard:fix
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
### Combined Testing and Linting
|
|
38
|
+
```bash
|
|
39
|
+
# Default rake task runs both tests and linting
|
|
40
|
+
bundle exec rake
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
### Benchmarking
|
|
44
|
+
```bash
|
|
45
|
+
# Run all benchmarks
|
|
46
|
+
bundle exec rake benchmark:all
|
|
47
|
+
|
|
48
|
+
# Run specific benchmark categories
|
|
49
|
+
bundle exec rake benchmark:success:simple
|
|
50
|
+
bundle exec rake benchmark:success:cpu_bound
|
|
51
|
+
bundle exec rake benchmark:success:io_bound
|
|
52
|
+
bundle exec rake benchmark:failure:simple
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
### Building and Releasing
|
|
56
|
+
```bash
|
|
57
|
+
# Build the gem
|
|
58
|
+
bundle exec rake build
|
|
59
|
+
|
|
60
|
+
# Install locally
|
|
61
|
+
bundle exec rake install
|
|
62
|
+
|
|
63
|
+
# Release to RubyGems (maintainers only)
|
|
64
|
+
bundle exec rake release
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
## Architecture
|
|
68
|
+
|
|
69
|
+
### Core Components
|
|
70
|
+
|
|
71
|
+
1. **Arbitrary** (`lib/pbt/arbitrary/`)
|
|
72
|
+
- Base classes and modules for generating random values
|
|
73
|
+
- Implements various arbitraries: integer, array, tuple, hash, etc.
|
|
74
|
+
- Each arbitrary knows how to generate values and shrink them
|
|
75
|
+
|
|
76
|
+
2. **Check** (`lib/pbt/check/`)
|
|
77
|
+
- `Property`: Defines properties to test
|
|
78
|
+
- `Runner`: Executes properties with different concurrency methods
|
|
79
|
+
- `Configuration`: Global and per-run configuration
|
|
80
|
+
- `Tosser`: Manages the generation and shrinking process
|
|
81
|
+
|
|
82
|
+
3. **Reporter** (`lib/pbt/reporter/`)
|
|
83
|
+
- Handles test result reporting
|
|
84
|
+
- Provides verbose mode for detailed output
|
|
85
|
+
|
|
86
|
+
### Key Design Patterns
|
|
87
|
+
|
|
88
|
+
- **Shrinking**: When a test fails, PBT attempts to find the minimal failing case by systematically reducing the input
|
|
89
|
+
- **Concurrency**: Supports both serial (`:none`) and parallel (`:ractor`) execution
|
|
90
|
+
- **Configuration**: Uses both global configuration and per-assertion overrides
|
|
91
|
+
|
|
92
|
+
### Testing Approach
|
|
93
|
+
|
|
94
|
+
The codebase uses RSpec for testing and follows these patterns:
|
|
95
|
+
- Unit tests for each arbitrary type in `spec/pbt/arbitrary/`
|
|
96
|
+
- Integration tests in `spec/e2e/`
|
|
97
|
+
- Property-based tests are used to test the library itself
|
|
98
|
+
|
|
99
|
+
### Important Notes
|
|
100
|
+
|
|
101
|
+
- Ruby 3.1+ is required due to Ractor usage
|
|
102
|
+
- When using `:ractor` worker, be aware of Ractor limitations (no shared state, limited object sharing)
|
|
103
|
+
- The gem uses Standard Ruby for code style (configured in `.standard.yml`)
|
|
104
|
+
- CI runs tests against Ruby 3.1, 3.2, 3.3, and 3.4
|
data/README.md
CHANGED
|
@@ -140,6 +140,99 @@ Pbt.one_of(:a, 1, 0.1).generate(rng) # => :a or 1 or 0.1
|
|
|
140
140
|
|
|
141
141
|
See [ArbitraryMethods](https://github.com/ohbarye/pbt/blob/main/lib/pbt/arbitrary/arbitrary_methods.rb) module for more details.
|
|
142
142
|
|
|
143
|
+
## Stateful Testing (Experimental)
|
|
144
|
+
|
|
145
|
+
`Pbt` also provides an experimental stateful property API for model-based / command-based testing.
|
|
146
|
+
It is designed as a property-compatible object (`generate`, `shrink`, `run`) so it works with the existing runner (`Pbt.assert` / `Pbt.check`) without changing the runner API.
|
|
147
|
+
|
|
148
|
+
This API is still experimental. Expect interface refinements and behavior changes in future releases.
|
|
149
|
+
|
|
150
|
+
### Minimal usage
|
|
151
|
+
|
|
152
|
+
```ruby
|
|
153
|
+
class CounterModel
|
|
154
|
+
def initialize
|
|
155
|
+
@inc = IncrementCommand.new
|
|
156
|
+
end
|
|
157
|
+
|
|
158
|
+
def initial_state
|
|
159
|
+
0
|
|
160
|
+
end
|
|
161
|
+
|
|
162
|
+
def commands(_state)
|
|
163
|
+
[@inc]
|
|
164
|
+
end
|
|
165
|
+
end
|
|
166
|
+
|
|
167
|
+
class IncrementCommand
|
|
168
|
+
def name
|
|
169
|
+
:increment
|
|
170
|
+
end
|
|
171
|
+
|
|
172
|
+
def arguments
|
|
173
|
+
Pbt.nil
|
|
174
|
+
end
|
|
175
|
+
|
|
176
|
+
def applicable?(_state)
|
|
177
|
+
true
|
|
178
|
+
end
|
|
179
|
+
|
|
180
|
+
def next_state(state, _args)
|
|
181
|
+
state + 1
|
|
182
|
+
end
|
|
183
|
+
|
|
184
|
+
def run!(sut, _args)
|
|
185
|
+
sut.increment
|
|
186
|
+
end
|
|
187
|
+
|
|
188
|
+
def verify!(before_state:, after_state:, args: _, result:, sut:)
|
|
189
|
+
raise "unexpected result" unless result == after_state
|
|
190
|
+
raise "state mismatch" unless after_state == before_state + 1
|
|
191
|
+
raise "sut mismatch" unless sut.value == after_state
|
|
192
|
+
end
|
|
193
|
+
end
|
|
194
|
+
|
|
195
|
+
class Counter
|
|
196
|
+
attr_reader :value
|
|
197
|
+
|
|
198
|
+
def initialize
|
|
199
|
+
@value = 0
|
|
200
|
+
end
|
|
201
|
+
|
|
202
|
+
def increment
|
|
203
|
+
@value += 1
|
|
204
|
+
end
|
|
205
|
+
end
|
|
206
|
+
|
|
207
|
+
Pbt.assert do
|
|
208
|
+
Pbt.stateful(
|
|
209
|
+
model: CounterModel.new,
|
|
210
|
+
sut: -> { Counter.new },
|
|
211
|
+
max_steps: 20
|
|
212
|
+
)
|
|
213
|
+
end
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
### Expected interfaces
|
|
217
|
+
|
|
218
|
+
`Pbt.stateful(model:, sut:, max_steps:)` expects the following duck-typed interfaces:
|
|
219
|
+
|
|
220
|
+
- `model.initial_state`
|
|
221
|
+
- `model.commands(state)` -> `Array<command>`
|
|
222
|
+
- `command.name`
|
|
223
|
+
- `command.arguments` (a `Pbt` arbitrary)
|
|
224
|
+
- `command.applicable?(state)` -> `true` / `false`
|
|
225
|
+
- `command.next_state(state, args)` -> next model state
|
|
226
|
+
- `command.run!(sut, args)` -> command result
|
|
227
|
+
- `command.verify!(before_state:, after_state:, args:, result:, sut:)`
|
|
228
|
+
|
|
229
|
+
### Current limitations (MVP)
|
|
230
|
+
|
|
231
|
+
- `Pbt.stateful` runs sequentially by default, even if the global worker is `:ractor`.
|
|
232
|
+
- You can still pass `worker: :none` explicitly if you want to make that choice obvious in a test.
|
|
233
|
+
- `worker: :ractor` is currently unsupported and raises `Pbt::InvalidConfiguration`.
|
|
234
|
+
- Shrinking supports shorter prefixes and command-argument shrinking (using `command.arguments.shrink(args)`).
|
|
235
|
+
|
|
143
236
|
## What if property-based tests fail?
|
|
144
237
|
|
|
145
238
|
Once a test fails it's time to debug. `Pbt` provides some features to help you debug.
|
|
@@ -2,6 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
module Pbt
|
|
4
4
|
module Arbitrary
|
|
5
|
+
class EmptyDomainError < ArgumentError; end
|
|
6
|
+
|
|
5
7
|
# Abstract class for generating random values on type `T`.
|
|
6
8
|
#
|
|
7
9
|
# @abstract
|
|
@@ -29,7 +31,7 @@ module Pbt
|
|
|
29
31
|
#
|
|
30
32
|
# @example
|
|
31
33
|
# integer_generator = Pbt.integer
|
|
32
|
-
# num_str_generator =
|
|
34
|
+
# num_str_generator = integer_generator.map(->(n){ n.to_s }, ->(s) {s.to_i})
|
|
33
35
|
#
|
|
34
36
|
# @param mapper [Proc] Proc to map generated values. Mainly used for generation.
|
|
35
37
|
# @param unmapper [Proc] Proc to unmap generated values. Used for shrinking.
|
|
@@ -45,8 +47,8 @@ module Pbt
|
|
|
45
47
|
#
|
|
46
48
|
# @example
|
|
47
49
|
# integer_generator = Pbt.integer
|
|
48
|
-
# even_integer_generator =
|
|
49
|
-
# # or `
|
|
50
|
+
# even_integer_generator = integer_generator.filter { |x| x.even? }
|
|
51
|
+
# # or `integer_generator.filter(&:even?)`
|
|
50
52
|
#
|
|
51
53
|
# @param refinement [Proc] Predicate proc to test each produced element. Return true to keep the element, false otherwise.
|
|
52
54
|
# @return [FilterArbitrary] New arbitrary filtered using `refinement`.
|
|
@@ -36,6 +36,10 @@ module Pbt
|
|
|
36
36
|
property = property.call
|
|
37
37
|
config = Pbt.configuration.to_h.merge(options.to_h)
|
|
38
38
|
|
|
39
|
+
if property.respond_to?(:stateful?) && property.stateful? && !options.key?(:worker)
|
|
40
|
+
config[:worker] = :none
|
|
41
|
+
end
|
|
42
|
+
|
|
39
43
|
initial_values = toss(property, config[:seed])
|
|
40
44
|
source_values = Enumerator.new(config[:num_runs]) do |y|
|
|
41
45
|
config[:num_runs].times do
|
|
@@ -115,14 +119,35 @@ module Pbt
|
|
|
115
119
|
runner.map.with_index { |val, index|
|
|
116
120
|
Case.new(val:, index:, ractor: property.run_in_ractor(val))
|
|
117
121
|
}.each do |c|
|
|
118
|
-
c.ractor
|
|
122
|
+
wait_ractor_result(c.ractor)
|
|
119
123
|
runner.handle_result(c)
|
|
120
124
|
rescue => e
|
|
121
|
-
c.exception = e
|
|
125
|
+
c.exception = unwrap_ractor_exception(e)
|
|
122
126
|
runner.handle_result(c)
|
|
123
127
|
break # Ignore the rest of the cases. Just pick up the first failure.
|
|
124
128
|
end
|
|
125
129
|
end
|
|
130
|
+
|
|
131
|
+
# Ractor errors can wrap the original exception in a `cause` (e.g. `Ractor::RemoteError`).
|
|
132
|
+
# If no cause exists, keep the original exception instance.
|
|
133
|
+
#
|
|
134
|
+
# @param error [Exception]
|
|
135
|
+
# @return [Exception]
|
|
136
|
+
def unwrap_ractor_exception(error)
|
|
137
|
+
error.cause || error
|
|
138
|
+
end
|
|
139
|
+
|
|
140
|
+
# Ruby 4 removed Ractor#take in favor of Ractor#value.
|
|
141
|
+
#
|
|
142
|
+
# @param ractor [Ractor]
|
|
143
|
+
# @return [Object]
|
|
144
|
+
def wait_ractor_result(ractor)
|
|
145
|
+
if ractor.respond_to?(:value)
|
|
146
|
+
ractor.value
|
|
147
|
+
else
|
|
148
|
+
ractor.take
|
|
149
|
+
end
|
|
150
|
+
end
|
|
126
151
|
end
|
|
127
152
|
end
|
|
128
153
|
end
|
|
@@ -0,0 +1,455 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Pbt
|
|
4
|
+
module Stateful
|
|
5
|
+
# Property-compatible wrapper for command-based stateful testing.
|
|
6
|
+
# It provides `generate`, `shrink` and `run`, so existing runners can execute it.
|
|
7
|
+
class Property
|
|
8
|
+
ARG_AWARE_GENERATION_ATTEMPTS = 5
|
|
9
|
+
|
|
10
|
+
REQUIRED_COMMAND_METHODS = %i[
|
|
11
|
+
name
|
|
12
|
+
arguments
|
|
13
|
+
applicable?
|
|
14
|
+
next_state
|
|
15
|
+
run!
|
|
16
|
+
verify!
|
|
17
|
+
].freeze
|
|
18
|
+
|
|
19
|
+
Step = Struct.new(:command, :args, keyword_init: true) do
|
|
20
|
+
def inspect
|
|
21
|
+
"#<Pbt::Stateful::Step command=#{command_label}, args=#{args.inspect}>"
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
private
|
|
25
|
+
|
|
26
|
+
def command_label
|
|
27
|
+
if command.respond_to?(:name)
|
|
28
|
+
command.name.inspect
|
|
29
|
+
else
|
|
30
|
+
command.class.name || command.class.inspect
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
# @param model [Object]
|
|
36
|
+
# @param sut [Proc]
|
|
37
|
+
# @param max_steps [Integer]
|
|
38
|
+
def initialize(model:, sut:, max_steps:)
|
|
39
|
+
validate_model!(model)
|
|
40
|
+
raise Pbt::InvalidConfiguration, "sut must be callable" unless sut.respond_to?(:call)
|
|
41
|
+
raise Pbt::InvalidConfiguration, "max_steps must be an Integer" unless max_steps.is_a?(Integer)
|
|
42
|
+
raise Pbt::InvalidConfiguration, "max_steps must be non-negative" if max_steps.negative?
|
|
43
|
+
|
|
44
|
+
@model = model
|
|
45
|
+
@sut_factory = sut
|
|
46
|
+
@max_steps = max_steps
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
# Generate a sequence of commands valid for the current model state.
|
|
50
|
+
#
|
|
51
|
+
# @param rng [Random]
|
|
52
|
+
# @return [Array<Step>]
|
|
53
|
+
def generate(rng)
|
|
54
|
+
length = rng.rand(0..@max_steps)
|
|
55
|
+
state = @model.initial_state
|
|
56
|
+
sequence = []
|
|
57
|
+
|
|
58
|
+
length.times do
|
|
59
|
+
candidates = generate_candidates_for(state, rng, context: "generate")
|
|
60
|
+
break if candidates.empty?
|
|
61
|
+
|
|
62
|
+
command, args = candidates[rng.rand(candidates.length)]
|
|
63
|
+
sequence << Step.new(command:, args:)
|
|
64
|
+
state = command.next_state(state, args)
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
sequence
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
# Shrink a sequence by trying shorter prefixes first.
|
|
71
|
+
#
|
|
72
|
+
# @param sequence [Array<Hash, Step>]
|
|
73
|
+
# @return [Enumerator<Array<Hash, Step>>]
|
|
74
|
+
def shrink(sequence)
|
|
75
|
+
Enumerator.new do |y|
|
|
76
|
+
seen = {}
|
|
77
|
+
state = @model.initial_state
|
|
78
|
+
|
|
79
|
+
(sequence.length - 1).downto(0) do |length|
|
|
80
|
+
yield_shrink_candidate(y, seen, sequence.first(length))
|
|
81
|
+
end
|
|
82
|
+
|
|
83
|
+
sequence.each_with_index do |step, index|
|
|
84
|
+
command, args = unpack_step(step)
|
|
85
|
+
validate_command_protocol!(command, state:, context: "shrink step #{index}")
|
|
86
|
+
break unless applicable?(command, state, args, context: "shrink step #{index}")
|
|
87
|
+
|
|
88
|
+
arbitrary_for(command, state, context: "shrink step #{index}").shrink(args).each do |shrunk_args|
|
|
89
|
+
candidate = replace_step(sequence, index, command:, args: shrunk_args)
|
|
90
|
+
next unless valid_sequence?(candidate)
|
|
91
|
+
|
|
92
|
+
yield_shrink_candidate(y, seen, candidate)
|
|
93
|
+
end
|
|
94
|
+
|
|
95
|
+
state = command.next_state(state, args)
|
|
96
|
+
end
|
|
97
|
+
end
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
# @return [Boolean]
|
|
101
|
+
def stateful?
|
|
102
|
+
true
|
|
103
|
+
end
|
|
104
|
+
|
|
105
|
+
# Stateful properties currently require sequential execution because the model,
|
|
106
|
+
# commands and SUT factory are ordinary Ruby objects and are not guaranteed to be
|
|
107
|
+
# Ractor-shareable.
|
|
108
|
+
#
|
|
109
|
+
# @param _sequence [Array<Hash, Step>]
|
|
110
|
+
# @raise [Pbt::InvalidConfiguration]
|
|
111
|
+
def run_in_ractor(_sequence)
|
|
112
|
+
raise Pbt::InvalidConfiguration, "Pbt.stateful does not support worker: :ractor yet; use worker: :none"
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
# Run the command sequence against a fresh SUT and verify each step.
|
|
116
|
+
#
|
|
117
|
+
# @param sequence [Array<Hash, Step>]
|
|
118
|
+
# @return [void]
|
|
119
|
+
def run(sequence)
|
|
120
|
+
state = @model.initial_state
|
|
121
|
+
sut = @sut_factory.call
|
|
122
|
+
|
|
123
|
+
sequence.each_with_index do |step, index|
|
|
124
|
+
command, args = unpack_step(step)
|
|
125
|
+
validate_command_protocol!(command, state:, context: "run step #{index}")
|
|
126
|
+
|
|
127
|
+
unless applicable?(command, state, args, context: "run step #{index}")
|
|
128
|
+
raise "invalid stateful sequence at step #{index}: #{command_name(command)}"
|
|
129
|
+
end
|
|
130
|
+
|
|
131
|
+
before_state = state
|
|
132
|
+
|
|
133
|
+
begin
|
|
134
|
+
after_state = command.next_state(before_state, args)
|
|
135
|
+
result = command.run!(sut, args)
|
|
136
|
+
command.verify!(
|
|
137
|
+
before_state:,
|
|
138
|
+
after_state:,
|
|
139
|
+
args:,
|
|
140
|
+
result:,
|
|
141
|
+
sut:
|
|
142
|
+
)
|
|
143
|
+
rescue Exception => e # standard:disable Lint/RescueException:
|
|
144
|
+
raise e.class,
|
|
145
|
+
"stateful step #{index} (#{command_name(command)}): #{e.message} [args=#{args.inspect}]",
|
|
146
|
+
e.backtrace
|
|
147
|
+
end
|
|
148
|
+
|
|
149
|
+
state = after_state
|
|
150
|
+
end
|
|
151
|
+
end
|
|
152
|
+
|
|
153
|
+
private
|
|
154
|
+
|
|
155
|
+
# @param step [Hash, Step]
|
|
156
|
+
# @return [Array<Object, Object>]
|
|
157
|
+
def unpack_step(step)
|
|
158
|
+
case step
|
|
159
|
+
in Step(command:, args:)
|
|
160
|
+
[command, args]
|
|
161
|
+
in {command:, args:}
|
|
162
|
+
[command, args]
|
|
163
|
+
else
|
|
164
|
+
raise ArgumentError, "invalid stateful step: #{step.inspect}"
|
|
165
|
+
end
|
|
166
|
+
end
|
|
167
|
+
|
|
168
|
+
# @param command [Object]
|
|
169
|
+
# @return [String]
|
|
170
|
+
def command_name(command)
|
|
171
|
+
command.respond_to?(:name) ? command.name.to_s : (command.class.name || command.class.inspect)
|
|
172
|
+
end
|
|
173
|
+
|
|
174
|
+
# @param model [Object]
|
|
175
|
+
# @return [void]
|
|
176
|
+
def validate_model!(model)
|
|
177
|
+
missing_methods = %i[initial_state commands].reject { |method_name| model.respond_to?(method_name) }
|
|
178
|
+
return if missing_methods.empty?
|
|
179
|
+
|
|
180
|
+
raise Pbt::InvalidConfiguration,
|
|
181
|
+
"Pbt.stateful model must respond to #{missing_methods.join(", ")}"
|
|
182
|
+
end
|
|
183
|
+
|
|
184
|
+
# @param state [Object]
|
|
185
|
+
# @param context [String]
|
|
186
|
+
# @return [Array<Object>]
|
|
187
|
+
def commands_for(state, context:)
|
|
188
|
+
commands = @model.commands(state)
|
|
189
|
+
|
|
190
|
+
unless commands.is_a?(Array)
|
|
191
|
+
raise Pbt::InvalidConfiguration,
|
|
192
|
+
"Pbt.stateful model.commands(state) must return Array, got #{commands.class} (context=#{context})"
|
|
193
|
+
end
|
|
194
|
+
|
|
195
|
+
commands.each { |command| validate_command_protocol!(command, state:, context:) }
|
|
196
|
+
commands
|
|
197
|
+
end
|
|
198
|
+
|
|
199
|
+
# @param command [Object]
|
|
200
|
+
# @param state [Object]
|
|
201
|
+
# @param context [String]
|
|
202
|
+
# @return [void]
|
|
203
|
+
def validate_command_protocol!(command, state:, context:)
|
|
204
|
+
missing_methods = REQUIRED_COMMAND_METHODS.reject { |method_name| command.respond_to?(method_name) }
|
|
205
|
+
unless missing_methods.empty?
|
|
206
|
+
raise Pbt::InvalidConfiguration,
|
|
207
|
+
"Pbt.stateful command protocol mismatch for #{command.class} " \
|
|
208
|
+
"(name=#{safe_command_label(command)}, missing: #{missing_methods.join(", ")}, context=#{context})"
|
|
209
|
+
end
|
|
210
|
+
|
|
211
|
+
validate_command_signature!(command, :arguments, valid_counts: [0, 1], expectation: "arguments or arguments(state)",
|
|
212
|
+
context:)
|
|
213
|
+
validate_command_signature!(command, :applicable?, valid_counts: [1, 2],
|
|
214
|
+
expectation: "applicable?(state) or applicable?(state, args)", context:)
|
|
215
|
+
end
|
|
216
|
+
|
|
217
|
+
# @param command [Object]
|
|
218
|
+
# @param arguments [Object]
|
|
219
|
+
# @param context [String]
|
|
220
|
+
# @return [void]
|
|
221
|
+
def validate_arguments_protocol!(command, arguments, context:)
|
|
222
|
+
missing_methods = %i[generate shrink].reject { |method_name| arguments.respond_to?(method_name) }
|
|
223
|
+
return if missing_methods.empty?
|
|
224
|
+
|
|
225
|
+
raise Pbt::InvalidConfiguration,
|
|
226
|
+
"Pbt.stateful command arguments protocol mismatch for #{command.class} " \
|
|
227
|
+
"(name=#{safe_command_label(command)}, missing: #{missing_methods.join(", ")}, context=#{context})"
|
|
228
|
+
end
|
|
229
|
+
|
|
230
|
+
# @param command [Object]
|
|
231
|
+
# @return [String]
|
|
232
|
+
def safe_command_label(command)
|
|
233
|
+
command.respond_to?(:name) ? command.name.inspect : "<unknown>"
|
|
234
|
+
end
|
|
235
|
+
|
|
236
|
+
# @param state [Object]
|
|
237
|
+
# @param rng [Random]
|
|
238
|
+
# @param context [String]
|
|
239
|
+
# @return [Array<Array<Object, Object>>]
|
|
240
|
+
def generate_candidates_for(state, rng, context:)
|
|
241
|
+
commands_for(state, context:).filter_map do |command|
|
|
242
|
+
args = generate_applicable_args(command, state, rng, context:)
|
|
243
|
+
next if args.equal?(NoApplicableArgs)
|
|
244
|
+
|
|
245
|
+
[command, args]
|
|
246
|
+
end
|
|
247
|
+
end
|
|
248
|
+
|
|
249
|
+
# @param command [Object]
|
|
250
|
+
# @param state [Object]
|
|
251
|
+
# @param rng [Random]
|
|
252
|
+
# @param context [String]
|
|
253
|
+
# @return [Object]
|
|
254
|
+
def generate_applicable_args(command, state, rng, context:)
|
|
255
|
+
unless arg_aware_command?(command)
|
|
256
|
+
return NoApplicableArgs unless applicable?(command, state, nil, context:)
|
|
257
|
+
|
|
258
|
+
return arbitrary_for(command, state, context:).generate(rng)
|
|
259
|
+
end
|
|
260
|
+
|
|
261
|
+
arbitrary = arbitrary_for(command, state, context:)
|
|
262
|
+
|
|
263
|
+
ARG_AWARE_GENERATION_ATTEMPTS.times do
|
|
264
|
+
args = generate_args_for(command, arbitrary, rng)
|
|
265
|
+
return NoApplicableArgs if args.equal?(NoApplicableArgs)
|
|
266
|
+
|
|
267
|
+
return args if applicable?(command, state, args, context:)
|
|
268
|
+
end
|
|
269
|
+
|
|
270
|
+
NoApplicableArgs
|
|
271
|
+
end
|
|
272
|
+
|
|
273
|
+
# @param command [Object]
|
|
274
|
+
# @param state [Object]
|
|
275
|
+
# @param context [String]
|
|
276
|
+
# @return [Object]
|
|
277
|
+
def arguments_for(command, state, context:)
|
|
278
|
+
method = command.method(:arguments)
|
|
279
|
+
|
|
280
|
+
if supports_argument_count?(method, 1)
|
|
281
|
+
command.arguments(state)
|
|
282
|
+
elsif supports_argument_count?(method, 0)
|
|
283
|
+
command.arguments
|
|
284
|
+
else
|
|
285
|
+
raise_invalid_signature!(command, :arguments, "arguments or arguments(state)", context)
|
|
286
|
+
end
|
|
287
|
+
end
|
|
288
|
+
|
|
289
|
+
# @param command [Object]
|
|
290
|
+
# @param state [Object]
|
|
291
|
+
# @param context [String]
|
|
292
|
+
# @return [Object]
|
|
293
|
+
def arbitrary_for(command, state, context:)
|
|
294
|
+
arguments = arguments_for(command, state, context:)
|
|
295
|
+
validate_arguments_protocol!(command, arguments, context:)
|
|
296
|
+
arguments
|
|
297
|
+
end
|
|
298
|
+
|
|
299
|
+
# @param command [Object]
|
|
300
|
+
# @param arbitrary [Object]
|
|
301
|
+
# @param rng [Random]
|
|
302
|
+
# @return [Object]
|
|
303
|
+
def generate_args_for(command, arbitrary, rng)
|
|
304
|
+
arbitrary.generate(rng)
|
|
305
|
+
rescue Pbt::Arbitrary::EmptyDomainError
|
|
306
|
+
raise unless state_aware_arg_aware_command?(command)
|
|
307
|
+
|
|
308
|
+
NoApplicableArgs
|
|
309
|
+
end
|
|
310
|
+
|
|
311
|
+
# @param command [Object]
|
|
312
|
+
# @param state [Object]
|
|
313
|
+
# @param args [Object]
|
|
314
|
+
# @param context [String]
|
|
315
|
+
# @return [Boolean]
|
|
316
|
+
def applicable?(command, state, args, context:)
|
|
317
|
+
method = command.method(:applicable?)
|
|
318
|
+
|
|
319
|
+
if supports_argument_count?(method, 2)
|
|
320
|
+
command.applicable?(state, args)
|
|
321
|
+
elsif supports_argument_count?(method, 1)
|
|
322
|
+
command.applicable?(state)
|
|
323
|
+
else
|
|
324
|
+
raise_invalid_signature!(command, :applicable?, "applicable?(state) or applicable?(state, args)", context)
|
|
325
|
+
end
|
|
326
|
+
end
|
|
327
|
+
|
|
328
|
+
# @param command [Object]
|
|
329
|
+
# @return [Boolean]
|
|
330
|
+
def arg_aware_command?(command)
|
|
331
|
+
supports_argument_count?(command.method(:applicable?), 2)
|
|
332
|
+
end
|
|
333
|
+
|
|
334
|
+
# @param command [Object]
|
|
335
|
+
# @return [Boolean]
|
|
336
|
+
def state_aware_arg_aware_command?(command)
|
|
337
|
+
arg_aware_command?(command) && supports_argument_count?(command.method(:arguments), 1)
|
|
338
|
+
end
|
|
339
|
+
|
|
340
|
+
# @param command [Object]
|
|
341
|
+
# @param method_name [Symbol]
|
|
342
|
+
# @param valid_counts [Array<Integer>]
|
|
343
|
+
# @param expectation [String]
|
|
344
|
+
# @param context [String]
|
|
345
|
+
# @return [void]
|
|
346
|
+
def validate_command_signature!(command, method_name, valid_counts:, expectation:, context:)
|
|
347
|
+
method = command.method(method_name)
|
|
348
|
+
return if valid_counts.any? { |count| supports_argument_count?(method, count) }
|
|
349
|
+
|
|
350
|
+
raise_invalid_signature!(command, method_name, expectation, context)
|
|
351
|
+
end
|
|
352
|
+
|
|
353
|
+
# @param command [Object]
|
|
354
|
+
# @param method_name [Symbol]
|
|
355
|
+
# @param expectation [String]
|
|
356
|
+
# @param context [String]
|
|
357
|
+
# @return [void]
|
|
358
|
+
def raise_invalid_signature!(command, method_name, expectation, context)
|
|
359
|
+
raise Pbt::InvalidConfiguration,
|
|
360
|
+
"Pbt.stateful command protocol mismatch for #{command.class} " \
|
|
361
|
+
"(name=#{safe_command_label(command)}, invalid #{method_name} signature; expected #{expectation}, context=#{context})"
|
|
362
|
+
end
|
|
363
|
+
|
|
364
|
+
# @param method [Method]
|
|
365
|
+
# @param count [Integer]
|
|
366
|
+
# @return [Boolean]
|
|
367
|
+
def supports_argument_count?(method, count)
|
|
368
|
+
return false if method.parameters.any? { |kind, _name| keyword_parameter?(kind) }
|
|
369
|
+
|
|
370
|
+
required = 0
|
|
371
|
+
optional = 0
|
|
372
|
+
rest = false
|
|
373
|
+
|
|
374
|
+
method.parameters.each do |kind, _name|
|
|
375
|
+
case kind
|
|
376
|
+
when :req
|
|
377
|
+
required += 1
|
|
378
|
+
when :opt
|
|
379
|
+
optional += 1
|
|
380
|
+
when :rest
|
|
381
|
+
rest = true
|
|
382
|
+
end
|
|
383
|
+
end
|
|
384
|
+
|
|
385
|
+
return false if count < required
|
|
386
|
+
return true if rest
|
|
387
|
+
|
|
388
|
+
count <= required + optional
|
|
389
|
+
end
|
|
390
|
+
|
|
391
|
+
# @param kind [Symbol]
|
|
392
|
+
# @return [Boolean]
|
|
393
|
+
def keyword_parameter?(kind)
|
|
394
|
+
%i[keyreq key keyrest].include?(kind)
|
|
395
|
+
end
|
|
396
|
+
|
|
397
|
+
# @param sequence [Array<Hash, Step>]
|
|
398
|
+
# @param index [Integer]
|
|
399
|
+
# @param command [Object]
|
|
400
|
+
# @param args [Object]
|
|
401
|
+
# @return [Array<Hash, Step>]
|
|
402
|
+
def replace_step(sequence, index, command:, args:)
|
|
403
|
+
candidate = sequence.dup
|
|
404
|
+
candidate[index] = rebuild_step(sequence[index], command:, args:)
|
|
405
|
+
candidate
|
|
406
|
+
end
|
|
407
|
+
|
|
408
|
+
# @param step [Hash, Step]
|
|
409
|
+
# @param command [Object]
|
|
410
|
+
# @param args [Object]
|
|
411
|
+
# @return [Hash, Step]
|
|
412
|
+
def rebuild_step(step, command:, args:)
|
|
413
|
+
case step
|
|
414
|
+
in Step
|
|
415
|
+
Step.new(command:, args:)
|
|
416
|
+
in Hash
|
|
417
|
+
{command:, args:}
|
|
418
|
+
else
|
|
419
|
+
raise ArgumentError, "invalid stateful step: #{step.inspect}"
|
|
420
|
+
end
|
|
421
|
+
end
|
|
422
|
+
|
|
423
|
+
# @param sequence [Array<Hash, Step>]
|
|
424
|
+
# @return [Boolean]
|
|
425
|
+
def valid_sequence?(sequence)
|
|
426
|
+
state = @model.initial_state
|
|
427
|
+
|
|
428
|
+
sequence.each do |step|
|
|
429
|
+
command, args = unpack_step(step)
|
|
430
|
+
return false unless applicable?(command, state, args, context: "validate sequence")
|
|
431
|
+
|
|
432
|
+
state = command.next_state(state, args)
|
|
433
|
+
end
|
|
434
|
+
|
|
435
|
+
true
|
|
436
|
+
rescue
|
|
437
|
+
false
|
|
438
|
+
end
|
|
439
|
+
|
|
440
|
+
# @param y [Enumerator::Yielder]
|
|
441
|
+
# @param seen [Hash{Array<Hash, Step> => true}]
|
|
442
|
+
# @param candidate [Array<Hash, Step>]
|
|
443
|
+
# @return [void]
|
|
444
|
+
def yield_shrink_candidate(y, seen, candidate)
|
|
445
|
+
return if seen[candidate]
|
|
446
|
+
|
|
447
|
+
seen[candidate] = true
|
|
448
|
+
y << candidate
|
|
449
|
+
end
|
|
450
|
+
|
|
451
|
+
NoApplicableArgs = Object.new
|
|
452
|
+
private_constant :NoApplicableArgs
|
|
453
|
+
end
|
|
454
|
+
end
|
|
455
|
+
end
|
data/lib/pbt/version.rb
CHANGED
data/lib/pbt.rb
CHANGED
|
@@ -5,6 +5,7 @@ require_relative "pbt/arbitrary/arbitrary_methods"
|
|
|
5
5
|
require_relative "pbt/check/runner_methods"
|
|
6
6
|
require_relative "pbt/check/property"
|
|
7
7
|
require_relative "pbt/check/configuration"
|
|
8
|
+
require_relative "pbt/stateful/property"
|
|
8
9
|
|
|
9
10
|
module Pbt
|
|
10
11
|
# Represents a property-based test failure.
|
|
@@ -44,6 +45,29 @@ module Pbt
|
|
|
44
45
|
Check::Property.new(arb, &predicate)
|
|
45
46
|
end
|
|
46
47
|
|
|
48
|
+
# Create a stateful property-based test backed by a model and a SUT factory.
|
|
49
|
+
# The returned object is compatible with `Pbt.assert` / `Pbt.check`.
|
|
50
|
+
#
|
|
51
|
+
# The model object is expected to provide:
|
|
52
|
+
# - `initial_state`
|
|
53
|
+
# - `commands(state)` -> Array of command objects
|
|
54
|
+
#
|
|
55
|
+
# Each command object is expected to provide:
|
|
56
|
+
# - `name`
|
|
57
|
+
# - `arguments` (an arbitrary) or `arguments(state)`
|
|
58
|
+
# - `applicable?(state)` -> bool or `applicable?(state, args)` -> bool
|
|
59
|
+
# - `next_state(state, args)`
|
|
60
|
+
# - `run!(sut, args)` -> result
|
|
61
|
+
# - `verify!(before_state:, after_state:, args:, result:, sut:)`
|
|
62
|
+
#
|
|
63
|
+
# @param model [Object]
|
|
64
|
+
# @param sut [Proc] Factory proc that returns a fresh SUT per run.
|
|
65
|
+
# @param max_steps [Integer]
|
|
66
|
+
# @return [Pbt::Stateful::Property]
|
|
67
|
+
def self.stateful(model:, sut:, max_steps: 20)
|
|
68
|
+
Stateful::Property.new(model:, sut:, max_steps:)
|
|
69
|
+
end
|
|
70
|
+
|
|
47
71
|
class << self
|
|
48
72
|
private
|
|
49
73
|
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: pbt
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.6.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- ohbarye
|
|
@@ -18,6 +18,7 @@ files:
|
|
|
18
18
|
- ".rspec"
|
|
19
19
|
- ".standard.yml"
|
|
20
20
|
- CHANGELOG.md
|
|
21
|
+
- CLAUDE.md
|
|
21
22
|
- CODE_OF_CONDUCT.md
|
|
22
23
|
- LICENSE.txt
|
|
23
24
|
- README.md
|
|
@@ -50,6 +51,7 @@ files:
|
|
|
50
51
|
- lib/pbt/reporter/run_details.rb
|
|
51
52
|
- lib/pbt/reporter/run_details_reporter.rb
|
|
52
53
|
- lib/pbt/reporter/run_execution.rb
|
|
54
|
+
- lib/pbt/stateful/property.rb
|
|
53
55
|
- lib/pbt/version.rb
|
|
54
56
|
- sig/pbt.rbs
|
|
55
57
|
homepage: https://github.com/ohbarye/pbt
|
|
@@ -73,7 +75,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
73
75
|
- !ruby/object:Gem::Version
|
|
74
76
|
version: '0'
|
|
75
77
|
requirements: []
|
|
76
|
-
rubygems_version: 3.6.
|
|
78
|
+
rubygems_version: 3.6.9
|
|
77
79
|
specification_version: 4
|
|
78
80
|
summary: Property-Based Testing tool for Ruby, utilizing Ractor for parallelizing
|
|
79
81
|
test cases.
|