pbt 0.6.0 → 0.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. checksums.yaml +4 -4
  2. data/.standard.yml +1 -1
  3. data/CHANGELOG.md +21 -0
  4. data/CLAUDE.md +2 -2
  5. data/README.md +116 -36
  6. data/benchmark/FIBER.md +97 -0
  7. data/benchmark/fiber_experiment.rb +39 -0
  8. data/benchmark/fiber_scheduler/Gemfile +7 -0
  9. data/benchmark/fiber_scheduler/experiment.rb +131 -0
  10. data/benchmark/fiber_scheduler/spec/experiment_spec.rb +38 -0
  11. data/docs/plans/fiber-experiment.md +12 -0
  12. data/docs/plans/generator-statistics.md +16 -0
  13. data/docs/plans/stateful-pbt-redesign.md +415 -0
  14. data/lib/pbt/arbitrary/arbitrary.rb +65 -12
  15. data/lib/pbt/arbitrary/arbitrary_methods.rb +51 -24
  16. data/lib/pbt/arbitrary/array_arbitrary.rb +46 -12
  17. data/lib/pbt/arbitrary/choose_arbitrary.rb +28 -1
  18. data/lib/pbt/arbitrary/constant.rb +0 -1
  19. data/lib/pbt/arbitrary/constant_arbitrary.rb +6 -1
  20. data/lib/pbt/arbitrary/constant_from_arbitrary.rb +39 -0
  21. data/lib/pbt/arbitrary/filter_arbitrary.rb +41 -3
  22. data/lib/pbt/arbitrary/fixed_hash_arbitrary.rb +17 -1
  23. data/lib/pbt/arbitrary/float_arbitrary.rb +31 -0
  24. data/lib/pbt/arbitrary/frequency_arbitrary.rb +61 -0
  25. data/lib/pbt/arbitrary/integer_arbitrary.rb +12 -1
  26. data/lib/pbt/arbitrary/map_arbitrary.rb +41 -3
  27. data/lib/pbt/arbitrary/one_of_arbitrary.rb +47 -14
  28. data/lib/pbt/arbitrary/sample.rb +97 -0
  29. data/lib/pbt/arbitrary/tuple_arbitrary.rb +23 -1
  30. data/lib/pbt/check/case.rb +1 -1
  31. data/lib/pbt/check/configuration.rb +5 -3
  32. data/lib/pbt/check/property.rb +24 -5
  33. data/lib/pbt/check/runner_iterator.rb +84 -10
  34. data/lib/pbt/check/runner_methods.rb +47 -17
  35. data/lib/pbt/check/tosser.rb +1 -1
  36. data/lib/pbt/reporter/run_details.rb +2 -2
  37. data/lib/pbt/reporter/run_details_reporter.rb +13 -0
  38. data/lib/pbt/reporter/run_execution.rb +47 -4
  39. data/lib/pbt/stateful/property.rb +8 -102
  40. data/lib/pbt/statistics.rb +28 -0
  41. data/lib/pbt/version.rb +1 -1
  42. data/lib/pbt.rb +45 -9
  43. metadata +16 -3
@@ -0,0 +1,415 @@
1
+ # Stateful Property-Based Testing Redesign
2
+
3
+ - **Tracking issue**: [#67](https://github.com/ohbarye/pbt/issues/67) (related to [#26](https://github.com/ohbarye/pbt/issues/26))
4
+ - **Baseline commit**: `main` at `c0ef6d0890636d0f21244b75584709e713ace1b2`
5
+ - **Ruby baseline**: `>= 3.3` (already enforced in `pbt.gemspec`; CI covers Ruby 3.3–4.1 preview)
6
+ - **Scope**: Clean-slate replacement of the experimental `Pbt.stateful` API and engine. Backward compatibility with the old `Pbt.stateful(model:, sut:)` protocol is intentionally dropped. Ractor support is optional.
7
+
8
+ ---
9
+
10
+ ## 1. Context & Baseline Capabilities on `main`
11
+
12
+ Recent merges on `main` directly shape this design:
13
+
14
+ 1. **Ruby `>= 3.3` (`pbt.gemspec`, PR #66)**: `Data.define` is available unconditionally in library code. Internal library files must avoid `it` (Ruby 3.4+) so Ruby 3.3 stays green, while single-argument callbacks allow Ruby 3.4 / 4.x users to write `it` idiomatically in their tests.
15
+ 2. **Contextual lazy shrink trees & weighted generators (`lib/pbt/arbitrary/sample.rb`, `frequency_arbitrary.rb`, PR #61)**:
16
+ - `Pbt::Arbitrary::Sample.capture(arb, rng)` preserves generator provenance (`#value` + lazy `#shrink`).
17
+ - `FrequencyArbitrary#shrink(bare_value)` intentionally returns an empty Enumerator because bare values lose branch provenance. Therefore, the stateful engine **must** generate and shrink `initial_state` and step `args` via `Sample.capture`, not bare `generate`/`shrink`.
18
+ - `Check::Tosser` already calls `property.generate_for_check(rng)` when available, and `Check::RunnerMethods#build_case` already extracts `Arbitrary::Sample` into `Case#sample`.
19
+ 3. **Generator statistics (`lib/pbt/statistics.rb`, PR #62)**: `Pbt.statistics(arbitrary, num_runs:, seed:, &classifier)` samples via `arbitrary.generate(rng)` and uses returned values/labels as Hash keys (requiring immutable values).
20
+ 4. **Workers (`lib/pbt/check/runner_methods.rb`, PR #65)**: v1 supports only `:none` and `:ractor` (bare Fiber worker was rejected). Stateful properties default to `:none` and reject explicit `worker: :ractor` before generation.
21
+
22
+ ---
23
+
24
+ ## 2. Prior-Art Research & Architectural Choice
25
+
26
+ ### 2.1 Taxonomy of Stateful PBT Paradigms
27
+
28
+ | Paradigm | Representative Libraries | Core Abstraction | Shrinking Requirement | Fit for `ohbarye/pbt` |
29
+ |---|---|---|---|---|
30
+ | **Imperative rule-based machine** | Python `Hypothesis` (`RuleBasedStateMachine`), Go `rapid` (`t.Repeat`), Ruby `zizek`, `Hegel` (2026) | Mutable state + rules calling `draw(arb)` + `assume` + `invariant` + `Bundle` | Requires internal choice-sequence (Conjecture/bitstream) shrinking | Poor fit for v1 without rewriting all of `Pbt::Arbitrary` into a choice-sequence engine; couples model and SUT |
31
+ | **Pure model + command transition** | Erlang `eqc_statem` / `proper_statem`, Elixir `PropCheck.StateM.ModelDSL`, Haskell `Hedgehog`, Rust `proptest-state-machine` | Pure model (`initial_state`, `guard`/`precondition`, `next_state`) + command (`arguments`, `run`, `verify`) + `invariant` | Works cleanly with value/sample-based shrinking (sequence reduction + `Sample#shrink` + offline model validation) | **Best fit**: aligns with `pbt`'s pure-Ruby zero-dep architecture, `Arbitrary::Sample`, and future parallel/linearizability testing |
32
+ | **Mutable-model command pattern** | TS/JS `fast-check` (`fc.commands` + `fc.modelRun`) | `Command<Model, Real>` with `check(m)`, `run(m, r)`, `toString()` | External/tree shrinking over command arrays | `model` and `real` updates are interleaved in `run`; awkward for state-dependent argument generation |
33
+ | **Fluent ActionChain** | Java/Kotlin `jqwik` (`ActionChain`, `Independent` / `Dependent`) | Fluent chain + `Transformer` + `withInvariant` | Integrated shrinking (`Shrinkable`) | Verbose and merges model/SUT into one state container |
34
+
35
+ ### 2.2 Key Lessons Imported from Prior Art
36
+
37
+ 1. **Two-stage applicability (Haskell `Hedgehog` & Rust `proptest-state-machine`)**: Separate a state-only `guard(state)` (evaluated *before* `arguments(state)`) from an optional arg-aware `precondition(state, args)`. Combine with `Arbitrary#cover?(value)` so state-dependent argument domains are automatically re-validated during shrinking without duplicating bounds in `precondition`.
38
+ 2. **Cohesive command blocks with sensible defaults (Elixir `PropCheck.StateM.ModelDSL`)**: Co-locate all hooks of one command inside `command :name do |c| ... end`; omit `guard`, `arguments`, `precondition`, `next_state`, and `verify` when trivial.
39
+ 3. **Global invariants & lifecycle hooks (Hypothesis, `rapid`, `proptest-state-machine`)**: Support `invariant` (checked after SUT initialization and after every command) and `sut`+`teardown` / `around` with guaranteed `ensure` cleanup.
40
+ 4. **Weighted commands & arbitrary initial state (`jqwik`, `proptest-state-machine`)**: First-class `weight:` on commands and arbitrary `initial_state` (including shrinking the initial state).
41
+ 5. **Symbolic references (Erlang `proper_statem`, Haskell `Hedgehog`)**: Keep model transitions strictly symbolic (`Ref` tokens) and resolve concrete SUT results only at execution/verification time so SUT bugs never corrupt model state.
42
+ 6. **Swarm testing & coverage diagnostics (Groce et al., Hypothesis, Antithesis)**: Planned for Phase 2 once the core model engine is rock-solid.
43
+
44
+ ---
45
+
46
+ ## 3. Audit of Current Experimental `Pbt.stateful`
47
+
48
+ The current implementation in `lib/pbt/stateful/property.rb` is being discarded due to these issues:
49
+
50
+ 1. **Severe boilerplate**: Even a 1-command counter requires a model class plus a command class implementing 6 mandatory methods (`name`, `arguments`, `applicable?`, `next_state`, `run!`, `verify!`).
51
+ 2. **Broken RSpec/Minitest ergonomics**: Separate command classes cannot call `expect(...)` or `assert_equal(...)` directly; all existing specs fall back to `raise "..." unless ...`.
52
+ 3. **Ordering bug between `arguments(state)` and `applicable?(state, args)`**: Because `applicable?` requires `args`, `arguments(state)` runs even in states where the command is illegal, requiring `EmptyDomainError` hacks and retry loops.
53
+ 4. **Missing essentials**: No `invariant`, `teardown`/`around`, command `weight`, arbitrary `initial_state`, symbolic references, or `replay`.
54
+ 5. **Shrinking flaws**:
55
+ - Prefix-only sequence shrinking (no post-failure truncation, no chunk/single-step subsequence deletion).
56
+ - Calling `arguments(new_state).shrink(old_args)` after state changes violates `Arbitrary#shrink`'s domain contract (e.g., `OneOfArbitrary#shrink` passes `nil` into `IntegerArbitrary#shrink`).
57
+ - Ignores `Arbitrary::Sample`, so `Pbt.frequency` in command arguments does not shrink.
58
+ - No failure-equivalence check during shrinking (a shrink candidate can slip into an unrelated bug).
59
+ - `valid_sequence?` has bare `rescue => false`, silently swallowing bugs in user model callbacks.
60
+ 6. **Exception rewrapping bug**: `raise e.class, "stateful step ...", e.backtrace` raises `ArgumentError` for exception classes whose `initialize` takes custom arguments.
61
+ 7. **Mutable `Step` & noisy reporting**: `Step` is a mutable `Struct`, and counterexamples print raw `Array#inspect`.
62
+
63
+ ---
64
+
65
+ ## 4. Public API Specification (v1)
66
+
67
+ ### 4.1 Entry Point
68
+
69
+ ```ruby
70
+ Pbt.stateful(max_steps: 20, min_steps: 1, max_shrink_attempts: 1_000) do |m|
71
+ ...
72
+ end
73
+ ```
74
+
75
+ - Returns a `Pbt::Stateful::Property` compatible with `Pbt.assert` and `Pbt.check`.
76
+ - Uses **yield builders (`|m|`, `|c|`)**, never `instance_eval`: preserves lexical `self` (RSpec `expect`/`let`, Minitest assertions, helper methods) and static tooling (RBS/Ruby LSP).
77
+ - `num_runs`, `seed`, and `verbose` remain options on `Pbt.assert` / `Pbt.check`.
78
+ - `min_steps` defaults to `1` (must be `>= 0`); `max_steps` defaults to `20` (must be `>= min_steps` and positive). Each generated run picks target length uniformly from `min_steps..max_steps`.
79
+ - Explicit `worker: :ractor` raises `Pbt::InvalidConfiguration` before generation begins. When `worker:` is omitted, stateful properties force `:none` even if global `Pbt.configuration.worker == :ractor`.
80
+
81
+ ### 4.2 Worked Examples
82
+
83
+ #### Minimal Stack Example (Ruby 3.4/4.x `it` style)
84
+
85
+ ```ruby
86
+ Pbt.assert do
87
+ Pbt.stateful(max_steps: 30) do |m|
88
+ m.initial_state { [] }
89
+ m.sut { Stack.new }
90
+
91
+ m.command :push, weight: 3 do |c|
92
+ c.arguments Pbt.integer(min: 0, max: 100)
93
+ c.next_state { |state, value| [*state, value] }
94
+ c.run { |sut, value| sut.push(value) }
95
+ end
96
+
97
+ m.command :pop, weight: 2 do |c|
98
+ c.guard { !it.empty? }
99
+ c.next_state { it[0...-1] }
100
+ c.run { it.pop }
101
+ c.verify { expect(it.result).to eq(it.before.last) }
102
+ end
103
+
104
+ m.command :peek do |c|
105
+ c.guard { !it.empty? }
106
+ c.run { it.peek }
107
+ c.verify { expect(it.result).to eq(it.after.last) }
108
+ end
109
+
110
+ m.invariant(:snapshot) { |state, sut| expect(sut.snapshot).to eq(state) }
111
+ end
112
+ end
113
+ ```
114
+
115
+ #### Full-Featured BankAccount Example (Arbitrary `initial_state`, `Data.define`, `around`/`teardown`, Symbolic `Ref`)
116
+
117
+ ```ruby
118
+ BankState = Data.define(:balance, :open, :accounts)
119
+
120
+ Pbt.assert(seed: 42, num_runs: 200) do
121
+ Pbt.stateful(max_steps: 25) do |m|
122
+ m.initial_state Pbt.integer(min: 0, max: 100).map { |b| BankState.new(balance: b, open: true, accounts: []) }
123
+
124
+ m.sut { |init| Bank.open(init.balance) }
125
+ m.teardown { |sut| sut.close }
126
+ # Alternative block-scoped lifecycle (mutually exclusive with sut/teardown):
127
+ # m.around { |init, run| Dir.mktmpdir { |dir| run.call(Bank.open(init.balance, dir:)) } }
128
+
129
+ m.command :deposit, weight: 3 do |c|
130
+ c.guard { |s| s.open }
131
+ c.arguments Pbt.integer(min: 1, max: 50)
132
+ c.next_state { |s, amount| s.with(balance: s.balance + amount) }
133
+ c.run { |sut, amount| sut.deposit(amount) }
134
+ end
135
+
136
+ m.command :withdraw, weight: 2 do |c|
137
+ c.guard { |s| s.open && s.balance.positive? }
138
+ c.arguments { |s| Pbt.integer(min: 1, max: s.balance) }
139
+ c.next_state { |s, amount| s.with(balance: s.balance - amount) }
140
+ c.run { |sut, amount| sut.withdraw(amount) }
141
+ c.verify { |ctx| expect(ctx.result).to eq(ctx.after.balance) }
142
+ end
143
+
144
+ m.command :create_sub_account do |c|
145
+ c.guard { |s| s.open }
146
+ c.next_state { |s, _args, ref| s.with(accounts: [*s.accounts, ref]) }
147
+ c.run { |sut| sut.create_sub_account }
148
+ end
149
+
150
+ m.command :close_sub_account do |c|
151
+ c.guard { |s| s.open && !s.accounts.empty? }
152
+ c.arguments { |s| Pbt.constant_from(*s.accounts) }
153
+ c.next_state { |s, ref| s.with(accounts: s.accounts - [ref]) }
154
+ c.run { |sut, account_id| sut.close_sub_account(account_id) }
155
+ end
156
+
157
+ m.invariant(:balance_matches) do |state, sut|
158
+ expect(sut.balance).to eq(state.balance)
159
+ expect(sut.balance).to be >= 0
160
+ end
161
+ end
162
+ end
163
+ ```
164
+
165
+ ### 4.3 Builder & Command Contracts
166
+
167
+ #### Model Builder (`m`)
168
+
169
+ | Method | Signature | Default | Contract |
170
+ |---|---|---|---|
171
+ | `m.initial_state` | `(value_or_arb)` or `{ ... }` | `nil` | Accepts a constant value, an `Arbitrary`, or a deterministic zero-arg factory block. Evaluated once per generated case and stored on `Run`. Always deep-frozen. |
172
+ | `m.sut` | `{ \|initial_state\| ... }` | Required (unless `around` is given) | Creates a fresh SUT per run and per shrink candidate. |
173
+ | `m.teardown` | `{ \|sut\| ... }` | None | Runs exactly once in `ensure` if `sut` returned normally. |
174
+ | `m.around` | `{ \|initial_state, run\| ... }` | None | Mutually exclusive with `sut`/`teardown`. Internal lifecycle uses `around` uniformly (`sut`+`teardown` desugars to `around`). Must call `run.call(sut)` exactly once (raises `InvalidSpec` if 0 or >1 calls). |
175
+ | `m.command` | `(name, weight: 1) { \|c\| ... }` | — | `name` must be a unique `Symbol`; `weight` must be a positive `Integer`. |
176
+ | `m.invariant` | `(name = nil) { \|state, sut\| ... }` | None (multiple allowed) | Assertion hook (signals failure only by raising; return value ignored). Runs after SUT setup (before step 1) and after every step's `verify`. |
177
+
178
+ #### Command Builder (`c`)
179
+
180
+ | Method | Signature | Default | Contract |
181
+ |---|---|---|---|
182
+ | `c.guard` | `{ \|state\| bool }` | `true` | State-only predicate evaluated **before** `arguments(state)` during generation and re-evaluated on every shrink candidate. |
183
+ | `c.arguments` | `(arb)`, `(**kw_arbs)`, or `{ \|state\| arb }` | `Pbt.nil` | Produces a single argument value (use `Pbt.tuple` or keyword shorthand `c.arguments(k: arb)` -> `Pbt.fixed_hash` for multiple fields; never splats). Must return an object responding to `generate` and `shrink` (or `sample`). |
184
+ | `c.precondition` | `{ \|state, args\| bool }` | `true` | Optional arg-aware predicate for cross-field/custom constraints not captured by `Arbitrary#cover?`. |
185
+ | `c.next_state` | `{ \|state, args\| new_state }` or `{ \|state, args, ref\| new_state }` | Identity (`state`) | Pure transition function. Always runs in the **symbolic** domain (`ref` is `Pbt::Stateful::Ref`, never a concrete SUT result). Return value is deep-frozen. |
186
+ | `c.run` | `{ \|sut, args\| result }` | **Required** | Executes the operation on SUT with resolved concrete `args`. |
187
+ | `c.verify` | `{ \|ctx\| ... }` | None | Assertion hook (signals failure only by raising; return value ignored). Receives `Context = Data.define(:step, :before, :after, :args, :result, :sut)` with all `Ref`s resolved to concrete values. |
188
+
189
+ ### 4.4 Public Value Objects (`Data.define`) & Replay
190
+
191
+ - `Pbt::Stateful::Step = Data.define(:id, :command, :args)`
192
+ - `id`: monotonic generation-time integer (never renumbered when steps are deleted).
193
+ - `command`: `Symbol` (never stores the Command object containing Procs).
194
+ - `args`: symbolic generated argument value.
195
+ - `Pbt::Stateful::Run = Data.define(:initial_state, :steps)`
196
+ - Pure, immutable, structurally comparable (`==`, `hash`, `eql?`).
197
+ - Helper methods: `#size`, `#empty?`, `#command_names` (`steps.map(&:command)`).
198
+ - `#to_s` / `#inspect`: program-style multi-line trace (numbering steps `1..n`, printing `v0 = ...` only for steps whose `Ref` is referenced later).
199
+ - `Pbt::Stateful::Ref = Data.define(:step_id)`
200
+ - Opaque symbolic token referencing the return value of the step with `id == step_id`.
201
+ - `Pbt::Stateful::Context = Data.define(:step, :before, :after, :args, :result, :sut)`
202
+ - `Pbt::Stateful::FailureSignature = Data.define(:stage, :target, :exception_class)`
203
+ - `property.replay(run)`
204
+ - Executes a `Pbt::Stateful::Run` (e.g., `details.counterexample` from `Pbt.check`) once against a fresh SUT with full `around`/`teardown`, `verify`, and `invariant` checks.
205
+ - `Pbt.statistics(property, num_runs: 100, seed: 42) { |run| ... }`
206
+ - Works out of the box because `property.generate(rng)` returns a pure immutable `Run`.
207
+
208
+ ---
209
+
210
+ ## 5. Core Semantics & Edge Cases
211
+
212
+ ### 5.1 `Arbitrary#cover?(value)` (3-Valued Membership Check)
213
+
214
+ Add `#cover?(value)` to `Pbt::Arbitrary::Arbitrary` returning:
215
+ - `true`: value is confirmed within this arbitrary's domain
216
+ - `false`: value is confirmed outside this arbitrary's domain
217
+ - `nil`: unknown / cannot determine reliably (default on base `Arbitrary` for custom user arbitraries)
218
+
219
+ Built-in implementations (normalized via `Arbitrary.cover_result` so non-boolean custom results collapse to `nil`):
220
+ - `IntegerArbitrary`: `value.is_a?(Integer) && value >= @min && value <= @max` (respecting unbounded `nil` bounds if any)
221
+ - `ChooseArbitrary`: `false` for unbounded ranges; checks `Integer` for integer ranges and finite `Float` for float ranges, then `@range.cover?(value)`
222
+ - `ConstantArbitrary`: `value.equal?(@val) || @val.eql?(value)`
223
+ - `ConstantFromArbitrary`: `@choices.any? { |c| c.equal?(value) || c.eql?(value) }`
224
+ - `OneOfArbitrary`: `true` if any branch returns `true`; `false` if all branches return `false`; otherwise `nil`
225
+ - `FrequencyArbitrary`: `true` if any branch returns `true`; `false` if all branches return `false`; otherwise `nil`
226
+ - `TupleArbitrary`: `false` unless `value.is_a?(Array) && value.size == @arbs.size`; aggregate element `cover?` (`false` if any `false`, `true` if all `true`, else `nil`)
227
+ - `ArrayArbitrary`: `false` unless `value.is_a?(Array) && value.size.between?(@min_length, @max_length)`; aggregate element `cover?`
228
+ - `FixedHashArbitrary`: `false` unless `value.is_a?(Hash) && value.size == @keys.size` with the same key set; aggregate value `cover?`
229
+ - `FilterArbitrary`: combine base `cover?(value)` with `!!@refinement.call(value)` (returning `false` if refinement is falsy, rescuing refinement errors as `nil`)
230
+ - `MapArbitrary`: return `nil` if `@unmapper.nil?`; otherwise unmap and check round-trip equality (`eql?`) and inner `cover?`. Returns `true` when round-trip and inner `cover?` are `true`; returns `false` only for known-bijective built-ins (`char`, `symbol`, `*_string`, `date`, `time`), and `nil` otherwise (so non-bijective maps such as `set`, `hash`, or user shrink-only unmappers never falsely reject generated values).
231
+ - `FloatArbitrary`: `false` for non-Float, non-finite, `abs > 1_000_000.0`, or `0 < abs < 1.0 / 1_000_001.0`; `true` for `0.0`; `nil` otherwise.
232
+
233
+ ### 5.2 Separating Pure `Run` from `Sample` Provenance
234
+
235
+ - `property.generate(rng)` returns pure `Pbt::Stateful::Run`.
236
+ - `property.generate_for_check(rng)` returns an internal provenance wrapper (compatible with `Case#sample` via `Arbitrary::Sample` or a dedicated stateful sample object whose `#value` is `Run`) containing:
237
+ - `run` (`Pbt::Stateful::Run`)
238
+ - `initial_state_sample` (`Arbitrary::Sample` or `nil` when `initial_state` is constant/block)
239
+ - `step_samples` (`Hash{Integer => Arbitrary::Sample}` keyed by `step.id`)
240
+ - Because `Sample` is never stored inside `Run` or `Step`:
241
+ - `Run#==`, `Run#hash`, `seen` deduplication, `Pbt.statistics`, `details.counterexample`, and `property.replay` remain pure and deterministic.
242
+ - `Pbt.frequency`, `map`, `filter`, `tuple`, `array`, and `fixed_hash` inside `initial_state` and `c.arguments` all retain full contextual shrinking.
243
+
244
+ ### 5.3 Generation Algorithm & Exhaustion Rules
245
+
246
+ For each case:
247
+ 1. Resolve `initial_state` (capturing `Sample` if `Arbitrary`) and deep-freeze it.
248
+ 2. Check eligible commands at step 0 (`guard(state)`):
249
+ - If `initial_state` is non-arbitrary and **no** command has a truthy `guard` at step 0, raise `Pbt::Stateful::InvalidSpec` immediately (prevents silent vacuous passes).
250
+ 3. Draw target length from `min_steps..max_steps` using `rng`.
251
+ 4. At each step `i`:
252
+ - Collect commands where `guard(state)` is truthy (exceptions in `guard` propagate immediately as model errors — never swallowed).
253
+ - If no command is eligible, stop generating further steps cleanly (if `steps.empty?` across all runs, raise `GenerationError` / `InvalidSpec`).
254
+ - Loop up to a bounded per-step attempt budget (e.g., `32` attempts):
255
+ - Pick an eligible command weighted by `weight:` using `rng`.
256
+ - Evaluate `arb = command.arguments_for(state)` (validate `generate`/`shrink` protocol; catch `Pbt::Arbitrary::EmptyDomainError` and mark that command ineligible for this state).
257
+ - Capture `sample = Arbitrary::Sample.capture(arb, rng)` (note: `FilterArbitrary` raises `Pbt::Arbitrary::FilterExhaustedError` if its predicate cannot be satisfied within `max_attempts` — prefer state-bounded generators over heavy `.filter`).
258
+ - Check `command.precondition(state, sample.value)`. If true, create `step_id = next_id`, `ref = Ref.new(step_id: step_id)`, compute `next_state = deep_freeze(command.next_state(state, sample.value, ref))`, append `Step.new(id: step_id, command: command.name, args: sample.value)`, record `step_samples[step_id] = sample`, and advance `state = next_state`.
259
+ - If eligible commands exist but all attempts are exhausted, raise `Pbt::Stateful::GenerationError` with step index and eligible command names.
260
+
261
+ ### 5.4 Symbolic References (`Ref`) Semantics
262
+
263
+ 1. **Monotonic IDs**: `step.id` is assigned `0, 1, 2, ...` at generation time and never mutated during shrinking.
264
+ 2. **Strict symbolic vs. concrete boundary**:
265
+ - `guard`, `arguments`, `precondition`, and `next_state` **always** see symbolic values (`Ref`). `next_state` is never re-run with concrete SUT return values.
266
+ - `run`, `verify`, and `invariant` **always** see resolved concrete values via `env` (`Hash{Integer => Object}`).
267
+ 3. **`nil` vs. unbound**: Resolution uses `env.fetch(ref.step_id)` so a step legitimately returning `nil` resolves to `nil`, while an unbound ref is a validation bug.
268
+ 4. **Scoped deep resolution**: Recursively resolve `Ref` only inside:
269
+ - `Ref` -> `env.fetch(ref.step_id)`
270
+ - `Array` -> `obj.map { |e| resolve(e, env) }`
271
+ - `Hash` -> `obj.transform_keys { |k| resolve(k, env) }.transform_values { |v| resolve(v, env) }`
272
+ - `Set` -> `Set.new(obj.map { |e| resolve(e, env) })`
273
+ - `Struct` -> reconstruct with resolved members
274
+ - `Data` -> `obj.with(**obj.to_h.transform_values { |v| resolve(v, env) })`
275
+ - Any other object passes through untouched.
276
+ 5. **Dependency checking**: During shrink validation, collect all `Ref` instances reachable inside `step.args` (using the same container traversal) and verify every `ref.step_id` is in the set of surviving earlier `step.id`s.
277
+ 6. **Limitation (documented)**: Because `Ref` is opaque to the pure model, the model cannot branch on the concrete value returned by the SUT.
278
+
279
+ ### 5.5 Shrinking Algorithm & `RunnerIterator` Integration
280
+
281
+ #### Minimal `RunnerIterator` Hook (`lib/pbt/check/runner_iterator.rb`)
282
+
283
+ Update `RunnerIterator#handle_result(c)` (~6 lines):
284
+ 1. Record `@initial_failure ||= c.exception` on first failure.
285
+ 2. On subsequent failures during shrinking, if `@property.respond_to?(:same_failure?) && !@property.same_failure?(@initial_failure, c.exception)`: treat candidate as non-matching (do **not** overwrite `@run_execution` failure; continue iterating `@next_values`).
286
+ 3. Choose the next shrink enumerator with `shrink_case` taking precedence over `c.sample`:
287
+ ```ruby
288
+ @next_values =
289
+ if @property.respond_to?(:shrink_case)
290
+ @property.shrink_case(c)
291
+ elsif c.sample
292
+ c.sample.shrink
293
+ else
294
+ @property.shrink(c.val)
295
+ end
296
+ ```
297
+
298
+ #### Failure Equivalence (`FailureSignature`)
299
+
300
+ - `StepFailure` carries `signature = FailureSignature.new(stage:, target:, exception_class: cause.class)` where:
301
+ - `stage`: `:run`, `:verify`, `:invariant`, `:sut`, or `:teardown`
302
+ - `target`: command `Symbol` or invariant name (`Symbol` / `String` / `nil`)
303
+ - `exception_class`: original cause exception class
304
+ - Deliberately **excludes** `step_index` because deleting earlier steps shifts indices.
305
+ - `same_failure?(initial_err, candidate_err)` compares `initial_err.signature == candidate_err.signature` (when both are `StepFailure`).
306
+ - If `teardown` fails (`teardown_error` present), stop shrinking immediately so cleanup breakage is never masked.
307
+
308
+ #### Shrink Candidate Order & Budget (`max_shrink_attempts`)
309
+
310
+ Shared per-failure budget counter (`max_shrink_attempts`, counting both invalid and valid attempts) and `seen` cache keyed by `[run.initial_state, run.steps]`:
311
+
312
+ 1. **Immediate post-failure truncation**: If `c.exception` is a `StepFailure` with `step_index` and `step_index < steps.length - 1`, yield `steps[0..step_index]` first (one step drops all unexecuted trailing commands).
313
+ 2. **Chunk deletions (ddmin-style)**: For chunk sizes `k = n/2, n/4, ..., 2`, scan windows from back to front and yield runs with that slice deleted.
314
+ 3. **Single-step deletions**: Scan steps from back to front and yield runs with step `i` removed.
315
+ 4. **`initial_state` shrinking**: If `initial_state_sample` exists, iterate `initial_state_sample.shrink` and yield runs with the smaller `initial_state` (preserving `steps`).
316
+ 5. **Step argument shrinking**: For each step `i` (from step `0` to `n - 1`), iterate `step_samples[step.id].shrink` and yield runs replacing step `i`'s `args` (and updated child `Sample`).
317
+
318
+ Every candidate in stages 2–5 is validated purely on the model **before** touching the SUT:
319
+ 1. `guard(state)` is truthy
320
+ 2. `arb = arguments(state)` builds without `EmptyDomainError`
321
+ 3. `arb.cover?(step.args) != false`
322
+ 4. `precondition(state, step.args)` is truthy
323
+ 5. Every `Ref` inside `step.args` is bound by an earlier surviving `step.id`
324
+ 6. `state = deep_freeze(next_state(state, step.args, Ref.new(step_id: step.id)))`
325
+
326
+ Because `RunnerIterator` greedily re-invokes `shrink_case(c)` on every accepted smaller failure, this naturally reaches a fixed point (e.g., shrinking an argument can unlock another step deletion on the next pass).
327
+
328
+ ### 5.6 Execution, Exceptions, State Freezing & Counterexample Formatting
329
+
330
+ 1. **Execution order** (inside `lifecycle.call(resolved_initial) { |sut| ... }`):
331
+ - Run all `invariant`s on `(resolved_initial, sut)` before step 1.
332
+ - For each step:
333
+ - Resolve `concrete_args = resolve(step.args, env)`.
334
+ - Compute `before_sym = sym_state`, `after_sym = deep_freeze(command.next_state(before_sym, step.args, Ref.new(step_id: step.id)))`.
335
+ - Execute `result = command.run!(sut, concrete_args)` (on failure -> raise `StepFailure` with `stage: :run`).
336
+ - Store `env[step.id] = result`.
337
+ - Resolve `before_concrete = resolve(before_sym, env)`, `after_concrete = resolve(after_sym, env)`.
338
+ - Run `command.verify!(Context.new(step:, before: before_concrete, after: after_concrete, args: concrete_args, result:, sut:))` (on failure -> `stage: :verify`).
339
+ - Run all `invariant`s on `(after_concrete, sut)` (on failure -> `stage: :invariant`).
340
+ - Advance `sym_state = after_sym`.
341
+ - Note: `after_sym` does not depend on `result`, and computing it after `run!` avoids unnecessary work when `run!` raises.
342
+ 2. **State freezing**:
343
+ - Deep-freeze `initial_state` and every `next_state` return value via `Ractor.make_shareable` (falling back to recursive container freeze / shallow `.freeze` if `Ractor::IsolationError` occurs).
344
+ - Catch `FrozenError` inside `next_state` and raise a clear `Pbt::Stateful::ModelError` guiding the user to return a new value (`state.with(...)`, `[*arr, x]`, `hash.merge(...)`).
345
+ 3. **Exception hygiene**:
346
+ - Never rescue `SystemExit`, `SignalException`, or `NoMemoryError`.
347
+ - Rescue `Exception` for `run`/`verify`/`invariant` so RSpec's `RSpec::Expectations::ExpectationNotMetError` (`< Exception`) is handled cleanly.
348
+ - Wrap failures in `Pbt::Stateful::StepFailure` preserving Ruby's exception `cause` and original backtrace.
349
+ - Run `teardown` in `ensure`. If `teardown` raises while a `StepFailure` is already propagating, keep the original `StepFailure` and attach `teardown_error` to it.
350
+ - Never swallow exceptions raised by user model callbacks (`guard`, `arguments`, `precondition`, `next_state`) during generation or execution; surface them immediately with callback name context.
351
+ 4. **Counterexample formatting (`Run#to_s`)**:
352
+ ```text
353
+ Property failed after 37 test(s)
354
+ seed: 42
355
+ counterexample:
356
+ initial_state: #<data BankState balance=0, open=true, accounts=[]>
357
+ 1. deposit(10)
358
+ 2. v0 = create_sub_account
359
+ 3. close_sub_account(v0)
360
+ 4. withdraw(10) # <- failed (verify)
361
+ Shrunk 12 time(s)
362
+ Got Pbt::Stateful::StepFailure: step 4 withdraw(10) [verify]: expected 0, got 10
363
+ (cause: RSpec::Expectations::ExpectationNotMetError)
364
+ ```
365
+ (When `Run` is rendered without failure annotation—e.g. from `generate`—omit the `# <- failed (...)` comment; `StepFailure` or annotated `Run` can supply the failing step marker cleanly.)
366
+
367
+ ---
368
+
369
+ ## 6. Phased Implementation Plan & Resumption Guide
370
+
371
+ Follow TDD (`bundle exec rspec` red -> green) and Standard Ruby (`bundle exec rake standard`) for every PR.
372
+
373
+ ### PR 0: Core Prerequisites (Small, Zero-Risk)
374
+ - **Files**:
375
+ - `lib/pbt/arbitrary/arbitrary.rb` + all subclasses (`integer`, `choose`, `constant`, `constant_from`, `one_of`, `frequency`, `tuple`, `array`, `fixed_hash`, `filter`, `map`)
376
+ - `lib/pbt/check/runner_iterator.rb`
377
+ - Unit specs under `spec/pbt/arbitrary/` and `spec/pbt/check/`
378
+ - **Checklist**:
379
+ - [ ] Implement 3-valued `Arbitrary#cover?(value)` on base class and all built-in arbitraries (including `FrequencyArbitrary`).
380
+ - [ ] Add `same_failure?` guard and `shrink_case(c)` precedence hook to `RunnerIterator#handle_result`.
381
+ - [ ] Verify full suite (`bundle exec rake`) passes with zero regressions on existing `Pbt.frequency` / `Sample` tests.
382
+
383
+ ### PR 1: Clean-Slate `Pbt.stateful` v1
384
+ - **Files**:
385
+ - Delete/replace `lib/pbt/stateful/property.rb`, `spec/pbt/stateful/property_spec.rb`, `spec/e2e/stateful_e2e_spec.rb`
386
+ - Create modular files under `lib/pbt/stateful/`:
387
+ - `errors.rb` (`InvalidSpec`, `GenerationError`, `ModelError`, `StepFailure`, `FailureSignature`)
388
+ - `types.rb` (`Ref`, `Step`, `Run`, `Context`, `Resolver`)
389
+ - `command.rb` (command spec & builder)
390
+ - `spec.rb` (model spec & yield builder)
391
+ - `shrinker.rb` (best-first shrinker + 6-step model validator + `seen` + `max_shrink_attempts`)
392
+ - `property.rb` (`Pbt::Stateful::Property` with `generate`, `generate_for_check`, `shrink_case`, `same_failure?`, `run`, `replay`, `stateful?`, `run_in_ractor`)
393
+ - Update `lib/pbt.rb` (`Pbt.stateful`) and `README.md`
394
+ - **Acceptance criteria**:
395
+ - [ ] Yield-builder DSL works inside RSpec with `expect(...)` and `let` (lexical `self` preserved).
396
+ - [ ] `initial_state` supports static value, block, and `Arbitrary` (shrunk when arbitrary); model states are deep-frozen with helpful `FrozenError` guidance.
397
+ - [ ] `sut`+`teardown` and `around` run cleanup in `ensure` on every trial and shrink candidate; teardown errors attach to original failure and halt shrinking.
398
+ - [ ] `guard` runs before `arguments`; `EmptyDomainError` makes a command ineligible for that state; vacuous step-0 raises `InvalidSpec`; argument exhaustion raises `GenerationError`.
399
+ - [ ] `Pbt.frequency` inside `c.arguments` and `m.initial_state` shrinks cleanly via `Sample.capture`.
400
+ - [ ] Symbolic `Ref` supports monotonic IDs, `nil` return values, scoped deep resolution, and dependency-safe shrinking.
401
+ - [ ] Shrinking performs post-failure truncation, chunk/single-step deletion, initial-state shrinking, and argument shrinking with `cover?` validation and `FailureSignature` preservation.
402
+ - [ ] `property.replay(details.counterexample)` deterministically replays the minimized `Run`.
403
+ - [ ] `Pbt.statistics(property, num_runs:, seed:) { |run| ... }` works directly.
404
+ - [ ] Explicit `worker: :ractor` raises `Pbt::InvalidConfiguration` before generation.
405
+ - [ ] Full suite and Standard Ruby pass (`bundle exec rake`).
406
+
407
+ ### PR 2: Exploration, Diagnostics & Reuse
408
+ - Seeded Swarm Testing (`swarm:` option driven by per-run `rng`; disabled during shrinking; retains full-command runs).
409
+ - Command execution & dead-guard coverage summary integrated with `Pbt.statistics`.
410
+ - Reusable class-based model DSL (`Pbt::Stateful::Model`) with optional Ruby 4 `Ractor.shareable_proc` support.
411
+ - Expected-error command helper and automatic final-counterexample replay to flag flaky SUTs.
412
+
413
+ ### PR 3: Future Experiments
414
+ - Concurrent / linearizability testing reusing the pure sequential model over Thread/Fiber/Ractor suffixes.
415
+ - `TapeRandom` choice-sequence integrated shrinking experiment (leveraging the fact that all randomness in `pbt` flows through `rng.rand(Range)` in `IntegerArbitrary` and `ChooseArbitrary`).
@@ -1,8 +1,11 @@
1
1
  # frozen_string_literal: true
2
2
 
3
+ require "pbt/arbitrary/sample"
4
+
3
5
  module Pbt
4
6
  module Arbitrary
5
7
  class EmptyDomainError < ArgumentError; end
8
+ class FilterExhaustedError < StandardError; end
6
9
 
7
10
  # Abstract class for generating random values on type `T`.
8
11
  #
@@ -11,9 +14,9 @@ module Pbt
11
14
  # Generate a value of type `T`, based on the provided random number generator.
12
15
  #
13
16
  # @abstract
14
- # @param rng [Random] Random number generator.
17
+ # @param rng [Random] Random number generator. Defaults to a new Random instance.
15
18
  # @return [Object] Random value of type `T`.
16
- def generate(rng)
19
+ def generate(rng = Random.new)
17
20
  raise NotImplementedError
18
21
  end
19
22
 
@@ -27,17 +30,66 @@ module Pbt
27
30
  raise NotImplementedError
28
31
  end
29
32
 
30
- # Create another arbitrary by applying `mapper` value by value.
33
+ # Normalize an arbitrary's 3-valued `cover?` result to `true`, `false`, or `nil`.
31
34
  #
32
- # @example
33
- # integer_generator = Pbt.integer
34
- # num_str_generator = integer_generator.map(->(n){ n.to_s }, ->(s) {s.to_i})
35
+ # @private
36
+ def self.cover_result(arbitrary, value)
37
+ return nil unless arbitrary.respond_to?(:cover?)
38
+
39
+ result = arbitrary.cover?(value)
40
+ (result == true || result == false) ? result : nil
41
+ end
42
+
43
+ # Check whether `value` belongs to this arbitrary's domain.
44
+ # Returns `true` if confirmed inside, `false` if confirmed outside,
45
+ # or `nil` if membership cannot be determined reliably (e.g. custom arbitraries).
46
+ #
47
+ # @param _value [Object]
48
+ # @return [Boolean, nil]
49
+ def cover?(_value)
50
+ nil
51
+ end
52
+
53
+ # Generate a value together with lazy shrinking provenance.
54
+ # @return [Sample]
55
+ def sample(rng = Random.new)
56
+ Sample.from_value(self, generate(rng))
57
+ end
58
+
59
+ # Whether property execution must retain a per-value shrink tree.
60
+ # @private
61
+ def contextual?
62
+ false
63
+ end
64
+
65
+ # Create another arbitrary by applying `mapper` (or a block) value by value.
66
+ # When `unmapper` is omitted, shrinking still works in `Pbt.property` and `sample`
67
+ # by preserving the underlying arbitrary's shrink tree.
68
+ #
69
+ # @example Block form (no unmapper required)
70
+ # Pbt.integer.map { |n| n.to_s }
35
71
  #
36
- # @param mapper [Proc] Proc to map generated values. Mainly used for generation.
37
- # @param unmapper [Proc] Proc to unmap generated values. Used for shrinking.
72
+ # @example Explicit mapper and unmapper (also supports bare `#shrink`)
73
+ # Pbt.integer.map(->(n) { n.to_s }, ->(s) { s.to_i })
74
+ #
75
+ # @param mapper [Proc, nil] Proc to map generated values.
76
+ # @param unmapper [Proc, nil] Optional proc to unmap generated values for bare `#shrink`.
38
77
  # @return [MapArbitrary] New arbitrary with mapped elements
39
- def map(mapper, unmapper)
40
- MapArbitrary.new(self, mapper, unmapper)
78
+ def map(mapper = nil, unmapper = nil, &block)
79
+ if mapper && block
80
+ raise ArgumentError, "map accepts either a mapper argument or a block, not both"
81
+ end
82
+
83
+ resolved_mapper = mapper || block
84
+ unless resolved_mapper.respond_to?(:call)
85
+ raise ArgumentError, "map requires a callable mapper or a block"
86
+ end
87
+
88
+ if !unmapper.nil? && !unmapper.respond_to?(:call)
89
+ raise ArgumentError, "unmapper must be callable or nil"
90
+ end
91
+
92
+ MapArbitrary.new(self, resolved_mapper, unmapper)
41
93
  end
42
94
 
43
95
  # Create another arbitrary by filtering values against `refinement`.
@@ -50,10 +102,11 @@ module Pbt
50
102
  # even_integer_generator = integer_generator.filter { |x| x.even? }
51
103
  # # or `integer_generator.filter(&:even?)`
52
104
  #
105
+ # @param max_attempts [Integer] Maximum generation attempts before raising `FilterExhaustedError`.
53
106
  # @param refinement [Proc] Predicate proc to test each produced element. Return true to keep the element, false otherwise.
54
107
  # @return [FilterArbitrary] New arbitrary filtered using `refinement`.
55
- def filter(&refinement)
56
- FilterArbitrary.new(self, &refinement)
108
+ def filter(max_attempts: FilterArbitrary::DEFAULT_MAX_ATTEMPTS, &refinement)
109
+ FilterArbitrary.new(self, max_attempts: max_attempts, &refinement)
57
110
  end
58
111
  end
59
112
  end