zizek 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +115 -0
- data/GUIDE.md +774 -0
- data/LICENSE +201 -0
- data/README.md +201 -0
- data/docs/assets/zizek-rb-logo-dark.svg +16 -0
- data/docs/assets/zizek-rb-logo.svg +16 -0
- data/examples/combinators.rb +93 -0
- data/examples/coverage_assertions.rb +138 -0
- data/examples/database_workflow.rb +120 -0
- data/examples/die_hard.rb +122 -0
- data/examples/hypothesis_quickstart.rb +44 -0
- data/examples/keyed_draws.rb +123 -0
- data/examples/ledger.rb +114 -0
- data/examples/min_stack.rb +108 -0
- data/examples/minitest_suite.rb +64 -0
- data/examples/multi.rb +46 -0
- data/examples/queue.rb +122 -0
- data/examples/roundtrip.rb +71 -0
- data/examples/stack.rb +68 -0
- data/examples/tour.rb +126 -0
- data/lib/zizek/database.rb +211 -0
- data/lib/zizek/gen.rb +10377 -0
- data/lib/zizek/generators.rb +879 -0
- data/lib/zizek/minitest.rb +94 -0
- data/lib/zizek/rspec.rb +67 -0
- data/lib/zizek/settings.rb +249 -0
- data/lib/zizek/shell.rb +2938 -0
- data/lib/zizek/stateful.rb +349 -0
- data/lib/zizek/version.rb +8 -0
- data/lib/zizek.rb +8 -0
- metadata +109 -0
data/GUIDE.md
ADDED
|
@@ -0,0 +1,774 @@
|
|
|
1
|
+
# Zizek for Ruby — feature guide
|
|
2
|
+
|
|
3
|
+
A walkthrough of every feature the Ruby runner currently supports, with
|
|
4
|
+
runnable examples for each. See [README.md](README.md) for installation and
|
|
5
|
+
[dev.md](dev.md) for the core/shell architecture; this guide is about *using*
|
|
6
|
+
the API.
|
|
7
|
+
|
|
8
|
+
## Public API
|
|
9
|
+
|
|
10
|
+
Use `require "zizek"` for the package, and `zizek/minitest` or `zizek/rspec`
|
|
11
|
+
for the optional framework integrations. The APIs described in this guide
|
|
12
|
+
are supported, including these extension protocols:
|
|
13
|
+
|
|
14
|
+
- Custom generators implement public `do_draw(test_case)`. Include
|
|
15
|
+
`Zizek::Generators::Combinators` to add `map`, `filter`, `flat_map`, `|`,
|
|
16
|
+
and `example` to your own generator.
|
|
17
|
+
- Custom state machines implement public `rules` and `invariants`, returning
|
|
18
|
+
arrays of `Zizek::Stateful::Rule` objects.
|
|
19
|
+
- Custom example databases implement public `fetch`, `save`, and `delete`.
|
|
20
|
+
|
|
21
|
+
A property's test case exposes `draw`, `integer`, `float`, `boolean`,
|
|
22
|
+
`weighted`, `bytes`, `string`, `clone_stream`, `next_key`, `assume`, `note`,
|
|
23
|
+
`always`, `unreachable`, `sometimes`, `cover`, `cost`, `target`, `start_span`,
|
|
24
|
+
and `stop_span`. Use `Failure` or `Result` for reports and reproduce blobs.
|
|
25
|
+
|
|
26
|
+
Create generators through `Zizek::Generators` factories and configure them
|
|
27
|
+
with their fluent builders. Concrete generator classes, engine modules,
|
|
28
|
+
recorded draws, mutable bookkeeping, and helper methods are internal and
|
|
29
|
+
hidden with Ruby visibility rules. They are not supported extension points.
|
|
30
|
+
|
|
31
|
+
## Properties and `Zizek.check`
|
|
32
|
+
|
|
33
|
+
A property is a callable (or block) taking a test case. It fails by raising
|
|
34
|
+
any exception or by returning `false`.
|
|
35
|
+
|
|
36
|
+
```ruby
|
|
37
|
+
require "zizek"
|
|
38
|
+
|
|
39
|
+
property = lambda do |test_case|
|
|
40
|
+
value = test_case.draw(
|
|
41
|
+
Zizek::Generators.integers(-100..100)
|
|
42
|
+
)
|
|
43
|
+
raise "value must be less than five: #{value}" unless value < 5
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
Zizek.check(property, examples: 200, seed: 1)
|
|
47
|
+
|
|
48
|
+
# Or with a block:
|
|
49
|
+
Zizek.check(examples: 200, seed: 1) do |test_case|
|
|
50
|
+
# ...
|
|
51
|
+
end
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
`Zizek.check` accepts keyword configuration:
|
|
55
|
+
|
|
56
|
+
| Keyword | Default | Meaning |
|
|
57
|
+
|---------|---------|---------|
|
|
58
|
+
| `examples:` | 100 | Valid (non-rejected) examples to run before declaring success. |
|
|
59
|
+
| `max_rejections:` | 458 | Initial rejection allowance; each valid example earns 100 more. |
|
|
60
|
+
| `max_shrinks:` | 500 | Accepted shrink candidates before the search stops. Zero disables the cap. |
|
|
61
|
+
| `max_stall:` | 200 | Consecutive rejected candidates before the search stops. Zero disables the cap. |
|
|
62
|
+
| `suppress_filter_too_much:` | `false` | Disable the early excessive-filtering health check. |
|
|
63
|
+
| `suppress_data_too_large:` | `false` | Disable the early choice-budget-overrun health check. |
|
|
64
|
+
| `suppress_large_base_example:` | `false` | Allow an all-simplest input that uses more than half the choice budget. |
|
|
65
|
+
| `suppress_too_slow:` | `false` | Disable the one-second cumulative generator-time health check. |
|
|
66
|
+
| `seed:` | random | Seed for random exploration. Reproduces a run on one machine; portable reproduction uses blobs instead. |
|
|
67
|
+
| `derandomize:` | `false` | Derive the seed from where the property was written, so a suite explores the same examples everywhere. |
|
|
68
|
+
| `must_cover:` | `[]` | Labels the completed run must have covered. See [Coverage assertions](#coverage-assertions). |
|
|
69
|
+
| `database:` | `nil` | Where counterexamples are stored between runs: a path, `":memory:"`, `true` for `.zizek/examples`, or a database object. See [The example database](#the-example-database). |
|
|
70
|
+
| `database_key:` | derived | The name a stored counterexample is filed under. Set it to share a directory with another language. |
|
|
71
|
+
| `reproduce:` | `nil` | A failure blob. When set, `check` replays exactly that example and runs nothing else. |
|
|
72
|
+
| `settings:` | active profile | A `Zizek::Settings` to start from; the keywords above override it. |
|
|
73
|
+
|
|
74
|
+
Every keyword except `reproduce:` and `settings:` is a field on
|
|
75
|
+
`Zizek::Settings`, and `check` takes its defaults from the active profile.
|
|
76
|
+
See [Settings and profiles](#settings-and-profiles).
|
|
77
|
+
|
|
78
|
+
On failure `check` raises:
|
|
79
|
+
|
|
80
|
+
- `Zizek::Failure` — a property failed; it is already shrunk and carries a
|
|
81
|
+
reproduce blob;
|
|
82
|
+
- `Zizek::Flaky` — a property failed, then did not fail the same way when
|
|
83
|
+
that example was replayed; it carries the original error and the unshrunk
|
|
84
|
+
example's blob (`Zizek.run` reports it as outcome `:flaky`);
|
|
85
|
+
- `Zizek::TooManyRejections` — assumptions rejected too many examples;
|
|
86
|
+
- `Zizek::InvalidReplay` / `Zizek::ReplayOverrun` — a reproduce blob is
|
|
87
|
+
malformed or does not match the property's draws.
|
|
88
|
+
|
|
89
|
+
```ruby
|
|
90
|
+
begin
|
|
91
|
+
Zizek.check(property, examples: 200, seed: 1)
|
|
92
|
+
rescue Zizek::Failure => failure
|
|
93
|
+
puts failure.blob
|
|
94
|
+
puts failure.property_cause # the original exception; aliased as #cause
|
|
95
|
+
puts failure.values # the named draws: [[:value, 5]]
|
|
96
|
+
puts failure.seed # the seed this search ran under
|
|
97
|
+
puts failure.notes
|
|
98
|
+
end
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
### `Zizek.run`, when the outcome is a value
|
|
102
|
+
|
|
103
|
+
`check` raises because a test that quietly ignored a counterexample would be
|
|
104
|
+
worse than no test. For code that wants the verdict instead — a harness
|
|
105
|
+
collecting blobs, a script comparing shrink quality, a property about a
|
|
106
|
+
property — `run` returns it:
|
|
107
|
+
|
|
108
|
+
```ruby
|
|
109
|
+
result = Zizek.run(property, examples: 200, seed: 1)
|
|
110
|
+
|
|
111
|
+
result.ok? # => false
|
|
112
|
+
result.outcome # => :failure
|
|
113
|
+
result.error # => the Zizek::Failure
|
|
114
|
+
result.blob # => "AAEAAAAACgEAAAAF"
|
|
115
|
+
result.values # => [[:value, 5]]
|
|
116
|
+
result.notes # => []
|
|
117
|
+
result.costs # => {}
|
|
118
|
+
result.examples_run # => 3
|
|
119
|
+
result.seed # => 1
|
|
120
|
+
result.raise! # raises what check would have raised
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
`outcome` is one of `:ok`, `:failure` (a counterexample), `:coverage` (a
|
|
124
|
+
`must_cover` label was missed), `:exhausted` (too many rejections), or
|
|
125
|
+
`:unhealthy` (a health check tripped). An invalid blob or a bad argument still
|
|
126
|
+
raises: those are mistakes in the call, not outcomes of the search.
|
|
127
|
+
|
|
128
|
+
## Settings and profiles
|
|
129
|
+
|
|
130
|
+
`Zizek::Settings` is a frozen value object. `with` returns a new one rather
|
|
131
|
+
than mutating, so a settings object can be shared without one caller
|
|
132
|
+
reconfiguring another:
|
|
133
|
+
|
|
134
|
+
```ruby
|
|
135
|
+
fast = Zizek::Settings.new(examples: 50)
|
|
136
|
+
thorough = fast.with(examples: 5_000, max_shrinks: 0)
|
|
137
|
+
|
|
138
|
+
Zizek.check(property, settings: thorough)
|
|
139
|
+
Zizek.check(property, settings: thorough, examples: 10) # keywords still win
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
Unknown names raise at construction, so a misspelled option is a loud error
|
|
143
|
+
rather than one that silently did nothing:
|
|
144
|
+
|
|
145
|
+
```ruby
|
|
146
|
+
Zizek::Settings.new(exampels: 50)
|
|
147
|
+
# => ArgumentError: unknown setting: :exampels
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
A suite names the shapes it runs in once:
|
|
151
|
+
|
|
152
|
+
```ruby
|
|
153
|
+
# test_helper.rb
|
|
154
|
+
Zizek.register_profile(:dev, examples: 100)
|
|
155
|
+
Zizek.register_profile(:ci, examples: 5_000, derandomize: true)
|
|
156
|
+
Zizek.register_profile(:deep, base: Zizek.profile(:ci), examples: 50_000)
|
|
157
|
+
Zizek.load_profile(ENV["CI"] ? :ci : :dev)
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
`Zizek.check` then uses the active profile, and any keyword overrides it for
|
|
161
|
+
that call. Inside a suite prefer `with_profile`, which restores the previous
|
|
162
|
+
profile even if the block raises:
|
|
163
|
+
|
|
164
|
+
```ruby
|
|
165
|
+
Zizek.with_profile(:deep) { Zizek.check(property) }
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
`Zizek.current_settings` and `Zizek.current_profile` report what is active.
|
|
169
|
+
The registry is process-global, like Hypothesis's.
|
|
170
|
+
|
|
171
|
+
### Repeating a search versus replaying an example
|
|
172
|
+
|
|
173
|
+
Two different things, worth keeping apart:
|
|
174
|
+
|
|
175
|
+
- The **blob** replays one recorded example. It is portable: the same blob
|
|
176
|
+
replays in the Go, Python, Rust, Zig, TypeScript, Java, C++, and Clojure
|
|
177
|
+
packages. This is what a failure report is for.
|
|
178
|
+
- The **seed** repeats the whole search — the same examples in the same order
|
|
179
|
+
— but only in this language and this version. Every failure now reports the
|
|
180
|
+
seed it ran under.
|
|
181
|
+
|
|
182
|
+
`derandomize: true` derives the seed from the property's source location, so
|
|
183
|
+
a suite explores the same examples on every machine and in every CI run
|
|
184
|
+
instead of a fresh random set each time. An explicit `seed:` beats it.
|
|
185
|
+
|
|
186
|
+
## Low-level draws
|
|
187
|
+
|
|
188
|
+
`Zizek::TestCase` exposes the recorded primitive choices directly:
|
|
189
|
+
|
|
190
|
+
```ruby
|
|
191
|
+
value = test_case.integer(0, 100) # shrinks toward 0
|
|
192
|
+
value = test_case.integer(0, 100, shrink_towards: 10)
|
|
193
|
+
flag = test_case.boolean # shrinks toward false
|
|
194
|
+
payload = test_case.bytes(1, 4) # 1..4 bytes, shortlex
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
Every draw is recorded, which is what makes shrinking and blob replay work.
|
|
198
|
+
Narrow integer ranges (fewer than 256 values) are drawn uniformly, so a draw
|
|
199
|
+
that indexes a list picks every option equally often. Wide ranges use a
|
|
200
|
+
per-case swarm: the generated core supplies candidates such as the interval
|
|
201
|
+
endpoints and values near zero, so bugs at boundaries (like `1` in
|
|
202
|
+
`0..=1_000_000`) are found reliably.
|
|
203
|
+
|
|
204
|
+
## Generator objects
|
|
205
|
+
|
|
206
|
+
`test_case.draw(generator)` routes any object with a `do_draw(test_case)`
|
|
207
|
+
method through the test case:
|
|
208
|
+
|
|
209
|
+
```ruby
|
|
210
|
+
value = test_case.draw(Zizek::Generators.integers(-100..100), name: :value)
|
|
211
|
+
name = test_case.draw(Zizek::Generators.text(min: 1, max: 8), name: :name)
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
Passing `name:` puts the value in the failure's `Falsifying example:` line. It
|
|
215
|
+
is a label for the report only: it records no choice, so naming a draw cannot
|
|
216
|
+
change the blob or what any other language package replays.
|
|
217
|
+
|
|
218
|
+
- `Zizek::Generators.integers(min:, max:, shrink_towards:)` — full-range
|
|
219
|
+
64-bit signed integers until bounded; `shrink_towards` moves the shrink
|
|
220
|
+
destination. Also takes a range: `integers(-100..100)`.
|
|
221
|
+
- `Zizek::Generators.text(min:, max:, alphabet:)` — text over an alphabet, by
|
|
222
|
+
default every Unicode scalar value. The bounds are lengths **in
|
|
223
|
+
characters**, and `alphabet: "abc"` or
|
|
224
|
+
`alphabet: Zizek.alphabet_range(0x20, 0x7E)` narrows the characters drawn.
|
|
225
|
+
Values shrink towards `"0"`, then the digits and uppercase letters, so a
|
|
226
|
+
minimal failing string is one you can retype.
|
|
227
|
+
- `Zizek::Generators.lists(element, min:, max:)` — a variable-length sequence.
|
|
228
|
+
Each element is recorded as its own span, so the shrinker deletes whole
|
|
229
|
+
elements — including ones in the middle — before it simplifies the values
|
|
230
|
+
inside them.
|
|
231
|
+
- `Zizek::Generators.sets(element)`, `.hashes(keys, values)`, and
|
|
232
|
+
`lists(element, unique_by: key)` — the same sequence with a distinctness
|
|
233
|
+
constraint. A value that repeats one already drawn costs the *element*, not
|
|
234
|
+
the example: its choices are discarded and another candidate is drawn. Asking
|
|
235
|
+
for more distinct values than the element generator can produce rejects the
|
|
236
|
+
example rather than looping, so
|
|
237
|
+
`sets(integers(max: 1), min: 5)` reports "no valid examples"
|
|
238
|
+
instead of hanging.
|
|
239
|
+
- `Zizek::Generators.floats(min:, max:, allow_nan:, allow_infinity:,
|
|
240
|
+
smallest_nonzero_magnitude:)` — IEEE-754 doubles with portable shrinking.
|
|
241
|
+
NaN and the infinities are off unless asked for: a property that has an
|
|
242
|
+
opinion about them should say so.
|
|
243
|
+
- `Zizek::Generators.booleans`, `.weighted_booleans(probability)`, and
|
|
244
|
+
`.binary(min:, max:)` — booleans shrinking towards `false` (the weight
|
|
245
|
+
biases generation only, so a blob replays the same whatever probability
|
|
246
|
+
produced it) and byte strings shrinking shortlex.
|
|
247
|
+
- `Zizek::Generators.builds(target, *parts, **keyword_parts)` — assemble an
|
|
248
|
+
object out of drawn parts. The target is any class or callable, and an
|
|
249
|
+
argument that is not a generator is passed through unchanged.
|
|
250
|
+
- `Zizek::Generators.recursive(leaf) { |subtrees| branch }` — bounded trees and
|
|
251
|
+
other recursively-defined values; see [Recursive generators](#recursive-generators).
|
|
252
|
+
|
|
253
|
+
Every generator carries `map`, `filter`, `flat_map`, and `|` (which is
|
|
254
|
+
`one_of`, and flattens, so `a | b | c` is one three-way choice rather than
|
|
255
|
+
nested pairs). `flat_map` builds the *next generator* from each drawn value —
|
|
256
|
+
monadic bind:
|
|
257
|
+
|
|
258
|
+
```ruby
|
|
259
|
+
lists_sized_by_draw = Zizek::Generators.integers(0..5).flat_map do |length|
|
|
260
|
+
Zizek::Generators.lists(Zizek::Generators.booleans, min: length, max: length)
|
|
261
|
+
end
|
|
262
|
+
```
|
|
263
|
+
|
|
264
|
+
Both draws share one `flat_map` span, so the shrinker can delete or simplify
|
|
265
|
+
the unit together, and the inner draws replay by the ordinary prefix rules.
|
|
266
|
+
|
|
267
|
+
The keyword form and the fluent builders are the same generator:
|
|
268
|
+
|
|
269
|
+
```ruby
|
|
270
|
+
Zizek::Generators.integers(min: 0, max: 100)
|
|
271
|
+
Zizek::Generators.integers.min_value(0).max_value(100)
|
|
272
|
+
```
|
|
273
|
+
|
|
274
|
+
Keywords are the preferred spelling at a creation site — the bounds are
|
|
275
|
+
self-documenting and there is no positional ambiguity between minimum,
|
|
276
|
+
maximum, and target. The builders remain for constructing a generator in
|
|
277
|
+
stages:
|
|
278
|
+
|
|
279
|
+
```ruby
|
|
280
|
+
generator = Zizek::Generators.integers
|
|
281
|
+
generator = generator.min_value(lower) if lower
|
|
282
|
+
generator = generator.max_value(upper) if upper
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
Builders are immutable: each configuration call returns a new generator, so
|
|
286
|
+
they can be shared and reused freely.
|
|
287
|
+
|
|
288
|
+
To see what a generator produces, without writing a test:
|
|
289
|
+
|
|
290
|
+
```ruby
|
|
291
|
+
Zizek::Generators.integers(0..100).example(seed: 3) # => 24
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
`example` records nothing and cannot be replayed; it is for a REPL or a
|
|
295
|
+
docstring. A filter that rejects every attempt raises `Zizek::Unsatisfiable`
|
|
296
|
+
rather than looping.
|
|
297
|
+
|
|
298
|
+
If a property draws often enough that the `Zizek::Generators.` prefix is
|
|
299
|
+
noise, include the module:
|
|
300
|
+
|
|
301
|
+
```ruby
|
|
302
|
+
class SplitBillTest < Minitest::Test
|
|
303
|
+
include Zizek::Generators # integers, text, lists, ... without the prefix
|
|
304
|
+
end
|
|
305
|
+
```
|
|
306
|
+
|
|
307
|
+
### Recursive generators
|
|
308
|
+
|
|
309
|
+
`recursive` takes a generator for leaf values and a block that receives the
|
|
310
|
+
generator for child subtrees. The block returns a generator that combines one
|
|
311
|
+
or more children into the same result type:
|
|
312
|
+
|
|
313
|
+
```ruby
|
|
314
|
+
Tree = Struct.new(:value, :left, :right, keyword_init: true)
|
|
315
|
+
|
|
316
|
+
leaves = Zizek::Generators.integers(0..9).map do |value|
|
|
317
|
+
Tree.new(value: value)
|
|
318
|
+
end
|
|
319
|
+
|
|
320
|
+
trees = Zizek::Generators.recursive(leaves) do |subtrees|
|
|
321
|
+
Zizek::Generators.composite do |test_case|
|
|
322
|
+
Tree.new(
|
|
323
|
+
left: test_case.draw(subtrees),
|
|
324
|
+
right: test_case.draw(subtrees)
|
|
325
|
+
)
|
|
326
|
+
end
|
|
327
|
+
end
|
|
328
|
+
trees = trees.max_depth(6).max_leaves(32)
|
|
329
|
+
|
|
330
|
+
tree = test_case.draw(trees)
|
|
331
|
+
```
|
|
332
|
+
|
|
333
|
+
The defaults are a maximum branch depth of 32 and at most 100 leaves.
|
|
334
|
+
`max_depth(0)` produces leaves only; `max_leaves` bounds the total base values,
|
|
335
|
+
not the number of branch nodes. An attempt that exceeds the leaf budget is
|
|
336
|
+
discarded and retried with a lower branching probability. If bounded retries
|
|
337
|
+
cannot construct a value, the example is rejected like a failed assumption.
|
|
338
|
+
|
|
339
|
+
Every subtree occupies a `recursive` span. The shrinker can therefore replace
|
|
340
|
+
an outer tree with one of its descendants, hoisting a deeply nested witness to
|
|
341
|
+
the root before simplifying its leaves. A leaf forced by `max_depth` is still a
|
|
342
|
+
recorded decision, so replay and the shrinker's choice positions stay aligned.
|
|
343
|
+
|
|
344
|
+
### Shrink targets
|
|
345
|
+
|
|
346
|
+
By default integers shrink toward 0 (clamped into the interval). With
|
|
347
|
+
`shrink_towards`, the reported counterexample is the failing value closest to
|
|
348
|
+
your target rather than the smallest one:
|
|
349
|
+
|
|
350
|
+
```ruby
|
|
351
|
+
value = test_case.draw(
|
|
352
|
+
Zizek::Generators.integers(min: -1000, max: 1000, shrink_towards: 500)
|
|
353
|
+
)
|
|
354
|
+
# if the property fails for value > 700, the failure shrinks to exactly 701
|
|
355
|
+
```
|
|
356
|
+
|
|
357
|
+
Run [`examples/tour.rb`](examples/tour.rb) to watch this happen.
|
|
358
|
+
|
|
359
|
+
## Assumptions
|
|
360
|
+
|
|
361
|
+
`test_case.assume(condition)` rejects the current example when the condition
|
|
362
|
+
is falsy. Rejection is control flow, not failure: `check` keeps generating
|
|
363
|
+
until it has run `examples` valid examples or exceeds the adaptive rejection
|
|
364
|
+
budget `max_rejections + 100 * valid`, at which point it raises
|
|
365
|
+
`Zizek::TooManyRejections`.
|
|
366
|
+
|
|
367
|
+
```ruby
|
|
368
|
+
value = test_case.integer(0, 100)
|
|
369
|
+
test_case.assume(value.even?) # odd draws are discarded, not failed
|
|
370
|
+
```
|
|
371
|
+
|
|
372
|
+
Shrinking respects assumptions too: a shrink candidate that gets rejected is
|
|
373
|
+
skipped, so the shrunk counterexample always satisfies every assumption.
|
|
374
|
+
|
|
375
|
+
## Coverage assertions
|
|
376
|
+
|
|
377
|
+
A property can pass vacuously: the branch it exists to test never executes,
|
|
378
|
+
and the run goes green. `test_case.sometimes(label, condition)` turns that
|
|
379
|
+
silent gap into a failure — the condition must hold in at least one valid
|
|
380
|
+
example of the run — and `test_case.cover(label, condition, percent:)`
|
|
381
|
+
requires a rate. The verdict is per run, not per example; observations from
|
|
382
|
+
rejected or overrun examples are discarded with the rest of the example.
|
|
383
|
+
Labels whose call site may itself never execute are declared up front:
|
|
384
|
+
|
|
385
|
+
```ruby
|
|
386
|
+
Zizek.check(property, must_cover: [Zizek::Coverage.new("overflow path"),
|
|
387
|
+
Zizek::Coverage.new("cache hit", percent: 30)])
|
|
388
|
+
```
|
|
389
|
+
|
|
390
|
+
A missed requirement raises `Zizek::FailedCoverage` naming the label and
|
|
391
|
+
counters — no counterexample, so no blob or shrinking; the same seed
|
|
392
|
+
reproduces the same verdict. The per-example half is sugar over ordinary
|
|
393
|
+
failure: `test_case.always(label, condition)` fails the example the moment
|
|
394
|
+
its condition does not hold (and, holding, counts as covering its label, so
|
|
395
|
+
pairing it with `must_cover:` also demands the site be reached), while
|
|
396
|
+
`test_case.unreachable(label)` fails the example that reaches it, shrinking
|
|
397
|
+
to the simplest reaching input. The accept rule is compiled from
|
|
398
|
+
`ir/lifecycle.zir`, so every language agrees on it.
|
|
399
|
+
|
|
400
|
+
## Notes
|
|
401
|
+
|
|
402
|
+
`test_case.note(message)` attaches a diagnostic line to the current example.
|
|
403
|
+
Notes are collected per example and reported on the shrunk failure — both in
|
|
404
|
+
`failure.notes` and in the rendered `Trace:` section — so they describe the
|
|
405
|
+
*minimal* example, not the noisy first one.
|
|
406
|
+
|
|
407
|
+
```ruby
|
|
408
|
+
test_case.note("transfer #{amount} from #{source.inspect} to #{destination.inspect}")
|
|
409
|
+
```
|
|
410
|
+
|
|
411
|
+
## Failures, blobs, and replay
|
|
412
|
+
|
|
413
|
+
A shrunk failure carries a portable, base64 reproduce blob:
|
|
414
|
+
|
|
415
|
+
```ruby
|
|
416
|
+
begin
|
|
417
|
+
Zizek.check(property, examples: 200, seed: 1)
|
|
418
|
+
rescue Zizek::Failure => failure
|
|
419
|
+
# Replays exactly the recorded choices — nothing is random.
|
|
420
|
+
Zizek.check(property, reproduce: failure.blob)
|
|
421
|
+
end
|
|
422
|
+
```
|
|
423
|
+
|
|
424
|
+
Blobs encode the recorded choice sequence, not the RNG seed, so they replay
|
|
425
|
+
bit-exactly across machines *and languages*: a blob printed by the Ruby runner
|
|
426
|
+
replays in the Go, Python, and Rust packages (the shared fixture
|
|
427
|
+
`AAEAAAAACgEAAAAF` replays the integer `5` in all four test suites, and the
|
|
428
|
+
ledger/queue examples pin cross-language blob fixtures of their own).
|
|
429
|
+
|
|
430
|
+
Replay is defensive:
|
|
431
|
+
|
|
432
|
+
- a structurally invalid blob raises `Zizek::InvalidReplay`;
|
|
433
|
+
- a blob that ends before the property stops drawing raises
|
|
434
|
+
`Zizek::ReplayOverrun`;
|
|
435
|
+
- a choice of the wrong kind (or out of range) puns to the requested draw's
|
|
436
|
+
unit value instead of failing, so replay stays deterministic while you edit
|
|
437
|
+
the property.
|
|
438
|
+
|
|
439
|
+
A pun keeps the replay running, but it means the values you get back are not
|
|
440
|
+
the ones the blob recorded — and nothing in the values themselves says so. It
|
|
441
|
+
bites hardest when a draw's bounds depend on earlier draws, because then the
|
|
442
|
+
same blob can fit one run and pun on the next. Every replay therefore counts
|
|
443
|
+
its puns: `failure.puns` reports how many, and a non-zero count adds a warning line
|
|
444
|
+
to the rendered failure. Treat it as "this blob no longer reproduces what it
|
|
445
|
+
recorded", not as a hard error.
|
|
446
|
+
|
|
447
|
+
## Keyed draws
|
|
448
|
+
|
|
449
|
+
Positional replay gives a draw its meaning from how many draws came before
|
|
450
|
+
it, which is exactly right until the property changes shape: insert one new
|
|
451
|
+
draw and every recorded choice after it answers the wrong question. A keyed
|
|
452
|
+
draw is addressed by a *name* instead, so a blob keeps meaning what it
|
|
453
|
+
recorded while the property around it evolves:
|
|
454
|
+
|
|
455
|
+
```ruby
|
|
456
|
+
timeout = test_case.integer(1, 30, key: Zizek.key_hash(:timeout))
|
|
457
|
+
retries = test_case.integer(0, 9, key: Zizek.key_hash(:retries))
|
|
458
|
+
```
|
|
459
|
+
|
|
460
|
+
- `Zizek.key_hash(*components)` names a decision made at most once per
|
|
461
|
+
example. Components are strings, symbols, or integers, hashed by the
|
|
462
|
+
compiled encoding (`ir/keys.zir`), so the same name means the same entry
|
|
463
|
+
in every language package.
|
|
464
|
+
- `test_case.next_key(*components)` appends an execution-local ordinal,
|
|
465
|
+
counted *within that name*, for a decision made repeatedly —
|
|
466
|
+
`next_key(:read, connection)` numbers that connection's reads
|
|
467
|
+
independently of every other sequence.
|
|
468
|
+
|
|
469
|
+
On replay a keyed draw looks its name up wherever it sits in the stream, and
|
|
470
|
+
the positional cursor steps over keyed entries, so the two addressing modes
|
|
471
|
+
never shift each other. A name the blob does not carry means the property
|
|
472
|
+
changed, not that the blob is malformed: the draw uses its simplest value,
|
|
473
|
+
and the failure counts and reports the miss (`failure.key_misses`) instead
|
|
474
|
+
of raising.
|
|
475
|
+
|
|
476
|
+
[`examples/keyed_draws.rb`](examples/keyed_draws.rb) runs the whole
|
|
477
|
+
contrast: the same edit to a property breaks a positional blob with
|
|
478
|
+
`ReplayOverrun` while the keyed blob still reproduces its failure, one
|
|
479
|
+
counted miss and all.
|
|
480
|
+
|
|
481
|
+
## The example database
|
|
482
|
+
|
|
483
|
+
Set `database:` and a failure outlives the run that found it:
|
|
484
|
+
|
|
485
|
+
```ruby
|
|
486
|
+
Zizek.check(property, database: ".zizek/examples", database_key: "split_is_exact")
|
|
487
|
+
```
|
|
488
|
+
|
|
489
|
+
The next run replays every stored entry for that key *before* it generates
|
|
490
|
+
anything, so a bug found yesterday reproduces on the first example. An entry
|
|
491
|
+
that no longer reproduces — because the bug was fixed — is deleted on the
|
|
492
|
+
replay that failed to reproduce it, so the directory keeps itself current.
|
|
493
|
+
|
|
494
|
+
The database is off until asked for, and a profile turns it on for a whole
|
|
495
|
+
suite (`Zizek.register_profile(:local, database: ".zizek/examples")`).
|
|
496
|
+
|
|
497
|
+
What makes it more than a cache is the layout: raw choice streams, filed under
|
|
498
|
+
a hash the IR computes, which the Python and Go packages write identically. A
|
|
499
|
+
counterexample your Go service's suite found replays in your Ruby client's
|
|
500
|
+
suite out of one checked-in directory, provided both name the property the same
|
|
501
|
+
with `database_key:`. Left unset, the key is derived from where the property
|
|
502
|
+
was written — fine for one language, meaningless to another.
|
|
503
|
+
|
|
504
|
+
[`examples/database_workflow.rb`](examples/database_workflow.rb) runs the
|
|
505
|
+
whole cycle in a temporary directory: find and store, replay on the first
|
|
506
|
+
attempt of the next run, and watch the fixed bug's entry collect itself.
|
|
507
|
+
|
|
508
|
+
## Shrink costs
|
|
509
|
+
|
|
510
|
+
The shrinker's default order answers "what is the simplest *input* that
|
|
511
|
+
fails". A fault-injecting property usually wants a different question
|
|
512
|
+
answered first: "how few *faults* suffice?" Under the default order, turning
|
|
513
|
+
one injected fault off is worth exactly as much as making one integer a
|
|
514
|
+
little smaller, so the search may keep four faults because the stream around
|
|
515
|
+
them happens to be shorter.
|
|
516
|
+
|
|
517
|
+
`test_case.cost(label, amount = 1)` declares what an example spent:
|
|
518
|
+
|
|
519
|
+
```ruby
|
|
520
|
+
if test_case.boolean(key: test_case.next_key(:inject, site))
|
|
521
|
+
test_case.cost(:reorder) # this example spent one fault
|
|
522
|
+
reorder(queue)
|
|
523
|
+
end
|
|
524
|
+
```
|
|
525
|
+
|
|
526
|
+
The shrinker then orders candidates by **total cost first** — a candidate
|
|
527
|
+
injecting one fewer fault wins even when its choice stream is longer — with
|
|
528
|
+
the ordinary stream order as the tiebreak, so a property that never calls
|
|
529
|
+
`cost` shrinks exactly as before. The failure reports the per-label
|
|
530
|
+
breakdown on `failure.costs` and in the rendered `cost:` line.
|
|
531
|
+
|
|
532
|
+
## Targeted generation
|
|
533
|
+
|
|
534
|
+
`cost` orders shrinking after a failure; `target` steers generation toward
|
|
535
|
+
one. A property records one finite score per label per example:
|
|
536
|
+
|
|
537
|
+
```ruby
|
|
538
|
+
value = test_case.integer(0, 100_000)
|
|
539
|
+
test_case.target(value)
|
|
540
|
+
raise "overflowed" if value >= 99_990
|
|
541
|
+
```
|
|
542
|
+
|
|
543
|
+
The run keeps the best-scoring valid example per label and, on a schedule,
|
|
544
|
+
hill-climbs its recorded choices — so a failing band at the far end of the
|
|
545
|
+
scored quantity is found in a fraction of the examples random search needs.
|
|
546
|
+
A failing climb trial shrinks and reports like any other counterexample, and
|
|
547
|
+
nothing new enters the blob. Target several quantities by naming them:
|
|
548
|
+
`test_case.target(depth, label: "depth")` — each label may be observed at
|
|
549
|
+
most once per example, and every score must be finite.
|
|
550
|
+
|
|
551
|
+
## Structural spans (advanced)
|
|
552
|
+
|
|
553
|
+
Framework-level code can group choices into labelled ranges so the shrinker
|
|
554
|
+
can delete whole semantic units at once:
|
|
555
|
+
|
|
556
|
+
```ruby
|
|
557
|
+
span_id = test_case.start_span("stateful.rule")
|
|
558
|
+
# ... draws ...
|
|
559
|
+
test_case.stop_span # or test_case.stop_span(discarded: true)
|
|
560
|
+
```
|
|
561
|
+
|
|
562
|
+
The state-machine runner uses spans for each rule, and
|
|
563
|
+
`Zizek::Generators.lists` uses one per element, which is why a 40-step failing
|
|
564
|
+
run shrinks to the three or six steps that matter and a 30-element list
|
|
565
|
+
shrinks to the one element that does. Most tests never call these directly.
|
|
566
|
+
|
|
567
|
+
## Cloned streams (advanced)
|
|
568
|
+
|
|
569
|
+
`test_case.clone_stream` returns a child test case that draws from a stream
|
|
570
|
+
of its own, occupying one choice position in the parent's. The child's draws
|
|
571
|
+
neither shift nor are shifted by the parent's, its stream is framed inside
|
|
572
|
+
the parent's blob with the engine's wire format, and the shrinker descends
|
|
573
|
+
into it with the same schedule — which is what makes it the seam a
|
|
574
|
+
concurrent runner would use, though today the child is drawn from
|
|
575
|
+
sequentially. Notes, costs, coverage, and the choice budget all belong to
|
|
576
|
+
the family rather than to one stream.
|
|
577
|
+
|
|
578
|
+
## Stateful testing
|
|
579
|
+
|
|
580
|
+
Subclass `Zizek::Stateful::Machine` and declare rules and invariants. A rule
|
|
581
|
+
body runs with `self` bound to the machine, so it reads and writes state
|
|
582
|
+
directly:
|
|
583
|
+
|
|
584
|
+
```ruby
|
|
585
|
+
class Counter < Zizek::Stateful::Machine
|
|
586
|
+
def initialize(start: 0)
|
|
587
|
+
@value = start
|
|
588
|
+
end
|
|
589
|
+
|
|
590
|
+
rule(:increment) do |test_case|
|
|
591
|
+
@value += test_case.draw(Zizek::Generators.integers(0..10), name: :amount)
|
|
592
|
+
end
|
|
593
|
+
|
|
594
|
+
rule(:decrement) do |test_case|
|
|
595
|
+
test_case.assume(@value.positive?) # a precondition
|
|
596
|
+
@value -= 1
|
|
597
|
+
end
|
|
598
|
+
|
|
599
|
+
invariant(:nonnegative) do
|
|
600
|
+
raise "negative" if @value.negative?
|
|
601
|
+
end
|
|
602
|
+
end
|
|
603
|
+
|
|
604
|
+
Zizek.check(Counter.property, examples: 500)
|
|
605
|
+
Zizek.check(Counter.property(start: 5), examples: 500) # constructor arguments
|
|
606
|
+
```
|
|
607
|
+
|
|
608
|
+
`Machine.property` builds a fresh machine per example. A machine holds mutable
|
|
609
|
+
state, so the instance cannot be shared between examples — or between shrink
|
|
610
|
+
candidates, which is why `property` returns a lambda rather than taking an
|
|
611
|
+
instance.
|
|
612
|
+
|
|
613
|
+
A subclass inherits its parent's rules and invariants, so a shared base
|
|
614
|
+
machine can declare what every variant has:
|
|
615
|
+
|
|
616
|
+
```ruby
|
|
617
|
+
class LoggingCounter < Counter
|
|
618
|
+
rule(:snapshot) { |_test_case| @log << @value }
|
|
619
|
+
end
|
|
620
|
+
```
|
|
621
|
+
|
|
622
|
+
`run` checks every invariant once before the first rule, then draws a capped
|
|
623
|
+
number of steps (at most 50, or `Counter.property(step_count: 200)`), picks a
|
|
624
|
+
rule per step, and re-checks every invariant after each successful rule. A
|
|
625
|
+
rule that calls `test_case.assume(false)` is recorded as rejected and skipped
|
|
626
|
+
without failing the run — that is what makes `assume` a precondition rather
|
|
627
|
+
than a rejection of the whole example.
|
|
628
|
+
|
|
629
|
+
Each rule invocation is its own structural span, so the shrinker deletes whole
|
|
630
|
+
commands and swaps adjacent ones before it simplifies the arguments inside
|
|
631
|
+
them. A six-step failure minimizes to the shortest sequence that still fails.
|
|
632
|
+
|
|
633
|
+
### The protocol underneath
|
|
634
|
+
|
|
635
|
+
`Stateful.run` asks a machine for `rules` and `invariants`, each an array of
|
|
636
|
+
`Zizek::Stateful::Rule` values whose lambdas receive the machine and the test
|
|
637
|
+
case. `Machine` builds those arrays from the class macros; a machine that
|
|
638
|
+
needs to compute its rules at runtime can define the two methods itself:
|
|
639
|
+
|
|
640
|
+
```ruby
|
|
641
|
+
class Counter
|
|
642
|
+
def rules
|
|
643
|
+
[Zizek::Stateful::Rule.new(
|
|
644
|
+
"increment",
|
|
645
|
+
->(machine, test_case) { machine.increment(test_case) }
|
|
646
|
+
)]
|
|
647
|
+
end
|
|
648
|
+
|
|
649
|
+
def invariants
|
|
650
|
+
[]
|
|
651
|
+
end
|
|
652
|
+
end
|
|
653
|
+
|
|
654
|
+
Zizek.check { |test_case| Zizek::Stateful.run(Counter.new, test_case) }
|
|
655
|
+
```
|
|
656
|
+
|
|
657
|
+
Both forms record the identical choice stream, so a blob from one replays in
|
|
658
|
+
the other — and in every other language package.
|
|
659
|
+
|
|
660
|
+
### Pools
|
|
661
|
+
|
|
662
|
+
`Zizek::Stateful.pool(test_case)` holds values produced by earlier rules
|
|
663
|
+
behind portable choices, so shrinking and replay stay deterministic even
|
|
664
|
+
though the values are arbitrary Ruby objects:
|
|
665
|
+
|
|
666
|
+
```ruby
|
|
667
|
+
accounts = Zizek::Stateful.pool(test_case)
|
|
668
|
+
accounts.add(test_case, "a")
|
|
669
|
+
|
|
670
|
+
account = test_case.draw(accounts.values_reusable) # stays in the pool
|
|
671
|
+
ticket = test_case.draw(accounts.values_consumed) # removed from the pool
|
|
672
|
+
```
|
|
673
|
+
|
|
674
|
+
Drawing from an empty pool rejects the current rule attempt (via `assume`), so
|
|
675
|
+
rules that need prerequisites simply retry on another step. Selection order is
|
|
676
|
+
generated-core semantics: selection by stable identifier.
|
|
677
|
+
|
|
678
|
+
## Using it from a test framework
|
|
679
|
+
|
|
680
|
+
### Minitest
|
|
681
|
+
|
|
682
|
+
```ruby
|
|
683
|
+
require "zizek/minitest"
|
|
684
|
+
|
|
685
|
+
class SplitBillTest < Minitest::Test
|
|
686
|
+
include Zizek::Minitest
|
|
687
|
+
|
|
688
|
+
property "the split is exact",
|
|
689
|
+
total: Zizek::Generators.integers(0..10_000),
|
|
690
|
+
people: Zizek::Generators.integers(1..50) do |total, people|
|
|
691
|
+
assert_equal total, Bill.split(total, people).sum
|
|
692
|
+
end
|
|
693
|
+
|
|
694
|
+
# No generators: the block is handed the test case.
|
|
695
|
+
property "the queue stays consistent", settings: { examples: 2_000 } do |test_case|
|
|
696
|
+
Zizek::Stateful.run(QueueMachine.new, test_case)
|
|
697
|
+
end
|
|
698
|
+
end
|
|
699
|
+
```
|
|
700
|
+
|
|
701
|
+
`property` defines a real test method — `test_the_split_is_exact` — so `rake
|
|
702
|
+
test`, `-n`, seeds, reporters, and `setup`/`teardown` all work unchanged. The
|
|
703
|
+
block runs with `self` bound to the test instance, so ordinary assertions are
|
|
704
|
+
available, and a passing property registers one so it is not
|
|
705
|
+
indistinguishable from a test that forgot to assert.
|
|
706
|
+
|
|
707
|
+
Each generator keyword becomes a named draw, so the failure names the values:
|
|
708
|
+
|
|
709
|
+
```
|
|
710
|
+
1) Failure:
|
|
711
|
+
SplitBillTest#test_the_split_is_exact:
|
|
712
|
+
Falsifying example: (total=1, people=2)
|
|
713
|
+
Minitest::Assertion: Expected: 1
|
|
714
|
+
Actual: 0
|
|
715
|
+
reproduce with: Zizek.check(property, reproduce: "AAIAAAAACgEAAAABAAoBAAAAAg==")
|
|
716
|
+
```
|
|
717
|
+
|
|
718
|
+
A failed assertion is reported as a Minitest **failure**; a raised exception
|
|
719
|
+
stays an **error**, so a bug in the code under test is not relabelled as a
|
|
720
|
+
failed assertion. `settings:` is reserved and cannot be a generator name.
|
|
721
|
+
|
|
722
|
+
Two properties in one class cannot share a description: the second would
|
|
723
|
+
silently overwrite the first's method, so it raises instead.
|
|
724
|
+
|
|
725
|
+
### RSpec
|
|
726
|
+
|
|
727
|
+
```ruby
|
|
728
|
+
require "zizek/rspec"
|
|
729
|
+
|
|
730
|
+
RSpec.describe Bill do
|
|
731
|
+
property "the split is exact",
|
|
732
|
+
total: Zizek::Generators.integers(0..10_000),
|
|
733
|
+
people: Zizek::Generators.integers(1..50) do |total, people|
|
|
734
|
+
expect(described_class.split(total, people).sum).to eq(total)
|
|
735
|
+
end
|
|
736
|
+
end
|
|
737
|
+
```
|
|
738
|
+
|
|
739
|
+
`property` defines a real example, so filters, tags, formatters, `let`, and
|
|
740
|
+
`before` hooks work unchanged, and the block runs in the example's own context.
|
|
741
|
+
A failed expectation is re-raised as `ExpectationNotMetError` carrying the
|
|
742
|
+
falsifying example and the blob, so RSpec renders it as a failed expectation
|
|
743
|
+
rather than an unexpected exception.
|
|
744
|
+
|
|
745
|
+
Requiring the file registers the DSL on every example group. To scope it,
|
|
746
|
+
skip the require and add `config.extend Zizek::RSpec::DSL, type: :prop`.
|
|
747
|
+
|
|
748
|
+
RSpec is an optional development dependency: `lib/zizek/rspec.rb` is loaded
|
|
749
|
+
only if you require it, and the gem depends on neither framework at runtime.
|
|
750
|
+
|
|
751
|
+
## Runnable examples
|
|
752
|
+
|
|
753
|
+
Run from the package root:
|
|
754
|
+
|
|
755
|
+
| Example | Features | Run |
|
|
756
|
+
|---------|----------|-----|
|
|
757
|
+
| [`examples/roundtrip.rb`](examples/roundtrip.rb) | text generator, notes, minimal shrinking, blob replay | `ruby -Ilib examples/roundtrip.rb` |
|
|
758
|
+
| [`examples/tour.rb`](examples/tour.rb) | shrink targets, assumptions, boolean+bytes draws, rejection limits | `ruby -Ilib examples/tour.rb` |
|
|
759
|
+
| [`examples/queue.rb`](examples/queue.rb) | state machine, rule assumptions, invariants, failure capture, span shrinking | `ruby -Ilib examples/queue.rb` |
|
|
760
|
+
| [`examples/ledger.rb`](examples/ledger.rb) | full stateful slice: pools, text, cross-language blob fixture | `ruby -Ilib examples/ledger.rb` |
|
|
761
|
+
| [`examples/coverage_assertions.rb`](examples/coverage_assertions.rb) | `always`, `unreachable`, `sometimes`, `cover(percent:)`, `must_cover:` | `ruby -Ilib examples/coverage_assertions.rb` |
|
|
762
|
+
| [`examples/keyed_draws.rb`](examples/keyed_draws.rb) | keyed vs positional replay: the blob that survives editing the property | `ruby -Ilib examples/keyed_draws.rb` |
|
|
763
|
+
| [`examples/database_workflow.rb`](examples/database_workflow.rb) | the example database: store, replay-first, self-clean | `ruby -Ilib examples/database_workflow.rb` |
|
|
764
|
+
| [`examples/combinators.rb`](examples/combinators.rb) | `builds`, `sampled_from`, `map`, `filter`, `\|`, `composite`, shrinking through composition | `ruby -Ilib examples/combinators.rb` |
|
|
765
|
+
| [`examples/minitest_suite.rb`](examples/minitest_suite.rb) | the Minitest integration: named draws, per-property settings, the failure report | `ruby -Ilib examples/minitest_suite.rb` |
|
|
766
|
+
|
|
767
|
+
Each example is deterministic, asserts its own expected outcome in
|
|
768
|
+
`test/test_zizek.rb`, and prints the shrunk failure with its reproduce blob. The Go
|
|
769
|
+
and Python packages ship the same first four, shrinking to identical blobs.
|
|
770
|
+
|
|
771
|
+
`examples/minitest_suite.rb` is the exception: it is a Minitest suite, so it
|
|
772
|
+
exits non-zero, and one of its three properties fails on purpose. That failure
|
|
773
|
+
is the demonstration — the gate asserts the example still fails, because an
|
|
774
|
+
example that stopped failing would have stopped showing anything.
|