hegeltest 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +103 -0
- data/README.md +72 -26
- data/Rakefile +81 -0
- data/docs/README.md +10 -0
- data/docs/adr/0014-name-a-drawn-value-only-when-the-draw-is-the-whole-assigned-value.md +74 -0
- data/docs/adr/0015-follow-the-hegeldev-interface-and-take-changes-upstream-first.md +101 -0
- data/docs/adr/0016-run-mutation-testing-with-mutineer.md +87 -0
- data/docs/adr/0017-raise-the-ruby-floor-to-3-4.md +43 -0
- data/docs/adr/0018-gate-mutation-testing-on-a-committed-baseline.md +74 -0
- data/docs/adr/0019-report-failures-from-the-cases-the-engine-stamps.md +70 -0
- data/docs/adr/0020-derive-span-labels-from-generator-names.md +64 -0
- data/docs/adr/0021-run-a-state-machine-in-rounds-with-its-own-step-count.md +59 -0
- data/docs/adr/0022-keep-microsecond-times-over-a-nanosecond-engine.md +45 -0
- data/docs/adr/0023-leave-unset-settings-to-the-engines-profile.md +50 -0
- data/docs/architecture.md +11 -6
- data/lib/hegel/draw_name.rb +24 -8
- data/lib/hegel/generator.rb +50 -9
- data/lib/hegel/generators.rb +134 -102
- data/lib/hegel/lib_hegel/real.rb +193 -94
- data/lib/hegel/lib_hegel.rb +30 -56
- data/lib/hegel/libhegel_version.rb +1 -1
- data/lib/hegel/report.rb +17 -8
- data/lib/hegel/runner.rb +153 -201
- data/lib/hegel/settings.rb +9 -14
- data/lib/hegel/state_machine.rb +15 -8
- data/lib/hegel/stateful/pool.rb +0 -2
- data/lib/hegel/stateful.rb +64 -36
- data/lib/hegel/syntax/methods.rb +3 -2
- data/lib/hegel/test_case.rb +29 -9
- data/lib/hegel/version.rb +1 -1
- data/lib/hegel.rb +11 -17
- data/lib/tasks/libhegel.rake +9 -2
- data/sig/hegel.rbs +60 -61
- data/skills/hegel-ruby/references/ruby/reference.md +147 -52
- metadata +12 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: fc58a6595c70e7dd1e873f76049109ceea93f5a7304ff4a6b4d2287072d0a0e6
|
|
4
|
+
data.tar.gz: 42099dfad36fa60a4a15b17d4713062a5224b4926a2ea15f56e84758f56e93f8
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: eb7cdb67a056b0db5e3803f915bb8b09001ac2ea5918ec1fd3763345f295e56140075aa02260a8efcc3bf3f6d6009b352066ab116a23797f02b42691cc93b018
|
|
7
|
+
data.tar.gz: 06bd1f1bfae6a4918adff338816c0d2f35cfca94fb063017ebcfdbd34c27581fe60b7006d7244fc12266e51a19e704a68053a3e568952aa5e28133703cae871f
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,106 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [Unreleased]
|
|
4
|
+
|
|
5
|
+
## [0.2.0] - 2026-10-07
|
|
6
|
+
|
|
7
|
+
The floor is Ruby 3.4. A project on Ruby 3.3 stays on 0.1.1. See
|
|
8
|
+
[ADR 0017](docs/adr/0017-raise-the-ruby-floor-to-3-4.md).
|
|
9
|
+
|
|
10
|
+
Each platform gem carries `libhegel` 0.45.0, up from 0.32.5. The engine
|
|
11
|
+
changed its C interface in several places between the two, and these
|
|
12
|
+
bindings follow each change.
|
|
13
|
+
|
|
14
|
+
### Changes a caller has to make
|
|
15
|
+
|
|
16
|
+
`Hegel.test` no longer takes `stateful_step_count:`. Pass `step_count:` to
|
|
17
|
+
`Hegel::Stateful.run` instead; it defaults to 50, as hegel-rust, hegel-go,
|
|
18
|
+
and hegel-java do. The engine no longer has a default of its own.
|
|
19
|
+
|
|
20
|
+
An invariant no longer runs after every rule. It runs before the first
|
|
21
|
+
rule and after the last, and between rules when the engine samples it,
|
|
22
|
+
about once in a test case that runs every step. Declare it with
|
|
23
|
+
`invariant :name, always_run: true` to keep the check after every rule.
|
|
24
|
+
See [ADR 0021](docs/adr/0021-run-a-state-machine-in-rounds-with-its-own-step-count.md).
|
|
25
|
+
|
|
26
|
+
### Failure reports
|
|
27
|
+
|
|
28
|
+
The report comes from the engine's own final replay of each failure, which
|
|
29
|
+
it runs inside the loop before the run ends. The body is no longer run
|
|
30
|
+
again after the run, so a failing run calls it once fewer per failure. A
|
|
31
|
+
`tc.note` block runs on each case the engine stamps for capture, which can
|
|
32
|
+
be more than one per failure. See
|
|
33
|
+
[ADR 0019](docs/adr/0019-report-failures-from-the-cases-the-engine-stamps.md).
|
|
34
|
+
|
|
35
|
+
A body that fails only some of the time is now reported as an ordinary
|
|
36
|
+
failure, with a `note:` line from the engine that says how often it
|
|
37
|
+
reproduced. Before, such a body ended the run in `Hegel::Error`. A failure the engine could not confirm comes
|
|
38
|
+
without a blob, and its report leaves out the reproduction line.
|
|
39
|
+
|
|
40
|
+
`reproduce_failure:` now replays its blob as a run, until a replay fails,
|
|
41
|
+
within the engine's own budget. A blob from a nondeterministic failure
|
|
42
|
+
reproduces this way too. When no replay fails, `Hegel.test` raises
|
|
43
|
+
`Hegel::Error` and names both causes: a fixed bug, or a body whose failure
|
|
44
|
+
did not recur.
|
|
45
|
+
|
|
46
|
+
A body that rejects its input with `tc.assume` or `tc.reject` before it
|
|
47
|
+
draws anything now raises `Hegel::Error` ("Unsatisfiable"). Before, the
|
|
48
|
+
run stopped after that one case and passed.
|
|
49
|
+
|
|
50
|
+
### Settings
|
|
51
|
+
|
|
52
|
+
A keyword left `nil` now takes its value from the engine's settings
|
|
53
|
+
profile: libhegel's default, a `hegel.toml`, or a `HEGEL_*` environment
|
|
54
|
+
variable such as `HEGEL_TEST_CASES`. On a CI server the engine picks its
|
|
55
|
+
`ci` profile, which derandomizes the run and turns the example database
|
|
56
|
+
off, so a run with `database_key:` and no `database:` stores nothing there.
|
|
57
|
+
A keyword you pass still wins. See
|
|
58
|
+
[ADR 0023](docs/adr/0023-leave-unset-settings-to-the-engines-profile.md).
|
|
59
|
+
|
|
60
|
+
### Generated values
|
|
61
|
+
|
|
62
|
+
The engine draws integers and floats from new distributions, and shrinks
|
|
63
|
+
further in several cases, so a seeded run draws different values
|
|
64
|
+
than it did on 0.32.5. `from_regex` follows Python's `re` more closely
|
|
65
|
+
under `(?i)` and `(?a)`, and its pattern may now hold a NUL character.
|
|
66
|
+
|
|
67
|
+
`times` and `datetimes` keep microsecond precision. The engine now draws
|
|
68
|
+
nanoseconds, and the binding rounds each one down. See
|
|
69
|
+
[ADR 0022](docs/adr/0022-keep-microsecond-times-over-a-nanosecond-engine.md).
|
|
70
|
+
|
|
71
|
+
A `datetimes` bound with a fraction of a microsecond could produce a value
|
|
72
|
+
outside it, because only the bound's whole microseconds reached the
|
|
73
|
+
engine. The lower bound now rounds up and the upper rounds down.
|
|
74
|
+
|
|
75
|
+
### Other changes
|
|
76
|
+
|
|
77
|
+
`HEGEL_LIBHEGEL_PATH` pointing at an engine that lacks a function these
|
|
78
|
+
bindings call raises `Hegel::Error` that names the function. Before, FFI
|
|
79
|
+
raised a `TypeError` that named nothing.
|
|
80
|
+
|
|
81
|
+
`rake libhegel:fetch` downloads from hegel-rust's `libhegel-v<version>`
|
|
82
|
+
release tags, which hegel-rust uses for engine releases since 0.42.1.
|
|
83
|
+
|
|
84
|
+
## [0.1.1] - 2026-08-31
|
|
85
|
+
|
|
86
|
+
The reference that ships inside the gem told a reader to install `hegeltest`
|
|
87
|
+
from git and to point `HEGEL_LIBHEGEL_PATH` at a local build. 0.1.0 shipped
|
|
88
|
+
that text, and 0.1.0 needs neither. A reader who followed it spent their first
|
|
89
|
+
minutes on a checkout and an engine build. The reference now says what the
|
|
90
|
+
README says.
|
|
91
|
+
|
|
92
|
+
A drawn value written inside a larger expression named itself after the
|
|
93
|
+
assignment target, so `term_start_date = Date.new(tc.draw(...), 2, 29)`
|
|
94
|
+
reported `term_start_date` beside a year. Such a draw now takes the generic
|
|
95
|
+
name, and `label:` names it when a name is wanted. See
|
|
96
|
+
[ADR 0014](docs/adr/0014-name-a-drawn-value-only-when-the-draw-is-the-whole-assigned-value.md).
|
|
97
|
+
|
|
98
|
+
The settings table names 100 as libhegel's own default for `test_cases`. The
|
|
99
|
+
README shows how to scope `Hegel::Syntax::Methods` to tagged example groups in
|
|
100
|
+
a suite that already exists, and how to share a group of draws through a plain
|
|
101
|
+
Ruby method. The `text` section says which generators cover the character-set
|
|
102
|
+
options it does not take, measured against libhegel 0.32.5.
|
|
103
|
+
|
|
3
104
|
## [0.1.0] - 2026-08-20
|
|
4
105
|
|
|
5
106
|
The first release.
|
|
@@ -19,4 +120,6 @@ The engine is called through the `ffi` gem. Each platform gem carries the
|
|
|
19
120
|
matching `libhegel` 0.32.5 build. The platform-independent gem carries none,
|
|
20
121
|
so a run on it needs `HEGEL_LIBHEGEL_PATH` pointing at a local build.
|
|
21
122
|
|
|
123
|
+
[0.2.0]: https://github.com/meganemura/hegel-ruby/releases/tag/v0.2.0
|
|
124
|
+
[0.1.1]: https://github.com/meganemura/hegel-ruby/releases/tag/v0.1.1
|
|
22
125
|
[0.1.0]: https://github.com/meganemura/hegel-ruby/releases/tag/v0.1.0
|
data/README.md
CHANGED
|
@@ -31,8 +31,7 @@ same process. There is no server and no Python dependency.
|
|
|
31
31
|
|
|
32
32
|
## Status
|
|
33
33
|
|
|
34
|
-
**
|
|
35
|
-
described below runs: Hegel finds a counterexample, shrinks it,
|
|
34
|
+
**Released as `hegeltest` 0.1.0.** Hegel finds a counterexample, shrinks it,
|
|
36
35
|
names the values it drew, and re-raises your own exception.
|
|
37
36
|
|
|
38
37
|
Work ran in three stages, and all three are done:
|
|
@@ -44,26 +43,34 @@ Work ran in three stages, and all three are done:
|
|
|
44
43
|
3. **The advanced features**: the example database, targeted testing,
|
|
45
44
|
stateful testing, phases, and health checks.
|
|
46
45
|
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
46
|
+
Install it the way you install any gem:
|
|
47
|
+
|
|
48
|
+
```ruby
|
|
49
|
+
# Gemfile
|
|
50
|
+
gem "hegeltest"
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Each platform gem carries the matching `libhegel` build, so an install
|
|
54
|
+
compiles nothing and needs no engine on the side. The Hegel maintainers
|
|
55
|
+
permit a third-party project to redistribute their release binaries
|
|
53
56
|
([hegeldev/hegel-rust#411]).
|
|
54
57
|
|
|
55
|
-
|
|
58
|
+
A project that already has a `Gemfile.lock` needs one more step. Bundler keeps
|
|
59
|
+
the platform list the lock already holds. It adds only the platform of the
|
|
60
|
+
machine that runs `bundle install`. A lock therefore names only the platforms
|
|
61
|
+
of the machines that have run it. A Linux CI job cannot find the gem when the
|
|
62
|
+
lock names no Linux platform. Add each platform your CI uses:
|
|
56
63
|
|
|
57
64
|
```bash
|
|
58
|
-
|
|
59
|
-
bundle exec rake libhegel:fetch # downloads the pinned build, checked against its SHA-256
|
|
60
|
-
bundle exec rake
|
|
65
|
+
bundle lock --add-platform x86_64-linux
|
|
61
66
|
```
|
|
62
67
|
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
68
|
+
A project with no lock yet reads the `PLATFORMS` section of the lock its first
|
|
69
|
+
`bundle install` writes, and adds whatever its CI needs the same way.
|
|
70
|
+
|
|
71
|
+
Every push to main runs the suite on Ruby 3.4 and 4.0, across Linux x64 and
|
|
72
|
+
arm64, macOS arm64, and Windows x64 and arm64. A pull request runs Linux x64
|
|
73
|
+
alone.
|
|
67
74
|
|
|
68
75
|
[hegeldev/hegel-rust#411]: https://github.com/hegeldev/hegel-rust/issues/411
|
|
69
76
|
|
|
@@ -75,17 +82,17 @@ pull request runs Linux x64 alone.
|
|
|
75
82
|
| Require path | `require "hegel"` (`require "hegeltest"` also works) |
|
|
76
83
|
| Namespace | `Hegel` |
|
|
77
84
|
| Binding | the `ffi` gem, which publishes a prebuilt binary for every platform above |
|
|
78
|
-
| Ruby | 3.
|
|
85
|
+
| Ruby | 3.4 and 4.0 |
|
|
79
86
|
| Platforms | Linux amd64/arm64, macOS arm64, Windows amd64/arm64 |
|
|
80
87
|
| Engine delivery | One prebuilt `libhegel` per platform-specific gem, built by `rake platform_gems:build` |
|
|
81
88
|
|
|
82
|
-
`HEGEL_LIBHEGEL_PATH`
|
|
89
|
+
`HEGEL_LIBHEGEL_PATH` overrides the bundled engine with a local build.
|
|
83
90
|
|
|
84
91
|
The gem name follows the Rust implementation, whose published crate is
|
|
85
92
|
`hegeltest` and whose library is `hegel`. The name `hegel` on RubyGems.org
|
|
86
93
|
stays free for whoever publishes an official Ruby implementation.
|
|
87
94
|
|
|
88
|
-
macOS on Intel has no published `libhegel` artifact, so that platform
|
|
95
|
+
macOS on Intel has no published `libhegel` artifact, so that platform needs
|
|
89
96
|
`HEGEL_LIBHEGEL_PATH` and a local build.
|
|
90
97
|
|
|
91
98
|
## Quickstart
|
|
@@ -135,6 +142,33 @@ Including `Hegel::Syntax::Methods` makes the generators available without a
|
|
|
135
142
|
prefix, the way FactoryBot makes `create` available. The same generators stay
|
|
136
143
|
reachable as `Hegel::Generators.arrays(...)` without the include.
|
|
137
144
|
|
|
145
|
+
An existing RSpec suite can limit the 25 generator names to tagged example
|
|
146
|
+
groups. This keeps the names out of other groups that have their own helpers:
|
|
147
|
+
|
|
148
|
+
```ruby
|
|
149
|
+
config.include Hegel::Syntax::Methods, :hegel
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
A new project can include the methods in every group, as the Quickstart does.
|
|
153
|
+
|
|
154
|
+
Hegel draws values imperatively, so an ordinary Ruby method can share a group
|
|
155
|
+
of draws between properties. Give the method parameter a descriptive name,
|
|
156
|
+
which the default RuboCop naming rule accepts where a block parameter `tc`
|
|
157
|
+
needs no change:
|
|
158
|
+
|
|
159
|
+
```ruby
|
|
160
|
+
def draw_sorted_pair(test_case)
|
|
161
|
+
low = test_case.draw(integers(min_value: 0, max_value: 100))
|
|
162
|
+
high = test_case.draw(integers(min_value: low, max_value: 200))
|
|
163
|
+
[low, high]
|
|
164
|
+
end
|
|
165
|
+
|
|
166
|
+
Hegel.test do |tc|
|
|
167
|
+
low, high = draw_sorted_pair(tc)
|
|
168
|
+
expect(low).to be <= high
|
|
169
|
+
end
|
|
170
|
+
```
|
|
171
|
+
|
|
138
172
|
The generators are `arrays`, `binary`, `booleans`, `characters`, `composite`,
|
|
139
173
|
`dates`, `datetimes`, `deferred`, `domains`, `emails`, `floats`, `from_regex`,
|
|
140
174
|
`hashes`, `integers`, `ip_addresses`, `just`, `one_of`, `optional`,
|
|
@@ -181,7 +215,7 @@ Expected: [0, 0]
|
|
|
181
215
|
|
|
182
216
|
Some bugs need a sequence of operations rather than one input. Declare the
|
|
183
217
|
operations as rules on a `Hegel::StateMachine`, and Hegel picks which runs
|
|
184
|
-
next, checks your invariants
|
|
218
|
+
next, checks your invariants along the way, and shrinks a failure to the
|
|
185
219
|
shortest sequence that still shows it:
|
|
186
220
|
|
|
187
221
|
```ruby
|
|
@@ -216,12 +250,23 @@ something an earlier rule produced, such as freeing a handle that some
|
|
|
216
250
|
`alloc` actually returned, put the value in a `Hegel::Stateful::Pool` and
|
|
217
251
|
draw it back out.
|
|
218
252
|
|
|
253
|
+
Every invariant runs before the first rule and after the last. Between rules,
|
|
254
|
+
the engine samples each one, so that it runs about once in a test case
|
|
255
|
+
that runs every step.
|
|
256
|
+
Declare an invariant with `always_run: true` to check it after every rule
|
|
257
|
+
instead. `Hegel::Stateful.run(machine, tc, step_count: 50)` sets the most
|
|
258
|
+
rules one test case runs; 50 is the default.
|
|
259
|
+
|
|
219
260
|
### Shaping a run
|
|
220
261
|
|
|
221
262
|
`Hegel.test` takes keywords for the rest: `test_cases`, `seed`,
|
|
222
|
-
`derandomize`, `verbosity`, `phases`, `suppress_health_check`,
|
|
223
|
-
`report_multiple_failures
|
|
224
|
-
|
|
263
|
+
`derandomize`, `verbosity`, `phases`, `suppress_health_check`, and
|
|
264
|
+
`report_multiple_failures`. `report_multiple_failures` defaults to `false`.
|
|
265
|
+
Each of the others left unset takes its value from the engine's settings
|
|
266
|
+
profile. That is libhegel's own default, unless a
|
|
267
|
+
`hegel.toml` or a `HEGEL_*` environment variable says otherwise. On a CI
|
|
268
|
+
server the engine picks its `ci` profile, which derandomizes the run and
|
|
269
|
+
turns the example database off.
|
|
225
270
|
|
|
226
271
|
Two more turn on libhegel's example database, which stores a failing case and
|
|
227
272
|
replays it first next time. `database_key` is the switch and `database`
|
|
@@ -241,9 +286,10 @@ Inside a test, `tc.note` records a message the failure report prints, and
|
|
|
241
286
|
## Development
|
|
242
287
|
|
|
243
288
|
```bash
|
|
244
|
-
bin/setup
|
|
245
|
-
bundle exec rake
|
|
246
|
-
|
|
289
|
+
bin/setup # install dependencies
|
|
290
|
+
bundle exec rake libhegel:fetch # download the pinned engine, checked against its SHA-256
|
|
291
|
+
bundle exec rake # run the tests and the linter
|
|
292
|
+
bin/console # open an interactive prompt
|
|
247
293
|
```
|
|
248
294
|
|
|
249
295
|
`just` recipes call the same Rake tasks, so `just test` and `just lint` work
|
data/Rakefile
CHANGED
|
@@ -17,3 +17,84 @@ task :coverage do
|
|
|
17
17
|
end
|
|
18
18
|
|
|
19
19
|
task default: %i[coverage standard]
|
|
20
|
+
|
|
21
|
+
# Mutation testing with mutineer, over lib/. Four settings make the suite
|
|
22
|
+
# run under it; ADR 0016 records why:
|
|
23
|
+
# - RUBYOPT gets -Itest, since mutineer puts only lib/ on the load path and
|
|
24
|
+
# every test file starts with `require "test_helper"`.
|
|
25
|
+
# - Each test file gets its own --test flag. The flag takes one file, and a
|
|
26
|
+
# second path after it becomes a source to mutate.
|
|
27
|
+
# - test_conformance.rb stays out. Its capture_subprocess_io reopens
|
|
28
|
+
# $stdout, and mutineer replaces $stdout with a StringIO, so the file fails
|
|
29
|
+
# before any mutant runs.
|
|
30
|
+
# - --strategy redefine reloads only the mutated method. The default,
|
|
31
|
+
# reload, loads the whole mutated file by a relative path, which breaks
|
|
32
|
+
# Hegel::Runner.origin_for and killed three mutants the suite does not
|
|
33
|
+
# catch.
|
|
34
|
+
# MUTINEER_ARGS passes more flags, for example
|
|
35
|
+
# MUTINEER_ARGS="--since origin/main" to mutate only the lines changed since
|
|
36
|
+
# origin/main.
|
|
37
|
+
#
|
|
38
|
+
# `rake mutation` fails on a survivor that the baseline does not list
|
|
39
|
+
# (ADR 0018). `rake mutation:baseline` rewrites the baseline from a full run.
|
|
40
|
+
MUTATION_BASELINE = "test/mutation-baseline.json"
|
|
41
|
+
|
|
42
|
+
def run_mutineer(*extra)
|
|
43
|
+
require "shellwords"
|
|
44
|
+
tests = Dir["test/*.rb", "test/hegel/*.rb"].sort -
|
|
45
|
+
%w[test/test_helper.rb test/hegel/test_conformance.rb]
|
|
46
|
+
args = ["lib", "--strategy", "redefine", *tests.flat_map { |t| ["--test", t] }, *extra, *Shellwords.split(ENV.fetch("MUTINEER_ARGS", ""))]
|
|
47
|
+
rubyopt = ["-Itest", ENV["RUBYOPT"]].compact.join(" ")
|
|
48
|
+
# mutineer keeps its cache here, and refuses an --output path whose
|
|
49
|
+
# directory does not exist yet.
|
|
50
|
+
mkdir_p ".mutineer", verbose: false
|
|
51
|
+
sh({"RUBYOPT" => rubyopt}, "bundle", "exec", "mutineer", "run", *args)
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
# mutineer writes its JSON report on one line, so a change to it reads as one
|
|
55
|
+
# changed line in a pull request. mutineer's --baseline reads only
|
|
56
|
+
# schema_version, summary.score, summary.scoped, and each survivor's id, so
|
|
57
|
+
# the committed file keeps those and writes one survivor per line, sorted.
|
|
58
|
+
# It leaves out line numbers: an edit above a survivor would move every
|
|
59
|
+
# line after it, and the id does not depend on the position. "change" holds
|
|
60
|
+
# the mutated line before and after, so a reviewer can read the survivor.
|
|
61
|
+
def write_mutation_baseline(report_path, baseline_path)
|
|
62
|
+
require "json"
|
|
63
|
+
report = JSON.parse(File.read(report_path))
|
|
64
|
+
# mutineer refuses a --since run as a baseline, since it covers only the
|
|
65
|
+
# changed lines.
|
|
66
|
+
abort "#{report_path} comes from a --since run; rewrite the baseline from a full run" if report.dig("summary", "scoped")
|
|
67
|
+
rows = report.fetch("survivors").map do |survivor|
|
|
68
|
+
change = survivor.fetch("diff").lines.grep(/^[-+] /).map { |line| line.chomp.sub(/^([-+]) +/, '\1 ') }.join(" | ")
|
|
69
|
+
{"id" => survivor.fetch("id"), "file" => survivor.fetch("file"), "subject" => survivor.fetch("subject"),
|
|
70
|
+
"operator" => survivor.fetch("operator"), "change" => change}
|
|
71
|
+
end
|
|
72
|
+
rows.sort_by! { |row| row.values_at("file", "subject", "id") }
|
|
73
|
+
text = +"{\n"
|
|
74
|
+
text << %( "schema_version": #{report.fetch("schema_version").to_json},\n)
|
|
75
|
+
text << %( "summary": #{report.fetch("summary").slice("score", "scoped").to_json},\n)
|
|
76
|
+
text << %( "survivors": [\n#{rows.map { |row| " #{row.to_json}" }.join(",\n")}\n ]\n}\n)
|
|
77
|
+
# The baseline is committed to a public repository, so a path from this
|
|
78
|
+
# machine must not reach it.
|
|
79
|
+
leak = [Dir.pwd, Dir.home].find { |path| text.include?(path) }
|
|
80
|
+
abort "#{baseline_path} would contain #{leak}; not written" if leak
|
|
81
|
+
File.write(baseline_path, text)
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
desc "Run mutation testing over lib/, and fail on a survivor that #{MUTATION_BASELINE} does not list"
|
|
85
|
+
task :mutation do
|
|
86
|
+
# --baseline also fails on a lower score. A mutant that times out on a slow
|
|
87
|
+
# machine leaves the score and can lower it by a fraction of a point, and
|
|
88
|
+
# it can never add a survivor. The epsilon absorbs that, and the survivor
|
|
89
|
+
# ids stay the gate.
|
|
90
|
+
run_mutineer("--baseline", MUTATION_BASELINE, "--baseline-epsilon", "1")
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
namespace :mutation do
|
|
94
|
+
desc "Rewrite #{MUTATION_BASELINE} from a full mutation testing run over lib/"
|
|
95
|
+
task :baseline do
|
|
96
|
+
report = ".mutineer/baseline-run.json"
|
|
97
|
+
run_mutineer("--format", "json", "--output", report)
|
|
98
|
+
write_mutation_baseline(report, MUTATION_BASELINE)
|
|
99
|
+
end
|
|
100
|
+
end
|
data/docs/README.md
CHANGED
|
@@ -20,6 +20,16 @@ An index of the design records for `hegel-ruby`.
|
|
|
20
20
|
- [0011: Let the test case own every pool drawn from it](adr/0011-let-the-test-case-own-every-pool-drawn-from-it.md)
|
|
21
21
|
- [0012: Build a failure origin from the caller's own frame](adr/0012-build-a-failure-origin-from-the-callers-own-frame.md)
|
|
22
22
|
- [0013: Bind libhegel through the ffi gem](adr/0013-bind-libhegel-through-the-ffi-gem.md)
|
|
23
|
+
- [0014: Name a drawn value only when the draw is the whole assigned value](adr/0014-name-a-drawn-value-only-when-the-draw-is-the-whole-assigned-value.md)
|
|
24
|
+
- [0015: Follow the hegeldev interface, and take changes upstream first](adr/0015-follow-the-hegeldev-interface-and-take-changes-upstream-first.md)
|
|
25
|
+
- [0016: Run mutation testing with mutineer](adr/0016-run-mutation-testing-with-mutineer.md)
|
|
26
|
+
- [0017: Raise the Ruby floor to 3.4](adr/0017-raise-the-ruby-floor-to-3-4.md)
|
|
27
|
+
- [0018: Gate mutation testing on a committed baseline](adr/0018-gate-mutation-testing-on-a-committed-baseline.md)
|
|
28
|
+
- [0019: Report failures from the cases the engine stamps](adr/0019-report-failures-from-the-cases-the-engine-stamps.md)
|
|
29
|
+
- [0020: Derive span labels from generator names](adr/0020-derive-span-labels-from-generator-names.md)
|
|
30
|
+
- [0021: Run a state machine in rounds, with its own step count](adr/0021-run-a-state-machine-in-rounds-with-its-own-step-count.md)
|
|
31
|
+
- [0022: Keep microsecond times over a nanosecond engine](adr/0022-keep-microsecond-times-over-a-nanosecond-engine.md)
|
|
32
|
+
- [0023: Leave unset settings to the engine's profile](adr/0023-leave-unset-settings-to-the-engines-profile.md)
|
|
23
33
|
|
|
24
34
|
A new decision gets a new record. A changed decision supersedes the old
|
|
25
35
|
record instead of editing it, so the history stays readable.
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
# 0014: Name a drawn value only when the draw is the whole assigned value
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted.
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
|
|
9
|
+
[ADR 0005](0005-name-drawn-values-from-the-callers-source-with-prism.md)
|
|
10
|
+
recovers a drawn value's name by parsing the caller's source and reading the
|
|
11
|
+
assignment that covers the draw's line. Its Consequences section lists three
|
|
12
|
+
cases it had not checked: a heredoc, several draws on one line, and a method
|
|
13
|
+
chain.
|
|
14
|
+
|
|
15
|
+
A first use of the gem outside this repository reached one of them. The
|
|
16
|
+
caller wrote a draw inside a larger expression:
|
|
17
|
+
|
|
18
|
+
```ruby
|
|
19
|
+
term_start_date = Date.new(tc.draw(integers(min_value: 1904, max_value: 1904)), 2, 29)
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Measured against libhegel 0.32.5, the failure report printed:
|
|
23
|
+
|
|
24
|
+
```
|
|
25
|
+
term_start_date = 1904
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
The name says "a date". The value is a year. The reader misread that report
|
|
29
|
+
twice before adding a `label:`.
|
|
30
|
+
|
|
31
|
+
`Hegel::DrawName` already carries the rule this breaks. Its own
|
|
32
|
+
documentation states it: a wrong name misdirects a reader of the failure
|
|
33
|
+
report more than a missing one does, so an ambiguous line returns nil rather
|
|
34
|
+
than choosing between candidates. The module applied that rule to two
|
|
35
|
+
assignments written side by side, and applied the opposite rule to a draw
|
|
36
|
+
buried in one assignment's value.
|
|
37
|
+
|
|
38
|
+
A draw pulled out into a plain method is a separate case, and it already
|
|
39
|
+
worked. Measured on the same engine, a helper whose body reads
|
|
40
|
+
`year = tc.draw(integers(min_value: 1900, max_value: 1910))` reported
|
|
41
|
+
`year = 1900`. `Hegel::TestCase::DRAW_CALLER_DEPTH` lands on the line that
|
|
42
|
+
called the draw, which is the helper's own line, and the name there is the
|
|
43
|
+
one the caller wrote.
|
|
44
|
+
|
|
45
|
+
## Decision
|
|
46
|
+
|
|
47
|
+
Recover a name only when the assignment's value is the draw call itself: the
|
|
48
|
+
value node is a `Prism::CallNode` whose name is one of the methods that reach
|
|
49
|
+
`Hegel::TestCase#record_draw`. Any other value node returns nil, and
|
|
50
|
+
`Hegel::TestCase#name_for` falls back to the generic name.
|
|
51
|
+
|
|
52
|
+
The three method names stay in `Hegel::TestCase`, which defines them, and
|
|
53
|
+
`Hegel::DrawName.for` takes them as an argument. A fourth draw method then
|
|
54
|
+
has one place to be added.
|
|
55
|
+
|
|
56
|
+
## Consequences
|
|
57
|
+
|
|
58
|
+
A draw written inside a larger expression reports the generic name and its
|
|
59
|
+
value. The reader loses a name and keeps a value that agrees with it. A
|
|
60
|
+
caller who wants the name back passes `label:`, which
|
|
61
|
+
[ADR 0005](0005-name-drawn-values-from-the-callers-source-with-prism.md)
|
|
62
|
+
already gives precedence over the recovered name.
|
|
63
|
+
|
|
64
|
+
A method chain answers the question ADR 0005 left open, and answers it the
|
|
65
|
+
same way: `xs = tc.draw_integer(0, 10).to_s` reports the generic name,
|
|
66
|
+
because `xs` holds a String and the drawn value is an Integer.
|
|
67
|
+
|
|
68
|
+
A draw written inside another draw's arguments shares the enclosing
|
|
69
|
+
assignment's name, because the enclosing assignment's value is a draw call
|
|
70
|
+
and one line carries both draws. `label:` separates them.
|
|
71
|
+
|
|
72
|
+
`Hegel::DrawName` now answers a narrower question than its name suggests on
|
|
73
|
+
its own, so its documentation states the question it answers and the reason
|
|
74
|
+
for the narrowing.
|
|
@@ -0,0 +1,101 @@
|
|
|
1
|
+
# 0015: Follow the hegeldev interface, and take changes upstream first
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted.
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
|
|
9
|
+
This gem is one binding of libhegel among several. The Hegel project
|
|
10
|
+
publishes implementations for Rust, Go, C++, TypeScript, Java, and OCaml,
|
|
11
|
+
and each one drives the same engine. A person who writes a property in one
|
|
12
|
+
language and then writes the same property in another expects the same call
|
|
13
|
+
to mean the same thing.
|
|
14
|
+
|
|
15
|
+
`CLAUDE.md` already names hegel-rust as the authority on meaning: what a
|
|
16
|
+
call does, what an error code means, how a value shrinks. That rule settles
|
|
17
|
+
disputes about behaviour. It does not say what to do when this gem's public
|
|
18
|
+
surface is shaped differently from every other binding while each shape
|
|
19
|
+
behaves as documented.
|
|
20
|
+
|
|
21
|
+
`floats` is that case. Three implementations write the same rule for its
|
|
22
|
+
defaults:
|
|
23
|
+
|
|
24
|
+
- hegel-rust, `src/generators/numeric.rs:315`:
|
|
25
|
+
`allow_nan.unwrap_or(!has_min && !has_max)`, and
|
|
26
|
+
`allow_infinity.unwrap_or(!has_min || !has_max)`
|
|
27
|
+
- hegel-go, `primitives.go:136`: the same two expressions, resolved through
|
|
28
|
+
a nil pointer check
|
|
29
|
+
- hegel-typescript, `src/generators/numeric.ts:126`: the same two
|
|
30
|
+
expressions, resolved through `??`
|
|
31
|
+
|
|
32
|
+
This gem instead fixed both to `false`, whatever the bounds. The comment on
|
|
33
|
+
`Hegel::Generators::FloatGenerator` carried the reasoning: a caller who
|
|
34
|
+
never asks for NaN never has to reason about it, a keyword can be added
|
|
35
|
+
later without breaking anyone, and a default flipped from `true` to `false`
|
|
36
|
+
would break someone.
|
|
37
|
+
|
|
38
|
+
That reasoning weighs this gem's own callers. It leaves out what the
|
|
39
|
+
difference costs. `floats` with no bounds is the ordinary way to ask for a
|
|
40
|
+
double. Under the sibling rule that draw yields NaN and both infinities;
|
|
41
|
+
here it yields neither. A property that passes here can fail the moment
|
|
42
|
+
somebody ports it to Rust, and the Ruby test was the weaker one all along,
|
|
43
|
+
because it never drew the values that break floating-point code. The
|
|
44
|
+
difference is invisible at the call site: the same six-word call means two
|
|
45
|
+
things.
|
|
46
|
+
|
|
47
|
+
The skill this repository ships states the difference as a gotcha, so a
|
|
48
|
+
reader who finds it is warned. A reader who does not find it gets a green
|
|
49
|
+
run and a false conclusion.
|
|
50
|
+
|
|
51
|
+
## Decision
|
|
52
|
+
|
|
53
|
+
**The public surface follows the interface the hegeldev implementations
|
|
54
|
+
share.** Names, arguments, defaults, and the meaning of each. A feature
|
|
55
|
+
that this gem alone has, and a feature that this gem alone lacks, are both
|
|
56
|
+
divergences.
|
|
57
|
+
|
|
58
|
+
**A change to that interface starts as a proposal upstream.** Do not
|
|
59
|
+
implement it here first and wait for the others to follow. When a proposal
|
|
60
|
+
is accepted, this gem takes the name and the semantics that were settled on,
|
|
61
|
+
rather than choosing again.
|
|
62
|
+
|
|
63
|
+
hegel-rust stays the authority. Where the sibling implementations disagree
|
|
64
|
+
with each other, hegel-rust decides, as it already does for behaviour.
|
|
65
|
+
|
|
66
|
+
Three things stay outside this rule, because they are mechanism rather than
|
|
67
|
+
interface. How the engine is reached (the `ffi` gem, ADR 0013). What Ruby
|
|
68
|
+
alone can offer a reader, such as recovering a drawn value's name from the
|
|
69
|
+
caller's source (ADR 0005, ADR 0014). Where a Ruby hazard forces a
|
|
70
|
+
different construction, such as the control exceptions that must inherit
|
|
71
|
+
from `Exception`.
|
|
72
|
+
|
|
73
|
+
This record replaces the reasoning in the comment on
|
|
74
|
+
`Hegel::Generators::FloatGenerator`, which decided the same question for
|
|
75
|
+
`floats` alone and decided it the other way.
|
|
76
|
+
|
|
77
|
+
## Consequences
|
|
78
|
+
|
|
79
|
+
`floats` changes its defaults to the shared rule, and gains the two checks
|
|
80
|
+
that come with it: `allow_nan: true` with either bound, and
|
|
81
|
+
`allow_infinity: true` with both bounds, each raise `Hegel::Error` at draw
|
|
82
|
+
time. hegel-rust and hegel-go both write those checks; the messages here
|
|
83
|
+
carry the same substring as theirs.
|
|
84
|
+
|
|
85
|
+
This breaks callers. A property that draws unbounded `floats` and passes
|
|
86
|
+
today will see NaN and infinity after the change, and code that does not
|
|
87
|
+
handle them will fail. That failure is the point: the property was not
|
|
88
|
+
testing what it appeared to test. A caller who wants the old behaviour
|
|
89
|
+
passes `allow_nan: false, allow_infinity: false`, and the call then says
|
|
90
|
+
what it does.
|
|
91
|
+
|
|
92
|
+
Growing the surface gets slower. A generator or an option that would be
|
|
93
|
+
useful here has to be worth proposing upstream before it can ship here, and
|
|
94
|
+
a proposal can be rejected. The gem trades the speed of deciding alone for
|
|
95
|
+
callers who can move between languages.
|
|
96
|
+
|
|
97
|
+
The comparison has to be made rather than assumed. Adding an option means
|
|
98
|
+
reading how the other implementations spell it and what they default it to.
|
|
99
|
+
`hegel-rust/src/generators/` holds the generator surface, and hegel-go and
|
|
100
|
+
hegel-typescript hold the two closest readings of it for a language with
|
|
101
|
+
keyword-style options.
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
# 0016: Run mutation testing with mutineer
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted. [ADR 0017](0017-raise-the-ruby-floor-to-3-4.md) supersedes the
|
|
6
|
+
Gemfile group `mutation` in the Consequences.
|
|
7
|
+
|
|
8
|
+
## Context
|
|
9
|
+
|
|
10
|
+
The suite holds line and branch coverage at 100%. Coverage shows that a line
|
|
11
|
+
ran. It does not show that a test would notice a change to that line.
|
|
12
|
+
Mutation testing measures that: it changes the source in small ways, one at
|
|
13
|
+
a time, and runs the tests against each change.
|
|
14
|
+
|
|
15
|
+
mutineer is a mutation testing tool for Ruby. It parses the source with
|
|
16
|
+
Prism and has no runtime dependency. It runs Minitest, and it selects, for
|
|
17
|
+
each mutant, the test files that cover the mutated line.
|
|
18
|
+
|
|
19
|
+
A first run over `lib/` met four problems, all measured on mutineer 1.0.2:
|
|
20
|
+
|
|
21
|
+
- mutineer puts only `lib/` on the load path. Every test file starts with
|
|
22
|
+
`require "test_helper"`, so each one failed to load. The run then reported
|
|
23
|
+
every mutant as uncovered, with a score of N/A and exit status 0. The load
|
|
24
|
+
error appears only with `--verbose`.
|
|
25
|
+
- mutineer pairs a source with `test/<name>_test.rb` only. This suite
|
|
26
|
+
names its tests `test_<name>.rb`, so each test file must be named with
|
|
27
|
+
`--test`. The flag takes one file, and it repeats. A first run gave one
|
|
28
|
+
flag several files, and the files after the first became sources to
|
|
29
|
+
mutate, so the score read 0.7%.
|
|
30
|
+
- mutineer replaces `$stdout` with a `StringIO` before it runs the tests.
|
|
31
|
+
Minitest's `capture_subprocess_io` reopens `$stdout` on a file, so
|
|
32
|
+
`test/hegel/test_conformance.rb` failed before any mutant ran.
|
|
33
|
+
- mutineer runs every mutant's tests in this one working directory. A
|
|
34
|
+
mutant that wrote `./.hegel` left the directory in place, and the
|
|
35
|
+
teardown check in five test classes then failed the tests of every later
|
|
36
|
+
mutant too.
|
|
37
|
+
|
|
38
|
+
## Decision
|
|
39
|
+
|
|
40
|
+
`bundle exec rake mutation` (and `just mutation`) runs mutineer 1.0.2 over
|
|
41
|
+
`lib/`, with four settings:
|
|
42
|
+
|
|
43
|
+
1. `RUBYOPT` gets `-Itest`, so `require "test_helper"` resolves.
|
|
44
|
+
2. Each test file gets its own `--test` flag.
|
|
45
|
+
3. `test/hegel/test_conformance.rb` stays out of the run. It still runs in
|
|
46
|
+
`rake test`.
|
|
47
|
+
4. `--strategy redefine`. mutineer's default, `reload`, loads the whole
|
|
48
|
+
mutated file from a tempfile by a relative path, so its backtrace lines
|
|
49
|
+
show a relative path. `Hegel::Runner.origin_for` then takes a line in
|
|
50
|
+
`lib/` for the caller's, and its test fails for every mutant of
|
|
51
|
+
`lib/hegel/test_case.rb`. That alone killed three mutants that the
|
|
52
|
+
suite does not catch: one of them, applied by hand, left all 377 tests
|
|
53
|
+
passing. `redefine` loads only the mutated method.
|
|
54
|
+
|
|
55
|
+
The five teardowns share `HegelDirectoryGuard#refute_new_hegel_directory`
|
|
56
|
+
(`test/test_helper.rb`). When a test leaves a `./.hegel` it did not find at
|
|
57
|
+
the start, the guard removes the directory, then fails the test. The mutant
|
|
58
|
+
that wrote it is still killed, and later mutants no longer fail on the
|
|
59
|
+
leftover. A `./.hegel` that existed before the test stays in place.
|
|
60
|
+
|
|
61
|
+
`mutineer` is a development dependency, pinned exact and never required at
|
|
62
|
+
load time. It needs Ruby 3.4 or later, and the gem supports 3.3, so it sits
|
|
63
|
+
in the Gemfile group `mutation`. CI leaves that group out on every job, and
|
|
64
|
+
a 3.3 install leaves it out with `bundle config set --local without
|
|
65
|
+
mutation`. Mutation testing then runs on 3.4 or later only.
|
|
66
|
+
`.mutineer/` (the cache and reports) and `.hegel/` are ignored by git.
|
|
67
|
+
|
|
68
|
+
## Consequences
|
|
69
|
+
|
|
70
|
+
The run over `lib/` covers 706 mutants in about a minute: 539 killed, 153
|
|
71
|
+
survived, 8 without coverage, 6 without a verdict, a score of 77.9%. Measured
|
|
72
|
+
with the reload strategy, the guard alone turned 10 former kills into
|
|
73
|
+
survivors, which were false kills before.
|
|
74
|
+
|
|
75
|
+
The survivors are a list of tests to add. A later change works through them
|
|
76
|
+
by file.
|
|
77
|
+
|
|
78
|
+
Settings 1, 3, and 4 work around mutineer itself, and setting 2 follows
|
|
79
|
+
from its pairing rule. Each is reported: the load path as
|
|
80
|
+
[davidteren/mutineer#119](https://github.com/davidteren/mutineer/issues/119),
|
|
81
|
+
the pairing rule as
|
|
82
|
+
[davidteren/mutineer#120](https://github.com/davidteren/mutineer/issues/120),
|
|
83
|
+
the `StringIO` as
|
|
84
|
+
[davidteren/mutineer#121](https://github.com/davidteren/mutineer/issues/121),
|
|
85
|
+
and the relative path as
|
|
86
|
+
[davidteren/mutineer#123](https://github.com/davidteren/mutineer/issues/123).
|
|
87
|
+
When a mutineer release fixes one, drop the setting that it replaces.
|