hegeltest 0.1.1-aarch64-linux → 0.2.1-aarch64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +91 -0
- data/README.md +32 -9
- data/Rakefile +81 -0
- data/docs/README.md +10 -0
- data/docs/adr/0015-follow-the-hegeldev-interface-and-take-changes-upstream-first.md +101 -0
- data/docs/adr/0016-run-mutation-testing-with-mutineer.md +87 -0
- data/docs/adr/0017-raise-the-ruby-floor-to-3-4.md +43 -0
- data/docs/adr/0018-gate-mutation-testing-on-a-committed-baseline.md +74 -0
- data/docs/adr/0019-report-failures-from-the-cases-the-engine-stamps.md +70 -0
- data/docs/adr/0020-derive-span-labels-from-generator-names.md +64 -0
- data/docs/adr/0021-run-a-state-machine-in-rounds-with-its-own-step-count.md +59 -0
- data/docs/adr/0022-keep-microsecond-times-over-a-nanosecond-engine.md +45 -0
- data/docs/adr/0023-leave-unset-settings-to-the-engines-profile.md +50 -0
- data/docs/adr/0024-release-when-a-merge-changes-the-gem-version.md +49 -0
- data/docs/architecture.md +11 -6
- data/lib/hegel/draw_name.rb +3 -1
- data/lib/hegel/generator.rb +50 -9
- data/lib/hegel/generators.rb +134 -102
- data/lib/hegel/lib_hegel/real.rb +193 -94
- data/lib/hegel/lib_hegel.rb +30 -56
- data/lib/hegel/libhegel/libhegel-linux-arm64.so +0 -0
- data/lib/hegel/libhegel_version.rb +1 -1
- data/lib/hegel/report.rb +17 -8
- data/lib/hegel/runner.rb +153 -201
- data/lib/hegel/settings.rb +9 -14
- data/lib/hegel/state_machine.rb +15 -8
- data/lib/hegel/stateful/pool.rb +0 -2
- data/lib/hegel/stateful.rb +64 -36
- data/lib/hegel/syntax/methods.rb +3 -2
- data/lib/hegel/test_case.rb +22 -8
- data/lib/hegel/version.rb +1 -1
- data/lib/hegel.rb +11 -17
- data/lib/tasks/libhegel.rake +9 -2
- data/sig/hegel.rbs +58 -60
- data/skills/hegel-ruby/references/ruby/reference.md +96 -52
- metadata +12 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: aaeb9d0de3eed73e101255a9737dcd411e7819dfd559b9b6654556563bd1a9ab
|
|
4
|
+
data.tar.gz: 1a8b52232cc7ea3b746c4224b5a16aebd910405a0b6a8403e067a23ce175ad6b
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 15fd917172691c48b448ba688ca00fae54cea30bf67408a4f34f96a09a95d37564e8c6cfb90eaf7e2dc68adf4d2285d8f6f3315e771a0838d7cf9e2f8fb0d2e4
|
|
7
|
+
data.tar.gz: 28c4f53b9dfeedf865f266940a27e8d012604bb4c32c3fc63d2db5af9af48449d08a933e24e2017d6a726512653b714c7c88bcacb55afe928cd8d5ea2e04ba29
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,94 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [Unreleased]
|
|
4
|
+
|
|
5
|
+
## [0.2.1] - 2026-10-08
|
|
6
|
+
|
|
7
|
+
Each platform gem carries `libhegel` 0.45.1, up from 0.45.0. A caller has
|
|
8
|
+
nothing to change. The engine now reads a NULL string buffer with length
|
|
9
|
+
zero as the empty string, and it rejects a NULL buffer with a nonzero
|
|
10
|
+
length. These bindings send neither case, so a drawn value does not
|
|
11
|
+
change: `from_regex("", fullmatch: true)` draws `""` on both engines.
|
|
12
|
+
|
|
13
|
+
## [0.2.0] - 2026-10-07
|
|
14
|
+
|
|
15
|
+
The floor is Ruby 3.4. A project on Ruby 3.3 stays on 0.1.1. See
|
|
16
|
+
[ADR 0017](docs/adr/0017-raise-the-ruby-floor-to-3-4.md).
|
|
17
|
+
|
|
18
|
+
Each platform gem carries `libhegel` 0.45.0, up from 0.32.5. The engine
|
|
19
|
+
changed its C interface in several places between the two, and these
|
|
20
|
+
bindings follow each change.
|
|
21
|
+
|
|
22
|
+
### Changes a caller has to make
|
|
23
|
+
|
|
24
|
+
`Hegel.test` no longer takes `stateful_step_count:`. Pass `step_count:` to
|
|
25
|
+
`Hegel::Stateful.run` instead; it defaults to 50, as hegel-rust, hegel-go,
|
|
26
|
+
and hegel-java do. The engine no longer has a default of its own.
|
|
27
|
+
|
|
28
|
+
An invariant no longer runs after every rule. It runs before the first
|
|
29
|
+
rule and after the last, and between rules when the engine samples it,
|
|
30
|
+
about once in a test case that runs every step. Declare it with
|
|
31
|
+
`invariant :name, always_run: true` to keep the check after every rule.
|
|
32
|
+
See [ADR 0021](docs/adr/0021-run-a-state-machine-in-rounds-with-its-own-step-count.md).
|
|
33
|
+
|
|
34
|
+
### Failure reports
|
|
35
|
+
|
|
36
|
+
The report comes from the engine's own final replay of each failure, which
|
|
37
|
+
it runs inside the loop before the run ends. The body is no longer run
|
|
38
|
+
again after the run, so a failing run calls it once fewer per failure. A
|
|
39
|
+
`tc.note` block runs on each case the engine stamps for capture, which can
|
|
40
|
+
be more than one per failure. See
|
|
41
|
+
[ADR 0019](docs/adr/0019-report-failures-from-the-cases-the-engine-stamps.md).
|
|
42
|
+
|
|
43
|
+
A body that fails only some of the time is now reported as an ordinary
|
|
44
|
+
failure, with a `note:` line from the engine that says how often it
|
|
45
|
+
reproduced. Before, such a body ended the run in `Hegel::Error`. A failure the engine could not confirm comes
|
|
46
|
+
without a blob, and its report leaves out the reproduction line.
|
|
47
|
+
|
|
48
|
+
`reproduce_failure:` now replays its blob as a run, until a replay fails,
|
|
49
|
+
within the engine's own budget. A blob from a nondeterministic failure
|
|
50
|
+
reproduces this way too. When no replay fails, `Hegel.test` raises
|
|
51
|
+
`Hegel::Error` and names both causes: a fixed bug, or a body whose failure
|
|
52
|
+
did not recur.
|
|
53
|
+
|
|
54
|
+
A body that rejects its input with `tc.assume` or `tc.reject` before it
|
|
55
|
+
draws anything now raises `Hegel::Error` ("Unsatisfiable"). Before, the
|
|
56
|
+
run stopped after that one case and passed.
|
|
57
|
+
|
|
58
|
+
### Settings
|
|
59
|
+
|
|
60
|
+
A keyword left `nil` now takes its value from the engine's settings
|
|
61
|
+
profile: libhegel's default, a `hegel.toml`, or a `HEGEL_*` environment
|
|
62
|
+
variable such as `HEGEL_TEST_CASES`. On a CI server the engine picks its
|
|
63
|
+
`ci` profile, which derandomizes the run and turns the example database
|
|
64
|
+
off, so a run with `database_key:` and no `database:` stores nothing there.
|
|
65
|
+
A keyword you pass still wins. See
|
|
66
|
+
[ADR 0023](docs/adr/0023-leave-unset-settings-to-the-engines-profile.md).
|
|
67
|
+
|
|
68
|
+
### Generated values
|
|
69
|
+
|
|
70
|
+
The engine draws integers and floats from new distributions, and shrinks
|
|
71
|
+
further in several cases, so a seeded run draws different values
|
|
72
|
+
than it did on 0.32.5. `from_regex` follows Python's `re` more closely
|
|
73
|
+
under `(?i)` and `(?a)`, and its pattern may now hold a NUL character.
|
|
74
|
+
|
|
75
|
+
`times` and `datetimes` keep microsecond precision. The engine now draws
|
|
76
|
+
nanoseconds, and the binding rounds each one down. See
|
|
77
|
+
[ADR 0022](docs/adr/0022-keep-microsecond-times-over-a-nanosecond-engine.md).
|
|
78
|
+
|
|
79
|
+
A `datetimes` bound with a fraction of a microsecond could produce a value
|
|
80
|
+
outside it, because only the bound's whole microseconds reached the
|
|
81
|
+
engine. The lower bound now rounds up and the upper rounds down.
|
|
82
|
+
|
|
83
|
+
### Other changes
|
|
84
|
+
|
|
85
|
+
`HEGEL_LIBHEGEL_PATH` pointing at an engine that lacks a function these
|
|
86
|
+
bindings call raises `Hegel::Error` that names the function. Before, FFI
|
|
87
|
+
raised a `TypeError` that named nothing.
|
|
88
|
+
|
|
89
|
+
`rake libhegel:fetch` downloads from hegel-rust's `libhegel-v<version>`
|
|
90
|
+
release tags, which hegel-rust uses for engine releases since 0.42.1.
|
|
91
|
+
|
|
3
92
|
## [0.1.1] - 2026-08-31
|
|
4
93
|
|
|
5
94
|
The reference that ships inside the gem told a reader to install `hegeltest`
|
|
@@ -39,5 +128,7 @@ The engine is called through the `ffi` gem. Each platform gem carries the
|
|
|
39
128
|
matching `libhegel` 0.32.5 build. The platform-independent gem carries none,
|
|
40
129
|
so a run on it needs `HEGEL_LIBHEGEL_PATH` pointing at a local build.
|
|
41
130
|
|
|
131
|
+
[0.2.1]: https://github.com/meganemura/hegel-ruby/releases/tag/v0.2.1
|
|
132
|
+
[0.2.0]: https://github.com/meganemura/hegel-ruby/releases/tag/v0.2.0
|
|
42
133
|
[0.1.1]: https://github.com/meganemura/hegel-ruby/releases/tag/v0.1.1
|
|
43
134
|
[0.1.0]: https://github.com/meganemura/hegel-ruby/releases/tag/v0.1.0
|
data/README.md
CHANGED
|
@@ -55,10 +55,22 @@ compiles nothing and needs no engine on the side. The Hegel maintainers
|
|
|
55
55
|
permit a third-party project to redistribute their release binaries
|
|
56
56
|
([hegeldev/hegel-rust#411]).
|
|
57
57
|
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
58
|
+
A project that already has a `Gemfile.lock` needs one more step. Bundler keeps
|
|
59
|
+
the platform list the lock already holds. It adds only the platform of the
|
|
60
|
+
machine that runs `bundle install`. A lock therefore names only the platforms
|
|
61
|
+
of the machines that have run it. A Linux CI job cannot find the gem when the
|
|
62
|
+
lock names no Linux platform. Add each platform your CI uses:
|
|
63
|
+
|
|
64
|
+
```bash
|
|
65
|
+
bundle lock --add-platform x86_64-linux
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
A project with no lock yet reads the `PLATFORMS` section of the lock its first
|
|
69
|
+
`bundle install` writes, and adds whatever its CI needs the same way.
|
|
70
|
+
|
|
71
|
+
Every push to main runs the suite on Ruby 3.4 and 4.0, across Linux x64 and
|
|
72
|
+
arm64, macOS arm64, and Windows x64 and arm64. A pull request runs Linux x64
|
|
73
|
+
alone.
|
|
62
74
|
|
|
63
75
|
[hegeldev/hegel-rust#411]: https://github.com/hegeldev/hegel-rust/issues/411
|
|
64
76
|
|
|
@@ -70,7 +82,7 @@ pull request runs Linux x64 alone.
|
|
|
70
82
|
| Require path | `require "hegel"` (`require "hegeltest"` also works) |
|
|
71
83
|
| Namespace | `Hegel` |
|
|
72
84
|
| Binding | the `ffi` gem, which publishes a prebuilt binary for every platform above |
|
|
73
|
-
| Ruby | 3.
|
|
85
|
+
| Ruby | 3.4 and 4.0 |
|
|
74
86
|
| Platforms | Linux amd64/arm64, macOS arm64, Windows amd64/arm64 |
|
|
75
87
|
| Engine delivery | One prebuilt `libhegel` per platform-specific gem, built by `rake platform_gems:build` |
|
|
76
88
|
|
|
@@ -203,7 +215,7 @@ Expected: [0, 0]
|
|
|
203
215
|
|
|
204
216
|
Some bugs need a sequence of operations rather than one input. Declare the
|
|
205
217
|
operations as rules on a `Hegel::StateMachine`, and Hegel picks which runs
|
|
206
|
-
next, checks your invariants
|
|
218
|
+
next, checks your invariants along the way, and shrinks a failure to the
|
|
207
219
|
shortest sequence that still shows it:
|
|
208
220
|
|
|
209
221
|
```ruby
|
|
@@ -238,12 +250,23 @@ something an earlier rule produced, such as freeing a handle that some
|
|
|
238
250
|
`alloc` actually returned, put the value in a `Hegel::Stateful::Pool` and
|
|
239
251
|
draw it back out.
|
|
240
252
|
|
|
253
|
+
Every invariant runs before the first rule and after the last. Between rules,
|
|
254
|
+
the engine samples each one, so that it runs about once in a test case
|
|
255
|
+
that runs every step.
|
|
256
|
+
Declare an invariant with `always_run: true` to check it after every rule
|
|
257
|
+
instead. `Hegel::Stateful.run(machine, tc, step_count: 50)` sets the most
|
|
258
|
+
rules one test case runs; 50 is the default.
|
|
259
|
+
|
|
241
260
|
### Shaping a run
|
|
242
261
|
|
|
243
262
|
`Hegel.test` takes keywords for the rest: `test_cases`, `seed`,
|
|
244
|
-
`derandomize`, `verbosity`, `phases`, `suppress_health_check`,
|
|
245
|
-
`report_multiple_failures
|
|
246
|
-
|
|
263
|
+
`derandomize`, `verbosity`, `phases`, `suppress_health_check`, and
|
|
264
|
+
`report_multiple_failures`. `report_multiple_failures` defaults to `false`.
|
|
265
|
+
Each of the others left unset takes its value from the engine's settings
|
|
266
|
+
profile. That is libhegel's own default, unless a
|
|
267
|
+
`hegel.toml` or a `HEGEL_*` environment variable says otherwise. On a CI
|
|
268
|
+
server the engine picks its `ci` profile, which derandomizes the run and
|
|
269
|
+
turns the example database off.
|
|
247
270
|
|
|
248
271
|
Two more turn on libhegel's example database, which stores a failing case and
|
|
249
272
|
replays it first next time. `database_key` is the switch and `database`
|
data/Rakefile
CHANGED
|
@@ -17,3 +17,84 @@ task :coverage do
|
|
|
17
17
|
end
|
|
18
18
|
|
|
19
19
|
task default: %i[coverage standard]
|
|
20
|
+
|
|
21
|
+
# Mutation testing with mutineer, over lib/. Four settings make the suite
|
|
22
|
+
# run under it; ADR 0016 records why:
|
|
23
|
+
# - RUBYOPT gets -Itest, since mutineer puts only lib/ on the load path and
|
|
24
|
+
# every test file starts with `require "test_helper"`.
|
|
25
|
+
# - Each test file gets its own --test flag. The flag takes one file, and a
|
|
26
|
+
# second path after it becomes a source to mutate.
|
|
27
|
+
# - test_conformance.rb stays out. Its capture_subprocess_io reopens
|
|
28
|
+
# $stdout, and mutineer replaces $stdout with a StringIO, so the file fails
|
|
29
|
+
# before any mutant runs.
|
|
30
|
+
# - --strategy redefine reloads only the mutated method. The default,
|
|
31
|
+
# reload, loads the whole mutated file by a relative path, which breaks
|
|
32
|
+
# Hegel::Runner.origin_for and killed three mutants the suite does not
|
|
33
|
+
# catch.
|
|
34
|
+
# MUTINEER_ARGS passes more flags, for example
|
|
35
|
+
# MUTINEER_ARGS="--since origin/main" to mutate only the lines changed since
|
|
36
|
+
# origin/main.
|
|
37
|
+
#
|
|
38
|
+
# `rake mutation` fails on a survivor that the baseline does not list
|
|
39
|
+
# (ADR 0018). `rake mutation:baseline` rewrites the baseline from a full run.
|
|
40
|
+
MUTATION_BASELINE = "test/mutation-baseline.json"
|
|
41
|
+
|
|
42
|
+
def run_mutineer(*extra)
|
|
43
|
+
require "shellwords"
|
|
44
|
+
tests = Dir["test/*.rb", "test/hegel/*.rb"].sort -
|
|
45
|
+
%w[test/test_helper.rb test/hegel/test_conformance.rb]
|
|
46
|
+
args = ["lib", "--strategy", "redefine", *tests.flat_map { |t| ["--test", t] }, *extra, *Shellwords.split(ENV.fetch("MUTINEER_ARGS", ""))]
|
|
47
|
+
rubyopt = ["-Itest", ENV["RUBYOPT"]].compact.join(" ")
|
|
48
|
+
# mutineer keeps its cache here, and refuses an --output path whose
|
|
49
|
+
# directory does not exist yet.
|
|
50
|
+
mkdir_p ".mutineer", verbose: false
|
|
51
|
+
sh({"RUBYOPT" => rubyopt}, "bundle", "exec", "mutineer", "run", *args)
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
# mutineer writes its JSON report on one line, so a change to it reads as one
|
|
55
|
+
# changed line in a pull request. mutineer's --baseline reads only
|
|
56
|
+
# schema_version, summary.score, summary.scoped, and each survivor's id, so
|
|
57
|
+
# the committed file keeps those and writes one survivor per line, sorted.
|
|
58
|
+
# It leaves out line numbers: an edit above a survivor would move every
|
|
59
|
+
# line after it, and the id does not depend on the position. "change" holds
|
|
60
|
+
# the mutated line before and after, so a reviewer can read the survivor.
|
|
61
|
+
def write_mutation_baseline(report_path, baseline_path)
|
|
62
|
+
require "json"
|
|
63
|
+
report = JSON.parse(File.read(report_path))
|
|
64
|
+
# mutineer refuses a --since run as a baseline, since it covers only the
|
|
65
|
+
# changed lines.
|
|
66
|
+
abort "#{report_path} comes from a --since run; rewrite the baseline from a full run" if report.dig("summary", "scoped")
|
|
67
|
+
rows = report.fetch("survivors").map do |survivor|
|
|
68
|
+
change = survivor.fetch("diff").lines.grep(/^[-+] /).map { |line| line.chomp.sub(/^([-+]) +/, '\1 ') }.join(" | ")
|
|
69
|
+
{"id" => survivor.fetch("id"), "file" => survivor.fetch("file"), "subject" => survivor.fetch("subject"),
|
|
70
|
+
"operator" => survivor.fetch("operator"), "change" => change}
|
|
71
|
+
end
|
|
72
|
+
rows.sort_by! { |row| row.values_at("file", "subject", "id") }
|
|
73
|
+
text = +"{\n"
|
|
74
|
+
text << %( "schema_version": #{report.fetch("schema_version").to_json},\n)
|
|
75
|
+
text << %( "summary": #{report.fetch("summary").slice("score", "scoped").to_json},\n)
|
|
76
|
+
text << %( "survivors": [\n#{rows.map { |row| " #{row.to_json}" }.join(",\n")}\n ]\n}\n)
|
|
77
|
+
# The baseline is committed to a public repository, so a path from this
|
|
78
|
+
# machine must not reach it.
|
|
79
|
+
leak = [Dir.pwd, Dir.home].find { |path| text.include?(path) }
|
|
80
|
+
abort "#{baseline_path} would contain #{leak}; not written" if leak
|
|
81
|
+
File.write(baseline_path, text)
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
desc "Run mutation testing over lib/, and fail on a survivor that #{MUTATION_BASELINE} does not list"
|
|
85
|
+
task :mutation do
|
|
86
|
+
# --baseline also fails on a lower score. A mutant that times out on a slow
|
|
87
|
+
# machine leaves the score and can lower it by a fraction of a point, and
|
|
88
|
+
# it can never add a survivor. The epsilon absorbs that, and the survivor
|
|
89
|
+
# ids stay the gate.
|
|
90
|
+
run_mutineer("--baseline", MUTATION_BASELINE, "--baseline-epsilon", "1")
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
namespace :mutation do
|
|
94
|
+
desc "Rewrite #{MUTATION_BASELINE} from a full mutation testing run over lib/"
|
|
95
|
+
task :baseline do
|
|
96
|
+
report = ".mutineer/baseline-run.json"
|
|
97
|
+
run_mutineer("--format", "json", "--output", report)
|
|
98
|
+
write_mutation_baseline(report, MUTATION_BASELINE)
|
|
99
|
+
end
|
|
100
|
+
end
|
data/docs/README.md
CHANGED
|
@@ -21,6 +21,16 @@ An index of the design records for `hegel-ruby`.
|
|
|
21
21
|
- [0012: Build a failure origin from the caller's own frame](adr/0012-build-a-failure-origin-from-the-callers-own-frame.md)
|
|
22
22
|
- [0013: Bind libhegel through the ffi gem](adr/0013-bind-libhegel-through-the-ffi-gem.md)
|
|
23
23
|
- [0014: Name a drawn value only when the draw is the whole assigned value](adr/0014-name-a-drawn-value-only-when-the-draw-is-the-whole-assigned-value.md)
|
|
24
|
+
- [0015: Follow the hegeldev interface, and take changes upstream first](adr/0015-follow-the-hegeldev-interface-and-take-changes-upstream-first.md)
|
|
25
|
+
- [0016: Run mutation testing with mutineer](adr/0016-run-mutation-testing-with-mutineer.md)
|
|
26
|
+
- [0017: Raise the Ruby floor to 3.4](adr/0017-raise-the-ruby-floor-to-3-4.md)
|
|
27
|
+
- [0018: Gate mutation testing on a committed baseline](adr/0018-gate-mutation-testing-on-a-committed-baseline.md)
|
|
28
|
+
- [0019: Report failures from the cases the engine stamps](adr/0019-report-failures-from-the-cases-the-engine-stamps.md)
|
|
29
|
+
- [0020: Derive span labels from generator names](adr/0020-derive-span-labels-from-generator-names.md)
|
|
30
|
+
- [0021: Run a state machine in rounds, with its own step count](adr/0021-run-a-state-machine-in-rounds-with-its-own-step-count.md)
|
|
31
|
+
- [0022: Keep microsecond times over a nanosecond engine](adr/0022-keep-microsecond-times-over-a-nanosecond-engine.md)
|
|
32
|
+
- [0023: Leave unset settings to the engine's profile](adr/0023-leave-unset-settings-to-the-engines-profile.md)
|
|
33
|
+
- [0024: Release when a merge changes the gem version](adr/0024-release-when-a-merge-changes-the-gem-version.md)
|
|
24
34
|
|
|
25
35
|
A new decision gets a new record. A changed decision supersedes the old
|
|
26
36
|
record instead of editing it, so the history stays readable.
|
|
@@ -0,0 +1,101 @@
|
|
|
1
|
+
# 0015: Follow the hegeldev interface, and take changes upstream first
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted.
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
|
|
9
|
+
This gem is one binding of libhegel among several. The Hegel project
|
|
10
|
+
publishes implementations for Rust, Go, C++, TypeScript, Java, and OCaml,
|
|
11
|
+
and each one drives the same engine. A person who writes a property in one
|
|
12
|
+
language and then writes the same property in another expects the same call
|
|
13
|
+
to mean the same thing.
|
|
14
|
+
|
|
15
|
+
`CLAUDE.md` already names hegel-rust as the authority on meaning: what a
|
|
16
|
+
call does, what an error code means, how a value shrinks. That rule settles
|
|
17
|
+
disputes about behaviour. It does not say what to do when this gem's public
|
|
18
|
+
surface is shaped differently from every other binding while each shape
|
|
19
|
+
behaves as documented.
|
|
20
|
+
|
|
21
|
+
`floats` is that case. Three implementations write the same rule for its
|
|
22
|
+
defaults:
|
|
23
|
+
|
|
24
|
+
- hegel-rust, `src/generators/numeric.rs:315`:
|
|
25
|
+
`allow_nan.unwrap_or(!has_min && !has_max)`, and
|
|
26
|
+
`allow_infinity.unwrap_or(!has_min || !has_max)`
|
|
27
|
+
- hegel-go, `primitives.go:136`: the same two expressions, resolved through
|
|
28
|
+
a nil pointer check
|
|
29
|
+
- hegel-typescript, `src/generators/numeric.ts:126`: the same two
|
|
30
|
+
expressions, resolved through `??`
|
|
31
|
+
|
|
32
|
+
This gem instead fixed both to `false`, whatever the bounds. The comment on
|
|
33
|
+
`Hegel::Generators::FloatGenerator` carried the reasoning: a caller who
|
|
34
|
+
never asks for NaN never has to reason about it, a keyword can be added
|
|
35
|
+
later without breaking anyone, and a default flipped from `true` to `false`
|
|
36
|
+
would break someone.
|
|
37
|
+
|
|
38
|
+
That reasoning weighs this gem's own callers. It leaves out what the
|
|
39
|
+
difference costs. `floats` with no bounds is the ordinary way to ask for a
|
|
40
|
+
double. Under the sibling rule that draw yields NaN and both infinities;
|
|
41
|
+
here it yields neither. A property that passes here can fail the moment
|
|
42
|
+
somebody ports it to Rust, and the Ruby test was the weaker one all along,
|
|
43
|
+
because it never drew the values that break floating-point code. The
|
|
44
|
+
difference is invisible at the call site: the same six-word call means two
|
|
45
|
+
things.
|
|
46
|
+
|
|
47
|
+
The skill this repository ships states the difference as a gotcha, so a
|
|
48
|
+
reader who finds it is warned. A reader who does not find it gets a green
|
|
49
|
+
run and a false conclusion.
|
|
50
|
+
|
|
51
|
+
## Decision
|
|
52
|
+
|
|
53
|
+
**The public surface follows the interface the hegeldev implementations
|
|
54
|
+
share.** Names, arguments, defaults, and the meaning of each. A feature
|
|
55
|
+
that this gem alone has, and a feature that this gem alone lacks, are both
|
|
56
|
+
divergences.
|
|
57
|
+
|
|
58
|
+
**A change to that interface starts as a proposal upstream.** Do not
|
|
59
|
+
implement it here first and wait for the others to follow. When a proposal
|
|
60
|
+
is accepted, this gem takes the name and the semantics that were settled on,
|
|
61
|
+
rather than choosing again.
|
|
62
|
+
|
|
63
|
+
hegel-rust stays the authority. Where the sibling implementations disagree
|
|
64
|
+
with each other, hegel-rust decides, as it already does for behaviour.
|
|
65
|
+
|
|
66
|
+
Three things stay outside this rule, because they are mechanism rather than
|
|
67
|
+
interface. How the engine is reached (the `ffi` gem, ADR 0013). What Ruby
|
|
68
|
+
alone can offer a reader, such as recovering a drawn value's name from the
|
|
69
|
+
caller's source (ADR 0005, ADR 0014). Where a Ruby hazard forces a
|
|
70
|
+
different construction, such as the control exceptions that must inherit
|
|
71
|
+
from `Exception`.
|
|
72
|
+
|
|
73
|
+
This record replaces the reasoning in the comment on
|
|
74
|
+
`Hegel::Generators::FloatGenerator`, which decided the same question for
|
|
75
|
+
`floats` alone and decided it the other way.
|
|
76
|
+
|
|
77
|
+
## Consequences
|
|
78
|
+
|
|
79
|
+
`floats` changes its defaults to the shared rule, and gains the two checks
|
|
80
|
+
that come with it: `allow_nan: true` with either bound, and
|
|
81
|
+
`allow_infinity: true` with both bounds, each raise `Hegel::Error` at draw
|
|
82
|
+
time. hegel-rust and hegel-go both write those checks; the messages here
|
|
83
|
+
carry the same substring as theirs.
|
|
84
|
+
|
|
85
|
+
This breaks callers. A property that draws unbounded `floats` and passes
|
|
86
|
+
today will see NaN and infinity after the change, and code that does not
|
|
87
|
+
handle them will fail. That failure is the point: the property was not
|
|
88
|
+
testing what it appeared to test. A caller who wants the old behaviour
|
|
89
|
+
passes `allow_nan: false, allow_infinity: false`, and the call then says
|
|
90
|
+
what it does.
|
|
91
|
+
|
|
92
|
+
Growing the surface gets slower. A generator or an option that would be
|
|
93
|
+
useful here has to be worth proposing upstream before it can ship here, and
|
|
94
|
+
a proposal can be rejected. The gem trades the speed of deciding alone for
|
|
95
|
+
callers who can move between languages.
|
|
96
|
+
|
|
97
|
+
The comparison has to be made rather than assumed. Adding an option means
|
|
98
|
+
reading how the other implementations spell it and what they default it to.
|
|
99
|
+
`hegel-rust/src/generators/` holds the generator surface, and hegel-go and
|
|
100
|
+
hegel-typescript hold the two closest readings of it for a language with
|
|
101
|
+
keyword-style options.
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
# 0016: Run mutation testing with mutineer
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted. [ADR 0017](0017-raise-the-ruby-floor-to-3-4.md) supersedes the
|
|
6
|
+
Gemfile group `mutation` in the Consequences.
|
|
7
|
+
|
|
8
|
+
## Context
|
|
9
|
+
|
|
10
|
+
The suite holds line and branch coverage at 100%. Coverage shows that a line
|
|
11
|
+
ran. It does not show that a test would notice a change to that line.
|
|
12
|
+
Mutation testing measures that: it changes the source in small ways, one at
|
|
13
|
+
a time, and runs the tests against each change.
|
|
14
|
+
|
|
15
|
+
mutineer is a mutation testing tool for Ruby. It parses the source with
|
|
16
|
+
Prism and has no runtime dependency. It runs Minitest, and it selects, for
|
|
17
|
+
each mutant, the test files that cover the mutated line.
|
|
18
|
+
|
|
19
|
+
A first run over `lib/` met four problems, all measured on mutineer 1.0.2:
|
|
20
|
+
|
|
21
|
+
- mutineer puts only `lib/` on the load path. Every test file starts with
|
|
22
|
+
`require "test_helper"`, so each one failed to load. The run then reported
|
|
23
|
+
every mutant as uncovered, with a score of N/A and exit status 0. The load
|
|
24
|
+
error appears only with `--verbose`.
|
|
25
|
+
- mutineer pairs a source with `test/<name>_test.rb` only. This suite
|
|
26
|
+
names its tests `test_<name>.rb`, so each test file must be named with
|
|
27
|
+
`--test`. The flag takes one file, and it repeats. A first run gave one
|
|
28
|
+
flag several files, and the files after the first became sources to
|
|
29
|
+
mutate, so the score read 0.7%.
|
|
30
|
+
- mutineer replaces `$stdout` with a `StringIO` before it runs the tests.
|
|
31
|
+
Minitest's `capture_subprocess_io` reopens `$stdout` on a file, so
|
|
32
|
+
`test/hegel/test_conformance.rb` failed before any mutant ran.
|
|
33
|
+
- mutineer runs every mutant's tests in this one working directory. A
|
|
34
|
+
mutant that wrote `./.hegel` left the directory in place, and the
|
|
35
|
+
teardown check in five test classes then failed the tests of every later
|
|
36
|
+
mutant too.
|
|
37
|
+
|
|
38
|
+
## Decision
|
|
39
|
+
|
|
40
|
+
`bundle exec rake mutation` (and `just mutation`) runs mutineer 1.0.2 over
|
|
41
|
+
`lib/`, with four settings:
|
|
42
|
+
|
|
43
|
+
1. `RUBYOPT` gets `-Itest`, so `require "test_helper"` resolves.
|
|
44
|
+
2. Each test file gets its own `--test` flag.
|
|
45
|
+
3. `test/hegel/test_conformance.rb` stays out of the run. It still runs in
|
|
46
|
+
`rake test`.
|
|
47
|
+
4. `--strategy redefine`. mutineer's default, `reload`, loads the whole
|
|
48
|
+
mutated file from a tempfile by a relative path, so its backtrace lines
|
|
49
|
+
show a relative path. `Hegel::Runner.origin_for` then takes a line in
|
|
50
|
+
`lib/` for the caller's, and its test fails for every mutant of
|
|
51
|
+
`lib/hegel/test_case.rb`. That alone killed three mutants that the
|
|
52
|
+
suite does not catch: one of them, applied by hand, left all 377 tests
|
|
53
|
+
passing. `redefine` loads only the mutated method.
|
|
54
|
+
|
|
55
|
+
The five teardowns share `HegelDirectoryGuard#refute_new_hegel_directory`
|
|
56
|
+
(`test/test_helper.rb`). When a test leaves a `./.hegel` it did not find at
|
|
57
|
+
the start, the guard removes the directory, then fails the test. The mutant
|
|
58
|
+
that wrote it is still killed, and later mutants no longer fail on the
|
|
59
|
+
leftover. A `./.hegel` that existed before the test stays in place.
|
|
60
|
+
|
|
61
|
+
`mutineer` is a development dependency, pinned exact and never required at
|
|
62
|
+
load time. It needs Ruby 3.4 or later, and the gem supports 3.3, so it sits
|
|
63
|
+
in the Gemfile group `mutation`. CI leaves that group out on every job, and
|
|
64
|
+
a 3.3 install leaves it out with `bundle config set --local without
|
|
65
|
+
mutation`. Mutation testing then runs on 3.4 or later only.
|
|
66
|
+
`.mutineer/` (the cache and reports) and `.hegel/` are ignored by git.
|
|
67
|
+
|
|
68
|
+
## Consequences
|
|
69
|
+
|
|
70
|
+
The run over `lib/` covers 706 mutants in about a minute: 539 killed, 153
|
|
71
|
+
survived, 8 without coverage, 6 without a verdict, a score of 77.9%. Measured
|
|
72
|
+
with the reload strategy, the guard alone turned 10 former kills into
|
|
73
|
+
survivors, which were false kills before.
|
|
74
|
+
|
|
75
|
+
The survivors are a list of tests to add. A later change works through them
|
|
76
|
+
by file.
|
|
77
|
+
|
|
78
|
+
Settings 1, 3, and 4 work around mutineer itself, and setting 2 follows
|
|
79
|
+
from its pairing rule. Each is reported: the load path as
|
|
80
|
+
[davidteren/mutineer#119](https://github.com/davidteren/mutineer/issues/119),
|
|
81
|
+
the pairing rule as
|
|
82
|
+
[davidteren/mutineer#120](https://github.com/davidteren/mutineer/issues/120),
|
|
83
|
+
the `StringIO` as
|
|
84
|
+
[davidteren/mutineer#121](https://github.com/davidteren/mutineer/issues/121),
|
|
85
|
+
and the relative path as
|
|
86
|
+
[davidteren/mutineer#123](https://github.com/davidteren/mutineer/issues/123).
|
|
87
|
+
When a mutineer release fixes one, drop the setting that it replaces.
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# 0017: Raise the Ruby floor to 3.4
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted. Supersedes the Gemfile group `mutation` in the Consequences of
|
|
6
|
+
[ADR 0016](0016-run-mutation-testing-with-mutineer.md).
|
|
7
|
+
|
|
8
|
+
## Context
|
|
9
|
+
|
|
10
|
+
The gem supported Ruby 3.3, 3.4, and 4.0. Ruby 3.3 has had security
|
|
11
|
+
maintenance only since 2026-04-01, and its end of life is expected on
|
|
12
|
+
2027-03-31.
|
|
13
|
+
|
|
14
|
+
mutineer 1.0.2, the mutation testing tool of ADR 0016, needs Ruby 3.4 or
|
|
15
|
+
later. With mutineer in the Gemfile, `bundle install` failed on every 3.3 job
|
|
16
|
+
in CI. The first fix put mutineer in a Gemfile group `mutation`. Every CI job
|
|
17
|
+
left that group out, and a developer on 3.3 had to leave it out by hand. That
|
|
18
|
+
fix gave the repository two install paths, and CI never installed a tool that
|
|
19
|
+
the repository runs.
|
|
20
|
+
|
|
21
|
+
## Decision
|
|
22
|
+
|
|
23
|
+
The floor is Ruby 3.4:
|
|
24
|
+
|
|
25
|
+
- `required_ruby_version` is `>= 3.4.0`.
|
|
26
|
+
- CI runs Ruby 3.4 and 4.0.
|
|
27
|
+
- `.standard.yml` targets Ruby 3.4.
|
|
28
|
+
- mutineer is back in the default Gemfile group, and CI installs it.
|
|
29
|
+
|
|
30
|
+
Keeping 3.3 with the group was refused, because of the two install paths above.
|
|
31
|
+
|
|
32
|
+
## Consequences
|
|
33
|
+
|
|
34
|
+
A project on Ruby 3.3 stays on 0.1.1, the last release that installs there.
|
|
35
|
+
The next release says so in the changelog.
|
|
36
|
+
|
|
37
|
+
Every CI job installs the same bundle, and a developer needs one Ruby for the
|
|
38
|
+
gem and its tools. The matrix loses its one exclusion: RubyInstaller has an
|
|
39
|
+
arm64 Windows build for every Ruby in it.
|
|
40
|
+
|
|
41
|
+
ADR 0001, ADR 0005, and ADR 0008 name 3.3 as the floor, which it was when
|
|
42
|
+
they were written. ADR 0005's reason still holds on 3.4: Ruby 3.4 also ships
|
|
43
|
+
Prism as a default gem.
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
# 0018: Gate mutation testing on a committed baseline
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted. It builds on
|
|
6
|
+
[ADR 0016](0016-run-mutation-testing-with-mutineer.md).
|
|
7
|
+
|
|
8
|
+
## Context
|
|
9
|
+
|
|
10
|
+
ADR 0016 runs mutineer over `lib/`: 706 mutants in about a minute, with 153
|
|
11
|
+
survivors. Nothing checked the result, so a change could add a survivor and
|
|
12
|
+
no one would see it.
|
|
13
|
+
|
|
14
|
+
mutineer has two ways to narrow what fails a run. `--since REF` mutates only
|
|
15
|
+
the lines changed since `REF`. `--baseline FILE` compares the run with an
|
|
16
|
+
earlier JSON report, and fails on a survivor whose stable id is not in the
|
|
17
|
+
report, or on a lower score. The id does not depend on the line number.
|
|
18
|
+
|
|
19
|
+
mutineer writes its JSON report on one line of about 55 KB. Committed as it
|
|
20
|
+
is, a change to the survivors reads as one changed line in a pull request.
|
|
21
|
+
|
|
22
|
+
## Decision
|
|
23
|
+
|
|
24
|
+
`test/mutation-baseline.json` holds the accepted survivors, and CI fails on
|
|
25
|
+
any other survivor:
|
|
26
|
+
|
|
27
|
+
- `rake mutation` runs over all of `lib/` with `--baseline` on that file.
|
|
28
|
+
`rake mutation:baseline` rewrites the file from a full run.
|
|
29
|
+
- The file keeps what `--baseline` reads: `schema_version`, `summary.score`,
|
|
30
|
+
`summary.scoped`, and each survivor's `id`. Each survivor is one line,
|
|
31
|
+
sorted by file, method, and id, with the mutated line before and after. It
|
|
32
|
+
leaves out line numbers.
|
|
33
|
+
- `--baseline-epsilon 1` absorbs a small score drop. A mutant that times out
|
|
34
|
+
on a slow machine leaves the score, and it can never add a survivor.
|
|
35
|
+
- The CI job `mutation` runs `rake mutation` once, on Linux with Ruby 4.0, for
|
|
36
|
+
every push and pull request.
|
|
37
|
+
- `rake mutation:baseline` refuses a `--since` run, which mutineer refuses as
|
|
38
|
+
a baseline too. It also refuses to write a file that contains the working
|
|
39
|
+
directory or the home directory, because the file is public.
|
|
40
|
+
|
|
41
|
+
A full run takes about a minute, so CI runs in full. `--since` stays for a
|
|
42
|
+
local run on a change in progress.
|
|
43
|
+
|
|
44
|
+
Two alternatives were refused:
|
|
45
|
+
|
|
46
|
+
- A baseline made on main by CI and passed to a pull request as an artifact.
|
|
47
|
+
It adds no file to the repository, but a reviewer cannot see a change to the
|
|
48
|
+
survivors, and the workflow needs to find the artifact of the right run.
|
|
49
|
+
- A gate on `--since` alone, which fails on any survivor on a changed line. An
|
|
50
|
+
edit to a line with an old survivor would then fail for a survivor that the
|
|
51
|
+
edit did not add.
|
|
52
|
+
|
|
53
|
+
## Consequences
|
|
54
|
+
|
|
55
|
+
A pull request that adds a survivor fails, until it adds a test or accepts
|
|
56
|
+
the survivor with `rake mutation:baseline`. The accepted survivor then shows
|
|
57
|
+
as one added line in the diff. A pull request that kills a survivor passes
|
|
58
|
+
without a change to the file, and the next rewrite shows the survivor as one
|
|
59
|
+
removed line.
|
|
60
|
+
|
|
61
|
+
A survivor that no test can kill, because the mutant behaves the same as the
|
|
62
|
+
original, gets a `# mutineer:disable-line <operator>` marker on its line,
|
|
63
|
+
with the reason in a comment on the line above. The judgment then moves
|
|
64
|
+
with the code through an edit or a move, which an id list in
|
|
65
|
+
`.mutineer.yml` does not do. The reason cannot follow the marker on the same
|
|
66
|
+
line, because mutineer reads every word after the marker as an operator
|
|
67
|
+
name. The survivor then leaves the baseline too.
|
|
68
|
+
|
|
69
|
+
A marker suppresses every mutant of its operator on its line. When a line
|
|
70
|
+
holds an equivalent mutant and a killed one of the same operator, a marker
|
|
71
|
+
would also hide the killed one. `Runner.classify` has three such lines,
|
|
72
|
+
and `TestCase#generate_boolean` has one. Those equivalent mutants go in
|
|
73
|
+
`.mutineer.yml` under `ignore:`, by their stable id. The reason stays in a
|
|
74
|
+
comment beside the code, and the comment names `.mutineer.yml`.
|