hegeltest 0.3.4 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +182 -0
- data/README.ja.md +408 -0
- data/README.md +208 -102
- data/Rakefile +6 -3
- data/docs/README.md +9 -0
- data/docs/adr/0027-report-each-runs-test-location-from-the-callers-frame.md +63 -0
- data/docs/adr/0028-expose-settings-profiles-as-a-keyword-and-module-functions.md +91 -0
- data/docs/adr/0029-keep-notes-local-and-defer-run-error-kinds.md +72 -0
- data/docs/adr/0030-represent-a-duration-as-integer-nanoseconds.md +48 -0
- data/docs/adr/0031-take-the-alphabet-options-from-hegel-rust.md +52 -0
- data/docs/adr/0032-suppress-span-closes-while-an-abandoned-recursion-attempt-unwinds.md +147 -0
- data/docs/adr/0033-cache-compiled-grammars-for-the-length-of-a-run.md +80 -0
- data/docs/adr/0034-repeat-a-test-body-with-a-ruby-side-indent.md +120 -0
- data/docs/adr/0035-defer-concurrent-state-machines-and-the-engine-printer.md +136 -0
- data/docs/architecture.md +16 -1
- data/lib/hegel/errors.rb +24 -0
- data/lib/hegel/generator.rb +52 -25
- data/lib/hegel/generators.rb +715 -45
- data/lib/hegel/grammar_cache.rb +36 -0
- data/lib/hegel/lib_hegel/real.rb +320 -17
- data/lib/hegel/lib_hegel.rb +31 -6
- data/lib/hegel/libhegel_version.rb +1 -1
- data/lib/hegel/report.rb +13 -10
- data/lib/hegel/runner.rb +61 -27
- data/lib/hegel/settings.rb +92 -16
- data/lib/hegel/state_machine.rb +17 -4
- data/lib/hegel/stateful/pool.rb +52 -0
- data/lib/hegel/stateful.rb +2 -2
- data/lib/hegel/syntax/methods.rb +99 -15
- data/lib/hegel/test_case.rb +256 -35
- data/lib/hegel/test_location.rb +49 -0
- data/lib/hegel/version.rb +1 -1
- data/lib/hegel.rb +21 -5
- data/sig/hegel.rbs +295 -42
- data/skills/hegel-ruby/references/ruby/reference.md +463 -41
- metadata +13 -1
data/README.md
CHANGED
|
@@ -27,78 +27,72 @@ failure to a minimal counterexample.
|
|
|
27
27
|
|
|
28
28
|
This gem drives [libhegel](https://hegel.dev/reference/libhegel), the native
|
|
29
29
|
engine that the official implementations also drive. The engine runs in the
|
|
30
|
-
same process. There is no
|
|
30
|
+
same process, through the [`ffi`](https://github.com/ffi/ffi) gem. There is no
|
|
31
|
+
server and no Python dependency.
|
|
31
32
|
|
|
32
|
-
##
|
|
33
|
+
## Installation
|
|
33
34
|
|
|
34
|
-
|
|
35
|
-
names the values it drew, and re-raises your own exception.
|
|
36
|
-
|
|
37
|
-
Work ran in three stages, and all three are done:
|
|
38
|
-
|
|
39
|
-
1. **The walking skeleton**: the libhegel binding, the run loop, and failure
|
|
40
|
-
reports.
|
|
41
|
-
2. **The full generator set**: twenty-five generators, composing through
|
|
42
|
-
`map` and `filter`.
|
|
43
|
-
3. **The advanced features**: the example database, targeted testing,
|
|
44
|
-
stateful testing, phases, and health checks.
|
|
45
|
-
|
|
46
|
-
Install it the way you install any gem:
|
|
35
|
+
The gem is published as [`hegeltest`](https://rubygems.org/gems/hegeltest):
|
|
47
36
|
|
|
48
37
|
```ruby
|
|
49
38
|
# Gemfile
|
|
50
|
-
gem "hegeltest"
|
|
39
|
+
gem "hegeltest", group: :test
|
|
51
40
|
```
|
|
52
41
|
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
42
|
+
The require path is `hegel`, and the namespace is `Hegel`.
|
|
43
|
+
`require "hegeltest"` also works, so Bundler's automatic require finds it.
|
|
44
|
+
|
|
45
|
+
The gem needs Ruby 3.4 or later. Each platform gem carries the matching
|
|
46
|
+
`libhegel` build, so an install compiles nothing and needs no engine on the
|
|
47
|
+
side:
|
|
48
|
+
|
|
49
|
+
| Platform | Gem platform |
|
|
50
|
+
| --- | --- |
|
|
51
|
+
| Linux x64 | `x86_64-linux` |
|
|
52
|
+
| Linux arm64 | `aarch64-linux` |
|
|
53
|
+
| macOS arm64 | `arm64-darwin` |
|
|
54
|
+
| Windows x64 | `x64-mingw-ucrt` |
|
|
55
|
+
| Windows arm64 | `aarch64-mingw-ucrt` |
|
|
56
|
+
|
|
57
|
+
The Hegel maintainers permit a third-party project to redistribute their
|
|
58
|
+
release binaries ([hegeldev/hegel-rust#411]).
|
|
57
59
|
|
|
58
|
-
A project that already has a `Gemfile.lock` needs one more step. Bundler
|
|
59
|
-
the platform
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
lock names no Linux platform. Add each platform your CI uses:
|
|
60
|
+
A project that already has a `Gemfile.lock` needs one more step. Bundler adds
|
|
61
|
+
only the platform of the machine that runs `bundle install`, so a lock made on
|
|
62
|
+
a Mac does not name Linux. A Linux CI job then cannot find the gem. Add each
|
|
63
|
+
platform your CI uses:
|
|
63
64
|
|
|
64
65
|
```bash
|
|
65
66
|
bundle lock --add-platform x86_64-linux
|
|
66
67
|
```
|
|
67
68
|
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
arm64, macOS arm64, and Windows x64 and arm64. A pull request runs Linux x64
|
|
73
|
-
alone.
|
|
69
|
+
On a platform with no published `libhegel` build, such as macOS on Intel, build
|
|
70
|
+
the engine yourself and set `HEGEL_LIBHEGEL_PATH` to the library file or to the
|
|
71
|
+
directory that holds it. The same variable makes any platform use a local
|
|
72
|
+
engine build in place of the bundled one.
|
|
74
73
|
|
|
75
74
|
[hegeldev/hegel-rust#411]: https://github.com/hegeldev/hegel-rust/issues/411
|
|
76
75
|
|
|
77
|
-
|
|
76
|
+
### Which libhegel a release carries
|
|
78
77
|
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
| Gem name | `hegeltest` |
|
|
82
|
-
| Require path | `require "hegel"` (`require "hegeltest"` also works) |
|
|
83
|
-
| Namespace | `Hegel` |
|
|
84
|
-
| Binding | the `ffi` gem, which publishes a prebuilt binary for every platform above |
|
|
85
|
-
| Ruby | 3.4 and 4.0 |
|
|
86
|
-
| Platforms | Linux amd64/arm64, macOS arm64, Windows amd64/arm64 |
|
|
87
|
-
| Engine delivery | One prebuilt `libhegel` per platform-specific gem, built by `rake platform_gems:build` |
|
|
78
|
+
Each gem release pins one libhegel release. To see the pinned version in your
|
|
79
|
+
installed gem:
|
|
88
80
|
|
|
89
|
-
|
|
81
|
+
```bash
|
|
82
|
+
ruby -rhegel -e 'puts Hegel::LIBHEGEL_VERSION'
|
|
83
|
+
```
|
|
90
84
|
|
|
91
|
-
The
|
|
92
|
-
|
|
93
|
-
|
|
85
|
+
The [CHANGELOG](CHANGELOG.md) section of each gem release starts with the
|
|
86
|
+
engine move, when there is one, and links the release notes of that engine
|
|
87
|
+
version. The [GitHub releases](https://github.com/meganemura/hegel-ruby/releases)
|
|
88
|
+
show the same text.
|
|
94
89
|
|
|
95
|
-
|
|
96
|
-
|
|
90
|
+
When libhegel publishes a new release, a pull request moves this gem to it.
|
|
91
|
+
The gem version and the engine version are independent: a gem release can
|
|
92
|
+
change the engine, the Ruby code, or both.
|
|
97
93
|
|
|
98
94
|
## Quickstart
|
|
99
95
|
|
|
100
|
-
Everything below runs today.
|
|
101
|
-
|
|
102
96
|
```ruby
|
|
103
97
|
# spec/spec_helper.rb
|
|
104
98
|
require "hegel"
|
|
@@ -123,7 +117,7 @@ end
|
|
|
123
117
|
```
|
|
124
118
|
|
|
125
119
|
That test fails, and Hegel shrinks the failure to the smallest input that
|
|
126
|
-
still shows the bug:
|
|
120
|
+
still shows the bug. The report looks like this:
|
|
127
121
|
|
|
128
122
|
```
|
|
129
123
|
Falsified after 11 test cases (0 discarded):
|
|
@@ -134,27 +128,26 @@ To reproduce this failure, pass the blob below to Hegel.test:
|
|
|
134
128
|
reproduce_failure: "AXicY2VgYGBkZOBiZEBhMAAAAd8AIQ=="
|
|
135
129
|
```
|
|
136
130
|
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
131
|
+
The count of test cases and the blob change from run to run, and between
|
|
132
|
+
engine releases. The shrunk value is the part to read: two duplicates are all
|
|
133
|
+
it takes. Hegel names the value `xs` by reading the line the draw was written
|
|
134
|
+
on. Then it re-raises RSpec's own expectation failure, so RSpec reports it as
|
|
135
|
+
its own.
|
|
140
136
|
|
|
141
137
|
Including `Hegel::Syntax::Methods` makes the generators available without a
|
|
142
|
-
prefix, the way FactoryBot makes `create` available. The same generators
|
|
143
|
-
|
|
138
|
+
prefix, the way FactoryBot makes `create` available. The same generators are
|
|
139
|
+
also available as `Hegel::Generators.arrays(...)` without the include.
|
|
144
140
|
|
|
145
|
-
|
|
146
|
-
|
|
141
|
+
To keep the generator names out of example groups that have their own
|
|
142
|
+
helpers, include the methods only in tagged groups:
|
|
147
143
|
|
|
148
144
|
```ruby
|
|
149
145
|
config.include Hegel::Syntax::Methods, :hegel
|
|
150
146
|
```
|
|
151
147
|
|
|
152
|
-
A new project can include the methods in every group, as the Quickstart does.
|
|
153
|
-
|
|
154
148
|
Hegel draws values imperatively, so an ordinary Ruby method can share a group
|
|
155
149
|
of draws between properties. Give the method parameter a descriptive name,
|
|
156
|
-
which the default RuboCop naming rule accepts
|
|
157
|
-
needs no change:
|
|
150
|
+
which the default RuboCop naming rule accepts:
|
|
158
151
|
|
|
159
152
|
```ruby
|
|
160
153
|
def draw_sorted_pair(test_case)
|
|
@@ -169,15 +162,9 @@ Hegel.test do |tc|
|
|
|
169
162
|
end
|
|
170
163
|
```
|
|
171
164
|
|
|
172
|
-
The generators are `arrays`, `binary`, `booleans`, `characters`, `composite`,
|
|
173
|
-
`dates`, `datetimes`, `deferred`, `domains`, `emails`, `floats`, `from_regex`,
|
|
174
|
-
`hashes`, `integers`, `ip_addresses`, `just`, `one_of`, `optional`,
|
|
175
|
-
`sampled_from`, `sets`, `text`, `times`, `tuples`, `urls`, and `uuids`. Each
|
|
176
|
-
one composes through `map` and `filter`.
|
|
177
|
-
|
|
178
165
|
### Minitest
|
|
179
166
|
|
|
180
|
-
Hegel needs nothing from your test framework.
|
|
167
|
+
Hegel needs nothing from your test framework. Include the generator methods
|
|
181
168
|
once, and write the property inside an ordinary test:
|
|
182
169
|
|
|
183
170
|
```ruby
|
|
@@ -201,8 +188,7 @@ class MySortTest < Minitest::Test
|
|
|
201
188
|
end
|
|
202
189
|
```
|
|
203
190
|
|
|
204
|
-
Minitest reports the shrunk case as its own assertion failure,
|
|
205
|
-
your line:
|
|
191
|
+
Minitest reports the shrunk case as its own assertion failure, at your line:
|
|
206
192
|
|
|
207
193
|
```
|
|
208
194
|
1) Failure:
|
|
@@ -211,10 +197,28 @@ Expected: [0, 0]
|
|
|
211
197
|
Actual: [0]
|
|
212
198
|
```
|
|
213
199
|
|
|
214
|
-
|
|
200
|
+
## Generators
|
|
201
|
+
|
|
202
|
+
| Kind | Generators |
|
|
203
|
+
| --- | --- |
|
|
204
|
+
| Numbers and booleans | `integers`, `floats`, `booleans` |
|
|
205
|
+
| Text and bytes | `text`, `characters`, `binary`, `from_regex`, `from_lark` |
|
|
206
|
+
| Collections | `arrays`, `sets`, `hashes`, `tuples` |
|
|
207
|
+
| Choice | `just`, `sampled_from`, `permutations`, `subsequences`, `samples`, `one_of`, `optional` |
|
|
208
|
+
| Formats | `emails`, `urls`, `domains`, `ip_addresses`, `uuids` |
|
|
209
|
+
| Time | `dates`, `times`, `datetimes`, `date_strings`, `time_strings`, `datetime_strings`, `durations` |
|
|
210
|
+
| Your own | `composite`, `deferred`, `recursive` |
|
|
211
|
+
|
|
212
|
+
Each one composes through `map` and `filter`. Options are keywords, with the
|
|
213
|
+
names the other Hegel implementations use, such as
|
|
214
|
+
`integers(min_value: 0, max_value: 9)`, `arrays(text, max_size: 5)`, or `arrays(integers, unique: true)`. An
|
|
215
|
+
invalid combination of options raises when the generator is drawn, not when it
|
|
216
|
+
is built.
|
|
217
|
+
|
|
218
|
+
## Stateful testing
|
|
215
219
|
|
|
216
220
|
Some bugs need a sequence of operations rather than one input. Declare the
|
|
217
|
-
operations as rules on a `Hegel::StateMachine
|
|
221
|
+
operations as rules on a `Hegel::StateMachine`. Hegel picks which rule runs
|
|
218
222
|
next, checks your invariants along the way, and shrinks a failure to the
|
|
219
223
|
shortest sequence that still shows it:
|
|
220
224
|
|
|
@@ -245,32 +249,48 @@ Hegel.test { |tc| Hegel::Stateful.run(StackMachine.new, tc) }
|
|
|
245
249
|
```
|
|
246
250
|
|
|
247
251
|
`tc.assume` inside a rule rejects that rule and lets Hegel choose another,
|
|
248
|
-
rather than throwing the whole test case away.
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
draw it back out.
|
|
252
|
+
rather than throwing the whole test case away. When a rule must act on a value
|
|
253
|
+
that an earlier rule made, such as freeing a handle that some `alloc` returned,
|
|
254
|
+
put the value in a `Hegel::Stateful::Pool` and draw it back out.
|
|
252
255
|
|
|
253
256
|
Every invariant runs before the first rule and after the last. Between rules,
|
|
254
|
-
the engine samples each
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
`
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
`
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
257
|
+
the engine samples each invariant. Declare an invariant with
|
|
258
|
+
`always_run: true` to check it after every rule.
|
|
259
|
+
`Hegel::Stateful.run(machine, tc, step_count: 50)` sets the largest number of
|
|
260
|
+
rules that one test case runs; 50 is the default. `rule :name, weight: 2.5`
|
|
261
|
+
asks Hegel to pick a rule more often than the others; the default weight is
|
|
262
|
+
1.0.
|
|
263
|
+
|
|
264
|
+
## Configuring a run
|
|
265
|
+
|
|
266
|
+
`Hegel.test` takes these keywords:
|
|
267
|
+
|
|
268
|
+
| Keyword | Effect |
|
|
269
|
+
| --- | --- |
|
|
270
|
+
| `profile` | The settings profile the run starts from; see [Profiles](#profiles) |
|
|
271
|
+
| `test_cases` | How many test cases to generate |
|
|
272
|
+
| `seed` | The seed for the random generation |
|
|
273
|
+
| `derandomize` | Make the generation deterministic |
|
|
274
|
+
| `verbosity` | How much the engine prints; `:quiet` also hides the failure report |
|
|
275
|
+
| `phases` | Which engine phases run, such as generation and shrinking |
|
|
276
|
+
| `suppress_health_check` | Which health checks to turn off |
|
|
277
|
+
| `report_multiple_failures` | Report each distinct failure; `false` by default |
|
|
278
|
+
| `stop_on_first_failure` | `false` spends the whole `test_cases` budget before reporting |
|
|
279
|
+
| `print_blob` | `false` leaves the reproduction line out of the failure report |
|
|
280
|
+
| `show_statistics` | Print what `tc.event` and `tc.event_value` recorded, at the end of the run |
|
|
281
|
+
| `nondeterminism_strictness` | `:quiet`, `:warn`, or `:error` when the test behaves differently on a replay |
|
|
282
|
+
| `backend` | `:default`, or `:urandom` to take fresh entropy on every draw, for Antithesis |
|
|
283
|
+
| `database_key` | Turn the example database on for this property |
|
|
284
|
+
| `database` | The directory of the example database |
|
|
285
|
+
| `reproduce_failure` | Replay one failure from the blob in its report |
|
|
286
|
+
|
|
287
|
+
A keyword that you do not set takes its value from the engine's settings
|
|
288
|
+
profile. That is libhegel's own default, unless a `hegel.toml` file or a
|
|
289
|
+
`HEGEL_*` environment variable changes it. On a CI server the engine uses its
|
|
290
|
+
`ci` profile, which derandomizes the run and turns the example database off.
|
|
291
|
+
|
|
292
|
+
The example database stores a failing case and replays it first next time.
|
|
293
|
+
`database_key` turns it on, and `database` chooses the directory:
|
|
274
294
|
|
|
275
295
|
```ruby
|
|
276
296
|
Hegel.test(database_key: "my_sort matches the builtin sort") { |tc| ... }
|
|
@@ -278,22 +298,104 @@ Hegel.test(database_key: "my_sort matches the builtin sort") { |tc| ... }
|
|
|
278
298
|
|
|
279
299
|
Give each property its own key. See
|
|
280
300
|
[ADR 0009](docs/adr/0009-turn-the-example-database-on-with-a-key.md) for why
|
|
281
|
-
the key,
|
|
301
|
+
the key, and not the directory, turns it on.
|
|
302
|
+
|
|
303
|
+
Inside a test, `tc.note` records a message that the failure report prints,
|
|
304
|
+
`tc.target` tells Hegel which inputs to search toward, and `tc.repeat` runs a
|
|
305
|
+
block for as many iterations as Hegel chooses.
|
|
306
|
+
|
|
307
|
+
`tc.event` and `tc.event_value` check that a test reaches the cases you care
|
|
308
|
+
about. With `show_statistics: true`, the engine prints how often each event
|
|
309
|
+
happened and a summary of each value. The numbers change from run to run:
|
|
310
|
+
|
|
311
|
+
```ruby
|
|
312
|
+
Hegel.test(show_statistics: true) do |tc|
|
|
313
|
+
xs = tc.draw(arrays(integers))
|
|
314
|
+
tc.event("empty") if xs.empty?
|
|
315
|
+
tc.event_value("size", xs.size)
|
|
316
|
+
end
|
|
317
|
+
```
|
|
318
|
+
|
|
319
|
+
```
|
|
320
|
+
Statistics (over 100 test cases):
|
|
321
|
+
* empty: 6.0% of test cases
|
|
322
|
+
* size: count 100, min 0, median 6, mean 7.99, p90 19, max 20
|
|
323
|
+
```
|
|
324
|
+
|
|
325
|
+
## Profiles
|
|
326
|
+
|
|
327
|
+
A profile is a named set of settings. The engine ships `development`, `ci`,
|
|
328
|
+
and `workload`, and a `hegel.toml` file can define more. Pass `profile:` to
|
|
329
|
+
start one run from a profile, and register your own with
|
|
330
|
+
`Hegel::Settings.register_profile`:
|
|
331
|
+
|
|
332
|
+
```ruby
|
|
333
|
+
Hegel::Settings.register_profile("nightly", profile: "ci", test_cases: 10_000)
|
|
334
|
+
|
|
335
|
+
Hegel.test(profile: "nightly") { |tc| ... }
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
A keyword passed to `Hegel.test` still beats the profile. A profile you
|
|
339
|
+
name extends the default profile, so what it does not set comes from
|
|
340
|
+
there. `Hegel::Settings.set_default_profile("nightly")` makes a profile the
|
|
341
|
+
default for every run in the process, and `set_default_profile("")` clears
|
|
342
|
+
it. Registration and the default are process-wide, and they do not change
|
|
343
|
+
runs that have already started. `register_profile` takes the keywords of
|
|
344
|
+
`Hegel.test` except `database_key`, which a profile does not carry, and
|
|
345
|
+
`Hegel.test` always passes `report_multiple_failures`, so a profile's value
|
|
346
|
+
for it never applies. An unknown name raises `Hegel::Error`. See
|
|
347
|
+
[ADR 0028](docs/adr/0028-expose-settings-profiles-as-a-keyword-and-module-functions.md).
|
|
282
348
|
|
|
283
|
-
|
|
284
|
-
|
|
349
|
+
## Agent skill
|
|
350
|
+
|
|
351
|
+
Two skills work together. The official
|
|
352
|
+
[hegel-skill](https://github.com/hegeldev/hegel-skill) from the Hegel
|
|
353
|
+
maintainers teaches the method: which properties to look for, and how to
|
|
354
|
+
check the tests an agent writes. It works with every Hegel implementation.
|
|
355
|
+
[`skills/hegel-ruby`](skills/hegel-ruby/SKILL.md) adds the Ruby part: this
|
|
356
|
+
gem's API and how to use it with RSpec and Minitest. Install both.
|
|
357
|
+
|
|
358
|
+
In Claude Code, add each repository as a marketplace and install its plugin:
|
|
359
|
+
|
|
360
|
+
```
|
|
361
|
+
/plugin marketplace add hegeldev/hegel-skill
|
|
362
|
+
/plugin install hegel-skill@hegeldev
|
|
363
|
+
/plugin marketplace add meganemura/hegel-ruby
|
|
364
|
+
/plugin install hegel-ruby@meganemura
|
|
365
|
+
```
|
|
366
|
+
|
|
367
|
+
For Codex and other agents, see the installation section of the
|
|
368
|
+
[hegel-skill README](https://github.com/hegeldev/hegel-skill#installation).
|
|
369
|
+
For this gem's skill, use the same steps with
|
|
370
|
+
<https://github.com/meganemura/hegel-ruby/tree/main/skills/hegel-ruby>.
|
|
285
371
|
|
|
286
372
|
## Development
|
|
287
373
|
|
|
288
374
|
```bash
|
|
289
375
|
bin/setup # install dependencies
|
|
290
376
|
bundle exec rake libhegel:fetch # download the pinned engine, checked against its SHA-256
|
|
291
|
-
bundle exec rake # run the tests and the linter
|
|
377
|
+
bundle exec rake # run the tests with coverage, and the linter
|
|
378
|
+
bundle exec rake mutation # run mutation testing
|
|
292
379
|
bin/console # open an interactive prompt
|
|
293
380
|
```
|
|
294
381
|
|
|
295
|
-
`just` recipes call the same Rake tasks, so `just
|
|
296
|
-
for anyone who
|
|
382
|
+
`just` recipes call the same Rake tasks, so `just check`, `just test`, and
|
|
383
|
+
`just lint` work for anyone who uses `just` with the other Hegel
|
|
384
|
+
implementations.
|
|
385
|
+
|
|
386
|
+
CI runs the suite on each pull request and on each push to main, on every
|
|
387
|
+
platform in the table above, with Ruby 3.4 and 4.0.
|
|
388
|
+
|
|
389
|
+
The design records are in [docs/](docs/README.md).
|
|
390
|
+
|
|
391
|
+
### Releasing
|
|
392
|
+
|
|
393
|
+
A merge into main that changes `lib/hegel/version.rb` starts a release. The
|
|
394
|
+
Release workflow waits for an approval of the `release` environment. After
|
|
395
|
+
the approval, it runs the tests, publishes the gems to RubyGems, and then
|
|
396
|
+
creates the tag `v<version>` and the GitHub release. The release notes come
|
|
397
|
+
from the version's section of [CHANGELOG.md](CHANGELOG.md). See
|
|
398
|
+
[ADR 0024](docs/adr/0024-release-when-a-merge-changes-the-gem-version.md).
|
|
297
399
|
|
|
298
400
|
## Contributing
|
|
299
401
|
|
|
@@ -308,3 +410,7 @@ This gem is available under the [MIT License](LICENSE.txt).
|
|
|
308
410
|
Released gems also carry `libhegel`'s own license, which is MIT and copyright
|
|
309
411
|
Antithesis, LLC. Each platform gem ships `NOTICE-libhegel.txt` next to the
|
|
310
412
|
engine, and that file holds the text.
|
|
413
|
+
|
|
414
|
+
---
|
|
415
|
+
|
|
416
|
+
[日本語版 README](README.ja.md)
|
data/Rakefile
CHANGED
|
@@ -18,8 +18,8 @@ end
|
|
|
18
18
|
|
|
19
19
|
task default: %i[coverage standard]
|
|
20
20
|
|
|
21
|
-
# Mutation testing with mutineer, over lib/.
|
|
22
|
-
# run under it; ADR 0016 records
|
|
21
|
+
# Mutation testing with mutineer, over lib/. Five settings make the suite
|
|
22
|
+
# run under it; ADR 0016 records the first four, and ADR 0032 the fifth:
|
|
23
23
|
# - RUBYOPT gets -Itest, since mutineer puts only lib/ on the load path and
|
|
24
24
|
# every test file starts with `require "test_helper"`.
|
|
25
25
|
# - Each test file gets its own --test flag. The flag takes one file, and a
|
|
@@ -31,6 +31,9 @@ task default: %i[coverage standard]
|
|
|
31
31
|
# reload, loads the whole mutated file by a relative path, which breaks
|
|
32
32
|
# Hegel::Runner.origin_for and killed three mutants the suite does not
|
|
33
33
|
# catch.
|
|
34
|
+
# - test_recursive_engine.rb stays out too. Its 30-leaf shrink takes about
|
|
35
|
+
# ten seconds, mutineer's limit for one mutant, so a mutant on a line it
|
|
36
|
+
# covers timed out without a verdict.
|
|
34
37
|
# MUTINEER_ARGS passes more flags, for example
|
|
35
38
|
# MUTINEER_ARGS="--since origin/main" to mutate only the lines changed since
|
|
36
39
|
# origin/main.
|
|
@@ -42,7 +45,7 @@ MUTATION_BASELINE = "test/mutation-baseline.json"
|
|
|
42
45
|
def run_mutineer(*extra)
|
|
43
46
|
require "shellwords"
|
|
44
47
|
tests = Dir["test/*.rb", "test/hegel/*.rb"].sort -
|
|
45
|
-
%w[test/test_helper.rb test/hegel/test_conformance.rb]
|
|
48
|
+
%w[test/test_helper.rb test/hegel/test_conformance.rb test/hegel/test_recursive_engine.rb]
|
|
46
49
|
args = ["lib", "--strategy", "redefine", *tests.flat_map { |t| ["--test", t] }, *extra, *Shellwords.split(ENV.fetch("MUTINEER_ARGS", ""))]
|
|
47
50
|
rubyopt = ["-Itest", ENV["RUBYOPT"]].compact.join(" ")
|
|
48
51
|
# mutineer keeps its cache here, and refuses an --output path whose
|
data/docs/README.md
CHANGED
|
@@ -33,6 +33,15 @@ An index of the design records for `hegel-ruby`.
|
|
|
33
33
|
- [0024: Release when a merge changes the gem version](adr/0024-release-when-a-merge-changes-the-gem-version.md)
|
|
34
34
|
- [0025: Copy each string input into a hegel_string_t](adr/0025-copy-each-string-input-into-a-hegel-string-t.md)
|
|
35
35
|
- [0026: Select the engine release repository by version](adr/0026-select-the-engine-release-repository-by-version.md)
|
|
36
|
+
- [0027: Report each run's test location from the caller's frame](adr/0027-report-each-runs-test-location-from-the-callers-frame.md)
|
|
37
|
+
- [0028: Expose settings profiles as a keyword and module functions](adr/0028-expose-settings-profiles-as-a-keyword-and-module-functions.md)
|
|
38
|
+
- [0029: Keep notes local, and defer run error kinds](adr/0029-keep-notes-local-and-defer-run-error-kinds.md)
|
|
39
|
+
- [0030: Represent a duration as Integer nanoseconds](adr/0030-represent-a-duration-as-integer-nanoseconds.md)
|
|
40
|
+
- [0031: Take the alphabet options from hegel-rust](adr/0031-take-the-alphabet-options-from-hegel-rust.md)
|
|
41
|
+
- [0032: Suppress span closes while an abandoned recursion attempt unwinds](adr/0032-suppress-span-closes-while-an-abandoned-recursion-attempt-unwinds.md)
|
|
42
|
+
- [0033: Cache compiled grammars for the length of a run](adr/0033-cache-compiled-grammars-for-the-length-of-a-run.md)
|
|
43
|
+
- [0034: Repeat a test body with a Ruby-side indent](adr/0034-repeat-a-test-body-with-a-ruby-side-indent.md)
|
|
44
|
+
- [0035: Defer concurrent state machines and the engine printer](adr/0035-defer-concurrent-state-machines-and-the-engine-printer.md)
|
|
36
45
|
|
|
37
46
|
A new decision gets a new record. A changed decision supersedes the old
|
|
38
47
|
record instead of editing it, so the history stays readable.
|
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
# 0027: Report each run's test location from the caller's frame
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted.
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
|
|
9
|
+
`hegel_settings_set_test_location` takes the file, the first line, the
|
|
10
|
+
enclosing class, and the name of the test function. Under Antithesis, which
|
|
11
|
+
the engine detects through `ANTITHESIS_OUTPUT_DIR`, the engine appends one
|
|
12
|
+
`always` assertion named `<class>::<function> passes properties` to
|
|
13
|
+
`sdk.jsonl` in that directory, located at `file:begin_line`. Outside
|
|
14
|
+
Antithesis it uses nothing. hegel-rust fills the four values in
|
|
15
|
+
`#[hegel::test]` from `file!()`, `module_path!()`, the function name, and
|
|
16
|
+
`line!()`, and takes no keyword for them.
|
|
17
|
+
|
|
18
|
+
Ruby has no macro that sees the definition. A `Hegel.test` call can read its
|
|
19
|
+
own caller's frame with `caller_locations`, and a frame label in Ruby 3.4
|
|
20
|
+
reads `Class#method` or `Class.method`, with `block in ` or
|
|
21
|
+
`block (2 levels) in ` in front of it inside a block. A call at the top of a
|
|
22
|
+
file, which is where an RSpec example sits, has the label `<main>` or
|
|
23
|
+
`<top (required)>`, and a frame whose method is defined on an anonymous
|
|
24
|
+
class has the bare method name.
|
|
25
|
+
|
|
26
|
+
## Decision
|
|
27
|
+
|
|
28
|
+
`Hegel.test` builds one `Hegel::TestLocation`, a `Data` of `file`,
|
|
29
|
+
`begin_line`, `class_name`, and `function`, and hands it to `Runner.run`.
|
|
30
|
+
`Runner.run` calls the setter after `Settings.apply` and before it starts
|
|
31
|
+
the run or replays a blob. The location is identity, not a setting, so
|
|
32
|
+
`Settings.apply` does not take it, and `Hegel.test` has no keyword for it.
|
|
33
|
+
|
|
34
|
+
The frame is the first one that `Runner.infrastructure?` does not call
|
|
35
|
+
infrastructure, the same rule that names the origin of a failure (see
|
|
36
|
+
[ADR 0012](0012-build-a-failure-origin-from-the-callers-own-frame.md)). A
|
|
37
|
+
wrapper in this gem, a gem, or the standard library is therefore never named
|
|
38
|
+
as the test.
|
|
39
|
+
|
|
40
|
+
`file` is the frame's absolute path made relative to `Dir.pwd` when it lies
|
|
41
|
+
under it, and absolute otherwise. `begin_line` is the frame's line, the line
|
|
42
|
+
of the `Hegel.test` call.
|
|
43
|
+
|
|
44
|
+
A label that splits as `Class#method` or `Class.method`, after the `block
|
|
45
|
+
... in ` prefix is stripped, gives `class_name` and `function`. Any other
|
|
46
|
+
label has no name of its own, so `class_name` is the `file` without `.rb`
|
|
47
|
+
and `function` is `line <n>`. Two examples in one spec file then report two
|
|
48
|
+
assertions, where one shared name would merge them into one that fails when
|
|
49
|
+
either fails.
|
|
50
|
+
|
|
51
|
+
## Consequences
|
|
52
|
+
|
|
53
|
+
The line is that of the call, where hegel-rust's is that of the function's
|
|
54
|
+
definition, so a method with two `Hegel.test` calls reports two assertions
|
|
55
|
+
with the same name at two lines.
|
|
56
|
+
|
|
57
|
+
A caller cannot choose the name. hegel-rust's macro derives it as well, so a
|
|
58
|
+
keyword would be a change to the shared interface, which starts as a proposal
|
|
59
|
+
upstream (see
|
|
60
|
+
[ADR 0015](0015-follow-the-hegeldev-interface-and-take-changes-upstream-first.md)).
|
|
61
|
+
|
|
62
|
+
The cost is one `caller_locations` call and one regular expression match for
|
|
63
|
+
each `Hegel.test` call, none for each test case.
|
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
# 0028: Expose settings profiles as a keyword and module functions
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted. Builds on [ADR 0023](0023-leave-unset-settings-to-the-engines-profile.md),
|
|
6
|
+
which left the profile to the engine, and applies
|
|
7
|
+
[ADR 0015](0015-follow-the-hegeldev-interface-and-take-changes-upstream-first.md).
|
|
8
|
+
|
|
9
|
+
## Context
|
|
10
|
+
|
|
11
|
+
libhegel 0.47.2 has three calls for profiles. `hegel_settings_new_for_profile`
|
|
12
|
+
builds a settings handle from a named profile. `hegel_settings_register_profile`
|
|
13
|
+
stores a complete snapshot of a handle under a name, for the whole process,
|
|
14
|
+
replacing an earlier registration of that name. `hegel_set_default_profile`
|
|
15
|
+
names the profile that `hegel_settings_new` resolves, for the whole process,
|
|
16
|
+
and an empty name clears it. A named profile still extends the default
|
|
17
|
+
profile, except `base`. An unknown name, a reserved name, a malformed
|
|
18
|
+
`hegel.toml`, and a malformed `HEGEL_*` variable return `HEGEL_E_INVALID_ARG`
|
|
19
|
+
with a message. The snapshot leaves out the database key, and a registration
|
|
20
|
+
does not change handles that already exist.
|
|
21
|
+
|
|
22
|
+
hegel-rust exposes them on its `Settings` value: `Settings::from_profile`
|
|
23
|
+
and `Settings::try_from_profile` return a `Settings` to adjust with builder
|
|
24
|
+
methods, `Settings::register_profile(name, settings)` registers one, and
|
|
25
|
+
`Settings::set_default_profile(name)` sets the default. Its documentation
|
|
26
|
+
example is `Settings::register_profile("nightly",
|
|
27
|
+
Settings::from_profile("ci").test_cases(10_000))`.
|
|
28
|
+
|
|
29
|
+
Ruby has no `Settings` value. `Hegel.test` takes keywords, and
|
|
30
|
+
`Hegel::Settings` is a module of functions that copy those keywords onto a
|
|
31
|
+
native handle that exists only for the length of one run (ADR 0023).
|
|
32
|
+
|
|
33
|
+
## Decision
|
|
34
|
+
|
|
35
|
+
**`Hegel.test` takes `profile:`, which stands in for `from_profile`.** A
|
|
36
|
+
`nil` profile calls `hegel_settings_new`, as before. A String calls
|
|
37
|
+
`hegel_settings_new_for_profile`. The keywords apply afterwards in both
|
|
38
|
+
cases, so one passed explicitly still beats the profile's value, and the
|
|
39
|
+
`reproduce_failure:` path builds its settings the same way. This is a
|
|
40
|
+
difference of mechanism under ADR 0015: the value that hegel-rust builds and
|
|
41
|
+
then adjusts is, here, the argument of the call that uses it.
|
|
42
|
+
|
|
43
|
+
**`Hegel::Settings.register_profile(name, profile: nil, **keywords)`
|
|
44
|
+
registers a profile.** It sits on `Hegel::Settings` because hegel-rust puts
|
|
45
|
+
`register_profile` on `Settings`. It opens a context of its own, builds a
|
|
46
|
+
handle from `hegel_settings_new` or from the named `profile:`, applies the
|
|
47
|
+
keywords, registers the handle, and frees the handle and then the context in
|
|
48
|
+
an `ensure`. It takes an `impl:` keyword, defaulting to `Hegel.default_impl`,
|
|
49
|
+
so a test can pass the Fake the way it does to `Hegel.test`.
|
|
50
|
+
|
|
51
|
+
The keywords are those of `Hegel.test`, each `nil` meaning that no setter
|
|
52
|
+
runs, with these exceptions:
|
|
53
|
+
|
|
54
|
+
- `database_key:` is not accepted. The engine leaves the key out of the
|
|
55
|
+
snapshot, so accepting it would promise something the profile cannot keep.
|
|
56
|
+
Ruby's own `ArgumentError` rejects it, as it does any unknown keyword.
|
|
57
|
+
- `database:` goes straight to `hegel_settings_set_database`, a path or `""`
|
|
58
|
+
to turn the database off. The table of ADR 0009 exists to decide what a
|
|
59
|
+
missing key means for a run, and a profile carries no key.
|
|
60
|
+
- `report_multiple_failures:` is `nil` by default here, where `Hegel.test`
|
|
61
|
+
defaults it to `false`.
|
|
62
|
+
|
|
63
|
+
Starting from `hegel_settings_new` matches `Settings::new()`, which resolves
|
|
64
|
+
the default profile. hegel-rust's documented example starts from another
|
|
65
|
+
profile instead, and `profile: "ci"` is the same thing here.
|
|
66
|
+
|
|
67
|
+
**`Hegel::Settings.set_default_profile(name)` sets the default profile,**
|
|
68
|
+
with the same `impl:` keyword. `""` clears it.
|
|
69
|
+
|
|
70
|
+
An error from the engine raises `Hegel::Error` with the engine's message,
|
|
71
|
+
through `LibHegel.check!`. hegel-rust panics with the same message.
|
|
72
|
+
|
|
73
|
+
## Consequences
|
|
74
|
+
|
|
75
|
+
`Hegel.test` always passes `report_multiple_failures`, as ADR 0023 decided,
|
|
76
|
+
so a profile's value for it is overridden on every run. A profile can carry
|
|
77
|
+
the setting, but it never changes what `Hegel.test` does.
|
|
78
|
+
|
|
79
|
+
Registrations and the default profile are state of the process, held by the
|
|
80
|
+
engine and not by any context. A test that registers a profile leaves it
|
|
81
|
+
registered, and the engine has no call to remove one. A test that sets the
|
|
82
|
+
default profile clears it again with `set_default_profile("")`. Neither
|
|
83
|
+
affects a run that is already in progress.
|
|
84
|
+
|
|
85
|
+
A `HEGEL_*` variable applies over a profile's snapshot, as the header says
|
|
86
|
+
for every handle, so `HEGEL_PRINT_BLOB` beats a `print_blob: false` that the
|
|
87
|
+
profile registered. The tests that observe a profile through the failure
|
|
88
|
+
report clear those variables first.
|
|
89
|
+
|
|
90
|
+
Each call to `register_profile` and `set_default_profile` opens and frees a
|
|
91
|
+
context. Both are meant for setup, run once, not once per test case.
|