hegeltest 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +22 -0
- data/CODE_OF_CONDUCT.md +10 -0
- data/LICENSE.txt +21 -0
- data/README.md +264 -0
- data/Rakefile +19 -0
- data/docs/README.md +25 -0
- data/docs/adr/0001-bind-libhegel-through-fiddle.md +54 -0
- data/docs/adr/0002-ship-one-prebuilt-engine-per-platform-specific-gem.md +48 -0
- data/docs/adr/0003-publish-as-hegeltest-require-as-hegel.md +39 -0
- data/docs/adr/0004-expose-generators-through-a-mixin-with-keyword-options.md +42 -0
- data/docs/adr/0005-name-drawn-values-from-the-callers-source-with-prism.md +40 -0
- data/docs/adr/0006-verify-the-binding-in-seven-layers-with-full-coverage.md +51 -0
- data/docs/adr/0007-ship-a-thin-ruby-skill-shaped-for-donation.md +56 -0
- data/docs/adr/0008-revisit-the-binding-after-milestone-c-on-measurement.md +81 -0
- data/docs/adr/0009-turn-the-example-database-on-with-a-key.md +89 -0
- data/docs/adr/0010-declare-stateful-rules-with-a-class-macro.md +113 -0
- data/docs/adr/0011-let-the-test-case-own-every-pool-drawn-from-it.md +83 -0
- data/docs/adr/0012-build-a-failure-origin-from-the-callers-own-frame.md +72 -0
- data/docs/adr/0013-bind-libhegel-through-the-ffi-gem.md +102 -0
- data/docs/architecture.md +182 -0
- data/lib/hegel/draw_name.rb +109 -0
- data/lib/hegel/errors.rb +47 -0
- data/lib/hegel/generator.rb +98 -0
- data/lib/hegel/generators.rb +865 -0
- data/lib/hegel/lib_hegel/real.rb +1149 -0
- data/lib/hegel/lib_hegel.rb +269 -0
- data/lib/hegel/libhegel_version.rb +9 -0
- data/lib/hegel/locate.rb +188 -0
- data/lib/hegel/report.rb +87 -0
- data/lib/hegel/runner.rb +464 -0
- data/lib/hegel/settings.rb +164 -0
- data/lib/hegel/state_machine.rb +89 -0
- data/lib/hegel/stateful/pool.rb +111 -0
- data/lib/hegel/stateful.rb +120 -0
- data/lib/hegel/syntax/methods.rb +173 -0
- data/lib/hegel/test_case.rb +523 -0
- data/lib/hegel/version.rb +5 -0
- data/lib/hegel.rb +92 -0
- data/lib/hegeltest.rb +7 -0
- data/lib/tasks/libhegel.rake +112 -0
- data/lib/tasks/platform_gems.rake +111 -0
- data/sig/hegel.rbs +563 -0
- data/skills/hegel-ruby/SKILL.md +30 -0
- data/skills/hegel-ruby/references/ruby/reference.md +1210 -0
- metadata +113 -0
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
# 0007: Ship a thin Ruby skill, shaped for donation
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted. The content arrives once milestone A lands a user-facing API.
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
|
|
9
|
+
The Hegel maintainers publish [hegel-skill](https://github.com/hegeldev/hegel-skill),
|
|
10
|
+
a Claude Code plugin that teaches agents to write property-based tests. Its
|
|
11
|
+
`skills/hegel/SKILL.md` carries language-agnostic methodology: how to find a
|
|
12
|
+
property worth testing, how to keep a generator from being over-constrained,
|
|
13
|
+
and a catalogue of property patterns. Step 1 of its workflow reads "load the
|
|
14
|
+
corresponding reference from `references/<language>/reference.md`".
|
|
15
|
+
|
|
16
|
+
That directory holds one subdirectory per language, and the front matter names
|
|
17
|
+
the set: Rust, Go, C++, TypeScript, Java, and OCaml. Each language contributes
|
|
18
|
+
two files. `reference.md` documents that language's API: setup, test
|
|
19
|
+
structure, settings, `TestCase` methods, every generator, the combinator
|
|
20
|
+
methods, composite generators, and gotchas. `porting.md` maps another
|
|
21
|
+
property-testing library onto Hegel; the TypeScript one covers fast-check,
|
|
22
|
+
jsverify, and testcheck-js.
|
|
23
|
+
|
|
24
|
+
Ruby has its own property-testing libraries. By RubyGems download count they
|
|
25
|
+
run rantly, prop_check, rubycheck, pbt, and propr.
|
|
26
|
+
|
|
27
|
+
Two shapes were available. A self-contained skill would copy hegel-skill's
|
|
28
|
+
methodology, which is MIT licensed, so that one install covers everything. A
|
|
29
|
+
thin skill carries only the Ruby half.
|
|
30
|
+
|
|
31
|
+
## Decision
|
|
32
|
+
|
|
33
|
+
Publish `skills/hegel-ruby/` holding a thin `SKILL.md` alongside
|
|
34
|
+
`references/ruby/reference.md` and `references/ruby/porting.md`, written in
|
|
35
|
+
the same shape hegel-skill uses for its six languages. The methodology stays
|
|
36
|
+
where its authors maintain it.
|
|
37
|
+
|
|
38
|
+
Write the content when milestone A lands `Hegel.test` and the first
|
|
39
|
+
generators, and grow it as each later generator arrives. A reference that
|
|
40
|
+
documents an API nobody has built yet records a guess.
|
|
41
|
+
|
|
42
|
+
Distribute it two ways. The gem already packages `skills/`, the way the
|
|
43
|
+
rubydex gem packages its own. A `.claude-plugin/` manifest makes the same
|
|
44
|
+
directory installable through the Claude Code marketplace.
|
|
45
|
+
|
|
46
|
+
## Consequences
|
|
47
|
+
|
|
48
|
+
Somebody who installs this skill alone gets the Ruby API and reaches
|
|
49
|
+
hegel-skill for the methodology. The `SKILL.md` says so.
|
|
50
|
+
|
|
51
|
+
The two reference files sit in the layout hegel-skill expects, so they can
|
|
52
|
+
move upstream unchanged if the Hegel maintainers ever want a Ruby entry.
|
|
53
|
+
|
|
54
|
+
A generator is finished when its entry in `reference.md` exists, which puts
|
|
55
|
+
the documentation next to the code that motivates it rather than at the end
|
|
56
|
+
of the milestone.
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
# 0008: Revisit the binding after milestone C, on measurement
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted, and carried out.
|
|
6
|
+
[ADR 0013](0013-bind-libhegel-through-the-ffi-gem.md)
|
|
7
|
+
records the measurement this scheduled and the decision it produced.
|
|
8
|
+
|
|
9
|
+
## Context
|
|
10
|
+
|
|
11
|
+
[ADR 0001](0001-bind-libhegel-through-fiddle.md) chose Fiddle over the `ffi`
|
|
12
|
+
gem, on the reading that Fiddle adds no third-party runtime dependency while
|
|
13
|
+
both avoid a compiler at install time. Building against it has since produced
|
|
14
|
+
evidence that decision could not have had.
|
|
15
|
+
|
|
16
|
+
**Fiddle has no struct-by-value argument.** `Fiddle::Function` converts each
|
|
17
|
+
argument type with `NUM2INT`, so a struct type cannot be expressed. Three ABI
|
|
18
|
+
draws take their bounds that way. They are reached by packing an eight-byte
|
|
19
|
+
all-integer struct into the register the ABI would have used for it, which is
|
|
20
|
+
sound for those three and is not a general technique: it puts the ABI
|
|
21
|
+
classification in this gem's own code, duplicates field offsets by hand,
|
|
22
|
+
cannot follow a struct holding a float, cannot express a struct return, and
|
|
23
|
+
diverges on Win64 for the sixteen-byte case. `ffi` has `FFI::Struct.by_value`
|
|
24
|
+
and lets libffi classify.
|
|
25
|
+
|
|
26
|
+
**Fiddle::Closure is still unverified here.** The engine's output callback is
|
|
27
|
+
passed as NULL, which the ABI documents as leaving output on stderr, so
|
|
28
|
+
nothing depends on the closure yet. Wiring the callback would put weight on a
|
|
29
|
+
Fiddle feature this project has never exercised across the fiddle versions
|
|
30
|
+
Ruby 3.3 through 4.0 ship.
|
|
31
|
+
|
|
32
|
+
**Ruby implementations.** `ffi` runs on JRuby and TruffleRuby; the Fiddle
|
|
33
|
+
choice scoped this gem to CRuby.
|
|
34
|
+
|
|
35
|
+
Against that: `ffi` is a runtime dependency, and this project's dependency
|
|
36
|
+
rule asks for a reason and an explicit decision before one is added.
|
|
37
|
+
|
|
38
|
+
None of the above is about speed, and speed is what a property-based testing
|
|
39
|
+
library spends. A run configured for twenty test cases was measured making
|
|
40
|
+
over a thousand iterations while shrinking, each with its own draws, so the
|
|
41
|
+
per-call cost of the binding is multiplied by a large and unpredictable
|
|
42
|
+
factor.
|
|
43
|
+
|
|
44
|
+
## Decision
|
|
45
|
+
|
|
46
|
+
Measure the two bindings against each other once milestone C is complete, and
|
|
47
|
+
choose on the result.
|
|
48
|
+
|
|
49
|
+
Measure a realistic property run, not a microbenchmark: a failing property
|
|
50
|
+
over `arrays(integers)` that shrinks, which is the shape that multiplies
|
|
51
|
+
per-call cost, plus a passing run of the same size for the no-shrink case.
|
|
52
|
+
Report wall-clock for each binding on the same machine, same engine build,
|
|
53
|
+
same seed, with the database disabled.
|
|
54
|
+
|
|
55
|
+
Library load time is measured too and reported separately. It happens once
|
|
56
|
+
per process and should not be added to the per-call figure.
|
|
57
|
+
|
|
58
|
+
A difference that shows on that run decides it. A difference visible only in
|
|
59
|
+
a tight loop around one native call does not: this gem's callers pay the
|
|
60
|
+
former and never the latter.
|
|
61
|
+
|
|
62
|
+
Milestone C is the trigger because it adds the last of the binding surface:
|
|
63
|
+
the example database, targeted testing, stateful testing, and pools. A
|
|
64
|
+
measurement taken before it would cover a fraction of the calls, and a
|
|
65
|
+
rewrite after it would be smaller than one before.
|
|
66
|
+
|
|
67
|
+
## Consequences
|
|
68
|
+
|
|
69
|
+
The binding stays Fiddle through milestones B and C, and the packing
|
|
70
|
+
technique carries the three struct-taking draws until then.
|
|
71
|
+
|
|
72
|
+
A switch stays cheap because of the seam: `Hegel::LibHegel::Real` is the only
|
|
73
|
+
file that names Fiddle, and a conformance test already holds both it and the
|
|
74
|
+
Fake to one method list. The cost is confined to that file, the gemspec,
|
|
75
|
+
and this record's supersession. The twenty-odd files built on top of the
|
|
76
|
+
seam do not need to change.
|
|
77
|
+
|
|
78
|
+
If the measurement favours `ffi`, adding it is still a dependency decision
|
|
79
|
+
under this project's own rule, and the reason recorded here is the argument
|
|
80
|
+
that decision would weigh: not `ffi` alone, but `ffi` together with
|
|
81
|
+
struct-by-value, an exercised closure, and JRuby and TruffleRuby.
|
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
# 0009: Turn the example database on with a key
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted. It replaces the interim rule that every run disables the example
|
|
6
|
+
database, which stood while milestone C was still deciding this.
|
|
7
|
+
|
|
8
|
+
## Context
|
|
9
|
+
|
|
10
|
+
libhegel can store a failing test case and replay it first on the next run.
|
|
11
|
+
Three settings govern it: `hegel_settings_set_database` chooses a directory
|
|
12
|
+
(NULL leaves the default `./.hegel/examples/`, `""` disables storage
|
|
13
|
+
entirely), `hegel_settings_set_database_key` scopes what a run stores and
|
|
14
|
+
replays, and the engine turns the whole feature off by itself under CI.
|
|
15
|
+
|
|
16
|
+
Until now every `Hegel.test` call passed `""`, because a run that left the
|
|
17
|
+
default in place looked like it would write into a contributor's working
|
|
18
|
+
copy and nowhere else. That is the hardest version of that mistake to
|
|
19
|
+
notice.
|
|
20
|
+
|
|
21
|
+
Measurement against 0.32.5 shows the hazard is narrower than that. A run
|
|
22
|
+
with the path left at its default and **no key set** wrote nothing at all:
|
|
23
|
+
no `./.hegel/`, under that name or any other, in a fresh empty directory,
|
|
24
|
+
across two runs of a property that did find and report a failure. The same
|
|
25
|
+
run with `hegel_settings_set_database_key(ctx, s, "probe")` produced
|
|
26
|
+
`.hegel/examples/` with three hash directories and twenty-nine files, and a
|
|
27
|
+
second run in that directory replayed the stored failure as its first and
|
|
28
|
+
only test case. A third run with a different key behaved like a fresh
|
|
29
|
+
search.
|
|
30
|
+
|
|
31
|
+
So the key, not the path, is what makes the database do anything.
|
|
32
|
+
`hegel_settings_set_database_key`'s own header entry describes NULL as
|
|
33
|
+
clearing the key. What a cleared key does to storage is what the measurement
|
|
34
|
+
above supplies.
|
|
35
|
+
|
|
36
|
+
The two implementations that ship this feature both leave the path at the
|
|
37
|
+
engine default and get a key from a name they already have: hegel-go passes
|
|
38
|
+
Go's own `t.Name()`, and hegel-java passes the `name` its `Settings` carries.
|
|
39
|
+
`Hegel.test` has no such name. It is a bare method call, not a named test,
|
|
40
|
+
so Ruby has nothing to key a run by unless the caller supplies one.
|
|
41
|
+
|
|
42
|
+
## Decision
|
|
43
|
+
|
|
44
|
+
`Hegel.test` gains two keywords.
|
|
45
|
+
|
|
46
|
+
`database_key:` is the switch. Given a String, the run stores and replays
|
|
47
|
+
examples under that key. Left nil, which is the default, the run passes `""`
|
|
48
|
+
to `hegel_settings_set_database` and stores nothing.
|
|
49
|
+
|
|
50
|
+
`database:` chooses the directory, and only means something alongside a key.
|
|
51
|
+
Given a String, the run uses that directory. Left nil, a keyed run leaves the
|
|
52
|
+
engine's own default in place, which is `./.hegel/examples/` outside CI and
|
|
53
|
+
disabled inside it. Given without `database_key:`, it raises `Hegel::Error`
|
|
54
|
+
rather than silently storing nothing.
|
|
55
|
+
|
|
56
|
+
Disabling explicitly rather than relying on the measurement above is
|
|
57
|
+
deliberate. "No key means nothing is written" is behaviour this project
|
|
58
|
+
measured, not behaviour the ABI promises, and the cost of being wrong is
|
|
59
|
+
directories appearing in working copies that never asked for them. Passing
|
|
60
|
+
`""` costs one call per run and depends on a documented sentence instead.
|
|
61
|
+
|
|
62
|
+
CI detection stays the engine's. hegel-go and hegel-java each run their own
|
|
63
|
+
check of the same environment variables; here, a run reaching the engine
|
|
64
|
+
default in CI is already disabled by the engine, and a run passing an
|
|
65
|
+
explicit path means the caller asked for that path.
|
|
66
|
+
|
|
67
|
+
## Consequences
|
|
68
|
+
|
|
69
|
+
An unkeyed run behaves exactly as every run behaved before this record, so
|
|
70
|
+
the change is invisible to a caller who does not ask for it.
|
|
71
|
+
|
|
72
|
+
A caller who wants replay writes the key themselves:
|
|
73
|
+
|
|
74
|
+
```ruby
|
|
75
|
+
Hegel.test(database_key: "sorting is idempotent") { |tc| ... }
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
That is more typing than hegel-go or hegel-java need, and the difference is
|
|
79
|
+
the name those two already have. A future integration that knows the
|
|
80
|
+
enclosing test's name, such as a Minitest or RSpec adapter, can supply the
|
|
81
|
+
key from it and close the gap. Nothing here forecloses that, because the
|
|
82
|
+
keyword takes any String.
|
|
83
|
+
|
|
84
|
+
Keying by hand also means two properties can collide by being given the same
|
|
85
|
+
key, which a name derived from a test could not do. The header describes the
|
|
86
|
+
key as what scopes stored and replayed examples, so two properties sharing a
|
|
87
|
+
key share one scope; what the engine then does with a case stored by one and
|
|
88
|
+
replayed against the other has not been measured here. The keyword's
|
|
89
|
+
documentation says to make the key unique, and this is the reason.
|
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
# 0010: Declare stateful rules with a class macro
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted.
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
|
|
9
|
+
A stateful test hands libhegel a list of rule names, then runs whichever rule
|
|
10
|
+
the engine picks, step after step, until the engine says the test case's step
|
|
11
|
+
budget is spent. The engine owns the choice and the shrinking. What a binding
|
|
12
|
+
decides is how its own users write the rules.
|
|
13
|
+
|
|
14
|
+
The four implementations that ship this each answered in their own language's
|
|
15
|
+
terms. hegel-rust's `StateMachine` trait returns `Vec<Rule<Self>>` from
|
|
16
|
+
`rules()`, and hegel-cpp does the same from a virtual `rules()`. hegel-go
|
|
17
|
+
finds methods by reflection, taking every method whose name begins with `Rule`
|
|
18
|
+
or `Invariant`. hegel-ocaml takes rules as a list argument to `Stateful.run`,
|
|
19
|
+
each built by `Rule.create ~name ~step`. All four then run the machine from
|
|
20
|
+
inside an ordinary test body rather than from a separate entry point.
|
|
21
|
+
|
|
22
|
+
Ruby has two idioms that fit, and they point at different answers. Minitest
|
|
23
|
+
finds tests by their `test_` prefix, which is hegel-go's shape. RSpec, Rake,
|
|
24
|
+
and ActiveRecord register work through class-level macros taking blocks, which
|
|
25
|
+
is closer to hegel-rust's explicit list. Being Ruby-like does not decide it.
|
|
26
|
+
|
|
27
|
+
What decides it is what happens to a mistake. Under prefix discovery, a rule
|
|
28
|
+
written `rules_push` instead of `rule_push` is not a rule. Nothing raises, the
|
|
29
|
+
machine runs with one fewer action, and the test passes: the suite reports
|
|
30
|
+
that a behaviour holds when the behaviour was never exercised. hegel-go keeps
|
|
31
|
+
that from happening by rejecting a method that takes a `TestCase` without the
|
|
32
|
+
prefix. In Ruby the same guard means reading parameter lists to infer intent,
|
|
33
|
+
where a declared name raises on the spot.
|
|
34
|
+
|
|
35
|
+
Writing all three shapes out inside a real Minitest file surfaced a second
|
|
36
|
+
difference that the shapes alone did not. Rules want their framework's
|
|
37
|
+
assertions. A machine written as its own class reaches neither the generator
|
|
38
|
+
methods nor `assert_equal` without saying so, because `Hegel::Syntax::Methods`
|
|
39
|
+
was included into `Minitest::Test` and the machine is not one.
|
|
40
|
+
|
|
41
|
+
## Decision
|
|
42
|
+
|
|
43
|
+
A machine is a class that inherits `Hegel::StateMachine` and declares its
|
|
44
|
+
rules and invariants with class-level macros:
|
|
45
|
+
|
|
46
|
+
```ruby
|
|
47
|
+
class StackMachine < Hegel::StateMachine
|
|
48
|
+
def initialize
|
|
49
|
+
@stack = BoundedStack.new(3)
|
|
50
|
+
@model = []
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
rule :push do |tc|
|
|
54
|
+
x = tc.draw(integers(min_value: 0, max_value: 9))
|
|
55
|
+
@stack.push(x)
|
|
56
|
+
@model.push(x) if @model.size < 3
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
rule :pop do |tc|
|
|
60
|
+
tc.assume(!@model.empty?)
|
|
61
|
+
raise "pop disagreed" unless @stack.pop == @model.pop
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
invariant :size_agrees do
|
|
65
|
+
raise "size disagreed" unless @stack.size == @model.size
|
|
66
|
+
end
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
Hegel.test { |tc| Hegel::Stateful.run(StackMachine.new, tc) }
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Each block runs against the machine instance, so a rule reads and writes
|
|
73
|
+
instance variables like any method. Both macros pass the test case to the
|
|
74
|
+
block; an invariant that does not want it declares no parameter, since a Ruby
|
|
75
|
+
block ignores arguments it does not name.
|
|
76
|
+
|
|
77
|
+
`Hegel::StateMachine` includes `Hegel::Syntax::Methods`, so a generator is
|
|
78
|
+
callable inside a rule without the machine repeating the include its test
|
|
79
|
+
class already has.
|
|
80
|
+
|
|
81
|
+
Declaring the same name twice in one class raises `Hegel::Error`. A subclass
|
|
82
|
+
declaring an inherited name replaces it, the way a redefined method does.
|
|
83
|
+
Silently keeping the last of two same-named rules would reintroduce exactly
|
|
84
|
+
the disappearing-rule failure this record exists to avoid, while a subclass
|
|
85
|
+
overriding one is the ordinary reason to write the name again.
|
|
86
|
+
|
|
87
|
+
`Hegel::Stateful.run(machine, tc)` runs it from inside an ordinary `Hegel.test`
|
|
88
|
+
body, as all four sibling implementations do. Nothing about a stateful test
|
|
89
|
+
takes over the run, so a body can draw values before or after running a
|
|
90
|
+
machine.
|
|
91
|
+
|
|
92
|
+
## Consequences
|
|
93
|
+
|
|
94
|
+
The typo that a Ruby prefix convention would swallow now raises. `rule :psuh`
|
|
95
|
+
registers a rule named `psuh`, the engine picks it, and the call fails on
|
|
96
|
+
whatever the block does. A `rule_` prefix over Ruby methods has no such
|
|
97
|
+
moment, because the misspelled method is simply not a rule. hegel-go closes
|
|
98
|
+
that gap in its own language, by rejecting a method that takes a `TestCase`
|
|
99
|
+
without the prefix.
|
|
100
|
+
|
|
101
|
+
The machine still has to reach its framework's assertions itself, with
|
|
102
|
+
`include Minitest::Assertions` or the RSpec equivalent, and that boilerplate
|
|
103
|
+
is real. This library does not include either one for a caller: it takes no
|
|
104
|
+
test framework as a dependency, and guessing which one is loaded would make
|
|
105
|
+
the base class behave differently depending on load order. A rule can also
|
|
106
|
+
just `raise`, which needs nothing, and the examples here do.
|
|
107
|
+
|
|
108
|
+
Rules registered on a class, rather than found on an instance, means a
|
|
109
|
+
machine built some other way has no supported path. That other way could
|
|
110
|
+
be rules assembled at run time, or a machine that is not a class. Nothing
|
|
111
|
+
here forecloses adding one; the engine only ever receives a list of names
|
|
112
|
+
and a way to invoke each, which a later record can decide how else to
|
|
113
|
+
supply.
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
# 0011: Let the test case own every pool drawn from it
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted. It builds on
|
|
6
|
+
[ADR 0010](0010-declare-stateful-rules-with-a-class-macro.md), which decided
|
|
7
|
+
how a state machine declares its rules.
|
|
8
|
+
|
|
9
|
+
## Context
|
|
10
|
+
|
|
11
|
+
A pool lets one rule act on a value an earlier rule produced. Without it, a
|
|
12
|
+
rule that frees a resource can only draw an arbitrary integer, and almost none
|
|
13
|
+
of those name a resource anything created. The engine tracks the pool as a set
|
|
14
|
+
of variable ids, chooses which id a draw returns, and shrinks that choice; the
|
|
15
|
+
caller keeps its own map from id to the value it generated.
|
|
16
|
+
|
|
17
|
+
`hegel_new_pool` hands back a handle that `hegel_pool_free` releases, and, like
|
|
18
|
+
every other free in this ABI, that call takes the context. So the context has
|
|
19
|
+
to outlive the pool. This project already answered the general form of that
|
|
20
|
+
question: release handles in an `ensure` in the code that owns the run, because
|
|
21
|
+
Ruby's garbage collector gives finalizer ordering no guarantee worth relying
|
|
22
|
+
on.
|
|
23
|
+
|
|
24
|
+
The difficulty is that a pool is created by the caller's own code, inside a
|
|
25
|
+
rule, and that code has no `ensure` wrapped around the run. Whoever writes
|
|
26
|
+
`Pool.new` is not in a position to free it.
|
|
27
|
+
|
|
28
|
+
hegel-ocaml, the other binding whose language collects garbage on its own
|
|
29
|
+
schedule, answers this by registering each pool on the test case that created
|
|
30
|
+
it and freeing them all when that case completes (`lib/internal.ml`,
|
|
31
|
+
`new_pool` and `free_owned_pools`). That is the same shape this library already
|
|
32
|
+
uses for the test-case handle itself.
|
|
33
|
+
|
|
34
|
+
## Decision
|
|
35
|
+
|
|
36
|
+
`Hegel::Stateful::Pool.new(tc)` creates a pool. The `Hegel::TestCase` records
|
|
37
|
+
every pool created from it, and `Hegel::Runner` frees them in the same `ensure`
|
|
38
|
+
that already frees the test-case handle, before that handle goes.
|
|
39
|
+
|
|
40
|
+
A finalizer backs nothing up here. The standing constraint allows one as a
|
|
41
|
+
backstop against a leak, but it must do nothing when the handle is already
|
|
42
|
+
free, and the three places that free a test case all run their `ensure` on
|
|
43
|
+
every path including a fatal exception. Adding a finalizer would add a second
|
|
44
|
+
owner for no case the first one misses.
|
|
45
|
+
|
|
46
|
+
A pool draws from the test case, so it must be built during one. The natural
|
|
47
|
+
place is the machine's own constructor, with the test case passed in from the
|
|
48
|
+
block that has it:
|
|
49
|
+
|
|
50
|
+
```ruby
|
|
51
|
+
Hegel.test { |tc| Hegel::Stateful.run(ResourceMachine.new(tc), tc) }
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
`Pool#add` records a value against a fresh variable id. Drawing goes through
|
|
55
|
+
two generators, not through the pool directly: `values_reusable` yields a
|
|
56
|
+
value and leaves it in the pool, `values_consumed` removes the value it
|
|
57
|
+
yields. Both are drawn with `tc.draw`, so the chosen value is named in the
|
|
58
|
+
failure report and its choice shrinks like any other draw. hegel-rust says
|
|
59
|
+
the same of its own two in `src/stateful.rs`, and this record follows
|
|
60
|
+
hegel-rust on meaning. hegel-ocaml builds its two the same way, as generators
|
|
61
|
+
(`Make_pool.pool_values` in `lib/generators_core.ml`). Those generators carry
|
|
62
|
+
no printer, so the choice shrinks like any other draw and the report leaves it
|
|
63
|
+
unnamed.
|
|
64
|
+
|
|
65
|
+
Drawing from an empty pool raises `Hegel::AssumeFailed`, translated from the
|
|
66
|
+
engine's own `HEGEL_E_ASSUME` by the existing result-code check. Inside a rule
|
|
67
|
+
that rejects the rule and the loop draws another, which is exactly what a rule
|
|
68
|
+
that frees a resource should do when nothing has been allocated yet.
|
|
69
|
+
|
|
70
|
+
## Consequences
|
|
71
|
+
|
|
72
|
+
A caller never frees a pool, and cannot leak one by forgetting to.
|
|
73
|
+
|
|
74
|
+
A caller also cannot outlive a pool past its test case on purpose. Holding one
|
|
75
|
+
in an instance variable and reusing it on the next case would reach a freed
|
|
76
|
+
handle; the machine is rebuilt per case for the same reason the model state is,
|
|
77
|
+
so the ordinary way of writing one does not run into this.
|
|
78
|
+
|
|
79
|
+
The engine records a pool draw as the chosen variable id rather than as an
|
|
80
|
+
index into the pool's current contents, so shrinking away an earlier `add`
|
|
81
|
+
never changes which variable a recorded choice refers to. The Ruby-side map
|
|
82
|
+
follows from that for free: an `add` that shrinks away never runs, so its id is
|
|
83
|
+
never in the map, and no draw can return it.
|
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
# 0012: Build a failure origin from the caller's own frame
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted.
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
|
|
9
|
+
libhegel groups failures by the origin string a caller passes to
|
|
10
|
+
`hegel_mark_complete`. The header is explicit: two failures with identical
|
|
11
|
+
origins are the same bug and get shrunk together, and each new origin is a new
|
|
12
|
+
bug. So the origin decides both how many failures a run reports and which
|
|
13
|
+
failures the shrinker minimises as one.
|
|
14
|
+
|
|
15
|
+
`Hegel::Runner.origin_for` built that string from the exception's first
|
|
16
|
+
backtrace frame. For a `raise` written in the caller's own test body, that is
|
|
17
|
+
the right line. For an assertion, it is not, because an assertion library
|
|
18
|
+
raises from inside itself. Measured against minitest 5.27.0 and
|
|
19
|
+
rspec-expectations 3.13.5:
|
|
20
|
+
|
|
21
|
+
```
|
|
22
|
+
assert_equal 0, 1 -> Raised at .../minitest-5.27.0/lib/minitest/assertions.rb:176
|
|
23
|
+
expect(1).to eq(0) -> Raised at .../rspec-support-3.13.7/lib/rspec/support.rb:110
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
Those two lines do not vary. Every failing assertion in a suite produced the
|
|
27
|
+
same origin as every other one, whichever property raised it and whichever
|
|
28
|
+
line the caller wrote. libhegel therefore saw one bug where there were
|
|
29
|
+
several, and shrank them together.
|
|
30
|
+
|
|
31
|
+
This gem exists to be driven from RSpec and Minitest, so that is not an edge
|
|
32
|
+
case; it is the ordinary path.
|
|
33
|
+
|
|
34
|
+
hegel-rust answers this at compile time: Rust's assertion macros expand at the
|
|
35
|
+
call site, so a panic already carries the caller's own location. hegel-java
|
|
36
|
+
meets the same problem this gem meets, and answers it in
|
|
37
|
+
`Runner.originOf` by walking the stack for the first frame that is not
|
|
38
|
+
infrastructure, where infrastructure is a list of class-name prefixes: its
|
|
39
|
+
own package, JUnit, opentest4j, and the JDK.
|
|
40
|
+
|
|
41
|
+
## Decision
|
|
42
|
+
|
|
43
|
+
Build the origin from the first backtrace frame that belongs to neither this
|
|
44
|
+
library, nor an installed gem, nor Ruby's own standard library. Keep the
|
|
45
|
+
exception's class out of it, as before.
|
|
46
|
+
|
|
47
|
+
Identifying infrastructure by location rather than by name is where this
|
|
48
|
+
departs from hegel-java. A name list needs an entry per framework, and this
|
|
49
|
+
gem is driven from frameworks it does not enumerate. Ruby offers location as
|
|
50
|
+
a handle that a name list does not need: Bundler installs a framework under a
|
|
51
|
+
gem directory and loads a caller's own `path:` or `git:` code from the working
|
|
52
|
+
copy, so one rule reaches every framework.
|
|
53
|
+
|
|
54
|
+
When every frame is filtered out, the first frame is used after all. An origin
|
|
55
|
+
that is merely coarse still groups consistently, where no origin at all would
|
|
56
|
+
lose the failure's identity entirely.
|
|
57
|
+
|
|
58
|
+
## Consequences
|
|
59
|
+
|
|
60
|
+
Two failures raised from one line in a caller's own code stay one bug. That is
|
|
61
|
+
the intended behaviour. A ternary that raises two different messages is one
|
|
62
|
+
line and one origin, and splitting it across an `if`/`else` is what makes it
|
|
63
|
+
two.
|
|
64
|
+
|
|
65
|
+
A caller whose code under test is itself an installed gem now gets their own
|
|
66
|
+
test line as the origin, rather than the line inside that gem. Two distinct
|
|
67
|
+
bugs in one installed dependency therefore group as one if the caller reaches
|
|
68
|
+
them from a single line. Distinguishing them is the caller's own to do, by
|
|
69
|
+
calling from separate lines, and the shrinker still reports the failure.
|
|
70
|
+
|
|
71
|
+
The origin string holds an absolute path, so it differs between machines. That
|
|
72
|
+
costs nothing: libhegel compares origins only within a single run.
|
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
# 0013: Bind libhegel through the ffi gem
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Accepted. Supersedes
|
|
6
|
+
[ADR 0001](0001-bind-libhegel-through-fiddle.md), and carries out the
|
|
7
|
+
measurement [ADR 0008](0008-revisit-the-binding-after-milestone-c-on-measurement.md)
|
|
8
|
+
scheduled for the end of milestone C.
|
|
9
|
+
|
|
10
|
+
## Context
|
|
11
|
+
|
|
12
|
+
ADR 0001 chose Fiddle over the `ffi` gem, on two grounds: Fiddle ships with
|
|
13
|
+
Ruby, so it adds no third-party runtime dependency, and neither one needs a
|
|
14
|
+
compiler at install time. ADR 0008 then recorded three things that building
|
|
15
|
+
against Fiddle had since shown: no struct-by-value argument, an unexercised
|
|
16
|
+
`Fiddle::Closure`, and no JRuby or TruffleRuby. It scheduled a measurement
|
|
17
|
+
rather than deciding on them, because none of those is about speed, and speed
|
|
18
|
+
is what a property-based testing library spends.
|
|
19
|
+
|
|
20
|
+
The whole binding surface exists now, so the measurement ran. An `ffi`
|
|
21
|
+
implementation of `Hegel::LibHegel` was written for the calls a failing
|
|
22
|
+
`arrays(integers)` property reaches, and checked against the Fiddle one before
|
|
23
|
+
anything was timed: same property, same seed, both bindings. The body call
|
|
24
|
+
count, every drawn value, the raised message, the failure report, and the
|
|
25
|
+
reproduction blob all matched. The two answer identically, so their times are
|
|
26
|
+
comparable.
|
|
27
|
+
|
|
28
|
+
Measured on Ruby 4.0.6, arm64-darwin, libhegel 0.32.5, `test_cases: 500`, same
|
|
29
|
+
seed, twelve reps, minimum of each:
|
|
30
|
+
|
|
31
|
+
| property | fiddle | ffi | ratio |
|
|
32
|
+
|---|---|---|---|
|
|
33
|
+
| 800-element fixed-length array, shrinking | 2140 ms | 727 ms | 2.9x |
|
|
34
|
+
| plain `arrays(integers)`, passing | 17.6 ms | 6.6 ms | 2.7x |
|
|
35
|
+
| plain `arrays(integers)`, shrinking | 3.4 ms | 2.0 ms | 1.7x |
|
|
36
|
+
|
|
37
|
+
The ratio varies with how many native calls a property makes per case, which
|
|
38
|
+
is what the first row exaggerates on purpose and the other two do not. The
|
|
39
|
+
direction does not vary. ADR 0008 asked for a difference visible on a
|
|
40
|
+
realistic property run rather than in a tight loop around one call, and 1.7x
|
|
41
|
+
to 2.9x on whole-run wall clock is that.
|
|
42
|
+
|
|
43
|
+
Two of ADR 0008's three non-speed items settled the same way. `FFI::Struct`
|
|
44
|
+
with `.by_value` expresses `hegel_date_t`, `hegel_time_t`, and
|
|
45
|
+
`hegel_datetime_t` directly, and a degenerate draw with every field distinct
|
|
46
|
+
round-tripped exactly for all three. The Fiddle binding, by contrast, packs
|
|
47
|
+
those structs into integer registers by hand, and its sixteen-byte case has
|
|
48
|
+
stayed unverified on Win64, since that ABI passes anything over eight bytes
|
|
49
|
+
by reference. An `FFI::Function` connected to `hegel_run_start`'s output
|
|
50
|
+
callback, the run completed, and six lines of engine output arrived in Ruby;
|
|
51
|
+
the Fiddle binding passes NULL there and has never exercised
|
|
52
|
+
`Fiddle::Closure`.
|
|
53
|
+
|
|
54
|
+
The third, JRuby and TruffleRuby, was not measured: neither is installed here.
|
|
55
|
+
`ffi` publishes a `java` platform build, and this remains a claim about
|
|
56
|
+
packaging rather than something this project has run.
|
|
57
|
+
|
|
58
|
+
Against all of that, `ffi` is a third-party runtime dependency. Two facts
|
|
59
|
+
narrow the gap ADR 0001 saw. Fiddle became a bundled gem rather than a default
|
|
60
|
+
one, so `hegeltest.gemspec` already declares `fiddle` as a dependency. The
|
|
61
|
+
change is which binding gem is declared, not whether one is. And `ffi` 1.17.4
|
|
62
|
+
publishes precompiled binary gems for every platform this gem targets:
|
|
63
|
+
`x86_64-linux-gnu`, `aarch64-linux-gnu`, both `-musl` variants, `arm64-darwin`,
|
|
64
|
+
`x86_64-darwin`, `x64-mingw-ucrt`, `aarch64-mingw-ucrt`, and `java`. Installing
|
|
65
|
+
still needs no compiler.
|
|
66
|
+
|
|
67
|
+
## Decision
|
|
68
|
+
|
|
69
|
+
Bind libhegel through the `ffi` gem. `Hegel::LibHegel::Real` is written
|
|
70
|
+
against `ffi` alone, with nothing left over from the Fiddle binding: the
|
|
71
|
+
hand-packed date, time, and datetime structs become `FFI::Struct` definitions
|
|
72
|
+
passed by value, and the tests that pinned those packings by field offset go
|
|
73
|
+
with them, because `ffi` is what computes the layout now.
|
|
74
|
+
|
|
75
|
+
Every raw binding call stays confined to that one file, as ADR 0001 decided
|
|
76
|
+
and this record keeps. That confinement is why this switch costs one file
|
|
77
|
+
rather than twenty: the conformance test holds `Real` and the Fake to a single
|
|
78
|
+
method list, and every other file in the library talks to that list.
|
|
79
|
+
|
|
80
|
+
The gemspec declares `ffi` with a pessimistic constraint, and `Gemfile.lock`
|
|
81
|
+
pins the exact version, matching how this project already handles a
|
|
82
|
+
dependency.
|
|
83
|
+
|
|
84
|
+
## Consequences
|
|
85
|
+
|
|
86
|
+
The struct-by-value entry in this project's open questions closes. There is no
|
|
87
|
+
longer a hand-computed field offset to drift from the header, and no Win64
|
|
88
|
+
divergence to confirm, because libffi classifies the argument.
|
|
89
|
+
|
|
90
|
+
The `Fiddle::Closure` open question closes too, by no longer being about
|
|
91
|
+
anything: nothing in this library uses Fiddle. Wiring the engine's output
|
|
92
|
+
callback is now possible rather than merely plausible, though this record does
|
|
93
|
+
not wire it. Output stays on stderr until something asks for it.
|
|
94
|
+
|
|
95
|
+
`ffi` reaches JRuby and TruffleRuby where Fiddle did not. That widening is
|
|
96
|
+
untested here, and naming it in a release note before someone has run the
|
|
97
|
+
suite on either would claim more than has been shown.
|
|
98
|
+
|
|
99
|
+
A caller now installs a gem with a native component. `ffi` publishes builds
|
|
100
|
+
for the platforms above, so the ordinary install compiles nothing; a platform
|
|
101
|
+
outside that list falls back to compiling `ffi` from source, which the Fiddle
|
|
102
|
+
binding never asked of anyone.
|