hegeltest 0.1.0

data/docs/architecture.md

@@ -0,0 +1,182 @@
+# Architecture
+
+## Status
+
+`Hegel.test` runs. It drives a property to a verdict, shrinks a failure to
+its minimal counterexample, and re-raises that case's own exception with
+its class intact.
+
+What exists: library resolution (`Hegel::Locate`, the pinned
+`Hegel::LIBHEGEL_VERSION`, and the development-only `libhegel:fetch` Rake
+task), the libhegel binding (`Hegel::LibHegel`, with a real implementation
+over `ffi` and a fake for tests), the settings mapping, `Hegel::TestCase`
+wrapping the engine's whole draw surface, and the run loop.
+
+What exists beyond that: the failure report, which names drawn values by
+reading the caller's own source, interleaves them with whatever `tc.note`
+recorded, and prints a blob that `Hegel.test(reproduce_failure:)` replays.
+The generator layer covers twenty-five generators composing through `map`
+and `filter`, reachable bare through `Hegel::Syntax::Methods`. A test case
+can discard itself with `tc.assume` or `tc.reject`. A run can be shaped by
+`phases:`, `suppress_health_check:`, `report_multiple_failures:`, and the
+example database, through `database_key:` and `database:`.
+[ADR 0009](adr/0009-turn-the-example-database-on-with-a-key.md) decides the
+database keywords.
+
+`tc.target` records an observation for the engine to search toward, and
+`Hegel::Stateful.run` drives a `Hegel::StateMachine`'s rules and invariants,
+drawing values an earlier rule produced back out of a
+`Hegel::Stateful::Pool`. [ADR 0010](adr/0010-declare-stateful-rules-with-a-class-macro.md)
+decides how a machine declares its rules and
+[ADR 0011](adr/0011-let-the-test-case-own-every-pool-drawn-from-it.md) who
+frees a pool.
+
+Every feature this binding set out to cover now has a Ruby surface. What
+remains open is not a feature but a measurement, and
+[ADR 0008](adr/0008-revisit-the-binding-after-milestone-c-on-measurement.md)
+schedules it.
+
+## Where meaning comes from
+
+libhegel is hegel-rust's native engine. hegel-rust's Cargo workspace builds
+it from the `hegel-c` directory, as the crate `hegeltest-c`, whose header is
+`hegel-c/include/hegel.h`. That header defines the ABI this gem calls: what
+each function does, what its error codes mean, and who owns each pointer.
+
+The Java, Go, TypeScript, OCaml, and C++ implementations bind the same
+engine. Each works under a constraint hegel-rust does not have: a dynamic
+loader, a garbage collector, or a package manager that ships prebuilt
+binaries. Where this document cites one of them, it cites a binding
+mechanism, not the meaning of a call.
+
+## libhegel and the Ruby boundary
+
+libhegel runs in the same process as its caller. Nearly every function in
+`hegel.h` takes a `hegel_context_t*` as its first argument and returns a
+`hegel_result_t` status code. `hegel_context_new` returns the context
+itself instead. `hegel_context_last_error` returns a borrowed message
+pointer instead of a status code.
+
+Ruby calls this ABI through the `ffi` gem. See
+[ADR 0013](adr/0013-bind-libhegel-through-the-ffi-gem.md), which supersedes
+[ADR 0001](adr/0001-bind-libhegel-through-fiddle.md).
+
+## The binding confined to one module
+
+Every raw `ffi` call lives in `Hegel::LibHegel`. The rest of the
+library calls that module's wrappers, never `ffi` directly. This gives
+the library one seam where a test can substitute a fake implementation,
+exercising libhegel's error codes without loading the real engine.
+
+## Run loop ownership
+
+The Ruby side owns the run loop. It starts a run, asks libhegel for
+each test case, calls the caller's test body, and reports the result back.
+A test body's own assertion failure must not cross the C boundary as if it
+were a crash.
+
+Ruby's test frameworks complicate this. With minitest 5.27.0, the version
+this gem's `Gemfile.lock` pins, `Minitest::Assertion.superclass` is
+`Exception`. With rspec-expectations 3.13.5,
+`RSpec::Expectations::ExpectationNotMetError.superclass` is `Exception`
+as well. By default, `rescue => e` catches only `StandardError`. A run
+loop written that way would let a failing assertion pass by uncaught,
+instead of being reported as a Hegel failure.
+
+Read the superclass rather than the full ancestor list. `ancestors` also
+reports what a given load order mixed into `Object`, so it answers a
+different question on a run that loaded `minitest/spec` than on one that
+did not.
+
+The run loop `rescue`s `Exception` and re-raises the library's own control
+exceptions first. Those control exceptions themselves descend from
+`Exception`, not `StandardError`, so a test body's own `rescue => e` cannot
+swallow them mid-run. `Hegel::Error`, which reports ordinary library errors
+to a caller outside a run, stays a `StandardError`.
+
+## Native handle lifetime
+
+Every `hegel_*_free` function in `hegel.h` takes the handle's owning
+context as an argument, so the context must outlive every handle allocated
+from it. Ruby's garbage collector does not guarantee finalizer ordering
+between a context and its handles.
+
+The code that owns a run releases its handles in an `ensure` block,
+freeing every other handle before the context. A finalizer may back up an
+`ensure` block that a raised exception skipped, but it must free nothing a
+prior `ensure` already freed.
+
+A pool is the one handle a caller's own code opens, inside a rule, where
+there is no `ensure` of its own to release it. `Hegel::TestCase` records
+every pool opened from it and `Hegel::Runner` releases them alongside the
+test-case handle, in nested `ensure`s so a failure releasing one cannot skip
+the other. See
+[ADR 0011](adr/0011-let-the-test-case-own-every-pool-drawn-from-it.md).
+
+## A rule's own control flow
+
+Inside a stateful rule, `tc.assume(false)` means something narrower than it
+does in a test body. `Hegel::Stateful` catches it, tells libhegel the rule
+was rejected so the attempt does not spend a step, and draws another rule;
+the test case continues. `Hegel::Runner.classify` never sees it, and no case
+is discarded.
+
+A rule is also the one place in this library where a process-ending
+exception can be raised inside an open span, which is why
+`Hegel::FATAL_EXCEPTIONS` lives beside the other control exceptions rather
+than beside the run loop: two `rescue Exception` sites need it now, and both
+have to let those four straight through before anything else.
+
+## Generator validation happens at draw time
+
+hegel-rust's own contributor documentation states the rule for its
+generators: every invalid combination of builder values must be caught at
+draw time. The check does not run when the generator is built. The same
+documentation treats a validation message as public API, asserted against
+by tests as a stable substring. `integers(min_value: 5, max_value: 1)`
+therefore returns a generator; the error arrives from `tc.draw`.
+
+## Passing a struct by value
+
+Exactly three ABI functions take a struct by value: `hegel_generate_date`,
+`hegel_generate_time`, and `hegel_generate_datetime`, each receiving two
+bound values.
+
+Each struct is declared as an `FFI::Struct` and passed with `.by_value`, so
+libffi classifies the argument from the same C-ABI rules the compiled
+library was built against. Nothing here computes a field offset or a size,
+and nothing here decides how a given ABI passes a struct of a given width.
+That matters most for the sixteen-byte `hegel_datetime_t`, where arm64 and
+System V use two registers and Win64 passes by reference.
+
+The layout is still worth a test, but a different one: the declaration is
+transcribed from `hegel.h` by hand, so a degenerate draw (`min == max`)
+against the real engine, with every field set to a distinct value, is what
+catches a transcription that disagrees with the header. Boundary values are
+not enough, since a value like `00:00:00` is symmetric under swapping two
+fields.
+
+## Open questions
+
+- Whether the vendored `linux` binaries run under musl (Alpine) has not
+  been checked. CI runs on Ubuntu, which is glibc.
+- This project has not traced which directories `ffi`'s `dlopen`-based
+  loading searches on Windows. The suite loads the engine from the path
+  `Hegel::Locate` resolves, on both Windows runners.
+
+## Answered
+
+A drawn string arrives as a pointer and a length, is not NUL-terminated,
+and can hold interior NUL bytes, because the drawn alphabet can include
+U+0000. Read by length. Measured against 0.32.5: an alphabet pinned to
+U+0000 produces a three-byte draw that reads as `""` when taken up to the
+first NUL and as three NUL bytes when taken by its reported length. Every
+string observed reports `valid_encoding?` after `force_encoding(UTF-8)`.
+
+Prism recovers a draw's assignment target through a method chain, an
+instance-variable assignment, and an assignment spanning several lines.
+Where several assignments cover the draw's line, the innermost one names
+it, so wrapping a body in `assert_raises do ... end` or an RSpec
+`expect { ... }` keeps the name. Two assignments side by side on one line
+name nothing, and so does a draw whose value is never assigned; both fall
+back to a number.

data/lib/hegel/draw_name.rb

@@ -0,0 +1,109 @@
+# frozen_string_literal: true
+
+require "prism"
+
+module Hegel
+  # Recovers a drawn value's variable name from the caller's own source (see
+  # docs/adr/0005), so a failure report can print `n = 501` instead of
+  # `draw = 501` when the caller never passed a label:. This module only
+  # answers "what name does path:lineno assign a value to"; deciding
+  # whether to call it, and what to fall back to when it answers nil, is
+  # Hegel::TestCase#name_for's job, not this one's.
+  module DrawName
+    # Node types #for treats as "this line names a drawn value". An
+    # explicit list, not every Prism::Node subclass whose name ends in
+    # WriteNode, because a wrong guess here (e.g. matching a constant
+    # assignment as if it named a draw) would misname a report entry -- the
+    # one outcome this feature must never risk. Local and instance
+    # variables both expose the assigned name the same way (a Symbol,
+    # including the leading "@" for an ivar), so one code path reads both.
+    ASSIGNMENT_NODE_TYPES = [Prism::LocalVariableWriteNode, Prism::InstanceVariableWriteNode].freeze
+
+    module_function
+
+    # The name +path+:+lineno+ assigns a drawn value to, or nil when that
+    # cannot be answered confidently: +path+ cannot be read, Prism cannot
+    # parse it, or the assignments covering +lineno+ do not single one out.
+    # A wrong name would misdirect a reader of the failure report more than
+    # a missing one would, so an ambiguous line returns nil rather than
+    # guessing between candidates.
+    #
+    # Several assignments can cover one line by nesting rather than by
+    # ambiguity. `error = assert_raises do ... n = tc.draw_integer(...) ...
+    # end` puts the draw inside both, and a reader has no doubt which one
+    # names it. So the innermost wins: an enclosing assignment is discarded
+    # whenever another candidate sits inside it. Two assignments written
+    # side by side on one line contain neither the other, nothing singles
+    # one out, and the answer is nil.
+    def for(path, lineno)
+      program = parse(path)
+      return nil unless program
+
+      matches = assignment_nodes(program).select { |node| covers?(node.location, lineno) }
+      innermost = matches.reject { |node| matches.any? { |other| encloses?(node, other) } }
+      innermost.one? ? innermost.first.name.to_s : nil
+    end
+
+    # Drops every cached parse. #for's only state; a fresh process would
+    # never need this, but a test process that calls #for many times over
+    # the life of the suite does, both to isolate one test's fixture file
+    # from another's and to prove #parse only reads a given path once (see
+    # #parse).
+    def reset_cache
+      @cache = {}
+    end
+
+    # +path+'s parsed Prism::ProgramNode, or nil if it could not be read or
+    # parsed. Cached either way, success or nil, because a single failure
+    # report names every draw the final replay recorded against the same
+    # one file, and nothing about that file changes between those lookups.
+    def parse(path)
+      cache.fetch(path) { cache[path] = read_and_parse(path) }
+    end
+
+    def cache
+      @cache ||= {}
+    end
+
+    # Reads and parses +path+. Returns nil, not the ParseResult, for either
+    # failure mode #for's caller cares about: +path+ raising SystemCallError
+    # (a nonexistent path -- eval, "-e", irb, or a file removed since the
+    # caller was compiled) and Prism reporting a syntax error (#success?
+    # false) for a path that did exist.
+    def read_and_parse(path)
+      result = Prism.parse(File.read(path))
+      result.success? ? result.value : nil
+    rescue SystemCallError
+      nil
+    end
+
+    # Every ASSIGNMENT_NODE_TYPES node under +node+, found by walking the
+    # whole tree: a drawn value's assignment can be nested arbitrarily deep
+    # (inside a block, a method body, a conditional), and this module has
+    # no way to know which nesting level to expect it at.
+    def assignment_nodes(node, matches = [])
+      matches << node if ASSIGNMENT_NODE_TYPES.include?(node.class)
+      node.compact_child_nodes.each { |child| assignment_nodes(child, matches) }
+      matches
+    end
+
+    # Range containment, not a start-line match: Ruby's own caller_locations
+    # reports the line a call's own last token is written on, and a
+    # multi-line assignment's start line can differ from that -- e.g.
+    # `n = tc\n  .draw_integer(...)` reports the second line, but the
+    # assignment node's own start_line is the first.
+    def covers?(location, lineno)
+      (location.start_line..location.end_line).cover?(lineno)
+    end
+
+    # Whether +outer+ strictly contains +inner+, by byte offset rather than
+    # by line: two assignments on one line share both line numbers, and only
+    # the offsets tell them apart from a genuine nesting.
+    def encloses?(outer, inner)
+      return false if outer.equal?(inner)
+
+      outer.location.start_offset <= inner.location.start_offset &&
+        inner.location.end_offset <= outer.location.end_offset
+    end
+  end
+end

data/lib/hegel/errors.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+module Hegel
+  # Raised for errors this library detects: a libhegel call that fails, a
+  # generator built with arguments the engine rejects, a missing native
+  # library. Control flow inside a running test case does not use this class;
+  # the control exceptions below descend from Exception instead, so that a
+  # `rescue => e` in a test body cannot swallow them.
+  #
+  # It lives here rather than in hegel.rb so that a file needing only the
+  # exception vocabulary can require this one file. Reaching it through
+  # hegel.rb would make every such file depend on the whole library, and
+  # hegel.rb requires the library back, which is a load-order cycle.
+  class Error < StandardError; end
+
+  # standard:disable Lint/InheritException
+
+  # Raised for HEGEL_E_STOP_TEST (-1): libhegel has exhausted its choice
+  # budget for the running test case and the caller must abort the test
+  # body immediately.
+  #
+  # Descends from Exception, not StandardError, so that a `rescue => e` (or
+  # a bare `rescue`) written in a user's test body cannot catch it. Only the
+  # code driving the test case is meant to catch this; letting a test body
+  # swallow it would turn "stop now" into "keep going".
+  class StopTest < Exception; end
+
+  # Raised for HEGEL_E_ASSUME (-2): an `assume` / `reject` precondition
+  # failed, so the current test case is invalid and must be discarded.
+  #
+  # Descends from Exception for the same reason as StopTest above.
+  class AssumeFailed < Exception; end
+
+  # standard:enable Lint/InheritException
+
+  # Say the process is ending, not that a property failed. Every
+  # `rescue Exception` in this library re-raises one of these before doing
+  # anything else: catching one as a counterexample would have the engine
+  # spend its shrink budget minimising an interrupt instead of letting the
+  # process exit, and NoMemoryError in particular must not be answered with
+  # another native call.
+  #
+  # Here rather than beside the run loop because two files now rescue
+  # Exception -- Hegel::Runner and Hegel::Stateful -- and this is the
+  # exception vocabulary both of them need, which is what this file is for.
+  FATAL_EXCEPTIONS = [Interrupt, SignalException, SystemExit, NoMemoryError].freeze
+end

data/lib/hegel/generator.rb

@@ -0,0 +1,98 @@
+# frozen_string_literal: true
+
+require_relative "errors"
+require_relative "lib_hegel"
+
+module Hegel
+  # Base class for every value generator (Hegel::Generators.integers,
+  # .booleans, .arrays, and so on) and the two combinators, #map and
+  # #filter, that build a new generator out of an existing one. Mirrors
+  # hegel-rust's Generator<T> trait: #do_draw is the one method a concrete
+  # generator implements, and #map/#filter come for free from this class
+  # rather than from each generator repeating them.
+  class Generator
+    # Draws one value using +tc+ (a Hegel::TestCase). Every concrete
+    # generator implements this; the base class only exists to raise a
+    # clear error for a subclass that forgot to.
+    def do_draw(_tc)
+      raise NotImplementedError, "#{self.class} must implement #do_draw"
+    end
+
+    # Returns a new Generator whose #do_draw runs +block+ on the value this
+    # generator drew, spanned with HEGEL_LABEL_MAPPED so the shrinker
+    # treats the source draw and the transform as one unit (see
+    # hegel-rust's Mapped, src/generators/generators.rs).
+    def map(&block)
+      Mapped.new(self, block)
+    end
+
+    # Returns a new Generator whose #do_draw retries this generator until
+    # +block+ returns true for the drawn value, or discards the test case
+    # if it never does. See Filtered for the retry budget.
+    def filter(&block)
+      Filtered.new(self, block)
+    end
+
+    # Generator#map's result. A private implementation detail of #map, not
+    # part of this library's public generator vocabulary (unlike the five
+    # under Hegel::Generators), so it lives here rather than in
+    # generators.rb.
+    class Mapped < Generator
+      def initialize(source, block)
+        super()
+        @source = source
+        @block = block
+      end
+
+      def do_draw(tc)
+        tc.start_span(LibHegel::HEGEL_LABEL_MAPPED)
+        @block.call(@source.do_draw(tc))
+      ensure
+        # discard is always false here: a span is only ever marked
+        # rejected to ask libhegel to retry it with different data (see
+        # Filtered below), and #map has no such retry -- an exception from
+        # +block+ or +source+ propagates past this method instead, and the
+        # span still has to close on the way out.
+        tc.stop_span(discard: false)
+      end
+    end
+
+    # Generator#filter's result. Retries the source generator, each
+    # attempt in its own span, up to MAX_ATTEMPTS times before discarding
+    # the test case (Hegel::AssumeFailed) -- the same budget hegel-rust's
+    # Filtered spends before calling assume(false)
+    # (src/generators/generators.rs), ported here rather than reinvented so
+    # a predicate that (almost) never holds fails the same way in both
+    # bindings.
+    #
+    # A predicate that never holds does not loop forever: libhegel's own
+    # health check aborts a run that discards too many test cases in a
+    # row, which Hegel::Runner.finish surfaces as Hegel::Error, the same
+    # path an engine-level failure takes for any other reason.
+    class Filtered < Generator
+      MAX_ATTEMPTS = 3
+
+      def initialize(source, block)
+        super()
+        @source = source
+        @block = block
+      end
+
+      def do_draw(tc)
+        MAX_ATTEMPTS.times do
+          accepted = false
+          value = nil
+          tc.start_span(LibHegel::HEGEL_LABEL_FILTER)
+          begin
+            value = @source.do_draw(tc)
+            accepted = @block.call(value)
+          ensure
+            tc.stop_span(discard: !accepted)
+          end
+          return value if accepted
+        end
+        raise Hegel::AssumeFailed
+      end
+    end
+  end
+end