megatest 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 81186c745613e5b5ffe4ef277acaed9763ea27b308a2d25c9f155d2369b15624
+  data.tar.gz: d6034a325bce53488239699554363bad937fb2c4e77a34f07e2862476ebfad0c
+SHA512:
+  metadata.gz: 1086dd76f6bf6ab4ae7e0974cf2c9e760bd20b104b5cbfba3e960d8fb9f3f0980c21c814ba09619a33be89edc6dadfae4edba8885c982354192fb54298301bb6
+  data.tar.gz: 06cd253892c7f76f28cf7feddf51f5135739040c6177abd9bc63ab5567b87832675a03aac3ad060831690df03824b0f07662b9d71ca7bd92419e70b764b66fd0

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+## [Unreleased]
+
+## [0.1.0] - 2024-08-20
+
+- Initial release

data/README.md

@@ -0,0 +1,156 @@
+# Megatest
+
+Megatest is a test-unit like framework with a focus on usability, and designed with continuous integration in mind.
+
+## Installation
+
+Install the gem and add to the application's Gemfile by executing:
+
+    $ bundle add megatest
+
+## Usage
+
+### Writing Tests
+
+Test suites are Ruby classes that inherit from `Megatest::Test`.
+
+Test cases are be defined with the `test` macro, or for compatibility with existing test suites,
+by defining a method starting with `test_`.
+
+All the classic `test-unit` and `minitest` assertion methods are available:
+
+```ruby
+# test/some_test.rb
+
+class SomeTest < MyApp::Test
+  setup do
+    @user = User.new("George")
+  end
+
+  test "the truth" do
+    assert_equal true, Some.truth
+  end
+
+  def test_it_works
+    assert_predicate 2, :even?
+  end
+end
+```
+
+By convention, all the `test_helper.rb` files are automatically loaded,
+which allows to centralize dependencies and define some helpers.
+
+```ruby
+# test/test_helper.rb
+
+require "some_dependency"
+
+module MyApp
+  class Test < Megatest::Test
+
+    def some_helper(arg)
+    end
+  end
+end
+```
+
+It also allow to define test inside `context` blocks, to make it easier to group
+related tests together and have them share a common name prefix.
+
+```ruby
+class SomeTest < MyApp::Test
+  context "when on earth" do
+    test "1 is odd" do
+      App.location = "earth"
+      assert_predicate 1, :odd?
+    end
+
+    test "2 is even" do
+      App.location = "earth"
+      assert_predicate 2, :even?
+    end
+  end
+end
+```
+
+Note however that context blocks aren't test suites, they don't have their own setup or teardown
+blocks, nor their own namespaces.
+
+### Command Line
+
+Contrary to many alternatives, `megatest` provide a convenient CLI interface to easily run specific tests.
+
+Run all tests in a directory:
+
+```bash
+$ megatest # Run all tests in `test/`
+$ megatest test/integration
+```
+
+Runs tests using 8 processes:
+
+```bash
+$ megatest -j 8
+```
+
+Run a test at the specific line:
+
+```bash
+$ megatest test/some_test.rb:42 test/other_test.rb:24
+```
+
+Run all tests matching a pattern:
+
+```bash
+$ megatest test/some_test.rb:/matching
+```
+
+For more detailed usage, run `megatest --help`.
+
+### CI Parallelization
+
+Megatest offer multiple feature to allow running test suites in parallel across
+many CI jobs.
+
+#### Sharding
+
+The simplest way is sharding. Each worker will run its share of the test cases.
+
+Many CI systems provide a way to run the same command on multiple nodes,
+and will generally expose environment variables to help split the workload.
+
+```yaml
+- label: "Run Unit Tests"
+  run: megatest --workers-count $CI_NODE_INDEX --worker-id $CI_NODE_TOTAL
+  parallel: 8
+```
+
+Note that Megatest makes no effort at balancing the shards as it has no
+information about how long each individual test case is expected to take.
+However it does shard test cases individually, so it avoids the most common issue which is
+very large test suites containing lots of slow test cases being sharded as one unit.
+
+If you are using CircleCI, Buildkite or HerokuCI, the workers count and worker id
+will be automatically inferred from the environment.
+
+### Redis Distribution
+
+A more efficient way to parallelize tests on CI is to use a Redis server to act as a queue.
+
+This allow to efficiently and dynamically ensure a near perfect test case balance across all
+the workers. And if for some reason one of the worker is lost or crashes, no test is lost,
+which for builds with hundreds of parallel jobs, is essential for stability.
+
+```yaml
+- label: "Run Unit Tests"
+  run: megatest --queue redis://redis-ci.example.com --build-id $CI_BUILD_ID --worker-id $CI_JOB_ID
+  parallel: 128
+  soft_fail: true # Doesn't matter if they fail or crash, only the "Results" job status matters
+
+- label: "Unit Test Results"
+  run: megatest report --queue redis://redis-ci.example.com --build-id $CI_BUILD_ID
+```
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/byroot/megatest.

data/TODO.md

@@ -0,0 +1,17 @@
+### Wants
+
+- Test leak bisect
+  - See ci-queue bisect.
+
+- List slow tests
+  - Not just X slowest test, but up to X tests that are significantly slower than average.
+  - Exclude them with `:slow` tag.
+
+- `-j` for forkless environments (Windows / JRuby / TruffleRuby)
+
+- `minitest/mocks`
+  - I'm not very fond of those, but could be worth offering it as a side gem or something, for easier transition.
+
+### Maybe
+
+- RSpec style pending?

data/exe/megatest

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "megatest"
+require "megatest/cli"
+
+Megatest::CLI.run!

data/lib/megatest/assertions.rb

@@ -0,0 +1,474 @@
+# frozen_string_literal: true
+
+module Megatest
+  class Assertion < Exception
+  end
+
+  class NoAssertion < Assertion
+    def initialize(message = "No assertions performed")
+      super
+    end
+  end
+
+  DidNotRun = Class.new(Assertion)
+
+  class LostTest < Assertion
+    def initialize(test_id)
+      super("#{test_id} never completed. Might be caused by a crash or early exit?")
+    end
+  end
+
+  Skip = Class.new(Assertion)
+
+  class UnexpectedError < Assertion
+    attr_reader :cause
+
+    def initialize(cause)
+      super("Unexpected exception")
+      @cause = cause
+    end
+
+    def backtrace
+      cause.backtrace
+    end
+
+    def backtrace_locations
+      cause.backtrace_locations
+    end
+  end
+
+  module Assertions
+    def pass
+      @__m.assert {}
+    end
+
+    def assert(result, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        return if result
+
+        if message
+          @__m.fail(message)
+        else
+          @__m.fail(message, "Expected", @__m.pp(result), "to be truthy")
+        end
+      end
+    end
+
+    def refute(result, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        return unless result
+
+        if message
+          @__m.fail(message)
+        else
+          @__m.fail(message, "Expected", @__m.pp(result), "to be falsy")
+        end
+      end
+    end
+
+    def assert_nil(actual, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        unless nil.equal?(actual)
+          @__m.fail(message, "Expected", @__m.pp(actual), "to be nil")
+        end
+      end
+    end
+
+    def refute_nil(actual, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        if nil.equal?(actual)
+          @__m.fail(message, "Expected", @__m.pp(actual), "to not be nil")
+        end
+      end
+    end
+
+    def assert_equal(expected, actual, msg = nil, message: nil, allow_nil: false)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        if !allow_nil && nil == expected
+          @__m.fail(nil, "Use assert_nil if expecting nil, or pass `allow_nil: true`")
+        end
+
+        if expected != actual
+          @__m.fail(
+            message,
+            @__m.diff(expected, actual) ||
+            "Expected: #{@__m.pp(expected)}\n" \
+            "  Actual: #{@__m.pp(actual)}",
+          )
+        end
+      end
+    end
+
+    def refute_equal(expected, actual, msg = nil, message: nil, allow_nil: false)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        if !allow_nil && nil == expected && !@__m.minitest_compatibility?
+          @__m.fail(nil, "Use refute_nil if expecting to not be nil, or pass `allow_nil: true`")
+        end
+
+        if expected == actual
+          @__m.fail(message, "Expected", @__m.pp(expected), "to not equal", @__m.pp(actual))
+        end
+      end
+    end
+
+    def assert_includes(collection, object, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        unless collection.include?(object)
+          @__m.fail message, "Expected", @__m.pp(collection), "to include", @__m.pp(object)
+        end
+      end
+    end
+
+    def refute_includes(collection, object, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        if collection.include?(object)
+          @__m.fail message, "Expected", @__m.pp(collection), "to not include", @__m.pp(object)
+        end
+      end
+    end
+
+    def assert_empty(object, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        unless object.empty?
+          @__m.fail message, "Expected", @__m.pp(object), "to be empty"
+        end
+      end
+    end
+
+    def refute_empty(object, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        if object.empty?
+          @__m.fail message, "Expected", @__m.pp(object), "to not be empty"
+        end
+      end
+    end
+
+    def assert_instance_of(klass, actual, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        unless actual.instance_of?(klass)
+          @__m.fail(message, "Expected", @__m.pp(actual), "to be an instance of", @__m.pp(klass), "not", @__m.pp(actual.class))
+        end
+      end
+    end
+
+    def refute_instance_of(klass, actual, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        if actual.instance_of?(klass)
+          @__m.fail(message, "Expected", @__m.pp(actual), "to not be an instance of", @__m.pp(klass))
+        end
+      end
+    end
+
+    def assert_kind_of(klass, actual, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        unless actual.kind_of?(klass)
+          @__m.fail(message, "Expected", @__m.pp(actual), "to be a kind of", @__m.pp(klass), "not", @__m.pp(actual.class))
+        end
+      end
+    end
+
+    def refute_kind_of(klass, actual, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        if actual.kind_of?(klass)
+          @__m.fail(message, "Expected", @__m.pp(actual), "to not be a kind of", @__m.pp(klass))
+        end
+      end
+    end
+
+    def assert_predicate(actual, predicate, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        unless @__m.expect_no_failures { actual.__send__(predicate) }
+          @__m.fail(message, "Expected", @__m.pp(actual), "to be #{predicate}")
+        end
+      end
+    end
+
+    def refute_predicate(actual, predicate, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        if @__m.expect_no_failures { actual.__send__(predicate) }
+          @__m.fail(message, "Expected", @__m.pp(actual), "to not be #{predicate}")
+        end
+      end
+    end
+
+    def assert_match(original_matcher, obj, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        matcher = if ::String === original_matcher
+          ::Regexp.new(::Regexp.escape(original_matcher))
+        else
+          original_matcher
+        end
+
+        unless match = matcher.match(obj)
+          @__m.fail(message, "Expected", @__m.pp(original_matcher), "to match", @__m.pp(obj))
+        end
+
+        match
+      end
+    end
+
+    def refute_match(original_matcher, obj, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        matcher = if ::String === original_matcher
+          ::Regexp.new(::Regexp.escape(original_matcher))
+        else
+          original_matcher
+        end
+
+        if matcher.match?(obj)
+          @__m.fail(message, "Expected", @__m.pp(original_matcher), "to not match", @__m.pp(obj))
+        end
+      end
+    end
+
+    def assert_respond_to(object, method, msg = nil, message: nil, include_all: false)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        unless object.respond_to?(method, include_all)
+          @__m.fail(message, "Expected", @__m.pp(object), "to respond to :#{method}")
+        end
+      end
+    end
+
+    def refute_respond_to(object, method, msg = nil, message: nil, include_all: false)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        if object.respond_to?(method, include_all)
+          @__m.fail(message, "Expected", @__m.pp(object), "to not respond to :#{method}")
+        end
+      end
+    end
+
+    def assert_same(expected, actual, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        unless expected.equal?(actual)
+          @__m.fail message, begin
+            actual_pp = @__m.pp(actual)
+            expected_pp = @__m.pp(expected)
+            if actual_pp == expected_pp
+              actual_pp += " (id: #{actual.object_id})"
+              expected_pp += " (id: #{expected.object_id})"
+            end
+
+            "Expected          #{actual_pp}\n" \
+            "To be the same as #{expected_pp}"
+          end
+        end
+      end
+    end
+
+    def refute_same(expected, actual, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        if expected.equal?(actual)
+          @__m.fail message, begin
+            actual_pp = @__m.pp(actual)
+            expected_pp = @__m.pp(expected)
+            if actual_pp == expected_pp
+              actual_pp += " (id: #{actual.object_id})"
+              expected_pp += " (id: #{expected.object_id})"
+            end
+
+            "Expected              #{actual_pp}\n" \
+            "To not be the same as #{expected_pp}"
+          end
+        end
+      end
+    end
+
+    def assert_raises(expected = StandardError, *expected_exceptions, message: nil)
+      msg = expected_exceptions.pop if expected_exceptions.last.is_a?(String)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        @__m.fail("assert_raises requires a block to capture errors.") unless block_given?
+
+        begin
+          yield
+        rescue expected, *expected_exceptions => exception
+          return exception
+        rescue ::Megatest::Assertion, *::Megatest::IGNORED_ERRORS
+          raise # Pass through
+        rescue ::Exception => unexepected_exception
+          error = @__m.strip_backtrace(unexepected_exception, __FILE__, __LINE__ - 6, 0)
+
+          expected_pp = if expected_exceptions.empty?
+            @__m.pp(expected)
+          else
+            expected_exceptions.map { |e| @__m.pp(e) }.join(", ") << " or #{@__m.pp(expected)}"
+          end
+
+          @__m.fail(message, "#{expected_pp} exception expected, not:\n#{@__m.pp(error)}")
+        end
+
+        expected_pp = if expected_exceptions.empty?
+          @__m.pp(expected)
+        else
+          expected_exceptions.map { |e| @__m.pp(e) }.join(", ") << " or #{@__m.pp(expected)}"
+        end
+
+        @__m.fail(message, "Expected", expected_pp, "but nothing was raised.")
+      end
+    end
+
+    def assert_throws(thrown_object, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        caught = true
+        value = catch(thrown_object) do
+          @__m.expect_no_failures do
+            yield
+          rescue UncaughtThrowError => error
+            @__m.fail(message, "Expected", @__m.pp(thrown_object), "to have been thrown, not:", @__m.pp(error.tag))
+          end
+          caught = false
+        end
+
+        unless caught
+          @__m.fail(message, "Expected", @__m.pp(thrown_object), "to have been thrown, but it wasn't")
+        end
+
+        value
+      end
+    end
+
+    def assert_operator(left, operator, right, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        unless left.__send__(operator, right)
+          @__m.fail(message, "Expected", @__m.pp(left), "to be #{operator}", @__m.pp(right))
+        end
+      end
+    end
+
+    def refute_operator(left, operator, right, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        if left.__send__(operator, right)
+          @__m.fail(message, "Expected", @__m.pp(left), "to not be #{operator}", @__m.pp(right))
+        end
+      end
+    end
+
+    def assert_in_delta(expected, actual, delta = 0.001, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        diff = (expected - actual).abs
+        unless delta >= diff
+          @__m.fail(message, "Expected", "|#{@__m.pp(expected)} - #{@__m.pp(actual)}| (#{diff})", "to be <= #{delta}")
+        end
+      end
+    end
+
+    def refute_in_delta(expected, actual, delta = 0.001, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        diff = (expected - actual).abs
+        if delta >= diff
+          @__m.fail(message, "Expected", "|#{@__m.pp(expected)} - #{@__m.pp(actual)}| (#{diff})", "to not be <= #{delta}")
+        end
+      end
+    end
+
+    def assert_in_epsilon(expected, actual, epsilon = 0.001, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        diff = (expected - actual).abs
+        delta = [expected.abs, actual.abs].min * epsilon
+        unless delta >= diff
+          @__m.fail(message, "Expected", "|#{@__m.pp(expected)} - #{@__m.pp(actual)}| (#{diff})", "to be <= #{delta}")
+        end
+      end
+    end
+
+    def refute_in_epsilon(expected, actual, epsilon = 0.001, msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        diff = (expected - actual).abs
+        delta = [expected.abs, actual.abs].min * epsilon
+        if delta >= diff
+          @__m.fail(message, "Expected", "|#{@__m.pp(expected)} - #{@__m.pp(actual)}| (#{diff})", "to not be <= #{delta}")
+        end
+      end
+    end
+
+    def skip(message = nil)
+      message ||= "Skipped, no message given"
+      ::Kernel.raise(::Megatest::Skip, message, nil)
+    end
+
+    def flunk(msg = nil, message: nil)
+      message = @__m.msg(msg, message)
+      @__m.assert do
+        @__m.fail(message || "Failed")
+      end
+    end
+
+    def assert_output(expected_stdout = nil, expected_stderr = nil, &block)
+      @__m.assert do
+        @__m.fail("assert_output requires a block to capture output.") unless block_given?
+
+        actual_stdout, actual_stderr = @__m.expect_no_failures do
+          capture_io(&block)
+        end
+
+        if expected_stderr
+          if Regexp === expected_stderr
+            assert_match(expected_stderr, actual_stderr, message: "In stderr")
+          else
+            assert_equal(expected_stderr, actual_stderr, message: "In stderr")
+          end
+        end
+
+        if expected_stdout
+          if Regexp === expected_stdout
+            assert_match(expected_stdout, actual_stdout, message: "In stdout")
+          else
+            assert_equal(expected_stdout, actual_stdout, message: "In stdout")
+          end
+        end
+      end
+    end
+
+    def assert_silent(&block)
+      @__m.assert do
+        assert_output("", "", &block)
+      end
+    end
+
+    def capture_io
+      require "stringio" unless defined?(::StringIO)
+      captured_stdout, captured_stderr = ::StringIO.new, ::StringIO.new
+
+      orig_stdout, orig_stderr = $stdout, $stderr
+      $stdout, $stderr = captured_stdout, captured_stderr
+
+      begin
+        yield
+
+        [captured_stdout.string, captured_stderr.string]
+      ensure
+        $stdout = orig_stdout
+        $stderr = orig_stderr
+      end
+    end
+  end
+end