minitest 1.7.2 → 2.0.0

data/History.txt

@@ -1,3 +1,30 @@
+=== 2.0.0 / 2010-11-11
+
+* 3 major enhancements:
+
+  * Added minitest/benchmark! Assert your performance! YAY!
+  * Refactored runner to allow for more extensibility. See minitest/benchmark.
+  * This makes the runner backwards incompatible in some ways!
+
+* 9 minor enhancements:
+
+  * Added MiniTest::Unit.after_tests { ... }
+  * Improved output by adding test rates and a more sortable verbose format
+  * Improved readme based on feedback from others
+  * Added io method to TestCase. If used, it'll supplant '.EF' output.
+  * Refactored IO in MiniTest::Unit.
+  * Refactored _run_anything to _run_suite to make it easier to wrap (ngauthier)
+  * Spec class names are now the unmunged descriptions (btakita)
+  * YAY for not having redundant rdoc/readmes!
+  * Help output is now generated from the flags you passed straight up.
+
+* 4 bug fixes:
+
+  * Fixed scoping issue on minitest/mock (srbaker/prosperity)
+  * Fixed some of the assertion default messages
+  * Fixes autorun when on windows with ruby install on different drive (larsch)
+  * Fixed rdoc output bug in spec.rb
+
 === 1.7.2 / 2010-09-23
 
 * 3 bug fixes:

data/Manifest.txt

@@ -5,9 +5,12 @@ README.txt
 Rakefile
 design_rationale.rb
 lib/minitest/autorun.rb
+lib/minitest/benchmark.rb
 lib/minitest/mock.rb
+lib/minitest/pride.rb
 lib/minitest/spec.rb
 lib/minitest/unit.rb
-test/test_mini_mock.rb
-test/test_mini_spec.rb
-test/test_mini_test.rb
+test/test_minitest_benchmark.rb
+test/test_minitest_mock.rb
+test/test_minitest_spec.rb
+test/test_minitest_unit.rb

data/README.txt

@@ -1,25 +1,44 @@
-= minitest/{unit,spec,mock}
+= minitest/*
 
 * http://rubyforge.org/projects/bfts
 
 == DESCRIPTION:
 
-minitest/unit is a small and fast replacement for ruby's huge and slow
-test/unit. This is meant to be clean and easy to use both as a regular
-test writer and for language implementors that need a minimal set of
-methods to bootstrap a working unit test suite.
+minitest provides a complete suite of testing facilities supporting
+TDD, BDD, mocking, and benchmarking.
 
-mini/spec is a functionally complete spec engine.
+minitest/unit is a small and incredibly fast unit testing framework.
+It provides a rich set of assertions to make your tests clean and
+readable.
 
-mini/mock, by Steven Baker, is a beautifully tiny mock object framework.
+minitest/spec is a functionally complete spec engine. It hooks onto
+minitest/unit and seamlessly bridges test assertions over to spec
+expectations.
 
-(This package was called miniunit once upon a time)
+minitest/benchmark is an awesome way to assert the performance of your
+algorithms in a repeatable manner. Now you can assert that your newb
+co-worker doesn't replace your linear algorithm with an exponential
+one!
+
+minitest/mock by Steven Baker, is a beautifully tiny mock object
+framework.
+
+minitest/pride shows pride in testing and adds coloring to your test
+output.
+
+minitest/unit is meant to have a clean implementation for language
+implementors that need a minimal set of methods to bootstrap a working
+test suite. For example, there is no magic involved for test-case
+discovery.
 
 == FEATURES/PROBLEMS:
 
-* Contains minitest/unit - a simple and clean test system (301 lines!).
-* Contains minitest/spec - a simple and clean spec system (52 lines!).
-* Contains minitest/mock - a simple and clean mock system (35 lines!).
+* minitest/autorun - the easy and explicit way to run all your tests.
+* minitest/unit - a very fast, simple, and clean test system.
+* minitest/spec - a very fast, simple, and clean spec system.
+* minitest/mock - a simple and clean mock system.
+* minitest/benchmark - an awesome way to assert your algorithm's performance.
+* minitest/pride - show your pride in testing!
 * Incredibly small and fast runner, but no bells and whistles.
 
 == RATIONALE:
@@ -30,100 +49,147 @@ See design_rationale.rb to see how specs and tests work in minitest.
 
 Given that you'd like to test the following class:
 
-    class Meme
-      def i_can_has_cheezburger?
-        "OHAI!"
-      end
-
-      def does_it_blend?
-        "YES!"
-      end
+  class Meme
+    def i_can_has_cheezburger?
+      "OHAI!"
     end
 
+    def does_it_blend?
+      "YES!"
+    end
+  end
 
 === Unit tests
 
-    require 'minitest/unit'
-    MiniTest::Unit.autorun
+  require 'minitest/autorun'
 
-    class TestMeme < MiniTest::Unit::TestCase
-      def setup
-        @meme = Meme.new
-      end
+  class TestMeme < MiniTest::Unit::TestCase
+    def setup
+      @meme = Meme.new
+    end
 
-      def test_that_kitty_can_eat
-        assert_equal "OHAI!", @meme.i_can_has_cheezburger?
-      end
+    def test_that_kitty_can_eat
+      assert_equal "OHAI!", @meme.i_can_has_cheezburger?
+    end
 
-      def test_that_it_doesnt_not_blend
-        refute_match /^no/i, @meme.does_it_blend?
-      end
+    def test_that_it_doesnt_not_blend
+      refute_match /^no/i, @meme.does_it_blend?
     end
+  end
 
 === Specs
 
-    require 'minitest/spec'
-    MiniTest::Unit.autorun
+  require 'minitest/autorun'
+
+  describe Meme do
+    before do
+      @meme = Meme.new
+    end
+
+    describe "when asked about cheeseburgers" do
+      it "must respond positively" do
+        @meme.i_can_has_cheezburger?.must_equal "OHAI!"
+      end
+    end
 
-    describe Meme do
-      before do
-        @meme = Meme.new
+    describe "when asked about blending possibilities" do
+      it "won't say no" do
+        @meme.does_it_blend?.wont_match /^no/i
       end
+    end
+  end
+
+=== Benchmarks
+
+Add benchmarks to your regular unit tests. If the unit tests fail, the
+benchmarks won't run.
 
-      describe "when asked about cheeseburgers" do
-        it "should respond positively" do
-          @meme.i_can_has_cheezburger?.must_equal "OHAI!"
+  # optionally run benchmarks, good for CI-only work!
+  require 'minitest/benchmark' if ENV["BENCH"]
+
+  class TestMeme < MiniTest::Unit::TestCase
+    # Override self.bench_range or default range is [1, 10, 100, 1_000, 10_000]
+    def bench_my_algorithm
+      assert_performance_linear 0.9999 do |n| # n is a range value
+        n.times do
+          @obj.my_algorithm
         end
       end
+    end
+  end
 
-      describe "when asked about blending possibilities" do
-        it "won't say no" do
-          @meme.does_it_blend?.wont_match /^no/i
+Or add them to your specs. If you make benchmarks optional, you'll
+need to wrap your benchmarks in a conditional since the methods won't
+be defined.
+
+  describe Meme do
+    if ENV["BENCH"] then
+      bench_performance_linear "my_algorithm", 0.9999 do |n|
+        100.times do
+          @obj.my_algorithm(n)
         end
       end
     end
+  end
+
+outputs something like:
+
+  # Running benchmarks:
+
+  TestBlah	100	1000	10000
+  bench_my_algorithm	 0.006167	 0.079279	 0.786993
+  bench_other_algorithm	 0.061679	 0.792797	 7.869932
+
+Output is tab-delimited to make it easy to paste into a spreadsheet.
 
 === Mocks
 
-    class MemeAsker
-      def initialize(meme)
-        @meme = meme
-      end
+  class MemeAsker
+    def initialize(meme)
+      @meme = meme
+    end
 
-      def ask(question)
-        method = question.tr(" ","_") + "?"
-        @meme.send(method)
-      end
+    def ask(question)
+      method = question.tr(" ","_") + "?"
+      @meme.send(method)
     end
+  end
 
-    require 'minitest/spec'
-    require 'minitest/mock'
-    MiniTest::Unit.autorun
+  require 'minitest/autorun'
 
-    describe MemeAsker do
-      before do
-        @meme = MiniTest::Mock.new
-        @meme_asker = MemeAsker.new @meme
-      end
+  describe MemeAsker do
+    before do
+      @meme = MiniTest::Mock.new
+      @meme_asker = MemeAsker.new @meme
+    end
 
-      describe "#ask" do
-        describe "when passed an unpunctuated question" do
-          it "should invoke the appropriate predicate method on the meme" do
-            @meme.expect :does_it_blend?, :return_value
-            @meme_asker.ask "does it blend"
-            @meme.verify
-          end
+    describe "#ask" do
+      describe "when passed an unpunctuated question" do
+        it "should invoke the appropriate predicate method on the meme" do
+          @meme.expect :does_it_blend?, :return_value
+          @meme_asker.ask "does it blend"
+          @meme.verify
         end
       end
     end
+  end
 
 == REQUIREMENTS:
 
-+ Ruby 1.8, maybe even 1.6 or lower. No magic is involved.
+* Ruby 1.8, maybe even 1.6 or lower. No magic is involved.
 
 == INSTALL:
 
-+ sudo gem install minitest
+  sudo gem install minitest
+
+On 1.9, you already have it. To get newer candy you can still install
+the gem, but you'll need to activate the gem explicitly to use it:
+
+  require 'rubygems'
+  gem 'minitest' # ensures you're using the gem, and not the built in MT
+  require 'minitest/autorun'
+  
+  # ... usual testing stuffs ...
 
 == LICENSE:
 

data/lib/minitest/benchmark.rb

@@ -0,0 +1,324 @@
+require 'minitest/unit'
+require 'minitest/spec'
+
+class MiniTest::Unit
+  attr_accessor :runner
+
+  def run_benchmarks
+    _run_anything :benchmark
+  end
+
+  def benchmark_suite_header suite
+    "\n#{suite}\t#{suite.bench_range.join("\t")}"
+  end
+
+  class TestCase
+    ##
+    # Returns a set of ranges stepped exponentially from +min+ to
+    # +max+ by powers of +base+. Eg:
+    #
+    #   bench_exp(2, 16, 2) # => [2, 4, 8, 16]
+
+    def self.bench_exp min, max, base = 10
+      min = (Math.log10(min) / Math.log10(base)).to_i
+      max = (Math.log10(max) / Math.log10(base)).to_i
+
+      (min..max).map { |m| base ** m }.to_a
+    end
+
+    ##
+    # Returns a set of ranges stepped linearly from +min+ to +max+ by
+    # +step+. Eg:
+    #
+    #   bench_linear(20, 40, 10) # => [20, 30, 40]
+
+    def self.bench_linear min, max, step = 10
+      (min..max).step(step).to_a
+    rescue LocalJumpError # 1.8.6
+      r = []; (min..max).step(step) { |n| r << n }; r
+    end
+
+    ##
+    # Returns the benchmark methods (methods that start with bench_)
+    # for that class.
+
+    def self.benchmark_methods # :nodoc:
+      public_instance_methods(true).grep(/^bench_/).map { |m| m.to_s }.sort
+    end
+
+    ##
+    # Returns all test suites that have benchmark methods.
+
+    def self.benchmark_suites
+      TestCase.test_suites.reject { |s| s.benchmark_methods.empty? }
+    end
+
+    ##
+    # Specifies the ranges used for benchmarking for that class.
+    # Defaults to exponential growth from 1 to 10k by powers of 10.
+    # Override if you need different ranges for your benchmarks.
+    #
+    # See also: ::bench_exp and ::bench_linear.
+
+    def self.bench_range
+      bench_exp 1, 10_000
+    end
+
+    ##
+    # Runs the given +work+, gathering the times of each run. Range
+    # and times are then passed to a given +validation+ proc. Outputs
+    # the benchmark name and times in tab-separated format, making it
+    # easy to paste into a spreadsheet for graphing or further
+    # analysis.
+    #
+    # Ranges are specified by ::bench_range.
+    #
+    # Eg:
+    #
+    #   def bench_algorithm
+    #     validation = proc { |x, y| ... }
+    #     assert_performance validation do |x|
+    #       @obj.algorithm
+    #     end
+    #   end
+
+    def assert_performance validation, &work
+      range = self.class.bench_range
+
+      io.print "#{__name__}"
+
+      times = []
+
+      range.each do |x|
+        GC.start
+        t0 = Time.now
+        instance_exec(x, &work)
+        t = Time.now - t0
+
+        io.print "\t%9.6f" % t
+        times << t
+      end
+      io.puts
+
+      validation[range, times]
+    end
+
+    ##
+    # Runs the given +work+ and asserts that the times gathered fit to
+    # match a constant rate (eg, linear slope == 0) within a given error
+    # +threshold+.
+    #
+    # Fit is calculated by #fit_constant.
+    #
+    # Ranges are specified by ::bench_range.
+    #
+    # Eg:
+    #
+    #   def bench_algorithm
+    #     assert_performance_constant 0.9999 do |x|
+    #       @obj.algorithm
+    #     end
+    #   end
+
+    def assert_performance_constant threshold = 0.99, &work
+      validation = proc do |range, times|
+        a, b, rr = fit_linear range, times
+        assert_in_delta 0, b, 1 - threshold
+        [a, b, rr]
+      end
+
+      assert_performance validation, &work
+    end
+
+    ##
+    # Runs the given +work+ and asserts that the times gathered fit to
+    # match a exponential curve within a given error +threshold+.
+    #
+    # Fit is calculated by #fit_exponential.
+    #
+    # Ranges are specified by ::bench_range.
+    #
+    # Eg:
+    #
+    #   def bench_algorithm
+    #     assert_performance_exponential 0.9999 do |x|
+    #       @obj.algorithm
+    #     end
+    #   end
+
+    def assert_performance_exponential threshold = 0.99, &work
+      assert_performance validation_for_fit(:exponential, threshold), &work
+    end
+
+    ##
+    # Runs the given +work+ and asserts that the times gathered fit to
+    # match a straight line within a given error +threshold+.
+    #
+    # Fit is calculated by #fit_linear.
+    #
+    # Ranges are specified by ::bench_range.
+    #
+    # Eg:
+    #
+    #   def bench_algorithm
+    #     assert_performance_linear 0.9999 do |x|
+    #       @obj.algorithm
+    #     end
+    #   end
+
+    def assert_performance_linear threshold = 0.99, &work
+      assert_performance validation_for_fit(:linear, threshold), &work
+    end
+
+    ##
+    # Runs the given +work+ and asserts that the times gathered curve
+    # fit to match a power curve within a given error +threshold+.
+    #
+    # Fit is calculated by #fit_power.
+    #
+    # Ranges are specified by ::bench_range.
+    #
+    # Eg:
+    #
+    #   def bench_algorithm
+    #     assert_performance_power 0.9999 do |x|
+    #       @obj.algorithm
+    #     end
+    #   end
+
+    def assert_performance_power threshold = 0.99, &work
+      assert_performance validation_for_fit(:power, threshold), &work
+    end
+
+    ##
+    # Takes an array of x/y pairs and calculates the general R^2 value.
+    #
+    # See: http://en.wikipedia.org/wiki/Coefficient_of_determination
+
+    def fit_error xys
+      y_bar  = sigma(xys) { |x, y| y } / xys.size.to_f
+      ss_tot = sigma(xys) { |x, y| (y    - y_bar) ** 2 }
+      ss_err = sigma(xys) { |x, y| (yield(x) - y) ** 2 }
+
+      1 - (ss_err / ss_tot)
+    end
+
+    ##
+    # To fit a functional form: y = ae^(bx).
+    #
+    # Takes x and y values and returns [a, b, r^2].
+    #
+    # See: http://mathworld.wolfram.com/LeastSquaresFittingExponential.html
+
+    def fit_exponential xs, ys
+      n     = xs.size
+      xys   = xs.zip(ys)
+      sxlny = sigma(xys) { |x,y| x * Math.log(y) }
+      slny  = sigma(xys) { |x,y| Math.log(y)     }
+      sx2   = sigma(xys) { |x,y| x * x           }
+      sx    = sigma xs
+
+      c = n * sx2 - sx ** 2
+      a = (slny * sx2 - sx * sxlny) / c
+      b = ( n * sxlny - sx * slny ) / c
+
+      return Math.exp(a), b, fit_error(xys) { |x| Math.exp(a + b * x) }
+    end
+
+    ##
+    # Fits the functional form: a + bx.
+    #
+    # Takes x and y values and returns [a, b, r^2].
+    #
+    # See: http://mathworld.wolfram.com/LeastSquaresFitting.html
+
+    def fit_linear xs, ys
+      n   = xs.size
+      xys = xs.zip(ys)
+      sx  = sigma xs
+      sy  = sigma ys
+      sx2 = sigma(xs)  { |x|   x ** 2 }
+      sxy = sigma(xys) { |x,y| x * y  }
+
+      c = n * sx2 - sx**2
+      a = (sy * sx2 - sx * sxy) / c
+      b = ( n * sxy - sx * sy ) / c
+
+      return a, b, fit_error(xys) { |x| a + b * x }
+    end
+
+    ##
+    # To fit a functional form: y = ax^b.
+    #
+    # Takes x and y values and returns [a, b, r^2].
+    #
+    # See: http://mathworld.wolfram.com/LeastSquaresFittingPowerLaw.html
+
+    def fit_power xs, ys
+      n       = xs.size
+      xys     = xs.zip(ys)
+      slnxlny = sigma(xys) { |x, y| Math.log(x) * Math.log(y) }
+      slnx    = sigma(xs)  { |x   | Math.log(x)               }
+      slny    = sigma(ys)  { |   y| Math.log(y)               }
+      slnx2   = sigma(xs)  { |x   | Math.log(x) ** 2          }
+
+      b = (n * slnxlny - slnx * slny) / (n * slnx2 - slnx ** 2);
+      a = (slny - b * slnx) / n
+
+      return Math.exp(a), b, fit_error(xys) { |x| (Math.exp(a) * (x ** b)) }
+    end
+
+    ##
+    # Enumerates over +enum+ mapping +block+ if given, returning the
+    # sum of the result. Eg:
+    #
+    #   sigma([1, 2, 3])                # => 1 + 2 + 3 => 7
+    #   sigma([1, 2, 3]) { |n| n ** 2 } # => 1 + 4 + 9 => 14
+
+    def sigma enum, &block
+      enum = enum.map(&block) if block
+      enum.inject { |sum, n| sum + n }
+    end
+
+    ##
+    # Returns a proc that calls the specified fit method and asserts
+    # that the error is within a tolerable threshold.
+
+    def validation_for_fit msg, threshold
+      proc do |range, times|
+        a, b, rr = send "fit_#{msg}", range, times
+        assert_operator rr, :>=, threshold
+        [a, b, rr]
+      end
+    end
+  end
+end
+
+class MiniTest::Spec
+  def self.bench name, &block
+    define_method "bench_#{name.gsub(/\W+/, '_')}", &block
+  end
+
+  def self.bench_range &block
+    meta = (class << self; self; end)
+    meta.send :define_method, "bench_range", &block
+  end
+
+  def self.bench_performance_linear name, threshold = 0.9, &work
+    bench name do
+      assert_performance_linear threshold, &work
+    end
+  end
+
+  def self.bench_performance_constant name, threshold = 0.99, &work
+    bench name do
+      assert_performance_constant threshold, &work
+    end
+  end
+
+  def self.bench_performance_exponential name, threshold = 0.99, &work
+    bench name do
+      assert_performance_exponential threshold, &work
+    end
+  end
+end