baretest 0.1.0 → 0.2.3

data/doc/quickref.rdoc

@@ -0,0 +1,261 @@
+= BareTest Quick Reference
+
+This is a very condensed overview over baretest. If you're new to testing and
+new to baretest, you may be more interested into doc/writing_tests.rdoc
+Also look into the examples and baretests own test directory.
+
+
+
+== Setup baretest
+
+1. Install baretest
+  a) official release: `sudo gem install baretest`
+  b) edge:
+    1. git clone git://github.com/apeiros/baretest.git
+    2. cd baretest
+    3. rake gem:install
+2. Change into the project directory
+3. `baretest --init` to create the basic test-directory layout
+
+
+
+== Location of testfiles
+
+Your tests for PROJECT/lib/foo.rb belong into PROJECT/test/suite/lib/foo.rb.
+Your tests for PROJECT/bin/bar belong into PROJECT/test/suite/bin/bar.
+In other words, for every path, insert /test/suite after PROJECT to get the
+path to the corresponding testfile.
+
+
+
+== Writing tests
+
+A testfile commonly looks like this:
+
+  BareTest.suite "module ModuleName" do
+    setup do
+      # This is run before each assert, that is in this suite or a nested suite
+    end
+
+    teardown do
+      # This is run after each assert, that is in this suite or a nested suite
+    end
+
+    suite "Class methods" do
+      setup do
+        # things used by most nested suites
+      end
+      
+      suite "ModuleName::class_method_name" do
+        assert "does this" do
+          ...
+        end
+
+        assert "does that" do
+        end
+      end
+
+    suite "Instance methods" do
+      suite "ModuleName#instance_method_name" do
+        ...
+      end
+    end
+    
+    suite "class ClassName" do # this is class ModuleName::ClassName
+      suite "Class methods" do
+        ...
+      end
+
+      suite "Instance methods" do
+        ...
+      end
+    end
+  end
+
+This layout makes it easy to figure where tests for something are, and thus
+makes maintaining the test-code base easier.
+
+Setup callbacks are invoked from outermost suite to innermost suite, and
+within the same suite, in the order of definition.
+
+Teardown callbacks are invoked from innermost suite to outermost suite, and
+within the same suite, in the order of definition.
+
+  BareTest.suite do
+    setup do puts 1 end
+    setup do puts 2 end
+    teardown do puts 7 end
+    teardown do puts 8 end
+
+    suite "Inner" do
+      setup do puts 3 end
+      setup do puts 4 end
+      teardown do puts 5 end
+      teardown do puts 6 end
+
+      assert "Inner - assert" do puts "Inner - assert" end
+    end
+
+    assert "Outer - assert" do puts "Outer - assert" end
+  end
+
+Running this suite will print:
+
+  1
+  2
+  3
+  4
+  Inner - assert
+  5
+  6
+  7
+  8
+  1
+  2
+  Outer - assert
+  7
+  8
+
+
+
+== Skipping Tests and Suites
+
+A test is skipped if it does not have a block or calls 'skip'
+
+== Assertion helper methods
+
+See BareTest::Assertion::Support
+All methods that have the method signature foo(expected, actual, message=nil)
+can alternatively be used with named arguments:
+  foo a, b, "message" # is equivalent to:
+  foo :expected => a, :actual => b, :message => "message"
+
+* skip(message, *args)
+  Description: Skips the assertion, uses sprintf with message and *args
+  Success:     not possible
+  Failure:     not possible
+
+* failure(message, *args)
+  Description: Lets the assertion fail, uses sprintf with message and *args
+  Success:     not possible
+  Failure:     failure("%p was not the inverse of %p", 2, 5)
+
+* same(expected, actual, message=nil)
+  Description: Uses expected.equal?(actual), which tests for object identity
+  Success:     same(:foo, :foo)
+  Failure:     same("foo", "foo")
+
+* hash_key_equal(expected, actual, message=nil)
+  Description: Uses expected.eql?(actual), which is used for hash key equality.
+  Success:     hash_key_equal("foo", "foo")
+  Failure:     hash_key_equal(1.0, 1)
+
+* equal(expected, actual, message=nil) (alias: order_equal)
+  Description: Uses expected == actual, which is used for order-equality.
+  Success:     equal(1.0, 1)
+  Failure:     equal(1, "1")
+
+* case_equal(expected, actual, message=nil)
+  Description: Uses expected === actual, which is used in case/whens.
+  Success:     case_equal(String, "foo")
+  Failure:     case_equal(String, 1)
+
+* equal_unordered(expected, actual, message=nil)
+  Description: Compares unordered enumerables, on the enumerable it uses each, 
+               on the items it uses hash and eql?
+  Success:     equal_unordered([1,2], [2,1])
+  Failure:     equal_unordered([1,2], [2,1,2])
+
+* within_delta(a, b, delta)
+  Description: Tests whether the difference between a and b is less than delta
+  Success:     within_delta(0.5, Math.sin(Math::PI/6), 1e-6)
+               # equal(0.5, Math.sin(Math::PI/6)) would fail
+  Failure:     within_delta(0.5, 0.6, 0.05)
+
+* kind_of(expected, actual, message=nil) (alias: is_a)
+  Description: Uses actual.kind_of?(expected)
+  Success:     kind_of(String, "foo")
+  Failure:     kind_of(String, 1)
+
+* throws(symbol)
+  Description: Test whether the block throws the given symbol
+  Success:     throws(:foo) do throw(:foo) end
+  Failure:     throws(:foo) do throw(:bar) end
+               throws(:foo) do nil end
+
+* throws_nothing
+  Description: Test whether a piece of code really throws nothing
+  Success:     throws_nothing do nil end
+  Failure:     throws_nothing do throw(:foo) end
+
+* raises(exception_class=StandardError, opts={})
+  Description: Test whether the block raises
+  Success:     raises do raise "foo" end
+               raises ArgumentError do "12".to_i(10, :superfluous) end
+  Failure:     raises do nil end
+               raises ArgumentError do "12".to_i(10) end
+
+* raises_nothing
+  Description: Test whether the block doesn't raise
+  Success:     raise_nothing do nil end
+  Failure:     raise_nothing do raise "foo" end
+
+* touch(thing=nil)
+  Description: Mark reaching a point in code, e.g. that a block was invoked
+  Success:     -> see touched
+  Failure:     -> see touched
+
+* touched(thing=nil, times=nil)
+  Description: test whether a mark for reached code was set, optionally test
+               whether it was set the expected number of times
+  Success:     touch; touched
+               touch :thing; touched :thing
+               touch :thing; touch :thing; touched :thing, 2
+  Failure:     touched
+               touch :thing; touched :something
+               touch :thing; touched :thing, 2
+
+* not_touched(thing=nil)
+  Description: same as touched(thing, 0)
+  Success:     see touched
+  Failure:     see touched
+
+
+
+== Running tests
+
+baretest's test-cycle is:
+
+1. Create Run instance and the toplevel suite
+2. Load everything as required by the command line flags
+3. Load PROJECT/test/setup.rb
+4. Find PROJECT/test/suite/**/*.rb
+5. Load each file found in 2., but for every file, see whether
+   PROJECT/test/helpers/suite/**/*.rb exists and load that first if it does
+6. Invoke run on the Run instance
+
+From there on it depends on the loaded formatters and extenders, what really
+will happen. But the norm is, that suites and assertions will be executed in
+order of definition.
+
+
+
+== Debugging tests
+
+Use `baretest -i` to run baretest in interactive mode. When a failure or an
+error occurs, you have access to the following commands:
+
+* s!          - original assertions' status
+* e!          - error message and full backtrace
+* em!         - error message
+* bt!         - full backtrace
+* iv!         - all available instance variables
+* cv!         - all available class variables
+* gv!         - all available global variables
+* file        - file this assertion was defined in
+* line        - line number in the file where this assertion's definition starts
+* nesting     - a '>'-separated list of suite descriptions this assertion is
+                nested in
+* description - this assertion's description
+* code        - code of this assertion
+* help        - overview over all the commands

data/doc/writing_tests.rdoc

@@ -0,0 +1,148 @@
+= Writing Tests
+
+
+
+This tutorial assumes you have basic ruby knowledge. It is an introduction into
+baretest and testing itself.
+If you're a quick study and have already good knowledge about ruby and testing,
+you may be more interested in just reading the examples and the
+doc/quickref.rdoc.
+
+
+
+== 1. In the beginning there was the project
+
+The first step is of course the project. Baretest was written with the
+assumption of a standard ruby project layout (should work fine without too,
+though - might just require a bit more work on your part).
+The standard directory layout looks like this:
+
+  |-- bin (executables)
+  |-- doc (additional documentation)
+  |-- ext (native extension code will be here)
+  |-- examples (for the users of the lib)
+  |-- lib (contains the library)
+  |-- rake (contains rake relevant stuff)
+  `-- Rakefile
+
+In your project directory, you can invoke `baretest --init`, this will
+create the 'test' directory. It will mirror your project directory. That is,
+it will recreate all directories nested in bin and lib within test/suite.
+The directory layout of 'test' is as follows:
+
+  `-- test
+      |-- external (baretest ignores this directory)
+      |-- helper (baretest loads helper/lib/foo.rb when loading suite/lib/foo.rb)
+      |-- setup.rb (setup.rb is loaded as the first file when running baretest)
+      `-- suite (in here are the tests itself)
+          |-- bin (the tests for bin, PROJECT/bin is replicated here)
+          `-- lib (the tests for lib, PROJECT/lib is replicated here)
+
+
+
+== 2. Writing the tests
+
+Assume you have `lib/foo.rb` containing the class 'Foo'. To test it, you create
+the file `test/suite/lib/foo.rb`. You start out by creating a suite for your
+class:
+
+  BareTest.suite "class Foo" do
+  end
+
+You're in no way limited in how you name the suites. It's an arbitrary String.
+Now lets assume 'lib/foo.rb' contains the following code:
+
+  class Foo
+    def bar
+      "bar"
+    end
+  end
+
+Then follows the next step, we write the first assertion:
+
+  BareTest.suite do
+    suite "class Foo" do
+      assert "bar returns 'bar'" do
+        Foo.new.bar == 'bar'
+      end
+    end
+  end
+
+As you can see, the assertion is plain ruby code. The return value of the block
+decides whether the assertion is considered a success (trueish value, that is
+all but false and nil) or a failure (falseish value, that is false or nil).
+
+
+
+== 3. Running the tests
+
+First you change the directory to your project's root directory.
+There you run `baretest`. That's it.
+Baretest will now load the 'test/setup.rb' file, then it'll search in
+'test/suite' for files and find 'test/suite/lib/foo.rb'. Before loading that
+file, it'll see if there's also a file 'test/helpers/suite/lib/foo.rb'. If
+there was, it'd load that first. After that, it loads the
+'test/suite/lib/foo.rb' file. When all testfiles are discovered and loaded,
+it'll run the tests.
+
+
+
+== 4. Separating parts of the test
+
+A classical test consists of four phases:
+
+1. setup
+2. exercise
+3. validate
+4. teardown
+
+Baretest has setup and teardown on suites, which will be run for every assertion
+the suite contains.
+Exercise and validate is currently combined in the 'assert' method.
+
+So let's make use of that and rewrite our previous test:
+
+  BareTest.suite do
+    suite "class Foo" do
+      setup do
+        @foo = Foo.new
+      end
+
+      assert "bar returns 'bar'" do
+        @foo.bar == 'bar'
+      end
+    end
+  end
+
+In this simplistic example, this may seem like wasted time. The more complex the
+setup becomes and the more assertions need the same setup, the more time a
+separate setup phase saves.
+It additionally helps in making intent clear: this is setup, and this is test.
+
+
+== 5. When troubles strike
+
+If one of your assertions fails or errors, you can use `baretest -i` to
+investigate the issue. It will throw you into an irb session, with
+self being the failing/erroring assertion and with several helpful
+methods (use `help` in the irb session to get a list of those).
+
+== Things left to be written out
+* toplevel suite may have a name/description too, it'll act the same as if
+  there was a suite in an unnamed toplevel suite
+* [setup] They will also be run for every nested suite's assertion,
+  where the outermost setup is run first, the innermost last.
+* using stubs & mocks
+* However, suites with the same name are considered the same.
+  
+  For example, this code:
+  
+    BareTest.suite "class Foo" do
+      suite "class Bar" do
+        assert "foo"
+    end
+  
+  
+  Just like ruby's
+  namespacing works. That is, if you twice do `module X; class Y; ...; end; end`,
+  it will both times open the same class X::Y.

data/examples/test.rake

@@ -1,37 +1,65 @@
+#--
+# Copyright 2009 by Stefan Rusterholz.
+# All rights reserved.
+# See LICENSE.txt for permissions.
+#++
+
+
+
 # This rake task expects to be in PROJECT_DIR/tasks/test.rake
 # It assumes that the tests are in PROJECT_DIR/test/**/*.rb
 # This is relevant as it calculates the paths accordingly.
-# Additionally it will add PROJECT_DIR/lib - if present - to $LOAD_PATH.
+# It uses BareTest.load_standard_test_files to load setup and test files.
+# This means it will also load a test/setup.rb file if present, where
+# you can add paths to $LOAD_PATH.
 
-desc "Run testsuite. Set FORMAT env variable to change the formatter used."
-task :test do
-	begin
-		require 'test'
-	rescue LoadError => e
-		puts "Could not run tests: #{e}"
-	else
-		# Prepare paths
-		rake_file = File.expand_path(__FILE__)
-		test_dir  = File.expand_path("#{rake_file}/../../test")
-		lib_dir   = File.expand_path("#{rake_file}/../../lib")
+namespace :test do
+  desc "Information about how your test directory should look."
+  task :structure do
+    wd        = File.expand_path(Dir.getwd)
+    rake_file = File.expand_path(__FILE__)
+    test_dir  = ['./test', "#{rake_file}/../../test"].map { |path|
+      full     = File.expand_path(path)
+      relative = full[(wd.size+1)..-1]
+      "* #{relative} (#{full})"
+    }
 
-		# Verify that the test directory exists
-		raise "Could not determine test directory, please adapt this rake task to " \
-					"your directory structure first." unless File.directory?(test_dir)
+    puts "rake test:run expects to find one of the these directories:", *test_dir
+  end
 
-		# Add PROJECT_DIR/lib to $LOAD_PATH if the dir exists
-		if File.directory?(lib_dir) && !$LOAD_PATH.include?(lib_dir) then
-			$LOAD_PATH.unshift(lib_dir)
-			puts "Added '#{lib_dir}' to $LOAD_PATH" if $VERBOSE
-		end
-
-		# Load all test definitions
-		Dir.glob(File.expand_path("#{test_dir}/**/*.rb")) { |path|
-			require path
-		}
-
-		# Run all tests
-		formatter = ENV["FORMAT"] || 'cli'
-		Test.run.run(formatter)
-	end
+  desc "Run testsuite. Set FORMAT env variable to change the formatter used, INTERACTIVE to have irb mode."
+  task :run do
+    begin
+      require 'baretest'
+    rescue LoadError => e
+      puts "Could not run tests: #{e}"
+    else
+      # Prepare paths
+      rake_file = File.expand_path(__FILE__)
+      test_dir  = ["#{rake_file}/../../test", './test'].map { |path|
+        File.expand_path(path)
+      }.find { |path|
+        File.directory?(path)
+      }
+  
+      # Verify that the test directory exists
+      raise "Could not determine test directory, please adapt this rake task to " \
+            "your directory structure first (see rake test:structure)." unless test_dir
+  
+      # Load all test definitions
+      BareTest.load_standard_test_files(
+        :verbose    => $VERBOSE,
+        :setup_file => 'test/setup.rb',
+        :chdir      => File.dirname(test_dir) # must chdir to 1 above the 'test' dir
+      )
+  
+      # Run all tests
+      format      = ENV["FORMAT"] || 'cli'
+      interactive = ENV["INTERACTIVE"] == 'true'
+      BareTest.run(:format => format, :interactive => interactive)
+    end
+  end
 end
+
+desc 'Alias for test:run'
+task :test => 'test:run'

data/examples/tests/irb_mode/failures.rb

@@ -0,0 +1,26 @@
+#--
+# Copyright 2009 by Stefan Rusterholz.
+# All rights reserved.
+# See LICENSE.txt for permissions.
+#++
+
+
+
+BareTest.suite "Failure" do
+  setup do
+    @a = 1
+    @b = 2
+  end
+
+  assert "This one should fail and thus drop you into irb" do
+    c = 3
+    d = 4
+    @a == d
+  end
+
+  assert "This one should error and thus drop you into irb" do
+    c = 3
+    d = 4
+    raise "error!"
+  end
+end

data/examples/tests/mock_developer/test/setup.rb

@@ -0,0 +1,57 @@
+$LOAD_PATH.unshift(File.expand_path("#{__FILE__}/../../lib"))
+require 'baretest'
+
+BareTest.toplevel_suite.setup do
+  @_mymockname = {
+    :mocks      => [],
+    :failures   => [],
+    :exceptions => []
+  }
+end
+BareTest.toplevel_suite.teardown do
+  @_mymockname[:mocks].each { |mock|
+    mock.teardown(@_mymockname)
+  }
+  unless @_mymockname[:exceptions].empty? then
+    @reason    = "An error occurred"
+    @exception = @_mymockname[:exceptions].first
+    @status    = :error
+  end
+  unless @status == :error || @_mymockname[:failures].empty?
+    @reason = @_mymockname[:failures].first
+    @status = :failure
+  end
+end
+
+class DemoMock
+  class Error < StandardError; end
+
+  def initialize(action, message)
+    @action, @message = action, message
+  end
+
+  def teardown(recorder)
+    case @action
+      when :fail
+        recorder[:failures] << @message
+      when :raise
+        raise @message
+    end
+  rescue Error => e
+    recorder[:exceptions] << e
+  end
+end
+
+module MockSupport
+  def demo_mock(*args, &block)
+    mock = DemoMock.new(*args, &block)
+    @_mymockname[:mocks] << mock
+    mock
+  end
+end
+
+module BareTest
+  class Assertion
+    include MockSupport
+  end
+end

data/examples/tests/mock_developer/test/suite/mock_demo.rb

@@ -0,0 +1,19 @@
+BareTest.suite "MockDemo" do
+  suite "nothing happens" do
+    assert "Success" do
+      demo_mock(nil, nil)
+    end
+  end
+
+  suite "mock fails" do
+    assert "Failure" do
+      demo_mock(:fail, "this is the mock failure message")
+    end
+  end
+
+  suite "mock errors" do
+    assert "Exception" do
+      demo_mock(:raise, DemoMock::Error.new("mock errors message"))
+    end
+  end
+end

data/examples/tests/overview/test.rb

@@ -0,0 +1,89 @@
+BareTest.suite do
+  # assertions and refutations can be grouped in suites. They will share
+  # setup and teardown
+  # they don't have to be in suites, though
+  suite "Success" do
+    assert "An assertion returning a trueish value (non nil/false) is a success" do
+      true
+    end
+  end
+
+  suite "Failure" do
+    assert "An assertion returning a falsish value (nil/false) is a failure" do
+      false
+    end
+  end
+
+  suite "Pending" do
+    assert "An assertion without a block is pending"
+  end
+
+  suite "Error" do
+    assert "Uncaught exceptions in an assertion are an error" do
+      raise "Error!"
+    end
+  end
+
+  suite "Special assertions" do
+    assert "Assert a block to raise" do
+      raises do
+        sleep(rand()/3+0.05)
+        raise "If this raises then the assertion is a success"
+      end
+    end
+
+    assert "Assert a float to be close to another" do
+      a = 0.18 - 0.01
+      b = 0.17
+      within_delta a, b, 0.001
+    end
+
+    suite "Nested suite" do
+      assert "Assert two randomly ordered arrays to contain the same values" do
+        a = [*"A".."Z"] # an array with values from A to Z
+        b = a.sort_by { rand }
+        equal_unordered(a, b) # can be used with any Enumerable, uses hash-key identity
+      end
+    end
+  end
+
+  suite "Setup & Teardown" do
+    setup do
+      @foo = "foo"
+      @bar = "bar"
+    end
+
+    assert "@foo should be set" do
+      @foo == "foo"
+    end
+
+    suite "Nested suite" do
+      setup do
+        @bar = "inner bar"
+        @baz = "baz"
+      end
+
+      assert "@foo is inherited" do
+        @foo == "foo"
+      end
+
+      assert "@bar is overridden" do
+        @bar == "inner bar"
+      end
+
+      assert "@baz is defined only for inner" do
+        @baz == "baz"
+      end
+    end
+
+    teardown do
+      @foo = nil # not that it'd make much sense, just to demonstrate
+    end
+  end
+
+  suite "Dependencies", :requires => ['foo', 'bar'] do
+    assert "Will be skipped, due to unsatisfied dependencies" do
+      failure "Why the heck do you have a 'foo/bar' file?"
+    end
+  end
+end