exemplor 2010.2.0 → 3000.1.0

data/README.md

@@ -0,0 +1,221 @@
+Exemplor (the exemplar)
+=======================
+
+Introduction
+------------
+
+Exemplor is testing framework for high level integration tests that read like example usage scenarios. It's designed with the minimum possible vocabulary, so you can spend more time writing tests and less time learning the testing framework.
+
+The ideal user of exemplor is a developer who is writing their program in small pieces, from the highest level of abstraction they have currently implemented.
+
+For example lets say you were writing a command-line utility to delete all those `.DS_store` files the OS X finder generates when it's going about its business.
+
+The approach the current crop of test frameworks encourage is to write a failing test that calls a method on an instance of a class. For example:
+
+    require 'test_framework'
+    require 'my_code'
+
+    context MyMainClass do
+
+      setup do
+        # fixture setup
+      end
+
+      it "deletes .DS_store files" do
+        MyMainClass.new(@fixture_directory).method_that_deletes_files
+        assert(not File(@fixture_directory + '/.DS_store').exist?)
+        assert(File(@fixture_directory + '/other_file.txt').exist?)
+      end
+
+    end
+
+No test framework forces the programmer to start this way. These are just the
+kind of examples that come up in the documentation and the open source
+projects that make use of the frameworks.
+
+There are a number of issues with this style of testing.
+
+  1.  The test file must reference:
+
+      * The main library file `my_code.rb`
+      * The main class `MyMainClass`
+      * The primary internal entry point - initialization of MyMainClass with a directory argument and then a call to `method_that_deletes_files` instance method
+
+      If any of these names change, the test must be updated accordingly.
+
+  2.  There is one compulsory level of nesting in the test file. Even if each file only has one context, all tests must be nested under it.
+
+  3.  If the `assert` calls fail, generally the output of the framework is not sufficient enough to determine the exact cause of failure. This is often partially solved with additional methods (or macros) such as `assert_size`, `should_exist` etc. These methods just increase the amount the user must learn to start effectively using the framework.
+
+  4.  The command-line utility - the stated goal of the program - is not yet under test.
+
+The exemplor approach would be something like:
+
+    require 'exemplor'
+
+    def run_inside directory
+      system "cd #{directory} && ./path/to/utility"
+    end
+
+    eg.setup do
+      # fixture setup
+    end
+
+    eg "deletes .DS_store files" do
+      run_inside @fixture_directory
+      Assert(not File(@fixture_directory + '/.DS_store').exist?)
+      Assert(File(@fixture_directory + '/other_file.txt').exist?)
+    end
+
+This tests the public interface (the command-line interface) of the utility without any information about its internal implementation. All the test knows about is the path to the actual command under test.
+
+This approach to testing allows the developer to write the first implementation without any classes, then refactor to instance methods on a class, then put that class inside a module, then move the library file into a `lib` directory, etc. All while continuing to run the tests that ensure the program works as expected.
+
+Exemplor is not just for command-line utilities. It's for testing the parts of your application that constitute the public interface. In a web app it would be the urls. In a library, it would be the public API. Exemplor can also be used to test your underlying implementation but if you find yourself with a bunch of those tests and nothing that covers the public interface then your doing it wrong.
+
+
+API Overview
+------------
+
+The api is tiny. Here it is
+
+    eg.setup do
+      # run before each example, set instance variables here
+    end
+
+    eg "an example" do
+      # example usage of the thing under test
+      Assert(expression_that.should.be(truthy))
+      Show(something_to_inspect)
+    end
+
+    eg.helpers do
+      # helper methods that need to run the context of an example
+    end
+
+When run from a terminal, the output optimised for human readability and a high signal-to-noise ratio.
+
+When standard out points to something other than a terminal, the output is information-rich YAML.
+
+
+Writing Examples
+----------------
+
+The simplest possible example:
+
+    eg 'example without Assert() or Show() calls' do
+      "foo"
+    end
+
+Will just print:
+
+    • example without Assert() or Show() calls: foo
+
+This is useful for "printf driven development" where you just want to check that your code runs and does something useful. To print more than one value per example you can use the `Show()` method:
+
+    eg 'using Show()' do
+      list = [1, 2, 3]
+      Show(list.last)
+      list << 4
+      Show(list.first)
+      Show(list.last)
+    end
+
+prints as:
+
+    ∙ Showing the value of some expression
+      • list.first: 1
+    ∙ using Show()
+      • list.last: 3
+      • list.first: 1
+      • list.last: 4
+
+`Show()` is like a fancy `Kernel#puts`, you don't need to worry about distinguishing between different calls like you would with normal `puts`:
+
+    puts "last item: #{item.first.to_yaml}"
+    puts "last item: #{item.last.to_yaml}"
+    # etc.
+
+because `Show()` works out the label from the source code and automatically pretty prints the value as yaml.
+
+Exemplor has only one kind of assertion: `Assert()`. It works like `Show()` in that the label will be read from the source:
+
+    eg 'using Assert()' do
+      list = [1, 2, 3]
+      Show(list.last)
+      Assert(list.last == 3)
+    end
+
+prints:
+
+    ∙ using Assert()
+      • list.last: 3
+      ✓ list.last == 3
+
+If the example contains no `Show()` calls and all the asserts are successful then the entire example is considered successful:
+
+    eg 'using Assert()' do
+      list = [1, 2, 3]
+      Assert(list.first == 1)
+      Assert(list.last == 3)
+    end
+
+prints:
+
+    ✓ using Assert()
+
+If an assertion fails then the name of the test is printed with an ✗ next to it:
+
+    eg 'The second Assert() will fail' do
+      list = [1, 2, 3]
+      Assert(list.first == 1)
+      Assert(list.last == 1)
+    end
+
+prints:
+
+    ✗ The second Assert() will fail
+      ✓ list.first == 1
+      ✗ list.last == 1
+
+Nothing fancy, no "expected" and "actual" values are printed, if you want to inspect those you can just add a `Show()` call before the assert.
+
+
+Running Examples
+----------------
+
+Run the example file through ruby
+
+    $> ruby examples.rb
+
+To run only examples that match the regex "location | setting/x"
+
+    $> ruby examples.rb "location | setting/x"
+
+Running with `--list` or `-l` lists all examples:
+
+    $> ruby examples.rb -l
+    - errors are caught and nicely displayed
+    - check_output_matches_expected_for :no_checks
+    - check_output_matches_expected_for :oneliner
+    - check_output_matches_expected_for :no_checks_non_string
+    - check_output_matches_expected_for :with_checks
+    - check_output_matches_expected_for :check_with_disambiguation
+    - check_output_matches_expected_for :assertion_success
+    - check_output_matches_expected_for :assertion_failure
+    - check_output_matches_expected_for :assertion_success_and_failure
+    - check_output_matches_expected_for :helpers
+    - check_output_matches_expected_for :with_setup
+    - called with --list arg
+    - called with --l arg
+    - called with some other arg (always interpreted as a regex)
+
+
+Thanks
+------
+
+Exemplor was inspired by [testy](http://github.com/ahoward/testy).
+
+What really kicked me over the line to this style of testing and eventually lead to writing exemplor was Yehuda Katz's "Writing Code That Doesn't Suck" presentation at RubyConf 2008. I wasn't there but luckily an excellent video of the presentation is available online[1]. If none of this readme made any sense to you or you have additional questions/concerns I strongly recommend watching that presentation.
+
+[1] http://rubyconf2008.confreaks.com/writing-code-that-doesnt-suck.html

data/Rakefile

@@ -8,16 +8,20 @@ begin
     gs.summary  = "A light-weight, low-fi way to provide executable usage examples of your code."
     gs.email    = "myles@myles.id.au"
     gs.authors  = ["Myles Byrne"]
-    gs.add_dependency('orderedhash', '>= 0.0.6')
-    gs.add_dependency('term-ansicolor', '>= 1.0.3')
   end
   Jeweler::GemcutterTasks.new
 rescue LoadError
   puts "Install jeweler to build gem"
 end
 
-task :examples do
-  ruby  "-rubygems", "examples.rb"
+desc "runs the examples with the rubygems version of exemplor (you must gem install the gem for this to work)"
+task :examples, [:filter] do |_,args|
+  ruby "-rubygems", "examples.rb", (args.filter || '')
+end
+
+desc "runs the examples with the development version (i.e. the one in this dir) of exemplor"
+task :dev, [:filter] do |_,args|
+  ruby '-rubygems', '-I', 'lib', 'examples.rb', (args.filter || '')
 end
 
 task :test => :examples

data/TODO

@@ -1,7 +1,14 @@
-* add symbol to_proc and tap
-* exemplor executable that can run multiple example files
-* TextMate bundle that does nice cmd+r output
-* smarter macros for examples.rb - this could be clearer
-* discuss example grouping in doc (philosophy)
+Stuff that's on my radar:
 
-* test requiring exemplor does *not* trigger at exit
+* Halt()
+* Run Assert() and Show() checks should appear before the error report
+* --version arg
+
+* Documentation on how to use with rack
+* Documentation on why the one-context-per-file approach is used
+
+* An exemplor executable that can run multiple example files
+
+* test stderr capturing
+* adding symbol to_proc and tap to ext.rb
+* test requiring an examples file from another file does *not* trigger at_exit

data/VERSION

@@ -1 +1 @@
-2010.2.0
+3000.1.0

data/examples.rb

@@ -1,74 +1,85 @@
-# uses the gem version, not the one being tested
 require 'exemplor'
 
-# slow because each test runs in a subshell
+# Each test runs in a subshell, exemplor is tested with exemplor but from
+# a version running in a different process. Exemplor hates unit tests.
 
-eg.helpers do
+eg "Exemplor.version comes from the version file" do
+  version = `ruby -Ilib -e "require 'exemplor' ; print Exemplor.version"`
+  version_from_file = File.read(__FILE__.sub('examples.rb','VERSION'))
+  Assert(version == version_from_file)
+end
 
-  def run_example(name, args = nil)
-    `ruby -rubygems -Ilib examples/#{name}.rb#{' ' + args if args}`
-  end
+# runs an example file (in /examples) using the development version of exemplor
+def run_example(name, args = nil)
+  `ruby -Ilib examples/#{name}.rb#{' ' + args if args}`
+end
 
-  def extract_expected(name)
-    File.read("examples/#{name}.rb").split('__END__').last
-  end
+# pulls out text after the __END__ in an example file
+def expected_output_for name
+  File.read("examples/#{name}.rb").split('__END__').last.lstrip + "\n"
+end
 
-  def expected_and_actual(example_name)
-    [extract_expected(example_name).strip, run_example(example_name).strip]
+# a macro that runs an example file and then asserts that the output matches
+# the expected output which is specified after the __END__ in that same file
+def examples filenames
+  filenames.each do |file|
+    eg("#{file}.rb") { Assert(run_example(file) == expected_output_for(file)) }
   end
+end
 
-  def check_output_matches_expected_for(example_name)
-    expected_output, output = expected_and_actual(example_name)
-    Check(output).is(expected_output)
-  end
+examples %w[
+  no_checks
+  no_checks_non_string
 
-end
+  simple_show
+  multi_show
+  show_with_disambiguation
 
-eg "version matches file" do
-  version = `ruby -rubygems -Ilib -e "require 'exemplor' ; print Exemplor.version"`
-  Check(version).is(File.read(__FILE__.sub('examples.rb','VERSION')))
-end
+  assertion_success
+  assertion_failure
+  assertion_success_and_failure
+  assertion_success_and_info
+  failure_halts_execution
+
+  helpers
+  with_setup
+  checking_nil
+  showing_classes
+  check_parsing
+]
+
+# I never use this guy, candidate for removal
+examples %w[oneliner]
 
-eg "errors are caught and nicely displayed" do
-  result = YAML.load(run_example(:an_error))[0]
-  Check(result['status']).is('error')
-  Check(result['result']['class']).is('RuntimeError')
-  Check(result['result']['message']).is('boom!')
-  Check(result['result']['backtrace'][0]).is('examples/an_error.rb:4')
+eg.helpers do
+  # Exemplor outputs valid yaml, for some of our assertions it's easier to use
+  # the parsed structure
+  def parse_run *args
+    YAML.load(run_example(*args))
+  end
 end
 
-eg { check_output_matches_expected_for :no_checks }
-eg { check_output_matches_expected_for :oneliner }
-eg { check_output_matches_expected_for :no_checks_non_string }
-eg { check_output_matches_expected_for :with_checks }
-eg { check_output_matches_expected_for :check_with_disambiguation }
-eg { check_output_matches_expected_for :assertion_success }
-eg { check_output_matches_expected_for :assertion_failure }
-eg { check_output_matches_expected_for :assertion_success_and_failure }
-eg { check_output_matches_expected_for :assertion_success_and_info }
-eg { check_output_matches_expected_for :failure_halts_execution }
-eg { check_output_matches_expected_for :helpers }
-eg { check_output_matches_expected_for :with_setup }
-eg { check_output_matches_expected_for :checking_nil }
-eg { check_output_matches_expected_for :dumping_classes }
-eg { check_output_matches_expected_for :check_parsing }
+eg "errors are caught and backtraces shown" do
+  result = parse_run(:an_error)[0]
+  Assert(result['status'] == 'error')
+  Assert(result['result']['class'] == 'RuntimeError')
+  Assert(result['result']['message'] == 'boom!')
+  Assert(result['result']['backtrace'][0] == 'examples/an_error.rb:4')
+end
 
 eg "exit status is percent of issues that failed or errored" do
   run_example :ten_percent_failures
-  Check($?.exitstatus).is(10)
+  Assert($?.exitstatus == 10)
 end
 
-eg "called with --list arg" do
-  list = YAML.load(run_example(:with_setup, '--list'))
-  Check(list).is(["Modified env", "Unmodified env"])
+eg "--list shows all the example names in the file" do
+  Assert(parse_run(:foobar, '--list') == ["foo", "bar"])
 end
 
-eg "called with --l arg" do
-  list = YAML.load(run_example(:with_setup, '--list'))
-  Check(list).is(["Modified env", "Unmodified env"])
+eg "-l is the same as --list" do
+  Assert(run_example(:foobar, '-l') == run_example(:foobar, '--list'))
 end
 
-eg "called with some other arg (always interpreted as a regex)" do
-  tests_run = YAML.load(run_example(:with_setup, 'Unmodified')).size
-  Check(tests_run).is(1)
+eg "any other arg is intepreted as a regex and the examples that match it are run" do
+  Assert(parse_run(:foobar, 'foo').size == 1)
 end

data/examples/assertion_failure.rb

@@ -2,7 +2,7 @@ require 'exemplor'
 
 eg 'Assertion failure' do
   list = [1, 2, 3]
-  Check(list.first).is(2)
+  Assert(list.first == 2)
 end
 
 __END__
@@ -10,7 +10,5 @@ __END__
 - name: Assertion failure
   status: failure
   result: 
-  - name: list.first
-    status: failure
-    expected: 2
-    actual: 1
+  - name: list.first == 2
+    status: failure

data/examples/assertion_success.rb

@@ -2,7 +2,7 @@ require 'exemplor'
 
 eg 'Asserting first is first' do
   list = [1, 2, 3]
-  Check(list.first).is(1)
+  Assert(list.first)
 end
 
 __END__
@@ -11,5 +11,4 @@ __END__
   status: success
   result: 
   - name: list.first
-    status: success
-    result: 1
+    status: success

data/examples/assertion_success_and_failure.rb

@@ -2,10 +2,10 @@ require 'exemplor'
 
 eg 'Some successes, then a fail' do
   list = [1, 2, 3]
-  Check(list.first).is(1)
-  Check(list[1]).is(2)
-  Check(list.last).is(1) # fail!
-  Check(list[2]).is(3) # would be successful but we never get here
+  Assert(list.first == 1)
+  Assert(list[1] == 2)
+  Assert(list.last == 1) # fail!
+  Assert(list[2] == 3) # would be successful but we never get here
 end
 
 __END__
@@ -13,13 +13,9 @@ __END__
 - name: Some successes, then a fail
   status: failure
   result: 
-  - name: list.first
+  - name: list.first == 1
     status: success
-    result: 1
-  - name: list[1]
+  - name: list[1] == 2
     status: success
-    result: 2
-  - name: list.last
-    status: failure
-    expected: 1
-    actual: 3
+  - name: list.last == 1
+    status: failure