tryouts 0.6.3 → 0.7.0

data/CHANGES.txt

@@ -1,6 +1,22 @@
 TRYOUTS, CHANGES
 
 
+#### 0.6.4 (2009-06-25) ###############################
+
+NOTE: command testing (:cli) is still disabled.
+
+* FIXED: Stash wasn't being displayed in the drill report
+* CHANGE: CLI is now formatted to be 80 characters wide. 
+* ADDED: Tryouts::Drill::Sergeant::Benchmark
+* ADDED: Tryouts::Stats
+* ADDED: new dream formats: :mean and :sdev which compare with <=.
+   dream 10, :mean      
+   drill "do something" do; ...; end  # stats.mean <= 10 
+* ADDED: new dream format: :proc
+    dream lambda { |x| x.real < 0.1 }, :proc
+    drill("array sort!") { @@array.dup.sort! }
+
+
 #### 0.6.3 (2009-06-25) ###############################
 
 NOTE: command testing (:cli) is still disabled.

data/README.rdoc

@@ -1,47 +1,104 @@
-= Tryouts - v0.6 BETA
+= Tryouts - v0.7 BETA
 
-Tryouts is a high-level testing library for your command-line applications and Ruby codes.
+Tryouts is a high-level testing library (DSL) for your Ruby codes.
 
-NOTE: The DSL syntax has changed significantly from 0.5 and command-line app tests are currently disabled. 
-
-== Syntax
-
-Tryouts is a bit different than other testing libraries. Test definitions are organized in a similar way as Shoulda tests (although the keywords in the syntax are different). 
+A tryout is made up of one of more drills. The return value of the drill block is compared to the expectations defined by one or more dreams. The goal is to achieve your dreams. 
 
 === Terminology
 
-* Tryout: a set of drills (just like basketball tryouts is a set of drills)
+* Tryout: a set of drills (think: basketball tryouts)
 * Drill: a single test. 
-* Dream: the expected outcome of a drill. There's always one or more dream per drill. 
+* Drill Sergeant: The class responsible for executing a drill.
+* Dream: the expected outcome of a drill. There's always one or more dream per drill.
+
+== Installation ==
 
-== Testing Ruby codes (an API)
+{{{
+    $ gem install tryouts
+}}}
 
-Tryouts employs the same approach for testing Ruby codes. The return value of the drill block is compared to the expectation defined by the dream. Here is an example of including dreams inside the tryout definition. 
+== Examples
+
+The examples below are a complete overview of Tryouts syntax. 
+
+=== Testing Ruby Codes (:api)
 
     library :caesars, "../path/to/caesars/lib"
 
-    tryout "Common Usage" do
-      dream 3
-      drill "Check the return value" do
+    tryouts "Common Usage" do
+    
+      # This drill block should return 3.
+      drill "Maths R Us", 3 do
         12 / 4
       end
-        
-      dream :Proc, :class
-      drill "Check the return value class" do
-        Proc.new do
-          :anything
-        end
+      
+      # You can specify a method to execute
+      # on the return value of the drill block. 
+      drill "We want a symbol", Symbol, :class do
+        orange.methods.first
       end
       
+      # Dreams can also be specified explicitly which is
+      # important b/c it's possible to specify multiple.
+      dream Array, :class
+      dream [:a, :b, :c]
+      drill "Should return a list of 3" do
+        Letters.list(3)
+      end
+      
+      # Drills can pass based on an exception too.
       dream NameError, :exception
-      drill "A test can pass based on the exception" do
+      drill "Something failed" do
         raise NameError
       end
       
-      dream /\w\d\w \d\w\d/, :match
-      drill "Knows where Santa Claus lives", 'H0H 0H0'
+      # We can even put simple drills on a single line. 
+      drill "Santa's postal code", 'H0H 0H0', /\w\d\w \d\w\d/, :match
+      
     end
 
+=== Benchmarks (:benchmark)
+
+You can also use Tryouts to run benchmarks. The tryouts method takes a second parameter which specifies which drill sergeant to use. Below we specify <tt>:benchmark</tt> so each drill is timed and executed multiple times.
+
+   tryouts "Benchmark examples", :benchmark do
+     
+     # We create an Array and store it in a class
+     # variable so it's available to other drills.
+     drill "Create test array" do
+       @@array = (1..100000).map { rand }
+     end
+     
+     dream 3.0, :mean        # The mean should be <= 3.0 seconds
+     dream 0.1, :sdev        # and the standard deviation <= 0.1
+     drill("Array sort!") { @@array.dup.sort! }
+     
+     # You can also include a dream inline
+     drill("Array sort", 3.0, :mean) { @@array.dup.sort }
+     
+     # The 3rd argument is the number of times to 
+     # execute the drill block. The mean and sdev
+     # are calculate based on all iterations. The
+     # default is 5 and the maximum is 30.
+     dream 0.1, :sdev, 15    
+     drill("Array sort") { @@array.dup.sort }
+      
+   end
+
+The drill blocks in a benchmark Tryout return Tryouts::Stats objects. See: Tryouts::Drill::Sergeant::Benchmark
+
+== Screenshots
+
+Here is an example of Tryouts output from a gibbler[http://github.com/delano/gibbler/blob/gibbler-0.2.1/tryouts/gibbler_tryouts.rb] tryout:
+
+http://github.com/delano/tryouts/raw/gh-pages/screens/tryouts-1-failure.png
+
+The drill that failed looks like this:
+
+    dream :to_gibble, :respond_to?
+    dream 'ab33b9dec202d136d0e695a3a7b06ee678134882', :to_gibble 
+    drill "Array", Array
+
 
 == BETA Notice
 
@@ -53,22 +110,28 @@ This library is very new (est. 2009-05-19) and has not been vetted by the scruti
 
 == On Threads
 
-Tryouts does some funky stuff to make it simple to write tests. This "funky stuff" means that this library is *not thread-safe at definition-time*. However, once all tryouts files are parsed (or in OO-syntax, once all objects are created), this class should be *thread-safe at drill-time*. 
+Tryouts does some funky stuff to make it simple to write tests. This "funky stuff" means that this library is <em>not thread-safe at definition-time</em>. However, once all tryouts files are parsed (or in OO-syntax, once all objects are created), this class should be <em>thread-safe at drill-time</em>. 
+
 
 == More Info
 
 * Check out the sourcecodes[http://github.com/delano/tryouts]
 * Read the rdocs[http://tryouts.rubyforge.org/]
 * About Solutious[http://solutious.com/about/]
+* Inspiration[http://www.youtube.com/watch?v=iB9nhyosDVs]
+
 
 == Thanks
 
-* Everyone at Utrecht.rb and Amsterdam.rb
+* Everyone at Utrecht.rb and Amsterdam.rb for putting up with my Ruby questions :]
+* Sam Aaron (http://sam.aaron.name) for the early feedback. 
+
 
 == Credits
 
 * Delano (@solutious.com)
 
+
 == Related Projects
 
 * Context[http://github.com/jeremymcanally/context/tree/master]

data/lib/tryouts/cli/run.rb

@@ -47,13 +47,13 @@ class Run < Drydock::Command
     
     passed, failed = 0, 0
     Tryouts.instances.each_pair do |group,tryouts_inst|
-      puts '', ' %-60s'.att(:reverse) % group  unless Tryouts.verbose < 0
+      puts '', ' %-80s'.att(:reverse) % group  unless Tryouts.verbose < 0
       puts "  #{tryouts_inst.paths.join("\n  ")}" if Tryouts.verbose > 0
       tryouts_inst.tryouts.each_pair do |name,to|
         begin
           to.run
           to.report
-        rescue SyntaxError, LoadError, Exception,
+        rescue SyntaxError, LoadError, Exception, TypeError,
                RuntimeError, NoMethodError, NameError => ex
           tryouts_inst.errors << ex
         end
@@ -63,12 +63,13 @@ class Run < Drydock::Command
       end
       
       unless tryouts_inst.errors.empty?
-        title = '%-61s' % " RUNTIME ERRORS !?"
-        puts $/, title.color(:red).att(:reverse).bright
+        title = '%-79s' % " RUNTIME ERRORS !?"
+        puts $/, ' ' << title.color(:red).att(:reverse).bright
         tryouts_inst.errors.each do |ex|
-          trace = Tryouts.verbose > 0 ? ex.backtrace : [ex.backtrace.first]
-          puts '%12s: %s (%s)' % ["error", ex.message.inspect, ex.class]
-          puts '%12s: %s' % ["trace", trace.join($/ + ' '*14)]
+          trace = Tryouts.verbose > 1 ? ex.backtrace : [ex.backtrace.first]
+          puts '%14s: %s' % [ex.class, ex.message.to_s.split($/).join($/ + ' '*16)]
+          puts
+          puts '%14s  %s' % ["", trace.join($/ + ' '*16)]
           puts 
         end
       end

data/lib/tryouts/drill/response.rb

@@ -27,6 +27,11 @@ class Tryouts::Drill
         when :match
           reality.output.respond_to?(:match) &&
           !reality.output.match(dream.output).nil?
+        when :proc
+          dream.output.is_a?(Proc) &&
+          reality.comparison_value(dream) == dream.comparison_value
+        when :mean, :sdev
+          reality.comparison_value(dream) <= dream.comparison_value
         when :gt
           reality.output > dream.output
         when :gte
@@ -66,8 +71,13 @@ class Tryouts::Drill
       
       begin
         case dream.format
+        when :proc
+          test = dream.output
+          test.arity > 0 ? "Proc.call(reality) == true" : "Proc.call == true"
         when :exception
           "#{reality.etype} == #{dream.output}"
+        when :mean, :sdev
+          "#{reality.comparison_value(dream)} <= #{dream.output}"
         when :match
           "#{reality.output.inspect}.match(#{dream.output.inspect})"
         when :gt, :gte, :lt, :lte, :ne
@@ -139,6 +149,11 @@ class Tryouts::Drill
     def comparison_value
       return @ret unless @ret.nil?
       @ret = case @format
+      when :gt, :gte, :lt, :lte, :ne
+        op = {:gt=>'>',:gte=>'>=', :lt=>'<', :lte => '<=', :ne => '!='}.find { |i| i[0] == @format }
+        @output
+      when :proc
+        true
       when :respond_to?, :is_a?, :kind_of?
         true
       else 
@@ -168,6 +183,9 @@ class Tryouts::Drill
     
     def comparison_value(dream)
       case dream.format
+      when :proc
+        test = dream.output
+        (test.arity > 0 ? test.call(@output) : test.call)
       when :exception
         @etype
       when :respond_to?, :is_a?, :kind_of?

data/lib/tryouts/drill.rb

@@ -12,6 +12,8 @@ class Tryouts
   require 'tryouts/drill/response'
   require 'tryouts/drill/sergeant/cli'
   require 'tryouts/drill/sergeant/api'
+  require 'tryouts/drill/sergeant/benchmark'
+  require 'tryouts/drill/sergeant/rbenchmark'
   
   class NoSergeant < Tryouts::Exception; end
   class UnknownFormat < Tryouts::Exception; end
@@ -29,27 +31,40 @@ class Tryouts
   attr_reader :dreams
     # A Reality object (the actual output of the test)
   attr_reader :reality
-      
+  
+  @@valid_dtypes = [:cli, :api, :benchmark]
+  
   def initialize(name, dtype, *args, &drill)
     @name, @dtype, @drill, @skip = name, dtype, drill, false
     @dreams = []
-    if @dtype == :cli
+    case @dtype 
+    when :cli
       @sergeant = Tryouts::Drill::Sergeant::CLI.new *args
-    elsif @dtype == :api
+    when :api
       default_output = drill.nil? ? args.shift : nil
       @sergeant = Tryouts::Drill::Sergeant::API.new default_output
       @dreams << Tryouts::Drill::Dream.new(*args) unless args.empty?
-    elsif @dtype == :skip
+    when :benchmark
+      default_output, format, reps = *args 
+      @sergeant = Tryouts::Drill::Sergeant::Benchmark.new reps || 1
+      @dreams << Tryouts::Drill::Dream.new(Tryouts::Stats, :class)
+      unless default_output.nil?
+        @dreams << Tryouts::Drill::Dream.new(default_output, format)
+      end
+    when :skip
       @skip = true
     else
       raise NoSergeant, "Weird drill sergeant: #{@dtype}"
     end
+    @clr = :red
     # For CLI drills, a block takes precedence over inline args. 
     # A block will contain multiple shell commands (see Rye::Box#batch)
     drill_args = [] if dtype == :cli && drill.is_a?(Proc)
     @reality = Tryouts::Drill::Reality.new
   end
   
+  def self.valid_dtype?(t); @@valid_dtypes.member?(t); end
+  
   def skip?; @skip; end
   
   def run(context=nil)
@@ -69,6 +84,69 @@ class Tryouts
     self.success?
   end
   
+  def flag
+    if skip?
+      "SKIP"
+    elsif success? 
+      "PASS".color(@clr).bright 
+    else
+      note = @dreams.empty? ? '[nodream]' : ''
+      "FAIL #{note}".color(@clr).bright
+    end
+  end
+    
+  def info
+    out = StringIO.new
+    if Tryouts.verbose > 1
+      if @dreams.empty?
+        out.puts '%6s%s'.color(@clr) % ['', @reality.output.inspect]
+      else
+        @dreams.each do |dream|
+          if dream != @reality
+            out.puts '%6s%s'.color(:red) % ['', @reality.output.inspect]
+          else
+            out.puts '%6s%s'.color(:green) % ["", dream.test_to_string(@reality)]
+          end
+        end
+      end
+    elsif Tryouts.verbose > 0
+      out.puts '%6s%s'.color(@clr) % ['', @reality.output.inspect]
+      unless @reality.stash.empty?
+        @reality.stash.each_pair do |n,v|
+          out.puts '%18s: %s'.color(@clr) % [n,v.inspect]
+        end
+        out.puts
+      end
+    end
+    out.rewind
+    out.read
+  end
+  
+  def report
+    return if skip?
+    out = StringIO.new
+    
+    @dreams.each do |dream|
+      next if dream == reality #? :normal : :red 
+      out.puts '%12s: %s'.color(@clr) % ["drill", dream.test_to_string(@reality)]
+      out.puts '%12s: %s' % ["returned", @reality.comparison_value(dream).inspect]
+      out.puts '%12s: %s' % ["expected", dream.comparison_value.inspect]
+      out.puts
+    end
+    
+    unless @reality.error.nil?
+      out.puts '%14s: %s' % [@reality.etype, @reality.error.to_s.split($/).join($/ + ' '*16)]
+    end
+    unless @reality.trace.nil?
+      trace = Tryouts.verbose > 1 ? @reality.trace : [@reality.trace.first]
+      out.puts '%14s  %s' % ['', trace.join($/ + ' '*16)]
+      out.puts
+    end
+    
+    out.rewind
+    out.read
+  end
+  
   def success?
     return false if @dreams.empty? && @reality.output != true
     begin
@@ -77,6 +155,7 @@ class Tryouts
       puts ex.message, ex.backtrace if Tryouts.debug?
       return false
     end
+    @clr = :green
     true
   end
   

data/lib/tryouts/tryout.rb

@@ -30,11 +30,10 @@ class Tryouts
     # A Hash of Dream objects for this Tryout. The keys are drill names. 
   attr_reader :dream_catcher
   
-  @@valid_dtypes = [:cli, :api]
      
   def initialize(name, dtype, command=nil, *args)
     raise "Must supply command for dtype :cli" if dtype == :cli && command.nil?
-    raise "#{dtype} is not a valid drill type" if !@@valid_dtypes.member?(dtype)
+    raise "#{dtype} is not a valid drill type" if !Drill.valid_dtype?(dtype)
     @name, @dtype, @command = name, dtype, command
     @drills, @dream_catcher = [], []
     @passed, @failed, @skipped = 0, 0, 0
@@ -56,36 +55,19 @@ class Tryouts
   # Execute all Drill objects
   def run
     DrillContext.module_eval &setup if setup.is_a?(Proc)
-    puts Tryouts::TRYOUT_MSG.bright % @name unless Tryouts.verbose < 0
+    puts "\n  %s ".bright % @name unless Tryouts.verbose < 0
     @drills.each do |drill|
-      print Tryouts::DRILL_MSG % drill.name unless Tryouts.verbose < 0
+      print '   %-70s ' % "\"#{drill.name}\"" unless Tryouts.verbose < 0
+      drill.run DrillContext.new
       if drill.skip?
-        puts "SKIP" if Tryouts.verbose >= 0
-        puts if Tryouts.verbose > 0
         @skipped += 1
-        next
-      end
-      drill.run DrillContext.new
-      drill.success? ? @passed += 1 : @failed += 1
-      next if Tryouts.verbose < 0
-      note = drill.dreams.empty? ? '[nodream]' : ''
-      c = drill.success? ? :green : :red
-      puts drill.success? ? "PASS".color(c).bright : "FAIL #{note}".color(c).bright
-      if Tryouts.verbose > 1
-        if drill.dreams.empty?
-          puts '%6s%s'.color(c) % ['', drill.reality.output.inspect]
-        else
-          drill.dreams.each do |dream|
-            if dream != drill.reality
-              puts '%6s%s'.color(c) % ['', drill.reality.output.inspect]
-            else
-              puts '%6s%s'.color(:green) % ["", dream.test_to_string(drill.reality)]
-            end
-          end
-        end
-      elsif Tryouts.verbose > 0
-        puts '%6s%s'.color(c) % ['', drill.reality.output.inspect]
+      elsif drill.success?
+        @passed += 1
+      else
+        @failed += 1
       end
+      puts drill.flag                           # PASS, FAIL, SKIP
+      puts drill.info if Tryouts.verbose > 0 && !drill.skip?  
     end
     DrillContext.module_eval &clean if clean.is_a?(Proc)
   end
@@ -93,40 +75,12 @@ class Tryouts
   # Prints error output. If there are no errors, it prints nothing. 
   def report
     return if Tryouts.verbose < 0
-    return true if success?
     failed = @drills.select { |d| !d.skip? && !d.success? }
     failed.each_with_index do |drill,index|
-      dreams, reality = drill.dreams, drill.reality
-      
-      unless dreams.empty?
-        title = ' %-51s %2d/%-2d ' % [drill.name, index+1, failed.size]
-        puts $/, ' ' << title.color(:red).att(:reverse)
-                
-        drill.reality.stash.each_pair do |n,v|
-          puts '%14s: %s' % [n,v.inspect]
-        end
-        
-        dreams.each do |dream|
-          next if dream == reality #? :normal : :red 
-          puts '%12s: %s'.color(:red) % ["test", dream.test_to_string(drill.reality)]
-          puts '%12s: %s' % ["returned", reality.comparison_value(dream).inspect]
-          puts '%12s: %s' % ["expected", dream.comparison_value.inspect]
-          puts
-        end
-        
-      end
-      
-      unless reality.error.nil?
-        puts '%12s: %s (%s)' % ["error", reality.error.inspect, reality.etype]
-      end
-      unless reality.trace.nil?
-        trace = Tryouts.verbose > 1 ? reality.trace : [reality.trace.first]
-        puts '%12s: %s' % ["trace", trace.join($/ + ' '*14)]
-        puts
-      end
-      
+      title = ' %-70s %2d/%-2d  ' % ["\"#{drill.name}\"", index+1, failed.size]
+      puts $/, ' ' << title.color(:red).att(:reverse)
+      puts drill.report
     end
-    false
   end
   
   # Did every Tryout finish successfully?

data/lib/tryouts.rb

@@ -32,18 +32,15 @@ class Tryouts
   # Raised when there is a problem loading or parsing a Tryouts::Drill::Dream object
   class BadDreams < Exception; end
   
-  VERSION = "0.6.3"
+  VERSION = "0.7.0"
   
   require 'tryouts/mixins'
   require 'tryouts/tryout'
   require 'tryouts/drill'
+  require 'tryouts/stats'
   
   require 'tryouts/orderedhash'
   HASH_TYPE = (RUBY_VERSION =~ /1.9/) ? ::Hash : Tryouts::OrderedHash
-  
-  TRYOUT_MSG = "\n  %s "
-  DRILL_MSG  = '    %-50s '
-  DRILL_ERR  = '    %s: '
 
     # An Array of +_tryouts.rb+ file paths that have been loaded.
   @@loaded_files = []
@@ -142,7 +139,7 @@ class Tryouts
     $LOAD_PATH.unshift path unless path.nil?
     begin
       require @library.to_s
-    rescue SyntaxError, LoadError, Exception,
+    rescue SyntaxError, LoadError, Exception, TypeError, 
            RuntimeError, NoMethodError, NameError => ex
       @errors << ex
       Tryouts.failed = true
@@ -191,7 +188,7 @@ class Tryouts
     # Process the rest of the DSL
     begin
       to.from_block block if block
-    rescue SyntaxError, LoadError, Exception,
+    rescue SyntaxError, LoadError, Exception, TypeError,
            RuntimeError, NoMethodError, NameError => ex
       @errors << ex
       Tryouts.failed = true
@@ -252,7 +249,7 @@ class Tryouts
         to.instance_eval file_content, fpath
       end
       to.paths << fpath
-    rescue SyntaxError, LoadError, Exception,
+    rescue SyntaxError, LoadError, Exception, TypeError,
            RuntimeError, NoMethodError, NameError => ex
       to.errors << ex
       Tryouts.failed = true

data/tryouts.gemspec

@@ -1,7 +1,7 @@
 @spec = Gem::Specification.new do |s|
 	s.name = "tryouts"
   s.rubyforge_project = "tryouts"
-	s.version = "0.6.3"
+	s.version = "0.7.0"
 	s.summary = "Tryouts are high-level tests for your Ruby code. May all your dreams come true!"
 	s.description = s.summary
 	s.author = "Delano Mandelbaum"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: tryouts
 version: !ruby/object:Gem::Version 
-  version: 0.6.3
+  version: 0.7.0
 platform: ruby
 authors: 
 - Delano Mandelbaum