rutema 1.3.0 → 2.0.0.pre

data/README.md

@@ -1,62 +1,56 @@
-[![Build Status](https://secure.travis-ci.org/damphyr/rutema.png)](http://travis-ci.org/damphyr/rutema) rutema [http://patir.rubyforge.org/rutema](http://patir.rubyforge.org/rutema)
-
-rutema is a test execution tool.
-It allows the  combination of  various test tools while it takes care of logging, reporting, archiving of results and formalizes execution of automated and manual tests.
-It's purpose is to make testing in heterogeneous environments easier. 
-
-###Why?
-Require consistency, repeatability and reliability from your test infrastructure while gathering data on every run.
-
-Whether running through a checklist of manual steps, or executing a sequence of fully automated commands we always want in the end to know if a test has failed, where it failed and what was the state of the system at that time.
-
-rutema will gather all logs, timestamp them, store them and report on them. 
-
-Using one of the database reporters we can then extract more information on the quality and state of our system by examining the behaviour of the tests over time.
-
-For more information look at [http://patir.rubyforge.org/rutema](http://patir.rubyforge.org/rutema)
-
-###The dry stuff
-* Unified test execution environment for automated and manual tests
-* Extensible reports and notifications in various formats (email, rss, pdf, html etc.)
-* Comprehensive history of test execution
-* A well defined way to create a project specific test specification format
-
-## Synopsis:
-See http://patir.rubyforge.org/rutema/examples.html for an introductory example.
-
-## Installation:
-* gem install rutema
-
-## Dependencies
-The core functionality of rutema depends on the following gems:
- * [patir](http://github.com/damphyr/patir)
- * [highline](http://highline.rubyforge.org/)
-
-Depending on which parser or reporter is used though, further dependencies might be needed.
-The reporters included in the gem depend on 
- * activerecord used by rutema/reporters/activerecord
- * [mailfactory](http://mailfactory.rubyforge.org/) used by rutema/reporters/email
-
-## License:
-(The MIT License)
-
-Copyright (c) 2007-2012 Vassilis Rizopoulos
-
-Permission is hereby granted, free of charge, to any person obtaining
-a copy of this software and associated documentation files (the
-'Software'), to deal in the Software without restriction, including
-without limitation the rights to use, copy, modify, merge, publish,
-distribute, sublicense, and/or sell copies of the Software, and to
-permit persons to whom the Software is furnished to do so, subject to
-the following conditions:
-
-The above copyright notice and this permission notice shall be
-included in all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
-EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
-IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
-CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
-TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+##rutema
+[![Build Status](https://secure.travis-ci.org/damphyr/rutema.png)](http://travis-ci.org/damphyr/rutema) [![Coverage Status](https://coveralls.io/repos/damphyr/rutema/badge.svg)](https://coveralls.io/r/damphyr/rutema) [![Code Climate](https://codeclimate.com/github/damphyr/rutema.png)](https://codeclimate.com/github/damphyr/rutema)
+
+rutema [http://github.com/damphyr/rutema](http://github.com/damphyr/rutema)
+
+rutema is a test execution tool and a framework for organizing and managing test execution across different tools.
+
+It enables the combination of different test tools while it takes care of logging, reporting, archiving of results and formalizes execution of automated and manual tests.
+
+It's purpose is to make testing in heterogeneous environments easier. 
+
+###Why?
+Require consistency, repeatability and reliability from your test infrastructure while gathering data on every run.
+
+Whether running through a checklist of manual steps, or executing a sequence of fully automated commands we always want to know if a test has failed, where it failed and what was the state of the system at that time.
+
+rutema will gather all logs, timestamp them, store them and report on them. 
+
+Rutema core provides a reference implementation of a parser for a simple but extensible XML test specification format which works well out of the box but the framework provides clearly defined interfaces so you can write the parser for your own format and add reporters that log wherever is needed.
+
+###The dry stuff
+* Unified test execution environment for automated and manual tests
+* Extensible reports and notifications in various formats (email, rss, pdf, html etc.)
+* A well defined way to create a project specific test specification format
+
+## Installation:
+* gem install rutema
+
+## Dependencies
+The core functionality of rutema depends on the following gems:
+ * [patir](http://github.com/damphyr/patir)
+ * [highline](http://highline.rubyforge.org/)
+
+## License:
+(The MIT License)
+
+Copyright (c) 2007-2015 Vassilis Rizopoulos
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
 SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/bin/rutema

@@ -1,8 +1,8 @@
-#  Copyright (c) 2007-2011 Vassilis Rizopoulos. All rights reserved.
-require 'rubygems'
-require 'rutema/system'
-begin
-	Rutema::RutemaX.new(ARGV)
-rescue
-	exit 1
+#  Copyright (c) 2007-2015 Vassilis Rizopoulos. All rights reserved.
+require 'rutema/application'
+begin
+	Rutema::App.new(ARGV)
+rescue Rutema::RutemaError
+ puts $!.message
+ exit 1
 end

data/lib/rutema/application.rb

@@ -0,0 +1,61 @@
+require  'optparse'
+require_relative "core/configuration"
+require_relative "core/engine"
+
+module Rutema
+  #Parses the commandline, sets up the configuration and launches Rutema::Engine
+  class App
+    def initialize command_line_args
+      parse_command_line(command_line_args)
+      @configuration=Rutema::Configuration.new(@config_file)
+      @configuration.context||={}
+      @configuration.context[:config_file]=File.basename(@config_file)
+      @configuration.reporters||={}
+      @configuration.reporters[Rutema::Reporters::Console]||={:class=>Rutema::Reporters::Console, "silent"=>@silent} unless @bare
+      @configuration.reporters[Rutema::Reporters::Summary]||={:class=>Rutema::Reporters::Summary, "silent"=>(@silent||@bare)}
+      Dir.chdir(File.dirname(@config_file)) do 
+        @engine=Rutema::Engine.new(@configuration)
+        application_flow
+      end
+    end
+    private
+    def parse_command_line args
+      args.options do |opt|
+        opt.on("rutema v#{Version::STRING}")
+        opt.on("Options:")
+        opt.on("--config FILE", "-c FILE",String,"Loads the configuration from FILE") { |config_file| @config_file=config_file}
+        opt.on("--check","Runs just the check test"){@check=true}
+        #opt.on("--step","Runs test cases step by step"){@step=true}
+        opt.on("--silent","Suppresses console output (only for the default reporters)") { @silent=true}
+        opt.on("--bare","No default reporters whatsoever") { @bare=true}
+        #opt.on("--color","Adds color to the Console reporter") { @color=true}
+        opt.on("-v", "--version","Displays the version") { $stdout.puts("rutema v#{Version::STRING}");exit 0 }
+        opt.on("--help", "-h", "-?", "This text") { $stdout.puts opt; exit 0 }
+        opt.on("You can provide a specification filename in order to run a single test")
+        opt.parse!
+        #and now the rest
+        unless @config_file
+          puts "No configuration file defined!\n"
+          $stdout.puts opt 
+          exit 1
+        end
+        if !args.empty?
+          @test_identifier=args.shift
+        end
+      end
+    end
+    def application_flow
+      if @check
+        #run just the check test
+        if @configuration.check
+          exit @engine.run(@configuration.check)
+        else
+          raise Rutema::RutemaError,"There is no check test defined in the configuration."
+        end
+      else
+        #run everything
+        exit @engine.run(@test_identifier)
+      end
+    end
+  end
+end

data/lib/rutema/core/configuration.rb

@@ -0,0 +1,195 @@
+  #  Copyright (c) 2007-2015 Vassilis Rizopoulos. All rights reserved.
+  require 'ostruct'
+  require_relative 'parser'
+  require_relative 'reporter'
+  module Rutema
+    #This module defines the "configuration directives" used in the configuration of Rutema
+    #
+    #Example
+    #A configuration file needs as a minimum to define which parser to use and which tests to run.
+    #
+    #Since rutema configuration files are valid Ruby code, you can use the full power of the Ruby language including require directives
+    #
+    # require 'rake'
+    # configuration.parser={:class=>Rutema::MinimalXMLParser}
+    # configuration.tests=FileList['all/of/the/tests/**/*.*']
+    module ConfigurationDirectives
+      attr_reader :parser,:runner,:tools,:paths,:tests,:context,:check,:setup,:teardown
+      attr_accessor :reporters
+      #Adds a hash of values to the tools hash of the configuration
+      #
+      #This hash is then accessible in the parser and reporters as a property of the configuration instance
+      #
+      #Required keys:
+      # :name - the name to use for accessing the path in code
+      #Example:
+      # configure do |cfg|
+      #  cfg.tool={:name=>"nunit",:path=>"/bin/nunit",:configuration=>{:important=>"info"}}
+      # end
+      #
+      #The path to nunit can then be accessed in the parser as
+      # @configuration.tools.nunit[:path]
+      #
+      #This way you can pass configuration information for the tools you use
+      def tool= definition
+        raise ConfigurationException,"required key :name is missing from #{definition}" unless definition[:name]
+        @tools[definition[:name]]=definition
+      end
+      #Adds a path to the paths hash of the configuration
+      #
+      #Required keys:
+      # :name - the name to use for accessing the path in code
+      # :path - the path
+      #Example:
+      # cfg.path={:name=>"sources",:path=>"/src"}
+      def path= definition
+        raise ConfigurationException,"required key :name is missing from #{definition}" unless definition[:name]
+        raise ConfigurationException,"required key :path is missing from #{definition}" unless definition[:path]
+        @paths[definition[:name]]=definition[:path]
+      end
+      
+      #Path to the setup specification. (optional)
+      #
+      #The setup test runs before every test.
+      def setup= path
+        @setup=check_path(path)
+      end
+      
+      #Path to the teardown specification. (optional)
+      #
+      #The teardown test runs after every test.    
+      def teardown= path
+        @teardown=check_path(path)
+      end
+      
+      #Path to the check specification. (optional)
+      #
+      #The check test runs once in the beginning before all the tests.
+      #
+      #If it fails no tests are run.
+      def check= path
+        @check=check_path(path)
+      end
+      
+      #Hash values for passing data to the system. It's supposed to be used in the reporters and contain 
+      #values such as version numbers, tester names etc.
+      def context= definition
+        @context||=Hash.new
+        raise ConfigurationException,"Only accepting hash values as context_data" unless definition.kind_of?(Hash)
+        @context.merge!(definition)
+      end
+      
+      #Adds the specification identifiers available to this instance of Rutema
+      #
+      #These will usually be files, but they can be anything.
+      #Essentially this is an Array of strings that mean something to your parser
+      def tests= array_of_identifiers
+        @tests+=array_of_identifiers.map{|f| full_path(f)}
+      end
+      
+      #A hash defining the parser to use.
+      #
+      #The hash is passed as is to the parser constructor and each parser should define the necessary configuration keys.
+      #
+      #The only required key from the configurator's point fo view is :class which should be set to the fully qualified name of the class to use.
+      #
+      #Example:
+      # cfg.parser={:class=>Rutema::MinimalXMLParser}
+      def parser= definition
+        raise ConfigurationException,"required key :class is missing from #{definition}" unless definition[:class]
+        @parser=definition
+      end
+      
+      #Adds a reporter to the configuration.
+      #
+      #As with the parser, the only required configuration key is :class and the definition hash is passed to the class' constructor.
+      # 
+      #Unlike the parser, you can define multiple reporters.
+      def reporter= definition
+        raise ConfigurationException,"required key :class is missing from #{definition}" unless definition[:class]
+        @reporters[definition[:class]]=definition
+      end
+
+      def init
+        @reporters={}
+        @context={}
+        @tests=[]
+        @tools=OpenStruct.new
+        @paths=OpenStruct.new
+      end
+      private 
+      #Checks if a path exists and raises a ConfigurationException if not
+      def check_path path
+        path=File.expand_path(path)
+        raise ConfigurationException,"#{path} does not exist" unless File.exists?(path)
+        return path
+      end
+      #Gives back a string of key=value,key=value for a hash
+      def definition_string definition
+        msg=Array.new
+        definition.each{|k,v| msg<<"#{k}=#{v}"}
+        return msg.join(",")
+      end
+
+      def full_path filename
+        return File.expand_path(filename) if File.exists?(filename)
+        return filename
+      end
+    end
+
+    class ConfigurationException<RuntimeError
+    end
+
+    class Configuration
+      include ConfigurationDirectives
+      attr_reader :filename
+      def initialize config_file
+        @filename=config_file
+        init
+        load_configuration(@filename)
+      end
+
+      def configure
+        if block_given?
+          yield self
+        end
+      end
+    
+      #Loads the configuration from a file
+      #
+      #Use this to chain configuration files together
+      #==Example
+      #Say you have on configuration file "first.rutema" that contains all the generic directives and several others that change only one or two things. 
+      #
+      #You can import the first.rutema file in the other configurations with
+      # import("first.rutema")
+      def import filename
+        fnm = File.expand_path(filename)
+        if File.exists?(fnm)          
+          load_configuration(fnm)
+        else
+          raise ConfigurationException, "Import error: Can't find #{fnm}"
+        end
+      end
+      private
+      def load_configuration filename
+        begin 
+          cfg_txt=File.read(filename)
+          cwd=File.expand_path(File.dirname(filename))
+          #evaluate in the working directory to enable relative paths in configuration
+          Dir.chdir(cwd){eval(cfg_txt,binding(),filename,__LINE__)}
+        rescue ConfigurationException
+          #pass it on, do not wrap again
+          raise
+        rescue SyntaxError
+          #Just wrap the exception so we can differentiate
+          raise ConfigurationException.new,"Syntax error in the configuration file '#{filename}':\n#{$!.message}"
+        rescue NoMethodError
+          raise ConfigurationException.new,"Encountered an unknown directive in configuration file '#{filename}':\n#{$!.message}"
+        rescue 
+          #Just wrap the exception so we can differentiate
+          raise ConfigurationException.new,"#{$!.message}"
+        end
+      end
+    end
+  end

data/lib/rutema/core/engine.rb

@@ -0,0 +1,186 @@
+#  Copyright (c) 2007-2015 Vassilis Rizopoulos. All rights reserved.
+require 'thread'
+require_relative 'parser'
+require_relative 'reporter'
+require_relative 'runner'
+require_relative '../version'
+
+module Rutema
+  class Engine
+    include Messaging
+    def initialize configuration
+      @queue=Queue.new
+      @parser=instantiate_class(configuration.parser,configuration) if configuration.parser
+      if configuration.runner
+        @runner=instantiate_class(configuration.runner,configuration) 
+      else
+        @runner=Rutema::Runners::Default.new(configuration.context,@queue)
+      end
+      raise RutemaError,"Could not instantiate parser" unless @parser
+      @dispatcher=Dispatcher.new(@queue,configuration)
+      @configuration=configuration
+    end
+    def run test_identifier=nil
+      @dispatcher.run!
+      #start
+      message("start")
+      check,setup,teardown,tests=*parse(test_identifier)
+      if tests.empty?
+        error(test_identifier,"Did not parse any tests succesfully")
+        @dispatcher.exit
+        raise RutemaError,"Did not parse any tests succesfully"
+      else
+        @runner.setup=setup
+        @runner.teardown=teardown
+        #running - at this point we've done any and all checks and we're stepping on the gas
+        message("running")
+        run_scenarios(tests,check)
+      end
+      message("end")
+      @dispatcher.exit
+      @dispatcher.report(tests)
+    end
+    def parse test_identifier=nil
+      specs=[]
+      #so, while we are parsing, we have a list of tests
+      #we're either parsing all of the tests, or just one
+      #make sure the one test is on the list
+      if test_identifier
+        if @configuration.tests.include?(File.expand_path(test_identifier))
+          specs<<parse_specification(File.expand_path(test_identifier))
+        else
+          error(test_identifier,"Does not exist in the configuration")  
+        end
+      else
+        specs=parse_specifications(@configuration.tests)
+      end
+      check,setup,teardown=parse_specials(@configuration)
+      return [check,setup,teardown,specs]
+    end
+    private
+    def parse_specifications tests
+      tests.map{|t| parse_specification(t)}.compact
+    end
+    def parse_specification spec_identifier
+      begin
+        @parser.parse_specification(spec_identifier)
+      rescue Rutema::ParserError
+        error(spec_identifier,$!.message)
+        nil
+      end
+    end
+    def parse_specials configuration
+      check=nil
+      setup=nil
+      teardown=nil
+      if configuration.check
+        check=parse_specification(configuration.check)
+      end
+      if configuration.setup
+        setup=parse_specification(configuration.setup)
+      end
+      if configuration.teardown
+        teardown=parse_specification(configuration.teardown)
+      end
+      return check,setup,teardown
+    end
+    def run_scenarios specs,check
+      if specs.empty?
+        error(nil,"No tests to run")
+      else
+        if check
+          if run_test(check)==:success
+            specs.each{|s| run_test(s)}
+          else
+            error(nil,"Check test failed")
+          end
+        else
+          specs.each{|spec| run_test(spec)}
+        end
+      end
+    end
+    def run_test specification
+      if specification.scenario
+        status=@runner.run(specification)["status"]
+      else
+        status=:not_executed
+        message(:test=>specification.name,:message=>"No scenario", :status=>status)
+      end
+      return status
+    end
+    def instantiate_class definition,configuration
+      if definition[:class]
+        klass=definition[:class]
+        return klass.new(configuration)
+      end
+      return nil
+    end
+  end
+  class Dispatcher
+    INTERVAL=0.01
+    def initialize queue,configuration
+      @queue = queue
+      @queues = {}
+      @streaming_reporters=[]
+      @block_reporters=[]
+      @collector=Rutema::Reporters::Collector.new(nil,self)
+      if configuration.reporters
+        instances=configuration.reporters.values.map{|v| instantiate_reporter(v,configuration) if v[:class] != Reporters::Summary}.compact
+        @streaming_reporters,_=instances.partition{|rep| rep.respond_to?(:update)}
+        @block_reporters,_=instances.partition{|rep| rep.respond_to?(:report)}
+      end
+      @streaming_reporters<<@collector
+      @configuration=configuration
+    end
+    def subscribe identifier
+      @queues[identifier]=Queue.new
+      return @queues[identifier]
+    end
+    
+    def run!
+      @streaming_reporters.each {|r| r.run!}
+      @thread=Thread.new do
+        while true do
+          dispatch()
+          sleep INTERVAL
+        end
+      end
+    end
+
+    def report specs
+      @block_reporters.each do |r|
+        r.report(specs,@collector.states,@collector.errors)
+      end
+      Reporters::Summary.new(@configuration,self).report(specs,@collector.states,@collector.errors)
+    end
+    def exit
+      if @thread
+        flush
+        @streaming_reporters.each {|r| r.exit}
+        Thread.kill(@thread)
+      end
+    end
+    private
+    def flush
+      if @thread
+        while @queue.size>0 do
+          dispatch()
+          sleep INTERVAL
+        end
+      end
+    end
+    def instantiate_reporter definition,configuration
+      if definition[:class]
+        klass=definition[:class]
+        return klass.new(configuration,self)
+      end
+      return nil
+    end
+    def dispatch
+      if @queue.size>0
+        data=@queue.pop
+        @queues.each{ |i,q| q.push(data) } if data
+      end
+    end
+  end
+end