chemistrykit 3.6.0 → 3.7.0

data/CHANGELOG.md

@@ -1,3 +1,15 @@
+#3.7.0 (2013-07-24)
+updated evidence to put in test based folders and added configuration for the retry functionality
+
+- Bumped version to 3.7.0 to prepare for release.
+- updated integration tests for new evidence directory format
+- updated config template and made some tests pass
+- updated readme to add usergroup link
+- added folder hierachy for evidence and slug formatter, fixed code quality issues
+- added slugification to junit formatter to support attachment plugin
+- added retry to configuration file and run time parameters updated docs and tests
+- updated sc version
+
 #3.6.0 (2013-07-20)
 Added the ability for tests to be retried on failure. A single retry will be attempted by default and the output will be verbose.
 

data/README.md

@@ -1,10 +1,11 @@
-#ChemistryKit 3.6.0 (2013-07-20)
+#ChemistryKit 3.7.0 (2013-07-24)
 
 [![Gem Version](https://badge.fury.io/rb/chemistrykit.png)](http://badge.fury.io/rb/chemistrykit) [![Build Status](https://travis-ci.org/arrgyle/chemistrykit.png?branch=develop)](https://travis-ci.org/jrobertfox/chef-broiler-platter) [![Code Climate](https://codeclimate.com/github/arrgyle/chemistrykit.png)](https://codeclimate.com/github/arrgyle/chemistrykit) [![Coverage Status](https://coveralls.io/repos/arrgyle/chemistrykit/badge.png?branch=develop)](https://coveralls.io/r/arrgyle/chemistrykit?branch=develop)
 
 ### A simple and opinionated web testing framework for Selenium WebDriver
 
-This framework was designed to help you get started with Selenium WebDriver quickly, to grow as needed, and to avoid common pitfalls by following convention over configuration.
+This framework was designed to help you get started with Selenium WebDriver quickly, to grow as needed, and to avoid common pitfalls by following convention over configuration. To checkout the user group go [here](https://groups.google.com/forum/#!forum/chemistrykit-users).
+
 
 ChemistryKit's inspiration comes from the Saunter Selenium framework which is available in Python and PHP. You can find more about it [here](http://element34.ca/products/saunter).
 
@@ -107,6 +108,8 @@ ChemistryKit is configured by default with a `config.yaml` file that is created
 
 `base_url:` The base url of your app, stored to the ENV for access in your beakers and formulas.
 
+`retries_on_failure:` Defaults to 1, set the number of times a test should be retried on failure
+
 `concurrency:` You may override the default concurrency of 1 to run the tests in parallel
 
 `log: path:` You may override the default log path 'evidence'
@@ -143,6 +146,7 @@ Available options for the `brew` command:
 -r, --results_file [NAME]   Specify the name of your results file.
 --tag [TAGS]                Specify a list of tags to run or exclude.
 --params [HASH]             Send a list of "key:value" parameters to the ENV.
+-x, --retry [INT]           How many times should a failing test be retried.
 ```
 
 ###generate forumla
@@ -168,11 +172,15 @@ Usage:
     ckit tags
 
 ##Contribution Guidelines
+
 This project conforms to the [neverstopbuilding/craftsmanship](https://github.com/neverstopbuilding/craftsmanship) guidelines. Please see them for details on:
 - Branching theory
 - Documentation expectations
 - Release process
 
+###Check out the user group!
+[https://groups.google.com/forum/#!forum/chemistrykit-users](https://groups.google.com/forum/#!forum/chemistrykit-users)
+
 ###It's simple
 
 1. Create a feature branch from develop: `git checkout -b feature/myfeature develop` or `git flow feature start myfeature`

data/chemistrykit.gemspec

@@ -2,13 +2,13 @@
 
 Gem::Specification.new do |s|
   s.name          = 'chemistrykit'
-  s.version       = '3.6.0'
+  s.version       = '3.7.0'
   s.platform      = Gem::Platform::RUBY
   s.authors       = ['Dave Haeffner', 'Jason Fox']
   s.email         = ['dave@arrgyle.com', 'jason@arrgyle.com']
   s.homepage      = 'https://github.com/arrgyle/chemistrykit'
   s.summary       = 'A simple and opinionated web testing framework for Selenium that follows convention over configuration.'
-  s.description   = 'Added the ability for tests to be retried on failure. A single retry will be attempted by default and the output will be verbose.'
+  s.description   = 'updated evidence to put in test based folders and added configuration for the retry functionality'
   s.license       = 'MIT'
 
   s.files         = `git ls-files`.split($/)
@@ -21,10 +21,10 @@ Gem::Specification.new do |s|
 
   s.add_dependency 'thor', '~> 0.17.0'
   s.add_dependency 'rspec', '~> 2.14.1'
-  s.add_dependency 'yarjuf', '~> 1.0.5'
+  s.add_dependency 'builder', '~> 3.2.2'
   s.add_dependency 'selenium-webdriver', '~> 2.29.0'
   s.add_dependency 'rest-client', '~> 1.6.7'
-  s.add_dependency 'selenium-connect', '~> 3.2.0'
+  s.add_dependency 'selenium-connect', '~> 3.3.1'
   s.add_dependency 'parallel_tests', '~> 0.15.0'
   s.add_dependency 'parallel', '~> 0.7.0'
   s.add_dependency 'rspec-retry', '~> 0.2.1'

data/features/brew.feature

@@ -40,8 +40,8 @@ Feature: Brewing a ChemistryKit project
     When I run `ckit brew`
     Then the stdout should contain "1 example, 0 failures"
     And the following files should exist:
-      | evidence/results_junit.xml       |
-      | evidence/bookie/server.log       |
+      | evidence/results_junit.xml |
+      | evidence/bookie/bookie_loads_an_external_web_page/server.log        |
 
   Scenario: Brew a single beaker
     Given a file named "config.yaml" with:
@@ -108,6 +108,54 @@ Feature: Brewing a ChemistryKit project
     """
     When I run `ckit brew --beakers=beakers/failure.rb`
     Then the stdout should contain "1 example, 1 failure"
-    And there should be "2" "failed image" log files in "evidence/failing_beaker"
-    And there should be "2" "report" log files in "evidence/failing_beaker"
-    And there should be "2" "sauce log" log files in "evidence/failing_beaker"
+    And there should be "1" "failed image" log files in "evidence/failing_beaker/failing_beaker_loads_an_external_web_page"
+    And there should be "1" "report" log files in "evidence/failing_beaker/failing_beaker_loads_an_external_web_page"
+    And there should be "1" "sauce log" log files in "evidence/failing_beaker/failing_beaker_loads_an_external_web_page"
+
+  Scenario: Retry a test on failure based on config file
+    Given a file named "config.yaml" with:
+      """
+      retries_on_failure: 3
+      selenium_connect:
+          log: 'evidence'
+          host: 'localhost'
+      """
+    And a file named "beakers/other_beaker.rb" with:
+      """
+      describe "Other" do
+        let(:book) { Formulas::Bookie.new(@driver) }
+
+        it "loads an external web page" do
+          book.open "http://www.google.com"
+          @driver.title.should_not include("Google")
+        end
+      end
+      """
+    When I run `ckit brew --beakers=beakers/other_beaker.rb`
+    Then the stdout should contain "RSpec::Retry: 2nd try"
+    Then the stdout should contain "RSpec::Retry: 3rd try"
+
+  Scenario: Retry a test on failure based on run time parameter
+    Given a file named "config.yaml" with:
+      """
+      retries_on_failure: 2
+      selenium_connect:
+          log: 'evidence'
+          host: 'localhost'
+      """
+    And a file named "beakers/other_beaker.rb" with:
+      """
+      describe "Other" do
+        let(:book) { Formulas::Bookie.new(@driver) }
+
+        it "loads an external web page" do
+          book.open "http://www.google.com"
+          @driver.title.should_not include("Google")
+        end
+      end
+      """
+    When I run `ckit brew --beakers=beakers/other_beaker.rb --retry=3`
+    Then the stdout should contain "RSpec::Retry: 2nd try"
+    Then the stdout should contain "RSpec::Retry: 3rd try"
+
+

data/features/concurrency.feature

@@ -97,4 +97,4 @@ Feature: Support for concurency
     Then the stdout should not contain "All examples were filtered out"
     And the stdout should not contain "0 examples, 0 failures"
     And there should be "4" unique results files in the "evidence" directory
-    And there should be "2" "report" log files in "evidence/cheese"
+    And there should be "2" "report" log files in "evidence/cheese/cheese_loads_an_external_web_page"

data/features/logging.feature

@@ -84,8 +84,8 @@ Feature: Log handling
         Then the stdout should contain "2 examples, 0 failures"
         And the following files should exist:
           | my_evidence/results_junit.xml |
-          | my_evidence/first/server.log  |
-          | my_evidence/second/server.log  |
+          | my_evidence/first/first_loads_an_external_web_page/server.log  |
+          | my_evidence/second/second_loads_an_external_web_page/server.log  |
 
     Scenario: I can runtime override the results output file name
       When I run `ckit brew --results_file results_junit_01.xml`

data/features/multi-config.feature

@@ -26,7 +26,7 @@ Feature: Support for multiple configuration files
     When I run `ckit brew`
     Then the stdout should contain "1 example, 0 failures"
     And the following files should exist:
-      | evidence_config/cheese/server.log       |
+      | evidence_config/cheese/cheese_loads_an_external_web_page/server.log |
 
   Scenario: I can specifiy an alternative configuration with --config
     Given a directory named "evidence_alternate"
@@ -39,7 +39,7 @@ Feature: Support for multiple configuration files
       When I run `ckit brew --config alternate.yaml`
       Then the stdout should contain "1 example, 0 failures"
       And the following files should exist:
-        | evidence_alternate/cheese/server.log       |
+        | evidence_alternate/cheese/cheese_loads_an_external_web_page/server.log |
 
   Scenario: I can specifiy an alternative configuration with -c
     Given a directory named "evidence_alternate"
@@ -52,8 +52,7 @@ Feature: Support for multiple configuration files
       When I run `ckit brew -c alternate.yaml`
       Then the stdout should contain "1 example, 0 failures"
       And the following files should exist:
-        | evidence_alternate/cheese/server.log       |
-
+       | evidence_alternate/cheese/cheese_loads_an_external_web_page/server.log |
   Scenario: I can specifiy an alternative configuration with --config with concurrency
     Given a directory named "evidence_alternate"
     And a file named "alternate.yaml" with:
@@ -68,4 +67,4 @@ Feature: Support for multiple configuration files
       Then the stdout should contain "1 example, 0 failures"
       And there should be "1" unique results files in the "evidence_alternate" directory
       And the following files should exist:
-        | evidence_alternate/cheese/server.log       |
+      | evidence_alternate/cheese/cheese_loads_an_external_web_page/server.log |

data/lib/chemistrykit/cli/cli.rb

@@ -3,7 +3,6 @@
 require 'thor'
 require 'rspec'
 require 'rspec/retry'
-require 'yarjuf'
 require 'chemistrykit/cli/new'
 require 'chemistrykit/cli/formula'
 require 'chemistrykit/cli/beaker'
@@ -14,6 +13,7 @@ require 'selenium_connect'
 require 'chemistrykit/configuration'
 require 'parallel_tests'
 require 'chemistrykit/parallel_tests_mods'
+require 'chemistrykit/j_unit'
 
 module ChemistryKit
   module CLI
@@ -66,6 +66,7 @@ module ChemistryKit
       method_option :parallel, default: false
       method_option :results_file, aliases: '-r', default: false, desc: 'Specifiy the name of your results file.'
       method_option :all, default: false, aliases: '-a', desc: 'Run every beaker.', type: :boolean
+      method_option :retry, default: false, aliases: '-x', desc: 'How many times should a failing test be retried.'
 
       def brew
         config = load_config options['config']
@@ -119,6 +120,9 @@ module ChemistryKit
         if options['results_file']
           config.log.results_file = options['results_file']
         end
+        if options['retry']
+          config.retries_on_failure = options['retry'].to_i
+        end
         config
       end
 
@@ -168,17 +172,22 @@ module ChemistryKit
           c.around(:each) do |example|
             # create the beaker name from the example data
             beaker_name = example.metadata[:example_group][:description_args].first.downcase.strip.gsub(' ', '_').gsub(/[^\w-]/, '')
+            test_name = example.metadata[:full_description].downcase.strip.gsub(' ', '_').gsub(/[^\w-]/, '')
+
             # override log path with be beaker sub path
             sc_config = @config.selenium_connect.dup
             sc_config[:log] += "/#{beaker_name}"
-            sub_evidence_path = File.join(Dir.getwd, sc_config[:log])
-            Dir.mkdir sub_evidence_path unless File.exists?(sub_evidence_path)
+            beaker_path = File.join(Dir.getwd, sc_config[:log])
+            Dir.mkdir beaker_path unless File.exists?(beaker_path)
+            sc_config[:log] += "/#{test_name}"
+            test_path = File.join(Dir.getwd, sc_config[:log])
+            Dir.mkdir test_path unless File.exists?(test_path)
 
             # configure and start sc
             configuration = SeleniumConnect::Configuration.new sc_config
             @sc = SeleniumConnect.start configuration
             @job = @sc.create_job # create a new job
-            @driver = @job.start name: example.metadata[:full_description]
+            @driver = @job.start name: test_name
             example.run
           end
           c.after(:each) do
@@ -197,7 +206,7 @@ module ChemistryKit
 
           # for rspec-retry
           c.verbose_retry = true # for rspec-retry
-          c.default_retry_count = 2
+          c.default_retry_count = config.retries_on_failure
 
           if config.concurrency == 1 || options['parallel']
             c.add_formatter(config.log.format, File.join(Dir.getwd, config.log.path, config.log.results_file))

data/lib/chemistrykit/configuration.rb

@@ -9,7 +9,8 @@ module ChemistryKit
 
     attr_accessor :base_url,
                   :concurrency,
-                  :screenshot_on_fail
+                  :screenshot_on_fail,
+                  :retries_on_failure
 
     attr_reader   :log
 
@@ -18,6 +19,7 @@ module ChemistryKit
     def initialize(hash)
       # set defaults
       @concurrency = 1
+      @retries_on_failure = 1
       @selenium_connect = {}
       @screenshot_on_fail = false
       @log = OpenStruct.new

data/lib/chemistrykit/j_unit.rb

@@ -0,0 +1,121 @@
+# Encoding: utf-8
+
+require 'time'
+require 'builder'
+require 'rspec/core/formatters/base_formatter'
+
+# An RSpec formatter for generating results in JUnit format
+# updated from https://github.com/natritmeyer/yarjuf
+class JUnit < RSpec::Core::Formatters::BaseFormatter
+
+  # rspec formatter methods we care about
+
+  def initialize(output)
+    super output
+    @test_suite_results = {}
+    @builder = Builder::XmlMarkup.new indent: 2
+  end
+
+  def example_passed(example)
+    add_to_test_suite_results example
+  end
+
+  def example_failed(example)
+    add_to_test_suite_results example
+  end
+
+  def example_pending(example)
+    add_to_test_suite_results example
+  end
+
+  def dump_summary(duration, example_count, failure_count, pending_count)
+    build_results duration, example_count, failure_count, pending_count
+    output.puts @builder.target!
+  end
+
+  protected
+
+  def add_to_test_suite_results(example)
+    suite_name = JUnit.root_group_name_for(example)
+    @test_suite_results[suite_name] = [] unless @test_suite_results.keys.include? suite_name
+    @test_suite_results[suite_name] << example
+  end
+
+  def failure_details_for(example)
+    exception = example.metadata[:execution_result][:exception]
+    exception.nil? ? '' : "#{exception.message}\n#{format_backtrace(exception.backtrace, example).join("\n")}"
+  end
+
+  # utility methods
+
+  def self.count_in_suite_of_type(suite, test_case_result_type)
+    suite.select { |example| example.metadata[:execution_result][:status] == test_case_result_type }.size
+  end
+
+  def self.root_group_name_for(example)
+    group_hierarchy = []
+    current_example_group = example.metadata[:example_group]
+    until current_example_group.nil?
+      group_hierarchy.unshift current_example_group
+      current_example_group = current_example_group[:example_group]
+    end
+    JUnit.slugify group_hierarchy.first[:description]
+  end
+
+  # methods to build the xml for test suites and individual tests
+
+  def build_results(duration, example_count, failure_count, pending_count)
+    @builder.instruct! :xml, version: '1.0', encoding: 'UTF-8'
+    @builder.testsuites errors: 0, failures: failure_count, skipped: pending_count, tests: example_count, time: duration, timestamp: Time.now.iso8601 do
+      build_all_suites
+    end
+  end
+
+  def build_all_suites
+    @test_suite_results.each do |suite_name, tests|
+      build_test_suite suite_name, tests
+    end
+  end
+
+  def build_test_suite(suite_name, tests)
+    failure_count = JUnit.count_in_suite_of_type tests, 'failed'
+    skipped_count = JUnit.count_in_suite_of_type tests, 'pending'
+
+    @builder.testsuite name: suite_name, tests: tests.size, errors: 0, failures: failure_count, skipped: skipped_count do
+      @builder.properties
+      build_all_tests tests
+    end
+  end
+
+  def build_all_tests(tests)
+    tests.each do |test|
+      build_test test
+    end
+  end
+
+  def build_test(test)
+    test_name = JUnit.slugify test.metadata[:full_description]
+    execution_time = test.metadata[:execution_result][:run_time]
+    test_status = test.metadata[:execution_result][:status]
+
+    @builder.testcase name: test_name, time: execution_time do
+      case test_status
+      when 'pending' then @builder.skipped
+      when 'failed' then build_failed_test test
+      end
+    end
+  end
+
+  def build_failed_test(test)
+    failure_message = "failed #{test.metadata[:full_description]}"
+
+    @builder.failure message: failure_message, type: 'failed' do
+      @builder.cdata! failure_details_for test
+    end
+  end
+
+  def self.slugify(string)
+    string.downcase.strip.gsub(' ', '_').gsub(/[^\w-]/, '')
+  end
+
+end

data/lib/templates/chemistrykit/config.yaml.tt

@@ -1,6 +1,7 @@
 # Default Chemisty Kit Configuration File
 
 base_url: http://google.com
+retries_on_failure: 1 #defaults to 1, set the number of times a test should be retried on failure
 concurrency: 1
 log:
     path: #you may override the default log path 'evidence'
@@ -9,24 +10,22 @@ log:
 selenium_connect:
     #Setup & Debugging
     jar:
-    log:
-
 
     #Where to run your tests
     host: 'localhost'
     port: #set to 4444 by default
 
-
     #Browser & Host
     browser:      'firefox'
     browser_path:
     profile_path:
     profile_name:
 
-
     #Saucelabs
-    os:
     sauce_username:
     sauce_api_key:
-    browser_version:
-    description:
+    api_timeout:
+    sauce_opts:
+        os:
+        browser_version:
+        job_name:

data/spec/unit/lib/chemistrykit/configuration_spec.rb

@@ -6,6 +6,7 @@ describe ChemistryKit::Configuration do
 
   VALID_BASE_URL = 'http://google.com'
   VALID_CONCURRENCY = 1
+  VALID_RETRIES_ON_FAILURE = 1
   VALID_CONFIG_FILE = 'config.yaml'
   VALID_LOG_PATH = 'evidence'
   VALID_JUNIT = 'results_junit.xml'
@@ -19,13 +20,15 @@ describe ChemistryKit::Configuration do
       base_url: VALID_BASE_URL,
       concurrency: VALID_CONCURRENCY,
       selenium_connect: @valid_selenium_connect_hash,
-      log: @valid_log_hash
+      log: @valid_log_hash,
+      retries_on_failure: VALID_RETRIES_ON_FAILURE
     }
   end
 
   def validate_config(config)
     config.screenshot_on_fail.should eq false
     config.concurrency.should eq VALID_CONCURRENCY
+    config.retries_on_failure.should eq VALID_RETRIES_ON_FAILURE
     config.base_url.should eq VALID_BASE_URL
     config.log.path.should eq VALID_LOG_PATH
     config.log.results_file.should eq VALID_JUNIT
@@ -36,6 +39,7 @@ describe ChemistryKit::Configuration do
   it 'should initialize with sane defaults' do
     config = ChemistryKit::Configuration.new({})
     config.concurrency.should eq VALID_CONCURRENCY
+    config.retries_on_failure.should eq VALID_RETRIES_ON_FAILURE
     config.log.path.should eq VALID_LOG_PATH
     config.log.results_file.should eq VALID_JUNIT
     config.log.format.should eq VALID_JUNIT_FORMAT_OUT
@@ -71,4 +75,9 @@ describe ChemistryKit::Configuration do
     config =  ChemistryKit::Configuration.new({ log: { path: 'main-path' }, selenium_connect: { log: 'sc-log' } })
     config.selenium_connect.should eq({ log: 'sc-log' })
   end
+
+  it 'should allow pass through of sauce options' do
+    config =  ChemistryKit::Configuration.new selenium_connect: { log: 'sc-log', sauce_opts: { job_name: 'test' } }
+    config.selenium_connect.should eq log: 'sc-log', sauce_opts: { job_name: 'test' }
+  end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: chemistrykit
 version: !ruby/object:Gem::Version
-  version: 3.6.0
+  version: 3.7.0
   prerelease: 
 platform: ruby
 authors:
@@ -10,7 +10,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-07-20 00:00:00.000000000 Z
+date: 2013-07-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: thor
@@ -45,13 +45,13 @@ dependencies:
       - !ruby/object:Gem::Version
         version: 2.14.1
 - !ruby/object:Gem::Dependency
-  name: yarjuf
+  name: builder
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 1.0.5
+        version: 3.2.2
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -59,7 +59,7 @@ dependencies:
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 1.0.5
+        version: 3.2.2
 - !ruby/object:Gem::Dependency
   name: selenium-webdriver
   requirement: !ruby/object:Gem::Requirement
@@ -99,7 +99,7 @@ dependencies:
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 3.2.0
+        version: 3.3.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -107,7 +107,7 @@ dependencies:
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 3.2.0
+        version: 3.3.1
 - !ruby/object:Gem::Dependency
   name: parallel_tests
   requirement: !ruby/object:Gem::Requirement
@@ -268,8 +268,8 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: 0.6.7
-description: Added the ability for tests to be retried on failure. A single retry
-  will be attempted by default and the output will be verbose.
+description: updated evidence to put in test based folders and added configuration
+  for the retry functionality
 email:
 - dave@arrgyle.com
 - jason@arrgyle.com
@@ -316,6 +316,7 @@ files:
 - lib/chemistrykit/cli/new.rb
 - lib/chemistrykit/configuration.rb
 - lib/chemistrykit/formula/base.rb
+- lib/chemistrykit/j_unit.rb
 - lib/chemistrykit/parallel_tests_mods.rb
 - lib/templates/beaker.tt
 - lib/templates/beaker_with_formula.tt
@@ -354,10 +355,10 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -2599168027342354598
+      hash: -324108040210892536
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.24
+rubygems_version: 1.8.25
 signing_key: 
 specification_version: 3
 summary: A simple and opinionated web testing framework for Selenium that follows