chemistrykit 3.2.0 → 3.3.0

data/CHANGELOG.md

@@ -1,3 +1,15 @@
+#3.3.0 (2013-07-09)
+Fixed bugs with tagging and concurrency
+
+- Bumped version to 3.3.0 to prepare for release.
+- added the ckit tags command to list all tags in the harnes
+- consolidated documentation into the readme file
+- fixed code quality issues and a bug with the tag filtering
+- added the --all flag and tests to run all beakers
+- fixed tagging issues for concurrent runs, now a beaker specification will override the default tag unless it is explicitly specified.
+- removed debug info
+- initial cut on tagging updates
+
 #3.2.0 (2013-07-08)
 Updated Selenium Connect
 

data/README.md

@@ -5,8 +5,7 @@ Master branch: [![Build Status](https://travis-ci.org/arrgyle/chemistrykit.png?b
 
 Develop branch: [![Build Status](https://travis-ci.org/arrgyle/chemistrykit.png?branch=develop)](https://travis-ci.org/jrobertfox/chef-broiler-platter)
 
-ChemistryKit
-============================================================
+#ChemistryKit
 
 ### A simple and opinionated web testing framework for Selenium WebDriver
 
@@ -14,7 +13,16 @@ This framework was designed to help you get started with Selenium WebDriver quic
 
 ChemistryKit's inspiration comes from the Saunter Selenium framework which is available in Python and PHP. You can find more about it [here](http://element34.ca/products/saunter).
 
-## Getting Started
+All the documentation for ChemistryKit can be found in this README, organized as follows:
+
+- [Getting Started](#getting-started)
+- [Building a Test Suite](#building-a-test-suite)
+- [Configuration](#configuration)
+- [Command Line Usage](#command-line-usage)
+- [Contribution Guidelines](#contribution-guidelines)
+- [Deployment](#deployment)
+
+##Getting Started
 
     $ gem install chemistrykit
     $ ckit new framework_name
@@ -28,10 +36,139 @@ This will generate a beaker file (a.k.a. test script) with the name you provide
 
     $ ckit brew
 
-This will run ckit and execute your beakers. By default it will run the tests locally by default. But you can change where the tests run and all other relevant bits in \_config.yaml. You can find out more about this [here](https://github.com/arrgyle/chemistrykit/wiki/Configs).
+This will run ckit and execute your beakers. By default it will run the tests locally by default. But you can change where the tests run and all other relevant bits in `config.yaml` file detailed below.
+
+##Building a Test Suite
+
+###Spec Discovery
+
+ChemistryKit is built on top of RSpec. All specs are in the _beaker_ directory and end in _beaker.rb. Rather than being discovered via class or file name as some systems they are by identified by tag.
+
+```ruby
+it 'with invalid credentials', :depth => 'shallow' do
+
+end
+
+it 'with invalid credentials', :depth => 'deep' do
+
+end
+```
+
+All specs should have at least a :depth tag. The depth should either be 'shallow' or 'deep'. Shallow specs are the ones that are the absolute-must-pass ones. And there will only be a few of them typically. Deep ones are everything else.
+
+You can add multiple tags as well.
+
+```ruby
+it 'with invalid credentials', :depth => 'shallow', :authentication => true do
+
+end
+```
+
+By default ChemistryKit will discover and run the _:depth => 'shallow'_ scripts. To run different ones you use the --tag option.
+
+    ckit brew --tag authentication
+
+    ckit brew --tag depth:shallow --tag authentication
+
+To exclude a tag, put a ~ in front of it.
+
+    ckit brew --tag depth:shallow --tag ~authentication
+
+During development it is often helpful to just run a specific beaker, this can be accomplished with the `--beakers` flag:
+
+    ckit brew --beakers=beakers/wip_beaker.rb
+
+
+###Formula Loading
+Code in the `formula` directory can be used to build out page objects and helper functions to facilitate your testing. The files are loaded in a particular way:
+
+- Files in any `lib` directory are loaded before other directories.
+- Files in child directories are loaded before those in parent directories.
+- Files are loaded in alphabetical order.
+
+So for example if you have a `alpha_page.rb` file in your formulas directory that depends on a `helpers.rb` file, then you best put the `helpers.rb` file in the `lib` directory so it is loaded before the file that depends on it.
+
+###Execution Order
+Chemistry Kit executes specs in a random order. This is intentional. Knowing the order a spec will be executed in allows for dependencies between them to creep in. Sometimes unintentionally. By having them go in a random order parallelization becomes a much easier.
+
+###Before and After
+Chemistry Kit uses the 4-phase model for scripts with a chunk of code that gets run before and after each method. By default, it does nothing more than launch a browser instance that your configuration says you want. If you want to do something more than that, just add it to your spec.
+
+```ruby
+before(:each) do
+    # something here
+end
+```
+
+You can even nest them inside different describe/context blocks and they will get executed from the outside-in.
+
+###Logs and CI Integration
+Each run of Chemistry Kit saves logging and test output to the _evidence_ directory by default. And in there will be the full set of JUnit Ant XML files that may be consumed by your CI.
+
+##Configuration
+ChemistryKit is configured by default with a `config.yaml` file that is created for you when you scaffold out a test harness. Relevant configuration options are detailed below:
+
+`base_url:` The base url of your app, stored to the ENV for access in your beakers and formulas.
+
+`concurrency:` You may override the default concurrency of 1 to run the tests in parallel
+
+`log: path:` You may override the default log path 'evidence'
+
+`log: results_file:` You may override the default file name 'results_junit.xml'
+
+`log: format:` You may override the default format 'junit' to an alternative like 'doc' or 'html'
+
+`selenium_connect:` Options in this node override the defaults for the [Selenium Connect](https://github.com/arrgyle/selenium-connect) gem.
+##Command Line Usage
+
+###new
+Creates a new ChemistryKit project.
+
+Usage:
+
+    ckit new [NAME]
+
+###brew
+Executes your test cases.
+
+Usage:
+
+    ckit brew [OPTIONS]
+
+Available options for the `brew` command:
+
+```
+-a, --all                   Run every beaker regardless of tag.
+-b, --beakers [BEAKERS]     Pass a list of beaker paths to be executed.
+-c, --config [PATH]         Pass the path to an alternative config.yaml file.
+-r, --results_file [NAME]   Specify the name of your results file.
+--tag [TAGS]                Specify a list of tags to run or exclude.
+--params [HASH]             Send a list of "key:value" parameters to the ENV.
+```
+
+###generate forumla
+Creates a new boilerplate formula object.
+
+Usage:
+
+    ckit generate formula [NAME]
+
+
+###generate beaker
+Creates a new boilerplate beaker object.
+
+Usage:
+
+    ckit generate beaker [NAME]
+
+###tags
+Lists all the tags you have used in your beakers.
+
+Usage:
 
+    ckit tags
 
-## Contributing
+##Contribution Guidelines
 This project conforms to the [neverstopbuilding/craftsmanship](https://github.com/neverstopbuilding/craftsmanship) guidelines. Please see them for details on:
 - Branching theory
 - Documentation expectations
@@ -57,7 +194,7 @@ All issues and questions related to this project should be logged using the [git
 
     ckit
 
-##Releaseing
+##Deployment
 The release process is rather automated, just use one rake task with the new version number:
 
     rake release_start['2.1.0']

data/chemistrykit.gemspec

@@ -2,13 +2,13 @@
 
 Gem::Specification.new do |s|
   s.name          = "chemistrykit"
-  s.version       = "3.2.0"
+  s.version       = "3.3.0"
   s.platform      = Gem::Platform::RUBY
   s.authors       = ["Dave Haeffner", "Jason Fox"]
   s.email         = ["dave@arrgyle.com", "jason@arrgyle.com"]
   s.homepage      = "https://github.com/arrgyle/chemistrykit"
   s.summary       = "A simple and opinionated web testing framework for Selenium that follows convention over configuration."
-  s.description   = "Updated Selenium Connect"
+  s.description   = "Fixed bugs with tagging and concurrency"
   s.license       = 'MIT'
 
   s.files         = `git ls-files`.split($/)

data/features/brew.feature

@@ -29,6 +29,7 @@ Feature: Brewing a ChemistryKit project
         end
       end
       """
+
   Scenario: Localhost
     Given a file named "config.yaml" with:
       """
@@ -68,7 +69,7 @@ Feature: Brewing a ChemistryKit project
       """
     And a file named "beakers/other_beaker.rb" with:
       """
-      describe "Other", :depth => 'shallow' do
+      describe "Other" do
         let(:book) { Formulas::Bookie.new(@driver) }
 
         it "loads an external web page" do
@@ -79,3 +80,23 @@ Feature: Brewing a ChemistryKit project
     When I run `ckit brew --beakers=beakers/other_beaker.rb`
     Then the stdout should contain "1 example, 0 failures"
 
+  Scenario: Run all the tests regardless of tag
+    Given a file named "config.yaml" with:
+      """
+      selenium_connect:
+          log: 'evidence'
+          host: 'localhost'
+      """
+    And a file named "beakers/other_beaker.rb" with:
+      """
+      describe "Other" do
+        let(:book) { Formulas::Bookie.new(@driver) }
+
+        it "loads an external web page" do
+          book.open "http://www.google.com"
+        end
+      end
+      """
+    When I run `ckit brew --all`
+    Then the stdout should contain "2 examples, 0 failures"
+

data/features/concurrency.feature

@@ -28,9 +28,7 @@ Feature: Support for concurency
       end
     end
     """
-
-  Scenario: I can run the tests in parallel
-    When I overwrite config.yaml with:
+    And I overwrite config.yaml with:
       """
       concurrency: 4
       selenium_connect:
@@ -41,6 +39,8 @@ Feature: Support for concurency
           sauce_api_key: 'ab7a6e17-16df-42d2-9ef6-c8d2539cc38a'
           description: 'concurrency check'
       """
+
+  Scenario: I can run the tests in parallel
     When I run `ckit brew`
     Then the stdout should contain "4 processes for 2 beakers"
     And the stdout should contain "3 examples, 0 failures"
@@ -48,17 +48,50 @@ Feature: Support for concurency
     And there should be "2" unique results files in the "evidence" directory
 
   Scenario: I can run a specific beaker in parallel
-    When I overwrite config.yaml with:
+    When I run `ckit brew --beakers=beakers/first_beaker.rb`
+    Then the stdout should contain "4 processes for 1 beakers"
+    And the stdout should contain "2 examples, 0 failures"
+
+  Scenario: I can run specific beakers by tag in parallel
+    Given a file named "beakers/tagged_beaker_1.rb" with:
       """
-      concurrency: 4
-      selenium_connect:
-          log: 'evidence'
-          host: 'saucelabs'
-          browser: 'firefox'
-          sauce_username: 'testing_arrgyle'
-          sauce_api_key: 'ab7a6e17-16df-42d2-9ef6-c8d2539cc38a'
-          description: 'concurrency check'
+      describe "Cheese 3", :item => 'test' do
+        it "loads an external web page" do
+          @driver.get "http://www.google.com"
+          @driver.title.should include("Google")
+        end
+      end
+      """
+    And a file named "beakers/tagged_beaker_2.rb" with:
+      """
+      describe "Cheese 4", :item => 'test' do
+        it "loads an external web page" do
+          @driver.get "http://www.google.com"
+          @driver.title.should include("Google")
+        end
+      end
+      """
+    When I run `ckit brew --tag item:test`
+    And the stdout should contain "2 examples, 0 failures"
+
+    Scenario: I can all beakers in parallel
+    Given a file named "beakers/tagged_beaker_1.rb" with:
+      """
+      describe "Cheese 3", :item => 'test' do
+        it "loads an external web page" do
+          @driver.get "http://www.google.com"
+          @driver.title.should include("Google")
+        end
+      end
+      """
+    And a file named "beakers/tagged_beaker_2.rb" with:
+      """
+      describe "Cheese 4", :item => 'test' do
+        it "loads an external web page" do
+          @driver.get "http://www.google.com"
+          @driver.title.should include("Google")
+        end
+      end
       """
-      When I run `ckit brew --beakers=beakers/first_beaker.rb`
-      Then the stdout should contain "4 processes for 1 beakers"
-      And the stdout should contain "2 examples, 0 failures"
+    When I run `ckit brew --all`
+    And there should be "4" unique results files in the "evidence" directory

data/features/tags.feature

@@ -0,0 +1,82 @@
+Feature: Listing all the tags
+  In order to see all the tags I'm using
+  As a harness developer
+  I want to run a command to list all the tags
+
+  Background: Setup the project
+    Given I run `ckit new tags-test`
+    And I cd to "tags-test"
+
+  Scenario: Get the tag for a single beaker
+    Given a file named "beakers/bookie_beaker.rb" with:
+      """
+      describe "Bookie", :depth => 'shallow' do
+        let(:book) { Formulas::Bookie.new(@driver) }
+
+        it "loads an external web page" do
+          book.open "http://www.google.com"
+        end
+      end
+      """
+    When I run `ckit tags`
+    Then the stdout should contain "depth:shallow"
+
+  Scenario: Get the tags inside a beaker
+    Given a file named "beakers/bookie_beaker.rb" with:
+      """
+      describe "Bookie" do
+        let(:book) { Formulas::Bookie.new(@driver) }
+
+        it "loads an external web page", :depth => 'shallow' do
+          book.open "http://www.google.com"
+        end
+        it "loads an external web page", :depth => 'deep' do
+          book.open "http://www.google.com"
+        end
+      end
+      """
+    When I run `ckit tags`
+    Then the stdout should contain "depth:shallow"
+    And the stdout should contain "depth:deep"
+
+  Scenario: The tags should be unique and in alpha order
+    Given a file named "beakers/bookie_beaker.rb" with:
+      """
+      describe "Bookie" do
+        let(:book) { Formulas::Bookie.new(@driver) }
+
+        it "loads an external web page", :aaa => 'ccc' do
+          book.open "http://www.google.com"
+        end
+        it "loads an external web page", :bbb => 'ddd' do
+          book.open "http://www.google.com"
+        end
+      end
+      """
+    And a file named "beakers/bookie_other.rb" with:
+      """
+      describe "Other", :depth => 'shallow' do
+        let(:book) { Formulas::Bookie.new(@driver) }
+
+        it "loads an external web page", :bbb => 'aaa' do
+          book.open "http://www.google.com"
+        end
+        it "loads an external web page", :aaa => 'ccc' do
+          book.open "http://www.google.com"
+        end
+      end
+      """
+    When I run `ckit tags`
+    Then the stdout from "ckit tags" should contain:
+      """
+      ....
+      Tags used in harness:
+
+      aaa:ccc
+      bbb:aaa
+      bbb:ddd
+      depth:shallow
+
+
+      Finished
+      """

data/lib/chemistrykit/cli/cli.rb

@@ -27,21 +27,44 @@ module ChemistryKit
     class CKitCLI < Thor
 
       register(ChemistryKit::CLI::New, 'new', 'new [NAME]', 'Creates a new ChemistryKit project')
+
       check_unknown_options!
       default_task :help
 
       desc 'generate SUBCOMMAND', 'generate <formula> or <beaker> [NAME]'
       subcommand 'generate', Generate
 
+      desc 'tags', 'Lists all tags in use in the test harness.'
+      def tags
+        beakers = Dir.glob(File.join(Dir.getwd, 'beakers/*'))
+        RSpec.configure do |c|
+          c.add_setting :used_tags
+          c.before(:suite) { RSpec.configuration.used_tags = [] }
+          c.around(:each) do |example|
+            standard_keys = [:example_group, :example_group_block, :description_args, :caller, :execution_result]
+            example.metadata.each do |key, value|
+              tag = "#{key}:#{value}" unless standard_keys.include?(key)
+              RSpec.configuration.used_tags.push tag unless RSpec.configuration.used_tags.include?(tag) || tag.nil?
+            end
+          end
+          c.after(:suite) do
+            puts "\nTags used in harness:\n\n"
+            puts RSpec.configuration.used_tags.sort
+          end
+        end
+        RSpec::Core::Runner.run(beakers)
+      end
+
       desc 'brew', 'Run ChemistryKit'
       method_option :params, type: :hash
-      method_option :tag, default: ['depth:shallow'], type: :array
+      method_option :tag, type: :array
       method_option :config, default: 'config.yaml', aliases: '-c', desc: 'Supply alternative config file.'
       # TODO there should be a facility to simply pass a path to this command
-      method_option :beakers, type: :array
+      method_option :beakers, aliases: '-a', type: :array
       # This is set if the thread is being run in parallel so as not to trigger recursive concurency
       method_option :parallel, default: false
-      method_option :results_file, default: false
+      method_option :results_file, aliases: '-r', default: false, desc: 'Specifiy the name of your results file.'
+      method_option :all, default: false, aliases: '-a', desc: 'Run every beaker.', type: :boolean
 
       def brew
         config = load_config options['config']
@@ -53,14 +76,35 @@ module ChemistryKit
         config = override_configs options, config
 
         load_page_objects
-        setup_tags
-        # configure rspec
-        rspec_config(config)
+
         # get those beakers that should be executed
         beakers = options['beakers'] ? options['beakers'] : Dir.glob(File.join(Dir.getwd, 'beakers/*'))
+
+        if options['beakers']
+          # if a beaker(s) defined use them
+          beakers = options['beakers']
+          # if tags are explicity defined, apply them to the selected beakers
+          setup_tags(options['tag'])
+        else
+          # beakers default to everything
+          beakers = Dir.glob(File.join(Dir.getwd, 'beakers/*'))
+
+          if options['tag']
+
+            # if tags are explicity defined, apply them to all beakers
+            setup_tags(options['tag'])
+          else
+            # else choose the default tag
+            setup_tags(['depth:shallow'])
+          end
+        end
+
+        # configure rspec
+        rspec_config(config)
+
         # based on concurrency parameter run tests
         if config.concurrency > 1 && ! options['parallel']
-          run_in_parallel beakers, config.concurrency
+          run_in_parallel beakers, config.concurrency, @tags
         else
           run_rspec beakers
         end
@@ -93,9 +137,9 @@ module ChemistryKit
         ChemistryKit::Configuration.initialize_with_yaml config_file
       end
 
-      def setup_tags
+      def setup_tags(selected_tags)
         @tags = {}
-        options['tag'].each do |tag|
+        selected_tags.each do |tag|
           filter_type = tag.start_with?('~') ? :exclusion_filter : :filter
 
           name, value = tag.gsub(/^(~@|~|@)/, '').split(':')
@@ -105,7 +149,7 @@ module ChemistryKit
 
           @tags[filter_type] ||= {}
           @tags[filter_type][name] = value
-        end
+        end unless selected_tags.nil?
       end
 
       def rspec_config(config) # Some of these bits work and others don't
@@ -114,23 +158,17 @@ module ChemistryKit
         end
         RSpec.configure do |c|
           c.treat_symbols_as_metadata_keys_with_true_values = true
-          c.filter_run @tags[:filter] unless @tags[:filter].nil?
-          c.filter_run_excluding @tags[:exclusion_filter] unless @tags[:exclusion_filter].nil?
-          c.before(:all) do
-            # set the config available globaly
-            @config = config
-            # assign base url to env variable for formulas
-            ENV['BASE_URL'] = config.base_url
-          end
-          c.before(:each) do
-            @driver = SeleniumConnect.start
+          unless options[:all]
+            c.filter_run @tags[:filter] unless @tags[:filter].nil?
+            c.filter_run_excluding @tags[:exclusion_filter] unless @tags[:exclusion_filter].nil?
           end
-          c.after(:each) do
-            @driver.quit
-          end
-          c.after(:all) do
-            SeleniumConnect.finish
+          c.before(:all) do
+            @config = config # set the config available globaly
+            ENV['BASE_URL'] = config.base_url # assign base url to env variable for formulas
           end
+          c.before(:each) { @driver = SeleniumConnect.start }
+          c.after(:each) { @driver.quit }
+          c.after(:all) { SeleniumConnect.finish }
           c.order = 'random'
           c.default_path = 'beakers'
           c.pattern = '**/*_beaker.rb'
@@ -142,9 +180,10 @@ module ChemistryKit
         end
       end
 
-      def run_in_parallel(beakers, concurrency)
-
-        ParallelTests::CLI.new.run(%w(--type rspec) + ['-n', concurrency.to_s] + %w(-o --beakers=) + beakers)
+      def run_in_parallel(beakers, concurrency, tags)
+        tag_string = tags.empty? ? nil : '--tag=' + tags[:filter].map { |k, v| "#{k}:#{v}" }.join(' ')
+        args = %w(--type rspec) + ['-n', concurrency.to_s] + ['-o', "#{tag_string} --beakers="] + beakers
+        ParallelTests::CLI.new.run(args)
       end
 
       def run_rspec(beakers)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: chemistrykit
 version: !ruby/object:Gem::Version
-  version: 3.2.0
+  version: 3.3.0
   prerelease: 
 platform: ruby
 authors:
@@ -10,7 +10,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-07-08 00:00:00.000000000 Z
+date: 2013-07-09 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: thor
@@ -220,7 +220,7 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: 0.9.0
-description: Updated Selenium Connect
+description: Fixed bugs with tagging and concurrency
 email:
 - dave@arrgyle.com
 - jason@arrgyle.com
@@ -255,6 +255,7 @@ files:
 - features/new.feature
 - features/step_definitions/steps.rb
 - features/support/env.rb
+- features/tags.feature
 - lib/chemistrykit.rb
 - lib/chemistrykit/catalyst.rb
 - lib/chemistrykit/cli/beaker.rb
@@ -301,7 +302,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 3641514315441920438
+      hash: -1364358405018714304
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.25
@@ -321,6 +322,7 @@ test_files:
 - features/new.feature
 - features/step_definitions/steps.rb
 - features/support/env.rb
+- features/tags.feature
 - spec/chemistrykit/catalyst_spec.rb
 - spec/chemistrykit/cli/helpers/formula_loader_spec.rb
 - spec/chemistrykit/configuration_spec.rb