controlrepo 2.0.0 → 2.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: d27924d8a371e6fcc60c20a749013cb8e9eb1240
-  data.tar.gz: 9425a314523f65fe3e5321958f5019b0e7452dab
+  metadata.gz: b60c1c6a0508a99a08fefac24672a0e78d117a78
+  data.tar.gz: a0ec19878c1fa9a05db40bbf9f8f0a214cae5e53
 SHA512:
-  metadata.gz: c428c645689a91b7e8bc939e6d69d993ddee30880173249b518fb853f94fc24def1b1994596204aca98d95e7139b3118a9322cef1d6798f2d7bd86c897c81e2e
-  data.tar.gz: 4ccca700d874ad4c42364c75270811d6b04b2050feb5096337a8b8444661acbf467be5ad929ea6fe52b2daea3b764a63b0140a06dab4137d5538f25d0e207f89
+  metadata.gz: 9b042a8f6fbc832b9113826c526dcc24025be5ef2f6087c159f0f51c8b60cee0b9b4300c01933a74ed49a6dfd084b62bd7693353935640133a282a4efc6cc03e
+  data.tar.gz: eb0d0b250c11997aef57348e2f4a595aff90c468b1ca8036351c5d2588fe619523a837f8fa8f5e6d29985a96a3fba6542d88280aa6fd8be6b73eb70c7b998a9c

data/README.md

@@ -45,6 +45,10 @@ If we are doing spec testing we need sets of facts to compile the puppet code ag
 
 If we are doing acceptance testing then we need information about how to spin up VMs to do the testing on, these are configured in [nodesets](#nodesets).
 
+There is one thing that is not configured using config files and that is the **environment** to test. To change the environment that tests run in simply use the `CONTROLREPO_env` environment variable e.g.
+
+`CONTROLREPO_env=development bundle exec rake controlrepo_spec`
+
 ### controlrepo.yaml 
 
 `spec/controlrepo.yaml`
@@ -55,11 +59,23 @@ Hopefully this config file will be fairly self explanatory once you see it, but
 
 **nodes:** The nodes that we want to test against. The nodes that we list here map directly to either a [factset](#factsets) or a [nodeset](#nodesets) depending on weather we are running spec or acceptance tests respectively.
 
-**groups:** The groups section is just for saving us some typing. Here we can set up groups of classes *or* nodes (not a combination of the two) which we can then refer to in our test matrix. There are also two default groups automatically created which you can use, **all_classes** and **all_nodes**. If you can't guess what they are go have a coffee and come back.
+**node_groups:** The `node_groups` section is just for saving us some typing. Here we can set up groups of nodes which we can then refer to in our test matrix. We can create groups by simply specifying an array of servers to be in the group, or we can use the subtractive *include/exclude* syntax.
 
-**test_matrix:** This where the action happens! This is the section where we set up which classes are going to be tested against which nodes. It should be a hash with the node/s as the key and class/es as the values. This is where we would use groups if we had them (you can see us using the `windows_roles` and `centos_servers` groups in the below example). We can also use groups to subtractively build up lists of classes by using `include` and `exclude` as per the example below.
+**class_groups:** The `class_groups` section is jmuch the same as the `node_groups` sections, except that it creates groups of classes, not groups of nodes (duh). All the same rules apply and you can also use the *include/exclude* syntax.
 
-Don't be afraid if your test matrix produces duplicate tests, the controlrepo gem de-duplicates the test matrix before flattening it and generating the tests, as a result there should never be any duplicate tests. This also means that tests might not come out in the order that you expect D:
+**test_matrix:** This where the action happens! This is the section where we set up which classes are going to be tested against which nodes. It should be an array of hashes with the following format:
+
+```yaml
+  - {nodes_to_test}:
+      classes: '{classes_to_test}'
+      tests: '{all_tests|acceptance|spec}' # One of the three
+      options: # Optional (haha, get it?)
+        {valid_option}: {value} # Check the doco for available options
+```
+
+Why an array of hashes? Well, that is so that we can refer to the same node or node group twice, which we may want/need to do. In the example below we have not referred to the same group twice but we have referred to `centos6a` and `centos7b` in all of out tests as they are in `all_nodes`, `non_windows_servers` and `centos_severs`. However we have left the more specific references to last. This is because entries in the test_matrix will override entries above them if applicable. Meaning that we are still only testing each class on the two centos servers once (Because the gem does de-duplication before running the tests), but also making sure we run `roles::frontend_webserver` twice before checking for idempotency. 
+
+A full example:
 
 ```yaml
 classes:
@@ -75,22 +91,60 @@ nodes:
   - server2008r2a
   - ubuntu1404a
 
-groups:
+node_groups:
+  centos_severs:
+    - centos6a
+    - centos7b
+  non_windows_servers:
+    include: 'all_nodes'
+    exclude: 'server2008r2a'
+
+class_groups:
   windows_roles:
     - 'roles::windows_server'
     - 'roles::backend_dbserver'
-  centos_servers:
-    - centos6a
-    - centos7b
+  non_windows_roles:
+    include: 'all_classes'
+    exclude: 'windows_roles'
 
 test_matrix:
-  server2008r2a: windows_roles
-  ubuntu1404a: 'roles::frontend_webserver'
-  centos_servers:
-    include: all_classes
-    exclude: windows_roles
+  - all_nodes:
+      classes: 'all_classes'
+      tests: 'spec'
+  - non_windows_servers:
+      classes: 'all_classes'
+      tests: 'all_tests'
+  - centos_severs:
+      classes: 'roles::frontend_webserver'
+      tests: 'acceptance'
+      options:
+        runs_before_idempotency: 2
+```
+
+**Include/Exclude syntax:** This can be used with either `node_groups` or `class_groups` and allows us to save some time by using existing groups to create new ones e.g.
+
+```yaml
+node_groups:
+  windows_nodes: # This has to be defined first
+    - sevrer2008r2
+    - server2012r2
+  non_windows:
+    include: 'all_nodes' # Start with all nodes
+    exclude: 'windows_nodes' # Then remove the windows ones from that list
 ```
 
+It's important to note that in order to reference a group using the *include/exclude* syntax is has to have been defined already i.e. it has to come above the group that references it (Makes sense right?)
+
+#### Test Options
+
+**check_idempotency** *Default: true*
+
+Weather or not to check that puppet will be idempotent
+
+**runs_before_idempotency** *Default: 1*
+
+The number of runs to try before checking that it is idempotent. Required for some things that require restarts of the server or restarts of puppet.
+
 ### factsets
 
 `spec/factsets/*.yaml`
@@ -103,7 +157,7 @@ Which will give raw json output of every fact which puppet knows about. Usually
 
 `puppet facts > fact_set_name.json`
 
-Once we have our factset all we need to do is copy it into `spec/factsets/` inside out controlrepo and commit it to version control. Factsets are named based on their filename, not the ane of the server they came from (Although you can, if you want). i.e the following factset file:
+Once we have our factset all we need to do is copy it into `spec/factsets/` inside out controlrepo and commit it to version control. Factsets are named based on their filename, not the name of the server they came from (Although you can, if you want). i.e the following factset file:
 
 `spec/factsets/server2008r2.yaml`
 
@@ -249,11 +303,11 @@ service { 'pe-puppetserver':
 }
 ```
 
-However this is going to pos an issue when we get to acceptance testing. Due to the fact that acceptance tests actually run the code, not just try to compile a catalog, it will not be able to find the 'pe-pupetserver' service and will fail. One way to get around this is to use some of the optional parameters to the service resource e.g.
+However this is going to pose an issue when we get to acceptance testing. Due to the fact that acceptance tests actually run the code, not just try to compile a catalog, it will not be able to find the 'pe-pupetserver' service and will fail. One way to get around this is to use some of the optional parameters to the service resource e.g.
 
 ```puppet
 # We are not going to actually have this service anywhere on our servers but
-# our code needs to refresh it. This is to trck puppet into doing nothing
+# our code needs to refresh it. This is to trick puppet into doing nothing
 service { 'pe-puppetserver':
   ensure     => 'running',
   enable     => false,

data/controlrepo.gemspec

@@ -3,7 +3,7 @@ $LOAD_PATH.unshift File.expand_path("../lib", __FILE__)
 
 Gem::Specification.new do |s|
   s.name        = "controlrepo"
-  s.version     = "2.0.0"
+  s.version     = "2.0.1"
   s.authors     = ["Dylan Ratcliffe"]
   s.email       = ["dylan.ratcliffe@puppetlabs.com"]
   s.homepage    = ""
@@ -24,4 +24,4 @@ Gem::Specification.new do |s|
   s.add_runtime_dependency 'bundler'
   s.add_runtime_dependency 'r10k'
   s.add_runtime_dependency 'puppet'
-end
+end

data/lib/controlrepo.rb

@@ -1,4 +1,3 @@
-require 'pry'
 require 'r10k/puppetfile'
 require 'erb'
 require 'json'
@@ -6,6 +5,12 @@ require 'yaml'
 require 'find'
 require 'pathname'
 require 'controlrepo/beaker'
+begin 
+  require 'pry' 
+rescue LoadError
+  # We don't care if i'ts not here, this is just used for
+  # debugging sometimes
+end
 
 class Controlrepo
   attr_accessor :root

data/lib/controlrepo/test.rb

@@ -5,11 +5,32 @@ class Controlrepo
     attr_accessor :nodes
     attr_accessor :classes
     attr_accessor :options
+    attr_reader   :default_options
 
     # This can accept a bunch of stuff. It can accept nodes, classes or groups anywhere
     # it will then detect them and expand them out into their respective objects so that
     # we just end up with a list of nodes and classes
     def initialize(on_these,test_this,options = {})
+      # Turn options into an empty hash is someone passed in a nil value
+      options ||= {}
+
+      # I copied this code off the internet, basically it allows us
+      # to refer to each key as either a string or an object
+      options.default_proc = proc do |h, k|
+        case k
+          when String then sym = k.to_sym; h[sym] if h.key?(sym)
+          when Symbol then str = k.to_s; h[str] if h.key?(str)
+        end
+      end
+
+      @default_options = {
+        'check_idempotency' => true,
+        'runs_before_idempotency' => 1
+      }
+
+      # Add defaults if they do not exist
+      options = @default_options.merge(options)
+
       @nodes = []
       @classes = []
       @options = options
@@ -92,13 +113,28 @@ class Controlrepo
       # This should take an array of tests and remove any duplicates from them
 
       # this will be an array of arrays, or maybe hashes
+      # TODO: Rewrite this so that it merges options hashes, or takes one, decide on the right behaviour
       combinations = []
       new_tests = []
       tests.each do |test|
         test.nodes.each do |node|
           test.classes.each do |cls|
             combo = {node => cls}
-            unless combinations.member?(combo)
+            if combinations.member?(combo)
+
+              # Find the right test object:
+              relevant_test = new_tests[new_tests.index do |a|
+                a.nodes[0] == node and a.classes[0] == cls
+              end]
+
+              # Delete all default values in the current options hash
+              test.options.delete_if do |key,value|
+                test.default_options[key] == value
+              end
+
+              # Merge the non-default options right on in there
+              relevant_test.options.merge!(test.options)
+            else
               combinations << combo
               new_tests << Controlrepo::Test.new(node,cls,test.options)
             end

data/lib/controlrepo/testconfig.rb

@@ -15,14 +15,16 @@ class Controlrepo
     attr_accessor :acceptance_tests
     attr_accessor :environment
 
-    def initialize(file, environment = 'production')
+    def initialize(file, environment = ENV['CONTROLREPO_env'])
       begin
         config = YAML.load(File.read(file))
       rescue YAML::ParserError
         raise "Could not parse the YAML file, check that it is valid YAML and that the encoding is correct"
       end
 
+
       @environment = environment
+      @environment ||= 'production'
       @classes = []
       @nodes = []
       @node_groups = []
@@ -39,19 +41,18 @@ class Controlrepo
       config['node_groups'].each { |name, members| @node_groups << Controlrepo::Group.new(name, members) }
       config['class_groups'].each { |name, members| @class_groups << Controlrepo::Group.new(name, members) }
 
-      config['test_matrix'].each do |machines, settings|
-        if settings['tests'] == 'spec'
-          @spec_tests << Controlrepo::Test.new(machines,settings['classes'],settings['options'])
-        elsif settings['tests'] == 'acceptance'
-          @acceptance_tests << Controlrepo::Test.new(machines,settings['classes'],settings['options'])
-        elsif settings['tests'] == 'all_tests'
-          test = Controlrepo::Test.new(machines,settings['classes'],settings['options'])
-          @spec_tests << test
-          @acceptance_tests << test
+      config['test_matrix'].each do |test_hash|
+        test_hash.each do |machines, settings|
+          if settings['tests'] == 'spec'
+            @spec_tests << Controlrepo::Test.new(machines,settings['classes'],settings['options'])
+          elsif settings['tests'] == 'acceptance'
+            @acceptance_tests << Controlrepo::Test.new(machines,settings['classes'],settings['options'])
+          elsif settings['tests'] == 'all_tests'
+            test = Controlrepo::Test.new(machines,settings['classes'],settings['options'])
+            @spec_tests << test
+            @acceptance_tests << test
+          end
         end
-        # TODO: Work out some way to set per-test options like idempotency
-        #@spec_tests << Controlrepo::Test.new(machines,roles)
-        #@acceptance_tests
       end
     end
 

data/templates/acceptance_test_spec.rb.erb

@@ -2,20 +2,24 @@ require 'spec_helper_acceptance'
 
 describe "Acceptance Testing" do
   before :each do |test|
+    # Steop out a bunch of time to make sure we are at 0
+    # Yes this stupid, i know
+    10.times { logger.step_out }
     #try to find the indentation level and mirror it in beaker logger
-    padding = ""
     # Get the depth and add that as padding
     (test.metadata[:scoped_id].split(':').count - 1).times do
-      padding << "  "
+      logger.step_in
     end
-    logger.line_prefix = padding
   end
 <% tests.each do |test| -%>
 <% test.nodes.each do |node| -%>
 <% test.classes.each do |cls| -%>
   describe "<%= cls.name %> on <%= node.name %>" do
     after :all do
-      logger.line_prefix = "    "
+      # We have to use step_in and step_out here
+      # otherwise everything will break, not my fault
+      10.times { logger.step_out }
+      2.times { logger.step_in }
       $nwm.cleanup
     end
 
@@ -71,7 +75,7 @@ describe "Acceptance Testing" do
         }.not_to raise_exception
       end
     end
-
+<% test.options['runs_before_idempotency'].times do %>
     describe "running puppet" do
       it "should run with no errors" do
         expect {
@@ -86,7 +90,8 @@ CODE
         }.not_to raise_exception
       end
     end
-
+<% end -%>
+<% if test.options['check_idempotency'] then %>
     describe "checking for idempotency" do
       it "should run with no changes" do
         expect {
@@ -101,6 +106,7 @@ CODE
         }.not_to raise_exception
       end
     end
+<% end -%>
   end
 <% end -%>
 <% end -%>

data/templates/spec_helper_acceptance.rb.erb

@@ -62,18 +62,6 @@ RSpec.configure do |c|
   #c.provision
   #c.validate
   #c.configure
-
-  # Destroy nodes if no preserve hosts
-  c.after :suite do
-    case options[:destroy]
-    when 'no'
-      # Don't cleanup
-    when 'onpass'
-      c.cleanup if RSpec.world.filtered_examples.values.flatten.none?(&:exception)
-    else
-      c.cleanup
-    end
-  end
 end
 
 # Set the number of lines it will print

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: controlrepo
 version: !ruby/object:Gem::Version
-  version: 2.0.0
+  version: 2.0.1
 platform: ruby
 authors:
 - Dylan Ratcliffe
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-11-11 00:00:00.000000000 Z
+date: 2015-11-16 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake