controlrepo 2.0.5 → 2.0.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ca0e652346d47d0383c8397521b62e99bc58a2f3
-  data.tar.gz: 1bccf7aea0f1edc08bf68e0e8ce3100cf122d53e
+  metadata.gz: fca8b974111f0429aa19983cbe5ec16c40949101
+  data.tar.gz: 9e13273d647f71d693933c8ec1271281e9cfdeb3
 SHA512:
-  metadata.gz: 01d3774044136a039da6e1af72aa98a9cf113f6e7c8dc1f983e18144e59aec0fb61633259472caeac4eb632ca813400523b97a1d1348a0116c295b817a974049
-  data.tar.gz: 37dbea15fb95d51478988387c534a2b0e36bf8a43c152e08c2934ff88a8fb19203461b9dc0a650698df8607a179ee40d4654f2b534aa884282b841c88c53bc15
+  metadata.gz: f8ce767884b96072657755a95cdd97da34abb221719f33b1f234bde60bedf4c956ae926ec0503582beea7e91e0c982452377287855d3c588d25713fe5da687bc
+  data.tar.gz: a0c4c56281e7e6d948e58ecb7ffc885c370d9bc88edbd203e71c93efbcdac8ca67a53ed3f66adb91dfaa0029f5e2f1de462cc6c4486dd2289bb573b9ef2780c9

data/README.md

@@ -84,9 +84,9 @@ This gem can just be installed using `gem install` however I would recommend usi
 
 ## Config Files
 
-This project uses one main config file to determine what classes we should be testing and how, this is [controlrepo.yaml](#controlrepo.yaml). The `controlrepo.yaml` config file provides information about what classes to test when, however it needs more information than that: 
+This project uses one main config file to determine what classes we should be testing and how, this is [controlrepo.yaml](#controlrepo.yaml). The `controlrepo.yaml` config file provides information about what classes to test when, however it needs more information than that:
 
-If we are doing spec testing we need sets of facts to compile the puppet code against, these are stored in [factsets](#factsets). 
+If we are doing spec testing we need sets of facts to compile the puppet code against, these are stored in [factsets](#factsets).
 
 If we are doing acceptance testing then we need information about how to spin up VMs to do the testing on, these are configured in [nodesets](#nodesets).
 
@@ -94,7 +94,7 @@ There is one thing that is not configured using config files and that is the **e
 
 `CONTROLREPO_env=development bundle exec rake controlrepo_spec`
 
-### controlrepo.yaml 
+### controlrepo.yaml
 
 `spec/controlrepo.yaml`
 
@@ -118,7 +118,7 @@ Hopefully this config file will be fairly self explanatory once you see it, but
         {valid_option}: {value} # Check the doco for available options
 ```
 
-Why an array of hashes? Well, that is so that we can refer to the same node or node group twice, which we may want/need to do. In the example below we have not referred to the same group twice but we have referred to `centos6a` and `centos7b` in all of out tests as they are in `all_nodes`, `non_windows_servers` and `centos_severs`. However we have left the more specific references to last. This is because entries in the test_matrix will override entries above them if applicable. Meaning that we are still only testing each class on the two centos servers once (Because the gem does de-duplication before running the tests), but also making sure we run `roles::frontend_webserver` twice before checking for idempotency. 
+Why an array of hashes? Well, that is so that we can refer to the same node or node group twice, which we may want/need to do. In the example below we have not referred to the same group twice but we have referred to `centos6a` and `centos7b` in all of out tests as they are in `all_nodes`, `non_windows_servers` and `centos_severs`. However we have left the more specific references to last. This is because entries in the test_matrix will override entries above them if applicable. Meaning that we are still only testing each class on the two centos servers once (Because the gem does de-duplication before running the tests), but also making sure we run `roles::frontend_webserver` twice before checking for idempotency.
 
 A full example:
 
@@ -180,6 +180,8 @@ node_groups:
 
 It's important to note that in order to reference a group using the *include/exclude* syntax is has to have been defined already i.e. it has to come above the group that references it (Makes sense right?)
 
+**NOTE:** You can change where the gem creates it's temporary directory for running the tests by exporting the `CONTROLREPO_temp` environment variable.
+
 #### Test Options
 
 **check_idempotency** *Default: true*
@@ -321,7 +323,7 @@ This will do the following things:
 ## Using workarounds
 
 There may be situations where you cannot test everything that is in your puppet code, some common reasons for this include:
-  
+
   - Code is destined for a Puppet Master but the VM image is not a Puppet Master which means we can't restart certain services etc.
   - A file is being pulled from somewhere that is only accessible in production
   - Something is trying to connect to something else that does not exist
@@ -373,122 +375,47 @@ Here we are specifying custom commands to run for starting, stopping and checkin
 
 **NOTE:** If you need to run some pre_conditions during acceptance tests but not spec tests or vice versa you can check the status of the `$controlrepo_accpetance` variable. It will be `true` when run as an acceptance test and `undef` otherwise. If you want to limit pre_conditions to only certain nodes just use conditional logic based on facts like you normally would.
 
-## Extra Tooling
-
-I have provided some extra tools to use if you would prefer to write your own tests which I will go over here.
-
-### Accessing fact sets in a traditional RSpec test
-
-We can access all of our fact sets using `Controlrepo.facts`. Normally it would be implemented something like this:
-
-```ruby
-Controlrepo.facts.each do |facts|
-  context "on #{facts['fqdn']}" do
-    let(:facts) { facts }
-    it { should compile }
-  end
-end
-```
-
-### Accessing Roles in a traditional RSpec test
-
-This gem allows us to easily and dynamically test all of the roles and profiles in our environment against fact sets from all of the nodes to which they will be applied. All we need to do is create a spec test that calls out to the `Controlrepo` ruby class:
+### Other Rake tasks
 
-```ruby
-require 'spec_helper'
-require 'controlrepo'
-
-Controlrepo.roles.each do |role|
-  describe role do
-    Controlrepo.facts.each do |facts|
-      context "on #{facts['fqdn']}" do
-        let(:facts) { facts }
-        it { should compile }
-      end
-    end
-  end
-end
-```
-
-This will iterate over each role in the controlrepo and test that it compiles with each set of facts.
-
-The same can also be done with profiles just by using the profiles method instead:
+I have included a couple of little rake tasks to help get you started with testing your control repos. Set them up by adding this to your `Rakefile`
 
 ```ruby
-require 'spec_helper'
-require 'controlrepo'
-
-Controlrepo.profiles.each do |profile|
-  describe profile do
-    Controlrepo.facts.each do |facts|
-      context "on #{facts['fqdn']}" do
-        let(:facts) { facts }
-        it { should compile }
-      end
-    end
-  end
-end
+require 'controlrepo/rake_tasks'
 ```
 
-It is not limited to just doing simple "It should compile" tests. You can put any tests you want in here.
-
-Also since the `profiles`, `roles` and `facts` methods simply return arrays, you can iterate over them however you would like i.e. you could write a different set of tests for each profile and then just use the `facts` method to run those tests on every fact set.
-
-### Filtering
-
-You can also filter your fact sets based on the value of any fact, including structured facts. (It will drill down into nested hashes and match those too, it's not just a dumb equality match)
+The tasks are as follows:
 
-Just pass a hash to the `facts` method and it will return only the fact sets with facts that match the hash e.g. Testing a certain profile on against only your Windows fact sets:
+#### generate_controlrepo_yaml
 
-```ruby
-require 'spec_helper'
-require 'controlrepo'
-
-describe 'profile::windows_appserver' do
-  Controlrepo.facts({
-    'kernel' => 'windows'
-    }).each do |facts|
-    context "on #{facts['fqdn']}" do
-      let(:facts) { facts }
-      it { should compile }
-    end
-  end
-end
-```
+`bundle exec rake generate_controlrepo_yaml`
 
-### Using hiera data (In manual tests)
+This will try to generate a `controlrepo.yaml` file, it will:
 
-You can also point these tests at your hiera data, you do this as you [normally would](https://github.com/rodjek/rspec-puppet#enabling-hiera-lookups) with rspec tests. However we do provide one helper to make this marginally easier. `Controlrepo.hiera_config` will look for hiera.yaml in the root of your control repo and also the spec directory, you will however need to set up the file itself e.g.
+  - Parse your environment.conf to work out where your roles and profiles might live
+  - Find your roles classes and pre-polulate them into the "classes" section
+  - Look though all of the factsets that ship with the gem, and also the ones you have created under `spec/factsets/*.json`
+  - Populate the "nodes" section with all of the factsets it finds
+  - Create node groups of windows and non-windows nodes
+  - Create a basic test_matrix
 
-```ruby
-require 'controlrepo'
 
-RSpec.configure do |c|
-  c.hiera_config = Controlrepo.hiera_config_file
-end
-```
+#### generate_nodesets
 
-### Extra Configuration
+`bundle exec rake generate_nodesets`
 
-You can modify the regexes that the gem uses to filter classes that it finds into roles and profiles. Just set up a Controlrepo object and pass regexes to the below settings.
+This task will generate nodeset file required by beaker, based on the factsets that exist in the repository. If you have any fact sets for which puppetlabs has a compatible vagrant box (i.e. centos, debian, ubuntu) it will detect the version specifics and set up the nodeset file, complete with box URL. If it doesn't know how to deal with a fact set it will output a boilerplate nodeset section and comment it out.
 
-```ruby
-repo = Controlrepo.new()
-repo.role_regex = /.*/ # Tells the class how to find roles, will be applied to repo.classes
-repo.profile_regex = /.*/ # Tells the class how to find profiles, will be applied to repo.classes
-```
+#### controlrepo_temp_create
 
-Note that you will need to call the `roles` and `profiles` methods on the object you just instantiated, not the main class e.g. `repo.roles` not `Controlrepo.roles`
+This will create a directory including the Controlrepo and all required modules where the environment variable `CONTROLREPO_temp` is set to. This is designed to make it easier to rebuild a cache when you are caching the modules on a build server.
 
-### Rake tasks
+#### hiera_setup
 
-I have included a couple of little rake tasks to help get you started with testing your control repos. Set them up by adding this to your `Rakefile`
+`bundle exec rake hiera_setup`
 
-```ruby
-require 'controlrepo/rake_tasks'
-```
+Automatically modifies your hiera.yaml to point at the hieradata relative to it's position.
 
-The tasks are as follows:
+This rake task will look for a hiera.yaml file (Using the same method we use for [this](#using-hiera-data)). It will then look for a hieradata directory in the root for your control repo (needs to match [this](http://rubular.com/?regex=%2Fhiera%28%3F%3A.%2Adata%29%3F%2Fi)). It will then modify the datadirs of any backends it finds in `hiera.yaml` to point at these directories.
 
 #### generate_fixtures
 
@@ -529,32 +456,3 @@ fixtures:
 ```
 
 Notice that the symlinks are not the ones that we provided in `environment.conf`? This is because the rake task will go into each of directories, find the modules and create a symlink for each of them (This is what rspec expects).
-
-#### generate_controlrepo_yaml
-
-`bundle exec rake generate_controlrepo_yaml`
-
-This will try to generate a `controlrepo.yaml` file, it will:
-
-  - Parse your environment.conf to work out where your roles and profiles might live
-  - Find your roles classes and pre-polulate them into the "classes" section
-  - Look though all of the factsets that ship with the gem, and also the ones you have created under `spec/factsets/*.json`
-  - Populate the "nodes" section with all of the factsets it finds
-  - Create node groups of windows and non-windows nodes
-  - Create a basic test_matrix
-
-
-#### generate_nodesets
-
-`bundle exec rake generate_nodesets`
-
-This task will generate nodeset file required by beaker, based on the factsets that exist in the repository. If you have any fact sets for which puppetlabs has a compatible vagrant box (i.e. centos, debian, ubuntu) it will detect the version specifics and set up the nodeset file, complete with box URL. If it doesn't know how to deal with a fact set it will output a boilerplate nodeset section and comment it out.
-
-#### hiera_setup
-
-`bundle exec rake hiera_setup`
-
-Automatically modifies your hiera.yaml to point at the hieradata relative to it's position.
-
-This rake task will look for a hiera.yaml file (Using the same method we use for [this](#using-hiera-data)). It will then look for a hieradata directory in the root for your control repo (needs to match [this](http://rubular.com/?regex=%2Fhiera%28%3F%3A.%2Adata%29%3F%2Fi)). It will then modify the datadirs of any backends it finds in `hiera.yaml` to point at these directories.
-

data/controlrepo.gemspec

@@ -3,7 +3,7 @@ $LOAD_PATH.unshift File.expand_path("../lib", __FILE__)
 
 Gem::Specification.new do |s|
   s.name        = "controlrepo"
-  s.version     = "2.0.5"
+  s.version     = "2.0.6"
   s.authors     = ["Dylan Ratcliffe"]
   s.email       = ["dylan.ratcliffe@puppetlabs.com"]
   s.homepage    = "https://github.com/dylanratcliffe/controlrepo_gem"

data/lib/controlrepo.rb

@@ -16,10 +16,9 @@ class Controlrepo
   attr_accessor :root
   attr_accessor :puppetfile
   attr_accessor :facts_files
+  attr_accessor :environmentpath
   attr_accessor :role_regex
   attr_accessor :profile_regex
-  attr_accessor :temp_environmentpath
-  attr_accessor :tempdir
   attr_accessor :spec_dir
   attr_accessor :temp_modulepath
   attr_accessor :nodeset_file
@@ -87,6 +86,7 @@ class Controlrepo
       raise e
     end
     @root = search_path
+    @environmentpath = 'etc/puppetlabs/code/environments'
     @puppetfile = File.expand_path('./Puppetfile',@root)
     @environment_conf = File.expand_path('./environment.conf',@root)
     @facts_dir = File.expand_path('./spec/factsets',@root)
@@ -95,8 +95,7 @@ class Controlrepo
     @nodeset_file = File.expand_path('./spec/acceptance/nodesets/controlrepo-nodes.yml',@root)
     @role_regex = /role[s]?:{2}/
     @profile_regex = /profile[s]?:{2}/
-    @temp_environmentpath = nil
-    @tempdir = nil
+    @tempdir = nil || ENV['CONTROLREPO_temp']
     $temp_modulepath = nil
     @manifest = config['manifest'] ? File.expand_path(config['manifest'],@root) : nil
   end
@@ -114,6 +113,20 @@ class Controlrepo
     END
   end
 
+  def tempdir
+    if ENV['CONTROLREPO_temp'] == ''
+      return nil
+    elsif ENV['CONTROLREPO_temp'] == nil
+      return nil
+    else
+      return File.absolute_path(ENV['CONTROLREPO_temp'])
+    end
+  end
+
+  def tempdir=(dir)
+    ENV['CONTROLREPO_temp'] = dir
+  end
+
   def roles
     classes.keep_if { |c| c =~ @role_regex }
   end

data/lib/controlrepo/rake_tasks.rb

@@ -86,6 +86,13 @@ task :controlrepo_autotest_prep do
     @config.acceptance_tests.each { |test| @config.verify_acceptance_test(@repo,test) }
   end
 
+  puts Dir["#{@repo.tempdir}/*"]
+  # Only deploy r10k of we don't already have a directory
+  if Dir["#{@repo.tempdir}/*"].empty?
+    # Deploy r10k to a temp dir
+    @config.r10k_deploy_local(@repo)
+  end
+
   # Deploy r10k to a temp dir
   @config.r10k_deploy_local(@repo)
 
@@ -106,10 +113,13 @@ task :controlrepo_autotest_prep do
   @config.write_spec_helper_acceptance("#{@repo.tempdir}/spec",@repo)
 
   # Deduplicate and write the tests (Spec and Acceptance)
+  FileUtils.rm_rf("#{@repo.tempdir}/spec/classes")
+  FileUtils.mkdir("#{@repo.tempdir}/spec/classes")
   Controlrepo::Test.deduplicate(@config.spec_tests).each do |test|
     @config.write_spec_test("#{@repo.tempdir}/spec/classes",test)
   end
 
+  FileUtils.rm_rf("#{@repo.tempdir}/spec/acceptance/*")
   @config.write_acceptance_tests("#{@repo.tempdir}/spec/acceptance",Controlrepo::Test.deduplicate(@config.acceptance_tests))
 
   # Parse the current hiera config, modify, and write it to the temp dir
@@ -118,11 +128,11 @@ task :controlrepo_autotest_prep do
     hiera_config.each do |setting,value|
       if value.is_a?(Hash)
         if value.has_key?(:datadir)
-          hiera_config[setting][:datadir] = "#{@repo.temp_environmentpath}/#{@config.environment}/#{value[:datadir]}"
+          hiera_config[setting][:datadir] = "#{@repo.tempdir}/#{@repo.environmentpath}/production/#{value[:datadir]}"
         end
       end
     end
-    File.write("#{@repo.temp_environmentpath}/#{@config.environment}/hiera.yaml",hiera_config.to_yaml)
+    File.write("#{@repo.tempdir}/#{@repo.environmentpath}/production/hiera.yaml",hiera_config.to_yaml)
   end
 
   @config.create_fixtures_symlinks(@repo)
@@ -154,13 +164,11 @@ task :controlrepo_acceptance => [
   :controlrepo_autotest_acceptance
   ]
 
-
-
-task :r10k_deploy_local do
+task :controlrepo_temp_create do
   require 'controlrepo/testconfig'
-  @repo = Controlrepo.new
-  @config = Controlrepo::TestConfig.new("#{repo.spec_dir}/controlrepo.yaml")
-
+  repo = Controlrepo.new
+  config = Controlrepo::TestConfig.new("#{repo.spec_dir}/controlrepo.yaml")
+  FileUtils.rm_rf(repo.tempdir)
   # Deploy r10k to a temp dir
   config.r10k_deploy_local(repo)
 end

data/lib/controlrepo/testconfig.rb

@@ -16,7 +16,7 @@ class Controlrepo
     attr_accessor :acceptance_tests
     attr_accessor :environment
 
-    def initialize(file, environment = ENV['CONTROLREPO_env'])
+    def initialize(file)
       begin
         config = YAML.load(File.read(file))
       rescue Errno::ENOENT
@@ -25,9 +25,6 @@ class Controlrepo
         raise "Could not parse the YAML file, check that it is valid YAML and that the encoding is correct"
       end
 
-
-      @environment = environment
-      @environment ||= 'production'
       @classes = []
       @nodes = []
       @node_groups = []
@@ -102,37 +99,38 @@ class Controlrepo
       puppetcode.join("\n")
     end
 
-    def checkout_branch(working_dir, branch) 
-      Dir.chdir(working_dir)     
-      g = Git.open(working_dir)
-      
-      # if we are already on the right branch do nothing
-      if ! g.branch.current == @environment then
-        if g.branches.include? branch
-          g.branch(@environment).checkout
-        else
-          puts "Unable to checkout requested environment #{@environment}: branch not found"
-        end
-      end
-    end
-
     def r10k_deploy_local(repo = Controlrepo.new)
       require 'controlrepo'
-      tempdir = Dir.mktmpdir('r10k')
-      repo.tempdir = tempdir
-      temp_code_dir = "#{tempdir}/etc/puppetlabs/code/environments/#{@environment}"
-      repo.temp_environmentpath = temp_code_dir.chomp("/#{@environment}")
-      FileUtils.mkdir_p(temp_code_dir)
-      FileUtils.cp_r("#{Dir.pwd}/.", temp_code_dir)
-      checkout_branch(temp_code_dir, @environment)
-
-      # Pull the trigger!
-      Dir.chdir(temp_code_dir) do
-        system("r10k puppetfile install --verbose")
+      require 'pathname'
+      if repo.tempdir == nil
+        repo.tempdir = Dir.mktmpdir('r10k')
+      else
+        FileUtils.mkdir_p(repo.tempdir)
+      end
+
+      # We need to do the copy to a tempdir then move the tempdir to the
+      # destination
+      temp_controlrepo = Dir.mktmpdir('controlrepo')
+      FileUtils.cp_r(Dir["#{repo.root}/*"], "#{temp_controlrepo}")
+      FileUtils.mkdir_p("#{repo.tempdir}/#{repo.environmentpath}/production")
+      FileUtils.mv(Dir["#{temp_controlrepo}/*"], "#{repo.tempdir}/#{repo.environmentpath}/production",:force => true)
+      FileUtils.rm_rf(temp_controlrepo)
+
+      # Pull the trigger! If it's not already been pulled
+      if repo.tempdir
+        if File.directory?(repo.tempdir)
+          if Dir["#{repo.tempdir}/#{repo.environmentpath}/production/modules/*"].empty?
+            Dir.chdir("#{repo.tempdir}/#{repo.environmentpath}/production") do
+              system("r10k puppetfile install --verbose")
+            end
+          end
+        else
+          raise "#{repo.tempdir} is not a directory"
+        end
       end
 
-      # Return tempdir for use
-      tempdir
+      # Return repo.tempdir for use
+      repo.tempdir
     end
 
     def write_spec_test(location, test)
@@ -162,11 +160,11 @@ class Controlrepo
     end
 
     def write_spec_helper(location, repo)
-      environmentpath = repo.temp_environmentpath
+      environmentpath = "#{repo.tempdir}/#{repo.environmentpath}"
       modulepath = repo.config['modulepath']
       modulepath.delete("$basemodulepath")
       modulepath.map! do |path|
-        "#{environmentpath}/#{@environment}/#{path}"
+        "#{environmentpath}/production/#{path}"
       end
       modulepath = modulepath.join(":")
       repo.temp_modulepath = modulepath
@@ -178,6 +176,7 @@ class Controlrepo
     end
 
     def create_fixtures_symlinks(repo)
+      FileUtils.rm_rf("#{repo.tempdir}/spec/fixtures/modules")
       FileUtils.mkdir_p("#{repo.tempdir}/spec/fixtures/modules")
       repo.temp_modulepath.split(':').each do |path|
         Dir["#{path}/*"].each do |mod|

data/templates/spec_helper.rb.erb

@@ -4,6 +4,6 @@ RSpec.configure do |c|
   c.parser = 'future'
   c.environmentpath = '<%= environmentpath %>'
   c.module_path = '<%= modulepath %>'
-  c.hiera_config = '<%= environmentpath %>/<%= @environment %>/hiera.yaml'
+  c.hiera_config = '<%= environmentpath %>/production/hiera.yaml'
   c.manifest = '<%= repo.temp_manifest %>'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: controlrepo
 version: !ruby/object:Gem::Version
-  version: 2.0.5
+  version: 2.0.6
 platform: ruby
 authors:
 - Dylan Ratcliffe
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-02-08 00:00:00.000000000 Z
+date: 2016-02-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake