r10k 2.0.3 → 2.1.0

data/doc/faq.mkd

@@ -50,3 +50,70 @@ on your remote repositories:
 [bitbucket-default-branch]: https://answers.atlassian.com/questions/280944/how-to-change-main-branch-in-bitbucket "Changing the default branch on Bitbucket"
 [gitolite-v2-default-branch]: http://stackoverflow.com/questions/7091599/git-default-remote-branch-with-gitolite "Changing the default branch on Gitolite v2"
 [gitolite-v3-default-branch]: http://stackoverflow.com/questions/13949093/git-change-default-branch-gitolite "Changing the default branch on Gitolite v3"
+
+### How do I prevent r10k from removing modules in the `/modules` directory of my Git repository?
+
+By default, r10k will install modules specified in the Puppetfile into the
+`/modules` directory of each environment, but if you already use that directory
+and keep modules in it, r10k may think those modules are not meant to exist and
+may remove them.
+
+There are three ways of fixing this: including your local modules in the
+Puppetfile, moving the directory where r10k install Puppetfile sourced modules,
+or moving your modules.
+
+#### Including your local modules in the Puppetfile
+
+The Puppetfile has a concept of a "local" module, otherwise known as a module
+that r10k did not directly placed there but should not be removed. If you want
+to continue keep your modules in the `/modules` directory and still install
+external modules from the Puppetfile into that directory, you can add a `mod`
+directive to the Puppetfile for each of your local modules.
+
+```
+
+mod 'my_ntp', :local => true
+mod 'roles', :local => true
+mod 'profiles', :local => true
+
+# Include your external modules as usual
+mod 'puppetlabs/stdlib'
+mod 'puppetlabs/apache'
+```
+
+#### Move where the Puppetfile installs external modules
+
+Instead of having to add a module entry for each of your local modules, you can
+simply move where the Puppetfile installs modules with the `moduledir` setting.
+
+```
+# The moduledir setting must be set before any modules are created
+moduledir "external-modules"
+
+mod 'puppetlabs/stdlib'
+mod 'puppetlabs/apache'
+```
+
+In Puppet 3.6 and later you can create an `environment.conf` in the root of your
+environment to indicate which directories contain modules:
+
+```
+# environment.conf
+modulepath: modules:external-modules
+```
+
+#### Move your local modules
+
+Lastly, you can simply move your locally versioned modules to a separate
+directory to avoid conflicting over the `/modules` directory entirely. With this
+example as well you can use the `environment.conf file to tell Puppet which
+directories contain modules.
+
+```
+# environment.conf
+modulepath: internal-modules::modules
+```
+
+#### What does the name mean?
+
+It's explained in [this blog post](http://somethingsinistral.net/blog/rethinking-puppet-deployment/#addendum).

data/doc/git/providers.mkd

@@ -50,6 +50,43 @@ git:
   username: 'git'
 ```
 
+#### Supported transports with Rugged
+
+Rugged compiles libgit2 and and the Ruby bindings when the gem is installed. You
+may need libraries installed before you install the gem to use certain protocols
+to access git remote repositories.
+
+For ssh support, you need to have libssh2 installed (along with the relevant dev
+package/headers) before you install the Rugged gem.
+
+For https support on Linux, you need to have OpenSSL installed (along with the
+relevant dev package/headers) before you install the Rugged gem. OS X and
+Windows support should automatically include https support.
+
+You can check whether https or ssh support is included in your Rugged
+installation by using the following in irb and making sure the required feature
+is listed:
+
+```ruby
+irb(main):001:0> require('rugged')
+=> true
+irb(main):002:0> Rugged.features
+=> [:threads, :https, :ssh]
+irb(main):003:0> 
+```
+You will require the ':https' or ':ssh' features to use the respective protocols
+in your Puppetfile module references or in r10k.yaml. R10K 2.0.0 and later will
+automatically issue a warning if either feature is missing.
+
+libssh2 on Debian and Ubuntu is compiled against libgcrypto instead of OpenSSL
+[due to licensing reasons](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=668271), and
+unfortunately libgcrypto does not support a number of required operations,
+including reading from a private key file. You will need to either use shellgit
+or recompile your own libssh2-1 package to use OpenSSL on these distributions.
+If you see the following error message, this is the likely cause:
+
+    Failed to authenticate SSH session: Unable to extract public key from private key file: Method unimplemented in libgcrypt backend at /var/cache/r10k/ssh---git@git.example.com-sys-puppet.git
+
 Configuration
 -------------
 

data/doc/puppetfile.mkd

@@ -184,6 +184,79 @@ credentials may be visible in the process table when r10k is running. If you
 choose to supply SVN credentials make sure that the system running r10k is
 appropriately secured.
 
+### Local
+
+In the event you want to store locally written modules in your r10k-managed
+repository in the Puppetfile managed path, you can use the `:local` type.
+
+For instance, if you have a Git repository with the following structure:
+
+```
+# tree -L 2
+.
+├── environment.conf
+├── modules
+│   └── local_module
+└── Puppetfile
+
+4 directories, 2 files
+```
+
+And you want to prevent `local_module` from being removed, you can add a 'local'
+module in your Puppetfile:
+
+```
+mod 'local_module', :local => true
+
+# Include your other modules as normal
+mod 'branan/eight_hundred'
+mod 'puppetlabs/apache'
+```
+
+If you run r10k against this Git branch, you'll get the following:
+
+```
+# tree -L 2
+.
+├── environment.conf
+├── modules
+│   ├── apache
+│   ├── eight_hundred
+│   └── local_module
+└── Puppetfile
+
+4 directories, 2 files
+```
+
+#### Caveats
+
+This is a workaround for r10k not being able to determine that modules
+created via VCS should not be purged, but is not meant to be a long term
+solution. The general practice around having local and remote modules in the
+same Git repository is to move modules versioned into a separate directory, like
+so:
+
+```
+# tree -L 2
+.
+├── environment.conf
+├── site-modules
+│   └── local_module
+├── modules
+│   ├── apache
+│   └── eight_hundred
+└── Puppetfile
+
+4 directories, 2 files
+```
+
+Moving modules stored in the Git repository into a separate directory will
+remove the need to have Puppetfile entries for every locally versioned Puppet
+module.
+
+For more information see the [FAQ entry](faq.mkd#how-do-i-prevent-r10k-from-removing-modules-in-the-modules-directory-of-my-git-repository)
+on managing internal and external modules in the same directory.
+
 ## Environment variables
 
 It is possible to set an alternate name/location for your `Puppetfile` and

data/integration/lib/r10k_utils.rb

@@ -97,12 +97,13 @@ def clean_up_r10k(master, commit_sha, git_repo_path)
   environment_path = on(master, puppet('config', 'print', 'environmentpath')).stdout.rstrip
   prod_env_modules_path = File.join(environment_path, 'production', 'modules')
   prod_env_site_path = File.join(environment_path, 'production', 'site')
+  r10k_fqp = get_r10k_fqp(master)
 
   step 'Reset Git Repo to Known Good State'
   r10k_revert_environment(master, commit_sha, git_repo_path)
 
   step 'Restore Original "production" Environment'
-  on(master, 'r10k deploy environment -v')
+  on(master, "#{r10k_fqp} deploy environment -v")
 
   step 'Verify "production" Environment is at Original State'
   verify_production_environment(master)
@@ -134,6 +135,7 @@ def init_r10k_source_from_prod(master, git_repo_parent_path, git_repo_name, git_
   #Init
   env_path = on(master, puppet('config print environmentpath')).stdout.rstrip
   prod_env_path = File.join(env_path, 'production')
+  r10k_fqp = get_r10k_fqp(master)
 
   local_files_root_path = ENV['FILES'] || 'files'
   prod_env_config_path = 'pre-suite/prod_env.config'
@@ -161,6 +163,49 @@ def init_r10k_source_from_prod(master, git_repo_parent_path, git_repo_name, git_
 
   #Attempt to deploy environments.
   if deploy
-    on(master, 'r10k deploy environment -v')
+    on(master, "#{r10k_fqp} deploy environment -v")
   end
 end
+
+# Get the fully qualified path for r10k.
+#
+# ==== Arguments
+#
+# * +host+ - The Beaker host to introspect
+#
+# ==== Returns
+#
+# +string+ - The fully qualified path for r10k
+#
+# ==== Examples
+#
+# get_r10k_fqp(master)
+def get_r10k_fqp(host)
+  puppet_version = get_puppet_version(host)
+  if puppet_version < 4.0
+    fqp = '/opt/puppet/bin/r10k'
+  else
+    fqp = '/opt/puppetlabs/puppet/bin/r10k'
+  end
+  fqp
+end
+
+# Get the version of puppet that is installed on a host.
+#
+# ==== Arguments
+#
+# * +host+ - The Beaker host to introspect
+#
+# ==== Returns
+#
+# +float+ - the version of puppet as a float
+#
+# ==== Examples
+#
+# get_puppet_version(master)
+def get_puppet_version(host)
+  on(host, puppet('--version')) do |result|
+    @version =  result.stdout.match(/(\d){1}.(\d){1,2}.(\d){1,2}/)[0].to_f
+  end
+  @version
+end

data/integration/pre-suite/01_git_config.rb

@@ -19,20 +19,21 @@ git_manifest_template_path = File.join(local_files_root_path, 'pre-suite', 'git_
 git_manifest = ERB.new(File.read(git_manifest_template_path)).result(binding)
 
 step 'Get PE Version'
-pe_major = on(master, 'facter -p pe_major_version').stdout.rstrip
-pe_minor = on(master, 'facter -p pe_minor_version').stdout.rstrip
-pe_version = "#{pe_major}.#{pe_minor}".to_f
-
-if pe_version < 3.7
-  fail_test('This pre-suite requires PE 3.7 or above!')
-end
+pe_version = get_puppet_version(master)
+fail_test('This pre-suite requires PE 3.7 or above!') if pe_version < 3.7
 
 #Setup
 step 'Stub Forge on Master'
 stub_forge_on(master)
 
+step 'Read module path'
+on(master, puppet('config print basemodulepath')) do |result|
+  (result.stdout.include? ':') ? separator = ':' : separator = ';'
+  @module_path = result.stdout.split(separator).first
+end
+
 step 'Install "git" Module'
-on(master, puppet('module install puppetlabs-git --modulepath /opt/puppet/share/puppet/modules'))
+on(master, puppet("module install  puppetlabs-git --modulepath #{@module_path}"))
 
 step 'Install and Configure Git'
 on(master, puppet('apply'), :stdin => git_manifest, :acceptable_exit_codes => [0,2]) do |result|

data/integration/pre-suite/02_pe_r10k.rb

@@ -13,14 +13,11 @@ git_repo_path = '/git_repos'
 git_repo_name = 'environments'
 git_control_remote = File.join(git_repo_path, "#{git_repo_name}.git")
 git_provider = ENV['GIT_PROVIDER'] || 'shellgit'
+r10k_fqp = get_r10k_fqp(master)
 
-pe_major = on(master, 'facter -p pe_major_version').stdout.rstrip
-pe_minor = on(master, 'facter -p pe_minor_version').stdout.rstrip
-pe_version = "#{pe_major}.#{pe_minor}".to_f
-
-if pe_version < 3.7
-  fail_test('This pre-suite requires PE 3.7 or above!')
-end
+step 'Get PE Version'
+pe_version = get_puppet_version(master)
+fail_test('This pre-suite requires PE 3.7 or above!') if pe_version < 3.7
 
 #In-line files
 r10k_conf = <<-CONF
@@ -42,7 +39,7 @@ create_remote_file(master, r10k_config_path, r10k_conf)
 on(master, "chmod 644 #{r10k_config_path}")
 
 step 'Deploy "production" Environment via r10k'
-on(master, 'r10k deploy environment -v')
+on(master, "#{r10k_fqp} deploy environment -v")
 
 step 'Disable Environment Caching on Master'
 on(master, puppet('config set environment_timeout 0 --section main'))

data/integration/scripts/setup_r10k_env_centos5.sh

@@ -7,7 +7,7 @@ if [ $SCRIPT_BASE_PATH = "scripts" ]; then
   cd ../
 fi
 
-export pe_dist_dir=http://neptune.puppetlabs.lan/3.8/ci-ready/
+export pe_dist_dir=http://neptune.puppetlabs.lan/4.0/ci-ready/
 export GIT_PROVIDER=shellgit
 
 bundle install --path .bundle/gems

data/integration/scripts/setup_r10k_env_centos6.sh

@@ -7,7 +7,7 @@ if [ $SCRIPT_BASE_PATH = "scripts" ]; then
   cd ../
 fi
 
-export pe_dist_dir=http://neptune.puppetlabs.lan/3.8/ci-ready/
+export pe_dist_dir=http://neptune.puppetlabs.lan/4.0/ci-ready/
 export GIT_PROVIDER=shellgit
 
 bundle install --path .bundle/gems

data/integration/scripts/setup_r10k_env_rhel7.sh

@@ -7,7 +7,7 @@ if [ $SCRIPT_BASE_PATH = "scripts" ]; then
   cd ../
 fi
 
-export pe_dist_dir=http://neptune.puppetlabs.lan/3.8/ci-ready/
+export pe_dist_dir=http://neptune.puppetlabs.lan/4.0/ci-ready/
 export GIT_PROVIDER=shellgit
 
 bundle install --path .bundle/gems

data/integration/scripts/setup_r10k_env_sles11.sh

@@ -7,7 +7,7 @@ if [ $SCRIPT_BASE_PATH = "scripts" ]; then
   cd ../
 fi
 
-export pe_dist_dir=http://neptune.puppetlabs.lan/3.8/ci-ready/
+export pe_dist_dir=http://neptune.puppetlabs.lan/4.0/ci-ready/
 export GIT_PROVIDER=shellgit
 
 bundle install --path .bundle/gems

data/integration/scripts/setup_r10k_env_sles12.sh

@@ -7,7 +7,7 @@ if [ $SCRIPT_BASE_PATH = "scripts" ]; then
   cd ../
 fi
 
-export pe_dist_dir=http://neptune.puppetlabs.lan/3.8/ci-ready/
+export pe_dist_dir=http://neptune.puppetlabs.lan/4.0/ci-ready/
 export GIT_PROVIDER=shellgit
 
 bundle install --path .bundle/gems

data/integration/scripts/setup_r10k_env_ubuntu1004.sh

@@ -7,7 +7,7 @@ if [ $SCRIPT_BASE_PATH = "scripts" ]; then
   cd ../
 fi
 
-export pe_dist_dir=http://neptune.puppetlabs.lan/3.8/ci-ready/
+export pe_dist_dir=http://neptune.puppetlabs.lan/4.0/ci-ready/
 export GIT_PROVIDER=shellgit
 
 bundle install --path .bundle/gems

data/integration/scripts/setup_r10k_env_ubuntu1204.sh

@@ -7,7 +7,7 @@ if [ $SCRIPT_BASE_PATH = "scripts" ]; then
   cd ../
 fi
 
-export pe_dist_dir=http://neptune.puppetlabs.lan/3.8/ci-ready/
+export pe_dist_dir=http://neptune.puppetlabs.lan/4.0/ci-ready/
 export GIT_PROVIDER=shellgit
 
 bundle install --path .bundle/gems

data/integration/scripts/setup_r10k_env_ubuntu1404.sh

@@ -7,7 +7,7 @@ if [ $SCRIPT_BASE_PATH = "scripts" ]; then
   cd ../
 fi
 
-export pe_dist_dir=http://neptune.puppetlabs.lan/3.8/ci-ready/
+export pe_dist_dir=http://neptune.puppetlabs.lan/4.0/ci-ready/
 export GIT_PROVIDER=shellgit
 
 bundle install --path .bundle/gems

data/integration/test_run_scripts/all_tests-rugged-pe-centos6.sh

@@ -7,7 +7,7 @@ if [ $SCRIPT_BASE_PATH = "test_run_scripts" ]; then
   cd ../
 fi
 
-export pe_dist_dir=http://neptune.puppetlabs.lan/3.8/ci-ready/
+export pe_dist_dir=http://neptune.puppetlabs.lan/2015.2/ci-ready/
 export GIT_PROVIDER=rugged
 
 beaker \

data/integration/test_run_scripts/all_tests-rugged-pe-rhel7.sh

@@ -7,7 +7,7 @@ if [ $SCRIPT_BASE_PATH = "test_run_scripts" ]; then
   cd ../
 fi
 
-export pe_dist_dir=http://neptune.puppetlabs.lan/3.8/ci-ready/
+export pe_dist_dir=http://neptune.puppetlabs.lan/2015.2/ci-ready/
 export GIT_PROVIDER=rugged
 
 beaker \

data/integration/test_run_scripts/all_tests-rugged-pe-sles11.sh

@@ -7,7 +7,7 @@ if [ $SCRIPT_BASE_PATH = "test_run_scripts" ]; then
   cd ../
 fi
 
-export pe_dist_dir=http://neptune.puppetlabs.lan/3.8/ci-ready/
+export pe_dist_dir=http://neptune.puppetlabs.lan/2015.2/ci-ready/
 export GIT_PROVIDER=rugged
 
 beaker \

data/integration/test_run_scripts/all_tests-rugged-pe-ubuntu1204.sh

@@ -7,7 +7,7 @@ if [ $SCRIPT_BASE_PATH = "test_run_scripts" ]; then
   cd ../
 fi
 
-export pe_dist_dir=http://neptune.puppetlabs.lan/3.8/ci-ready/
+export pe_dist_dir=http://neptune.puppetlabs.lan/2015.2/ci-ready/
 export GIT_PROVIDER=rugged
 
 beaker \

data/integration/test_run_scripts/all_tests-rugged-pe-ubuntu1404.sh

@@ -7,7 +7,7 @@ if [ $SCRIPT_BASE_PATH = "test_run_scripts" ]; then
   cd ../
 fi
 
-export pe_dist_dir=http://neptune.puppetlabs.lan/3.8/ci-ready/
+export pe_dist_dir=http://neptune.puppetlabs.lan/2015.2/ci-ready/
 export GIT_PROVIDER=rugged
 
 beaker \