r10k 3.11.0 → 3.12.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 34dc5753178d4777a6d36e6806515345b8b0e133807d46b21a9468de0325a1ed
-  data.tar.gz: 830085cca185bb00292af40f5cc13e8a46d00645a7bdc7f16e0d48b98ca4df23
+  metadata.gz: f378346234a9e8a8d92b3acb64735f88ca6b8e8edd60aa33adad9a09d7efd537
+  data.tar.gz: 1b94403eb2e5cb420939d85b73ec299c709418afabb45705f70b1cfb8606a1f5
 SHA512:
-  metadata.gz: 26e01c3ae5cdd4b4c873aa04d233dbe8fbe1f1f9fa193add5fe03aba50d1db09604fc117fd26190b7a1c350a7a64df9bdeba1b33ee0697d90420813075fe7811
-  data.tar.gz: 84c8c37b4daa9f11a864c54df12834d7bc920207f5f1219516d49583ff88537b7a13772a0c812d2bc9ebb4ebd636da4124dd35fbc105681e36db1aee4f540961
+  metadata.gz: a3eda38f3fc6e79ea62082c7c71bd65383089dce82e87dc6473c1f278f1560239f3e2189df28ef23a9ce8e47f3446cdd63e8794344ee62895914353bdcdec6ee
+  data.tar.gz: 0dc89955ede8226bf59c7cd4f8e03d9a89688700f5767d0578884fce97643c937f3363dd0d965404fd2b900128b181874dfb129952d64891a3700516239a2af6

data/CHANGELOG.mkd

@@ -4,6 +4,15 @@ CHANGELOG
 Unreleased
 ----------
 
+3.12.0
+------
+
+- (RK-308) Provide a `forge.allow_puppetfile_override` setting that, when true, causes a `forge` declaration in the Puppetfile to override `forge.baseurl`. [#1214](https://github.com/puppetlabs/r10k/pull/1214)
+- (CODEMGMT-1415) Provide an `--incremental` flag to only sync those modules in a Puppetfile whose definitions have changed since last sync, or those whose versions could change. [#1200](https://github.com/puppetlabs/r10k/pull/1200)
+- (CODEMGMT-1454) Ensure missing repo caches are re-synced [#1210](https://github.com/puppetlabs/r10k/pull/1210)
+- (PF-2437) Allow token authentication to be used with the Forge. [#1192](https://github.com/puppetlabs/r10k/pull/1192)
+- Only run the module postrun command for environments in which the module was modified. [#1215](https://github.com/puppetlabs/r10k/issues/1215)
+
 3.11.0
 ------
 

data/doc/dynamic-environments/configuration.mkd

@@ -167,9 +167,8 @@ This defaults to 'https://forgeapi.puppetlabs.com'
 
 #### authorization_token
 
-The 'authorization_token' setting allows you to provide a token for authenticating to a
-custom Forge server. When set, 'baseurl' must also be set.
-You will need to prepend your token with 'Bearer ' if using Artifactory as your Forge server.
+The 'authorization_token' setting allows you to provide a token for authenticating to a Forge server.
+You will need to prepend your token with 'Bearer ' to authenticate to the Forge or when using your own Artifactory server.
 
 ```yaml
 forge:
@@ -177,6 +176,11 @@ forge:
   authorization_token: 'Bearer mysupersecretauthtoken'
 ```
 
+#### allow_puppetfile_override
+
+The `allow_puppetfile_override` setting causes r10k to respect [`forge` declarations](https://github.com/puppetlabs/r10k/blob/main/doc/puppetfile.mkd#forge)
+in Puppetfiles, overriding the `baseurl` setting and allowing per-environment configuration of the Forge URL.
+
 Deployment options
 ------------------
 

data/doc/dynamic-environments/usage.mkd

@@ -62,6 +62,32 @@ useful if you want to make sure that a given environment is fully up to date.
 
 - - -
 
+There is also a middle ground between updating all modules and updating no modules.
+It is often desirable to update the environment and then update only those modules
+whose definitions have changed in the Puppetfile, or whose content _could_ have
+changed since the last deployment (eg, Forge modules with their version set to
+`:latest` or Git modules who point to a `branch` ref).
+
+This can be achieved by assuming content is unchanged locally on disk. This is the
+opposite of what one would assume during a module development cycle, when a user
+might be making local edits to test code changes. However, in production, access
+to puppet code is usually locked down, and updates are deployed through automated
+invocations of R10K.
+
+In these cases, deploys where most modules are unchanged and reference exact
+versions (ie, not `:latest` or a branch as mentioned above), this invocation
+may shorten deployment times dozens of seconds if not minutes depending on how
+many modules meet the above criteria (approximately 1 minute for every 400 modules).
+
+To take advantage of this, set as many modules as possible in the Puppetfile to
+explicit, static version. These are released Forge versions, or Git modules using
+the `:tag`, or `:commit` keys. Git `:ref`s containing only the full 40 character
+commit SHA will also be treated as static versions. Then invoke a deploy with:
+
+    r10k deploy environment production --modules --incremental
+
+- - -
+
 Update a single environment and specify a default branch override:
 
     r10k deploy environment my_working_environment --modules --default-branch-override default_branch_override

data/doc/puppetfile.mkd

@@ -51,10 +51,9 @@ handles modules.
 ### forge
 
 The `forge` setting specifies which server that Forge based modules are fetched
-from. This is currently a noop and is provided for compatibility with
-librarian-puppet.
-
-R10k supports setting the Forge to use _globally_ in `r10k.yaml`. see [Configuration](/doc/dynamic-environments/configuration.mkd#baseurl) for details.
+from. This declaration is only respected if [`forge.allow_puppetfile_override`](/dynamic-environments/configuration.mkd#allow_puppetfile_override)
+is set to true in the main `r10k.yaml`. Otherwise, use [`forge.baseurl`](/doc/dynamic-environments/configuration.mkd#baseurl)
+to globally configure where modules should be downloaded from.
 
 ### moduledir
 

data/integration/tests/basic_functionality/basic_deployment.rb

@@ -0,0 +1,176 @@
+require 'git_utils'
+require 'r10k_utils'
+require 'master_manipulator'
+
+
+test_name 'Basic Environment Deployment Workflows'
+
+# This isn't a block because we want to use the local variables throughout the file
+step 'init'
+  @env_path             = on(master, puppet('config print environmentpath')).stdout.rstrip
+  r10k_fqp              = get_r10k_fqp(master)
+
+  control_repo_gitdir   = '/git_repos/environments.git'
+  control_repo_worktree = '/root/environments'
+  last_commit           = git_last_commit(master, control_repo_worktree)
+  git_provider          = ENV['GIT_PROVIDER']
+
+  config_path           = get_r10k_config_file_path(master)
+  config_backup_path    = "#{config_path}.bak"
+
+  puppetfile1 =<<-EOS
+    mod 'puppetlabs/apache', '0.10.0'
+    mod 'puppetlabs/stdlib', '8.0.0'
+    EOS
+
+  r10k_conf = <<-CONF
+    cachedir: '/var/cache/r10k'
+    git:
+      provider: '#{git_provider}'
+    sources:
+      control:
+        basedir: "#{@env_path}"
+        remote: "#{control_repo_gitdir}"
+    deploy:
+      purge_levels: ['deployment','environment','puppetfile']
+
+    CONF
+
+
+def and_stdlib_is_correct
+  metadata_path = "#{@env_path}/production/modules/stdlib/metadata.json"
+  on(master, "test -f #{metadata_path}", accept_all_exit_codes: true) do |result|
+    assert(result.exit_code == 0, 'stdlib content has been inappropriately purged')
+  end
+  metadata_info = JSON.parse(on(master, "cat #{metadata_path}").stdout)
+  assert(metadata_info['version'] == '8.0.0', 'stdlib deployed to wrong version')
+end
+
+teardown do
+  on(master, "mv #{config_backup_path} #{config_path}")
+  clean_up_r10k(master, last_commit, control_repo_worktree)
+end
+
+step 'Set up r10k and control repo' do
+
+  # Backup and replace r10k config
+  on(master, "mv #{config_path} #{config_backup_path}")
+  create_remote_file(master, config_path, r10k_conf)
+
+  # Place our Puppetfile in the control repo's production branch
+  git_on(master, 'checkout production', control_repo_worktree)
+  create_remote_file(master, "#{control_repo_worktree}/Puppetfile", puppetfile1)
+  git_add_commit_push(master, 'production', 'add Puppetfile for Basic Deployment test', control_repo_worktree)
+
+  # Ensure the production environment will be deployed anew
+  on(master, "rm -rf #{@env_path}/production")
+end
+
+test_path = "#{@env_path}/production/modules/apache/metadata.json"
+step 'Test initial environment deploy works' do
+  on(master, "#{r10k_fqp} deploy environment production --verbose=info") do |result|
+    assert(result.output =~ /.*Deploying module to .*apache.*/, 'Did not log apache deployment')
+    assert(result.output =~ /.*Deploying module to .*stdlib.*/, 'Did not log stdlib deployment')
+  end
+  on(master, "test -f #{test_path}", accept_all_exit_codes: true) do |result|
+    assert(result.exit_code == 0, 'Expected module in Puppetfile was not installed')
+  end
+
+  and_stdlib_is_correct
+end
+
+original_apache_info = JSON.parse(on(master, "cat #{test_path}").stdout)
+
+step 'Test second run of deploy updates control repo, but leaves moduledir untouched' do
+  puppetfile2 =<<-EOS
+    # Current latest of apache is 6.5.1 as of writing this test
+    mod 'puppetlabs/apache', :latest
+    mod 'puppetlabs/stdlib', '8.0.0'
+    mod 'puppetlabs/concat', '7.0.0'
+    EOS
+
+  git_on(master, 'checkout production', control_repo_worktree)
+  create_remote_file(master, "#{control_repo_worktree}/Puppetfile", puppetfile2)
+  git_add_commit_push(master, 'production', 'add Puppetfile for Basic Deployment test', control_repo_worktree)
+
+  on(master, "#{r10k_fqp} deploy environment production --verbose=info") do |result|
+    refute(result.output =~ /.*Deploying module to .*apache.*/, 'Inappropriately updated apache')
+    refute(result.output =~ /.*Deploying module to .*stdlib.*/, 'Inappropriately updated stdlib')
+  end
+
+  on(master, "test -f #{test_path}", accept_all_exit_codes: true) do |result|
+    assert(result.exit_code == 0, 'Expected module content in Puppetfile was inappropriately purged')
+  end
+
+  new_apache_info = JSON.parse(on(master, "cat #{test_path}").stdout)
+  on(master, "cat #{@env_path}/production/Puppetfile | grep ':latest'", accept_all_exit_codes: true) do |result|
+    assert(result.exit_code == 0, 'Puppetfile not updated on subsequent r10k deploys')
+  end
+
+  assert(original_apache_info['version'] == new_apache_info['version'] &&
+         new_apache_info['version'] == '0.10.0',
+         'Module content updated on subsequent r10k invocations w/o providing --modules')
+
+  on(master, "test -f #{@env_path}/production/modules/concat/metadata.json", accept_all_exit_codes: true) do |result|
+    assert(result.exit_code == 1, 'Module content deployed on subsequent r10k invocation w/o providing --modules')
+  end
+
+  and_stdlib_is_correct
+end
+
+step 'Test --modules updates modules' do
+  on(master, "#{r10k_fqp} deploy environment production --modules --verbose=info") do |result|
+    assert(result.output =~ /.*Deploying module to .*apache.*/, 'Did not log apache deployment')
+    assert(result.output =~ /.*Deploying module to .*stdlib.*/, 'Did not log stdlib deployment')
+    assert(result.output =~ /.*Deploying module to .*concat.*/, 'Did not log concat deployment')
+  end
+
+  on(master, "test -f #{test_path}", accept_all_exit_codes: true) do |result|
+    assert(result.exit_code == 0, 'Expected module content in Puppetfile was inappropriately purged')
+  end
+
+  on(master, "test -f #{@env_path}/production/modules/concat/metadata.json", accept_all_exit_codes: true) do |result|
+    assert(result.exit_code == 0, 'New module content was not deployed when providing --modules')
+  end
+
+  new_apache_info = JSON.parse(on(master, "cat #{test_path}").stdout)
+  apache_major_version = new_apache_info['version'].split('.').first.to_i
+  assert(apache_major_version > 5, 'Module not updated correctly using --modules')
+
+  and_stdlib_is_correct
+end
+
+step 'Test --modules --incremental deploys changed & dynamic modules, but not unchanged, static modules' do
+  puppetfile3 =<<-EOS
+    # Current latest of apache is 6.5.1 as of writing this test
+    mod 'puppetlabs/apache', :latest
+    mod 'puppetlabs/stdlib', '8.0.0'
+    mod 'puppetlabs/concat', '7.1.0'
+    EOS
+
+  git_on(master, 'checkout production', control_repo_worktree)
+  create_remote_file(master, "#{control_repo_worktree}/Puppetfile", puppetfile3)
+  git_add_commit_push(master, 'production', 'add Puppetfile for Basic Deployment test', control_repo_worktree)
+
+  on(master, "#{r10k_fqp} deploy environment production --modules --incremental --verbose=debug1") do |result|
+    assert(result.output =~ /.*Deploying module to .*apache.*/, 'Did not log apache deployment')
+    assert(result.output =~ /.*Deploying module to .*concat.*/, 'Did not log concat deployment')
+    assert(result.output =~ /.*Not updating module stdlib, assuming content unchanged.*/, 'Did not log notice of skipping stdlib')
+  end
+
+  on(master, "test -f #{test_path}", accept_all_exit_codes: true) do |result|
+    assert(result.exit_code == 0, 'Expected module content in Puppetfile was inappropriately purged')
+  end
+
+  new_apache_info = JSON.parse(on(master, "cat #{test_path}").stdout)
+  apache_major_version = new_apache_info['version'].split('.').first.to_i
+  assert(apache_major_version > 5, 'Module not updated correctly using --modules & --incremental')
+
+  concat_info = JSON.parse(on(master, "cat #{@env_path}/production/modules/concat/metadata.json").stdout)
+  concat_minor_version = concat_info['version'].split('.')[1].to_i
+  assert(concat_minor_version == 1, 'Module not updated correctly using --modules & --incremental')
+
+  and_stdlib_is_correct
+end
+
+

data/lib/r10k/action/deploy/environment.rb

@@ -41,12 +41,14 @@ module R10K
                 requested_environments: @argv.map { |arg| arg.gsub(/\W/,'_') },
                 default_branch_override: @default_branch_override,
                 generate_types: @generate_types || settings.dig(:deploy, :generate_types) || false,
-                preload_environments: true
+                preload_environments: true,
+                incremental: @incremental
               },
               modules: {
                 exclude_spec: settings.dig(:deploy, :exclude_spec),
                 requested_modules: [],
                 deploy_modules: @modules,
+                pool_size: @settings[:pool_size] || 4,
                 force: !@no_force, # force here is used to make it easier to reason about
               },
               purging: {
@@ -54,6 +56,9 @@ module R10K
                 purge_allowlist: read_purge_allowlist(settings.dig(:deploy, :purge_whitelist) || [],
                                                       settings.dig(:deploy, :purge_allowlist) || [])
               },
+              forge: {
+                allow_puppetfile_override: settings.dig(:forge, :allow_puppetfile_override) || false
+              },
               output: {}
             }
           })
@@ -130,6 +135,7 @@ module R10K
               envs.reject! { |e| !requested_envs.include?(e) } if requested_envs.any?
               postcmd = postcmd.map { |e| e.gsub('$modifiedenvs', envs.join(' ')) }
             end
+            logger.debug _("Executing postrun command.")
             subproc = R10K::Util::Subprocess.new(postcmd)
             subproc.logger = logger
             subproc.execute
@@ -237,6 +243,7 @@ module R10K
           super.merge(puppetfile: :modules,
                       modules: :self,
                       cachedir: :self,
+                      incremental: :self,
                       'no-force': :self,
                       'exclude-spec': :self,
                       'generate-types': :self,

data/lib/r10k/action/deploy/module.rb

@@ -39,10 +39,14 @@ module R10K
               },
               modules: {
                 exclude_spec: settings.dig(:deploy, :exclude_spec),
+                pool_size: @settings[:pool_size] || 4,
                 requested_modules: @argv.map.to_a,
                 # force here is used to make it easier to reason about
                 force: !@no_force
               },
+              forge: {
+                allow_puppetfile_override: settings.dig(:forge, :allow_puppetfile_override) || false
+              },
               purging: {},
               output: {}
             }
@@ -74,16 +78,18 @@ module R10K
         ensure
           if (postcmd = @settings[:postrun])
             if @modified_envs.any?
+              envs_to_run = @modified_envs.join(' ')
+              logger.debug _("Running postrun command for environments: %{envs_to_run}.") % { envs_to_run: envs_to_run }
+
               if postcmd.grep('$modifiedenvs').any?
-                envs_to_run = @modified_envs.join(' ')
-                logger.debug "Running postrun command for environments #{envs_to_run}."
                 postcmd = postcmd.map { |e| e.gsub('$modifiedenvs', envs_to_run) }
               end
+
               subproc = R10K::Util::Subprocess.new(postcmd)
               subproc.logger = logger
               subproc.execute
             else
-              logger.debug "No environments were modified, not executing postrun command."
+              logger.debug _("No environments were modified, not executing postrun command.")
             end
           end
         end
@@ -99,11 +105,10 @@ module R10K
           else
             logger.debug1(_("Updating modules %{modules} in environment %{env_path}") % {modules: @settings.dig(:overrides, :modules, :requested_modules).inspect, env_path: environment.path})
 
-            environment.deploy
+            updated_modules = environment.deploy
 
-            requested_mods = @settings.dig(:overrides, :modules, :requested_modules) || []
             # We actually synced a module in this env
-            if !((environment.modules.map(&:name) & requested_mods).empty?)
+            if !updated_modules.nil? && !updated_modules.empty?
               # Record modified environment for postrun command
               @modified_envs << environment.dirname
 

data/lib/r10k/action/puppetfile/check.rb

@@ -1,6 +1,6 @@
-require 'r10k/puppetfile'
 require 'r10k/action/base'
 require 'r10k/errors/formatting'
+require 'r10k/module_loader/puppetfile'
 
 module R10K
   module Action
@@ -8,11 +8,13 @@ module R10K
       class Check < R10K::Action::Base
 
         def call
-          pf = R10K::Puppetfile.new(@root,
-                                    {moduledir: @moduledir,
-                                     puppetfile_path: @puppetfile})
+          options = { basedir: @root }
+          options[:moduledir] = @moduledir if @moduledir
+          options[:puppetfile] = @puppetfile if @puppetfile
+
+          loader = R10K::ModuleLoader::Puppetfile.new(**options)
           begin
-            pf.load!
+            loader.load!
             $stderr.puts _("Syntax OK")
             true
           rescue => e

data/lib/r10k/action/puppetfile/install.rb

@@ -1,7 +1,7 @@
-require 'r10k/puppetfile'
-require 'r10k/errors/formatting'
-require 'r10k/action/visitor'
 require 'r10k/action/base'
+require 'r10k/content_synchronizer'
+require 'r10k/errors/formatting'
+require 'r10k/module_loader/puppetfile'
 require 'r10k/util/cleaner'
 
 module R10K
@@ -10,25 +10,31 @@ module R10K
       class Install < R10K::Action::Base
 
         def call
-          @ok = true
           begin
-            pf = R10K::Puppetfile.new(@root,
-                                      {moduledir: @moduledir,
-                                       puppetfile_path: @puppetfile,
-                                       force: @force || false})
-            pf.load!
-            pf.sync
+            options = { basedir: @root, overrides: { force: @force || false } }
+            options[:moduledir]  = @moduledir  if @moduledir
+            options[:puppetfile] = @puppetfile if @puppetfile
+
+            loader = R10K::ModuleLoader::Puppetfile.new(**options)
+            loaded_content = loader.load!
 
-            R10K::Util::Cleaner.new(pf.managed_directories,
-                                    pf.desired_contents,
-                                    pf.purge_exclusions).purge!
+            pool_size = @settings[:pool_size] || 4
+            modules   = loaded_content[:modules]
+            if pool_size > 1
+              R10K::ContentSynchronizer.concurrent_sync(modules, pool_size, logger)
+            else
+              R10K::ContentSynchronizer.serial_sync(modules, logger)
+            end
 
+            R10K::Util::Cleaner.new(loaded_content[:managed_directories],
+                                    loaded_content[:desired_contents],
+                                    loaded_content[:purge_exclusions]).purge!
+
+            true
           rescue => e
-            @ok = false
             logger.error R10K::Errors::Formatting.format_exception(e, @trace)
+            false
           end
-
-          @ok
         end
 
         private