r10k 2.2.2 → 2.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (38) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.mkd +25 -0
  3. data/Gemfile +1 -1
  4. data/doc/dynamic-environments/configuration.mkd +36 -11
  5. data/integration/tests/basic_functionality/proxy_with_pe_only_module.rb +128 -0
  6. data/lib/r10k/errors.rb +18 -0
  7. data/lib/r10k/forge/module_release.rb +3 -3
  8. data/lib/r10k/git.rb +53 -0
  9. data/lib/r10k/git/rugged/bare_repository.rb +16 -7
  10. data/lib/r10k/git/rugged/base_repository.rb +16 -2
  11. data/lib/r10k/git/rugged/credentials.rb +9 -0
  12. data/lib/r10k/git/rugged/working_repository.rb +18 -5
  13. data/lib/r10k/git/shellgit/bare_repository.rb +12 -3
  14. data/lib/r10k/git/shellgit/base_repository.rb +19 -3
  15. data/lib/r10k/git/shellgit/working_repository.rb +13 -3
  16. data/lib/r10k/initializers.rb +1 -0
  17. data/lib/r10k/settings.rb +41 -22
  18. data/lib/r10k/settings/collection.rb +25 -21
  19. data/lib/r10k/settings/definition.rb +14 -2
  20. data/lib/r10k/settings/helpers.rb +38 -0
  21. data/lib/r10k/settings/list.rb +107 -0
  22. data/lib/r10k/util/symbolize_keys.rb +6 -2
  23. data/lib/r10k/version.rb +1 -1
  24. data/r10k.gemspec +1 -1
  25. data/r10k.yaml.example +38 -12
  26. data/spec/shared-examples/git/bare_repository.rb +62 -0
  27. data/spec/shared-examples/git/working_repository.rb +57 -11
  28. data/spec/shared-examples/settings/ancestry.rb +44 -0
  29. data/spec/unit/forge/module_release_spec.rb +1 -1
  30. data/spec/unit/git/rugged/credentials_spec.rb +6 -0
  31. data/spec/unit/settings/collection_spec.rb +2 -1
  32. data/spec/unit/settings/definition_spec.rb +6 -5
  33. data/spec/unit/settings/inheritance_spec.rb +38 -0
  34. data/spec/unit/settings/list_spec.rb +88 -0
  35. data/spec/unit/settings_spec.rb +52 -23
  36. data/spec/unit/util/attempt_spec.rb +1 -1
  37. data/spec/unit/util/symbolize_keys_spec.rb +25 -9
  38. metadata +11 -4
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 79e256f57d1cd80b724c490632856750dcf368f4
4
- data.tar.gz: 146b029e45575f462cf7438b4a795c9e68fcb5c7
3
+ metadata.gz: 79a29967cc56c3178634ae325404b4b201aa9841
4
+ data.tar.gz: 5ddfd39f4a437e52ed87adc06f08945a8c2eef07
5
5
  SHA512:
6
- metadata.gz: 8479890443ec53087bf5b8469d2d04fd6c0c325f6215a428b536a9bad1a91e67ac7d67d5dccf4786936e39bdc192b09925bfb74a39f262cfedbcd2fc881453d0
7
- data.tar.gz: baf085c12deb3a95664a3db9c90475dfba51ca6c9d39c8d0945c2572d040b0fb181daaf40bc74ebec06cf77dd1a4e1e596da472b0b63ae76e7cebf4d97c271e5
6
+ metadata.gz: 1da246a1e621b9b47f9a5baca4641160c8a71a0ba66f86849f28bac73a1341b586987b154e5e4b97aca66f074fae813c4a153119f2ccc508f34b8fe2ea4c1241
7
+ data.tar.gz: 64f55efe4f54dae8da9d6ff5c8d5ed28201dd80ca1202e73c9af22c9867e309cb009d3c4bc6454b3b88870656bb751bed68866816d711ad47c64c0dc295d83db
@@ -1,6 +1,31 @@
1
1
  CHANGELOG
2
2
  =========
3
3
 
4
+ 2.3.0
5
+ -----
6
+
7
+ 2016/05/17
8
+
9
+ ### New Features
10
+
11
+ (RK-236/RK-237) Added HTTP proxy support for Git operations.
12
+
13
+ Previously, r10k only supported the use of HTTP proxies for connecting to the Puppet
14
+ Forge. With these changes, r10k can now be configured to use an HTTP proxy for both
15
+ Forge and Git operations. Configuration can be specified globally, for Forge or Git
16
+ only, or on a per-Git repository basis. See [configuration documentation](https://github.com/puppetlabs/r10k/blob/master/doc/dynamic-environments/configuration.mkd)
17
+ for more details.
18
+
19
+ ### Bug Fixes
20
+
21
+ (RK-238) When r10k encounters and ignores invalid file types in a module archive, it
22
+ will now log the message at the DEBUG1 level instead of at WARN.
23
+
24
+ (RK-243) In certain cases, when using the "rugged" Git provider, specifying invalid HTTP
25
+ credentials for a repository could result in an infinite loop. Authentication retry
26
+ for HTTP repositories is now capped at 50 attempts which matches the existing behavior
27
+ for SSH.
28
+
4
29
  2.2.2
5
30
  -----
6
31
 
data/Gemfile CHANGED
@@ -3,7 +3,7 @@ source 'https://rubygems.org'
3
3
  gemspec
4
4
 
5
5
  group :extra do
6
- gem 'rugged', '~> 0.21.4', :platforms => :ruby
6
+ gem 'rugged', '~> 0.24.0', :platforms => :ruby
7
7
  end
8
8
 
9
9
  group :development do
@@ -52,6 +52,28 @@ The cachedir setting defaults to `~/.r10k`. If the HOME environment variable is
52
52
  unset r10k will assume that r10k is being run with the Puppet [`prerun_command`][prerun_command]
53
53
  setting and will set the cachedir default to `/root/.r10k`.
54
54
 
55
+ ### proxy
56
+
57
+ The 'proxy' setting configures a proxy server to use for all operations which occur over
58
+ an HTTP(S) transport. You can override this setting for Git or Forge operations only by
59
+ setting the 'proxy' setting under the 'git' or 'forge' settings. You can also override
60
+ for a specific Git repository by setting a proxy in the 'repositories' list of the 'git'
61
+ setting. By default, r10k will look for and use the first environment variable it finds
62
+ in this list: HTTPS\_PROXY, https\_proxy, HTTP\_PROXY, http\_proxy. If no proxy setting
63
+ is found in the environment, this setting will default to use no proxy.
64
+
65
+ ```yaml
66
+ proxy: 'http://proxy.example.com:3128'
67
+ ```
68
+
69
+ r10k also supports using authenticated proxies with either Basic or Digest authentication:
70
+
71
+ ```yaml
72
+ proxy: 'http://user:password@proxy.example.com:3128'
73
+ ```
74
+
75
+ The proxy server being used will be logged at the "debug" level when r10k runs.
76
+
55
77
  ### git
56
78
 
57
79
  The 'git' setting is a hash that contains Git specific settings.
@@ -68,6 +90,12 @@ git:
68
90
  See the [git provider documentation](../git/providers.mkd) for more information
69
91
  regarding Git providers.
70
92
 
93
+ #### proxy
94
+
95
+ The 'proxy' setting allows you to set or override the global proxy setting specifically
96
+ for Git operations that use an HTTP(S) transport. See the global proxy setting documentation
97
+ for more information and examples.
98
+
71
99
  #### username
72
100
 
73
101
  The username setting is only used by the Rugged git provider.
@@ -107,10 +135,15 @@ overrides the global private_key setting.
107
135
  ```yaml
108
136
  git:
109
137
  repositories:
110
- "ssh://tessier-ashpool.freeside/protected-repo.git":
138
+ - remote: "ssh://tessier-ashpool.freeside/protected-repo.git"
111
139
  private_key: "/etc/puppetlabs/r10k/ssh/id_rsa-protected-repo-deploy-key"
112
140
  ```
113
141
 
142
+ ##### proxy
143
+
144
+ The 'proxy' setting allows you to set or override the global proxy setting for a single, specific
145
+ repository. See the global proxy setting documentation for more information and examples.
146
+
114
147
  ### forge
115
148
 
116
149
  The 'forge' setting is a hash that contains settings for downloading modules
@@ -118,16 +151,8 @@ from the Puppet Forge.
118
151
 
119
152
  #### proxy
120
153
 
121
- The proxy option sets an optional HTTP proxy to use when downloading modules
122
- from the Forge.
123
-
124
- ```yaml
125
- forge:
126
- proxy: 'http://my-site.proxy:3128'
127
- ```
128
-
129
- If no proxy is given, r10k will check the environment variables 'HTTPS_PROXY',
130
- 'https_proxy', 'HTTP_PROXY', and 'http_proxy' in that order for a proxy URL.
154
+ The 'proxy' setting allows you to set or override the global proxy setting for all Forge
155
+ interactions. See the global proxy setting documentation for more information and examples.
131
156
 
132
157
  #### baseurl
133
158
 
@@ -0,0 +1,128 @@
1
+ require 'git_utils'
2
+ require 'r10k_utils'
3
+ require 'master_manipulator'
4
+ test_name 'RK-242 '#'- C87652 - Specify the proxy in the r10k.yaml'
5
+
6
+ confine(:to, :platform => ['el', 'sles'])
7
+
8
+ #Init
9
+ master_platform = fact_on(master, 'osfamily')
10
+ master_certname = on(master, puppet('config', 'print', 'certname')).stdout.rstrip
11
+ env_path = on(master, puppet('config print environmentpath')).stdout.rstrip
12
+ r10k_fqp = get_r10k_fqp(master)
13
+
14
+ git_repo_path = '/git_repos'
15
+ git_repo_name = 'environments'
16
+ git_control_remote = File.join(git_repo_path, "#{git_repo_name}.git")
17
+ git_environments_path = '/root/environments'
18
+ last_commit = git_last_commit(master, git_environments_path)
19
+ git_provider = ENV['GIT_PROVIDER']
20
+
21
+ local_files_root_path = ENV['FILES'] || 'files'
22
+
23
+ git_manifest_template_path = File.join(local_files_root_path, 'pre-suite', 'git_config.pp.erb')
24
+ git_manifest = ERB.new(File.read(git_manifest_template_path)).result(binding)
25
+
26
+ r10k_config_path = get_r10k_config_file_path(master)
27
+ r10k_config_bak_path = "#{r10k_config_path}.bak"
28
+
29
+ case master_platform
30
+ when 'RedHat'
31
+ pkg_manager = 'yum'
32
+ when 'Suse'
33
+ pkg_manager = 'zypper'
34
+ end
35
+
36
+ install_squid = "#{pkg_manager} install -y squid"
37
+ remove_squid = "#{pkg_manager} remove -y squid"
38
+ squid_log = "/var/log/squid/access.log"
39
+
40
+ #In-line files
41
+ r10k_conf = <<-CONF
42
+ cachedir: '/var/cache/r10k'
43
+ git:
44
+ provider: '#{git_provider}'
45
+ sources:
46
+ control:
47
+ basedir: "#{env_path}"
48
+ remote: "#{git_control_remote}"
49
+ forge:
50
+ proxy: "http://#{master.hostname}:3128"
51
+ CONF
52
+
53
+ #Manifest
54
+ site_pp_path = File.join(git_environments_path, 'manifests', 'site.pp')
55
+ site_pp = create_site_pp(master_certname, ' include peonly')
56
+
57
+ #Verification
58
+ squid_log_regex = /CONNECT forgeapi.puppetlabs.com:443/
59
+ notify_message_regex = /I am in the production environment, this is a PE only module/
60
+
61
+ #Teardown
62
+ teardown do
63
+ step 'remove license file'
64
+ on(master, 'rm -f /etc/puppetlabs/license.key')
65
+
66
+ step 'Restore "git" Package'
67
+ on(master, puppet('apply'), :stdin => git_manifest, :acceptable_exit_codes => [0,2])
68
+
69
+ step 'Restore Original "r10k" Config'
70
+ on(master, "mv #{r10k_config_bak_path} #{r10k_config_path}")
71
+
72
+ clean_up_r10k(master, last_commit, git_environments_path)
73
+
74
+ step 'Remove Squid'
75
+ on(master, puppet("apply -e 'service {'squid' : ensure => stopped}'"))
76
+ on(master, remove_squid)
77
+ end
78
+
79
+ #Setup
80
+ step 'Stub the forge'
81
+ stub_forge_on(master)
82
+
83
+ step 'Backup Current "r10k" Config'
84
+ on(master, "mv #{r10k_config_path} #{r10k_config_bak_path}")
85
+
86
+ step 'Update the "r10k" Config'
87
+ create_remote_file(master, r10k_config_path, r10k_conf)
88
+
89
+ step 'Download license file from int-resources'
90
+ curl_on(master, 'http://int-resources.ops.puppetlabs.net/QA_resources/r10k/license.key -o /etc/puppetlabs/license.key')
91
+
92
+ step 'Checkout "production" Branch'
93
+ git_on(master, 'checkout production', git_environments_path)
94
+
95
+ step 'Inject New "site.pp" to the "production" Environment'
96
+ inject_site_pp(master, site_pp_path, site_pp)
97
+
98
+ step 'Copy Puppetfile to "production" Environment with PE only module'
99
+ create_remote_file(master, "#{git_environments_path}/Puppetfile", 'mod "ztr-peonly"')
100
+
101
+ step 'Push Changes'
102
+ git_add_commit_push(master, 'production', 'add Puppetfile', git_environments_path)
103
+
104
+ step 'Install and configure squid proxy'
105
+ on(master, install_squid)
106
+
107
+ step 'turn off the firewall'
108
+ on(master, puppet("apply -e 'service {'iptables' : ensure => stopped}'"))
109
+
110
+ step 'start squid proxy'
111
+ on(master, puppet("apply -e 'service {'squid' : ensure => running}'"))
112
+
113
+ #Tests
114
+ step 'Deploy "production" Environment via r10k'
115
+ on(master, "#{r10k_fqp} deploy environment -p")
116
+
117
+ step 'Read the squid logs'
118
+ on(master, "cat #{squid_log}") do |result|
119
+ assert_match(squid_log_regex, result.stdout, 'Proxy logs did not indicate use of the proxy.')
120
+ end
121
+
122
+ agents.each do |agent|
123
+ step "Run Puppet Agent"
124
+ on(agent, puppet('agent', '--test', '--environment production'), :acceptable_exit_codes => [0,2]) do |result|
125
+ assert_no_match(/Error:/, result.stderr, 'Unexpected error was detected!')
126
+ assert_match(notify_message_regex, result.stdout, 'Expected message not found!')
127
+ end
128
+ end
@@ -39,5 +39,23 @@ module R10K
39
39
 
40
40
  @options = options
41
41
  end
42
+
43
+ protected
44
+
45
+ def structure_exception(name, exc)
46
+ struct = []
47
+ struct << "#{name}:"
48
+ if exc.respond_to?(:format)
49
+ struct << indent(exc.format)
50
+ else
51
+ struct << indent(exc.message)
52
+ end
53
+ struct.join("\n")
54
+ end
55
+
56
+ def indent(str, level = 4)
57
+ prefix = ' ' * level
58
+ str.gsub(/^/, prefix)
59
+ end
42
60
  end
43
61
  end
@@ -95,11 +95,11 @@ module R10K
95
95
  file_lists = PuppetForge::Unpacker.unpack(download_path.to_s, target_dir.to_s, unpack_path.to_s)
96
96
  logger.debug2 "Valid files unpacked: #{file_lists[:valid]}"
97
97
  if !file_lists[:invalid].empty?
98
- logger.warn "These files existed in the module's tar file, but are invalid filetypes and were not " +
99
- "unpacked: #{file_lists[:invalid]}"
98
+ logger.debug1 "These files existed in the module's tar file, but are invalid filetypes and were not " +
99
+ "unpacked: #{file_lists[:invalid]}"
100
100
  end
101
101
  if !file_lists[:symlinks].empty?
102
- raise R10K::Error, "Symlinks are unsupported and were not unpacked from the module tarball. " +
102
+ raise R10K::Error, "Symlinks are unsupported and were not unpacked from the module tarball. " +
103
103
  "#{@forge_release.slug} contained these ignored symlinks: #{file_lists[:symlinks]}"
104
104
  end
105
105
  end
@@ -1,3 +1,4 @@
1
+ require 'uri'
1
2
  require 'r10k/features'
2
3
  require 'r10k/errors'
3
4
  require 'r10k/settings'
@@ -133,11 +134,63 @@ module R10K
133
134
  extend R10K::Settings::Mixin::ClassMethods
134
135
 
135
136
  def_setting_attr :private_key
137
+ def_setting_attr :proxy
136
138
  def_setting_attr :username
137
139
  def_setting_attr :repositories, {}
138
140
 
139
141
  def self.get_repo_settings(remote)
140
142
  self.settings[:repositories].find {|r| r[:remote] == remote }
141
143
  end
144
+
145
+ def self.get_proxy_for_remote(remote)
146
+ # We only support proxy for HTTP(S) transport
147
+ return nil unless remote =~ /^http(s)?/i
148
+
149
+ repo_settings = self.get_repo_settings(remote)
150
+
151
+ if repo_settings && repo_settings.has_key?(:proxy)
152
+ proxy = repo_settings[:proxy] unless repo_settings[:proxy].nil? || repo_settings[:proxy].empty?
153
+ else
154
+ proxy = self.settings[:proxy]
155
+ end
156
+
157
+ R10K::Git.log_proxy_for_remote(proxy, remote) if proxy
158
+
159
+ proxy
160
+ end
161
+
162
+ def self.log_proxy_for_remote(proxy, remote)
163
+ # Sanitize passwords out of the proxy URI for loggging.
164
+ proxy_uri = URI.parse(proxy)
165
+ proxy_str = "#{proxy_uri.scheme}://"
166
+ proxy_str << "#{proxy_uri.userinfo.gsub(/:(.*)$/, ':<FILTERED>')}@" if proxy_uri.userinfo
167
+ proxy_str << "#{proxy_uri.host}:#{proxy_uri.port}"
168
+
169
+ logger.debug { "Using HTTP proxy '#{proxy_str}' for '#{remote}'." }
170
+
171
+ nil
172
+ end
173
+
174
+ # Execute block with given proxy configured in ENV
175
+ def self.with_proxy(new_proxy)
176
+ unless new_proxy.nil?
177
+ old_proxy = Hash[
178
+ ['HTTPS_PROXY', 'HTTP_PROXY', 'https_proxy', 'http_proxy'].collect do |var|
179
+ old_value = ENV[var]
180
+ ENV[var] = new_proxy
181
+
182
+ [var, old_value]
183
+ end
184
+ ]
185
+ end
186
+
187
+ begin
188
+ yield
189
+ ensure
190
+ ENV.update(old_proxy) if old_proxy
191
+ end
192
+
193
+ nil
194
+ end
142
195
  end
143
196
  end
@@ -32,14 +32,15 @@ class R10K::Git::Rugged::BareRepository < R10K::Git::Rugged::BaseRepository
32
32
  # @return [void]
33
33
  def clone(remote)
34
34
  logger.debug1 { "Cloning '#{remote}' into #{@path}" }
35
- @_rugged_repo = ::Rugged::Repository.init_at(@path.to_s, true)
36
- with_repo do |repo|
35
+
36
+ @_rugged_repo = ::Rugged::Repository.init_at(@path.to_s, true).tap do |repo|
37
37
  config = repo.config
38
38
  config['remote.origin.url'] = remote
39
39
  config['remote.origin.fetch'] = '+refs/*:refs/*'
40
40
  config['remote.origin.mirror'] = 'true'
41
- fetch
42
41
  end
42
+
43
+ fetch('origin')
43
44
  rescue Rugged::SshError, Rugged::NetworkError => e
44
45
  raise R10K::Git::GitError.new(e.message, :git_dir => git_dir, :backtrace => e.backtrace)
45
46
  end
@@ -47,13 +48,21 @@ class R10K::Git::Rugged::BareRepository < R10K::Git::Rugged::BaseRepository
47
48
  # Fetch refs and objects from the origin remote
48
49
  #
49
50
  # @return [void]
50
- def fetch
51
+ def fetch(remote_name='origin')
51
52
  backup_branches = wipe_branches
52
- logger.debug1 { "Fetching remote 'origin' at #{@path}" }
53
+ logger.debug1 { "Fetching remote '#{remote_name}' at #{@path}" }
53
54
  options = {:credentials => credentials}
54
55
  refspecs = ['+refs/*:refs/*']
55
- results = with_repo { |repo| repo.fetch('origin', refspecs, options) }
56
- report_transfer(results, 'origin')
56
+
57
+ remote = remotes[remote_name]
58
+ proxy = R10K::Git.get_proxy_for_remote(remote)
59
+ results = nil
60
+
61
+ R10K::Git.with_proxy(proxy) do
62
+ results = with_repo { |repo| repo.fetch(remote_name, refspecs, options) }
63
+ end
64
+
65
+ report_transfer(results, remote_name)
57
66
  rescue Rugged::SshError, Rugged::NetworkError => e
58
67
  restore_branches(backup_branches)
59
68
  raise R10K::Git::GitError.new(e.message, :git_dir => git_dir, :backtrace => e.backtrace)
@@ -45,10 +45,24 @@ class R10K::Git::Rugged::BaseRepository
45
45
  end
46
46
  end
47
47
 
48
+ def remotes
49
+ remotes_hash = {}
50
+
51
+ if @_rugged_repo
52
+ @_rugged_repo.remotes.each do |remote|
53
+ remotes_hash[remote.name] = remote.url
54
+ end
55
+ end
56
+
57
+ remotes_hash
58
+ end
59
+
48
60
  private
49
61
 
50
- def with_repo
51
- yield @_rugged_repo if @_rugged_repo
62
+ def with_repo(opts={})
63
+ if @_rugged_repo
64
+ yield @_rugged_repo
65
+ end
52
66
  ensure
53
67
  @_rugged_repo.close if @_rugged_repo
54
68
  end
@@ -12,9 +12,18 @@ class R10K::Git::Rugged::Credentials
12
12
  # @param repository [R10K::Git::Rugged::BaseRepository]
13
13
  def initialize(repository)
14
14
  @repository = repository
15
+ @called = 0
15
16
  end
16
17
 
17
18
  def call(url, username_from_url, allowed_types)
19
+ @called += 1
20
+
21
+ # Break out of infinite HTTP auth retry loop introduced in libgit2/rugged 0.24.0, libssh
22
+ # auth seems to already abort after ~50 attempts.
23
+ if @called > 50
24
+ raise R10K::Git::GitError.new("Authentication failed for Git remote #{url.inspect}.")
25
+ end
26
+
18
27
  if allowed_types.include?(:ssh_key)
19
28
  get_ssh_key_credentials(url, username_from_url)
20
29
  elsif allowed_types.include?(:plaintext)