puppet-armature 0.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 1cf1d192c7f3bd7e27b82ded9f4a8fa5e7277647
4
+ data.tar.gz: d5882f18aa5102f3adb6068bf7c0cc97194c0ecf
5
+ SHA512:
6
+ metadata.gz: 28a7b7c5e6937603dca25c7a63bf9bf33fb03258414397c5a722eff52df61ab5211c830296e05c6ef64a91688d26b1c6a04c6761fa923b328f0d84186cca7051
7
+ data.tar.gz: cfa8e3b8f55bbe670f870bb3b773277ffbf4009e0ca6728b9e4f727d8963c258ffbb737c8e43fa898a1bcef04536461cf713af1a944f735d2b0556d85f8eb0e0
data/Gemfile ADDED
@@ -0,0 +1,2 @@
1
+ source 'https://rubygems.org'
2
+ gemspec
data/Gemfile.lock ADDED
@@ -0,0 +1,25 @@
1
+ PATH
2
+ remote: .
3
+ specs:
4
+ armature (0.1.0)
5
+ gli (= 2.14.0)
6
+ logging (~> 2)
7
+
8
+ GEM
9
+ remote: https://rubygems.org/
10
+ specs:
11
+ gli (2.14.0)
12
+ little-plugger (1.1.3)
13
+ logging (2.0.0)
14
+ little-plugger (~> 1.1)
15
+ multi_json (~> 1.10)
16
+ multi_json (1.12.1)
17
+
18
+ PLATFORMS
19
+ ruby
20
+
21
+ DEPENDENCIES
22
+ armature!
23
+
24
+ BUNDLED WITH
25
+ 1.10.4
data/LICENSE ADDED
@@ -0,0 +1,24 @@
1
+ Simplified BSD License
2
+
3
+ Copyright (c) 2016, Daniel Parks
4
+ All rights reserved.
5
+
6
+ Redistribution and use in source and binary forms, with or without
7
+ modification, are permitted provided that the following conditions are met:
8
+
9
+ 1. Redistributions of source code must retain the above copyright notice, this
10
+ list of conditions and the following disclaimer.
11
+ 2. Redistributions in binary form must reproduce the above copyright notice,
12
+ this list of conditions and the following disclaimer in the documentation
13
+ and/or other materials provided with the distribution.
14
+
15
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
16
+ ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
17
+ WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
18
+ DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
19
+ ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
20
+ (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
21
+ LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
22
+ ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
23
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
24
+ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
data/README.md ADDED
@@ -0,0 +1,50 @@
1
+ # Armature
2
+
3
+ A tool for deploying Puppet environments and modules.
4
+
5
+ ~~~
6
+ $ armature deploy-branch my-puppet-code.git '*'
7
+ ~~~
8
+
9
+ Armature sets up Puppet environments for each branch in your control repo and
10
+ installs modules as specified by the Puppetfile for each environment.
11
+
12
+ Armature is designed to replace [r10k](https://github.com/puppetlabs/r10k) for
13
+ certain, very specific use cases. It does not have nearly as many features, but
14
+ it is _much_ faster.
15
+
16
+ _This is an alpha release. The interface is likely to change significantly._
17
+
18
+ ## Glossary
19
+
20
+ * **Control repository (or repo):** The main git repository containing your
21
+ Puppet code.
22
+ * **Puppetfile:** A file listing modules needed by the Puppet code in your
23
+ control repo. See the [syntax documentation](docs/puppetfile-syntax.md).
24
+ * **Environment:** A directory containing Puppet code and resources. The master
25
+ can serve different environments to different nodes.
26
+
27
+ For example, you might have a dev environment that contains Puppet code in
28
+ development, and a prod environment that runs on the production nodes. See
29
+ the [official Puppet documentation.
30
+ ](https://docs.puppet.com/puppet/latest/reference/environments.html)
31
+
32
+ ## Usage
33
+
34
+ There are three commands you need to use. Run `armature help` to learn about
35
+ options, or `armature help <command>` to learn about a specific command.
36
+
37
+ ### `armature deploy-branch <git-url> <branch>`
38
+
39
+ Deploys branches from a git repository as environments.
40
+
41
+ ### `armature update`
42
+
43
+ Updates all branches in the cache. This will update all environments to their
44
+ latest commit in git, as well as all modules that were specified with a branch
45
+ ref (this will not update tags).
46
+
47
+ ### `armature gc`
48
+
49
+ Removes unused objects from the cache. For example, old commits in the control
50
+ repo that are no longer used as environments.
data/TODO.md ADDED
@@ -0,0 +1,44 @@
1
+ # Future development
2
+
3
+ ### Update module branches when updating an environment
4
+
5
+ Rather than waiting for a periodic `update-branches` job to run, update module
6
+ branches when their environment is deployed.
7
+
8
+ This will require some caching so that work is not duplicated when updating all
9
+ environments.
10
+
11
+ ### Webhook endpoints
12
+
13
+ I'm unsure if I want this. Webhooks are definitely useful, but I can get that
14
+ functionality from other services, like Jenkins.
15
+
16
+ Perhaps a separate tool that acts as a shim would be good.
17
+
18
+ ### Support updating module branches separately
19
+
20
+ This is useful if we can get a webhook for a module branch, or if we happen
21
+ to know that it is updated more frequently than other things and thus should
22
+ be checked more frequently.
23
+
24
+ ### Manage multiple masters
25
+
26
+ 1. Separate updates into three steps:
27
+
28
+ 1. _Prepare_: determine what repos and refs to update
29
+ 2. _Stage_: update repos an check out the needed shas
30
+ 3. _Activate_: make the ref changes
31
+
32
+ _Prepare_ will be run on the armature master (the Puppet MoM, presumably)
33
+ and will generate a data object to pass to other nodes for the _stage_ and
34
+ _activate_ steps.
35
+
36
+ We have to be careful that a poorly timed garbage collect doesn't wipe out
37
+ our work. That means keeping a process running to hold a lock, or adding some
38
+ sort of expiring lock on staged refs.
39
+
40
+ 2. Add interface for passing data between nodes
41
+
42
+ * HTTP can be proxied through NGINX for encryption and access control.
43
+
44
+ * SSH provides encryption and access control natively.
data/bin/armature ADDED
@@ -0,0 +1,142 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require 'gli'
4
+ require 'armature'
5
+ require 'logging'
6
+
7
+ include GLI::App
8
+
9
+ program_desc 'Deploy Puppet environments and manage modules'
10
+ version Armature::VERSION
11
+ subcommand_option_handling :normal
12
+ arguments :strict
13
+
14
+ @logger = Logging.logger['armature']
15
+ Logging.logger.root.level = :warn
16
+
17
+ desc 'Show INFO level output'
18
+ switch [:v,:verbose]
19
+
20
+ desc 'Show DEBUG level output'
21
+ switch [:d,:debug]
22
+
23
+ if Process.uid == 0
24
+ environments_default = '/etc/puppetlabs/code/environments'
25
+ cache_default = '/srv/armature-cache'
26
+ else
27
+ environments_default = "#{ENV['HOME']}/.puppetlabs/etc/code/environments"
28
+ cache_default = "#{ENV['HOME']}/.armature/cache"
29
+ end
30
+
31
+ desc 'Path to the environments directory'
32
+ default_value environments_default
33
+ arg_name 'DIR'
34
+ flag [:e,:environments]
35
+
36
+ desc 'Path to armature cache'
37
+ default_value cache_default
38
+ arg_name 'DIR'
39
+ flag [:c,:cache]
40
+
41
+ desc 'Deploy a ref as an environment'
42
+ arg 'GIT_URL'
43
+ arg 'REF'
44
+ arg 'NAME'
45
+ command "deploy-ref" do |c|
46
+ c.action do |global_options, options, arguments|
47
+ # This should fail as early as possible (e.g. if the path isn't correct)
48
+ environments = Armature::Environments.new(@environments_path, @cache)
49
+
50
+ repo = @cache.get_repo(arguments.shift)
51
+ ref = arguments.shift
52
+ name = arguments.shift
53
+
54
+ begin
55
+ environments.checkout_ref(repo, ref, name)
56
+ rescue
57
+ @logger.error("Error deploying ref '#{ref}' as '#{name}'")
58
+ raise
59
+ end
60
+ end
61
+ end
62
+
63
+ desc 'Deploy branches as environments'
64
+ long_desc 'This accepts globs to match branch names. For example, specify "*"
65
+ to deploy all branches.'
66
+ arg 'GIT_URL'
67
+ arg 'BRANCH', :multiple=>true
68
+ command "deploy-branch" do |c|
69
+ c.desc "Delete environments that aren't being deployed"
70
+ c.switch [:d,"delete-old"]
71
+ c.action do |global_options, options, arguments|
72
+ # This should fail as early as possible (e.g. if the path isn't correct)
73
+ environments = Armature::Environments.new(@environments_path, @cache)
74
+
75
+ repo = @cache.get_repo(arguments.shift)
76
+ all_branches = repo.get_branches()
77
+ branches = Set.new()
78
+
79
+ arguments.each do |glob|
80
+ found = all_branches.select do |branch|
81
+ File.fnmatch(glob, branch, File::FNM_PATHNAME)
82
+ end
83
+
84
+ if found.empty?
85
+ @logger.warn("No branches match '#{glob}'")
86
+ else
87
+ branches.merge(found)
88
+ end
89
+ end
90
+
91
+ if options["delete-old"]
92
+ (Set.new(environments.names()) - branches).each do |name|
93
+ environments.remove(name)
94
+ end
95
+ end
96
+
97
+ branches.each do |branch|
98
+ begin
99
+ environments.checkout_ref(repo, branch)
100
+ rescue => e
101
+ @logger.error("Error deploying branch '#{branch}' (skipping): #{e}")
102
+ end
103
+ end
104
+ end
105
+ end
106
+
107
+ desc 'Remove unused objects from cache'
108
+ command :gc do |c|
109
+ c.action { @cache.garbage_collect(@environments_path) }
110
+ end
111
+
112
+ desc 'Update all branches in the cache'
113
+ command :update do |c|
114
+ c.action { @cache.update_branches() }
115
+ end
116
+
117
+ pre do |global_options, command, options, arguments|
118
+ Logging.logger.root.add_appenders Logging.appenders.stdout
119
+
120
+ Logging.logger.root.level = :info if global_options[:verbose]
121
+ Logging.logger.root.level = :debug if global_options[:debug]
122
+
123
+ # Don't log all the git commands
124
+ Logging.logger["Armature::Run"].level = :info if global_options[:debug]
125
+
126
+ @logger.debug "Using environments directory '#{global_options[:environments]}'"
127
+ @logger.debug "Using cache directory '#{global_options[:cache]}'"
128
+
129
+ @environments_path = global_options[:environments]
130
+ @cache = Armature::Cache.new(global_options[:cache])
131
+
132
+ true
133
+ end
134
+
135
+ on_error do |exception|
136
+ if exception.is_a? SystemExit
137
+ raise
138
+ end
139
+ @logger.error(exception)
140
+ end
141
+
142
+ exit run(ARGV)
@@ -0,0 +1,33 @@
1
+ # Puppetfile syntax
2
+
3
+ The Puppetfile is just ruby. Armature provides one important function:
4
+
5
+ ### `mod 'name', :git=>'url', :ref=>'ref'`
6
+ Specifies a module to install.
7
+
8
+ * **name:** The name of the module. You use this in your Puppet code to
9
+ reference the module's classes and defined types.
10
+ * **url:** The URL of the git repo holding the module.
11
+ * **ref:** The ref in the repo to check out. May be a branch, a tag, or a sha.
12
+ Defaults to "master".
13
+
14
+ ## Example
15
+
16
+ ~~~ ruby
17
+ mod 'autosign',
18
+ :git => 'git://github.com/danieldreier/puppet-autosign.git',
19
+
20
+ mod 'aws',
21
+ :git => 'git://github.com/puppetlabs/puppetlabs-aws.git',
22
+ :ref => '1.4.0'
23
+ ~~~
24
+
25
+ ## Compatibility
26
+
27
+ Armature does not support the full syntax of either
28
+ [r10k](https://github.com/puppetlabs/r10k/blob/master/doc/puppetfile.mkd) or
29
+ [librarian-puppet](http://librarian-puppet.com). It will likely support more of
30
+ r10k's syntax at some point.
31
+
32
+ It does provide a `forge` function which accepts any arguments and is ignored.
33
+ This is only for compatility with the Puppetfile I'm using.
data/lib/armature.rb ADDED
@@ -0,0 +1,10 @@
1
+ require "armature/cache.rb"
2
+ require "armature/environments.rb"
3
+ require "armature/gitrepo.rb"
4
+ require "armature/puppetfile.rb"
5
+ require "armature/run.rb"
6
+ require "armature/util.rb"
7
+ require "armature/version.rb"
8
+
9
+ module Armature
10
+ end
@@ -0,0 +1,371 @@
1
+ require 'fileutils'
2
+ require 'pathname'
3
+ require 'set'
4
+
5
+ module Armature
6
+ class Cache
7
+ def initialize(path)
8
+ FileUtils.mkdir_p(path)
9
+ @path = File.realpath(path)
10
+ @repos = {}
11
+ @process_prefix = "#{Time.now.to_i}.#{Process.pid}"
12
+ @sequence = 0
13
+ @logger = Logging.logger[self]
14
+
15
+ %w{repo mutable immutable object tmp}.each do |subdir|
16
+ FileUtils.mkdir_p("#{@path}/#{subdir}")
17
+ end
18
+ end
19
+
20
+ # Get GitRepo object for a local clone of a remote repo at a URL
21
+ #
22
+ # This will clone the repo if it doesn't already exist.
23
+ def get_repo(url)
24
+ safe_url = fs_sanitize(url)
25
+
26
+ repo_path = "#{@path}/repo/#{safe_url}"
27
+ if ! Dir.exist? repo_path
28
+ @logger.info("Cloning '#{url}' for the first time")
29
+ Armature::Util::lock repo_path, File::LOCK_EX, "clone" do
30
+ if Dir.exist? repo_path
31
+ @logger.info("Another process cloned '#{url}' while we were blocked")
32
+ else
33
+ # Mirror copies *all* refs, not just branches. Ignore output.
34
+ Armature::Run.command(
35
+ "git", "clone", "--quiet", "--mirror", url, repo_path)
36
+ @logger.debug("Done cloning '#{url}'")
37
+ end
38
+ end
39
+ end
40
+
41
+ get_repo_by_name(safe_url)
42
+ end
43
+
44
+ # Get a GitRepo object for an existing local repo by its santized URL
45
+ def get_repo_by_name(safe_url)
46
+ @repos[safe_url] ||= GitRepo.new("#{@path}/repo/#{safe_url}", safe_url)
47
+ end
48
+
49
+ # Check out a ref from a repo and return the path
50
+ def checkout(repo, ref, options={})
51
+ options = Armature::Util.process_options(options,
52
+ { :name=>nil }, { :refresh=>false })
53
+
54
+ safe_ref = fs_sanitize(ref)
55
+
56
+ if options[:refresh]
57
+ # Don't check the cache; refresh it from source.
58
+ repo.freshen()
59
+ else
60
+ # Check cache first
61
+ ["immutable", "mutable"].each do |type|
62
+ ref_path = "#{@path}/#{type}/#{repo.name}/#{safe_ref}"
63
+ if Dir.exist? ref_path
64
+ return ref_path
65
+ end
66
+ end
67
+
68
+ # Special case, since branches may be refered to by name
69
+ safe_branch = fs_sanitize("refs/heads/#{ref}")
70
+ ref_path = "#{@path}/mutable/#{repo.name}/#{safe_branch}"
71
+ if Dir.exist? ref_path
72
+ return ref_path
73
+ end
74
+
75
+ # Special case, since tags may be refered to by name
76
+ safe_tag = fs_sanitize("refs/tags/#{ref}")
77
+ ref_path = "#{@path}/immutable/#{repo.name}/#{safe_tag}"
78
+ if Dir.exist? ref_path
79
+ return ref_path
80
+ end
81
+ end
82
+
83
+ # This will raise if the ref doesn't exist
84
+ begin
85
+ type, sha, real_ref = repo.ref_info(ref)
86
+ rescue
87
+ repo.freshen()
88
+ type, sha, real_ref = repo.ref_info(ref)
89
+ end
90
+
91
+ repo_dir = "#{@path}/#{type}/#{repo.name}"
92
+ if ! Dir.exist? repo_dir
93
+ Dir.mkdir(repo_dir)
94
+ end
95
+
96
+ ref_path = "#{repo_dir}/#{safe_ref}"
97
+ sha_path = checkout_sha(repo, sha, options[:name])
98
+ if sha_path != ref_path
99
+ atomic_symlink(sha_path, ref_path)
100
+ end
101
+
102
+ ref_path
103
+ end
104
+
105
+ # Creates a symlink atomically
106
+ #
107
+ # That is, if a symlink or file exists at new_path, then this will replace
108
+ # it with the newly created symlink atomically.
109
+ #
110
+ # Both target_path and new_path must be absolute.
111
+ def atomic_symlink(target_path, new_path)
112
+ new_path.chomp!("/")
113
+
114
+ @logger.debug("#{new_path} -> #{target_path}")
115
+
116
+ begin
117
+ if File.readlink(new_path) == target_path
118
+ return
119
+ end
120
+ rescue Errno::ENOENT
121
+ end
122
+
123
+ temp_path = new_temp_path()
124
+ File.symlink(target_path, temp_path)
125
+ File.rename(temp_path, new_path)
126
+ rescue
127
+ @logger.error("Error in atomic_symlink('#{target_path}', '#{new_path}')")
128
+ raise
129
+ end
130
+
131
+ def update_branches()
132
+ Dir.glob("#{@path}/mutable/*/*") do |path|
133
+ ref = File.basename(path)
134
+ ### FIXME decode
135
+ repo = get_repo_by_name(File.basename(File.dirname(path)))
136
+ @logger.info("Updating #{ref} ref from #{repo.url}")
137
+
138
+ begin
139
+ checkout(repo, ref, :refresh=>true)
140
+ rescue RefError
141
+ # The ref no longer exists, so we can't update it. Leave the old
142
+ # checkout in place for safety; garbage collection will remove it if
143
+ # it's no longer used.
144
+ @logger.info("#{ref} ref missing in remote; leaving untouched")
145
+ end
146
+ end
147
+ end
148
+
149
+ def garbage_collect(environments_path)
150
+ lock File::LOCK_EX, "garbage_collect #{environments_path}" do
151
+ # Remove all object locks
152
+ FileUtils.rm Dir.glob("#{@path}/*/.*.lock")
153
+ FileUtils.rm Dir.glob("#{@path}/*/*/.*.lock")
154
+
155
+ referenced_paths = find_all_references(environments_path)
156
+
157
+ garbage_collect_refs(referenced_paths)
158
+ garbage_collect_objects(referenced_paths)
159
+ garbage_collect_repos()
160
+
161
+ ### remove excess modules directories
162
+ end
163
+ ensure
164
+ empty_trash()
165
+ end
166
+
167
+ def lock(mode, message=nil)
168
+ if @lock_file
169
+ raise "Cannot re-lock cache"
170
+ end
171
+
172
+ Armature::Util::lock_file "#{@path}/lock", mode, message do |lock_file|
173
+ @lock_file = lock_file
174
+ yield
175
+ end
176
+ ensure
177
+ @lock_file = nil
178
+ end
179
+
180
+ private
181
+
182
+ def new_temp_path
183
+ @sequence += 1
184
+ "#{@path}/tmp/#{@process_prefix}.#{@sequence}"
185
+ end
186
+
187
+ def new_object_path(name=nil)
188
+ @sequence += 1
189
+ if name
190
+ name = "." + name.tr("/", " ")
191
+ end
192
+
193
+ "#{@path}/object/#{@process_prefix}.#{@sequence}#{name}"
194
+ end
195
+
196
+ def fs_sanitize(ref)
197
+ # Escape | and replace / with |. Also escape a leading .
198
+ ref.sub(/\A\./, "\\.").gsub(/[\\|]/, '\\\0').gsub(/\//, '|')
199
+ end
200
+
201
+ # Assumes sha exists. Use checkout() if it might not.
202
+ def checkout_sha(repo, sha, name=nil)
203
+ safe_sha = fs_sanitize(sha)
204
+
205
+ repo_path = "#{@path}/immutable/#{repo.name}"
206
+ sha_path = "#{repo_path}/#{safe_sha}"
207
+ if Dir.exist? sha_path
208
+ return sha_path
209
+ end
210
+
211
+ FileUtils.mkdir_p(repo_path)
212
+
213
+ Armature::Util::lock sha_path, File::LOCK_EX, "checkout" do
214
+ # Another process may have created the object before we got the lock
215
+ if Dir.exist? sha_path
216
+ return sha_path
217
+ end
218
+
219
+ object_path = new_object_path(name)
220
+ FileUtils.mkdir_p object_path
221
+
222
+ @logger.debug(
223
+ "Checking out '#{sha}' from '#{repo.url}' into '#{object_path}'")
224
+ repo.git "reset", "--hard", sha, :work_dir=>object_path
225
+ atomic_symlink(object_path, sha_path)
226
+ end
227
+
228
+ sha_path
229
+ end
230
+
231
+ # Put a directory into a trash directory for off-line deletion
232
+ #
233
+ # Directories take a while to delete, so move them into a temporary
234
+ # directory and then delete them outside the lock.
235
+ def trash(path)
236
+ if not @trash_path or not File.directory? @trash_path
237
+ @trash_path = new_temp_path()
238
+ Dir.mkdir(@trash_path)
239
+ @trash_sequence = 1
240
+ end
241
+
242
+ File.rename(path, "#{@trash_path}/#{@trash_sequence}")
243
+ @trash_sequence += 1
244
+ end
245
+
246
+ def empty_trash
247
+ if @trash_path and File.exist? @trash_path
248
+ @logger.info("Deleting trashed objects")
249
+ FileUtils.remove_entry(@trash_path)
250
+ @logger.debug("Finished deleting trashed objects")
251
+ end
252
+
253
+ @trash_path = nil
254
+ @trash_sequence = nil
255
+ end
256
+
257
+ # Part of find_all_references
258
+ def follow_reference(path, referenced, visited=[])
259
+ if not File.symlink? path
260
+ raise "Expected a symlink: #{path}"
261
+ end
262
+
263
+ target = Pathname.new(File.readlink(path)).cleanpath.to_s
264
+ if referenced.include? target
265
+ # Short cut. We've already seen this path.
266
+ @logger.debug("follow_reference: shortcutting #{path} -> #{target}")
267
+ return nil
268
+ end
269
+
270
+ @logger.debug("follow_reference: #{path} -> #{target}")
271
+
272
+ visited << target
273
+
274
+ if File.symlink? target
275
+ if visited.size() >= 6
276
+ raise "Symlink path more than 6 links deep: #{visited}"
277
+ end
278
+
279
+ target = follow_reference(target, referenced, visited)
280
+ elsif File.directory? target
281
+ # Assume this is an object
282
+ @logger.debug("follow_reference: object: #{target}")
283
+ elsif not File.exist? target
284
+ @logger.warn("follow_reference: does not exist: #{target}")
285
+ else
286
+ @logger.error("follow_reference: not an object or symlink: #{target}")
287
+ end
288
+
289
+ # Delay updating referenced until now so that we don't interfere with
290
+ # loop detection. (Adding links as we find them would cause loops to
291
+ # short circuit, resulting in a return of nil.)
292
+ referenced.merge(visited)
293
+ return target
294
+ end
295
+
296
+ def find_all_references(environments_path)
297
+ referenced_paths = Set.new
298
+ environments = Dir.glob("#{environments_path}/*")
299
+
300
+ @logger.debug(
301
+ "Looking for references in #{environments.count} environments")
302
+
303
+ environments.each do |env_path|
304
+ object_path = follow_reference(env_path, referenced_paths)
305
+ if object_path
306
+ # We could get a nil if two branches evaluate to the same sha.
307
+ referenced_paths << object_path
308
+
309
+ Dir.glob("#{object_path}/modules/*") do |module_path|
310
+ follow_reference(module_path, referenced_paths)
311
+ end
312
+ end
313
+ end
314
+
315
+ referenced_paths
316
+ end
317
+
318
+ def garbage_collect_refs(referenced_paths)
319
+ # Must be run from garbage_collect, since that handles the lock as well
320
+ # as emptying the trash
321
+ all_references = Set.new(Dir.glob("#{@path}/{im,}mutable/*/*"))
322
+ difference = all_references - referenced_paths
323
+ @logger.info(
324
+ "Deleting #{difference.size} of #{all_references.size} references")
325
+ difference.each do |path|
326
+ @logger.debug("Deleting #{path} (unused)")
327
+ File.delete(path)
328
+ end
329
+ end
330
+
331
+ def garbage_collect_objects(referenced_paths)
332
+ # Must be run from garbage_collect, since that handles the lock as well
333
+ # as emptying the trash
334
+ all_objects = Set.new(Dir.glob("#{@path}/object/*"))
335
+ difference = all_objects - referenced_paths
336
+ @logger.info(
337
+ "Trashing #{difference.size} of #{all_objects.size} objects")
338
+ difference.each do |path|
339
+ @logger.debug("Trashing #{path} (unused)")
340
+ trash(path)
341
+ end
342
+ end
343
+
344
+ def garbage_collect_repos
345
+ # Must be run from garbage_collect, since that handles the lock as well
346
+ # as emptying the trash
347
+ all_repos = Dir.glob("#{@path}/repo/*")
348
+ all_repos = Set.new(all_repos.map { |path| File.basename(path) })
349
+ used_repos = Set.new()
350
+
351
+ referenced_repos = Dir.glob("#{@path}/{im,}mutable/*")
352
+ referenced_repos.each do |path|
353
+ # No refs within the repo in use (for this ref type)
354
+ if Dir.glob("#{path}/*").empty?
355
+ @logger.debug("Trashing #{path} (empty)")
356
+ trash(path)
357
+ else
358
+ used_repos << File.basename(path)
359
+ end
360
+ end
361
+
362
+ difference = all_repos - used_repos
363
+ @logger.info(
364
+ "Trashing #{difference.size} of #{all_repos.size} repos")
365
+ difference.each do |repo|
366
+ @logger.debug("Trashing #{@path}/repo/#{repo} (unused)")
367
+ trash("#{@path}/repo/#{repo}")
368
+ end
369
+ end
370
+ end
371
+ end
@@ -0,0 +1,104 @@
1
+ module Armature
2
+ class Environments
3
+ attr_reader :path
4
+
5
+ # path is the path to the directory containing all the environments
6
+ def initialize(path, cache)
7
+ @cache = cache
8
+ @logger = Logging.logger[self]
9
+
10
+ if not File.directory? path
11
+ abort "Puppet environments path does not exist: '#{path}'"
12
+ end
13
+
14
+ @path = File.realpath(path)
15
+ end
16
+
17
+ def names()
18
+ Dir["#{@path}/*"].map { |path| File.basename(path) }
19
+ end
20
+
21
+ def remove(name)
22
+ File.delete("#{@path}/#{name}")
23
+ @logger.debug "Environment '#{name}' deleted"
24
+ rescue Errno::ENOENT
25
+ @logger.debug "Environment '#{name}' does not exist"
26
+ end
27
+
28
+ def checkout_ref(repo, ref, name=ref)
29
+ # This will add and update a modules dir in any repo, even if the repo is
30
+ # used in a Puppetfile. (Perhaps the cache is used for multiple repos?)
31
+
32
+ # https://docs.puppet.com/puppet/latest/reference/lang_reserved.html#environments
33
+ # The docs are slightly wrong; A-Z are allowed.
34
+ if name !~ /\A[a-z0-9_]+\Z/i
35
+ raise "Invalid environment name '#{name}'"
36
+ end
37
+
38
+ @cache.lock File::LOCK_SH do
39
+ @logger.info "Deploying ref '#{ref}' from '#{repo.url}' as" \
40
+ " environment '#{name}'"
41
+
42
+ begin
43
+ ref_path = @cache.checkout(repo, ref, :name=>ref, :refresh=>true)
44
+ rescue RefError
45
+ @logger.info "Ref '#{ref}' does not exist; ensuring environment" \
46
+ " '#{name}' is gone"
47
+ remove(name)
48
+ return
49
+ end
50
+
51
+ puppetfile_path = "#{ref_path}/Puppetfile"
52
+ if File.exist?(puppetfile_path)
53
+ @logger.debug "Found Puppetfile in environment '#{name}'"
54
+ puppetfile = Armature::Puppetfile.new()
55
+ puppetfile.include(puppetfile_path)
56
+ module_refs = puppetfile.results
57
+ @logger.debug "Loaded Puppetfile in environment '#{name}' with" \
58
+ " #{module_refs.length} modules"
59
+ else
60
+ @logger.debug "No Puppetfile in environment '#{name}'"
61
+ module_refs = {}
62
+ end
63
+
64
+ update_modules(ref_path, module_refs)
65
+
66
+ @cache.atomic_symlink(ref_path, "#{@path}/#{name}")
67
+ @logger.debug "Done deploying ref '#{ref}' from '#{repo.url}' as" \
68
+ " environment '#{name}'"
69
+ end
70
+ end
71
+
72
+ private
73
+
74
+ # Apply the results of the Puppetfile to a ref (e.g. an environment)
75
+ def update_modules(target_path, module_refs)
76
+ modules_path = "#{target_path}/modules"
77
+ if ! Dir.exist? modules_path
78
+ Dir.mkdir(modules_path)
79
+ end
80
+
81
+ module_refs.each do |name, info|
82
+ if name =~ /\A\./
83
+ raise "Module name may not start with period: '#{name}'"
84
+ elsif name =~ /\//
85
+ raise "Module name may not contain /: '#{name}'"
86
+ end
87
+
88
+ ref_path = @cache.checkout(
89
+ @cache.get_repo(info[:git]),
90
+ info[:ref],
91
+ :name=>"#{name}.#{info[:ref]}")
92
+
93
+ @cache.atomic_symlink(ref_path, "#{modules_path}/#{name}")
94
+ end
95
+
96
+ Dir.foreach(modules_path) do |name|
97
+ if ! module_refs.has_key? name and name != "." and name != ".."
98
+ # All paths should be symlinks.
99
+ File.delete("#{modules_path}/#{name}")
100
+ end
101
+ end
102
+ end
103
+ end
104
+ end
@@ -0,0 +1,99 @@
1
+ module Armature
2
+ class RefError < StandardError
3
+ end
4
+
5
+ class GitRepo
6
+ attr_reader :name
7
+
8
+ def initialize(git_dir, name)
9
+ @git_dir = git_dir
10
+ @name = name
11
+ @fetched = false
12
+ @logger = Logging.logger[self]
13
+ @ref_cache = {}
14
+ end
15
+
16
+ def url
17
+ @url ||= git("config", "--get", "remote.origin.url").chomp()
18
+ end
19
+
20
+ def freshen
21
+ if ! @fetched
22
+ @logger.info("Fetching from #{url}")
23
+ Armature::Util::lock @git_dir, File::LOCK_EX, "fetch" do
24
+ git "remote", "update", "--prune"
25
+ end
26
+ @fetched = true
27
+ true
28
+ else
29
+ false
30
+ end
31
+ end
32
+
33
+ def git(*arguments)
34
+ # This accepts a hash of options as the last argument
35
+ options = if arguments.last.is_a? Hash then arguments.pop else {} end
36
+ options = Armature::Util.process_options(options, { :work_dir => nil }, {})
37
+
38
+ if options[:work_dir]
39
+ work_dir_arguments = [ "--work-tree=" + options[:work_dir] ]
40
+ else
41
+ work_dir_arguments = []
42
+ end
43
+
44
+ command = [ "git", "--git-dir=" + @git_dir ] \
45
+ + work_dir_arguments \
46
+ + arguments
47
+
48
+ Armature::Run.command(*command)
49
+ end
50
+
51
+ def get_branches()
52
+ freshen()
53
+ data = git("for-each-ref",
54
+ "--format", "%(objectname) %(refname:strip=2)",
55
+ "refs/heads")
56
+ lines = data.split(/[\r\n]/).reject { |line| line == "" }
57
+
58
+ lines.map do |line|
59
+ sha, branch = line.split(' ', 2)
60
+ ref = "refs/heads/#{branch}"
61
+ @ref_cache[ref] = [:mutable, sha, ref]
62
+ branch
63
+ end
64
+ end
65
+
66
+ # Get the type of a ref (:branch, :tag, or :sha) and its sha as [type, sha]
67
+ def ref_info(ref)
68
+ if result = check_cache(ref)
69
+ result
70
+ elsif sha = rev_parse("refs/heads/#{ref}")
71
+ @ref_cache["refs/heads/#{ref}"] = [:mutable, sha, "refs/heads/#{ref}"]
72
+ elsif sha = rev_parse("refs/tags/#{ref}")
73
+ @ref_cache["refs/tags/#{ref}"] = [:immutable, sha, "refs/tags/#{ref}"]
74
+ elsif sha = rev_parse(ref)
75
+ if sha == ref
76
+ @ref_cache[ref] = [:immutable, sha, ref]
77
+ else
78
+ @ref_cache[ref] = [:mutable, sha, ref]
79
+ end
80
+ else
81
+ raise RefError.new("no such ref '#{ref}' in repo '#{url}'")
82
+ end
83
+ end
84
+
85
+ private
86
+ def check_cache(ref)
87
+ @ref_cache[ref] \
88
+ || @ref_cache["refs/heads/#{ref}"] \
89
+ || @ref_cache["refs/tags/#{ref}"]
90
+ end
91
+
92
+ # Get the sha for a ref, or nil if it doesn't exist
93
+ def rev_parse(ref)
94
+ git("rev-parse", "--verify", "#{ref}^{commit}").chomp
95
+ rescue Armature::Run::CommandFailureError
96
+ nil
97
+ end
98
+ end
99
+ end
@@ -0,0 +1,34 @@
1
+ module Armature
2
+ class Puppetfile
3
+ attr_reader :results
4
+
5
+ def initialize()
6
+ @results = {}
7
+ end
8
+
9
+ def include(path)
10
+ instance_eval(IO.read(path), path)
11
+ end
12
+
13
+ def mod(name, options={})
14
+ if name =~ /\A\./
15
+ raise "Module name may not start with period: '#{name}'"
16
+ elsif name =~ /\//
17
+ raise "Module name may not contain /: '#{name}'"
18
+ end
19
+
20
+ if @results[name]
21
+ raise "Module #{name} declared twice"
22
+ end
23
+
24
+ @results[name] = Armature::Util.process_options(options, {}, {
25
+ :git => nil,
26
+ :ref => "master",
27
+ })
28
+ end
29
+
30
+ def forge(*arguments)
31
+ ### error?
32
+ end
33
+ end
34
+ end
@@ -0,0 +1,42 @@
1
+ require 'open3'
2
+ require 'shellwords'
3
+
4
+ module Armature::Run
5
+ extend self
6
+
7
+ class CommandFailureError < StandardError
8
+ attr_reader :status
9
+ attr_reader :command
10
+ attr_reader :output
11
+
12
+ def initialize(status, command, output)
13
+ @status = status
14
+ @command = command
15
+ @output = output
16
+ super("Command '#{@command.first}' failed with #{@status}")
17
+ end
18
+
19
+ def to_s
20
+ command_str = Armature::Run.command_to_string(*@command)
21
+ "Command failed: #{command_str}\nReturn: #{@status}\nOutput:\n#{@output}\n"
22
+ end
23
+ end
24
+
25
+ def command_to_string(*command)
26
+ command.shelljoin
27
+ end
28
+
29
+ def command(*command)
30
+ logger = Logging.logger[self]
31
+
32
+ logger.debug(command_to_string(*command))
33
+ out, status = Open3.capture2e(*command)
34
+ logger.debug(command_to_string(command.first) + ": #{status}")
35
+
36
+ if ! status.success?
37
+ raise CommandFailureError.new(status, command, out)
38
+ end
39
+
40
+ out
41
+ end
42
+ end
@@ -0,0 +1,74 @@
1
+ require 'json'
2
+
3
+ module Armature::Util
4
+ extend self
5
+
6
+ # Validate options passed to a function
7
+ #
8
+ # options - the options that were passed to the function
9
+ # optional - options that are not required as a hash of keys and defaults
10
+ # required - options that are required as a hash of keys and defaults
11
+ #
12
+ # Returns the options hash with defaults applied.
13
+ #
14
+ # ~~~ ruby
15
+ # options = process_options(options, { :output => nil }, { :work_dir => "." })
16
+ # ~~~
17
+ def process_options(options, optional, required={})
18
+ options.each_key do |key|
19
+ if ! optional.has_key? key and ! required.has_key? key
20
+ raise ArgumentError.new("invalid option '#{key}'")
21
+ end
22
+ end
23
+
24
+ options = required.merge(optional).merge(options)
25
+ required.each_key do |key|
26
+ if options[key].nil?
27
+ raise ArgumentError.new("required option '#{key}' not set")
28
+ end
29
+ end
30
+
31
+ options
32
+ end
33
+
34
+ # Lock a path with a .name.lock file
35
+ #
36
+ # For example, `lock("foo/bar")` creates a `foo/.bar.lock` lock file.
37
+ def lock(path, mode, message=nil)
38
+ lock_path = File.dirname(path) + "/." + File.basename(path) + ".lock"
39
+ lock_file lock_path, mode, message do |lock|
40
+ yield lock
41
+ end
42
+ end
43
+
44
+ def lock_file(path, mode, message=nil)
45
+ # Any user that can open the lock file can flock it, causing armature
46
+ # operations to block.
47
+ File.open(path, File::RDWR|File::CREAT, 0600) do |lock|
48
+ if not lock.flock(mode | File::LOCK_NB)
49
+ logger = Logging.logger[self]
50
+ logger.info("Waiting for lock on #{path}")
51
+
52
+ start_time = Time.now
53
+ lock.flock(mode)
54
+ seconds = Time.now - start_time
55
+
56
+ logger.info("Got lock on #{path} after #{seconds} seconds")
57
+ end
58
+
59
+ if mode == File::LOCK_EX
60
+ lock.rewind()
61
+ lock.write({
62
+ "pid" => Process.pid,
63
+ "message" => message,
64
+ }.to_json)
65
+ lock.flush()
66
+ lock.truncate(lock.pos)
67
+ elsif message
68
+ raise "message parameter may only be set in File::LOCK_EX mode"
69
+ end
70
+
71
+ yield lock
72
+ end
73
+ end
74
+ end
@@ -0,0 +1,3 @@
1
+ module Armature
2
+ VERSION = '0.2.1'
3
+ end
@@ -0,0 +1,20 @@
1
+ # Ensure we require the local version and not one we might have installed already
2
+ require File.join([File.dirname(__FILE__),'lib','armature','version.rb'])
3
+ spec = Gem::Specification.new do |s|
4
+ s.name = 'puppet-armature'
5
+ s.version = Armature::VERSION
6
+ s.author = 'Daniel Parks'
7
+ s.email = 'dp-os-armature@oxidized.org'
8
+ s.homepage = 'https://github.com/danielparks/armature'
9
+ s.license = 'BSD-2-Clause'
10
+ s.platform = Gem::Platform::RUBY
11
+ s.summary = 'Deploy Puppet environments and manage modules'
12
+ s.description = 'Armature sets up Puppet environments for each branch in your control repo and installs modules as specified by the Puppetfile for each environment. It is designed as a replacement for r10k.'
13
+ s.files = `git ls-files`.split("\n")
14
+ s.require_paths << 'lib'
15
+ s.has_rdoc = false
16
+ s.bindir = 'bin'
17
+ s.executables << 'armature'
18
+ s.add_runtime_dependency('gli','2.14.0')
19
+ s.add_runtime_dependency('logging','~> 2')
20
+ end
metadata ADDED
@@ -0,0 +1,92 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: puppet-armature
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.2.1
5
+ platform: ruby
6
+ authors:
7
+ - Daniel Parks
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2016-08-29 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: gli
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - '='
18
+ - !ruby/object:Gem::Version
19
+ version: 2.14.0
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - '='
25
+ - !ruby/object:Gem::Version
26
+ version: 2.14.0
27
+ - !ruby/object:Gem::Dependency
28
+ name: logging
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - "~>"
32
+ - !ruby/object:Gem::Version
33
+ version: '2'
34
+ type: :runtime
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - "~>"
39
+ - !ruby/object:Gem::Version
40
+ version: '2'
41
+ description: Armature sets up Puppet environments for each branch in your control
42
+ repo and installs modules as specified by the Puppetfile for each environment. It
43
+ is designed as a replacement for r10k.
44
+ email: dp-os-armature@oxidized.org
45
+ executables:
46
+ - armature
47
+ extensions: []
48
+ extra_rdoc_files: []
49
+ files:
50
+ - Gemfile
51
+ - Gemfile.lock
52
+ - LICENSE
53
+ - README.md
54
+ - TODO.md
55
+ - bin/armature
56
+ - docs/puppetfile-syntax.md
57
+ - lib/armature.rb
58
+ - lib/armature/cache.rb
59
+ - lib/armature/environments.rb
60
+ - lib/armature/gitrepo.rb
61
+ - lib/armature/puppetfile.rb
62
+ - lib/armature/run.rb
63
+ - lib/armature/util.rb
64
+ - lib/armature/version.rb
65
+ - puppet-armature.gemspec
66
+ homepage: https://github.com/danielparks/armature
67
+ licenses:
68
+ - BSD-2-Clause
69
+ metadata: {}
70
+ post_install_message:
71
+ rdoc_options: []
72
+ require_paths:
73
+ - lib
74
+ - lib
75
+ required_ruby_version: !ruby/object:Gem::Requirement
76
+ requirements:
77
+ - - ">="
78
+ - !ruby/object:Gem::Version
79
+ version: '0'
80
+ required_rubygems_version: !ruby/object:Gem::Requirement
81
+ requirements:
82
+ - - ">="
83
+ - !ruby/object:Gem::Version
84
+ version: '0'
85
+ requirements: []
86
+ rubyforge_project:
87
+ rubygems_version: 2.4.5
88
+ signing_key:
89
+ specification_version: 4
90
+ summary: Deploy Puppet environments and manage modules
91
+ test_files: []
92
+ has_rdoc: false