puppet-armature 0.2.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 1cf1d192c7f3bd7e27b82ded9f4a8fa5e7277647
+  data.tar.gz: d5882f18aa5102f3adb6068bf7c0cc97194c0ecf
+SHA512:
+  metadata.gz: 28a7b7c5e6937603dca25c7a63bf9bf33fb03258414397c5a722eff52df61ab5211c830296e05c6ef64a91688d26b1c6a04c6761fa923b328f0d84186cca7051
+  data.tar.gz: cfa8e3b8f55bbe670f870bb3b773277ffbf4009e0ca6728b9e4f727d8963c258ffbb737c8e43fa898a1bcef04536461cf713af1a944f735d2b0556d85f8eb0e0

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/Gemfile.lock

@@ -0,0 +1,25 @@
+PATH
+  remote: .
+  specs:
+    armature (0.1.0)
+      gli (= 2.14.0)
+      logging (~> 2)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    gli (2.14.0)
+    little-plugger (1.1.3)
+    logging (2.0.0)
+      little-plugger (~> 1.1)
+      multi_json (~> 1.10)
+    multi_json (1.12.1)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  armature!
+
+BUNDLED WITH
+   1.10.4

data/LICENSE

@@ -0,0 +1,24 @@
+Simplified BSD License
+
+Copyright (c) 2016, Daniel Parks
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

data/README.md

@@ -0,0 +1,50 @@
+# Armature
+
+A tool for deploying Puppet environments and modules.
+
+~~~
+$ armature deploy-branch my-puppet-code.git '*'
+~~~
+
+Armature sets up Puppet environments for each branch in your control repo and
+installs modules as specified by the Puppetfile for each environment.
+
+Armature is designed to replace [r10k](https://github.com/puppetlabs/r10k) for
+certain, very specific use cases. It does not have nearly as many features, but
+it is _much_ faster.
+
+_This is an alpha release. The interface is likely to change significantly._
+
+## Glossary
+
+* **Control repository (or repo):** The main git repository containing your
+  Puppet code.
+* **Puppetfile:** A file listing modules needed by the Puppet code in your
+  control repo. See the [syntax documentation](docs/puppetfile-syntax.md).
+* **Environment:** A directory containing Puppet code and resources. The master
+  can serve different environments to different nodes.
+
+  For example, you might have a dev environment that contains Puppet code in
+  development, and a prod environment that runs on the production nodes. See
+  the [official Puppet documentation.
+  ](https://docs.puppet.com/puppet/latest/reference/environments.html)
+
+## Usage
+
+There are three commands you need to use. Run `armature help` to learn about
+options, or `armature help <command>` to learn about a specific command.
+
+### `armature deploy-branch <git-url> <branch>`
+
+Deploys branches from a git repository as environments.
+
+### `armature update`
+
+Updates all branches in the cache. This will update all environments to their
+latest commit in git, as well as all modules that were specified with a branch
+ref (this will not update tags).
+
+### `armature gc`
+
+Removes unused objects from the cache. For example, old commits in the control
+repo that are no longer used as environments.

data/TODO.md

@@ -0,0 +1,44 @@
+# Future development
+
+### Update module branches when updating an environment
+
+Rather than waiting for a periodic `update-branches` job to run, update module
+branches when their environment is deployed.
+
+This will require some caching so that work is not duplicated when updating all
+environments.
+
+### Webhook endpoints
+
+I'm unsure if I want this. Webhooks are definitely useful, but I can get that
+functionality from other services, like Jenkins.
+
+Perhaps a separate tool that acts as a shim would be good.
+
+### Support updating module branches separately
+
+This is useful if we can get a webhook for a module branch, or if we happen
+to know that it is updated more frequently than other things and thus should
+be checked more frequently.
+
+### Manage multiple masters
+
+1. Separate updates into three steps:
+
+   1. _Prepare_: determine what repos and refs to update
+   2. _Stage_: update repos an check out the needed shas
+   3. _Activate_: make the ref changes
+
+   _Prepare_ will be run on the armature master (the Puppet MoM, presumably)
+   and will generate a data object to pass to other nodes for the _stage_ and
+   _activate_ steps.
+
+   We have to be careful that a poorly timed garbage collect doesn't wipe out
+   our work. That means keeping a process running to hold a lock, or adding some
+   sort of expiring lock on staged refs.
+
+2. Add interface for passing data between nodes
+
+   * HTTP can be proxied through NGINX for encryption and access control.
+
+   * SSH provides encryption and access control natively.

data/bin/armature

@@ -0,0 +1,142 @@
+#!/usr/bin/env ruby
+
+require 'gli'
+require 'armature'
+require 'logging'
+
+include GLI::App
+
+program_desc 'Deploy Puppet environments and manage modules'
+version Armature::VERSION
+subcommand_option_handling :normal
+arguments :strict
+
+@logger = Logging.logger['armature']
+Logging.logger.root.level = :warn
+
+desc 'Show INFO level output'
+switch [:v,:verbose]
+
+desc 'Show DEBUG level output'
+switch [:d,:debug]
+
+if Process.uid == 0
+  environments_default = '/etc/puppetlabs/code/environments'
+  cache_default = '/srv/armature-cache'
+else
+  environments_default = "#{ENV['HOME']}/.puppetlabs/etc/code/environments"
+  cache_default = "#{ENV['HOME']}/.armature/cache"
+end
+
+desc 'Path to the environments directory'
+default_value environments_default
+arg_name 'DIR'
+flag [:e,:environments]
+
+desc 'Path to armature cache'
+default_value cache_default
+arg_name 'DIR'
+flag [:c,:cache]
+
+desc 'Deploy a ref as an environment'
+arg 'GIT_URL'
+arg 'REF'
+arg 'NAME'
+command "deploy-ref" do |c|
+  c.action do |global_options, options, arguments|
+    # This should fail as early as possible (e.g. if the path isn't correct)
+    environments = Armature::Environments.new(@environments_path, @cache)
+
+    repo = @cache.get_repo(arguments.shift)
+    ref = arguments.shift
+    name = arguments.shift
+
+    begin
+      environments.checkout_ref(repo, ref, name)
+    rescue
+      @logger.error("Error deploying ref '#{ref}' as '#{name}'")
+      raise
+    end
+  end
+end
+
+desc 'Deploy branches as environments'
+long_desc 'This accepts globs to match branch names. For example, specify "*"
+  to deploy all branches.'
+arg 'GIT_URL'
+arg 'BRANCH', :multiple=>true
+command "deploy-branch" do |c|
+  c.desc "Delete environments that aren't being deployed"
+  c.switch [:d,"delete-old"]
+  c.action do |global_options, options, arguments|
+    # This should fail as early as possible (e.g. if the path isn't correct)
+    environments = Armature::Environments.new(@environments_path, @cache)
+
+    repo = @cache.get_repo(arguments.shift)
+    all_branches = repo.get_branches()
+    branches = Set.new()
+
+    arguments.each do |glob|
+      found = all_branches.select do |branch|
+        File.fnmatch(glob, branch, File::FNM_PATHNAME)
+      end
+
+      if found.empty?
+        @logger.warn("No branches match '#{glob}'")
+      else
+        branches.merge(found)
+      end
+    end
+
+    if options["delete-old"]
+      (Set.new(environments.names()) - branches).each do |name|
+        environments.remove(name)
+      end
+    end
+
+    branches.each do |branch|
+      begin
+        environments.checkout_ref(repo, branch)
+      rescue => e
+        @logger.error("Error deploying branch '#{branch}' (skipping): #{e}")
+      end
+    end
+  end
+end
+
+desc 'Remove unused objects from cache'
+command :gc do |c|
+  c.action { @cache.garbage_collect(@environments_path) }
+end
+
+desc 'Update all branches in the cache'
+command :update do |c|
+  c.action { @cache.update_branches() }
+end
+
+pre do |global_options, command, options, arguments|
+  Logging.logger.root.add_appenders Logging.appenders.stdout
+
+  Logging.logger.root.level = :info if global_options[:verbose]
+  Logging.logger.root.level = :debug if global_options[:debug]
+
+  # Don't log all the git commands
+  Logging.logger["Armature::Run"].level = :info if global_options[:debug]
+
+  @logger.debug "Using environments directory '#{global_options[:environments]}'"
+  @logger.debug "Using cache directory '#{global_options[:cache]}'"
+
+  @environments_path = global_options[:environments]
+  @cache = Armature::Cache.new(global_options[:cache])
+
+  true
+end
+
+on_error do |exception|
+  if exception.is_a? SystemExit
+    raise
+  end
+  @logger.error(exception)
+end
+
+exit run(ARGV)

data/docs/puppetfile-syntax.md

@@ -0,0 +1,33 @@
+# Puppetfile syntax
+
+The Puppetfile is just ruby. Armature provides one important function:
+
+### `mod 'name', :git=>'url', :ref=>'ref'`
+Specifies a module to install.
+
+* **name:** The name of the module. You use this in your Puppet code to
+  reference the module's classes and defined types.
+* **url:** The URL of the git repo holding the module.
+* **ref:** The ref in the repo to check out. May be a branch, a tag, or a sha.
+  Defaults to "master".
+
+## Example
+
+~~~ ruby
+mod 'autosign',
+  :git => 'git://github.com/danieldreier/puppet-autosign.git',
+
+mod 'aws',
+  :git => 'git://github.com/puppetlabs/puppetlabs-aws.git',
+  :ref => '1.4.0'
+~~~
+
+## Compatibility
+
+Armature does not support the full syntax of either
+[r10k](https://github.com/puppetlabs/r10k/blob/master/doc/puppetfile.mkd) or
+[librarian-puppet](http://librarian-puppet.com). It will likely support more of
+r10k's syntax at some point.
+
+It does provide a `forge` function which accepts any arguments and is ignored.
+This is only for compatility with the Puppetfile I'm using.

data/lib/armature.rb

@@ -0,0 +1,10 @@
+require "armature/cache.rb"
+require "armature/environments.rb"
+require "armature/gitrepo.rb"
+require "armature/puppetfile.rb"
+require "armature/run.rb"
+require "armature/util.rb"
+require "armature/version.rb"
+
+module Armature
+end

data/lib/armature/cache.rb

@@ -0,0 +1,371 @@
+require 'fileutils'
+require 'pathname'
+require 'set'
+
+module Armature
+  class Cache
+    def initialize(path)
+      FileUtils.mkdir_p(path)
+      @path = File.realpath(path)
+      @repos = {}
+      @process_prefix = "#{Time.now.to_i}.#{Process.pid}"
+      @sequence = 0
+      @logger = Logging.logger[self]
+
+      %w{repo mutable immutable object tmp}.each do |subdir|
+        FileUtils.mkdir_p("#{@path}/#{subdir}")
+      end
+    end
+
+    # Get GitRepo object for a local clone of a remote repo at a URL
+    #
+    # This will clone the repo if it doesn't already exist.
+    def get_repo(url)
+      safe_url = fs_sanitize(url)
+
+      repo_path = "#{@path}/repo/#{safe_url}"
+      if ! Dir.exist? repo_path
+        @logger.info("Cloning '#{url}' for the first time")
+        Armature::Util::lock repo_path, File::LOCK_EX, "clone" do
+          if Dir.exist? repo_path
+            @logger.info("Another process cloned '#{url}' while we were blocked")
+          else
+            # Mirror copies *all* refs, not just branches. Ignore output.
+            Armature::Run.command(
+              "git", "clone", "--quiet", "--mirror", url, repo_path)
+            @logger.debug("Done cloning '#{url}'")
+          end
+        end
+      end
+
+      get_repo_by_name(safe_url)
+    end
+
+    # Get a GitRepo object for an existing local repo by its santized URL
+    def get_repo_by_name(safe_url)
+      @repos[safe_url] ||= GitRepo.new("#{@path}/repo/#{safe_url}", safe_url)
+    end
+
+    # Check out a ref from a repo and return the path
+    def checkout(repo, ref, options={})
+      options = Armature::Util.process_options(options,
+        { :name=>nil }, { :refresh=>false })
+
+      safe_ref = fs_sanitize(ref)
+
+      if options[:refresh]
+        # Don't check the cache; refresh it from source.
+        repo.freshen()
+      else
+        # Check cache first
+        ["immutable", "mutable"].each do |type|
+          ref_path = "#{@path}/#{type}/#{repo.name}/#{safe_ref}"
+          if Dir.exist? ref_path
+            return ref_path
+          end
+        end
+
+        # Special case, since branches may be refered to by name
+        safe_branch = fs_sanitize("refs/heads/#{ref}")
+        ref_path = "#{@path}/mutable/#{repo.name}/#{safe_branch}"
+        if Dir.exist? ref_path
+          return ref_path
+        end
+
+        # Special case, since tags may be refered to by name
+        safe_tag = fs_sanitize("refs/tags/#{ref}")
+        ref_path = "#{@path}/immutable/#{repo.name}/#{safe_tag}"
+        if Dir.exist? ref_path
+          return ref_path
+        end
+      end
+
+      # This will raise if the ref doesn't exist
+      begin
+        type, sha, real_ref = repo.ref_info(ref)
+      rescue
+        repo.freshen()
+        type, sha, real_ref = repo.ref_info(ref)
+      end
+
+      repo_dir = "#{@path}/#{type}/#{repo.name}"
+      if ! Dir.exist? repo_dir
+        Dir.mkdir(repo_dir)
+      end
+
+      ref_path = "#{repo_dir}/#{safe_ref}"
+      sha_path = checkout_sha(repo, sha, options[:name])
+      if sha_path != ref_path
+        atomic_symlink(sha_path, ref_path)
+      end
+
+      ref_path
+    end
+
+    # Creates a symlink atomically
+    #
+    # That is, if a symlink or file exists at new_path, then this will replace
+    # it with the newly created symlink atomically.
+    #
+    # Both target_path and new_path must be absolute.
+    def atomic_symlink(target_path, new_path)
+      new_path.chomp!("/")
+
+      @logger.debug("#{new_path} -> #{target_path}")
+
+      begin
+        if File.readlink(new_path) == target_path
+          return
+        end
+      rescue Errno::ENOENT
+      end
+
+      temp_path = new_temp_path()
+      File.symlink(target_path, temp_path)
+      File.rename(temp_path, new_path)
+    rescue
+      @logger.error("Error in atomic_symlink('#{target_path}', '#{new_path}')")
+      raise
+    end
+
+    def update_branches()
+      Dir.glob("#{@path}/mutable/*/*") do |path|
+        ref = File.basename(path)
+        ### FIXME decode
+        repo = get_repo_by_name(File.basename(File.dirname(path)))
+        @logger.info("Updating #{ref} ref from #{repo.url}")
+
+        begin
+          checkout(repo, ref, :refresh=>true)
+        rescue RefError
+          # The ref no longer exists, so we can't update it. Leave the old
+          # checkout in place for safety; garbage collection will remove it if
+          # it's no longer used.
+          @logger.info("#{ref} ref missing in remote; leaving untouched")
+        end
+      end
+    end
+
+    def garbage_collect(environments_path)
+      lock File::LOCK_EX, "garbage_collect #{environments_path}" do
+        # Remove all object locks
+        FileUtils.rm Dir.glob("#{@path}/*/.*.lock")
+        FileUtils.rm Dir.glob("#{@path}/*/*/.*.lock")
+
+        referenced_paths = find_all_references(environments_path)
+
+        garbage_collect_refs(referenced_paths)
+        garbage_collect_objects(referenced_paths)
+        garbage_collect_repos()
+
+        ### remove excess modules directories
+      end
+    ensure
+      empty_trash()
+    end
+
+    def lock(mode, message=nil)
+      if @lock_file
+        raise "Cannot re-lock cache"
+      end
+
+      Armature::Util::lock_file "#{@path}/lock", mode, message do |lock_file|
+        @lock_file = lock_file
+        yield
+      end
+    ensure
+      @lock_file = nil
+    end
+
+  private
+
+    def new_temp_path
+      @sequence += 1
+      "#{@path}/tmp/#{@process_prefix}.#{@sequence}"
+    end
+
+    def new_object_path(name=nil)
+      @sequence += 1
+      if name
+        name = "." + name.tr("/", " ")
+      end
+
+      "#{@path}/object/#{@process_prefix}.#{@sequence}#{name}"
+    end
+
+    def fs_sanitize(ref)
+      # Escape | and replace / with |. Also escape a leading .
+      ref.sub(/\A\./, "\\.").gsub(/[\\|]/, '\\\0').gsub(/\//, '|')
+    end
+
+    # Assumes sha exists. Use checkout() if it might not.
+    def checkout_sha(repo, sha, name=nil)
+      safe_sha = fs_sanitize(sha)
+
+      repo_path = "#{@path}/immutable/#{repo.name}"
+      sha_path = "#{repo_path}/#{safe_sha}"
+      if Dir.exist? sha_path
+        return sha_path
+      end
+
+      FileUtils.mkdir_p(repo_path)
+
+      Armature::Util::lock sha_path, File::LOCK_EX, "checkout" do
+        # Another process may have created the object before we got the lock
+        if Dir.exist? sha_path
+          return sha_path
+        end
+
+        object_path = new_object_path(name)
+        FileUtils.mkdir_p object_path
+
+        @logger.debug(
+          "Checking out '#{sha}' from '#{repo.url}' into '#{object_path}'")
+        repo.git "reset", "--hard", sha, :work_dir=>object_path
+        atomic_symlink(object_path, sha_path)
+      end
+
+      sha_path
+    end
+
+    # Put a directory into a trash directory for off-line deletion
+    #
+    # Directories take a while to delete, so move them into a temporary
+    # directory and then delete them outside the lock.
+    def trash(path)
+      if not @trash_path or not File.directory? @trash_path
+        @trash_path = new_temp_path()
+        Dir.mkdir(@trash_path)
+        @trash_sequence = 1
+      end
+
+      File.rename(path, "#{@trash_path}/#{@trash_sequence}")
+      @trash_sequence += 1
+    end
+
+    def empty_trash
+      if @trash_path and File.exist? @trash_path
+        @logger.info("Deleting trashed objects")
+        FileUtils.remove_entry(@trash_path)
+        @logger.debug("Finished deleting trashed objects")
+      end
+
+      @trash_path = nil
+      @trash_sequence = nil
+    end
+
+    # Part of find_all_references
+    def follow_reference(path, referenced, visited=[])
+      if not File.symlink? path
+        raise "Expected a symlink: #{path}"
+      end
+
+      target = Pathname.new(File.readlink(path)).cleanpath.to_s
+      if referenced.include? target
+        # Short cut. We've already seen this path.
+        @logger.debug("follow_reference: shortcutting #{path} -> #{target}")
+        return nil
+      end
+
+      @logger.debug("follow_reference: #{path} -> #{target}")
+
+      visited << target
+
+      if File.symlink? target
+        if visited.size() >= 6
+          raise "Symlink path more than 6 links deep: #{visited}"
+        end
+
+        target = follow_reference(target, referenced, visited)
+      elsif File.directory? target
+        # Assume this is an object
+        @logger.debug("follow_reference: object: #{target}")
+      elsif not File.exist? target
+        @logger.warn("follow_reference: does not exist: #{target}")
+      else
+        @logger.error("follow_reference: not an object or symlink: #{target}")
+      end
+
+      # Delay updating referenced until now so that we don't interfere with
+      # loop detection. (Adding links as we find them would cause loops to
+      # short circuit, resulting in a return of nil.)
+      referenced.merge(visited)
+      return target
+    end
+
+    def find_all_references(environments_path)
+      referenced_paths = Set.new
+      environments = Dir.glob("#{environments_path}/*")
+
+      @logger.debug(
+        "Looking for references in #{environments.count} environments")
+
+      environments.each do |env_path|
+        object_path = follow_reference(env_path, referenced_paths)
+        if object_path
+          # We could get a nil if two branches evaluate to the same sha.
+          referenced_paths << object_path
+
+          Dir.glob("#{object_path}/modules/*") do |module_path|
+            follow_reference(module_path, referenced_paths)
+          end
+        end
+      end
+
+      referenced_paths
+    end
+
+    def garbage_collect_refs(referenced_paths)
+      # Must be run from garbage_collect, since that handles the lock as well
+      # as emptying the trash
+      all_references = Set.new(Dir.glob("#{@path}/{im,}mutable/*/*"))
+      difference = all_references - referenced_paths
+      @logger.info(
+        "Deleting #{difference.size} of #{all_references.size} references")
+      difference.each do |path|
+        @logger.debug("Deleting #{path} (unused)")
+        File.delete(path)
+      end
+    end
+
+    def garbage_collect_objects(referenced_paths)
+      # Must be run from garbage_collect, since that handles the lock as well
+      # as emptying the trash
+      all_objects = Set.new(Dir.glob("#{@path}/object/*"))
+      difference = all_objects - referenced_paths
+      @logger.info(
+        "Trashing #{difference.size} of #{all_objects.size} objects")
+      difference.each do |path|
+        @logger.debug("Trashing #{path} (unused)")
+        trash(path)
+      end
+    end
+
+    def garbage_collect_repos
+      # Must be run from garbage_collect, since that handles the lock as well
+      # as emptying the trash
+      all_repos = Dir.glob("#{@path}/repo/*")
+      all_repos = Set.new(all_repos.map { |path| File.basename(path) })
+      used_repos = Set.new()
+
+      referenced_repos = Dir.glob("#{@path}/{im,}mutable/*")
+      referenced_repos.each do |path|
+        # No refs within the repo in use (for this ref type)
+        if Dir.glob("#{path}/*").empty?
+          @logger.debug("Trashing #{path} (empty)")
+          trash(path)
+        else
+          used_repos << File.basename(path)
+        end
+      end
+
+      difference = all_repos - used_repos
+      @logger.info(
+        "Trashing #{difference.size} of #{all_repos.size} repos")
+      difference.each do |repo|
+        @logger.debug("Trashing #{@path}/repo/#{repo} (unused)")
+        trash("#{@path}/repo/#{repo}")
+      end
+    end
+  end
+end

data/lib/armature/environments.rb

@@ -0,0 +1,104 @@
+module Armature
+  class Environments
+    attr_reader :path
+
+    # path is the path to the directory containing all the environments
+    def initialize(path, cache)
+      @cache = cache
+      @logger = Logging.logger[self]
+
+      if not File.directory? path
+        abort "Puppet environments path does not exist: '#{path}'"
+      end
+
+      @path = File.realpath(path)
+    end
+
+    def names()
+      Dir["#{@path}/*"].map { |path| File.basename(path) }
+    end
+
+    def remove(name)
+      File.delete("#{@path}/#{name}")
+      @logger.debug "Environment '#{name}' deleted"
+    rescue Errno::ENOENT
+      @logger.debug "Environment '#{name}' does not exist"
+    end
+
+    def checkout_ref(repo, ref, name=ref)
+      # This will add and update a modules dir in any repo, even if the repo is
+      # used in a Puppetfile. (Perhaps the cache is used for multiple repos?)
+
+      # https://docs.puppet.com/puppet/latest/reference/lang_reserved.html#environments
+      # The docs are slightly wrong; A-Z are allowed.
+      if name !~ /\A[a-z0-9_]+\Z/i
+        raise "Invalid environment name '#{name}'"
+      end
+
+      @cache.lock File::LOCK_SH do
+        @logger.info "Deploying ref '#{ref}' from '#{repo.url}' as" \
+          " environment '#{name}'"
+
+        begin
+          ref_path = @cache.checkout(repo, ref, :name=>ref, :refresh=>true)
+        rescue RefError
+          @logger.info "Ref '#{ref}' does not exist; ensuring environment" \
+            " '#{name}' is gone"
+          remove(name)
+          return
+        end
+
+        puppetfile_path = "#{ref_path}/Puppetfile"
+        if File.exist?(puppetfile_path)
+          @logger.debug "Found Puppetfile in environment '#{name}'"
+          puppetfile = Armature::Puppetfile.new()
+          puppetfile.include(puppetfile_path)
+          module_refs = puppetfile.results
+          @logger.debug "Loaded Puppetfile in environment '#{name}' with" \
+            " #{module_refs.length} modules"
+        else
+          @logger.debug "No Puppetfile in environment '#{name}'"
+          module_refs = {}
+        end
+
+        update_modules(ref_path, module_refs)
+
+        @cache.atomic_symlink(ref_path, "#{@path}/#{name}")
+        @logger.debug "Done deploying ref '#{ref}' from '#{repo.url}' as" \
+          " environment '#{name}'"
+      end
+    end
+
+  private
+
+    # Apply the results of the Puppetfile to a ref (e.g. an environment)
+    def update_modules(target_path, module_refs)
+      modules_path = "#{target_path}/modules"
+      if ! Dir.exist? modules_path
+        Dir.mkdir(modules_path)
+      end
+
+      module_refs.each do |name, info|
+        if name =~ /\A\./
+          raise "Module name may not start with period: '#{name}'"
+        elsif name =~ /\//
+          raise "Module name may not contain /: '#{name}'"
+        end
+
+        ref_path = @cache.checkout(
+          @cache.get_repo(info[:git]),
+          info[:ref],
+          :name=>"#{name}.#{info[:ref]}")
+
+        @cache.atomic_symlink(ref_path, "#{modules_path}/#{name}")
+      end
+
+      Dir.foreach(modules_path) do |name|
+        if ! module_refs.has_key? name and name != "." and name != ".."
+          # All paths should be symlinks.
+          File.delete("#{modules_path}/#{name}")
+        end
+      end
+    end
+  end
+end

data/lib/armature/gitrepo.rb

@@ -0,0 +1,99 @@
+module Armature
+  class RefError < StandardError
+  end
+
+  class GitRepo
+    attr_reader :name
+
+    def initialize(git_dir, name)
+      @git_dir = git_dir
+      @name = name
+      @fetched = false
+      @logger = Logging.logger[self]
+      @ref_cache = {}
+    end
+
+    def url
+      @url ||= git("config", "--get", "remote.origin.url").chomp()
+    end
+
+    def freshen
+      if ! @fetched
+        @logger.info("Fetching from #{url}")
+        Armature::Util::lock @git_dir, File::LOCK_EX, "fetch" do
+          git "remote", "update", "--prune"
+        end
+        @fetched = true
+        true
+      else
+        false
+      end
+    end
+
+    def git(*arguments)
+      # This accepts a hash of options as the last argument
+      options = if arguments.last.is_a? Hash then arguments.pop else {} end
+      options = Armature::Util.process_options(options, { :work_dir => nil }, {})
+
+      if options[:work_dir]
+        work_dir_arguments = [ "--work-tree=" + options[:work_dir] ]
+      else
+        work_dir_arguments = []
+      end
+
+      command = [ "git", "--git-dir=" + @git_dir ] \
+        + work_dir_arguments \
+        + arguments
+
+      Armature::Run.command(*command)
+    end
+
+    def get_branches()
+      freshen()
+      data = git("for-each-ref",
+          "--format", "%(objectname) %(refname:strip=2)",
+          "refs/heads")
+      lines = data.split(/[\r\n]/).reject { |line| line == "" }
+
+      lines.map do |line|
+        sha, branch = line.split(' ', 2)
+        ref = "refs/heads/#{branch}"
+        @ref_cache[ref] = [:mutable, sha, ref]
+        branch
+      end
+    end
+
+    # Get the type of a ref (:branch, :tag, or :sha) and its sha as [type, sha]
+    def ref_info(ref)
+      if result = check_cache(ref)
+        result
+      elsif sha = rev_parse("refs/heads/#{ref}")
+        @ref_cache["refs/heads/#{ref}"] = [:mutable, sha, "refs/heads/#{ref}"]
+      elsif sha = rev_parse("refs/tags/#{ref}")
+        @ref_cache["refs/tags/#{ref}"] = [:immutable, sha, "refs/tags/#{ref}"]
+      elsif sha = rev_parse(ref)
+        if sha == ref
+          @ref_cache[ref] = [:immutable, sha, ref]
+        else
+          @ref_cache[ref] = [:mutable, sha, ref]
+        end
+      else
+        raise RefError.new("no such ref '#{ref}' in repo '#{url}'")
+      end
+    end
+
+  private
+    def check_cache(ref)
+      @ref_cache[ref] \
+      || @ref_cache["refs/heads/#{ref}"] \
+      || @ref_cache["refs/tags/#{ref}"]
+    end
+
+    # Get the sha for a ref, or nil if it doesn't exist
+    def rev_parse(ref)
+      git("rev-parse", "--verify", "#{ref}^{commit}").chomp
+    rescue Armature::Run::CommandFailureError
+      nil
+    end
+  end
+end

data/lib/armature/puppetfile.rb

@@ -0,0 +1,34 @@
+module Armature
+  class Puppetfile
+    attr_reader :results
+
+    def initialize()
+      @results = {}
+    end
+
+    def include(path)
+      instance_eval(IO.read(path), path)
+    end
+
+    def mod(name, options={})
+      if name =~ /\A\./
+        raise "Module name may not start with period: '#{name}'"
+      elsif name =~ /\//
+        raise "Module name may not contain /: '#{name}'"
+      end
+
+      if @results[name]
+        raise "Module #{name} declared twice"
+      end
+
+      @results[name] = Armature::Util.process_options(options, {}, {
+        :git => nil,
+        :ref => "master",
+      })
+    end
+
+    def forge(*arguments)
+      ### error?
+    end
+  end
+end

data/lib/armature/run.rb

@@ -0,0 +1,42 @@
+require 'open3'
+require 'shellwords'
+
+module Armature::Run
+  extend self
+
+  class CommandFailureError < StandardError
+    attr_reader :status
+    attr_reader :command
+    attr_reader :output
+
+    def initialize(status, command, output)
+      @status = status
+      @command = command
+      @output = output
+      super("Command '#{@command.first}' failed with #{@status}")
+    end
+
+    def to_s
+      command_str = Armature::Run.command_to_string(*@command)
+      "Command failed: #{command_str}\nReturn: #{@status}\nOutput:\n#{@output}\n"
+    end
+  end
+
+  def command_to_string(*command)
+    command.shelljoin
+  end
+
+  def command(*command)
+    logger = Logging.logger[self]
+
+    logger.debug(command_to_string(*command))
+    out, status = Open3.capture2e(*command)
+    logger.debug(command_to_string(command.first) + ": #{status}")
+
+    if ! status.success?
+      raise CommandFailureError.new(status, command, out)
+    end
+
+    out
+  end
+end

data/lib/armature/util.rb

@@ -0,0 +1,74 @@
+require 'json'
+
+module Armature::Util
+  extend self
+
+  # Validate options passed to a function
+  #
+  # options - the options that were passed to the function
+  # optional - options that are not required as a hash of keys and defaults
+  # required - options that are required as a hash of keys and defaults
+  #
+  # Returns the options hash with defaults applied.
+  #
+  # ~~~ ruby
+  # options = process_options(options, { :output => nil }, { :work_dir => "." })
+  # ~~~
+  def process_options(options, optional, required={})
+    options.each_key do |key|
+      if ! optional.has_key? key and ! required.has_key? key
+        raise ArgumentError.new("invalid option '#{key}'")
+      end
+    end
+
+    options = required.merge(optional).merge(options)
+    required.each_key do |key|
+      if options[key].nil?
+        raise ArgumentError.new("required option '#{key}' not set")
+      end
+    end
+
+    options
+  end
+
+  # Lock a path with a .name.lock file
+  #
+  # For example, `lock("foo/bar")` creates a `foo/.bar.lock` lock file.
+  def lock(path, mode, message=nil)
+    lock_path = File.dirname(path) + "/." + File.basename(path) + ".lock"
+    lock_file lock_path, mode, message do |lock|
+      yield lock
+    end
+  end
+
+  def lock_file(path, mode, message=nil)
+    # Any user that can open the lock file can flock it, causing armature
+    # operations to block.
+    File.open(path, File::RDWR|File::CREAT, 0600) do |lock|
+      if not lock.flock(mode | File::LOCK_NB)
+        logger = Logging.logger[self]
+        logger.info("Waiting for lock on #{path}")
+
+        start_time = Time.now
+        lock.flock(mode)
+        seconds = Time.now - start_time
+
+        logger.info("Got lock on #{path} after #{seconds} seconds")
+      end
+
+      if mode == File::LOCK_EX
+        lock.rewind()
+        lock.write({
+          "pid" => Process.pid,
+          "message" => message,
+        }.to_json)
+        lock.flush()
+        lock.truncate(lock.pos)
+      elsif message
+        raise "message parameter may only be set in File::LOCK_EX mode"
+      end
+
+      yield lock
+    end
+  end
+end

data/lib/armature/version.rb

@@ -0,0 +1,3 @@
+module Armature
+  VERSION = '0.2.1'
+end

data/puppet-armature.gemspec

@@ -0,0 +1,20 @@
+# Ensure we require the local version and not one we might have installed already
+require File.join([File.dirname(__FILE__),'lib','armature','version.rb'])
+spec = Gem::Specification.new do |s|
+  s.name = 'puppet-armature'
+  s.version = Armature::VERSION
+  s.author = 'Daniel Parks'
+  s.email = 'dp-os-armature@oxidized.org'
+  s.homepage = 'https://github.com/danielparks/armature'
+  s.license = 'BSD-2-Clause'
+  s.platform = Gem::Platform::RUBY
+  s.summary = 'Deploy Puppet environments and manage modules'
+  s.description = 'Armature sets up Puppet environments for each branch in your control repo and installs modules as specified by the Puppetfile for each environment. It is designed as a replacement for r10k.'
+  s.files = `git ls-files`.split("\n")
+  s.require_paths << 'lib'
+  s.has_rdoc = false
+  s.bindir = 'bin'
+  s.executables << 'armature'
+  s.add_runtime_dependency('gli','2.14.0')
+  s.add_runtime_dependency('logging','~> 2')
+end

metadata

@@ -0,0 +1,92 @@
+--- !ruby/object:Gem::Specification
+name: puppet-armature
+version: !ruby/object:Gem::Version
+  version: 0.2.1
+platform: ruby
+authors:
+- Daniel Parks
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2016-08-29 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: gli
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 2.14.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 2.14.0
+- !ruby/object:Gem::Dependency
+  name: logging
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2'
+description: Armature sets up Puppet environments for each branch in your control
+  repo and installs modules as specified by the Puppetfile for each environment. It
+  is designed as a replacement for r10k.
+email: dp-os-armature@oxidized.org
+executables:
+- armature
+extensions: []
+extra_rdoc_files: []
+files:
+- Gemfile
+- Gemfile.lock
+- LICENSE
+- README.md
+- TODO.md
+- bin/armature
+- docs/puppetfile-syntax.md
+- lib/armature.rb
+- lib/armature/cache.rb
+- lib/armature/environments.rb
+- lib/armature/gitrepo.rb
+- lib/armature/puppetfile.rb
+- lib/armature/run.rb
+- lib/armature/util.rb
+- lib/armature/version.rb
+- puppet-armature.gemspec
+homepage: https://github.com/danielparks/armature
+licenses:
+- BSD-2-Clause
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.5
+signing_key: 
+specification_version: 4
+summary: Deploy Puppet environments and manage modules
+test_files: []
+has_rdoc: false