pupistry 0.0.1 → 0.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 2c47a743b2bf72dc35a18a04e4c907029773f9cd
-  data.tar.gz: 0ffd31d0773fac92db4989fcd52bafb4b25e2215
+  metadata.gz: 732d292847fab362692d6118e33d286e1c50f4bb
+  data.tar.gz: 5992f7c0698262d35b2a357b978821c23da7cbb8
 SHA512:
-  metadata.gz: 36fa8ed2a0f3b0fc34d548649582143193fa78e29420bda0f69d5a0c27488c93e4f96a4c102cdcc533cdf9f2414ba8bbb75c28da8363822c59ff8b386987a750
-  data.tar.gz: 715fa89d66398222226527be295da1532d17c9f32a35e2d1d878ac7564a5f3cb5a3632f819cdd6a052d1e3fefed4e6de6381549591904a8014b411219baa29ee
+  metadata.gz: 2591da8ee00435744fa2cb97357f88cba20e7bd3f1ab7b7148a158f4dcaea8d60f806dcb93a320995c7738b7a41684d225688cc60ab4cd00a5a1c72fdb709c3b
+  data.tar.gz: bda64095f11c6ce61ceb6fffa42238f9d029ab07d829805a91d207f8efb86cdda39496252dd2c869f02b92a8f48c68cd9a907294ede2fbfc57906d8d2ebbcf67

data/README.md

@@ -47,7 +47,26 @@ a number of issues with it.
 
 Build a new artifact:
 
-    pupistry build
+    $ pupistry build
+    I, [2015-04-08T22:19:30.419392 #52534]  INFO -- : Using r10k utility to fetch the latest Puppet code
+    [R10K::Action::Deploy::Environment - INFO] Deploying environment /Users/jethro/.pupistry/cache/puppetcode/master
+    [R10K::Action::Deploy::Environment - INFO] Deploying module /Users/jethro/.pupistry/cache/puppetcode/master/modules/stdlib
+    [R10K::Action::Deploy::Environment - INFO] Deploying module /Users/jethro/.pupistry/cache/puppetcode/master/modules/ruby
+    [R10K::Action::Deploy::Environment - INFO] Deploying module /Users/jethro/.pupistry/cache/puppetcode/master/modules/gcc
+    [R10K::Action::Deploy::Environment - INFO] Deploying module /Users/jethro/.pupistry/cache/puppetcode/master/modules/inifile
+    [R10K::Action::Deploy::Environment - INFO] Deploying module /Users/jethro/.pupistry/cache/puppetcode/master/modules/vcsrepo
+    [R10K::Action::Deploy::Environment - INFO] Deploying module /Users/jethro/.pupistry/cache/puppetcode/master/modules/git
+    [R10K::Action::Deploy::Environment - INFO] Deploying module /Users/jethro/.pupistry/cache/puppetcode/master/modules/ntp
+    [R10K::Action::Deploy::Environment - INFO] Deploying module /Users/jethro/.pupistry/cache/puppetcode/master/modules/firewall
+    [R10K::Action::Deploy::Environment - INFO] Deploying module /Users/jethro/.pupistry/cache/puppetcode/master/modules/soe
+    I, [2015-04-08T22:21:21.705315 #52534]  INFO -- : r10k run completed
+    I, [2015-04-08T22:21:21.706023 #52534]  INFO -- : Creating artifact...
+    I, [2015-04-08T22:21:21.999753 #52534]  INFO -- : Compressing artifact...
+    I, [2015-04-08T22:21:22.103131 #52534]  INFO -- : Building manifest information for artifact...
+    I, [2015-04-08T22:21:22.107012 #52534]  INFO -- : New artifact version 3f29c324aab076cd81667f9031a675e7 ready for pushing
+    --
+    Tip: Run pupistry diff to see what changed since the last artifact version
+
 
 Note that artifact builds are done from the upstream git repos, so if you
 have made changes, remember to git push first before generating. The tool will
@@ -56,13 +75,29 @@ remind you if it detects nothing has changed since the last run.
 Once your artifact is built, you can double check what has changed in the
 Puppet modules since the last run with:
 
-    pupistry diff
+    $ pupistry diff
+    diff -Nuar unpacked.3f29c324aab076cd81667f9031a675e7/puppetcode/master/README.md unpacked.4a522dd22c0453e1e3ec3d17dfed151b/puppetcode/master/README.md
+    --- unpacked.3f29c324aab076cd81667f9031a675e7/puppetcode/master/README.md	2015-04-08 22:19:42.000000000 +1200
+    +++ unpacked.4a522dd22c0453e1e3ec3d17dfed151b/puppetcode/master/README.md	2015-04-08 23:01:14.000000000 +1200
+    @@ -1 +1,4 @@
+     Personal Puppet Repo
+     +
+     +Example of a changed file in a module somewhere, nice and visible for all to see.
+     +
+     --
+     Tip: Run pupistry push to GPG sign & upload if happy to go live
 
 
 Finally when you're happy, push it to S3 to be delivered to all your servers. 
-If you have gpg signing enabled, it will ask you to sign here.
+If you have gpg signing enabled, it will ask you to sign here... or tell you
+off if you have it disabled. :-)
+
+    $ pupistry push
+    I, [2015-04-08T22:52:01.020865 #53037]  INFO -- : Uploading artifact version latest (3f29c324aab076cd81667f9031a675e7)
+    W, [2015-04-08T22:52:01.888356 #53037]  WARN -- : You have GPG signing *disabled*, whilst not critical it does weaken your security.
+    W, [2015-04-08T22:52:01.888418 #53037]  WARN -- : Skipping signing step...
+    I, [2015-04-08T22:52:03.043886 #53037]  INFO -- : Upload of artifact version 3f29c324aab076cd81667f9031a675e7 completed and is now latest
 
-    pupistry push
 
 
 ## Bootstrapping nodes
@@ -81,26 +116,39 @@ data field of most cloud providers like AWS or Digital Ocean.
 
 ## Running Puppet on target nodes
 
-Check what is going to be applied (Puppet in --noop mode)
+Pupistry replaces the need to call Puppet directly. Instead, call Pupistry with
+and it will handle getting the artifact and then executing Puppet for you. It
+respects some parameters like --environment and --noop for easy testing of new
+manifests and modules.
 
-    pupistry apply --noop
+At it's simpliest, to apply the current Puppet manifests:
 
+    $ pupistry apply
+    I, [2015-04-10T00:44:40.623101 #6726]  INFO -- : Pulling latest artifact....
+    I, [2015-04-10T00:44:42.700540 #6726]  INFO -- : Executing Puppet...
+    Notice: Compiled catalog for testhost1 in environment master in 2.21 seconds
+    Notice: Finished catalog run in 3.07 seconds
 
-Apply the current Puppet manifests:
 
-    pupistry apply
+Check what is going to be applied (Puppet in --noop mode)
+
+    pupistry apply --noop
 
 Specify an alternative environment:
 
     pupistry apply --environment staging
 
-
 Run pupistry as a system daemon. When you use the companion Puppet module, a
 system init file gets installed that sets this daemon up for you automatically.
 
     pupistry apply --daemon
 
 
+Alternatively, if you don't wish to use Pupistry to run the nodes, you don't
+have to. You can use Pupistry to build the artifacts and then pull them down
+and unpack via any means you find appropiate. It's just standard S3 + tar with
+some YAML and optional GPG signing.
+
 
 # Installation
 
@@ -118,6 +166,7 @@ Alternatively if you like living on the edge, download this repository and run:
     gem install pupistry-VERSION.gem
     pupistry setup
 
+TODO: Currently setup not implemented, copy the sample file provided.
 
 ## 2. S3 Bucket
 
@@ -138,7 +187,53 @@ needing to create explicit IAM credentials for the agents/servers.
 
 
 
-## 3. Helper Puppet Module
+## 3. Puppet Manifests & Configuration
+
+### Puppet Code Structure
+
+The following is the expected minmum structure of the Puppetcode repository to
+enable it to work with Pupistry:
+    /Puppetfile
+    /hiera.yaml
+    /manifests/site.pp
+
+Puppetfile is standard r10k and site.pp is standard Puppet. The Hiera config
+is generally normal, but you do need to define a datadir to tell Puppet to look
+where the puppetcode gets unpacked to. Generally the following sample Hiera
+will do the trick:
+    ---
+    :backends: yaml
+    :yaml:
+      :datadir: "%{::settings::confdir}/environments/%{::environment}/hieradata"
+    :hierarchy:
+     - "environments/%{::environment}"
+     - "nodes/%{::hostname}"
+     - common
+
+Then in Pupistry, the following configuration should be used for the agent (or
+subsitute /etc/puppet/ for wherever your platform has %{::settings::confdir}
+set to).
+      agent:
+        puppetcode: /etc/puppet/environments
+
+Pupistry will default to applying the "master" branch if one is not listed, if
+you are doing branch-based environments, you can specifiy when bootstrapping
+and override on a per-execution basis.
+
+You'll notice pretty quickly if something is broken when doing `puppet apply`
+
+
+
+TODO: Longer term intend to add support for various popular structure, but
+for now it is what it is. It's not hard, check out bin/puppistry and send
+pull requests.
+
+TODO: Provide an example repo to build from.... or maybe have smarter
+assistance in the app to check for files and find them or generate defaults
+if missing (eg hiera)
+
+
+### Helper Module
 
 Whilst you can use Pupistry to roll out any particular design of Puppet
 manifests, you will save yourself a lot of pain by also including the Pupistry
@@ -149,12 +244,13 @@ up the system service and configuring Puppet and Hiera correctly for masterless
 operation.
 
 You can fetch the module from:
-https://github.com/jethrocarr/pupistry-puppet
+https://github.com/jethrocarr/puppet-pupistry
 
 If you're doing r10k and Puppet masterless from scratch, this is probably
 something you want to make life easy.
 
 
+
 ## 4. Bootstrapping Nodes
 
 No need for manual configuration of your servers/nodes, you just need to build
@@ -162,6 +258,7 @@ your first artifact with Pupistry (`pupistry build && pupistry push`) and then
 generate a bootstrap script for your particular OS with `pupistry bootstrap`
 
 The bootstrap script will:
+
 1. Install Puppet and Pupistry for the particular OS.
 2. Download the latest artifact
 3. Trigger a Puppet run to build your server.
@@ -169,6 +266,11 @@ The bootstrap script will:
 Once done, it's up to your Puppet manifests to build your machine how you want
 it - enjoy!
 
+TODO: Currently being worked on, for now the following is a rough example of
+what you can do to bootstrap a RHEL/CentOS7 box:
+
+
+
 
 
 # Tutorials
@@ -183,8 +285,6 @@ and running masterless Puppet environment using Pupistry. It covers the very
 basics of setting up your r10k environment.
 
 
-
-
 # Caveats & Future Plans
 
 ## Use r10k
@@ -265,10 +365,11 @@ When developing Pupistry, you can run the git repo copy with:
     ruby -Ilib/ -r rubygems bin/pupistry
 
 By default Pupistry will try to load a settings.yaml file in the current
-working directory, before then trying ~/.pupistry/settings.yaml and then
-finally /etc/pupistry/settings.yaml. You can also override with --config.
+working directory, before then trying `~/.pupistry/settings.yaml` and then
+finally `/etc/pupistry/settings.yaml`. You can also override with `--config`.
 
-Add --verbose for additional debugging information.
+Add `--verbose` for additional debugging information. If you have a bug this
+is the first thing you should run to get more context for reports.
 
 
 # Contributions

data/bin/pupistry

@@ -19,6 +19,87 @@ class CLI < Thor
   class_option :verbose, :type => :boolean
   class_option :config, :type => :string
 
+
+  ## Agent Commands
+  
+  desc "apply", "Apply the latest Puppet artifact"
+  method_option :noop, :type => :boolean
+  method_option :daemon, :type => :boolean
+  method_option :environment, :type => :string
+  def apply
+
+    # Thor seems to force class options to be defined repeatedly? :-/
+    if options[:verbose]
+      $logger.level = Logger::DEBUG
+    else
+      $logger.level = Logger::INFO
+    end
+
+    if options[:config]
+      Pupistry::Config.load(options[:config])
+    else
+      Pupistry::Config.find_and_load
+    end
+
+    # Muppet Check
+    if options[:noop] and options[:daemon]
+      $logger.warn "A daemon running in noop will do nothing except log what changes it could apply"
+    end
+
+    # Muppet dev check
+    if options[:daemon]
+      $logger.fatal "Daemon mode not implemented yet"
+      exit 0
+    end
+
+    # Install the artifact
+    # TODO: Check if we've already installed it or not... :-)
+    $logger.info "Pulling latest artifact...."
+
+    artifact = Pupistry::Artifact.new
+    artifact.checksum = artifact.fetch_latest
+    artifact.fetch_artifact
+    artifact.unpack
+    unless artifact.install
+      $logger.fatal "An unexpected error happened when installing the latest artifact, cancelling Puppet run"
+      exit 0
+    end
+
+    # Remove temporary unpacked files
+    artifact.clean_unpack
+
+
+    # Execute Puppet. At this point
+    puppet_cmd = "puppet apply"
+
+    if options[:noop]
+      puppet_cmd += " --noop"
+    end
+
+    if options[:environment]
+      environment = options[:environment]
+    else
+      environment = 'master'
+    end
+
+    puppet_cmd += " --environment #{environment}"
+    puppet_cmd += " --modulepath #{$config["agent"]["puppetcode"]}/#{environment}/modules/"
+    puppet_cmd += " --hiera_config #{$config["agent"]["puppetcode"]}/#{environment}/hiera.yaml"
+    puppet_cmd += " #{$config["agent"]["puppetcode"]}/#{environment}/manifests/site.pp"
+
+    $logger.info "Executing Puppet..."
+    $logger.debug "With: #{puppet_cmd}"
+
+    unless system puppet_cmd
+      $logger.fatal "An unexpected issue occured when running puppet"
+    end
+
+  end
+
+
+
+  ## Workstation Commands
+
   desc "build", "Build a new archive file"
   def build
 
@@ -51,7 +132,6 @@ class CLI < Thor
       raise e
     end
 
-
   end
 
 

data/lib/pupistry/artifact.rb

@@ -0,0 +1,452 @@
+require 'rubygems'
+require 'yaml'
+require 'time'
+require 'digest'
+require 'fileutils'
+
+module Pupistry
+  # Pupistry::Artifact
+
+  class Artifact
+    # All the functions needed for manipulating the artifats
+    attr_accessor :checksum
+
+
+    def fetch_r10k
+      $logger.info "Using r10k utility to fetch the latest Puppet code"
+
+      unless defined? $config["build"]["puppetcode"]
+        $logger.fatal "You must configure the build:puppetcode config option in settings.yaml"
+        raise "Invalid Configuration"
+      end
+
+      # https://github.com/puppetlabs/r10k
+      #
+      # r10k does a fantastic job with all the git stuff and we want to use it
+      # to download the Puppet code from all the git modules (based on following
+      # the master one provided), then we can steal the Puppet code from the
+      # artifact generated.
+      #
+      # TODO: We should re-write this to hook directly into r10k's libraries,
+      # given that both Pupistry and r10k are Ruby, presumably it should be
+      # doable and much more polished approach. For now the MVP is to just run
+      # it via system, pull requests/patches to fix very welcome!
+
+
+      # Build the r10k config to instruct it to use our cache path for storing
+      # it's data and exporting the finished result.
+      $logger.debug "Generating an r10k configuration file..."
+      r10k_config = {
+        "cachedir" => "#{$config["general"]["app_cache"]}/r10kcache",
+        "sources"  => {
+          "puppet" => {
+            "remote"  => $config["build"]["puppetcode"],
+            "basedir" => $config["general"]["app_cache"] + "/puppetcode",
+          }
+        }
+      }
+
+      begin
+        File.open("#{$config["general"]["app_cache"]}/r10kconfig.yaml",'w') do |fh|
+          fh.write YAML::dump(r10k_config)
+        end
+      rescue Exception => e
+        $logger.fatal "Unexpected error when trying to write the r10k configuration file"
+        raise e
+      end
+
+      
+      # Execute R10k with the provided configuration
+      $logger.debug "Executing r10k"
+
+      if system "r10k deploy environment -c #{$config["general"]["app_cache"]}/r10kconfig.yaml -pv"
+        $logger.info "r10k run completed"
+      else
+        $logger.error "r10k run failed, unable to generate artifact"
+        raise "r10k run did not complete, unable to generate artifact"
+      end
+
+    end
+
+    def fetch_latest
+      # Fetch the latest S3 YAML file and check the version metadata without writing
+      # it to disk. Returns the version. Useful for quickly checking for updates :-)
+
+      $logger.debug "Checking latest artifact version..."
+
+      s3        = Pupistry::Storage_AWS.new 'agent'
+      contents  = s3.download 'manifest.latest.yaml'
+
+      if contents
+        manifest = YAML::load(contents)
+
+        if defined? manifest['version']
+          return manifest['version']
+        else
+          return false
+        end
+
+      else
+        # download did not work
+        return false
+      end
+
+
+    end
+
+
+    def fetch_current
+      # Fetch the latest on-disk YAML file and check the version metadata, used
+      # to determine the latest artifact that has not yet been pushed to S3.
+      # Returns the version.
+
+      # Read the symlink information to get the latest version
+      if File.exists?($config["general"]["app_cache"] + "/artifacts/manifest.latest.yaml")
+          manifest    = YAML::load(File.open($config["general"]["app_cache"] + "/artifacts/manifest.latest.yaml"))
+          @checksum   = manifest['version']
+        else
+          $logger.error "No artifact has been built yet. You need to run pupistry build first?"
+          return 0
+        end
+    end
+
+
+    def fetch_artifact
+
+      # Figure out which version to fetch (if not explicitly defined)
+      if defined? @checksum
+        $logger.debug "Downloading artifact version #{@checksum}"
+      else
+        @checksum = fetch_latest
+
+        if defined? @checksum
+          $logger.debug "Downloading latest artifact (#{@checksum})"
+        else
+          $logger.error "There is not current artifact that can be fetched"
+          return false
+        end
+
+      end
+
+      # Make sure the download dir/cache exists
+      unless Dir.exists?($config["general"]["app_cache"] + "/artifacts/")
+        FileUtils.mkdir_p $config["general"]["app_cache"] + "/artifacts/"
+      end
+
+      # Download files if they don't already exist
+      if File.exists?($config["general"]["app_cache"] + "/artifacts/manifest.#{@checksum}.yaml") and File.exists?($config["general"]["app_cache"] + "/artifacts/artifact.#{@checksum}.tar.gz")
+        $logger.debug "This artifact is already present, no download required."
+      else
+        s3 = Pupistry::Storage_AWS.new 'agent'
+        s3.download "manifest.#{@checksum}.yaml", $config["general"]["app_cache"] + "/artifacts/manifest.#{@checksum}.yaml"
+        s3.download "artifact.#{@checksum}.tar.gz", $config["general"]["app_cache"] + "/artifacts/artifact.#{@checksum}.tar.gz"
+      end
+
+    end
+
+
+
+    def push_artifact
+      # The push step involves 2 steps:
+      # 1. GPG sign the artifact and write it into the manifest file
+      # 2. Upload the manifest and archive files to S3.
+      # 3. Upload a copy as the "latest" manifest file which will be hit by clients.
+
+
+      # Determine which version we are uploading. Either one specifically
+      # selected, otherwise find the latest one to push
+
+      if defined? @checksum
+        $logger.info "Uploading artifact version #{@checksum}."
+      else
+        @checksum = fetch_current
+
+        if @checksum
+          $logger.info "Uploading artifact version latest (#{@checksum})"
+        else
+          # If there is no current version, we can't do much....
+          exit 0
+        end
+      end
+
+
+      # Do we even need to upload? If nothing has changed....
+      if @checksum == fetch_latest
+        $logger.error "You've already pushed this artifact version, nothing to do."
+        exit 0
+      end
+
+
+      # Make sure the files actually exist...
+      unless File.exists?($config["general"]["app_cache"] + "/artifacts/manifest.#{@checksum}.yaml")
+        $logger.error "The files expected for #{@checksum} do not appear to exist or are not readable"
+        raise "Fatal unexpected error"
+      end
+
+      unless File.exists?($config["general"]["app_cache"] + "/artifacts/artifact.#{@checksum}.tar.gz")
+        $logger.error "The files expected for #{@checksum} do not appear to exist or are not readable"
+        raise "Fatal unexpected error"
+      end
+
+
+      # GPG sign the files
+      if $config["general"]["gpg_disable"] == true
+        $logger.warn "You have GPG signing *disabled*, whilst not critical it does weaken your security."
+        $logger.warn "Skipping signing step..."
+      else
+        $logger.info "GPG signing the artifact with configured key"
+
+        # TODO: should probably write this bit!
+      end
+
+
+      # Upload the artifact & manifests to S3. We also make an additional copy 
+      # as the "latest" file which will be downloaded by all the agents checking
+      # for new updates.
+
+      s3 = Pupistry::Storage_AWS.new 'build'
+      s3.upload $config["general"]["app_cache"] + "/artifacts/artifact.#{@checksum}.tar.gz", "artifact.#{@checksum}.tar.gz"
+      s3.upload $config["general"]["app_cache"] + "/artifacts/manifest.#{@checksum}.yaml", "manifest.#{@checksum}.yaml"
+      s3.upload $config["general"]["app_cache"] + "/artifacts/manifest.#{@checksum}.yaml", "manifest.latest.yaml"
+
+
+      # Test a read of the manifest, we do this to make sure the S3 ACLs setup
+      # allow downloading of the uploaded files - helps avoid user headaches if
+      # they misconfigure and then blindly trust their bootstrap config.
+      #
+      # Only worth doing this step if they've explicitly set their AWS IAM credentials
+      # for the agent, which should be everyone except for IAM role users.
+
+      if $config["agent"]["aws_access_id"]
+        fetch_artifact
+      else
+        $logger.warn "The agent's AWS credentials are unset on this machine, unable to do download test to check permissions for you."
+        $logger.warn "Assuming you know what you're doing, please set if unsure."
+      end
+
+      $logger.info "Upload of artifact version #{@checksum} completed and is now latest"
+    end
+    
+
+    def build_artifact
+      # r10k has done all the heavy lifting for us, we just need to generate a
+      # tarball from the app_cache /puppetcode directory. There are some Ruby
+      # native libraries, but really we might as well just use the native tools
+      # since we don't want to do anything clever like in-memory assembly of
+      # the file. Like r10k, if you want to convert to a nicely polished native
+      # Ruby solution, patches welcome.
+
+      $logger.info "Creating artifact..."
+
+      Dir.chdir($config["general"]["app_cache"]) do
+
+        # Make sure there is a directory to write artifacts into
+        FileUtils.mkdir_p('artifacts')
+
+        # Build the tar file - we delibertly don't compress in a single step
+        # so that we can grab the checksum, since checksum will always differ
+        # post-compression.
+        unless system "tar -c --exclude '.git' -f artifacts/artifact.temp.tar puppetcode/*"
+          $logger.error "Unable to create tarball"
+          raise "An unexpected error occured when executing tar"
+        end
+
+        # The checksum is important, we use it as our version for each artifact
+        # so we can tell them apart in a unique way.
+        @checksum = Digest::MD5.file($config["general"]["app_cache"] + "/artifacts/artifact.temp.tar").hexdigest
+
+        # Now we have the checksum, check if it's the same as any existing
+        # artifacts. If so, drop out here, good to give feedback to the user
+        # if nothing has changed since it's easy to forget to git push a single
+        # module/change.
+
+        if File.exists?($config["general"]["app_cache"] + "/artifacts/manifest.#{@checksum}.yaml")
+          $logger.error "This artifact version (#{@checksum}) has already been built, nothing todo."
+          $logger.error "Did you remember to \"git push\" your module changes?"
+
+          # Cleanup temp file
+          FileUtils.rm($config["general"]["app_cache"] + "/artifacts/artifact.temp.tar")
+          exit 0
+        end
+
+        # Compress the artifact now that we have taken it's checksum
+        $logger.info "Compressing artifact..."
+
+        if system "gzip artifacts/artifact.temp.tar"
+        else
+          $logger.error "An unexpected error occured during compression of the artifact"
+          raise "An unexpected error occured during compression of the artifact"
+        end
+      end
+
+
+      # We have the checksum, so we can now rename the artifact file
+      FileUtils.mv($config["general"]["app_cache"] + "/artifacts/artifact.temp.tar.gz", $config["general"]["app_cache"] + "/artifacts/artifact.#{@checksum}.tar.gz")
+
+
+      $logger.info "Building manifest information for artifact..."
+
+      # Create the manifest file, this is used by clients for pulling details about
+      # the latest artifacts. We don't GPG sign here, but we do put in a placeholder.
+      manifest = {
+        "version"   => @checksum, 
+        "date"      => Time.new.inspect,
+        "builduser" => ENV['USER'] || 'unlabled',
+        "gpgsig"    => 'unsighed',
+      }
+
+      begin
+        File.open("#{$config["general"]["app_cache"]}/artifacts/manifest.#{@checksum}.yaml",'w') do |fh|
+          fh.write YAML::dump(manifest)
+        end
+      rescue Exception => e
+        $logger.fatal "Unexpected error when trying to write the manifest file"
+        raise e
+      end
+
+      # This is the latest artifact, create some symlinks pointing the latest to it
+      begin
+        FileUtils.ln_s("manifest.#{@checksum}.yaml", "#{$config["general"]["app_cache"]}/artifacts/manifest.latest.yaml", :force => true)
+        FileUtils.ln_s("artifact.#{@checksum}.tar.gz",  "#{$config["general"]["app_cache"]}/artifacts/artifact.latest.tar.gz",  :force => true)
+      rescue Exception => e
+        $logger.fatal "Something weird went really wrong trying to symlink the latest artifacts"
+        raise e
+      end
+
+
+      $logger.info "New artifact version #{@checksum} ready for pushing"
+    end
+
+
+    def unpack
+      # Unpack the currently selected artifact to the archives directory.
+    
+      # An application version must be specified
+      unless defined? @checksum
+        raise "Application bug, trying to unpack no artifact"
+      end
+
+      # Make sure the files actually exist...
+      unless File.exists?($config["general"]["app_cache"] + "/artifacts/manifest.#{@checksum}.yaml")
+        $logger.error "The files expected for #{@checksum} do not appear to exist or are not readable"
+        raise "Fatal unexpected error"
+      end
+
+      unless File.exists?($config["general"]["app_cache"] + "/artifacts/artifact.#{@checksum}.tar.gz")
+        $logger.error "The files expected for #{@checksum} do not appear to exist or are not readable"
+        raise "Fatal unexpected error"
+      end
+
+      # Clean up an existing unpacked copy - in *theory* it should be same, but
+      # a mistake like running out of disk could have left it in an unclean state
+      # so let's make sure it's gone
+      clean_unpack
+
+      # Unpack the archive file
+      FileUtils.mkdir_p($config["general"]["app_cache"] + "/artifacts/unpacked.#{@checksum}")
+      Dir.chdir($config["general"]["app_cache"] + "/artifacts/unpacked.#{@checksum}") do
+
+        unless system "tar -xf ../artifact.#{@checksum}.tar.gz"
+          $logger.error "Unable to unpack artifact files to #{Dir.pwd}"
+          raise "An unexpected error occured when executing tar"
+        else
+          $logger.debug "Successfully unpacked artifact #{@checksum}"
+        end
+      end
+
+    end
+
+
+    def install
+      # Copy the unpacked artifact into the agent's configured location. Generally all the
+      # heavy lifting is done by fetch_latest and unpack methods.
+
+      # An application version must be specified
+      unless defined? @checksum
+        raise "Application bug, trying to install no artifact"
+      end
+
+      # Make sure the artifact has been unpacked
+      unless Dir.exists?($config["general"]["app_cache"] + "/artifacts/unpacked.#{@checksum}")
+        $logger.error "The unpacked directory expected for #{@checksum} does not appear to exist or is not readable"
+        raise "Fatal unexpected error"
+      end
+
+      # Purge any currently installed files in the directory. See clean_install
+      # TODO notes for how this could be improved.
+      unless clean_install
+        $logger.error "Installation not proceeduing due to issues cleaning/prepping destination dir"
+      end
+
+      # Make sure the destination directory exists
+      unless Dir.exists?($config["agent"]["puppetcode"])
+        $logger.error "The destination path of #{$config["agent"]["puppetcode"]} does not appear to exist or is not readable"
+        raise "Fatal unexpected error"
+      end
+    
+      # Clone unpacked contents to the installation directory
+      begin
+        FileUtils.cp_r $config["general"]["app_cache"] + "/artifacts/unpacked.#{@checksum}/puppetcode/.", $config["agent"]["puppetcode"]
+        return true
+      rescue
+        $logger.fatal "An unexpected error occured when copying the unpacked artifact to #{$config["agent"]["puppetcode"]}"
+        raise e
+      end
+
+    end
+
+
+    def clean_install
+      # Cleanup the destination installation directory before we unpack the artifact
+      # into it, otherwise long term we will end up with old deprecated files hanging
+      # around.
+      #
+      # TODO: Do this smarter, we should track what files we drop in, and then remove
+      # any that weren't touched. Need to avoid rsync and stick with native to make
+      # support easier for weird/minimilistic distributions.
+      
+      if defined? $config["agent"]["puppetcode"]
+        if $config["agent"]["puppetcode"].empty?
+          $logger.error "You must configure a location for the agent's Puppet code to be deployed to"
+          return false
+        else
+          $logger.debug "Cleaning up #{$config["agent"]["puppetcode"]} directory"
+
+          if Dir.exists?($config["agent"]["puppetcode"])
+            FileUtils.rm_r Dir.glob($config["agent"]["puppetcode"] + "/*"), :secure => true
+          else
+            FileUtils.mkdir_p $config["agent"]["puppetcode"]
+          end
+
+          return true
+        end
+      end
+
+    end
+
+
+    def clean_unpack
+      # Cleanup/remove any unpacked archive directories. Requires that the
+      # checksum be set to the version to be purged.
+
+      unless defined? @checksum
+        raise "Application bug, trying to unpack no artifact"
+      end
+
+      if Dir.exists?($config["general"]["app_cache"] + "/artifacts/unpacked.#{@checksum}/")
+        $logger.debug "Cleaning up #{$config["general"]["app_cache"]}/artifacts/unpacked.#{@checksum}..."
+        FileUtils.rm_r $config["general"]["app_cache"] + "/artifacts/unpacked.#{@checksum}", :secure => true
+        return true
+      else
+        $logger.debug "Nothing to cleanup (selected artifact is not currently unpacked)"
+        return true
+      end
+
+      return false
+
+    end
+
+  end
+end
+
+# vim:shiftwidth=2:tabstop=2:softtabstop=2:expandtab:smartindent

data/lib/pupistry/bootstrap.rb

@@ -0,0 +1,22 @@
+require 'rubygems'
+
+module Pupistry
+  # Pupistry::Bootstrap
+
+  class Bootstrap
+    attr_accessor :template_dir
+
+    def initalize
+      template_dir = "dir"
+    end
+    
+    def self.templates_list
+      # glob all the templates
+      puts "Template"
+    end
+
+
+  end
+end
+
+# vim:shiftwidth=2:tabstop=2:softtabstop=2:expandtab:smartindent

data/lib/pupistry/config.rb

@@ -0,0 +1,80 @@
+require 'rubygems'
+require 'fileutils'
+require 'tempfile'
+require 'yaml'
+
+module Pupistry
+  # Pupistry::Config
+  #
+  # Provides loading of configuration.
+  #
+
+  class Config
+
+    def self.load file
+      $logger.debug "Loading configuration file #{file}"
+
+      unless File.exists?(file)
+        $logger.fatal "The configuration file provided does not exist, or cannot be accessed"
+        exit 0
+      end
+
+      $config = YAML::load(File.open(file))
+
+      # Make sure cache directory exists, create it otherwise
+      $config["general"]["app_cache"] = File.expand_path($config["general"]["app_cache"]).chomp('/')
+
+      unless Dir.exists?($config["general"]["app_cache"])
+        begin
+          FileUtils.mkdir_p($config["general"]["app_cache"])
+        rescue Exception => e
+          $logger.fatal "Unable to create cache directory at \"#{$config["general"]["app_cache"]}\"."
+          raise e
+        end
+      end
+
+      # Write test file to confirm writability
+      begin
+        FileUtils.touch($config["general"]["app_cache"] + "/testfile")
+        FileUtils.rm($config["general"]["app_cache"] + "/testfile")
+      rescue Exception => e
+        $logger.fatal "Unexpected exception when creating testfile in cache directory at \"#{$config["general"]["app_cache"]}\", is the directory writable?"
+        raise e
+      end
+
+    end
+
+    def self.find_and_load
+      $logger.debug "Looking for configuration file in common locations"
+  
+      # Locations in order of preference:
+      # settings.yaml (current dir)
+      # ~/.pupistry/settings.yaml
+      # /etc/pupistry/settings.yaml
+
+      config    = ''
+      local_dir = Dir.pwd
+
+      if File.exists?("#{local_dir}/settings.yaml")
+        config = "#{local_dir}/settings.yaml"
+
+      elsif File.exists?( File.expand_path "~/.pupistry/settings.yaml" )
+        config = File.expand_path "~/.pupistry/settings.yaml"
+
+      elsif File.exists?("/etc/pupistry/settings.yaml")
+        config = "/etc/pupistry/settings.yaml"
+
+      else
+        $logger.error "No configuration file provided."
+        $logger.error "See pupistry help for information on configuration"
+        exit 0
+      end
+
+      self.load(config)
+
+    end
+
+
+  end
+end
+# vim:shiftwidth=2:tabstop=2:softtabstop=2:expandtab:smartindent

data/lib/pupistry/storage_aws.rb

@@ -0,0 +1,134 @@
+require 'rubygems'
+require 'yaml'
+require 'aws-sdk-v1'
+
+module Pupistry
+  # Pupistry::Storage_AWS
+
+  class Storage_AWS
+    attr_accessor :s3
+    attr_accessor :bucket
+
+    def initialize mode
+      # mode is either "build" or "agent", depending which we load a different
+      # set of permissions. Awareness of both is intentional, since we want the
+      # build machines to known the agent creds so we can generate bootstrap
+      # template files.
+
+      unless defined? $config["general"]["s3_bucket"]
+        $logger.fatal "You must set the AWS s3_bucket"
+        exit 0
+      end
+
+      # Define AWS configuration
+      if defined? $config[mode]["access_key_id"]
+        unless $config[mode]["access_key_id"] == ''
+          $logger.debug "Loading AWS credentials from configuration file"
+
+          AWS.config(
+            :access_key_id     => $config[mode]["access_key_id"],
+            :secret_access_key => $config[mode]["secret_access_key"],
+            :region            => $config[mode]["region"],
+           :proxy_uri         => $config[mode]["proxy_uri"],
+          )
+        else
+          $logger.debug "No AWS IAM credentials specified, defaulting to environmental discovery"
+          $logger.debug "If you get weird permissions errors, try setting the credentials explicity in config first."
+        end
+      else
+        $logger.debug "No AWS IAM credentials specified, defaulting to environmental discovery"
+        $logger.debug "If you get weird permissions errors, try setting the credentials explicity in config first."
+      end
+
+      # Setup S3 bucket
+      @s3     = AWS::S3.new
+      @bucket = @s3.buckets[ $config[mode]["s3_bucket"] ]
+
+    end
+
+
+
+    def upload src, dest
+      $logger.debug "Pushing file #{src} to s3://#{$config["general"]["s3_bucket"]}/#{$config["general"]["s3_prefix"]}#{dest}"
+
+      begin
+        # Generate the object name/key based on the relative file name and path.
+        s3_obj_name = "#{$config["general"]["s3_prefix"]}#{dest}"
+        s3_obj      = @s3.buckets[$config["general"]["s3_bucket"]].objects[s3_obj_name]
+
+        # Perform S3 upload
+        s3_obj.write(:file => src)
+
+      rescue AWS::S3::Errors::NoSuchBucket => e
+        $logger.fatal "S3 bucket #{$config["general"]["s3_bucket"]} does not exist"
+        exit 0
+
+      rescue AWS::S3::Errors::AccessDenied => e
+        $logger.fatal "Access to S3 bucket #{$config["general"]["s3_bucket"]} denied"
+        exit 0
+
+      rescue AWS::S3::Errors::PermanentRedirect => e
+        $logger.error "The wrong endpoint has been specified (or autodetected) for #{$config["general"]["s3_bucket"]}."
+        raise e
+
+      rescue AWS::S3::Errors::SignatureDoesNotMatch => e
+        $logger.error "IAM signature error when accessing #{$config["general"]["s3_bucket"]}, probably invalid IAM credentials"
+        raise e
+      
+      rescue Exception => e
+        raise e
+      end
+    end
+
+
+
+   def download src, dest = 'stream'
+      $logger.debug "Downloading file s3://#{$config["general"]["s3_bucket"]}/#{$config["general"]["s3_prefix"]}#{src} to #{dest}"
+
+      begin
+        # Generate the object name/key based on the relative file name and path.
+        s3_obj_name = "#{$config["general"]["s3_prefix"]}#{src}"
+        s3_obj      = @s3.buckets[$config["general"]["s3_bucket"]].objects[s3_obj_name]
+
+        # Download the file
+        if dest == 'stream'
+          # Return the contents rather than writing to disk. We assume stream mode
+          # if the dest filename was unspecified
+          return s3_obj.read
+        else
+          # Download to an ondisk file
+          File.open(dest, 'wb') do |file|
+            s3_obj.read do |chunk|
+              file.write(chunk)
+            end
+          end
+        end
+
+      rescue AWS::S3::Errors::NoSuchKey => e
+        $logger.debug "No such file exists for download, this is normal at times."
+        return false
+
+      rescue AWS::S3::Errors::NoSuchBucket => e
+        $logger.fatal "S3 bucket #{$config["general"]["s3_bucket"]} does not exist"
+        exit 0
+
+      rescue AWS::S3::Errors::AccessDenied => e
+        $logger.fatal "Access to S3 bucket #{$config["general"]["s3_bucket"]} denied"
+        exit 0
+
+      rescue AWS::S3::Errors::PermanentRedirect => e
+        $logger.error "The wrong endpoint has been specified (or autodetected) for #{$config["general"]["s3_bucket"]}."
+        raise e
+
+      rescue AWS::S3::Errors::SignatureDoesNotMatch => e
+        $logger.error "IAM signature error when accessing #{$config["general"]["s3_bucket"]}, probably invalid IAM credentials"
+        raise e
+
+      rescue Exception => e
+        raise e
+      end
+    end
+
+  end
+end
+# vim:shiftwidth=2:tabstop=2:softtabstop=2:expandtab:smartindent

data/settings.example.yaml

@@ -0,0 +1,70 @@
+## Configuration file for Pupistry.
+
+
+# The following settings apply for all use cases of Pupistry and must be set
+general:
+
+  # We need somewhere to cache files like archives and git repos. This will
+  # be as big as the total size of all your git repos when being used to
+  # build artifacts. Agent-only systems will be far smaller as it only includes
+  # the latest version of the artifacts.
+  app_cache: ~/.pupistry/cache
+
+  # The S3 bucket must be set in order to have a place to push and
+  # pull artifact and manifests from. This bucket should be PRIVATE, we
+  # only want your servers accessing the files!
+  #
+  # REMEMBER - S3 buckets are a global namespace, other people might have
+  # already picked the name you want. Make sure you update this default
+  # with something you actually own :-)
+  s3_bucket: example
+
+  # S3 prefix is entirely optional, useful if you're reusing/sharing an S3
+  # bucket with other applications. Leave blank if not needed.
+  s3_prefix:
+
+  # GPG key to use for signing & validating the artifacts. It is possible to
+  # run pupistry in an unsigned mode, but you will lose the protection against
+  # someone with access to the S3 bucket tampering with the files and pushing
+  # malicious puppet manifests to your servers
+  gpg_disable: true
+  gpg_signing_key: XXXXXX
+
+
+# Settings for agents, these are required on the machines that will be
+# downloading and applying artifacts but also need to be set for build
+# machines so we can popular bootstrap templates for you and automatically
+# check stuff like IAM permissions before you roll your hosts.
+agent:
+  puppetcode: /etc/puppet/environents/
+
+  # The AWS credentials with READ permission to the S3 bucket for downloading
+  # artifact files. If unset, we try to figure it out from any AWS creds
+  access_key_id:
+  secret_access_key:
+  region: ap-southeast-2
+  proxy_uri:
+
+
+
+# The following settings are only needed on the build machines (people building
+# new artifacts) and are not needed on the actual agent servers that will be
+# downloading and applying them.
+build:
+
+  # Define the Git repo for the Puppet manifest & r10k data
+  # (ie repo where your Puppetfile & site.pp is)
+  puppetcode: git@bitbucket.org:user/example.git
+
+
+  # The AWS credentials with write permission to the S3 bucket for uploading
+  # new artifact files. If unset, we try to figure it out from any AWS creds
+  # set in the environmnt, but you're best to make it explicit here to avoid
+  # surprises....
+  #
+  access_key_id:
+  secret_access_key:
+  region: ap-southeast-2
+  proxy_uri:
+
+

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pupistry
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.0.2
 platform: ruby
 authors:
 - Jethro Carr
@@ -61,7 +61,12 @@ extra_rdoc_files: []
 files:
 - bin/pupistry
 - lib/pupistry.rb
+- lib/pupistry/artifact.rb
+- lib/pupistry/bootstrap.rb
+- lib/pupistry/config.rb
+- lib/pupistry/storage_aws.rb
 - README.md
+- settings.example.yaml
 homepage: https://github.com/jethrocarr/pupistry
 licenses:
 - Apache