pupistry 0.0.3 → 0.0.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 040c9a706127947909dc602972b7e9cbd28ebe42
-  data.tar.gz: 864c4368a6ae0d3103b4ca50d9cc987e135b077c
+  metadata.gz: 5ff9e16fb23fac8d09a0bf86e4bd88c452f52b29
+  data.tar.gz: f944f6ea67120033772613d9e88186ca9cacdbf8
 SHA512:
-  metadata.gz: 82f9c81beee09687e742c95fd49107b1e13edb75752f24546770a72d05f944b35f4ee3513c27b6e91f9f0bd017169ce874a52cb41866c6fe6f3dbf0bb0b10135
-  data.tar.gz: af932b06e1cef3f9f64dec3dc074e5276eb8aab4897854d8013b693c8e92aa463452b384f6041d06821e8c6fa3ba20dcb0cafcb7812fcb84124c4f46139d598b
+  metadata.gz: beb20b7264be3c5a105dc6cb5e405944b51126d0a9880f998ec226c058e1869bb05037fa829e962fd487d2281292d355d2bcefbe5bbb2ac2039e8bd016cad278
+  data.tar.gz: 0bc78b2c6f99795c877b75da67068844c89c7fc091b1c371109986d0fce1cc03d32c7ccd2b320ec412d3fe8cd02ad12d97e0719bb714d52f90cea00fb250d10a

data/README.md

@@ -109,16 +109,56 @@ off if you have it disabled. :-)
 
 ## Bootstrapping nodes
 
-You need to bootstrap your masterless nodes, which involves installing Pupistry
-and setting up Puppet configuration accordingly.
-
-    pupistry bootstrap
+New machines need to be bootstrapped in order to install Pupistry, configure it
+and be able to download configuration. Generally this is a step done differently
+site-by-site (and you can still do it that way if you want), but if you want a
+nice easy life, Pupistry can generate you a bootstrap script for your platform.
+
+    $ pupistry bootstrap
+    - rhel-7
+    - ubuntu-14.04
+
+    $ pupistry boostrap --template rhel-7
+    # Bootstrap for Red Hat Enterprise Linux Platform
+    # Compatible with RHEL 7, CentOS 7 and maybe other variations.
+    
+    rpm -ivh http://yum.puppetlabs.com/puppetlabs-release-el-7.noarch.rpm
+    rpm -ivh http://download.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
+    
+    yum update --assumeyes
+    yum install --assumeyes puppet ruby-devel rubygems
+    yum install --assumeyes gcc zlib-devel libxml2-devel patch
+    
+    gem install pupistry
+    mkdir /etc/pupistry
+    cat > /etc/pupistry/settings.yaml << "EOF"
+    general:
+      app_cache: ~/.pupistry/cache
+      s3_bucket: example
+      s3_prefix:
+      gpg_disable: true
+      gpg_signing_key: XYZXYZ
+    agent:
+      puppetcode: /etc/puppet/environents/
+      access_key_id: 
+      secret_access_key: 
+      region: ap-southeast-2
+      proxy_uri:
+    EOF
+    pupistry apply --verbose
 
-    pupistry boostrap --template rhel7
+You generally can run this on a new non-Puppetised machine, or paste into the
+user data field of most cloud providers like AWS or Digital Ocean. If using CFN
+with AWS, you can make it part of the stack itself.
 
+These bootstraps aren't mandatory, if you prefer a different approach you can
+use these as an example and write your own - generally the essential bit is to
+get puppet installed, get pupistry (and deps to build it's gems) installed and
+write the config before finally executing your first Pupistry/Puppet run.
 
-You generally can run this on a new non-Puppetised machine, or into the user
-data field of most cloud providers like AWS or Digital Ocean.
+If using AWS and IAM Roles feature, it is acceptable for access_key_id and
+secret_access_key to be blank, if not you will need to have these set to an
+account with read-only access to the configured S3 bucket!
 
 
 ## Running Puppet on target nodes
@@ -230,7 +270,7 @@ Pupistry will default to applying the "master" branch if one is not listed, if
 you are doing branch-based environments, you can specifiy when bootstrapping
 and override on a per-execution basis.
 
-You'll notice pretty quickly if something is broken when doing `puppet apply`
+You'll notice pretty quickly if something is broken when doing `pupistry apply`
 
 Confused? No worried, check out the sample repo that shows a very simple setup.
 You can copy this and start your own Puppet adventure, just add in your modules
@@ -258,11 +298,15 @@ You can fetch the module from:
 https://github.com/jethrocarr/puppet-pupistry
 
 If you're doing r10k and Puppet masterless from scratch, this is probably
-something you want to make life easy.
+something you want to make life easy. With r10k, just add the following to your
+Puppetfile:
 
+    # Install the Pupistry companion module
+    mod 'jethrocarr/pupistry'
 
 
-## 4. Bootstrapping Nodes
+
+## 4. Building you first node (Bootstrapping)
 
 No need for manual configuration of your servers/nodes, you just need to build
 your first artifact with Pupistry (`pupistry build && pupistry push`) and then
@@ -274,41 +318,12 @@ The bootstrap script will:
 2. Download the latest artifact
 3. Trigger a Puppet run to build your server.
 
-Once done, it's up to your Puppet manifests to build your machine how you want
-it - enjoy!
-
-TODO: Currently being worked on, for now the following is a rough example of
-what you can do to bootstrap a RHEL/CentOS7 box:
-
-    rpm -ivh http://yum.puppetlabs.com/puppetlabs-release-el-7.noarch.rpm
-    rpm -ivh http://download.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
-
-    yum update --assumeyes
-    yum install --assumeyes puppet ruby-devel rubygems
-    yum install --assumeyes gcc zlib-devel libxml2-devel patch
-
-    gem install pupistry
-    mkdir /etc/pupistry
-    cat > /etc/pupistry/settings.yaml << "EOF"
-    general:
-      app_cache: ~/.pupistry/cache
-      s3_bucket: example
-      s3_prefix:
-      gpg_disable: true
-      gpg_signing_key: XXXXX
-    agent:
-      puppetcode: /etc/puppet/environments
-      access_key_id: 
-      secret_access_key: 
-      region: ap-southeast-2
-      proxy_uri:
-    EOF
-    pupistry apply --verbose
-
-It will setup the Puppet repo for RHEL 7, install updates (remember that security
-thing?) and then install the gem (and deps to build it). Then we generate the
-minimal configuration file needed, and kick off a Puppet run!
+These bootstrap scripts can be generated for you. Refer to "Bootstrapping" under
+"Usage" instructions above for details.
 
+The bootstrap script goal is to get you from stock OS to running Pupistry and
+doing your first Puppet run. After that - it's up to you and your Puppet
+skills to make your node actually do something useful. :-)
 
 
 # Tutorials

data/bin/pupistry

@@ -23,9 +23,11 @@ class CLI < Thor
   ## Agent Commands
   
   desc "apply", "Apply the latest Puppet artifact"
-  method_option :noop, :type => :boolean
-  method_option :daemon, :type => :boolean
-  method_option :environment, :type => :string
+  method_option :noop,        :type => :boolean, :desc => "No changes mode (note: does change checked out artifact, but no Puppet changes)"
+  method_option :force,       :type => :boolean, :desc => "Ignore existing versions, re-apply every time"
+  method_option :minimal,     :type => :boolean, :desc => "Don't run Puppet unless the artifact has changed"
+  method_option :daemon,      :type => :boolean, :desc => "Run as a system daemon"
+  method_option :environment, :type => :string,  :desc => "Specifiy which environment to deploy (default: master)"
   def apply
 
     # Thor seems to force class options to be defined repeatedly? :-/
@@ -52,24 +54,66 @@ class CLI < Thor
       exit 0
     end
 
-    # Install the artifact
-    # TODO: Check if we've already installed it or not... :-)
-    $logger.info "Pulling latest artifact...."
+
+    # Fetch artifact versions
+    $logger.info "Checking version of artifact available..."
 
     artifact = Pupistry::Artifact.new
     artifact.checksum = artifact.fetch_latest
-    artifact.fetch_artifact
-    artifact.unpack
-    unless artifact.install
-      $logger.fatal "An unexpected error happened when installing the latest artifact, cancelling Puppet run"
-      exit 0
-    end
 
-    # Remove temporary unpacked files
-    artifact.clean_unpack
+    artifact_installed = Pupistry::Artifact.new
+    artifact_installed.checksum = artifact_installed.fetch_installed
 
 
-    # Execute Puppet. At this point
+    # Download the new artifact if one has changed. If we already have this
+    # version, then we should skip downloading and go straight to running
+    # Puppet - unless the user runs with --force (eg to fix a corrupted
+    # artifact).
+    $logger.debug "Currently on #{artifact_installed.checksum}"
+    
+    if artifact.checksum != artifact_installed.checksum or options[:force]
+      if options[:force]
+        $logger.warn "Forcing download of latest artifact regardless of current one."
+      end
+
+      # Install the artifact
+      $logger.info "Downloading latest artifact (#{artifact.checksum})..."
+
+      artifact.fetch_artifact
+      artifact.unpack
+
+      unless artifact.install
+        $logger.fatal "An unexpected error happened when installing the latest artifact, cancelling Puppet run"
+        exit 0
+      end
+
+      # Remove temporary unpacked files
+      artifact.clean_unpack
+    else
+      $logger.info "Already have latest artifact applied."
+
+      # By default we run Puppet even if we have the latest artifact. There's
+      # some grounds for debate about whether this is the right thing - in some
+      # ways it is often a waste of CPU, since if the artifact hasn't changed,
+      # then it's unlikley anything else has changed.
+      #
+      # But that's not always 100% true - Puppet will undo local changes or
+      # upgrade package versions (ensure => latest) if appropiate, so we should
+      # act like the standard command and attempt to apply whatever we can.
+      #
+      # To provide users with options, we provide the --lazy parameter to avoid
+      # running Puppet except when the artifact changes. By default, Puppet
+      # runs every thing to avoid surprise.
+      
+      if options[:minimal]
+        $logger.info "Running with minimal effort mode enabled, not running Puppet since artifact version already applied"
+        exit 0
+      end
+
+    end
+
+
+    # Execute Puppet.
     puppet_cmd = "puppet apply"
 
     if options[:noop]
@@ -242,11 +286,15 @@ class CLI < Thor
 
     if options[:template]
       $logger.info "Generating bootstrap template #{options[:template]}"
-    else
-      puts "Listing all avaible templates"
-      Pupistry::Bootstrap.templates_list
 
-      
+      templates = Pupistry::Bootstrap.new
+      templates.build options[:template]
+    else
+      templates = Pupistry::Bootstrap.new
+      templates.list
+     
+      puts "--"
+      puts "Tip: Run `pupistry bootstrap --template example` to generate a specific template"
     end
       
   end

data/lib/pupistry/artifact.rb

@@ -111,6 +111,27 @@ module Pupistry
     end
 
 
+    def fetch_installed
+      # Fetch the current version that is installed.
+      
+      # Make sure the Puppetcode install directory exists
+      unless Dir.exists?($config["agent"]["puppetcode"])
+        $logger.warn "The destination path of #{$config["agent"]["puppetcode"]} does not appear to exist or is not readable"
+        return 0
+      end
+
+      # Look for a manifest file in the directory and read the version from it.
+      if File.exists?($config["agent"]["puppetcode"] + "/manifest.pupistry.yaml")
+        manifest = YAML::load(File.open($config["agent"]["puppetcode"] + "/manifest.pupistry.yaml"))
+
+        return manifest['version']
+      else
+        $logger.warn "No current version installed"
+        return 0
+      end
+    end
+
+
     def fetch_artifact
 
       # Figure out which version to fetch (if not explicitly defined)
@@ -387,6 +408,7 @@ module Pupistry
       # Clone unpacked contents to the installation directory
       begin
         FileUtils.cp_r $config["general"]["app_cache"] + "/artifacts/unpacked.#{@checksum}/puppetcode/.", $config["agent"]["puppetcode"]
+        FileUtils.cp   $config["general"]["app_cache"] + "/artifacts/manifest.#{@checksum}.yaml", $config["agent"]["puppetcode"] + "/manifest.pupistry.yaml"
         return true
       rescue
         $logger.fatal "An unexpected error occured when copying the unpacked artifact to #{$config["agent"]["puppetcode"]}"
@@ -395,7 +417,6 @@ module Pupistry
 
     end
 
-
     def clean_install
       # Cleanup the destination installation directory before we unpack the artifact
       # into it, otherwise long term we will end up with old deprecated files hanging

data/lib/pupistry/bootstrap.rb

@@ -1,4 +1,5 @@
 require 'rubygems'
+require 'erubis'
 
 module Pupistry
   # Pupistry::Bootstrap
@@ -6,16 +7,82 @@ module Pupistry
   class Bootstrap
     attr_accessor :template_dir
 
-    def initalize
-      template_dir = "dir"
+    def initialize
+
+      # We need to find where the templates are located - either it should be
+      # in the current working directory, or if we are an installed gem, we
+      # can try the gem's installed path.
+
+      if Dir.exists?("resources/bootstrap/")
+        # Use local PWD version first if possible
+        @template_dir = Dir.pwd
+      else
+        # Check for GEM installed location
+        begin
+          @template_dir = Gem::Specification.find_by_name("pupistry").gem_dir
+        rescue Gem::LoadError
+          $logger.error "Unable to find templates/ directory, doesn't appear we are running from project dir nor as a Gem"
+          return false
+        end
+      end
+
+      @template_dir = @template_dir.chomp("/") + "/resources/bootstrap/"
+
+      unless Dir.exists?(@template_dir)
+        $logger.error "Unable to find templates dir at #{@template_dir}, unable to proceed."
+        return false
+      else
+        $logger.debug "Using directory #{@template_dir} for bootstrap templates"
+      end
+
     end
     
-    def self.templates_list
-      # glob all the templates
-      puts "Template"
+
+    def list
+      # Simply glob the templates directory and list their names.
+      $logger.debug "Finding all available templates"
+
+      Dir.glob("#{@template_dir}/*.erb").each do |file|
+        puts "- #{File.basename(file, ".erb")}"
+      end
     end
 
 
+    def build template
+      # Build a template with the configured parameters already to go
+      $logger.debug "Generating a bootstrap script for #{template}"
+
+      unless File.exists?("#{@template_dir}/#{template}.erb")
+        $logger.error "The requested template does not exist, unable to build"
+        return 0
+      end
+
+      # Assume values we care about
+      template_values = {
+        s3_bucket: $config["general"]["s3_bucket"],
+        s3_prefix: $config["general"]["s3_prefix"],
+        gpg_disable: $config["general"]["gpg_disable"],
+        gpg_signing_key: $config["general"]["gpg_signing_key"],
+        puppetcode: $config["agent"]["puppetcode"],
+        access_key_id: $config["agent"]["access_key_id"],
+        secret_access_key: $config["agent"]["secret_access_key"],
+        region: $config["agent"]["region"],
+        proxy_uri: $config["agent"]["proxy_uri"],
+      }
+
+      # Generate template using ERB
+      begin
+        template_contents = Erubis::Eruby.new(File.read("#{@template_dir}/#{template}.erb")).result(template_values)
+      rescue Exception => e
+        $logger.error "An unexpected error occured when trying to generate the bootstrap template"
+        raise e
+      end
+
+      # Output template
+      puts "-- Bootstrap Start --"
+      puts template_contents
+      puts "-- Bootstrap End --"
+    end
   end
 end
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: pupistry
 version: !ruby/object:Gem::Version
-  version: 0.0.3
+  version: 0.0.4
 platform: ruby
 authors:
 - Jethro Carr
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-04-10 00:00:00.000000000 Z
+date: 2015-04-14 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: aws-sdk-v1
@@ -38,6 +38,20 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: erubis
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: r10k
   requirement: !ruby/object:Gem::Requirement