pupistry 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 2c47a743b2bf72dc35a18a04e4c907029773f9cd
+  data.tar.gz: 0ffd31d0773fac92db4989fcd52bafb4b25e2215
+SHA512:
+  metadata.gz: 36fa8ed2a0f3b0fc34d548649582143193fa78e29420bda0f69d5a0c27488c93e4f96a4c102cdcc533cdf9f2414ba8bbb75c28da8363822c59ff8b386987a750
+  data.tar.gz: 715fa89d66398222226527be295da1532d17c9f32a35e2d1d878ac7564a5f3cb5a3632f819cdd6a052d1e3fefed4e6de6381549591904a8014b411219baa29ee

data/README.md

@@ -0,0 +1,296 @@
+# WORK IN PROGRESS
+
+This project is currently in progress and not all of this documentation reflects
+where it is really at. Use this at your own peril/madness.
+
+# pupistry
+
+Pupistry (puppet + artistry) is a solution for implementing reliable and secure
+masterless puppet deployments by taking Puppet modules assembled by r10k and
+generating compresed and signed archives for distribution to the masterless
+servers.
+
+Pupistry builds on the functionality offered by the r10k workflow but rather
+than requiring the implementing of site-specific custom bootstrap and custom
+workflow mechanisms, Pupistry executes r10k, assembles the combined modules
+and then generates a compress artifact file. It then signs the artifact with
+GPG and uploads it into an Amazon S3 bucket along with a manifest file.
+
+The masterless Puppet machines then just run a Pupistry job which checks for a
+new version of the manifest file. If there is, it downloads the new artifact
+and does a GPG validation before applying it and running Puppet. To make life
+even easier, Pupistry will even spit out bootstrap files for your platform
+which sets up each server from scratch to pull and run the artifacts.
+
+Essentially Pupistry is intended to be a robust solution for masterless Puppet
+deployments and makes it trivial for beginners to get started with Puppet.
+
+
+# Why Pupistry?
+
+Masterless Puppet is a great solution for anyone wanting to avoid scaling issues
+and risk of centralised failure due to a central Puppet master, but it does bring
+a number of issues with it.
+
+1. Having to setup deployer keys to every git repo used is a maintainance headache. Pupistry means only your workstation needs access, which presumably will have access to most/all repos already.
+2. Your system build success is dependent on all the git repos you've used, including any third parties that could vanish. A single missing or broken repo could prevent autoscaling or new machine builds at a critical time. Pupistry's use of artifact files prevents surprises - if you can hit S3, you're sorted.
+3. It is easy for malicious code in the third party repos to slip in without noticing. Even if the author themselves is honest, not all repos have proper security like two-factor. Pupistry prevents surprise updates of modules and also has an easy diff feature to see what changed since you last generated an artifact.
+4. Puppet masterless tends to be implemented in many different ways using everyone's own hacky scripts. Pupistry's goal is to create a singular standard/approach to masterless, in the same way that r10k created a standard approach to git-based Puppet workflows. And this makes things easy - install Pupistry, add the companion Puppet module and run the bootstrap script. Easy!
+5. No dodgy cronjobs running r10k and Puppet in weird ways. A simple clean agent with daemon or run-once functionality.
+6. Performance - Go from 30+ seconds r10k update checks to 2 second Pupistry update checks. And when there is a change, it's a fast efficent compressed file download from S3 rather than pulling numerious git repos.
+
+
+
+# Usage
+
+## Building new artifacts
+
+Build a new artifact:
+
+    pupistry build
+
+Note that artifact builds are done from the upstream git repos, so if you
+have made changes, remember to git push first before generating. The tool will
+remind you if it detects nothing has changed since the last run.
+
+Once your artifact is built, you can double check what has changed in the
+Puppet modules since the last run with:
+
+    pupistry diff
+
+
+Finally when you're happy, push it to S3 to be delivered to all your servers. 
+If you have gpg signing enabled, it will ask you to sign here.
+
+    pupistry push
+
+
+## Bootstrapping nodes
+
+You need to bootstrap your masterless nodes, which involves installing Pupistry
+and setting up Puppet configuration accordingly.
+
+    pupistry bootstrap
+
+    pupistry boostrap --template rhel7
+
+
+You generally can run this on a new non-Puppetised machine, or into the user
+data field of most cloud providers like AWS or Digital Ocean.
+
+
+## Running Puppet on target nodes
+
+Check what is going to be applied (Puppet in --noop mode)
+
+    pupistry apply --noop
+
+
+Apply the current Puppet manifests:
+
+    pupistry apply
+
+Specify an alternative environment:
+
+    pupistry apply --environment staging
+
+
+Run pupistry as a system daemon. When you use the companion Puppet module, a
+system init file gets installed that sets this daemon up for you automatically.
+
+    pupistry apply --daemon
+
+
+
+# Installation
+
+## 1. Application
+
+First install Pupistry onto your workstation. You can make pupistry generate
+you a config file if you've never used it before
+
+    gem install pupistry
+    pupistry setup
+
+Alternatively if you like living on the edge, download this repository and run:
+
+    gembuild pupistry.gemspec
+    gem install pupistry-VERSION.gem
+    pupistry setup
+
+
+## 2. S3 Bucket
+
+Pupistry uses S3 for storing and pulling the artifact files. You need to
+configure the following:
+
+* A *private* S3 bucket (you'll get this by default).
+* An IAM account with access to write that bucket (for your build workstation)
+* An IAM account with access to read that bucket (for your servers)
+
+If you're not already using IAM with your AWS account you want to be - your
+servers should only ever have read access to the bucket and only your build
+workstation should be permitted to write new artifacts. IE, don't share your
+AWS root account around the place. :-)
+
+Note that if you're running EC2 instances and using IAM roles, you can avoid
+needing to create explicit IAM credentials for the agents/servers.
+
+
+
+## 3. Helper Puppet Module
+
+Whilst you can use Pupistry to roll out any particular design of Puppet
+manifests, you will save yourself a lot of pain by also including the Pupistry
+companion Puppet module in your manifests.
+
+The companion Puppet module will configure Pupistry for you, including setting
+up the system service and configuring Puppet and Hiera correctly for masterless
+operation.
+
+You can fetch the module from:
+https://github.com/jethrocarr/pupistry-puppet
+
+If you're doing r10k and Puppet masterless from scratch, this is probably
+something you want to make life easy.
+
+
+## 4. Bootstrapping Nodes
+
+No need for manual configuration of your servers/nodes, you just need to build
+your first artifact with Pupistry (`pupistry build && pupistry push`) and then
+generate a bootstrap script for your particular OS with `pupistry bootstrap`
+
+The bootstrap script will:
+1. Install Puppet and Pupistry for the particular OS.
+2. Download the latest artifact
+3. Trigger a Puppet run to build your server.
+
+Once done, it's up to your Puppet manifests to build your machine how you want
+it - enjoy!
+
+
+
+# Tutorials
+
+If you're looking for a more complete introduction to doing masterless Puppet
+and want to use Pupistry, check out a tutorial by the author:
+
+TUTORIAL LINK HERE
+
+By following this tutorial you can go from nothing, to having a complete up
+and running masterless Puppet environment using Pupistry. It covers the very
+basics of setting up your r10k environment.
+
+
+
+
+# Caveats & Future Plans
+
+## Use r10k
+
+Currently only an r10k workflow is supported. Pull requests for others (eg
+Librarian Puppet) are welcome, but it's not a priority for this author as r10k
+is working nicely.
+
+
+## Bootstrap Functionality
+
+Currently Pupistry only supports generation of bootstrap for CentOS 7 & Ubuntu
+14.04. Other distributions will be added, but it may take time to get to your
+particular favourite distribution.
+
+Note that it isn't a show stopper if support for your platform of choice
+doesn't yet exist -  you can use pupistry with pretty much any nix platform,
+you'll just not have the handy advantage of automatically generated bootstrap
+for your servers.
+
+If you do customise it for a different platform, pull requests are VERY
+welcome, I'll add pretty much any OS if you write a decent bootstrap template
+for it.
+
+
+## Continious Deployment
+
+A lot of what Pupistry does can also be accomplished by various home-grown
+Continious Deployment (CD) solutions using platforms like Jenkins or Bamboo. CD
+is an excellent approach for larger organisations, but Pupistry has been
+designed for both large and small users so does not mandate it.
+
+It would be possible to use Pupistry as part of your CD process and if you
+decide to do so, a pull request to better support CD systems out-of-the-box
+would be welcome.
+
+
+## Hiera Security Still Sucks
+
+In a standard Puppet master situation, the Puppet master parses the Hiera data
+and then passes only the values that apply to a particular host to it. But with
+masterless Puppet, all machines get a full copy of Hiera data, which could be a
+major issue if one box gets expoited and the contents leaked. Generally it goes
+against good practise and damanges the isolation ability of VMs if you give all
+the VMs enough information to do some serious damage to themselves.
+
+Pupistry does not yet have any solution for it and it remains a fundamental
+limitation of the Puppet masterless approach. Longer term, we could potentially
+craft a solution that customises the artifacts per-machine to fix this security
+gap, but there's no proper solution currently.
+
+If you have an environment where you need to send lots of sensitive values to
+your servers, a traditional master-full Puppet environment may be a better
+solution for this reason. But if you can architect to avoid this or have no
+critical secrets in Hiera, Pupistry should be good for you.
+
+
+## PuppetDB
+
+There's nothing stopping you from using PuppetDB other than Pupistry has no
+automatic setup hooks in the bootstrap config. Pull requests to support
+PuppetDB for masterless machines are welcome, although masterless users tend
+to want to avoid dependencies on a central point.
+
+
+## Windows
+
+No idea whether this works under Windows, or what would be required to make it
+do so. Again, pull requests always welcome but it's not a priority for the
+author.
+
+
+
+# Developing
+
+When developing Pupistry, you can run the git repo copy with:
+
+    ruby -Ilib/ -r rubygems bin/pupistry
+
+By default Pupistry will try to load a settings.yaml file in the current
+working directory, before then trying ~/.pupistry/settings.yaml and then
+finally /etc/pupistry/settings.yaml. You can also override with --config.
+
+Add --verbose for additional debugging information.
+
+
+# Contributions
+
+Pull requests are very welcome. Pupistry is a very young app and there is
+plenty of work that can be done to improve it's code quality, enhance existing
+features and add handy new features. Constructive feedback/requests via the
+issue tracker is fine, but pull requests speak louder than words. :-)
+
+If you find a bug or need support, please use the issue tracker rather than
+personal emails to the author.
+
+
+# Author
+
+Pupistry is developed by Jethro Carr. Blog posts about Pupistry and new
+features can be found at http://www.jethrocarr.com/tag/pupistry
+
+Beer welcome.
+
+
+# License
+
+Pupistry is licensed under the Apache License, Version 2.0.
+

data/bin/pupistry

@@ -0,0 +1,177 @@
+#!/usr/bin/env ruby
+# Lancher for Pupistry CLI
+
+require 'rubygems'
+require 'thor'
+require 'logger'
+require 'pupistry'
+
+# Ensure all output is real time - this is a long running process with
+# continual output, we want it to sync ASAP
+STDOUT.sync = true
+
+# Logging - STDOUT only
+$logger = Logger.new(STDOUT)
+
+
+# Thor is a toolkit for producing command line applications, see http://whatisthor.com/
+class CLI < Thor
+  class_option :verbose, :type => :boolean
+  class_option :config, :type => :string
+
+  desc "build", "Build a new archive file"
+  def build
+
+    # Thor seems to force class options to be defined repeatedly? :-/
+    if options[:verbose]
+      $logger.level = Logger::DEBUG
+    else
+      $logger.level = Logger::INFO
+    end
+
+    if options[:config]
+      Pupistry::Config.load(options[:config])
+    else
+      Pupistry::Config.find_and_load
+    end
+
+
+    begin
+      # Fetch the latest data with r10k
+      artifact = Pupistry::Artifact.new
+
+      artifact.fetch_r10k
+      artifact.build_artifact
+
+      puts "--"
+      puts "Tip: Run pupistry diff to see what changed since the last artifact version"
+
+    rescue Exception => e
+      $logger.fatal "An unexpected error occured when trying to generate the new artifact file"
+      raise e
+    end
+
+
+  end
+
+
+  desc "diff", "Show what has changed between now and the current live artifact"
+  def diff
+
+    # Thor seems to force class options to be defined repeatedly? :-/
+    if options[:verbose]
+      $logger.level = Logger::DEBUG
+    else
+      $logger.level = Logger::INFO
+    end
+
+    if options[:config]
+      Pupistry::Config.load(options[:config])
+    else
+      Pupistry::Config.find_and_load
+    end
+
+    # Fetch the latest artifact
+    artifact_upstream          = Pupistry::Artifact.new
+    artifact_upstream.checksum = artifact_upstream.fetch_latest
+
+    unless artifact_upstream.checksum
+      $logger.error "There is no upstream artifact to compare to."
+      exit 0
+    end
+
+    artifact_upstream.fetch_artifact
+
+    # Fetch the current artifact
+    artifact_current          = Pupistry::Artifact.new
+    artifact_current.checksum = artifact_current.fetch_current
+
+    unless artifact_current.checksum
+      $logger.error "There is no current artifact to compare to, run \"pupistry build\" first to generate one with current changes"
+      exit 0
+    end
+
+    artifact_current.fetch_artifact
+
+    # Unpack the archives
+    artifact_current.unpack
+    artifact_upstream.unpack
+
+    # Diff the contents. This is actually bit of a pain, there's no native way
+    # of diffing an entire directory and a lot of the gems out there that promise
+    # to do diffing a) can't handle dirs and b) generally exec out to native diff
+    # anyway. :-(
+    #
+    # So given this, we might as well go native and just rely on the system
+    # diff command to do the job.
+    
+    Dir.chdir("#{$config["general"]["app_cache"]}/artifacts/") do
+      unless system "diff -Nuar unpacked.#{artifact_upstream.checksum} unpacked.#{artifact_current.checksum}"
+      end
+    end
+
+
+    # Cleanup
+    artifact_current.clean_unpack
+    artifact_upstream.clean_unpack
+
+    puts "--"
+    puts "Tip: Run pupistry push to GPG sign & upload if happy to go live"
+  end
+
+
+  desc "push", "Sign & Upload a new artifact version"
+  def push
+
+    # Thor seems to force class options to be defined repeatedly? :-/
+    if options[:verbose]
+      $logger.level = Logger::DEBUG
+    else
+      $logger.level = Logger::INFO
+    end
+
+    if options[:config]
+      Pupistry::Config.load(options[:config])
+    else
+      Pupistry::Config.find_and_load
+    end
+
+    # Push the artifact to S3
+    artifact = Pupistry::Artifact.new
+    artifact.push_artifact
+  end
+
+
+  desc "bootstrap", "Generate a user-data bootstrap script for a node"
+  method_option :template, :type => :string
+  def bootstrap
+
+    # Thor seems to force class options to be defined repeatedly? :-/
+    if options[:verbose]
+      $logger.level = Logger::DEBUG
+    else
+      $logger.level = Logger::INFO
+    end
+
+    if options[:config]
+      Pupistry::Config.load(options[:config])
+    else
+      Pupistry::Config.find_and_load
+    end
+
+
+    if options[:template]
+      $logger.info "Generating bootstrap template #{options[:template]}"
+    else
+      puts "Listing all avaible templates"
+      Pupistry::Bootstrap.templates_list
+
+      
+    end
+      
+  end
+end
+
+CLI.start(ARGV)
+
+# vim:shiftwidth=2:tabstop=2:softtabstop=2:expandtab:smartindent

data/lib/pupistry.rb

@@ -0,0 +1,8 @@
+require 'pupistry/config'
+require 'pupistry/bootstrap'
+require 'pupistry/artifact'
+require 'pupistry/storage_aws'
+
+
+
+# vim:shiftwidth=2:tabstop=2:softtabstop=2:expandtab:smartindent

metadata

@@ -0,0 +1,89 @@
+--- !ruby/object:Gem::Specification
+name: pupistry
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- Jethro Carr
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2015-04-04 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: aws-sdk-v1
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: thor
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: r10k
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Provides security, reliability and consistency to Puppet masterless environments
+email: jethro.carr@jethrocarr.com
+executables:
+- pupistry
+extensions: []
+extra_rdoc_files: []
+files:
+- bin/pupistry
+- lib/pupistry.rb
+- README.md
+homepage: https://github.com/jethrocarr/pupistry
+licenses:
+- Apache
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.0.14
+signing_key: 
+specification_version: 4
+summary: A workflow tool for Puppet Masterless Deployments
+test_files: []