environmate 0.1.0

data/README.md

@@ -0,0 +1,190 @@
+# Environmate
+
+Environmate is a ruby sinatra web application to deploy
+Puppet code to a Puppet master.
+
+This web application will completely take control of your
+environments directory and purge every other environment already in there
+or which is deployed by other means.
+
+Currently it supports webhook events from Gitlab, but this may be easily
+extended in the future to more GIT web frontends.
+
+## Installation
+
+Environmate comes as ruby gem. You can install it on
+your puppet master by running:
+
+    $ /opt/puppetlabs/puppet/bin/gem install environmate
+
+## Usage
+
+Environmate comes with puma included and no additional setup
+is required. Simply run environmate from the console:
+
+    $ /opt/puppetlabs/puppet/bin/environmate --help
+
+## Environment Management
+
+Environmate provides two different ways to deploy code which represent two
+different use cases and are explained in the following section.
+
+### Dynamic Environments
+
+Dynamic environments are puppet environments which are derived from a
+GIT branch which starts with a defined prefix. The default prefix is
+'env/'. All other branches will simply be ignored.
+
+To deploy such branches as environments a web hook has to registered in Gitlab
+with the API endpoint '/gitlab_push' on push events. The web application will
+examine each push event and deploy the specified revision.
+
+The typical use case for this type of deployment is for development purposes:
+
+Start a new branch on a local development machine and push it to your GIT
+server:
+
+    $ git checkout -b env/mynewfeature
+    $ vim modules/foo/manifests/init.pp
+    $ git commit -a -m "my awesome new feature"
+    $ git push -u origin env/mynewfeature
+
+Environmate will now automatically create a new environment with the name
+'mynewfeature' which you can test on any node with:
+
+    $ puppet agent --test --environment mynewfeature
+
+If you delete the branch the environment will be removed again.
+
+### Static Environments
+
+Static environments are predefined environments which can be deployed with
+a different API endpoint '/deploy'. The main purpose is to deploy arbitrary
+revisions from a deployment pipeline outside the version control system.
+The push request to this endpoint has to contain the following data:
+
+    {
+      "environment": "name_of_the_puppet_environment",
+      "token":       "secret_token",
+      "revision":    "git_commit_revision_(sha1)"
+    }
+
+Those environments and their tokens have to be preconfigured in order to work.
+Dynamic environments can not have the same name as a static environment and will
+fail to deploy if created.
+
+## Features
+
+### Atomic Deployments
+
+Environmate deploys the code for the revisions in directories with the name
+of the SHA1 reference. Only after the environment is completely deployed it will add
+a link with the puppet environment name or switch an existing name in an atomic operation
+which guarantees that there is never an invalid environment.
+
+If the deployment of a revision fails for some reason, the link will remain on the old
+revision and the puppet master stays unaffected.
+
+### Notifications
+
+Since the Gitlab hook provides information about the users email it is possible to
+configure a mapping of email to jabber accounts to inform the user in a timely
+fashion about the progress of the deployment.
+
+However this will only work for ruby > 1.9.3 since xmpp4r uses classes not
+available in previous ruby versions.
+
+## Configuration
+
+Environmate will attempt to load the configuration in the following order.
+It will use the first yaml file it finds:
+
+- '/etc/environmate.yml'
+- '~/.environmate.yml'
+
+Additionally you can provide a configuration file when starting the environmate
+service:
+
+    $ /opt/puppetlabs/puppet/bin/environmate --config /path/to/my/conf.yml
+
+Here is a complete example config:
+
+    production:
+      environment_path: '/etc/puppetlabs/code/environments'
+
+      lockfile_path: '/var/run/lock/puppet_deploy_lock'
+      lockfile_options:
+        timeout: 300
+
+      logfile: '/var/log/environmate.log'
+      loglevel: 'INFO'
+
+      master_repository: 'http://gitlab.exmple.com/puppet/control'
+      master_branch: 'origin/master'
+      master_path: '/etc/puppetlabs/code/environmate_master'
+
+      dynamic_environments_prefix: 'env/'
+
+      static_environments:
+        nonprod:
+          token: 'abc123'
+        prod:
+          token: '123abc'
+
+      xmpp:
+        username: 'foo@jabber.example.com'
+        password: 'foofoofoo'
+        users:
+          - bob@example.com: 'bob@jabber.example.com'
+          - alice@example.com: 'alice@jabber.example.com'
+
+      server_settings:
+        :Port: 4443
+        :Bind: '0.0.0.0'
+        :SSLEnable: true
+        :SSLCertificate: '/path/to/your/cert.pem'
+        :SSLPrivateKey: '/path/to/your/key.pem'
+
+## Internals
+
+### Locking
+
+To prevent deployment processes from interfering with each other, only one deployment
+can happen at a certain time. Environmate will halt additional deployment requests
+until the deployment in progress is finished in which case the next deployment waiting will
+be started automatically.
+
+### Master Repository
+
+The master repository is only there to work as a starting point for new branches, so we
+don't have to clone the whole environment from scratch each time. Environmate
+will try to find the shortest way to deploy an environment from the already deployed
+revisions. If no good starting point can be evaluated it will default to the master.
+
+## Debugging
+
+To easily debug environmate you can start it manually with the foreground flag to get all
+the log output to the console. This way you don't have to adjust the config:
+
+    $ /opt/puppetlabs/puppet/bin/environmate --foreground --verbosity DEBUG
+
+If you want to see stacktraces add the trace flag:
+
+    $ /opt/puppetlabs/puppet/bin/environmate --foreground --verbosity DEBUG --trace
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/puzzle/environmate.
+This project is intended to be a safe, welcoming space for collaboration, and contributors are
+expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+
+## License
+
+The gem is available as open source under the terms of the GNU General Public License 3.0
+(https://www.gnu.org/licenses/gpl-3.0.en.html)
+
+## Code of Conduct
+
+Everyone interacting in the Environmate project’s codebases, issue trackers, chat rooms
+and mailing lists is expected to follow the
+[code of conduct](https://github.com/puzzle/environmate/blob/master/CODE_OF_CONDUCT.md).

data/Rakefile

@@ -0,0 +1,6 @@
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require 'bundler/setup'
+require 'environmate'
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require 'irb'
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/environmate.gemspec

@@ -0,0 +1,39 @@
+
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'environmate/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = 'environmate'
+  spec.version       = Environmate::VERSION
+  spec.authors       = ['Andreas Zuber']
+  spec.email         = ['zuber@puzzle.ch']
+
+  spec.summary       = 'Manage Puppet environments with a GIT workflow'
+  spec.description   = 'Environmate is a Webhook receiver for various GIT '\
+                       'web frontends for deloying Puppet environments to'\
+                       'the master/server'
+  spec.homepage      = 'https://github.com/puzzle/environmate'
+  spec.license       = 'GPL-3.0'
+
+  spec.files         = `git ls-files -z`.split("\x0").reject do |f|
+    f.match(%r{^(test|spec|features)/})
+  end
+  spec.bindir        = 'exe'
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ['lib']
+
+  spec.add_dependency 'sinatra', '~> 2'
+  spec.add_dependency 'lockfile', '~> 2'
+
+  # TODO: make this dependencies optional
+  spec.add_dependency 'xmpp4r', '~> 0'
+  spec.add_dependency 'librarian-puppet', '~> 3'
+
+  spec.add_development_dependency 'bundler', '~> 1.16'
+  spec.add_development_dependency 'rake', '~> 10.0'
+  spec.add_development_dependency 'rspec', '~> 3.0'
+  spec.add_development_dependency 'rack-test', '~> 0.8'
+  spec.add_development_dependency 'simplecov', '~> 0.15'
+  spec.add_development_dependency 'rubocop', '~> 0.51'
+end

data/exe/environmate

@@ -0,0 +1,51 @@
+#!/usr/bin/env ruby
+
+$:.push File.join(File.dirname(__FILE__), '..', 'lib')
+
+require 'optparse'
+require 'environmate/app'
+
+options = {
+  verbosity:  nil,
+  foreground: false,
+  trace:      false,
+}
+
+parser = OptionParser.new do|opts|
+	opts.banner = "Usage: environmate [options]"
+
+	opts.on('-c', '--config CONFIGFILE.YML', 'Location of the configuration file') do |config_file|
+		options[:config_file] = config_file
+	end
+
+  opts.on('-e', '--rack_env RAILS_ENV', 'Rack environment') do |rack_env|
+		ENV['RACK_ENV'] = rack_env
+	end
+
+	opts.on('-v', '--verbosity VERBOSITY', 'Log Verbosity: ERROR, WARN, INFO, DEBUG') do |verbosity|
+		options[:verbosity] = verbosity
+	end
+
+	opts.on('-f', '--foreground', 'Log to console instead of logfile and don\'t daemonize') do
+		options[:foreground] = true
+	end
+
+  opts.on('-t', '--trace', 'Print backtrace on error') do
+		options[:trace] = true
+	end
+
+	opts.on('-h', '--help', 'Displays Help') do
+		puts opts
+		exit
+	end
+end
+
+parser.parse!
+
+begin
+  Environmate::App.run!(options)
+rescue => e
+  puts e.message
+  puts e.backtrace if options[:trace]
+  exit(-1)
+end

data/lib/environmate.rb

@@ -0,0 +1,14 @@
+require 'environmate/version'
+require 'environmate/errors'
+require 'environmate/configuration'
+require 'environmate/log'
+require 'environmate/user'
+require 'environmate/xmpp'
+require 'environmate/command'
+require 'environmate/git_repository'
+require 'environmate/deployment'
+require 'environmate/environment'
+require 'environmate/environment_manager'
+
+module Environmate
+end

data/lib/environmate/app.rb

@@ -0,0 +1,83 @@
+#
+# This is the main application class
+#
+require 'json'
+require 'sinatra/base'
+require 'environmate'
+
+module Environmate
+  class App < Sinatra::Base
+
+    def self.run!(options = {})
+      Environmate.load_configuration(settings.environment.to_s, options[:config_file])
+      configuration = Environmate.configuration
+
+      logfile  = options[:foreground] ? STDOUT : configuration['logfile']
+      loglevel = options[:verbosity] || configuration['loglevel']
+      Environmate.logger = Logger.new(logfile)
+      Environmate.log.level = Logger.const_get(loglevel.upcase)
+
+      Environmate::Xmpp.init
+
+      server_settings = configuration['server_settings']
+
+      if server_settings[:SSLEnable]
+        require 'webrick/https'
+
+        ssl_cert = server_settings[:SSLCertificate] || ""
+        ssl_key  = server_settings[:SSLPrivateKey] || ""
+        # replace cert filename with content
+        if File.exists?(ssl_cert)
+          server_settings[:SSLCertificate] = OpenSSL::X509::Certificate.new(File.open(ssl_cert).read)
+        end
+        if File.exists?(ssl_key)
+          server_settings[:SSLPrivateKey] = OpenSSL::PKey::RSA.new(File.open(ssl_key).read)
+        end
+      end
+
+      Rack::Handler::WEBrick.run(self, server_settings)
+    end
+
+    #
+    # push hook for deployment of dynamic environments
+    # from gitlab
+    #
+    post '/gitlab_push' do
+      data         = JSON.parse(request.body.read)
+      user         = Environmate::User.new(data['user_email'])
+      branch       = data['ref'].gsub('refs/heads/', '')
+      old_revision = data['before'].to_s
+      new_revision = data['after'].to_s
+
+      # gitlab uses a rev with only 0 to signal branch removal
+      new_revision = nil if new_revision[/^0+$/]
+      puppet_env = Environmate::EnvironmentManager.env_from_branch(branch)
+
+      unless puppet_env.nil?
+        deployment = Environmate::Deployment.new(user, puppet_env, new_revision, old_revision)
+        deployment.deploy_dynamic
+      end
+
+      content_type :json
+      user.response.to_json
+    end
+
+    #
+    # Deploy hook for static environments
+    #
+    post '/deploy' do
+      data       = JSON.parse(request.body.read)
+      user       = Environmate::User.new
+      puppet_env = data['environment']
+      revision   = data['revision']
+      token      = data['token']
+
+      deployment = Environmate::Deployment.new(user, puppet_env, revision)
+      deployment.deploy_static(token)
+
+      content_type :json
+      user.response.to_json
+    end
+
+  end
+end

data/lib/environmate/command.rb

@@ -0,0 +1,20 @@
+require 'open3'
+
+module Environmate
+  module Command
+
+    def command(cmd)
+      stdout, stderr, status = Open3.capture3(cmd)
+      unless status.success?
+        message = []
+        message << "Command '#{cmd}' failed"
+        message << 'Status:' + status.exitstatus.to_s
+        message << "Stdout:\n" + stdout.strip
+        message << "Stderr:\n" + stderr.strip
+        raise Environmate::DeployError, message.join("\n")
+      end
+      return stdout
+    end
+
+  end
+end

data/lib/environmate/configuration.rb

@@ -0,0 +1,52 @@
+#
+# Simple configuration loader from yaml
+#
+require 'yaml'
+
+module Environmate
+
+  def self.load_configuration(environment, config_file = nil)
+    config_file ||= config_location
+    if config_file.nil?
+      raise "No configuration file was provided"
+    end
+    unless File.exists?(config_file)
+      raise "Configuration file #{config_file} does not exist"
+    end
+    config = YAML.load_file(config_file)[environment]
+    @configuration = config_defaults.merge(config)
+  end
+
+  def self.configuration
+    @configuration
+  end
+
+  private
+
+  def self.config_defaults
+    {
+      'logfile'                     => '/var/log/enviromate.log',
+      'loglevel'                    => 'WARN',
+      'environment_path'            => '/etc/puppetlabs/code/environments',
+      'lockfile_path'               => '/var/run/lock/environmate',
+      'lockfile_options'            => {
+        'timeout' => 300
+      },
+      'master_repository'           => 'http://gitlab.example.com/puppet/control',
+      'master_branch'               => 'origin/master',
+      'master_path'                 => '/etc/puppetlabs/code/environmate',
+      'dynamic_environments_prefix' => 'env/',
+      'static_environments'         => {},
+      'install_modules_command'     => 'librarian-puppet install --destructive',
+      'server_settings'             => {},
+    }
+  end
+
+  def self.config_location
+    [
+      '/etc/environmate.yml',
+      File.expand_path('~/.environmate.yml'),
+    ].find{|c| File.exist?(c)}
+  end
+
+end