etsy-deployinator 1.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: cd2670b5a14fac9a1b92c16876c4d361f5d6f49a
+  data.tar.gz: a23addf61909a7feb8b9f56bdfbaff2faf30fcef
+SHA512:
+  metadata.gz: be256bfe8ee18d7c03d4f92424f37ed98eb6ff2e92d776feafafd351f382bfdb981de5c5766a4a25d8819b4e035b447ba25b19a5c91bf6e17c6e4961eb604b29
+  data.tar.gz: 6f638ef633763f9ac4a4f4a2b097cb508031eae0fb2482e14d654991c42375c2942ef50d85b56f6ca9a01d6c6057042e39b1b26f6c4c9f622807343fead188b2

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in deployinator.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2014 Etsy
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,368 @@
+<pre>
+_________               ______                _____                 _____
+______  /_____ ________ ___  /______ _____  _____(_)_______ ______ ___  /_______ ________
+_  __  / _  _ \___  __ \__  / _  __ \__  / / /__  / __  __ \_  __ `/_  __/_  __ \__  ___/
+/ /_/ /  /  __/__  /_/ /_  /  / /_/ /_  /_/ / _  /  _  / / // /_/ / / /_  / /_/ /_  / 
+\__,_/   \___/ _  .___/ /_/   \____/ _\__, /  /_/   /_/ /_/ \__,_/  \__/  \____/ /_/ 
+                /_/                   /____/             Deploy with style!
+</pre>
+
+Deployinator - Deploy code like Etsy
+====================================
+
+Deployinator is a deployment framework extracted from Etsy. We've been using it since late 2009 / early 2010. This has been revamped into a ruby gem.
+
+**Table of Contents**
+
+- [Stacks](#stacks)
+- [Installation](#installation)
+- [Usage](#usage)
+  - [Example Stack](#example-stack)
+  - [Customizing your stack](#customizing-your-stack)
+  - [Useful helper methods](#useful-helper-methods)
+  - [Plugins](#plugins)
+  - [Template Hooks](#template-hooks)
+- [Hacking on the gem](#hacking-on-the-gem)
+  - [Contributing](#contributing)
+
+## Stacks
+Deployments are grouped by "stacks". You might have a "web" and "search" stack.
+
+Each of those stacks might have different deployment environments, such as "staging" or "production". 
+
+You can map a button to each of these environments,  to create multi-stage pushes within each stack. 
+
+## Installation
+
+This demo assumes you are using bundler to install deployinator. If you aren't
+you can skip the bundler steps.
+
+- Create a directory for your project. `mkdir test_stacks`
+
+- Create a Gemfile for bundler:
+
+```ruby
+    source 'https://rubygems.org'
+    gem 'etsy-deployinator', :git => 'https://github.com/etsy/deployinator.git', :branch => 'master', :require => 'deployinator'
+```
+
+- Install all required gems with bundler:
+
+```sh
+    $ bundle install --path vendor/bundle
+```
+
+- Run the following command to make deployinator gem's rake tasks available to you:
+
+```sh
+    $ echo "require 'deployinator'\nload 'deployinator/tasks/initialize.rake' " > Rakefile
+```
+
+- Create a binstub for the deploy log tailing backend:
+
+```sh
+    bundle install --binstubs deployinator
+```
+
+- Initialize the project directory by running the following command replacing ___Company___ with the name of your company/organization. This must start with a capital letter.
+
+```sh
+    $ bundle exec rake 'deployinator:init[Company]'
+```
+
+- If you are using bundler add the following to the top of the config.ru that
+shipped with deployinator
+```ruby
+    require 'rubygems'
+    require 'bundler'
+    Bundler.require
+```
+
+- Run the tailer as a background service (using whatever init flavor you like)
+
+```sh
+    ./bin/deployinator-tailer.rb &
+```
+
+
+## Usage
+
+
+### Example Stack
+- Use the deployinator rake task to create the stub for your stack. Replace
+___test_stack___ with the name of your stack. This should be all lowercase with
+underscores but if you forget the rake task will convert from camelcase for you. The commands run by the rake tasks are logged to stderr.
+
+```sh
+    $ bundle exec rake 'deployinator:new_stack[test_stack]'
+```
+
+- We need a server to run our Sinatra application. For the purpose of this demo, we will use [shotgun](https://github.com/rtomayko/shotgun). Let's install shotgun into our bundle. Add the following to your Gemfile:
+
+```ruby
+    gem 'shotgun'
+```
+
+- Now update your bundler:
+
+```sh
+    bundle install --path vendor/bundle --no-deployment && bundle install --path vendor/bundle --deployment
+```
+
+- Start the server by running:
+  - The host could be localhost or the dns name (or ip address of the server you are using). You can set any port you want that's not in use using the `-p` flag.
+
+```sh
+    $ bundle exec shotgun --host localhost -p 7777 config.ru
+```
+
+- You will probably want a robust server like apache to handle production traffic. 
+
+- The `config/base.rb` file is the base config for the application. Replace all
+occurences of ___test_stack___ with the name you chose above. Also the example
+below uses a git repository of http://github.com/etsy/deployinator, feel free to
+replace this with your specific repository
+
+```ruby                                                                                                         
+Deployinator.app_context['test_stack_config'] = {
+   :prod_host               => "localhost",
+   :checkout_path           => "/tmp/deployinator_dev/"
+ }
+Deployinator.git_info_for_stack = {
+    :test_stack => {
+        :user => "etsy",
+        :repository => "deployinator"
+    }
+}
+```
+
+- Edit the stacks/test_stack.rb file to look like this (adding git version bumping and checkout)
+
+```ruby
+require 'helpers/test_stack'
+module Deployinator
+  module Stacks
+    class TestStackDeploy < Deployinator::Deploy
+        include Deployinator::Helpers::TestStackHelpers,
+            Deployinator::Helpers::GitHelpers
+
+      def test_stack_production(options={})
+        # save old version for announcement
+        old_version = test_stack_production_version
+
+        # Clone and update copy of git repository
+        git_freshen_or_clone(stack, "ssh #{Deployinator.app_context['test_stack_info'][:prod_host]}", stack_config[:checkout_path], "master")
+
+        # bump version
+        version = git_bump_version(stack, checkout_path, "ssh #{Deployinator.app_context['test_stack_info'][:prod_host]}", checkout_path)
+
+        # Write the sha1s of the different versions out to the logs for safe keeping.
+        log_and_stream "Updating application to #{version} from #{old_version}"
+
+        # log the deploy
+        log_and_shout :old_build => get_build(old_version), :build => get_build(version)
+      end
+    end
+  end
+end
+```
+
+- Next, edit the helpers/test_stack.rb file. You can delete the test_stack_head_build function since you are using the GitHelpers and that is automatically taken care of for you. Here is the final version:
+
+```ruby
+module Deployinator
+  module Helpers
+    module TestStackHelpers
+      def test_stack_production_version
+        %x{ssh #{Deployinator.app_context["test_stack_config"][:prod_host]} cat #{Deployinator.app_context["test_stack_config"][:checkout_path]}/#{stack}/version.txt}
+      end
+    end
+  end
+end
+```
+
+- Create the directory that will contain the checkout if it doesn't exist already
+(defined in `config/base.rb`)
+
+- Load up deployinator and deploy your stack!
+
+### Customizing your stack
+
+A stack can be customized so that you have flexibility over the different environments within it (which correspond to buttons) and the methods that correspond to each button press.
+
+By default, you will see a button called "deploy _stackname_" where _stackname_ is the stack defined in the rake command. In your helpers file, you can add a function called _stackname_\_environments that returns an array of hashes.  Each hash will correspond to a new environment, or button. For example if your stack is called web, you can define a function like so in helpers/web.rb to define qa and production environments within your web stack:
+
+      def web_environments
+        [
+          {
+            :name            => "qa",
+            :method          => "qa_rsync",
+            :current_version => proc{send(:qa_version)},
+            :current_build   => proc{send(:current_qa_build)},
+            :next_build      => proc{send(:next_qa_build)}
+          },
+          {
+            :name            => "production",
+            :method          => "prod_rsync",
+            :current_version => proc{send(:prod_version)},
+            :current_build   => proc{send(:current_prod_build)},
+            :next_build      => proc{send(:next_prod_build)}
+          }
+        ]
+      end
+
+The keys of each hash describe what you will be pushing for that environment:
+
+* __:name__ - name of the environment
+* __:method__ - method name (string) that gets invoked when you press the button (this is defined in the stack)
+* __:current_version__ - method that returns the version that is currently deployed in this environment (defined in the helper)
+* __:current_build__ - method that returns the build that is currently deployed (usually inferred from the version, also defined in the helper)
+* __:next_build__ - method that returns the next build that is about to be deployed (defined in the helper)
+
+### Useful helper methods
+There are a few helpers built in that you can use after creating a new stack to assist you
+
+#### run_cmd
+
+Shell out to run a command line program.
+Includes timing information streams and logs the output of the command.
+
+For example you could wrap your capistrano deploy:
+
+    run_cmd %Q{cap deploy}
+
+
+#### log_and_stream 
+
+Output information to the log file, and the streaming output handler.
+The real time output console renders HTML so you should use markup here.
+
+    log_and_stream "starting deploy<br>"
+
+#### log_and_shout
+
+Output an announcement message with build related information.  
+Also includes hooks for Email.
+
+    log_and_shout({
+        :old_build  => old_build,
+        :build      => build,
+        :send_email => true
+    });
+
+The supported keys for log_and_shout are:
+
+* __:env__ - the environment that is being pushed
+* __:user__ - the user that pushed
+* __:start__ - the start time of the push (if provided the command will log timing output)
+* __:end__ - the end time of the push (defaults to "now")
+* __:old_build__ - the existing version to be replaced
+* __:build__ - the new version to be pushed
+* __:send_email__ - true if you want it to email the announcement (make sure to define settings in config)
+
+### Plugins
+Deployinator provides various entry points to execute plugins without having to
+modify the gem code. Here is a list of current pluggable events:
+
+- __:logout_url__ for defining your own authentications logout url
+- __:deploy_start__ for any actions to be performed at the start of a deploy
+- __:deploy_end__ for any actions to be performed at the end of a deploy
+- __:deploy_error__ for any actions to be performed when an error occurs during a
+deploy
+- __:run_command_start__ for any actions to be performed at the start of a run_cmd
+call
+- __:run_command_end__ for any actions to be performed at the end of a run_cmd call
+- __:run_command_error__ for any actions to be performed when an error occurs during a
+run_cmd call
+- __:timeout__ for any actions to be performed when a with_timeout calls times out
+- __:announce__ for any actions to be performed when announcing a deploy (IRC
+        integration)
+- __:diff__ for generating diff links
+- __:timing_log__ for sending timing log information to anywhere besides the log
+file (Graphite for example)
+- __:auth__ for handling auth however you please
+
+To create a plugin simply create a new class (example from our code) that
+defined a run method taking event and state. __event__ is a symbol from the
+table above and __state__ is a hash of state data which varies from event to
+event
+
+```ruby
+require 'deployinator/plugin'
+require 'helpers/etsy'
+require 'deployinator/helpers'
+
+module Deployinator
+  class GraphitePlugin < Plugin
+    include Deployinator::Helpers::EtsyHelpers,
+      Deployinator::Helpers
+
+    def run(event, state)
+      case event
+      when :run_command_end
+        unless state[:timing_metric].nil?
+          graphite_timing "deploylong.#{state[:stack]}.#{state[:timing_metric]}", "#{state[:time]}", state[:start_time].to_i
+        end
+      when :timing_log
+        graphite_timing("deploylong.#{state[:stack]}.#{state[:type]}", "#{state[:duration]}", state[:timestamp])
+      end
+      return nil
+    end
+  end
+end
+```
+
+Then simply require your plugin in `lib/app.rb` and add it to your
+`config/base.rb` like this:
+
+```ruby
+Deployinator.global_plugins = []
+Deployinator.global_plugins << "GraphitePlugin"
+```
+
+You can also configure plugins to only apply to a single stack like this:
+
+```ruby
+Deployinator.stack_plugins = {}
+Deployinator.stack_plugins["test_stack"] << "TestStackPlugin"
+```
+
+### Template Hooks
+Since the main layout page is contained within the gem, there are tags provided
+to allow you to add things to it in the header and body. List of points:
+
+- __tailer_loading_message__ To customize the default deploy tailer loading
+message
+- __additional_header_html__ Additional html to add to the header
+- __additional_body_top_html__ Additional html to add to the top of the body
+- __additional_body_bottom_html__ Additional html to add to the bottom of the body
+
+To set these simple override the methods in your view class. For example:
+
+```ruby
+     def additional_bottom_body_html
+       '<script src="/js/check_push_status.js"></script>'
+     end
+```
+
+This can be done on a global layout that extends the gem's default layout or on
+a stack by stack basis in their own view.
+
+## Hacking on the gem
+If you find issues with the gem, or would like to play around with it, you can check it out from git and start hacking on it. 
+First tell bundler to use your local copy instead by running:
+```sh
+    $ bundle config local.deployinator /path/to/DeployinatorGem
+```
+Next, on every code change, you can install from the checked out gem by running (you will want to make commits to the gem to update the sha in the Gemfile.lock)
+```sh
+    $ bundle install --no-deployment && bundle install --deployment
+```
+
+### Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1,16 @@
+
+require 'rake/testtask'
+require 'rdoc/task'
+require 'bundler/gem_tasks'
+
+#
+# Helpers
+#
+
+def command?(command)
+  system("type #{command} &> /dev/null")
+end
+
+task :default => 'deployinator:test:unit'
+
+load 'deployinator/tasks/tests.rake'

data/bin/deployinator-tailer.rb

@@ -0,0 +1,78 @@
+#!/bin/env ruby
+
+require "deployinator"
+require "deployinator/base"
+require "deployinator/helpers"
+require 'deployinator/stack-tail'
+require 'deployinator/helpers/stack-tail'
+
+# Note: If you change the protocol of how this communicates with the front end,
+# please also update the version located in helpers/stack-tail.rb
+EM.run do
+  @@tailers = Hash.new
+  @@channels = Hash.new
+  @@channels_count = Hash.new
+
+  # reload the list of stacks from Deployinator without affecting
+  # existing channels. does not remove stacks that no longer exist`
+  def refresh_stacks
+    Deployinator.get_stacks.each do |stack|
+      unless @@channels.key?(stack)
+        @@channels[stack] = EM::Channel.new
+        @@channels_count[stack] = 0
+      end
+    end
+  end
+
+  # refresh the available stacks when we receive a HUP signal
+  Signal.trap('HUP')  { refresh_stacks }
+  refresh_stacks
+
+  EM::WebSocket.start(:host => "0.0.0.0", :port => Deployinator.stack_tailer_port) do |ws|
+
+    # This attempts to attach a tailer to the stack symlink. If it doesn't
+    # exist yet it will attempt next tick of the EventMachine
+    def attach_tailer(stack)
+      if @@channels.key?(stack) 
+        tailer = Tailer.stack_tail(stack, @@channels[stack], @@channels_count[stack])
+        if tailer
+          @@tailers[stack] = tailer
+        else
+          EM.next_tick do
+            attach_tailer(stack)
+          end
+        end
+      end
+    end
+
+    ws.onopen do |handshake|
+      session_id = false
+      stack = false
+      ws.send(Deployinator::Helpers::StackTailHelpers::get_stack_tail_version())
+
+      ws.onclose do
+        if stack && @@channels.key?(stack)
+          @@channels[stack].unsubscribe(session_id)
+          @@channels_count[stack] -= 1
+          if @@channels_count[stack] == 0
+            @@tailers[stack].close
+            @@tailers.delete(stack)
+          end
+        end
+      end
+
+      ws.onmessage do |msg|
+        stack = msg
+        if @@channels.key?(stack)
+          unless @@tailers.key?(stack)
+            attach_tailer(stack)
+          end
+          @@channels_count[stack] += 1
+          session_id = @@channels[stack].subscribe do |data|
+            ws.send(data)
+          end
+        end
+      end
+    end
+  end
+end