sshkit 0.0.1

data/README.md

@@ -0,0 +1,181 @@
+![SSHKit Logo](/Users/leehambley/Projects/deployrb-deploy/assets/images/logo.png)
+
+
+**SSHKit** is a toolkit for running commands in a structured way on one or
+more servers.
+
+## How might it work?
+
+The typical use-case looks something like this:
+
+    require 'sshkit/dsl'
+
+    on %w{1.example.com 2.example.com}, in: :sequence, wait: 5 do
+      within "/opt/sites/example.com" do
+        as :deploy  do
+          with rails_env: :production do
+            rake   "assets:precompile"
+            runner "S3::Sync.notify"
+          end
+        end
+      end
+    end
+
+One will notice that it's quite low level, but exposes a convenient API, the
+`as()`/`within()`/`with()` are nestable in any order, repeatable, and stackable.
+
+When used inside a block in this way, `as()` and `within()` will guard
+the block they are given with a check.
+
+In the case of `within()`, an error-raising check will be made that the directory
+exists; for `as()` a simple call to `sudo su -<user> whoami` wrapped in a check for
+success, raising an error if unsuccessful.
+
+The directory check is implemented like this:
+
+    if test ! -d <directory>; then echo "Directory doesn't exist" 2>&1; false; fi
+
+And the user switching test implemented like this:
+
+    if ! sudo su <user> -c whoami > /dev/null; then echo "Can't switch user" 2>&1; false; fi
+
+According to the defaults, any command that exits with a status other than 0
+raises an error (this can be changed). The body of the message is whatever was
+written to *stdout* by the process. The `1>&2` redirects the standard output
+of echo to the standard error channel, so that it's available as the body of
+the raised error.
+
+Helpers such as `runner()` and `rake()` which expand to `execute(:rails, "runner", ...)` and
+`execute(:rake, ...)` are convenience helpers for Ruby, and Rails based apps.
+
+## Parallel
+
+Notice on the `on()` call the `in: :sequence` option, the following will do
+what you might expect:
+
+    on(in: :parallel, limit: 2) { ...}
+    on(in: :sequence, wait: 5) { ... }
+    on(in: :groups, limit: 2, wait: 5) { ... }
+
+The default is to run `in: :parallel` with no limit, if you have 400 servers,
+this might be a problem, and you might better look at changing that to run in
+groups, or sequence.
+
+Groups were designed in this case to relieve problems (mass Git checkouts)
+where you rely on a contested resource that you don't want to DDOS by hitting
+it too hard.
+
+Sequential runs were intended to be used for rolling restarts, amongst other
+similar use-cases.
+
+## Synchronisation
+
+The `on()` block is the unit of synchronisation, one `on()` block will wait
+for all servers to complete before it returns.
+
+For example:
+
+    all_servers = %w{one.example.com two.example.com three.example.com}
+    site_dir    = '/opt/sites/example.com'
+
+    # Let's simulate a backup task, assuming that some servers take longer
+    # then others to complete
+    on servers do |host|
+      in site_dir do
+        execute :tar, '-czf', "backup-#{host.hostname}.tar.gz", 'current'
+        # Will run: "/usr/bin/env tar -czf backup-one.example.com.tar.gz current"
+      end
+    end
+
+    # Now we can do something with those backups, safe in the knowledge that
+    # they will all exist (all tar commands exited with a success status, or
+    # that we will have raised an exception if one of them failed.
+    on servers do |host|
+      in site_dir do
+        backup_filename = "backup-#{host.hostname}.tar.gz"
+        target_filename = "backups/#{Time.now.utc.iso8601}/#{host.hostname}.tar.gz"
+        puts capture(:s3cmd, 'put', backup_filename, target_filename)
+      end
+    end
+
+## The Command Map
+
+It's often a problem that programatic SSH sessions don't share the same environmental
+variables as sessions that are started interactively.
+
+This problem often comes when calling out to executables, expected to be on
+the `$PATH` which, under conditions without dotfiles or other environmental
+configuration are not where they are expected to be.
+
+To try and solve this there is the `with()` helper which takes a hash of variables and makes them
+available to the environment.
+
+    with path: '/usr/local/bin/rbenv/shims:$PATH' do
+      execute :ruby, '--version'
+    end
+
+Will execute:
+
+    ( PATH=/usr/local/bin/rbenv/shims:$PATH /usr/bin/env ruby --version )
+
+**Often more preferable is to use the *command map*.**
+
+The *command map* is used by default when instantiating a *Command* object
+
+The *command map* exists on the configuration object, and in principle is
+quite simple, it's a *Hash* structure with a default key factory block
+specified, for example:
+
+    puts SSHKit.config.command_map[:ruby]
+    # => /usr/bin/env ruby
+
+The `/usr/bin/env` prefix is applied to all commands, to make clear that the
+environment is being deferred to to make the decision, this is what happens
+anyway when one would simply attempt to execute `ruby`, however by making it
+explicit, it was hoped that it might lead people to explore the documentation.
+
+One can override the hash map for individual commands:
+
+    SSHKit.config.command_map[:rake] = "/usr/local/rbenv/shims/rake"
+    puts SSHKit.config.command_map[:rake]
+    # => /usr/local/rbenv/shims/rake
+
+One can also override the command map completely, this may not be wise, but it
+would be possible, for example:
+
+    SSHKit.config.command_map = Hash.new do |hash, command|
+      hash[command] = "/usr/local/rbenv/shims/#{command}"
+    end
+
+This would effectively make it impossible to call any commands which didn't
+provide an executable in that directory, but in some cases that might be
+desirable.
+
+*Note:* All keys should be symbolised, as the *Command* object will symbolize it's
+first argument before attempting to find it in the *command map*.
+
+## Output Handling
+
+The output handling comprises two objects, first is the output itself, by
+default this is *$stdout*, but can be any object responding to a
+*StringIO*-like interface. The second part is the *formatter*.
+
+The *formatter* and *output* have a strange relationship:
+
+    SSHKit.config.output = SSHKit.config.formatter.new($stdout)
+
+The *formatter* will typically delegate all calls to the *output*, depending
+on it's implementation it will almost certainly override the implementation of
+`write()` (alias `<<()`) and query the objects it receives to determine what
+should be printed.
+
+
+## Known Issues
+
+* No handling of slow / timed out connections
+* No handling ot slow / hung remote commands
+* No built-in way to background() something (execute and background the
+  process)
+* No environment handling
+* No arbitrary `Host` properties
+

data/Rakefile

@@ -0,0 +1,37 @@
+#!/usr/bin/env rake
+require "bundler/gem_tasks"
+require 'debugger'
+require 'rake/testtask'
+
+namespace :test do
+
+  Rake::TestTask.new(:units) do |t|
+    t.libs << "test"
+    t.test_files = FileList['test/unit/**/test*.rb']
+  end
+
+  Rake::TestTask.new(:functional) do |t|
+    t.libs << "test"
+    t.test_files = FileList['test/functional/**/test*.rb']
+  end
+
+  Rake::TestTask.new(:integration) do |t|
+    t.libs << "test"
+    t.test_files = FileList['test/integration/**/test*.rb']
+  end
+
+  task :test do
+    Rake::Task['test:units'].execute
+    Rake::Task['test:integration'].execute
+    Rake::Task['test:functional'].execute unless ENV['TRAVIS']
+  end
+
+  task default: :test
+
+end
+
+task :default => 'test:default'
+
+Rake::Task["test:functional"].enhance do
+  warn "Remember there are still some VMs running, kill them with `vagrant halt` if you are finished using them."
+end

data/Vagrantfile

@@ -0,0 +1,18 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+Vagrant::Config.run do |config|
+
+  config.vm.define :one do |one|
+    one.vm.box = "lucid32"
+  end
+
+  config.vm.define :two do |one|
+    one.vm.box = "lucid32"
+  end
+
+  config.vm.define :three do |one|
+    one.vm.box = "lucid32"
+  end
+
+end

data/example.rb

@@ -0,0 +1,70 @@
+#!/usr/bin/env ruby
+
+# Ruby 1.9 doesn't include the current
+# working directory on the load path.
+$: << Dir.pwd + '/lib/'
+
+# Automatically sucks in the `sshkit`
+# files so that you don't need to.
+require 'sshkit/dsl'
+require 'forwardable'
+require 'term/ansicolor'
+
+directory = '/opt/sites/web_application'
+hosts     = SSHKit::Host.new("root@example.com")
+
+#
+# Custom output formatter!
+#
+class ColorizedFormatter < StringIO
+  extend Forwardable
+  attr_reader :original_output
+  def_delegators :@original_output, :read, :rewind
+
+  def initialize(oio)
+    @original_output = oio
+  end
+
+  def write(obj)
+    if obj.is_a? SSHKit::Command
+      unless obj.started?
+        original_output << "[#{c.green(obj.uuid)}] Running #{c.yellow(c.bold(String(obj)))} on #{c.yellow(obj.host.to_s)}\n"
+      end
+      if obj.complete? && !obj.stdout.empty?
+        obj.stdout.lines.each do |line|
+          original_output << c.green("\t" + line)
+        end
+      end
+      if obj.complete? && !obj.stderr.empty?
+        obj.stderr.lines.each do |line|
+          original_output << c.red("\t" + line)
+        end
+      end
+      if obj.finished?
+        original_output << "[#{c.green(obj.uuid)}] Finished in #{sprintf('%5.3f seconds', obj.runtime)} command #{c.bold { obj.failure? ? c.red('failed') : c.green('successful') }}.\n"
+      end
+    else
+      original_output << c.black(c.on_yellow("Output formatter doesn't know how to handle #{obj.inspect}\n"))
+    end
+  end
+  private
+  def c
+    @c ||= Term::ANSIColor
+  end
+end
+
+SSHKit.config.output = ColorizedFormatter.new($stdout)
+
+on hosts do |host|
+  target = '/opt/rack-rack-repository'
+  if host.hostname =~ /seven/
+    target = '/var/rack-rack-repository'
+  end
+  if execute(:test, "-d #{target}")
+    within target do
+      execute :git, :pull
+    end
+  else
+    execute :git, :clone, 'git://github.com/rack/rack.git', target
+  end
+end

data/lib/core_ext/array.rb

@@ -0,0 +1,5 @@
+class Array
+  def extract_options!
+    last.is_a?(::Hash) ? pop : {}
+  end
+end

data/lib/core_ext/hash.rb

@@ -0,0 +1,11 @@
+class Hash
+  def symbolize_keys
+    inject({}) do |options, (key, value)|
+      options[(key.to_sym rescue key) || key] = value
+      options
+    end
+  end
+  def symbolize_keys!
+    self.replace(self.symbolize_keys)
+  end
+end

data/lib/sshkit/all.rb

@@ -0,0 +1,13 @@
+require_relative '../core_ext/array'
+require_relative '../core_ext/hash'
+
+require_relative 'dsl'
+require_relative 'host'
+
+require_relative 'command'
+require_relative 'configuration'
+require_relative 'connection_manager'
+
+require_relative 'backends/abstract'
+require_relative 'backends/printer'
+require_relative 'backends/netssh'

data/lib/sshkit/backends/abstract.rb

@@ -0,0 +1,83 @@
+module SSHKit
+  module Backend
+
+    MethodUnavailableError = Class.new(RuntimeError)
+
+    class Abstract
+
+      attr_reader :host
+
+      def run
+        # Nothing to do
+      end
+
+      def initialize(host, &block)
+        raise "Must pass a Host object" unless host.is_a? Host
+        @host  = host
+        @block = block
+      end
+
+      def make(commands=[])
+        raise MethodUnavailableError
+      end
+
+      def rake(commands=[])
+        raise MethodUnavailableError
+      end
+
+      def execute(command, args=[])
+        raise MethodUnavailableError
+      end
+
+      def capture(command, args=[])
+        raise MethodUnavailableError
+      end
+
+      def within(directory, &block)
+        (@pwd ||= []).push directory.to_s
+        execute <<-EOTEST
+          if test ! -d #{File.join(@pwd)}
+            then echo "Directory does not exist '#{File.join(@pwd)}'" 1>&2
+            false
+          fi
+        EOTEST
+        yield
+      ensure
+        @pwd.pop
+      end
+
+      def with(environment, &block)
+        @_env = (@env ||= {})
+        @env = @_env.merge environment
+        yield
+      ensure
+        @env = @_env
+        remove_instance_variable(:@_env)
+      end
+
+      def as(user, &block)
+        @user = user
+        execute <<-EOTEST
+          if ! sudo su #{user} -c whoami > /dev/null
+            then echo "You cannot switch to user '#{user}' using sudo, please check the sudoers file" 1>&2
+            false
+          fi
+        EOTEST
+        yield
+      ensure
+        remove_instance_variable(:@user)
+      end
+
+      private
+
+      def command(*args)
+        SSHKit::Command.new(*args, in: @pwd.nil? ? nil : File.join(@pwd), env: @env, host: @host, user: @user)
+      end
+
+      def connection
+        raise "No Connection Pool Implementation"
+      end
+
+    end
+  end
+end

data/lib/sshkit/backends/netssh.rb

@@ -0,0 +1,82 @@
+require 'net/ssh'
+
+module SSHKit
+  module Backend
+
+    class Netssh < Printer
+
+      include SSHKit::CommandHelper
+
+      def run
+        instance_exec(host, &@block)
+      end
+
+      def execute(*args)
+        _execute(*args).success?
+      end
+
+      def capture(*args)
+        _execute(*args).stdout.strip
+      end
+
+      private
+
+      def _execute(*args)
+        command(*args).tap do |cmd|
+          output << cmd
+          cmd.started = true
+          ssh.open_channel do |chan|
+            chan.exec cmd.to_s do |ch, success|
+              chan.on_data do |ch, data|
+                cmd.stdout += data
+                output << cmd
+              end
+              chan.on_extended_data do |ch, type, data|
+                cmd.stderr += data
+                output << cmd
+              end
+              chan.on_request("exit-status") do |ch, data|
+                exit_status = data.read_long
+                cmd.exit_status = exit_status
+                output << cmd
+              end
+              #chan.on_request("exit-signal") do |ch, data|
+              #  # TODO: This gets called if the program is killed by a signal
+              #  # might also be a worthwhile thing to report
+              #  exit_signal = data.read_string.to_i
+              #  warn ">>> " + exit_signal.inspect
+              #  output << cmd
+              #end
+              chan.on_open_failed do |ch|
+                # TODO: What do do here?
+                # I think we should raise something
+              end
+              chan.on_process do |ch|
+                # TODO: I don't know if this is useful
+              end
+              chan.on_eof do |ch|
+                # TODO: chan sends EOF before the exit status has been
+                # writtend
+              end
+            end
+            chan.wait
+          end
+          ssh.loop
+        end
+      end
+
+      def ssh
+        @ssh ||= begin
+          Net::SSH.start(
+            host.hostname,
+            host.username,
+            port: host.port,
+            password: host.password,
+          )
+        end
+      end
+
+    end
+  end
+
+end

data/lib/sshkit/backends/printer.rb

@@ -0,0 +1,28 @@
+module SSHKit
+  module Backend
+
+    class Printer < Abstract
+
+      include SSHKit::CommandHelper
+
+      def run
+        instance_exec(host, &@block)
+      end
+
+      def execute(*args)
+        output << command(*args)
+      end
+
+      def capture(command, args=[])
+        raise MethodUnavailableError
+      end
+
+      private
+
+      def output
+        SSHKit.config.output
+      end
+
+    end
+  end
+end

data/lib/sshkit/command.rb

@@ -0,0 +1,153 @@
+require 'shellwords'
+require 'digest/sha1'
+require 'securerandom'
+
+# @author Lee Hambley
+module SSHKit
+
+  # @author Lee Hambley
+  module CommandHelper
+
+    def rake(tasks=[])
+      execute :rake, tasks
+    end
+
+    def make(tasks=[])
+      execute :make, tasks
+    end
+
+    def execute(command, args=[])
+      Command.new(command, args)
+    end
+
+    private
+
+      def map(command)
+        SSHKit.config.command_map[command.to_sym]
+      end
+
+  end
+
+  # @author Lee Hambley
+  class Command
+
+    attr_reader :command, :args, :options, :started_at, :started, :exit_status
+
+    attr_accessor :stdout, :stderr
+
+    # Initialize a new Command object
+    #
+    # @param  [Array] A list of arguments, the first is considered to be the
+    # command name, with optional variadaric args
+    # @return [Command] An un-started command object with no exit staus, and
+    # nothing in stdin or stdout
+    #
+    def initialize(*args)
+      raise ArgumentError, "May not pass no arguments to Command.new" if args.empty?
+      @options = args.extract_options!
+      @command = args.shift.to_s.strip.to_sym
+      @args    = args
+      @options.symbolize_keys!
+      sanitize_command!
+      @stdout, @stderr = String.new, String.new
+    end
+
+    def complete?
+      !exit_status.nil?
+    end
+    alias :finished? :complete?
+
+    def started?
+      started
+    end
+
+    def started=(new_started)
+      @started_at = Time.now
+      @started = new_started
+    end
+
+    def uuid
+      @uuid ||= Digest::SHA1.hexdigest(SecureRandom.random_bytes(10))[0..7]
+    end
+
+    def success?
+      exit_status.nil? ? false : exit_status.to_i == 0
+    end
+    alias :successful? :success?
+
+    def failure?
+      exit_status.to_i > 0
+    end
+    alias :failed? :failure?
+
+    def exit_status=(new_exit_status)
+      @finished_at = Time.now
+      @exit_status = new_exit_status
+    end
+
+    def runtime
+      return nil unless complete?
+      @finished_at - @started_at
+    end
+
+    def to_hash
+      {
+        command:     command,
+        args:        args,
+        options:     options,
+        exit_status: exit_status,
+        stdout:      stdout,
+        stderr:      stderr
+      }
+    end
+
+    def host
+      options[:host]
+    end
+
+    def to_s
+      return command.to_s if command.match /\s/
+      String.new.tap do |cs|
+        if options[:in]
+          cs << sprintf("cd %s && ", options[:in])
+        end
+        if options[:env]
+          cs << '( '
+          options[:env].each do |k,v|
+            cs << k.to_s.upcase
+            cs << "="
+            cs << v.to_s.shellescape
+          end
+          cs << ' '
+        end
+        if options[:user]
+          cs << "sudo su #{options[:user]} -c "
+        end
+        cs << SSHKit.config.command_map[command.to_sym]
+        if args.any?
+          cs << ' '
+          cs << args.join(' ')
+        end
+        if options[:env]
+          cs << ' )'
+        end
+      end
+    end
+
+    private
+
+      def sanitize_command!
+        command.to_s.strip!
+        if command.to_s.match("\n")
+          @command = String.new.tap do |cs|
+            command.to_s.lines.each do |line|
+              cs << line.strip
+              cs << '; ' unless line == command.to_s.lines.to_a.last
+            end
+          end
+        end
+      end
+
+  end
+
+end