sshotgun 1.0.4

data/README.rdoc

@@ -0,0 +1,523 @@
+== SSHotgun - Utility library for writing scripts to administer and provision multiple servers or server farms.
+
+The SSHotgun project homepage is http://sshotgun.rubyforge.org
+
+To get help, post your questions on forums on RubyForge at http://rubyforge.org/forum/?group_id=6867
+
+The author of SSHotgun is Vick Perry (vick.perry @nospam@ gmail.com)
+
+== DESCRIPTION:
+
+SShotgun is a utility library for writing Unix/Linux server management and
+provisioning scripts in Ruby. I use it for remotely managing machines in server
+clusters/farms.  SShotgun uses your locally installed SSH client and expects
+that you have your SSH public key installed on the servers.  
+
+== FEATURES:
+
+* Uses OpenSSH compatible SSH client and keys already installed on your local machine
+* Optional use of a gateway machine for access to servers behind a firewall
+* Logs in to remote machines via your public key already installed on all remote machines
+* Automatically logs to the console and to a timestamped log file
+* stdout and stderr from each remote machine is displayed and logged
+* Can run sudo commands (your sudo password must be same on all remote machines)
+* Multithreaded with configurable thread pool to service multiple machines simultaneously
+* At end of processing run, a status report is displayed indicating successes, failures, timeouts, incompletes, etc.
+
+== EXAMPLE:
+
+You don't need to know much about Ruby to write powerful SSHotgun scripts. Just
+copy the sample script below and edit the hostnames and the doProcessing method
+as necessary.
+
+The steps for writing a SSHotgun Ruby script are:
+
+1. Create an array of fully qualified hostnames (fqdn).
+2. Write your own custom processing class and doProcess method that will be
+   invoked on all remote servers. You can copy the example for this.
+3. Instantiate a SSHotgun object in the script.
+4. Run the script. A Ruby script can be invoked two different ways:
+
+Run Ruby and specify the script
+
+  ruby ./myrubyscript.rb
+
+or
+
+include this line as the first line of the script
+
+ #!/usr/bin/env ruby
+
+then make the script executable 
+
+ chmod 755 myrubyscript.rb 
+
+and run it directly from the command line 
+
+  ./myrubyscript.rb
+
+This is a simple example of a SSHotgun Ruby script:
+
+  #!/usr/bin/env ruby
+
+  # Required libraries
+  require "highline/import"
+  require 'sshotgun'
+
+  # Create an array to contain hosts for processing
+  hostlist = Array::new()
+
+  # Create a host definition
+  host = HostDefinition.new("localhost")
+
+  # Add the host definition to the array of hosts
+  hostlist << host
+
+  # Create and add more hosts...
+  host = HostDefinition.new("another.host.junk")
+  hostlist << host
+
+  # Create your own custom processing class that subclasses 
+  # BaseProcessingClass.  In this class, define your own doProcessing method 
+  # where you write commands to execute on each remote server. 
+  class MyProcessingClass < BaseProcessingClass
+    def doProcessing
+      # Execute a command on all remote servers. The run command has a return
+      # object from which you can obtain the output and exit status. See the
+      # user guide for more details.
+      run "ls -aF"
+
+      # There are other sshotgun commands available for copying files, running
+      # sudo, invoking local commands, etc.
+    end
+  end
+
+  # Create an instance of SSHotgun and pass it the list of hosts and your
+  # custom processing class.
+  sshotgun = SSHotgun.new(hostlist, MyProcessingClass)
+
+  # Start processing. The start call should be the very last line in a SSHotgun
+  # script.
+  sshotgun.start
+
+== REQUIREMENTS:
+
+* You must have an OpenSSH-compatible SSH client installed on your local machine.
+* You must have correctly installed your SSH public keys on all remote servers that you access. 
+* For most advanced administrative tasks, you'll also need sudo privleges on the remote servers.
+* You'll need write permissions in your present working directory (pwd) since SSHotgun will need to write its logs.
+
+== INSTALL:
+
+Use gem to install SSHotgun. Note that you probably need to run this command with sudo or as root.
+
+  gem install sshotgun
+
+== USER GUIDE:
+
+Here are snippets of SSHotgun Ruby code to illustrate SSHotgun's capabilities.
+
+At a minimum, a host definition contains the fully qualified domain name (fqdn)
+of a machine. As a best practice, DNS aliases are not generally referenced -
+only the fqdn. There are also several optional fields in a host definition that
+can be used for more selectivity when running a script. For example, you may
+wish to do some additional configuration for those machines categorized as
+"ldap" or "proxy"
+
+A host definition has the following fields: hostname, category, os, desc
+
+* hostname is the primary hostname or fqdn and NOT an alias.
+* category is a freeform identifier for the type of the host. e.g. "wiki", "ldap" (optional)
+* os is a freeform string indicating os name + server/workstation + version.  e.g. ubuntu_server_7.10 (optional)
+* desc is a one sentence string describing main purpose of the host (optional)
+
+An example of a complete host definition is:
+
+  host = HostDefinition.new("localhost", "test", "ubuntu_server_7.10", "test vm")
+  hostlist << host
+
+You can share a list of hosts among several SSHotgun scripts. To do this,
+create a Ruby module (not a class) in a separate file. The module will contain
+a method that returns an array of host definitions. The module is referenced by
+any script that needs it.
+
+  # When you create your own modules, name the filename and module name with a similar name.  
+  # Note: This module filename is 'listofhosts.rb' - this is the reference in the require statement in the calling script.
+  module ListOfHosts
+
+    # Create a method named <modulename>.getHostlist
+    def ListOfHosts.getHostlist
+      # Define, load and return the hostlist array
+      hostlist = []
+
+      host = HostDefinition.new("localhost", "test", "ubuntu_server_7.10", "test vm")
+      hostlist << host
+
+      host = HostDefinition.new("other.host.junk", "test", "ubuntu_server_7.10", "test vm")
+      hostlist << host
+
+      # return the array
+      return hostlist
+    end
+
+  end
+
+
+It is a good practice to name the module file name similar or identical to the
+module name. 
+
+Here is a SSHotgun Ruby script that uses the list of hosts:
+
+  #!/usr/bin/env ruby
+
+  # Required libraries
+  require "highline/import"
+  require 'sshotgun'
+
+  # Create array to contain hosts for processing
+  hostlist = Array::new()
+
+  # Read hosts from an external Ruby module.
+  require 'listofhosts'
+
+  # In the listofhosts.rb file, the ListOfHosts getHostlist method returns an
+  # array of hosts. Concatenate all of the multiple arrays into a single array of
+  # hosts.
+  hostlist = hostlist + ListOfHosts.getHostlist
+
+  class MyProcessingClass < BaseProcessingClass
+    def doProcessing
+      run "ls -aF"
+    end
+  end
+
+  sshotgun = SSHotgun.new(hostlist, MyProcessingClass)
+  sshotgun.start
+
+Here's how to prompt for the sudo password (Don't hardcode a password into your
+script). See the advanced example for details.
+
+ sshotgun.sudopassword = ask(">>> Enter your sudo password:  ") { |q| q.echo = "*" } # or q.echo = false
+
+Below is an advanced example that illustrates additional capabilities.
+
+  #!/usr/bin/env ruby
+  
+  # Required libraries
+  require "highline/import"
+  require 'sshotgun'
+  
+  # Create array to contain hosts for processing
+  hostlist = Array::new()
+  host = HostDefinition.new("someserver.junk", "test", "ubuntu_server_7.10", "test vm")
+  hostlist << host
+  
+  # Import a list of host definitions
+  require 'listofhosts'
+  hostlist = hostlist + ListOfHosts.getHostlist
+  
+  class MyProcessingClass < BaseProcessingClass
+  
+    # You can add your own Ruby instance variables. These are useful if you
+    # want to share data across your own custom methods.
+    attr_accessor :myInstanceVariable
+  
+    # You MUST define a doProcessing method. Note that to make the doProcessing
+    # method less cluttered, you can also create your own methods that are
+    # called from within the doProcessing method. You can use your own accessor
+    # variables (see above) for data sharing between methods. The doProcess method
+    # is the only method called by the SSHotgun framework for each host.
+    def doProcessing
+      # Log the beginning of this method. This marker is useful for debugging problems.
+      log "[" + @hostdef.hostname + "] info: " + "Processing started"
+  
+      # Execute a command on all remote servers.  Remote output is always written
+      # to this script's stdout.
+      # run <command>
+      run "ls -aF | grep -i .bash"
+  
+      # The runstatus object is returned from all run and runSudo calls.
+      # runstatus contains information about exit status, stdout and stderr from
+      # the remote server.
+      runstatus = run "ls -aF"
+  
+      # Use the log method to write to the log (the console). Don't use the Ruby
+      # 'puts' method to write to the console because it isn't threadsafe and may
+      # mix and garble simultaneous output by multiple threads.
+      log "[" + @hostdef.hostname + "] info: " + "The exit status of the run command is " + runstatus.exitstatus.to_s
+  
+      # Remote stderr and stdout are captured and stored in the stdoutstr and
+      # stderrstr variables. Learn about Ruby's regular expressions for examining
+      # the output from a remote command.
+      runstatus = run "id rumplestiltskin"
+  
+      # Check for "No such user" contained in stdoutstr. That's what is displayed
+      # when the unix "id" command can't find the specified user. Note that the
+      # the exit status of a failed 'id' command is not equal to zero either (see
+      # below). This is also a Ruby regex comparison.
+      if /No such user/ =~ runstatus.stdoutstr
+        log "I failed to find user rumplestiltskin on this server"
+      end
+  
+      # In some cases a remote command fails and you must stop processing for
+      # that one server. Stop the execution of the doProcessing method by calling
+      # Ruby's "return" statement. Don't call Ruby's "exit" statement because
+      # that will halt all running threads.
+  #    runstatus = run "id rumplestiltskin"
+  #    if runstatus.exitstatus > 0
+  #      log "The exit code was not 0, therefore I failed to find user rumplestiltskin"
+  #      return  # stop processing for this host
+  #    end
+  #    log "You'll never get here...unless you have a user named rumplestiltskin"
+  
+      # The @hostdef instance variable contains the host definition for the
+      # current host being processed. If you set category, os and desc to
+      # meaningful information then you can do fancier branching logic in your
+      # script. The @hostdef.hostname is very useful in log statements. The other
+      # @hostdef fields are useful for treating a category or os differently than
+      # the others.
+      log "The current host is: " + @hostdef.hostname
+      log "The current host's category is: " + @hostdef.category
+      log "The current host's os is: " + @hostdef.os
+      log "The current host's description is: " + @hostdef.desc
+  
+      # Target a specified host. This is a Ruby string comparison.
+      if @hostdef.hostname == "192.168.1.224"
+        log "[" + @hostdef.hostname + "] info: " + "Special processing for this host"
+      end
+  
+      # If necessary, escape any quotes within the command.  that must be passed
+      # through to the remote server. This isn't a great example but you'll need
+      # to do this for tricky shell scripting.
+      run "ls -aF | grep -i \".bash\""
+  
+      # If you are running any runSudo commands then the SSHotgun.sudoPassword
+      # must be set - uncomment the user prompt and setting of the sudoPassword
+      # at the bottom of this script. Don't set up your servers to permit a
+      # password-less sudo - it is a security risk. Watch out, the "runSudo"
+      # command and Ruby in general, are case sensitive.
+      # runSudo <command>
+  #    runSudo "touch /usr/local/bin/mytest.txt"
+  #    run "ls -alF /usr/local/bin/"
+  #    runSudo "rm /usr/local/bin/mytest.txt"
+  
+      # Run a command locally (on your local machine). The runstatus variable is
+      # returned if you need the data - same as run or runSudo.
+      # runLocal <command> 
+  #    runLocal "ls -alF"
+  
+      # Run a sudo command locally. The runstatus variable is returned if you
+      # need data from it. Set a sudoPassword if you are calling runLocalSudo.
+      # runLocalSudo <command> 
+  #    runLocalSudo "ls -alF"
+  
+      # Create a remote file from a string parameter.
+      # createRemoteFile <content string>, <remotefile>
+  #    createRemoteFile "this is your content here", "/home/vickp/testfile.txt"
+  
+      # Create a remote file and set mode
+      # createRemoteFile <content string>, <remotefile>, <mode string>
+  #    createRemoteFile "this is your content here", "/home/vickp/testfile.sh", "0755"
+      
+      # Copy a remote file to the local machine. Note that you should vary the
+      # name of the incoming local file else it will be overwritten when each
+      # host's file is copied.
+      # copyRemoteFileToLocal <remotefile>, <localfile>
+  #    copyRemoteFileToLocal "/home/vickp/testfile.txt", "/home/vickp/temp/" + @hostdef.hostname + "_testfile.txt" 
+      
+      # Copy a remote file to the local machine and set the mode of local file.
+      # copyRemoteFileToLocal <remotefile>, <localfile>, <mode string>
+  #    copyRemoteFileToLocal "/home/vickp/testfile.txt", "/home/vickp/temp/" + @hostdef.hostname + "_testfile.txt", "0644"
+      
+      # Copy a local file to the remote machine.
+      # copyRemoteFileToLocal <localfile>, <remotefile>
+  #    copyLocalFileToRemote "/home/vickp/temp/testfile.txt", "/home/vickp/testfile2.txt"
+      
+      # Copy a local file to the remote machine and set the mode of remote file.
+      # copyRemoteFileToLocal <localfile>, <remotefile>, <mode string>
+  #    copyLocalFileToRemote "/home/vickp/temp/testfile.txt", "/home/vickp/testfile2.txt", "0644"
+  
+      # Other helpful SSHotgun and Ruby tips below
+      # ===========================================
+  
+      # Ruby allows you to concatenate multiple command lines
+      # Note use of shell command separator ';'
+  #    cmd = "ls -alF;"
+  #    cmd << "ls -alF | grep -i .bash;"
+  #    cmd << "ls -alF | grep -i .profile"
+  #    run cmd
+  
+      # Delete a file on the remote server.
+      # Note use of -f with rm command
+  #    run "rm -f /path/to/remote/file"
+  
+      # Call your own methods to unclutter and better organize your doProcess
+      # method code. In most of my SSHotgun scripts the doProcess method is
+      # fairly sparse - most of the work is done is various custom methods that
+      # are called within the doProcess method
+  #    myOwnMethod
+  
+      # Sometimes I either install unsigned packages of my own (shell and ruby
+      # scripts) or need to force the installation of files (overwrite) that
+      # belong to another package. Aptitude doesn't yet support forcing yes for
+      # this sort of thing but apt-get does...
+  #    runSudo "apt-get install -y --force-yes myUntrustedPackage -o DPkg::options::='--force-overwrite'"
+  
+      # Ruby's ENV contains the current user (you)
+      currentUser = ENV["USER"].to_s
+  
+      # For double quoted strings, Ruby will do inline substitution for a
+      # variable enbedded within "#{myvariablehere}". This is useful when you
+      # want a variable substituted inside of double quotes
+  #    createRemoteFile "deb http://debrepo/global ./", "/home/#{currentUser}/sources.list", "0644"
+  
+      # Create a global variable if you need to prompt for additional information
+      # (such as userid, date range, etc) and want to use that information in
+      # your doProcessing method. Global variables begin with '$'. See bottom of
+      # script where the variable is created. Note that "id" returns with an exit
+      # status of 1 if the user does not exist.
+      runstatus = run "id #{$userid}"
+      if runstatus.exitstatus == 0
+        log "[" + @hostdef.hostname + "] info: " + "Found user " + $userid
+      else
+        log "[" + @hostdef.hostname + "] info: " + "DID NOT find user " + $userid
+      end
+  
+      # To pass escaped characters in strings, tell Ruby not to interpolate them
+      # by using single quotes or the %q() delimiter to enclose the escaped
+      # string.  This means that Ruby will not attempt to resolve the escaped
+      # string but merely pass it on to the ssh client.
+  #    run %q("mycommand \"some\ string\ with\ escapes\" more")
+  
+      # Example of how to update a debian/ubuntu sources.list package repository file
+      # Also see use of case statement in the method body.
+  #    installSourcesListAndUpdate
+  
+      # Log the end of this method.
+      log "[" + @hostdef.hostname + "] info: " + "Processing completed"
+    end
+  end
+  
+  # Your own custom methods can be called from within the doProcess method
+  def myOwnMethod
+    log "[" + @hostdef.hostname + "] info: " + "Calling my own method"
+  end
+  
+  # Update a debian/ubuntu /etc/apt/sources.list
+  def installSourcesListAndUpdate
+    # WARNING: This only works if you fill in the category field in your hostdefs.
+    #
+    # Copy production sources.list into your home directory, then copy it into
+    # place. Note this is a two step operation because you can't create the
+    # remote file in a place where you don't have perms. The second step is to
+    # use sudo and copy the file into place.
+    s = ""
+    case @hostdef.category
+    when "production"
+      f = File.new("/usr/local/bin/adminscripts_templates/production_ubuntuserver710i386.sources.list")
+    when "devint"
+      f = File.new("/usr/local/bin/adminscripts_templates/devint_ubuntuserver710i386.sources.list")
+    when "qaint"
+      f = File.new("/usr/local/bin/adminscripts_templates/qaint_ubuntuserver710i386.sources.list")
+    when "staging"
+      f = File.new("/usr/local/bin/adminscripts_templates/staging_ubuntuserver710i386.sources.list")
+    else
+      log "[" + @hostdef.hostname + "] ERROR: " + "Unknown hostdef category = " + @hostdef.category
+      return
+    end
+    # read the file and concatenate it into a single string
+    f.each_line do |line|
+      s << line
+    end
+    createRemoteFile s, "/home/#{$currentUser}/sources.list", "0644"
+    runSudo "cp /home/#{$currentUser}/sources.list /etc/apt/sources.list"
+    run "rm /home/#{$currentUser}/sources.list"
+  
+    # after changing sources.list, update package manager to refresh package list
+    runSudo "aptitude update"
+  end
+  
+  # Create an instance of SSHotgun and pass it the list of hosts and your custom
+  # processing class.
+  sshotgun = SSHotgun.new(hostlist, MyProcessingClass)
+  
+  # If you are calling a sudo command then prompt the user for the sudo password.
+  # I recommend that you NEVER HARDCODE A PASSWORD IN A SCRIPT!  Configure the
+  # ask command to hide the password as the user types it.  Note that the you
+  # must have the highline gem package installed for the "ask" command
+  #sshotgun.sudopassword = ask(">>> Enter your sudo password:  ") { |q| q.echo = "*" } # or q.echo = false
+  
+  # If you need to forward all calls via a gateway ssh server, set it here.  Once
+  # set, all ssh connections go through the gateway machine
+  #sshotgun.gatewayhost = "someserver.somedomain.com"
+  
+  # You can also use the 'ask' command to prompt the user for any other
+  # information. Configure the ask command to display the string as the user
+  # types it. Be sure to test password-less login from your gateway server to
+  # clear out any prompts asking you to verify the authenticity of the remote
+  # host.
+  #gatewayhost = ask(">>> Enter the hostname of the gateway server:  ") { |q| q.echo = true }
+  #sshotgun.gatewayhost = gatewayhost
+  
+  # Create and use a global variable to get additional data into your doProcessing method.
+  $userid = ask(">>> Enter the userid to find:  ") { |q| q.echo = true }
+  
+  # Number of simultaneous processing threads (hosts) to run. In terms of load to
+  # your local machine, each processing thread equates to about one ssh client
+  # running. Default is 50
+  # sshotgun.maxThreads = 30    
+  
+  # A processing thread will be killed after this amount of time. Default is 30 minutes.
+  # sshotgun.threadTimeoutInSeconds = 900
+  
+  # Period for displaying status information about processing threads that
+  # are still running. Default is 30
+  # sshotgun.monitorPollIntervalInSeconds = 20
+  
+  # Any output is also written to a file logger. You can disable the file logging
+  # by setting isFileLogging to false.
+  # sshotgun.isFileLogging = false.
+  
+  # What should the behavior be when a command returns a non-zero exit status? A
+  # non-zero exit status usually indicates failure of the command. Some sshotgun
+  # users wish to immediately stop the processing for that server and stop it's
+  # thread. Other people will check the exit status after each command and decide
+  # whether to continue or return from the processing method.  Note that if
+  # isStopOnNonZeroExit is true then the processing thread exits immediately and
+  # you never have the opportunity to check the exit status in your processing
+  # method code. The default is false (don't stop on non-zero exit).
+  # sshotgun.isStopOnNonZeroExit = true
+  
+  # Start processing. The start call should be the very last line in a SSHotgun
+  # script.
+  sshotgun.start
+  
+== LICENSE:
+
+(Simplified BSD License)
+
+Copyright (c) 2008, Vick Perry
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+* Neither the name of the <ORGANIZATION> nor the names of its contributors may
+  be used to endorse or promote products derived from this software without
+  specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

data/lib/sshotgun.rb

@@ -0,0 +1,475 @@
+#:title: SSHotgun
+#Author::    Vick Perry (vick.perry @nospam@ gmail.com)
+#Copyright:: Copyright (c) 2008, Vick Perry
+#License::   Simplified BSD License.
+#
+require 'open4'
+require 'logger'
+
+# Class to contain information about each server
+#
+class HostDefinition
+  # hostname is the primary hostname or fqdn (NOT an alias). It is better to only refer to the primary DNS entry for a machine or VM.
+  attr_accessor :hostname
+
+  # category is a freeform identifier for the type of the host. e.g. "production", "devint", "qaint", "staging" (optional)
+  attr_accessor :category
+
+  # os is a freeform string indicating os name + server/workstation + version. e.g. ubuntu_server_7.10 (optional)
+  attr_accessor :os
+
+  # desc is a one sentence string describing main purpose of the host (optional)
+  attr_accessor :desc
+
+  # For internal use. True if processing was done for this server.
+  attr_accessor :startedProcess
+
+  # For internal use. True if processing was finished for this server. This means that doProcess exited
+  # normally with no exceptions. 
+  attr_accessor :finishedProcess
+
+  # For internal use. True if one or more commands on a server returned with a non-zero exit status
+  attr_accessor :hadNonZeroExitStatus
+
+  def initialize(hostname, category = "", os = "", desc = "")
+    @hostname = hostname
+    @category = category
+    @os = os
+    @desc = desc
+    @startedProcess = false
+    @finishedProcess = false
+    @hadNonZeroExitStatus = false
+  end
+end
+
+# This class contains the returned information from commands such as run, runSudo, runLocal, runLocalSudo
+#
+class RunStatus
+  # The exit status of the command
+  attr_accessor :exitstatus
+
+  # The stdout stream data from the command stored in a string
+  attr_accessor :stdoutstr
+
+  # The stderr stream data from the command stored in a string
+  attr_accessor :stderrstr
+
+  def initialize
+    @exitstatus = 0
+    @stdoutstr = ""
+    @stderrstr = ""
+  end
+end
+
+# This is the base processing class. Your custom processing class MUST inherit from this base class.
+#
+class BaseProcessingClass
+  # A SSHotgun object
+  attr_accessor :sshotgun
+
+  # The current HostDefinition object (represents the remote server for processing)
+  attr_accessor :hostdef
+
+  # Default constructor. Don't create a constructor in your custom processing
+  # class so that this constructor will be called.
+  #
+  def initialize(sshotgun, hostdef)
+    @sshotgun = sshotgun
+    @hostdef = hostdef
+  end
+
+  # You must override the doProcessing method in your own custom processing class
+  #
+  def doProcessing
+    log "[" + @hostdef.hostname + "] info: " + "The doProcessing method must be created in your custom processing class"
+  end
+
+  # Run a command on a remote server
+  #
+  # cmd is a string containing the command to run on a remote server. e.g. "ls -alF"
+  #
+  # Returns a RunStatus object
+  #
+  def run(cmd)
+    log "[" + @hostdef.hostname + "] run: " + cmd
+    if @sshotgun.gatewayhost
+      cmdline = "ssh -A -t -x " + @sshotgun.gatewayhost + " 'ssh -A -x " +  @hostdef.hostname + " \"" + cmd + "\"'"
+    else
+      cmdline = "ssh -A -t -x " +  @hostdef.hostname + " \"" + cmd + "\""
+    end
+    runLocal cmdline, false
+  end
+
+  # Sudo a command on a remote server
+  #
+  # cmd is a string containing the sudo command to run on a remote server. e.g. "ls -alF"
+  #
+  # Returns a RunStatus object
+  #
+  def runSudo(cmd)
+    log "[" + @hostdef.hostname + "] runSudo: " + cmd
+    if @sshotgun.gatewayhost
+      cmdline = "ssh -A -t -x " + @sshotgun.gatewayhost + " 'ssh -A -x " + @hostdef.hostname + " sudo -S \"" + cmd + "\"'"
+    else
+      cmdline = "ssh -A -t -x " + @hostdef.hostname + " sudo -S \"" + cmd + "\""
+    end
+    runLocal cmdline, true
+  end
+
+  # Copy a remote file to local
+  #
+  # remotefile is the remote file to fetch. e.g. "/usr/local/bin/myscript.sh"
+  #
+  # localfile is the name of the destination local file. e.g. "/home/vickp/temp/myscript.sh"
+  #
+  # modestr is the string containing parameters for the chmod operation.
+  # The string must be in the NNNN or ugo format. e.g. "0755" or "u=rwx,g=rw,o=r"
+  #
+  def copyRemoteFileToLocal(remotefile, localfile, modestr = "0644")
+    log "[" + @hostdef.hostname + "] copyRemoteFileToLocal: " + @hostdef.hostname + ":" + remotefile + " " + localfile
+    if @sshotgun.gatewayhost
+      cmdline = "ssh -A -t -x " + @sshotgun.gatewayhost + " 'ssh -A -x " + @hostdef.hostname + " \"cat " + remotefile + " | gzip - \"' | gzip -d - > " + localfile + "; chmod " + modestr + " " + localfile
+    else
+      cmdline = "ssh -A -t -x " + @hostdef.hostname + " \"cat " + remotefile + " | gzip - \" | gzip -d - > " + localfile + "; chmod " + modestr + " " + localfile
+    end
+    runLocal cmdline, false
+  end
+
+  # Copy a local file to remote
+  #
+  # localfile is the local file to copy. e.g. "/home/vickp/temp/myscript.sh"
+  #
+  # remotefile is the name of the destination remote file. e.g. "/usr/local/bin/myscript.sh"
+  #
+  # modestr is the string containing parameters for the chmod operation.
+  # The string must be in the NNNN or ugo format. e.g. "0755" or "u=rwx,g=rw,o=r"
+  #
+  def copyLocalFileToRemote(localfile, remotefile, modestr = "0644")
+    log "[" + @hostdef.hostname + "] copyLocalFileToRemote: " + localfile + " " + @hostdef.hostname + ":" + remotefile
+    if @sshotgun.gatewayhost
+      cmdline = "cat " + localfile + " | gzip - | ssh -A -t -x " + @sshotgun.gatewayhost + " 'ssh -A -x " + @hostdef.hostname + " \"cat | gzip -d > " + remotefile + "; chmod " + modestr + " " + remotefile + "\"'"
+    else
+      cmdline = "cat " + localfile + " | gzip - | ssh -A -t -x " + @hostdef.hostname + " \"cat | gzip -d > " + remotefile + "; chmod " + modestr + " " + remotefile + "\""
+    end
+    runLocal cmdline, false
+  end
+
+  # Create a remote text file from a content string.
+  #
+  # contentstr is the content to write to the file.
+  #
+  # remotefile is the name of the destination remote file. e.g. "/usr/local/bin/myscript.sh"
+  #
+  # modestr is the string containing parameters for the chmod operation.
+  # The string must be in the NNNN or ugo format. e.g. "0755" or "u=rwx,g=rw,o=r"
+  #
+  def createRemoteFile(contentstr, remotefile, modestr = "0644")
+    log "[" + @hostdef.hostname + "] createRemoteFile: " + @hostdef.hostname + ":" + remotefile
+    if @sshotgun.gatewayhost
+      cmdline = "echo \"" + contentstr + "\" | gzip - | ssh -A -t -x " + @sshotgun.gatewayhost + " 'ssh -A -x " + @hostdef.hostname + " \"cat | gzip -d > " + remotefile + "; chmod " + modestr + " " + remotefile + "\"'"
+    else
+      cmdline = "echo \"" + contentstr + "\" | gzip - | ssh -A -t -x " + @hostdef.hostname + " \"cat | gzip -d > " + remotefile + "; chmod " + modestr + " " + remotefile + "\""
+    end
+    runLocal cmdline, false
+  end
+
+  # Execute a command locally without launching ssh. Note that this will be executed for EACH host.
+  #
+  # cmdline is the command line to run
+  #
+  # isSudo is the flag to launch as a sudo command
+  #
+  def runLocal (cmdline, isSudo=false)
+    #log "[" + @hostdef.hostname + "] debug: " + cmdline
+    currentThread = Thread.current
+    runstatus = RunStatus.new
+    retval = Open4::popen4( cmdline ) do |pid, stdin, stdout, stderr|
+      # isSudo and if sudo password is defined, insert the sudo password via stdin
+      if isSudo
+        if @sshotgun.sudopassword
+          stdin.puts @sshotgun.sudopassword
+        end
+      end
+      runstatus.stdoutstr = stdout.read.strip
+      runstatus.stdoutstr.each do |line|
+        log "[" + @hostdef.hostname + "] stdout: " + line
+      end
+      runstatus.stderrstr = stderr.read.strip
+      runstatus.stderrstr.each do |line|
+        log "[" + @hostdef.hostname + "] stderr: " + line
+      end
+    end
+    runstatus.exitstatus = retval.exitstatus
+
+    if runstatus.exitstatus != 0
+      # set flag to indicate a non-zero exit status
+      # this flag is needed by the status report at the end of the script run
+      @hostdef.hadNonZeroExitStatus = true
+      
+      # am I supposed to stop thread?
+      if @sshotgun.isStopOnNonZeroExit
+        log "[" + @hostdef.hostname + "] stderr: Non-zero exit status - Stop processing for this server (" + runstatus.exitstatus.to_s + ")"
+        currentThread.exit
+      else
+        # did not exit thread so just log that a non-zero exit occured
+        log "[" + @hostdef.hostname + "] stderr: Non-zero exit status (" + runstatus.exitstatus.to_s + ")"
+      end
+    end
+
+    # return runstatus to caller
+    return runstatus
+  end
+
+  # Execute a sudo command locally without launching ssh. Note that this will be executed for EACH host.
+  #
+  # cmdline is the command line to run
+  #
+  def runLocalSudo (cmdline)
+    runLocal(cmdline, true)
+  end
+
+  # Log a string to the console.
+  #
+  # s is the string to log
+  #
+  def log(s)
+    @sshotgun.log s
+  end
+end
+
+# This is the main class of SSHotgun. This class stores the configuration and launches the processing threads.
+# 
+class SSHotgun
+
+  # Array containing all of the HostDefinitions (hosts to process)
+  attr_accessor :hostlist
+
+  # Your custom processing class
+  attr_accessor :processingclass
+
+  # The sudo password, if needed. (optional)
+  attr_accessor :sudopassword
+
+  # The gateway host for use as an ssh relay. (optional)
+  attr_accessor :gatewayhost
+
+  # The maximum number of processing threads to run simultaneously. Each
+  # processing thread roughly equates to one ssh client running on your local
+  # machine. Default is 30. (optional)
+  attr_accessor :maxThreads
+
+  # Timeout for processing threads. Default is 30 minutes. (optional)
+  attr_accessor :threadTimeoutInSeconds
+
+  # Interval time to periodically display thread status information. Default is 30 sec. (optional)
+  attr_accessor :monitorPollIntervalInSeconds
+
+  # For internal use. Mutex for writing to the log
+  attr_reader :logmutex
+
+  # For internal use. Mutex for launching processing threads
+  attr_reader :threadStartMutex
+
+  # File logger. Set your own if desired. (optional)
+  attr_accessor :filelogger
+
+  # Turn on/off file logging. Default is on. (optional)
+  attr_accessor :isFileLogging
+
+  # For internal use. Script start time
+  attr_reader :startTime
+
+  # Turn on/off behavior where processing for a server is stopped if any
+  # command (run, runSudo, etc) returns with a non-zero exit code. Default is
+  # off (don't stop). Note that if isStopOnNonZeroExit is true then the
+  # processing thread exits immediately, you never have the opportunity to
+  # check the exit status in your processing code.
+  attr_accessor :isStopOnNonZeroExit
+
+  def initialize(hostlist, processingclass)
+    @hostlist = hostlist
+    @processingclass = processingclass
+    @logmutex = Mutex.new
+    @threadStartMutex = Mutex.new
+    @maxThreads = 30  # allow NN processing threads to run at any time
+    @threadTimeoutInSeconds = 1800  # timeout after 30 minutes
+    @monitorPollIntervalInSeconds = 30 # period for displaying thread monitor
+    timestr = Time.now.strftime("%Y%m%d%H%M%S")
+    logfilename = File.basename($0) + ".#{timestr}.log"
+    @filelogger = Logger.new(logfilename)
+    @isFileLogging = true
+    @isStopOnNonZeroExit = false
+  end
+
+  # Start the processing. In most cases, this call will be the last line in
+  # your SSHotgun processing script.
+  #
+  def start
+    # trap ctrl-c, show stats and exit
+    trap("INT") {
+      log "[_sshotgun_] info: Script terminated early by ctrl-c"
+      showRunStats
+      exit
+    }
+    @startTime = Time.now
+    log "[_sshotgun_] info: Started script " + $0 + " at " + @startTime.to_s
+
+    # Display configuration info
+    log "[_sshotgun_] info: User=" + ENV["USER"].to_s
+    log "[_sshotgun_] info: Gateway host=" + @gatewayhost.to_s
+    log "[_sshotgun_] info: Thread timeout=" + @threadTimeoutInSeconds.to_s + "s"
+    log "[_sshotgun_] info: Max concurrent threads=" + @maxThreads.to_s
+    log "[_sshotgun_] info: Monitor poll interval=" + @monitorPollIntervalInSeconds.to_s + "s"
+    log "[_sshotgun_] info: File logging is=" + @isFileLogging.to_s
+    log "[_sshotgun_] info: Will stop thread upon non-zero exit=" + @isStopOnNonZeroExit.to_s
+
+    hostarr = []
+    @hostlist.each do |hostdef| 
+      hostarr << hostdef.hostname
+    end
+    log "[_sshotgun_] info: Host list=" + hostarr.join(", ")
+
+    currentHostIndex = 0
+    processingThreads = []
+    checkRunningThreadsCounter = 0
+    while true
+
+      # how many active threads now?
+      numThreadsAlive = 0
+      processingThreads.each { |aThread|  
+        if aThread.alive?
+          numThreadsAlive += 1
+        end
+      }
+
+      # calc number of threads to launch with this 
+      # pass through the loop
+      numThreadsToLaunch = @maxThreads - numThreadsAlive 
+      numThreadsLeftToLaunch = @hostlist.length - currentHostIndex
+      if numThreadsToLaunch > numThreadsLeftToLaunch
+        numThreadsToLaunch = numThreadsLeftToLaunch
+      end
+
+      # from current index in hosts array loop and launch threads up to maxThreads running
+      i = 0
+      while i < numThreadsToLaunch do
+        # I hate to admit it but I don't know why i needed to synchronize this block
+        # Without it, some threads seemed to freeze at startup
+        @threadStartMutex.synchronize {
+          hostdef = @hostlist[currentHostIndex]
+          processingThreads << Thread.new {
+            aThread = Thread.current
+            aThread["threadstarttime"] = Time.now  # store start time in threadlocal variable
+            aThread["hostname"] = hostdef.hostname  # store hostname time in threadlocal variable
+            log "[_sshotgun_] info: Started thread for host = " + hostdef.hostname + " at " + aThread["threadstarttime"].to_s
+            begin 
+              hostdef.startedProcess = true
+              o = @processingclass.new(self, hostdef)
+              o.doProcessing
+              hostdef.finishedProcess = true
+            rescue => ex
+              log "[_sshotgun_] stderr: Thread blew up for host = " + hostdef.hostname + ". ex = " + ex.inspect + "\n" + ex.backtrace.join("\n")
+            end
+          }
+
+          # bump loop counter
+          i += 1
+          currentHostIndex += 1
+        }
+      end
+
+      # kill expired threads
+      processingThreads.each { |aThread|  
+        if aThread.alive?
+          # ensure that the threadlocal var is actually set before
+          # doing a calc with it. This is race condition where you
+          # get here so fast and threadstarttime is nil
+          if aThread["threadstarttime"]
+            if Time.now - aThread["threadstarttime"] > @threadTimeoutInSeconds
+              log "[_sshotgun_] info: Killed thread for host = " + aThread["hostname"]
+              aThread.kill
+            end
+          end
+        end
+      }
+
+      # display list of running threads as visual indicator to user
+      checkRunningThreadsCounter += 1
+      if checkRunningThreadsCounter > @monitorPollIntervalInSeconds
+        # display running threads
+        processingThreads.each { |aThread|  
+          if aThread.alive?
+            log "[_sshotgun_] monitor: Thread still running for host = " + aThread["hostname"] + ". Elapsed time is " + (Time.now - aThread["threadstarttime"] ).to_s + "s"
+          end
+        }
+        checkRunningThreadsCounter = 0
+      end
+
+      # any processing threads still alive?
+      aThreadIsAlive = false
+      processingThreads.each { |aThread|  
+        if aThread.alive?
+          aThreadIsAlive = true
+          break
+        end
+      }
+
+      # break out of this main loop if no processing threads are alive
+      if aThreadIsAlive
+        sleep 1 
+      else
+        break
+      end
+    end
+
+    # ended normally, show stats
+    log "[_sshotgun_] info: Script terminated normally"
+    showRunStats
+  end
+
+  # Display these status at end of script or when control-c is pressed
+  def showRunStats
+    # Display script ending message
+    stopTime = Time.now
+    log "[_sshotgun_] info: Ended script " + $0 + " at " + stopTime.to_s
+    log "[_sshotgun_] info: Elapsed script " + $0 + " time " + (stopTime - @startTime).to_s + "s"
+
+    hostarr = []
+    @hostlist.each do |hostdef| 
+      if hostdef.startedProcess == false
+        # did not start processing
+        hostarr << (hostdef.hostname + "(np)")
+      else
+        # started processing
+        if hostdef.hadNonZeroExitStatus
+          # one or more exit status were non-zero
+          hostarr << (hostdef.hostname + "(*)")
+        else
+          if hostdef.finishedProcess
+            # finished with no errors during processing
+            hostarr << hostdef.hostname
+          else
+            # Incomplete processing.  This usually means that the method threw
+            # an exception
+            hostarr << (hostdef.hostname + "(i)")
+          end
+        end
+      end
+    end
+    log "[_sshotgun_] info: Status codes are '*'=had one|more non-zero exit status, 'i'=incomplete, 'np'=not started, blank=finished"
+    log "[_sshotgun_] info: Hosts summary = " + hostarr.join(", ")
+  end
+
+  # Log a string to the console and to the file logger
+  #
+  # s is the string to log
+  #
+  def log(s)
+    @logmutex.synchronize {
+      puts s.chomp
+      if @isFileLogging
+        @filelogger.info s.chomp
+      end
+    }
+  end
+end

metadata

@@ -0,0 +1,76 @@
+--- !ruby/object:Gem::Specification 
+name: sshotgun
+version: !ruby/object:Gem::Version 
+  version: 1.0.4
+platform: ruby
+authors: 
+- Vick Perry
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2008-09-08 00:00:00 -07:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: open4
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 0.9.6
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: highline
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 1.4.0
+    version: 
+description: 
+email: vick.perry @nospam@ gmail.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: 
+- README.rdoc
+files: 
+- lib/sshotgun.rb
+- README.rdoc
+has_rdoc: true
+homepage: http://sshotgun.rubyforge.org
+post_install_message: 
+rdoc_options: 
+- --line-numbers
+- --inline-source
+- --title
+- sshotgun
+- --main
+- README.rdoc
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: sshotgun
+rubygems_version: 1.1.1
+signing_key: 
+specification_version: 2
+summary: A lib for writing server management scripts in Ruby. Useful for managing lots of servers.
+test_files: []
+