sshotgun 1.0.4
Sign up to get free protection for your applications and to get access to all the features.
- data/README.rdoc +523 -0
- data/lib/sshotgun.rb +475 -0
- metadata +76 -0
data/README.rdoc
ADDED
@@ -0,0 +1,523 @@
|
|
1
|
+
== SSHotgun - Utility library for writing scripts to administer and provision multiple servers or server farms.
|
2
|
+
|
3
|
+
The SSHotgun project homepage is http://sshotgun.rubyforge.org
|
4
|
+
|
5
|
+
To get help, post your questions on forums on RubyForge at http://rubyforge.org/forum/?group_id=6867
|
6
|
+
|
7
|
+
The author of SSHotgun is Vick Perry (vick.perry @nospam@ gmail.com)
|
8
|
+
|
9
|
+
== DESCRIPTION:
|
10
|
+
|
11
|
+
SShotgun is a utility library for writing Unix/Linux server management and
|
12
|
+
provisioning scripts in Ruby. I use it for remotely managing machines in server
|
13
|
+
clusters/farms. SShotgun uses your locally installed SSH client and expects
|
14
|
+
that you have your SSH public key installed on the servers.
|
15
|
+
|
16
|
+
== FEATURES:
|
17
|
+
|
18
|
+
* Uses OpenSSH compatible SSH client and keys already installed on your local machine
|
19
|
+
* Optional use of a gateway machine for access to servers behind a firewall
|
20
|
+
* Logs in to remote machines via your public key already installed on all remote machines
|
21
|
+
* Automatically logs to the console and to a timestamped log file
|
22
|
+
* stdout and stderr from each remote machine is displayed and logged
|
23
|
+
* Can run sudo commands (your sudo password must be same on all remote machines)
|
24
|
+
* Multithreaded with configurable thread pool to service multiple machines simultaneously
|
25
|
+
* At end of processing run, a status report is displayed indicating successes, failures, timeouts, incompletes, etc.
|
26
|
+
|
27
|
+
== EXAMPLE:
|
28
|
+
|
29
|
+
You don't need to know much about Ruby to write powerful SSHotgun scripts. Just
|
30
|
+
copy the sample script below and edit the hostnames and the doProcessing method
|
31
|
+
as necessary.
|
32
|
+
|
33
|
+
The steps for writing a SSHotgun Ruby script are:
|
34
|
+
|
35
|
+
1. Create an array of fully qualified hostnames (fqdn).
|
36
|
+
2. Write your own custom processing class and doProcess method that will be
|
37
|
+
invoked on all remote servers. You can copy the example for this.
|
38
|
+
3. Instantiate a SSHotgun object in the script.
|
39
|
+
4. Run the script. A Ruby script can be invoked two different ways:
|
40
|
+
|
41
|
+
Run Ruby and specify the script
|
42
|
+
|
43
|
+
ruby ./myrubyscript.rb
|
44
|
+
|
45
|
+
or
|
46
|
+
|
47
|
+
include this line as the first line of the script
|
48
|
+
|
49
|
+
#!/usr/bin/env ruby
|
50
|
+
|
51
|
+
then make the script executable
|
52
|
+
|
53
|
+
chmod 755 myrubyscript.rb
|
54
|
+
|
55
|
+
and run it directly from the command line
|
56
|
+
|
57
|
+
./myrubyscript.rb
|
58
|
+
|
59
|
+
This is a simple example of a SSHotgun Ruby script:
|
60
|
+
|
61
|
+
#!/usr/bin/env ruby
|
62
|
+
|
63
|
+
# Required libraries
|
64
|
+
require "highline/import"
|
65
|
+
require 'sshotgun'
|
66
|
+
|
67
|
+
# Create an array to contain hosts for processing
|
68
|
+
hostlist = Array::new()
|
69
|
+
|
70
|
+
# Create a host definition
|
71
|
+
host = HostDefinition.new("localhost")
|
72
|
+
|
73
|
+
# Add the host definition to the array of hosts
|
74
|
+
hostlist << host
|
75
|
+
|
76
|
+
# Create and add more hosts...
|
77
|
+
host = HostDefinition.new("another.host.junk")
|
78
|
+
hostlist << host
|
79
|
+
|
80
|
+
# Create your own custom processing class that subclasses
|
81
|
+
# BaseProcessingClass. In this class, define your own doProcessing method
|
82
|
+
# where you write commands to execute on each remote server.
|
83
|
+
class MyProcessingClass < BaseProcessingClass
|
84
|
+
def doProcessing
|
85
|
+
# Execute a command on all remote servers. The run command has a return
|
86
|
+
# object from which you can obtain the output and exit status. See the
|
87
|
+
# user guide for more details.
|
88
|
+
run "ls -aF"
|
89
|
+
|
90
|
+
# There are other sshotgun commands available for copying files, running
|
91
|
+
# sudo, invoking local commands, etc.
|
92
|
+
end
|
93
|
+
end
|
94
|
+
|
95
|
+
# Create an instance of SSHotgun and pass it the list of hosts and your
|
96
|
+
# custom processing class.
|
97
|
+
sshotgun = SSHotgun.new(hostlist, MyProcessingClass)
|
98
|
+
|
99
|
+
# Start processing. The start call should be the very last line in a SSHotgun
|
100
|
+
# script.
|
101
|
+
sshotgun.start
|
102
|
+
|
103
|
+
== REQUIREMENTS:
|
104
|
+
|
105
|
+
* You must have an OpenSSH-compatible SSH client installed on your local machine.
|
106
|
+
* You must have correctly installed your SSH public keys on all remote servers that you access.
|
107
|
+
* For most advanced administrative tasks, you'll also need sudo privleges on the remote servers.
|
108
|
+
* You'll need write permissions in your present working directory (pwd) since SSHotgun will need to write its logs.
|
109
|
+
|
110
|
+
== INSTALL:
|
111
|
+
|
112
|
+
Use gem to install SSHotgun. Note that you probably need to run this command with sudo or as root.
|
113
|
+
|
114
|
+
gem install sshotgun
|
115
|
+
|
116
|
+
== USER GUIDE:
|
117
|
+
|
118
|
+
Here are snippets of SSHotgun Ruby code to illustrate SSHotgun's capabilities.
|
119
|
+
|
120
|
+
At a minimum, a host definition contains the fully qualified domain name (fqdn)
|
121
|
+
of a machine. As a best practice, DNS aliases are not generally referenced -
|
122
|
+
only the fqdn. There are also several optional fields in a host definition that
|
123
|
+
can be used for more selectivity when running a script. For example, you may
|
124
|
+
wish to do some additional configuration for those machines categorized as
|
125
|
+
"ldap" or "proxy"
|
126
|
+
|
127
|
+
A host definition has the following fields: hostname, category, os, desc
|
128
|
+
|
129
|
+
* hostname is the primary hostname or fqdn and NOT an alias.
|
130
|
+
* category is a freeform identifier for the type of the host. e.g. "wiki", "ldap" (optional)
|
131
|
+
* os is a freeform string indicating os name + server/workstation + version. e.g. ubuntu_server_7.10 (optional)
|
132
|
+
* desc is a one sentence string describing main purpose of the host (optional)
|
133
|
+
|
134
|
+
An example of a complete host definition is:
|
135
|
+
|
136
|
+
host = HostDefinition.new("localhost", "test", "ubuntu_server_7.10", "test vm")
|
137
|
+
hostlist << host
|
138
|
+
|
139
|
+
You can share a list of hosts among several SSHotgun scripts. To do this,
|
140
|
+
create a Ruby module (not a class) in a separate file. The module will contain
|
141
|
+
a method that returns an array of host definitions. The module is referenced by
|
142
|
+
any script that needs it.
|
143
|
+
|
144
|
+
# When you create your own modules, name the filename and module name with a similar name.
|
145
|
+
# Note: This module filename is 'listofhosts.rb' - this is the reference in the require statement in the calling script.
|
146
|
+
module ListOfHosts
|
147
|
+
|
148
|
+
# Create a method named <modulename>.getHostlist
|
149
|
+
def ListOfHosts.getHostlist
|
150
|
+
# Define, load and return the hostlist array
|
151
|
+
hostlist = []
|
152
|
+
|
153
|
+
host = HostDefinition.new("localhost", "test", "ubuntu_server_7.10", "test vm")
|
154
|
+
hostlist << host
|
155
|
+
|
156
|
+
host = HostDefinition.new("other.host.junk", "test", "ubuntu_server_7.10", "test vm")
|
157
|
+
hostlist << host
|
158
|
+
|
159
|
+
# return the array
|
160
|
+
return hostlist
|
161
|
+
end
|
162
|
+
|
163
|
+
end
|
164
|
+
|
165
|
+
|
166
|
+
It is a good practice to name the module file name similar or identical to the
|
167
|
+
module name.
|
168
|
+
|
169
|
+
Here is a SSHotgun Ruby script that uses the list of hosts:
|
170
|
+
|
171
|
+
#!/usr/bin/env ruby
|
172
|
+
|
173
|
+
# Required libraries
|
174
|
+
require "highline/import"
|
175
|
+
require 'sshotgun'
|
176
|
+
|
177
|
+
# Create array to contain hosts for processing
|
178
|
+
hostlist = Array::new()
|
179
|
+
|
180
|
+
# Read hosts from an external Ruby module.
|
181
|
+
require 'listofhosts'
|
182
|
+
|
183
|
+
# In the listofhosts.rb file, the ListOfHosts getHostlist method returns an
|
184
|
+
# array of hosts. Concatenate all of the multiple arrays into a single array of
|
185
|
+
# hosts.
|
186
|
+
hostlist = hostlist + ListOfHosts.getHostlist
|
187
|
+
|
188
|
+
class MyProcessingClass < BaseProcessingClass
|
189
|
+
def doProcessing
|
190
|
+
run "ls -aF"
|
191
|
+
end
|
192
|
+
end
|
193
|
+
|
194
|
+
sshotgun = SSHotgun.new(hostlist, MyProcessingClass)
|
195
|
+
sshotgun.start
|
196
|
+
|
197
|
+
Here's how to prompt for the sudo password (Don't hardcode a password into your
|
198
|
+
script). See the advanced example for details.
|
199
|
+
|
200
|
+
sshotgun.sudopassword = ask(">>> Enter your sudo password: ") { |q| q.echo = "*" } # or q.echo = false
|
201
|
+
|
202
|
+
Below is an advanced example that illustrates additional capabilities.
|
203
|
+
|
204
|
+
#!/usr/bin/env ruby
|
205
|
+
|
206
|
+
# Required libraries
|
207
|
+
require "highline/import"
|
208
|
+
require 'sshotgun'
|
209
|
+
|
210
|
+
# Create array to contain hosts for processing
|
211
|
+
hostlist = Array::new()
|
212
|
+
host = HostDefinition.new("someserver.junk", "test", "ubuntu_server_7.10", "test vm")
|
213
|
+
hostlist << host
|
214
|
+
|
215
|
+
# Import a list of host definitions
|
216
|
+
require 'listofhosts'
|
217
|
+
hostlist = hostlist + ListOfHosts.getHostlist
|
218
|
+
|
219
|
+
class MyProcessingClass < BaseProcessingClass
|
220
|
+
|
221
|
+
# You can add your own Ruby instance variables. These are useful if you
|
222
|
+
# want to share data across your own custom methods.
|
223
|
+
attr_accessor :myInstanceVariable
|
224
|
+
|
225
|
+
# You MUST define a doProcessing method. Note that to make the doProcessing
|
226
|
+
# method less cluttered, you can also create your own methods that are
|
227
|
+
# called from within the doProcessing method. You can use your own accessor
|
228
|
+
# variables (see above) for data sharing between methods. The doProcess method
|
229
|
+
# is the only method called by the SSHotgun framework for each host.
|
230
|
+
def doProcessing
|
231
|
+
# Log the beginning of this method. This marker is useful for debugging problems.
|
232
|
+
log "[" + @hostdef.hostname + "] info: " + "Processing started"
|
233
|
+
|
234
|
+
# Execute a command on all remote servers. Remote output is always written
|
235
|
+
# to this script's stdout.
|
236
|
+
# run <command>
|
237
|
+
run "ls -aF | grep -i .bash"
|
238
|
+
|
239
|
+
# The runstatus object is returned from all run and runSudo calls.
|
240
|
+
# runstatus contains information about exit status, stdout and stderr from
|
241
|
+
# the remote server.
|
242
|
+
runstatus = run "ls -aF"
|
243
|
+
|
244
|
+
# Use the log method to write to the log (the console). Don't use the Ruby
|
245
|
+
# 'puts' method to write to the console because it isn't threadsafe and may
|
246
|
+
# mix and garble simultaneous output by multiple threads.
|
247
|
+
log "[" + @hostdef.hostname + "] info: " + "The exit status of the run command is " + runstatus.exitstatus.to_s
|
248
|
+
|
249
|
+
# Remote stderr and stdout are captured and stored in the stdoutstr and
|
250
|
+
# stderrstr variables. Learn about Ruby's regular expressions for examining
|
251
|
+
# the output from a remote command.
|
252
|
+
runstatus = run "id rumplestiltskin"
|
253
|
+
|
254
|
+
# Check for "No such user" contained in stdoutstr. That's what is displayed
|
255
|
+
# when the unix "id" command can't find the specified user. Note that the
|
256
|
+
# the exit status of a failed 'id' command is not equal to zero either (see
|
257
|
+
# below). This is also a Ruby regex comparison.
|
258
|
+
if /No such user/ =~ runstatus.stdoutstr
|
259
|
+
log "I failed to find user rumplestiltskin on this server"
|
260
|
+
end
|
261
|
+
|
262
|
+
# In some cases a remote command fails and you must stop processing for
|
263
|
+
# that one server. Stop the execution of the doProcessing method by calling
|
264
|
+
# Ruby's "return" statement. Don't call Ruby's "exit" statement because
|
265
|
+
# that will halt all running threads.
|
266
|
+
# runstatus = run "id rumplestiltskin"
|
267
|
+
# if runstatus.exitstatus > 0
|
268
|
+
# log "The exit code was not 0, therefore I failed to find user rumplestiltskin"
|
269
|
+
# return # stop processing for this host
|
270
|
+
# end
|
271
|
+
# log "You'll never get here...unless you have a user named rumplestiltskin"
|
272
|
+
|
273
|
+
# The @hostdef instance variable contains the host definition for the
|
274
|
+
# current host being processed. If you set category, os and desc to
|
275
|
+
# meaningful information then you can do fancier branching logic in your
|
276
|
+
# script. The @hostdef.hostname is very useful in log statements. The other
|
277
|
+
# @hostdef fields are useful for treating a category or os differently than
|
278
|
+
# the others.
|
279
|
+
log "The current host is: " + @hostdef.hostname
|
280
|
+
log "The current host's category is: " + @hostdef.category
|
281
|
+
log "The current host's os is: " + @hostdef.os
|
282
|
+
log "The current host's description is: " + @hostdef.desc
|
283
|
+
|
284
|
+
# Target a specified host. This is a Ruby string comparison.
|
285
|
+
if @hostdef.hostname == "192.168.1.224"
|
286
|
+
log "[" + @hostdef.hostname + "] info: " + "Special processing for this host"
|
287
|
+
end
|
288
|
+
|
289
|
+
# If necessary, escape any quotes within the command. that must be passed
|
290
|
+
# through to the remote server. This isn't a great example but you'll need
|
291
|
+
# to do this for tricky shell scripting.
|
292
|
+
run "ls -aF | grep -i \".bash\""
|
293
|
+
|
294
|
+
# If you are running any runSudo commands then the SSHotgun.sudoPassword
|
295
|
+
# must be set - uncomment the user prompt and setting of the sudoPassword
|
296
|
+
# at the bottom of this script. Don't set up your servers to permit a
|
297
|
+
# password-less sudo - it is a security risk. Watch out, the "runSudo"
|
298
|
+
# command and Ruby in general, are case sensitive.
|
299
|
+
# runSudo <command>
|
300
|
+
# runSudo "touch /usr/local/bin/mytest.txt"
|
301
|
+
# run "ls -alF /usr/local/bin/"
|
302
|
+
# runSudo "rm /usr/local/bin/mytest.txt"
|
303
|
+
|
304
|
+
# Run a command locally (on your local machine). The runstatus variable is
|
305
|
+
# returned if you need the data - same as run or runSudo.
|
306
|
+
# runLocal <command>
|
307
|
+
# runLocal "ls -alF"
|
308
|
+
|
309
|
+
# Run a sudo command locally. The runstatus variable is returned if you
|
310
|
+
# need data from it. Set a sudoPassword if you are calling runLocalSudo.
|
311
|
+
# runLocalSudo <command>
|
312
|
+
# runLocalSudo "ls -alF"
|
313
|
+
|
314
|
+
# Create a remote file from a string parameter.
|
315
|
+
# createRemoteFile <content string>, <remotefile>
|
316
|
+
# createRemoteFile "this is your content here", "/home/vickp/testfile.txt"
|
317
|
+
|
318
|
+
# Create a remote file and set mode
|
319
|
+
# createRemoteFile <content string>, <remotefile>, <mode string>
|
320
|
+
# createRemoteFile "this is your content here", "/home/vickp/testfile.sh", "0755"
|
321
|
+
|
322
|
+
# Copy a remote file to the local machine. Note that you should vary the
|
323
|
+
# name of the incoming local file else it will be overwritten when each
|
324
|
+
# host's file is copied.
|
325
|
+
# copyRemoteFileToLocal <remotefile>, <localfile>
|
326
|
+
# copyRemoteFileToLocal "/home/vickp/testfile.txt", "/home/vickp/temp/" + @hostdef.hostname + "_testfile.txt"
|
327
|
+
|
328
|
+
# Copy a remote file to the local machine and set the mode of local file.
|
329
|
+
# copyRemoteFileToLocal <remotefile>, <localfile>, <mode string>
|
330
|
+
# copyRemoteFileToLocal "/home/vickp/testfile.txt", "/home/vickp/temp/" + @hostdef.hostname + "_testfile.txt", "0644"
|
331
|
+
|
332
|
+
# Copy a local file to the remote machine.
|
333
|
+
# copyRemoteFileToLocal <localfile>, <remotefile>
|
334
|
+
# copyLocalFileToRemote "/home/vickp/temp/testfile.txt", "/home/vickp/testfile2.txt"
|
335
|
+
|
336
|
+
# Copy a local file to the remote machine and set the mode of remote file.
|
337
|
+
# copyRemoteFileToLocal <localfile>, <remotefile>, <mode string>
|
338
|
+
# copyLocalFileToRemote "/home/vickp/temp/testfile.txt", "/home/vickp/testfile2.txt", "0644"
|
339
|
+
|
340
|
+
# Other helpful SSHotgun and Ruby tips below
|
341
|
+
# ===========================================
|
342
|
+
|
343
|
+
# Ruby allows you to concatenate multiple command lines
|
344
|
+
# Note use of shell command separator ';'
|
345
|
+
# cmd = "ls -alF;"
|
346
|
+
# cmd << "ls -alF | grep -i .bash;"
|
347
|
+
# cmd << "ls -alF | grep -i .profile"
|
348
|
+
# run cmd
|
349
|
+
|
350
|
+
# Delete a file on the remote server.
|
351
|
+
# Note use of -f with rm command
|
352
|
+
# run "rm -f /path/to/remote/file"
|
353
|
+
|
354
|
+
# Call your own methods to unclutter and better organize your doProcess
|
355
|
+
# method code. In most of my SSHotgun scripts the doProcess method is
|
356
|
+
# fairly sparse - most of the work is done is various custom methods that
|
357
|
+
# are called within the doProcess method
|
358
|
+
# myOwnMethod
|
359
|
+
|
360
|
+
# Sometimes I either install unsigned packages of my own (shell and ruby
|
361
|
+
# scripts) or need to force the installation of files (overwrite) that
|
362
|
+
# belong to another package. Aptitude doesn't yet support forcing yes for
|
363
|
+
# this sort of thing but apt-get does...
|
364
|
+
# runSudo "apt-get install -y --force-yes myUntrustedPackage -o DPkg::options::='--force-overwrite'"
|
365
|
+
|
366
|
+
# Ruby's ENV contains the current user (you)
|
367
|
+
currentUser = ENV["USER"].to_s
|
368
|
+
|
369
|
+
# For double quoted strings, Ruby will do inline substitution for a
|
370
|
+
# variable enbedded within "#{myvariablehere}". This is useful when you
|
371
|
+
# want a variable substituted inside of double quotes
|
372
|
+
# createRemoteFile "deb http://debrepo/global ./", "/home/#{currentUser}/sources.list", "0644"
|
373
|
+
|
374
|
+
# Create a global variable if you need to prompt for additional information
|
375
|
+
# (such as userid, date range, etc) and want to use that information in
|
376
|
+
# your doProcessing method. Global variables begin with '$'. See bottom of
|
377
|
+
# script where the variable is created. Note that "id" returns with an exit
|
378
|
+
# status of 1 if the user does not exist.
|
379
|
+
runstatus = run "id #{$userid}"
|
380
|
+
if runstatus.exitstatus == 0
|
381
|
+
log "[" + @hostdef.hostname + "] info: " + "Found user " + $userid
|
382
|
+
else
|
383
|
+
log "[" + @hostdef.hostname + "] info: " + "DID NOT find user " + $userid
|
384
|
+
end
|
385
|
+
|
386
|
+
# To pass escaped characters in strings, tell Ruby not to interpolate them
|
387
|
+
# by using single quotes or the %q() delimiter to enclose the escaped
|
388
|
+
# string. This means that Ruby will not attempt to resolve the escaped
|
389
|
+
# string but merely pass it on to the ssh client.
|
390
|
+
# run %q("mycommand \"some\ string\ with\ escapes\" more")
|
391
|
+
|
392
|
+
# Example of how to update a debian/ubuntu sources.list package repository file
|
393
|
+
# Also see use of case statement in the method body.
|
394
|
+
# installSourcesListAndUpdate
|
395
|
+
|
396
|
+
# Log the end of this method.
|
397
|
+
log "[" + @hostdef.hostname + "] info: " + "Processing completed"
|
398
|
+
end
|
399
|
+
end
|
400
|
+
|
401
|
+
# Your own custom methods can be called from within the doProcess method
|
402
|
+
def myOwnMethod
|
403
|
+
log "[" + @hostdef.hostname + "] info: " + "Calling my own method"
|
404
|
+
end
|
405
|
+
|
406
|
+
# Update a debian/ubuntu /etc/apt/sources.list
|
407
|
+
def installSourcesListAndUpdate
|
408
|
+
# WARNING: This only works if you fill in the category field in your hostdefs.
|
409
|
+
#
|
410
|
+
# Copy production sources.list into your home directory, then copy it into
|
411
|
+
# place. Note this is a two step operation because you can't create the
|
412
|
+
# remote file in a place where you don't have perms. The second step is to
|
413
|
+
# use sudo and copy the file into place.
|
414
|
+
s = ""
|
415
|
+
case @hostdef.category
|
416
|
+
when "production"
|
417
|
+
f = File.new("/usr/local/bin/adminscripts_templates/production_ubuntuserver710i386.sources.list")
|
418
|
+
when "devint"
|
419
|
+
f = File.new("/usr/local/bin/adminscripts_templates/devint_ubuntuserver710i386.sources.list")
|
420
|
+
when "qaint"
|
421
|
+
f = File.new("/usr/local/bin/adminscripts_templates/qaint_ubuntuserver710i386.sources.list")
|
422
|
+
when "staging"
|
423
|
+
f = File.new("/usr/local/bin/adminscripts_templates/staging_ubuntuserver710i386.sources.list")
|
424
|
+
else
|
425
|
+
log "[" + @hostdef.hostname + "] ERROR: " + "Unknown hostdef category = " + @hostdef.category
|
426
|
+
return
|
427
|
+
end
|
428
|
+
# read the file and concatenate it into a single string
|
429
|
+
f.each_line do |line|
|
430
|
+
s << line
|
431
|
+
end
|
432
|
+
createRemoteFile s, "/home/#{$currentUser}/sources.list", "0644"
|
433
|
+
runSudo "cp /home/#{$currentUser}/sources.list /etc/apt/sources.list"
|
434
|
+
run "rm /home/#{$currentUser}/sources.list"
|
435
|
+
|
436
|
+
# after changing sources.list, update package manager to refresh package list
|
437
|
+
runSudo "aptitude update"
|
438
|
+
end
|
439
|
+
|
440
|
+
# Create an instance of SSHotgun and pass it the list of hosts and your custom
|
441
|
+
# processing class.
|
442
|
+
sshotgun = SSHotgun.new(hostlist, MyProcessingClass)
|
443
|
+
|
444
|
+
# If you are calling a sudo command then prompt the user for the sudo password.
|
445
|
+
# I recommend that you NEVER HARDCODE A PASSWORD IN A SCRIPT! Configure the
|
446
|
+
# ask command to hide the password as the user types it. Note that the you
|
447
|
+
# must have the highline gem package installed for the "ask" command
|
448
|
+
#sshotgun.sudopassword = ask(">>> Enter your sudo password: ") { |q| q.echo = "*" } # or q.echo = false
|
449
|
+
|
450
|
+
# If you need to forward all calls via a gateway ssh server, set it here. Once
|
451
|
+
# set, all ssh connections go through the gateway machine
|
452
|
+
#sshotgun.gatewayhost = "someserver.somedomain.com"
|
453
|
+
|
454
|
+
# You can also use the 'ask' command to prompt the user for any other
|
455
|
+
# information. Configure the ask command to display the string as the user
|
456
|
+
# types it. Be sure to test password-less login from your gateway server to
|
457
|
+
# clear out any prompts asking you to verify the authenticity of the remote
|
458
|
+
# host.
|
459
|
+
#gatewayhost = ask(">>> Enter the hostname of the gateway server: ") { |q| q.echo = true }
|
460
|
+
#sshotgun.gatewayhost = gatewayhost
|
461
|
+
|
462
|
+
# Create and use a global variable to get additional data into your doProcessing method.
|
463
|
+
$userid = ask(">>> Enter the userid to find: ") { |q| q.echo = true }
|
464
|
+
|
465
|
+
# Number of simultaneous processing threads (hosts) to run. In terms of load to
|
466
|
+
# your local machine, each processing thread equates to about one ssh client
|
467
|
+
# running. Default is 50
|
468
|
+
# sshotgun.maxThreads = 30
|
469
|
+
|
470
|
+
# A processing thread will be killed after this amount of time. Default is 30 minutes.
|
471
|
+
# sshotgun.threadTimeoutInSeconds = 900
|
472
|
+
|
473
|
+
# Period for displaying status information about processing threads that
|
474
|
+
# are still running. Default is 30
|
475
|
+
# sshotgun.monitorPollIntervalInSeconds = 20
|
476
|
+
|
477
|
+
# Any output is also written to a file logger. You can disable the file logging
|
478
|
+
# by setting isFileLogging to false.
|
479
|
+
# sshotgun.isFileLogging = false.
|
480
|
+
|
481
|
+
# What should the behavior be when a command returns a non-zero exit status? A
|
482
|
+
# non-zero exit status usually indicates failure of the command. Some sshotgun
|
483
|
+
# users wish to immediately stop the processing for that server and stop it's
|
484
|
+
# thread. Other people will check the exit status after each command and decide
|
485
|
+
# whether to continue or return from the processing method. Note that if
|
486
|
+
# isStopOnNonZeroExit is true then the processing thread exits immediately and
|
487
|
+
# you never have the opportunity to check the exit status in your processing
|
488
|
+
# method code. The default is false (don't stop on non-zero exit).
|
489
|
+
# sshotgun.isStopOnNonZeroExit = true
|
490
|
+
|
491
|
+
# Start processing. The start call should be the very last line in a SSHotgun
|
492
|
+
# script.
|
493
|
+
sshotgun.start
|
494
|
+
|
495
|
+
== LICENSE:
|
496
|
+
|
497
|
+
(Simplified BSD License)
|
498
|
+
|
499
|
+
Copyright (c) 2008, Vick Perry
|
500
|
+
All rights reserved.
|
501
|
+
|
502
|
+
Redistribution and use in source and binary forms, with or without
|
503
|
+
modification, are permitted provided that the following conditions are met:
|
504
|
+
|
505
|
+
* Redistributions of source code must retain the above copyright notice, this
|
506
|
+
list of conditions and the following disclaimer.
|
507
|
+
* Redistributions in binary form must reproduce the above copyright notice,
|
508
|
+
this list of conditions and the following disclaimer in the documentation
|
509
|
+
and/or other materials provided with the distribution.
|
510
|
+
* Neither the name of the <ORGANIZATION> nor the names of its contributors may
|
511
|
+
be used to endorse or promote products derived from this software without
|
512
|
+
specific prior written permission.
|
513
|
+
|
514
|
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
|
515
|
+
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
|
516
|
+
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
517
|
+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
|
518
|
+
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
|
519
|
+
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
|
520
|
+
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
|
521
|
+
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
522
|
+
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
|
523
|
+
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
data/lib/sshotgun.rb
ADDED
@@ -0,0 +1,475 @@
|
|
1
|
+
#:title: SSHotgun
|
2
|
+
#Author:: Vick Perry (vick.perry @nospam@ gmail.com)
|
3
|
+
#Copyright:: Copyright (c) 2008, Vick Perry
|
4
|
+
#License:: Simplified BSD License.
|
5
|
+
#
|
6
|
+
require 'open4'
|
7
|
+
require 'logger'
|
8
|
+
|
9
|
+
# Class to contain information about each server
|
10
|
+
#
|
11
|
+
class HostDefinition
|
12
|
+
# hostname is the primary hostname or fqdn (NOT an alias). It is better to only refer to the primary DNS entry for a machine or VM.
|
13
|
+
attr_accessor :hostname
|
14
|
+
|
15
|
+
# category is a freeform identifier for the type of the host. e.g. "production", "devint", "qaint", "staging" (optional)
|
16
|
+
attr_accessor :category
|
17
|
+
|
18
|
+
# os is a freeform string indicating os name + server/workstation + version. e.g. ubuntu_server_7.10 (optional)
|
19
|
+
attr_accessor :os
|
20
|
+
|
21
|
+
# desc is a one sentence string describing main purpose of the host (optional)
|
22
|
+
attr_accessor :desc
|
23
|
+
|
24
|
+
# For internal use. True if processing was done for this server.
|
25
|
+
attr_accessor :startedProcess
|
26
|
+
|
27
|
+
# For internal use. True if processing was finished for this server. This means that doProcess exited
|
28
|
+
# normally with no exceptions.
|
29
|
+
attr_accessor :finishedProcess
|
30
|
+
|
31
|
+
# For internal use. True if one or more commands on a server returned with a non-zero exit status
|
32
|
+
attr_accessor :hadNonZeroExitStatus
|
33
|
+
|
34
|
+
def initialize(hostname, category = "", os = "", desc = "")
|
35
|
+
@hostname = hostname
|
36
|
+
@category = category
|
37
|
+
@os = os
|
38
|
+
@desc = desc
|
39
|
+
@startedProcess = false
|
40
|
+
@finishedProcess = false
|
41
|
+
@hadNonZeroExitStatus = false
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
45
|
+
# This class contains the returned information from commands such as run, runSudo, runLocal, runLocalSudo
|
46
|
+
#
|
47
|
+
class RunStatus
|
48
|
+
# The exit status of the command
|
49
|
+
attr_accessor :exitstatus
|
50
|
+
|
51
|
+
# The stdout stream data from the command stored in a string
|
52
|
+
attr_accessor :stdoutstr
|
53
|
+
|
54
|
+
# The stderr stream data from the command stored in a string
|
55
|
+
attr_accessor :stderrstr
|
56
|
+
|
57
|
+
def initialize
|
58
|
+
@exitstatus = 0
|
59
|
+
@stdoutstr = ""
|
60
|
+
@stderrstr = ""
|
61
|
+
end
|
62
|
+
end
|
63
|
+
|
64
|
+
# This is the base processing class. Your custom processing class MUST inherit from this base class.
|
65
|
+
#
|
66
|
+
class BaseProcessingClass
|
67
|
+
# A SSHotgun object
|
68
|
+
attr_accessor :sshotgun
|
69
|
+
|
70
|
+
# The current HostDefinition object (represents the remote server for processing)
|
71
|
+
attr_accessor :hostdef
|
72
|
+
|
73
|
+
# Default constructor. Don't create a constructor in your custom processing
|
74
|
+
# class so that this constructor will be called.
|
75
|
+
#
|
76
|
+
def initialize(sshotgun, hostdef)
|
77
|
+
@sshotgun = sshotgun
|
78
|
+
@hostdef = hostdef
|
79
|
+
end
|
80
|
+
|
81
|
+
# You must override the doProcessing method in your own custom processing class
|
82
|
+
#
|
83
|
+
def doProcessing
|
84
|
+
log "[" + @hostdef.hostname + "] info: " + "The doProcessing method must be created in your custom processing class"
|
85
|
+
end
|
86
|
+
|
87
|
+
# Run a command on a remote server
|
88
|
+
#
|
89
|
+
# cmd is a string containing the command to run on a remote server. e.g. "ls -alF"
|
90
|
+
#
|
91
|
+
# Returns a RunStatus object
|
92
|
+
#
|
93
|
+
def run(cmd)
|
94
|
+
log "[" + @hostdef.hostname + "] run: " + cmd
|
95
|
+
if @sshotgun.gatewayhost
|
96
|
+
cmdline = "ssh -A -t -x " + @sshotgun.gatewayhost + " 'ssh -A -x " + @hostdef.hostname + " \"" + cmd + "\"'"
|
97
|
+
else
|
98
|
+
cmdline = "ssh -A -t -x " + @hostdef.hostname + " \"" + cmd + "\""
|
99
|
+
end
|
100
|
+
runLocal cmdline, false
|
101
|
+
end
|
102
|
+
|
103
|
+
# Sudo a command on a remote server
|
104
|
+
#
|
105
|
+
# cmd is a string containing the sudo command to run on a remote server. e.g. "ls -alF"
|
106
|
+
#
|
107
|
+
# Returns a RunStatus object
|
108
|
+
#
|
109
|
+
def runSudo(cmd)
|
110
|
+
log "[" + @hostdef.hostname + "] runSudo: " + cmd
|
111
|
+
if @sshotgun.gatewayhost
|
112
|
+
cmdline = "ssh -A -t -x " + @sshotgun.gatewayhost + " 'ssh -A -x " + @hostdef.hostname + " sudo -S \"" + cmd + "\"'"
|
113
|
+
else
|
114
|
+
cmdline = "ssh -A -t -x " + @hostdef.hostname + " sudo -S \"" + cmd + "\""
|
115
|
+
end
|
116
|
+
runLocal cmdline, true
|
117
|
+
end
|
118
|
+
|
119
|
+
# Copy a remote file to local
|
120
|
+
#
|
121
|
+
# remotefile is the remote file to fetch. e.g. "/usr/local/bin/myscript.sh"
|
122
|
+
#
|
123
|
+
# localfile is the name of the destination local file. e.g. "/home/vickp/temp/myscript.sh"
|
124
|
+
#
|
125
|
+
# modestr is the string containing parameters for the chmod operation.
|
126
|
+
# The string must be in the NNNN or ugo format. e.g. "0755" or "u=rwx,g=rw,o=r"
|
127
|
+
#
|
128
|
+
def copyRemoteFileToLocal(remotefile, localfile, modestr = "0644")
|
129
|
+
log "[" + @hostdef.hostname + "] copyRemoteFileToLocal: " + @hostdef.hostname + ":" + remotefile + " " + localfile
|
130
|
+
if @sshotgun.gatewayhost
|
131
|
+
cmdline = "ssh -A -t -x " + @sshotgun.gatewayhost + " 'ssh -A -x " + @hostdef.hostname + " \"cat " + remotefile + " | gzip - \"' | gzip -d - > " + localfile + "; chmod " + modestr + " " + localfile
|
132
|
+
else
|
133
|
+
cmdline = "ssh -A -t -x " + @hostdef.hostname + " \"cat " + remotefile + " | gzip - \" | gzip -d - > " + localfile + "; chmod " + modestr + " " + localfile
|
134
|
+
end
|
135
|
+
runLocal cmdline, false
|
136
|
+
end
|
137
|
+
|
138
|
+
# Copy a local file to remote
|
139
|
+
#
|
140
|
+
# localfile is the local file to copy. e.g. "/home/vickp/temp/myscript.sh"
|
141
|
+
#
|
142
|
+
# remotefile is the name of the destination remote file. e.g. "/usr/local/bin/myscript.sh"
|
143
|
+
#
|
144
|
+
# modestr is the string containing parameters for the chmod operation.
|
145
|
+
# The string must be in the NNNN or ugo format. e.g. "0755" or "u=rwx,g=rw,o=r"
|
146
|
+
#
|
147
|
+
def copyLocalFileToRemote(localfile, remotefile, modestr = "0644")
|
148
|
+
log "[" + @hostdef.hostname + "] copyLocalFileToRemote: " + localfile + " " + @hostdef.hostname + ":" + remotefile
|
149
|
+
if @sshotgun.gatewayhost
|
150
|
+
cmdline = "cat " + localfile + " | gzip - | ssh -A -t -x " + @sshotgun.gatewayhost + " 'ssh -A -x " + @hostdef.hostname + " \"cat | gzip -d > " + remotefile + "; chmod " + modestr + " " + remotefile + "\"'"
|
151
|
+
else
|
152
|
+
cmdline = "cat " + localfile + " | gzip - | ssh -A -t -x " + @hostdef.hostname + " \"cat | gzip -d > " + remotefile + "; chmod " + modestr + " " + remotefile + "\""
|
153
|
+
end
|
154
|
+
runLocal cmdline, false
|
155
|
+
end
|
156
|
+
|
157
|
+
# Create a remote text file from a content string.
|
158
|
+
#
|
159
|
+
# contentstr is the content to write to the file.
|
160
|
+
#
|
161
|
+
# remotefile is the name of the destination remote file. e.g. "/usr/local/bin/myscript.sh"
|
162
|
+
#
|
163
|
+
# modestr is the string containing parameters for the chmod operation.
|
164
|
+
# The string must be in the NNNN or ugo format. e.g. "0755" or "u=rwx,g=rw,o=r"
|
165
|
+
#
|
166
|
+
def createRemoteFile(contentstr, remotefile, modestr = "0644")
|
167
|
+
log "[" + @hostdef.hostname + "] createRemoteFile: " + @hostdef.hostname + ":" + remotefile
|
168
|
+
if @sshotgun.gatewayhost
|
169
|
+
cmdline = "echo \"" + contentstr + "\" | gzip - | ssh -A -t -x " + @sshotgun.gatewayhost + " 'ssh -A -x " + @hostdef.hostname + " \"cat | gzip -d > " + remotefile + "; chmod " + modestr + " " + remotefile + "\"'"
|
170
|
+
else
|
171
|
+
cmdline = "echo \"" + contentstr + "\" | gzip - | ssh -A -t -x " + @hostdef.hostname + " \"cat | gzip -d > " + remotefile + "; chmod " + modestr + " " + remotefile + "\""
|
172
|
+
end
|
173
|
+
runLocal cmdline, false
|
174
|
+
end
|
175
|
+
|
176
|
+
# Execute a command locally without launching ssh. Note that this will be executed for EACH host.
|
177
|
+
#
|
178
|
+
# cmdline is the command line to run
|
179
|
+
#
|
180
|
+
# isSudo is the flag to launch as a sudo command
|
181
|
+
#
|
182
|
+
def runLocal (cmdline, isSudo=false)
|
183
|
+
#log "[" + @hostdef.hostname + "] debug: " + cmdline
|
184
|
+
currentThread = Thread.current
|
185
|
+
runstatus = RunStatus.new
|
186
|
+
retval = Open4::popen4( cmdline ) do |pid, stdin, stdout, stderr|
|
187
|
+
# isSudo and if sudo password is defined, insert the sudo password via stdin
|
188
|
+
if isSudo
|
189
|
+
if @sshotgun.sudopassword
|
190
|
+
stdin.puts @sshotgun.sudopassword
|
191
|
+
end
|
192
|
+
end
|
193
|
+
runstatus.stdoutstr = stdout.read.strip
|
194
|
+
runstatus.stdoutstr.each do |line|
|
195
|
+
log "[" + @hostdef.hostname + "] stdout: " + line
|
196
|
+
end
|
197
|
+
runstatus.stderrstr = stderr.read.strip
|
198
|
+
runstatus.stderrstr.each do |line|
|
199
|
+
log "[" + @hostdef.hostname + "] stderr: " + line
|
200
|
+
end
|
201
|
+
end
|
202
|
+
runstatus.exitstatus = retval.exitstatus
|
203
|
+
|
204
|
+
if runstatus.exitstatus != 0
|
205
|
+
# set flag to indicate a non-zero exit status
|
206
|
+
# this flag is needed by the status report at the end of the script run
|
207
|
+
@hostdef.hadNonZeroExitStatus = true
|
208
|
+
|
209
|
+
# am I supposed to stop thread?
|
210
|
+
if @sshotgun.isStopOnNonZeroExit
|
211
|
+
log "[" + @hostdef.hostname + "] stderr: Non-zero exit status - Stop processing for this server (" + runstatus.exitstatus.to_s + ")"
|
212
|
+
currentThread.exit
|
213
|
+
else
|
214
|
+
# did not exit thread so just log that a non-zero exit occured
|
215
|
+
log "[" + @hostdef.hostname + "] stderr: Non-zero exit status (" + runstatus.exitstatus.to_s + ")"
|
216
|
+
end
|
217
|
+
end
|
218
|
+
|
219
|
+
# return runstatus to caller
|
220
|
+
return runstatus
|
221
|
+
end
|
222
|
+
|
223
|
+
# Execute a sudo command locally without launching ssh. Note that this will be executed for EACH host.
|
224
|
+
#
|
225
|
+
# cmdline is the command line to run
|
226
|
+
#
|
227
|
+
def runLocalSudo (cmdline)
|
228
|
+
runLocal(cmdline, true)
|
229
|
+
end
|
230
|
+
|
231
|
+
# Log a string to the console.
|
232
|
+
#
|
233
|
+
# s is the string to log
|
234
|
+
#
|
235
|
+
def log(s)
|
236
|
+
@sshotgun.log s
|
237
|
+
end
|
238
|
+
end
|
239
|
+
|
240
|
+
# This is the main class of SSHotgun. This class stores the configuration and launches the processing threads.
|
241
|
+
#
|
242
|
+
class SSHotgun
|
243
|
+
|
244
|
+
# Array containing all of the HostDefinitions (hosts to process)
|
245
|
+
attr_accessor :hostlist
|
246
|
+
|
247
|
+
# Your custom processing class
|
248
|
+
attr_accessor :processingclass
|
249
|
+
|
250
|
+
# The sudo password, if needed. (optional)
|
251
|
+
attr_accessor :sudopassword
|
252
|
+
|
253
|
+
# The gateway host for use as an ssh relay. (optional)
|
254
|
+
attr_accessor :gatewayhost
|
255
|
+
|
256
|
+
# The maximum number of processing threads to run simultaneously. Each
|
257
|
+
# processing thread roughly equates to one ssh client running on your local
|
258
|
+
# machine. Default is 30. (optional)
|
259
|
+
attr_accessor :maxThreads
|
260
|
+
|
261
|
+
# Timeout for processing threads. Default is 30 minutes. (optional)
|
262
|
+
attr_accessor :threadTimeoutInSeconds
|
263
|
+
|
264
|
+
# Interval time to periodically display thread status information. Default is 30 sec. (optional)
|
265
|
+
attr_accessor :monitorPollIntervalInSeconds
|
266
|
+
|
267
|
+
# For internal use. Mutex for writing to the log
|
268
|
+
attr_reader :logmutex
|
269
|
+
|
270
|
+
# For internal use. Mutex for launching processing threads
|
271
|
+
attr_reader :threadStartMutex
|
272
|
+
|
273
|
+
# File logger. Set your own if desired. (optional)
|
274
|
+
attr_accessor :filelogger
|
275
|
+
|
276
|
+
# Turn on/off file logging. Default is on. (optional)
|
277
|
+
attr_accessor :isFileLogging
|
278
|
+
|
279
|
+
# For internal use. Script start time
|
280
|
+
attr_reader :startTime
|
281
|
+
|
282
|
+
# Turn on/off behavior where processing for a server is stopped if any
|
283
|
+
# command (run, runSudo, etc) returns with a non-zero exit code. Default is
|
284
|
+
# off (don't stop). Note that if isStopOnNonZeroExit is true then the
|
285
|
+
# processing thread exits immediately, you never have the opportunity to
|
286
|
+
# check the exit status in your processing code.
|
287
|
+
attr_accessor :isStopOnNonZeroExit
|
288
|
+
|
289
|
+
def initialize(hostlist, processingclass)
|
290
|
+
@hostlist = hostlist
|
291
|
+
@processingclass = processingclass
|
292
|
+
@logmutex = Mutex.new
|
293
|
+
@threadStartMutex = Mutex.new
|
294
|
+
@maxThreads = 30 # allow NN processing threads to run at any time
|
295
|
+
@threadTimeoutInSeconds = 1800 # timeout after 30 minutes
|
296
|
+
@monitorPollIntervalInSeconds = 30 # period for displaying thread monitor
|
297
|
+
timestr = Time.now.strftime("%Y%m%d%H%M%S")
|
298
|
+
logfilename = File.basename($0) + ".#{timestr}.log"
|
299
|
+
@filelogger = Logger.new(logfilename)
|
300
|
+
@isFileLogging = true
|
301
|
+
@isStopOnNonZeroExit = false
|
302
|
+
end
|
303
|
+
|
304
|
+
# Start the processing. In most cases, this call will be the last line in
|
305
|
+
# your SSHotgun processing script.
|
306
|
+
#
|
307
|
+
def start
|
308
|
+
# trap ctrl-c, show stats and exit
|
309
|
+
trap("INT") {
|
310
|
+
log "[_sshotgun_] info: Script terminated early by ctrl-c"
|
311
|
+
showRunStats
|
312
|
+
exit
|
313
|
+
}
|
314
|
+
@startTime = Time.now
|
315
|
+
log "[_sshotgun_] info: Started script " + $0 + " at " + @startTime.to_s
|
316
|
+
|
317
|
+
# Display configuration info
|
318
|
+
log "[_sshotgun_] info: User=" + ENV["USER"].to_s
|
319
|
+
log "[_sshotgun_] info: Gateway host=" + @gatewayhost.to_s
|
320
|
+
log "[_sshotgun_] info: Thread timeout=" + @threadTimeoutInSeconds.to_s + "s"
|
321
|
+
log "[_sshotgun_] info: Max concurrent threads=" + @maxThreads.to_s
|
322
|
+
log "[_sshotgun_] info: Monitor poll interval=" + @monitorPollIntervalInSeconds.to_s + "s"
|
323
|
+
log "[_sshotgun_] info: File logging is=" + @isFileLogging.to_s
|
324
|
+
log "[_sshotgun_] info: Will stop thread upon non-zero exit=" + @isStopOnNonZeroExit.to_s
|
325
|
+
|
326
|
+
hostarr = []
|
327
|
+
@hostlist.each do |hostdef|
|
328
|
+
hostarr << hostdef.hostname
|
329
|
+
end
|
330
|
+
log "[_sshotgun_] info: Host list=" + hostarr.join(", ")
|
331
|
+
|
332
|
+
currentHostIndex = 0
|
333
|
+
processingThreads = []
|
334
|
+
checkRunningThreadsCounter = 0
|
335
|
+
while true
|
336
|
+
|
337
|
+
# how many active threads now?
|
338
|
+
numThreadsAlive = 0
|
339
|
+
processingThreads.each { |aThread|
|
340
|
+
if aThread.alive?
|
341
|
+
numThreadsAlive += 1
|
342
|
+
end
|
343
|
+
}
|
344
|
+
|
345
|
+
# calc number of threads to launch with this
|
346
|
+
# pass through the loop
|
347
|
+
numThreadsToLaunch = @maxThreads - numThreadsAlive
|
348
|
+
numThreadsLeftToLaunch = @hostlist.length - currentHostIndex
|
349
|
+
if numThreadsToLaunch > numThreadsLeftToLaunch
|
350
|
+
numThreadsToLaunch = numThreadsLeftToLaunch
|
351
|
+
end
|
352
|
+
|
353
|
+
# from current index in hosts array loop and launch threads up to maxThreads running
|
354
|
+
i = 0
|
355
|
+
while i < numThreadsToLaunch do
|
356
|
+
# I hate to admit it but I don't know why i needed to synchronize this block
|
357
|
+
# Without it, some threads seemed to freeze at startup
|
358
|
+
@threadStartMutex.synchronize {
|
359
|
+
hostdef = @hostlist[currentHostIndex]
|
360
|
+
processingThreads << Thread.new {
|
361
|
+
aThread = Thread.current
|
362
|
+
aThread["threadstarttime"] = Time.now # store start time in threadlocal variable
|
363
|
+
aThread["hostname"] = hostdef.hostname # store hostname time in threadlocal variable
|
364
|
+
log "[_sshotgun_] info: Started thread for host = " + hostdef.hostname + " at " + aThread["threadstarttime"].to_s
|
365
|
+
begin
|
366
|
+
hostdef.startedProcess = true
|
367
|
+
o = @processingclass.new(self, hostdef)
|
368
|
+
o.doProcessing
|
369
|
+
hostdef.finishedProcess = true
|
370
|
+
rescue => ex
|
371
|
+
log "[_sshotgun_] stderr: Thread blew up for host = " + hostdef.hostname + ". ex = " + ex.inspect + "\n" + ex.backtrace.join("\n")
|
372
|
+
end
|
373
|
+
}
|
374
|
+
|
375
|
+
# bump loop counter
|
376
|
+
i += 1
|
377
|
+
currentHostIndex += 1
|
378
|
+
}
|
379
|
+
end
|
380
|
+
|
381
|
+
# kill expired threads
|
382
|
+
processingThreads.each { |aThread|
|
383
|
+
if aThread.alive?
|
384
|
+
# ensure that the threadlocal var is actually set before
|
385
|
+
# doing a calc with it. This is race condition where you
|
386
|
+
# get here so fast and threadstarttime is nil
|
387
|
+
if aThread["threadstarttime"]
|
388
|
+
if Time.now - aThread["threadstarttime"] > @threadTimeoutInSeconds
|
389
|
+
log "[_sshotgun_] info: Killed thread for host = " + aThread["hostname"]
|
390
|
+
aThread.kill
|
391
|
+
end
|
392
|
+
end
|
393
|
+
end
|
394
|
+
}
|
395
|
+
|
396
|
+
# display list of running threads as visual indicator to user
|
397
|
+
checkRunningThreadsCounter += 1
|
398
|
+
if checkRunningThreadsCounter > @monitorPollIntervalInSeconds
|
399
|
+
# display running threads
|
400
|
+
processingThreads.each { |aThread|
|
401
|
+
if aThread.alive?
|
402
|
+
log "[_sshotgun_] monitor: Thread still running for host = " + aThread["hostname"] + ". Elapsed time is " + (Time.now - aThread["threadstarttime"] ).to_s + "s"
|
403
|
+
end
|
404
|
+
}
|
405
|
+
checkRunningThreadsCounter = 0
|
406
|
+
end
|
407
|
+
|
408
|
+
# any processing threads still alive?
|
409
|
+
aThreadIsAlive = false
|
410
|
+
processingThreads.each { |aThread|
|
411
|
+
if aThread.alive?
|
412
|
+
aThreadIsAlive = true
|
413
|
+
break
|
414
|
+
end
|
415
|
+
}
|
416
|
+
|
417
|
+
# break out of this main loop if no processing threads are alive
|
418
|
+
if aThreadIsAlive
|
419
|
+
sleep 1
|
420
|
+
else
|
421
|
+
break
|
422
|
+
end
|
423
|
+
end
|
424
|
+
|
425
|
+
# ended normally, show stats
|
426
|
+
log "[_sshotgun_] info: Script terminated normally"
|
427
|
+
showRunStats
|
428
|
+
end
|
429
|
+
|
430
|
+
# Display these status at end of script or when control-c is pressed
|
431
|
+
def showRunStats
|
432
|
+
# Display script ending message
|
433
|
+
stopTime = Time.now
|
434
|
+
log "[_sshotgun_] info: Ended script " + $0 + " at " + stopTime.to_s
|
435
|
+
log "[_sshotgun_] info: Elapsed script " + $0 + " time " + (stopTime - @startTime).to_s + "s"
|
436
|
+
|
437
|
+
hostarr = []
|
438
|
+
@hostlist.each do |hostdef|
|
439
|
+
if hostdef.startedProcess == false
|
440
|
+
# did not start processing
|
441
|
+
hostarr << (hostdef.hostname + "(np)")
|
442
|
+
else
|
443
|
+
# started processing
|
444
|
+
if hostdef.hadNonZeroExitStatus
|
445
|
+
# one or more exit status were non-zero
|
446
|
+
hostarr << (hostdef.hostname + "(*)")
|
447
|
+
else
|
448
|
+
if hostdef.finishedProcess
|
449
|
+
# finished with no errors during processing
|
450
|
+
hostarr << hostdef.hostname
|
451
|
+
else
|
452
|
+
# Incomplete processing. This usually means that the method threw
|
453
|
+
# an exception
|
454
|
+
hostarr << (hostdef.hostname + "(i)")
|
455
|
+
end
|
456
|
+
end
|
457
|
+
end
|
458
|
+
end
|
459
|
+
log "[_sshotgun_] info: Status codes are '*'=had one|more non-zero exit status, 'i'=incomplete, 'np'=not started, blank=finished"
|
460
|
+
log "[_sshotgun_] info: Hosts summary = " + hostarr.join(", ")
|
461
|
+
end
|
462
|
+
|
463
|
+
# Log a string to the console and to the file logger
|
464
|
+
#
|
465
|
+
# s is the string to log
|
466
|
+
#
|
467
|
+
def log(s)
|
468
|
+
@logmutex.synchronize {
|
469
|
+
puts s.chomp
|
470
|
+
if @isFileLogging
|
471
|
+
@filelogger.info s.chomp
|
472
|
+
end
|
473
|
+
}
|
474
|
+
end
|
475
|
+
end
|
metadata
ADDED
@@ -0,0 +1,76 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: sshotgun
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 1.0.4
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Vick Perry
|
8
|
+
autorequire:
|
9
|
+
bindir: bin
|
10
|
+
cert_chain: []
|
11
|
+
|
12
|
+
date: 2008-09-08 00:00:00 -07:00
|
13
|
+
default_executable:
|
14
|
+
dependencies:
|
15
|
+
- !ruby/object:Gem::Dependency
|
16
|
+
name: open4
|
17
|
+
version_requirement:
|
18
|
+
version_requirements: !ruby/object:Gem::Requirement
|
19
|
+
requirements:
|
20
|
+
- - ">="
|
21
|
+
- !ruby/object:Gem::Version
|
22
|
+
version: 0.9.6
|
23
|
+
version:
|
24
|
+
- !ruby/object:Gem::Dependency
|
25
|
+
name: highline
|
26
|
+
version_requirement:
|
27
|
+
version_requirements: !ruby/object:Gem::Requirement
|
28
|
+
requirements:
|
29
|
+
- - ">="
|
30
|
+
- !ruby/object:Gem::Version
|
31
|
+
version: 1.4.0
|
32
|
+
version:
|
33
|
+
description:
|
34
|
+
email: vick.perry @nospam@ gmail.com
|
35
|
+
executables: []
|
36
|
+
|
37
|
+
extensions: []
|
38
|
+
|
39
|
+
extra_rdoc_files:
|
40
|
+
- README.rdoc
|
41
|
+
files:
|
42
|
+
- lib/sshotgun.rb
|
43
|
+
- README.rdoc
|
44
|
+
has_rdoc: true
|
45
|
+
homepage: http://sshotgun.rubyforge.org
|
46
|
+
post_install_message:
|
47
|
+
rdoc_options:
|
48
|
+
- --line-numbers
|
49
|
+
- --inline-source
|
50
|
+
- --title
|
51
|
+
- sshotgun
|
52
|
+
- --main
|
53
|
+
- README.rdoc
|
54
|
+
require_paths:
|
55
|
+
- lib
|
56
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
57
|
+
requirements:
|
58
|
+
- - ">="
|
59
|
+
- !ruby/object:Gem::Version
|
60
|
+
version: "0"
|
61
|
+
version:
|
62
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
63
|
+
requirements:
|
64
|
+
- - ">="
|
65
|
+
- !ruby/object:Gem::Version
|
66
|
+
version: "0"
|
67
|
+
version:
|
68
|
+
requirements: []
|
69
|
+
|
70
|
+
rubyforge_project: sshotgun
|
71
|
+
rubygems_version: 1.1.1
|
72
|
+
signing_key:
|
73
|
+
specification_version: 2
|
74
|
+
summary: A lib for writing server management scripts in Ruby. Useful for managing lots of servers.
|
75
|
+
test_files: []
|
76
|
+
|