kostya-bluepill 0.0.60.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (42) hide show
  1. data/.gitignore +10 -0
  2. data/.rspec +1 -0
  3. data/DESIGN.md +10 -0
  4. data/Gemfile +10 -0
  5. data/LICENSE +22 -0
  6. data/README.md +305 -0
  7. data/Rakefile +38 -0
  8. data/bin/bluepill +104 -0
  9. data/bluepill.gemspec +37 -0
  10. data/examples/example.rb +87 -0
  11. data/examples/new_example.rb +89 -0
  12. data/examples/new_runit_example.rb +29 -0
  13. data/examples/runit_example.rb +26 -0
  14. data/lib/bluepill.rb +38 -0
  15. data/lib/bluepill/application.rb +201 -0
  16. data/lib/bluepill/application/client.rb +8 -0
  17. data/lib/bluepill/application/server.rb +23 -0
  18. data/lib/bluepill/condition_watch.rb +50 -0
  19. data/lib/bluepill/controller.rb +110 -0
  20. data/lib/bluepill/dsl.rb +12 -0
  21. data/lib/bluepill/dsl/app_proxy.rb +25 -0
  22. data/lib/bluepill/dsl/process_factory.rb +122 -0
  23. data/lib/bluepill/dsl/process_proxy.rb +44 -0
  24. data/lib/bluepill/group.rb +72 -0
  25. data/lib/bluepill/process.rb +480 -0
  26. data/lib/bluepill/process_conditions.rb +14 -0
  27. data/lib/bluepill/process_conditions/always_true.rb +18 -0
  28. data/lib/bluepill/process_conditions/cpu_usage.rb +19 -0
  29. data/lib/bluepill/process_conditions/http.rb +58 -0
  30. data/lib/bluepill/process_conditions/mem_usage.rb +32 -0
  31. data/lib/bluepill/process_conditions/process_condition.rb +22 -0
  32. data/lib/bluepill/process_statistics.rb +27 -0
  33. data/lib/bluepill/socket.rb +58 -0
  34. data/lib/bluepill/system.rb +236 -0
  35. data/lib/bluepill/trigger.rb +59 -0
  36. data/lib/bluepill/triggers/flapping.rb +56 -0
  37. data/lib/bluepill/util/rotational_array.rb +20 -0
  38. data/lib/bluepill/version.rb +4 -0
  39. data/spec/lib/bluepill/process_statistics_spec.rb +24 -0
  40. data/spec/lib/bluepill/system_spec.rb +36 -0
  41. data/spec/spec_helper.rb +19 -0
  42. metadata +304 -0
data/.gitignore ADDED
@@ -0,0 +1,10 @@
1
+ *.sw?
2
+ .DS_Store
3
+ coverage/
4
+ .yardoc/
5
+ doc/
6
+ pkg/
7
+ *.gem
8
+ .bundle/
9
+ Gemfile.lock
10
+ .idea
data/.rspec ADDED
@@ -0,0 +1 @@
1
+ -r ./spec/spec_helper.rb -c -f progress
data/DESIGN.md ADDED
@@ -0,0 +1,10 @@
1
+ ## Bluepill Design
2
+ Here are just some bullet points of the design. We'll add details later.
3
+
4
+ * Each process monitors a single _application_, so you can have multiple bluepill processes on a system
5
+ * Use rotational arrays for storing historical data for monitoring process conditions
6
+ * Memo-ize output of _ps_ per tick as an optimization for applications with many processes
7
+ * Use socket files to communicate between CLI and daemon
8
+ * DSL is a separate layer, the core of the monitoring just uses regular initializers, etc. DSL is simply for ease of use and should not interfere with business logic
9
+ * Sequentially process user issued commands so no weird race cases occur
10
+ * Triggers are notified by the state machine on any process state transitions
data/Gemfile ADDED
@@ -0,0 +1,10 @@
1
+ source :rubygems
2
+
3
+ # Specify your gem's dependencies in bluepill.gemspec
4
+ gemspec
5
+
6
+ # YARD helper for ruby 1.8 (already embedded into ruby 1.9)
7
+ gem "ripper", :platforms => :ruby_18, :group => :development
8
+
9
+ # Code coverage tool that works on Ruby 1.9
10
+ gem "simplecov", ">= 0.4.0", :platforms => :ruby_19, :group => :test
data/LICENSE ADDED
@@ -0,0 +1,22 @@
1
+ Copyright (c) 2009 Arya Asemanfar, Rohith Ravi, Gary Tsang
2
+
3
+ Permission is hereby granted, free of charge, to any person
4
+ obtaining a copy of this software and associated documentation
5
+ files (the "Software"), to deal in the Software without
6
+ restriction, including without limitation the rights to use,
7
+ copy, modify, merge, publish, distribute, sublicense, and/or sell
8
+ copies of the Software, and to permit persons to whom the
9
+ Software is furnished to do so, subject to the following
10
+ conditions:
11
+
12
+ The above copyright notice and this permission notice shall be
13
+ included in all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
16
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
17
+ OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
18
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
19
+ HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
20
+ WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
21
+ FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
22
+ OTHER DEALINGS IN THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,305 @@
1
+ # Bluepill
2
+ Bluepill is a simple process monitoring tool written in Ruby.
3
+
4
+ ## Installation
5
+ It's hosted on [rubygems.org][rubygems].
6
+
7
+ sudo gem install bluepill
8
+
9
+ In order to take advantage of logging with syslog, you also need to setup your syslog to log the local6 facility. Edit the appropriate config file for your syslogger (/etc/syslog.conf for syslog) and add a line for local6:
10
+
11
+ local6.* /var/log/bluepill.log
12
+
13
+ You'll also want to add _/var/log/bluepill.log_ to _/etc/logrotate.d/syslog_ so that it gets rotated.
14
+
15
+ Lastly, create the _/var/run/bluepill_ directory for bluepill to store its pid and sock files.
16
+
17
+ ## Usage
18
+ ### Config
19
+ Bluepill organizes processes into 3 levels: application -> group -> process. Each process has a few attributes that tell bluepill how to start, stop, and restart it, where to look or put the pid file, what process conditions to monitor and the options for each of those.
20
+
21
+ The minimum config file looks something like this:
22
+
23
+ ```ruby
24
+ Bluepill.application("app_name") do |app|
25
+ app.process("process_name") do |process|
26
+ process.start_command = "/usr/bin/some_start_command"
27
+ process.pid_file = "/tmp/some_pid_file.pid"
28
+ end
29
+ end
30
+ ```
31
+
32
+ Note that since we specified a PID file and start command, bluepill assumes the process will daemonize itself. If we wanted bluepill to daemonize it for us, we can do (note we still need to specify a PID file):
33
+
34
+ ```ruby
35
+ Bluepill.application("app_name") do |app|
36
+ app.process("process_name") do |process|
37
+ process.start_command = "/usr/bin/some_start_command"
38
+ process.pid_file = "/tmp/some_pid_file.pid"
39
+ process.daemonize = true
40
+ end
41
+ end
42
+ ```
43
+
44
+ If you don't specify a stop command, a TERM signal will be sent by default. Similarly, the default restart action is to issue stop and then start.
45
+
46
+ Now if we want to do something more meaningful, like actually monitor the process, we do:
47
+
48
+ ```ruby
49
+ Bluepill.application("app_name") do |app|
50
+ app.process("process_name") do |process|
51
+ process.start_command = "/usr/bin/some_start_command"
52
+ process.pid_file = "/tmp/some_pid_file.pid"
53
+
54
+ process.checks :cpu_usage, :every => 10.seconds, :below => 5, :times => 3
55
+ end
56
+ end
57
+ ```
58
+
59
+ We added a line that checks every 10 seconds to make sure the cpu usage of this process is below 5 percent; 3 failed checks results in a restart. We can specify a two-element array for the _times_ option to say that it 3 out of 5 failed attempts results in a restart.
60
+
61
+ To watch memory usage, we just add one more line:
62
+
63
+ ```ruby
64
+ Bluepill.application("app_name") do |app|
65
+ app.process("process_name") do |process|
66
+ process.start_command = "/usr/bin/some_start_command"
67
+ process.pid_file = "/tmp/some_pid_file.pid"
68
+
69
+ process.checks :cpu_usage, :every => 10.seconds, :below => 5, :times => 3
70
+ process.checks :mem_usage, :every => 10.seconds, :below => 100.megabytes, :times => [3,5]
71
+ end
72
+ end
73
+ ```
74
+
75
+ We can tell bluepill to give a process some grace time to start/stop/restart before resuming monitoring:
76
+
77
+ ```ruby
78
+ Bluepill.application("app_name") do |app|
79
+ app.process("process_name") do |process|
80
+ process.start_command = "/usr/bin/some_start_command"
81
+ process.pid_file = "/tmp/some_pid_file.pid"
82
+ process.start_grace_time = 3.seconds
83
+ process.stop_grace_time = 5.seconds
84
+ process.restart_grace_time = 8.seconds
85
+
86
+ process.checks :cpu_usage, :every => 10.seconds, :below => 5, :times => 3
87
+ process.checks :mem_usage, :every => 10.seconds, :below => 100.megabytes, :times => [3,5]
88
+ end
89
+ end
90
+ ```
91
+
92
+ We can group processes by name:
93
+
94
+ ```ruby
95
+ Bluepill.application("app_name") do |app|
96
+ 5.times do |i|
97
+ app.process("process_name_#{i}") do |process|
98
+ process.group = "mongrels"
99
+ process.start_command = "/usr/bin/some_start_command"
100
+ process.pid_file = "/tmp/some_pid_file.pid"
101
+ end
102
+ end
103
+ end
104
+ ```
105
+
106
+ If you want to run the process as someone other than root:
107
+
108
+ ```ruby
109
+ Bluepill.application("app_name") do |app|
110
+ app.process("process_name") do |process|
111
+ process.start_command = "/usr/bin/some_start_command"
112
+ process.pid_file = "/tmp/some_pid_file.pid"
113
+ process.uid = "deploy"
114
+ process.gid = "deploy"
115
+
116
+ process.checks :cpu_usage, :every => 10.seconds, :below => 5, :times => 3
117
+ process.checks :mem_usage, :every => 10.seconds, :below => 100.megabytes, :times => [3,5]
118
+ end
119
+ end
120
+ ```
121
+
122
+ If you want to include one or more supplementary groups:
123
+
124
+ ```ruby
125
+ Bluepill.application("app_name") do |app|
126
+ app.process("process_name") do |process|
127
+ process.start_command = "/usr/bin/some_start_command"
128
+ process.pid_file = "/tmp/some_pid_file.pid"
129
+ process.uid = "deploy"
130
+ process.gid = "deploy"
131
+ process.supplementary_groups = ['rvm']
132
+
133
+ process.checks :cpu_usage, :every => 10.seconds, :below => 5, :times => 3
134
+ process.checks :mem_usage, :every => 10.seconds, :below => 100.megabytes, :times => [3,5]
135
+ end
136
+ end
137
+ ```
138
+
139
+ You can also set an app-wide uid/gid:
140
+
141
+ ```ruby
142
+ Bluepill.application("app_name") do |app|
143
+ app.uid = "deploy"
144
+ app.gid = "deploy"
145
+ app.process("process_name") do |process|
146
+ process.start_command = "/usr/bin/some_start_command"
147
+ process.pid_file = "/tmp/some_pid_file.pid"
148
+ end
149
+ end
150
+ ```
151
+
152
+ To check for flapping:
153
+
154
+ ```ruby
155
+ process.checks :flapping, :times => 2, :within => 30.seconds, :retry_in => 7.seconds
156
+ ```
157
+
158
+ To set the working directory to _cd_ into when starting the command:
159
+
160
+ ```ruby
161
+ Bluepill.application("app_name") do |app|
162
+ app.process("process_name") do |process|
163
+ process.start_command = "/usr/bin/some_start_command"
164
+ process.pid_file = "/tmp/some_pid_file.pid"
165
+ process.working_dir = "/path/to/some_directory"
166
+ end
167
+ end
168
+ ```
169
+
170
+ You can also have an app-wide working directory:
171
+
172
+ ```ruby
173
+ Bluepill.application("app_name") do |app|
174
+ app.working_dir = "/path/to/some_directory"
175
+ app.process("process_name") do |process|
176
+ process.start_command = "/usr/bin/some_start_command"
177
+ process.pid_file = "/tmp/some_pid_file.pid"
178
+ end
179
+ end
180
+ ```
181
+
182
+ Note: We also set the PWD in the environment to the working dir you specify. This is useful for when the working dir is a symlink. Unicorn in particular will cd into the environment variable in PWD when it re-execs to deal with a change in the symlink.
183
+
184
+ By default, bluepill will send a SIGTERM to your process when stopping.
185
+ To change the stop command:
186
+
187
+ ```ruby
188
+ Bluepill.application("app_name") do |app|
189
+ app.process("process_name") do |process|
190
+ process.start_command = "/usr/bin/some_start_command"
191
+ process.pid_file = "/tmp/some_pid_file.pid"
192
+ process.stop_command = "/user/bin/some_stop_command"
193
+ end
194
+ end
195
+ ```
196
+
197
+ If you'd like to send a signal or signals to your process to stop it:
198
+
199
+ ```ruby
200
+ Bluepill.application("app_name") do |app|
201
+ app.process("process_name") do |process|
202
+ process.start_command = "/usr/bin/some_start_command"
203
+ process.pid_file = "/tmp/some_pid_file.pid"
204
+ process.stop_signals = [:quit, 30.seconds, :term, 5.seconds, :kill]
205
+ end
206
+ end
207
+ ```
208
+
209
+ We added a line that will send a SIGQUIT, wait 30 seconds and check to
210
+ see if the process is still up, send a SIGTERM, wait 5 seconds and check
211
+ to see if the process is still up, and finally send a SIGKILL.
212
+
213
+ And lastly, to monitor child processes:
214
+
215
+ ```ruby
216
+ process.monitor_children do |child_process|
217
+ child_process.checks :cpu_usage, :every => 10, :below => 5, :times => 3
218
+ child_process.checks :mem_usage, :every => 10, :below => 100.megabytes, :times => [3, 5]
219
+
220
+ child_process.stop_command = "kill -QUIT {{PID}}"
221
+ end
222
+ ```
223
+
224
+ Note {{PID}} will be substituted for the pid of process in both the stop and restart commands.
225
+
226
+ ### A Note About Output Redirection
227
+
228
+ While you can specify shell tricks like the following in the start_command of a process:
229
+
230
+ ```ruby
231
+ Bluepill.application("app_name") do |app|
232
+ app.process("process_name") do |process|
233
+ process.start_command = "cd /tmp/some_dir && SOME_VAR=1 /usr/bin/some_start_command > /tmp/server.log 2>&1"
234
+ process.pid_file = "/tmp/some_pid_file.pid"
235
+ end
236
+ end
237
+ ```
238
+
239
+ We recommend that you _not_ do that and instead use the config options to capture output from your daemons. Like so:
240
+
241
+ ```ruby
242
+ Bluepill.application("app_name") do |app|
243
+ app.process("process_name") do |process|
244
+ process.start_command = "/usr/bin/env SOME_VAR=1 /usr/bin/some_start_command"
245
+
246
+ process.working_dir = "/tmp/some_dir"
247
+ process.stdout = process.stderr = "/tmp/server.log"
248
+
249
+ process.pid_file = "/tmp/some_pid_file.pid"
250
+ end
251
+ end
252
+ ```
253
+
254
+ The main benefit of using the config options is that Bluepill will be able to monitor the correct process instead of just watching the shell that spawned your actual server.
255
+
256
+ ### CLI
257
+ To start a bluepill process and load a config:
258
+
259
+ sudo bluepill load /path/to/production.pill
260
+
261
+ To act on a process or group:
262
+
263
+ sudo bluepill <start|stop|restart|unmonitor> <process_or_group_name>
264
+
265
+ To view process statuses:
266
+
267
+ sudo bluepill status
268
+
269
+ To view the log for a process or group:
270
+
271
+ sudo bluepill log <process_or_group_name>
272
+
273
+ To quit bluepill:
274
+
275
+ sudo bluepill quit
276
+
277
+ ### Logging
278
+ By default, bluepill uses syslog local6 facility as described in the installation section. But if for any reason you don&apos;t want to use syslog, you can use a log file. You can do this by setting the :log\_file option in the config:
279
+
280
+ ```ruby
281
+ Bluepill.application("app_name", :log_file => "/path/to/bluepill.log") do |app|
282
+ # ...
283
+ end
284
+ ```
285
+
286
+ Keep in mind that you still need to set up log rotation (described in the installation section) to keep the log file from growing huge.
287
+
288
+ ### Extra options
289
+ You can run bluepill in the foreground:
290
+
291
+ ```ruby
292
+ Bluepill.application("app_name", :foreground => true) do |app|
293
+ # ...
294
+ end
295
+ ```
296
+
297
+ Note that You must define only one application per config when using foreground mode.
298
+
299
+ ## Links
300
+ Code: [http://github.com/arya/bluepill](http://github.com/arya/bluepill)
301
+ Bugs/Features: [http://github.com/arya/bluepill/issues](http://github.com/arya/bluepill/issues)
302
+ Mailing List: [http://groups.google.com/group/bluepill-rb](http://groups.google.com/group/bluepill-rb)
303
+
304
+
305
+ [rubygems]: http://rubygems.org/gems/bluepill
data/Rakefile ADDED
@@ -0,0 +1,38 @@
1
+ # encoding: utf-8
2
+
3
+ begin
4
+ require 'bundler'
5
+ Bundler::GemHelper.install_tasks
6
+ rescue LoadError
7
+ $stderr.puts "Bundler not installed. You should install it with: gem install bundler"
8
+ end
9
+
10
+ $LOAD_PATH << File.expand_path('./lib', File.dirname(__FILE__))
11
+ require 'bluepill/version'
12
+
13
+ begin
14
+ require 'rspec/core/rake_task'
15
+
16
+ RSpec::Core::RakeTask.new
17
+
18
+ if RUBY_VERSION >= '1.9'
19
+ RSpec::Core::RakeTask.new(:cov) do |t|
20
+ ENV['ENABLE_SIMPLECOV'] = '1'
21
+ t.ruby_opts = '-w'
22
+ t.rcov_opts = %q[-Ilib --exclude "spec/*,gems/*"]
23
+ end
24
+ end
25
+ rescue LoadError
26
+ $stderr.puts "RSpec not available. Install it with: gem install rspec-core rspec-expectations rr faker"
27
+ end
28
+
29
+ begin
30
+ require 'yard'
31
+ YARD::Rake::YardocTask.new do |yard|
32
+ yard.options << "--title='bluepill #{Bluepill::VERSION}'"
33
+
34
+ end
35
+ rescue LoadError
36
+ $stderr.puts "Please install YARD with: gem install yard"
37
+ end
38
+
data/bin/bluepill ADDED
@@ -0,0 +1,104 @@
1
+ #! /usr/bin/env ruby
2
+ require 'optparse'
3
+ require 'bluepill'
4
+
5
+ begin
6
+ require 'rbconfig'
7
+ rescue LoadError
8
+ end
9
+
10
+ RbConfig = Config unless Object.const_defined?(:RbConfig)
11
+
12
+ # Default options
13
+ options = {
14
+ :base_dir => File.join(ENV['HOME'], '.bluepill'),
15
+ :timeout => 10,
16
+ :attempts => 1
17
+ }
18
+
19
+ OptionParser.new do |opts|
20
+ opts.banner = "Usage: bluepill [app] cmd [options]"
21
+ opts.on('-c', "--base-dir DIR", "Directory to store bluepill socket and pid files, defaults to #{options[:base_dir]}") do |base_dir|
22
+ options[:base_dir] = base_dir
23
+ end
24
+
25
+ opts.on("-v", "--version") do
26
+ puts "bluepill, version #{Bluepill::VERSION}"
27
+ exit
28
+ end
29
+
30
+ opts.on('-t', '--timeout Seconds', Integer, "Timeout for commands sent to the daemon, in seconds. Defaults to 10.") do |timeout|
31
+ options[:timeout] = timeout
32
+ end
33
+
34
+ opts.on('--attempts Count', Integer, "Attempts for commands sent to the daemon, in seconds. Defaults to 1.") do |attempts|
35
+ options[:attempts] = attempts
36
+ end
37
+
38
+ help = proc do
39
+ puts opts
40
+ puts
41
+ puts "Commands:"
42
+ puts " load CONFIG_FILE\t\tLoads new instance of bluepill using the specified config file"
43
+ puts " status\t\t\tLists the status of the proceses for the specified app"
44
+ puts " start [TARGET]\t\tIssues the start command for the target process or group, defaults to all processes"
45
+ puts " stop [TARGET]\t\tIssues the stop command for the target process or group, defaults to all processes"
46
+ puts " restart [TARGET]\t\tIssues the restart command for the target process or group, defaults to all processes"
47
+ puts " unmonitor [TARGET]\t\tStop monitoring target process or group, defaults to all processes"
48
+ puts " quit\t\t\tStop bluepill"
49
+ puts
50
+ puts "See http://github.com/arya/bluepill for README"
51
+ exit
52
+ end
53
+
54
+ opts.on_tail('-h','--help', 'Show this message', &help)
55
+ help.call if ARGV.empty?
56
+ end.parse!
57
+
58
+ APPLICATION_COMMANDS = %w(status start stop restart unmonitor quit)
59
+
60
+ controller = Bluepill::Controller.new(options.slice(:base_dir))
61
+
62
+ if controller.running_applications.include?(File.basename($0)) && File.symlink?($0)
63
+ # bluepill was called as a symlink with the name of the target application
64
+ options[:application] = File.basename($0)
65
+ elsif controller.running_applications.include?(ARGV.first)
66
+ # the first arg is the application name
67
+ options[:application] = ARGV.shift
68
+ elsif APPLICATION_COMMANDS.include?(ARGV.first)
69
+ if controller.running_applications.length == 1
70
+ # there is only one, let's just use that
71
+ options[:application] = controller.running_applications.first
72
+ elsif controller.running_applications.length > 1
73
+ # There is more than one, tell them the list and exit
74
+ $stderr.puts "You must specify an application name to run that command. Here's the list of running applications:"
75
+ controller.running_applications.each_with_index do |app, index|
76
+ $stderr.puts " #{index + 1}. #{app}"
77
+ end
78
+ $stderr.puts "Usage: bluepill [app] cmd [options]"
79
+ exit(1)
80
+ else
81
+ # There are none running AND they aren't trying to start one
82
+ $stderr.puts "Error: There are no running bluepill daemons.\nTo start a bluepill daemon, use: bluepill load <config file>"
83
+ exit(2)
84
+ end
85
+ end
86
+
87
+ options[:command] = ARGV.shift
88
+
89
+ if options[:command] == "load"
90
+ file = ARGV.shift
91
+ if File.exists?(file)
92
+ # Restart the ruby interpreter for the config file so that anything loaded here
93
+ # does not stay in memory for the daemon
94
+ ruby = File.join(RbConfig::CONFIG['bindir'], RbConfig::CONFIG['ruby_install_name'])
95
+ load_path = File.expand_path("#{File.dirname(__FILE__)}/../lib")
96
+ file_path = File.expand_path(file)
97
+ exec(ruby, "-I#{load_path}", '-rbluepill', file_path)
98
+ else
99
+ $stderr.puts "Can't find file: #{file}"
100
+ end
101
+ else
102
+ target = ARGV.shift
103
+ controller.handle_command(options[:application], options[:command], target)
104
+ end