ryansch-bluepill 0.0.53

Sign up to get free protection for your applications and to get access to all the features.
Files changed (41) hide show
  1. data/.gitignore +8 -0
  2. data/DESIGN.md +10 -0
  3. data/Gemfile +4 -0
  4. data/LICENSE +22 -0
  5. data/README.md +306 -0
  6. data/Rakefile +9 -0
  7. data/bin/bluepill +111 -0
  8. data/bin/bpsv +3 -0
  9. data/bin/sample_forking_server +53 -0
  10. data/bluepill.gemspec +29 -0
  11. data/examples/example.rb +87 -0
  12. data/examples/new_example.rb +89 -0
  13. data/examples/new_runit_example.rb +29 -0
  14. data/examples/runit_example.rb +26 -0
  15. data/lib/bluepill.rb +36 -0
  16. data/lib/bluepill/application.rb +203 -0
  17. data/lib/bluepill/application/client.rb +8 -0
  18. data/lib/bluepill/application/server.rb +23 -0
  19. data/lib/bluepill/condition_watch.rb +50 -0
  20. data/lib/bluepill/controller.rb +121 -0
  21. data/lib/bluepill/dsl.rb +12 -0
  22. data/lib/bluepill/dsl/app_proxy.rb +25 -0
  23. data/lib/bluepill/dsl/process_factory.rb +122 -0
  24. data/lib/bluepill/dsl/process_proxy.rb +44 -0
  25. data/lib/bluepill/group.rb +72 -0
  26. data/lib/bluepill/logger.rb +63 -0
  27. data/lib/bluepill/process.rb +478 -0
  28. data/lib/bluepill/process_conditions.rb +14 -0
  29. data/lib/bluepill/process_conditions/always_true.rb +18 -0
  30. data/lib/bluepill/process_conditions/cpu_usage.rb +19 -0
  31. data/lib/bluepill/process_conditions/http.rb +58 -0
  32. data/lib/bluepill/process_conditions/mem_usage.rb +32 -0
  33. data/lib/bluepill/process_conditions/process_condition.rb +22 -0
  34. data/lib/bluepill/process_statistics.rb +23 -0
  35. data/lib/bluepill/socket.rb +47 -0
  36. data/lib/bluepill/system.rb +235 -0
  37. data/lib/bluepill/trigger.rb +60 -0
  38. data/lib/bluepill/triggers/flapping.rb +56 -0
  39. data/lib/bluepill/util/rotational_array.rb +20 -0
  40. data/lib/bluepill/version.rb +4 -0
  41. metadata +192 -0
data/.gitignore ADDED
@@ -0,0 +1,8 @@
1
+ *.sw?
2
+ .DS_Store
3
+ coverage
4
+ rdoc
5
+ pkg/*
6
+ *.gem
7
+ .bundle
8
+ Gemfile.lock
data/DESIGN.md ADDED
@@ -0,0 +1,10 @@
1
+ ## Bluepill Design
2
+ Here are just some bullet points of the design. We'll add details later.
3
+
4
+ * Each process monitors a single _application_, so you can have multiple bluepill processes on a system
5
+ * Use rotational arrays for storing historical data for monitoring process conditions
6
+ * Memo-ize output of _ps_ per tick as an optimization for applications with many processes
7
+ * Use socket files to communicate between CLI and daemon
8
+ * DSL is a separate layer, the core of the monitoring just uses regular initializers, etc. DSL is simply for ease of use and should not interfere with business logic
9
+ * Sequentially process user issued commands so no weird race cases occur
10
+ * Triggers are notified by the state machine on any process state transitions
data/Gemfile ADDED
@@ -0,0 +1,4 @@
1
+ source :rubygems
2
+
3
+ # Specify your gem's dependencies in bluepill.gemspec
4
+ gemspec
data/LICENSE ADDED
@@ -0,0 +1,22 @@
1
+ Copyright (c) 2009 Arya Asemanfar, Rohith Ravi, Gary Tsang
2
+
3
+ Permission is hereby granted, free of charge, to any person
4
+ obtaining a copy of this software and associated documentation
5
+ files (the "Software"), to deal in the Software without
6
+ restriction, including without limitation the rights to use,
7
+ copy, modify, merge, publish, distribute, sublicense, and/or sell
8
+ copies of the Software, and to permit persons to whom the
9
+ Software is furnished to do so, subject to the following
10
+ conditions:
11
+
12
+ The above copyright notice and this permission notice shall be
13
+ included in all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
16
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
17
+ OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
18
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
19
+ HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
20
+ WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
21
+ FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
22
+ OTHER DEALINGS IN THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,306 @@
1
+ # Bluepill
2
+ Bluepill is a simple process monitoring tool written in Ruby.
3
+
4
+ ## Installation
5
+ It's hosted on [rubygems.org][rubygems].
6
+
7
+ sudo gem install bluepill
8
+
9
+ In order to take advantage of logging with syslog, you also need to setup your syslog to log the local6 facility. Edit the appropriate config file for your syslogger (/etc/syslog.conf for syslog) and add a line for local6:
10
+
11
+ local6.* /var/log/bluepill.log
12
+
13
+ You'll also want to add _/var/log/bluepill.log_ to _/etc/logrotate.d/syslog_ so that it gets rotated.
14
+
15
+ Lastly, create the _/var/bluepill_ directory for bluepill to store its pid and sock files.
16
+
17
+ ## Usage
18
+ ### Config
19
+ Bluepill organizes processes into 3 levels: application -> group -> process. Each process has a few attributes that tell bluepill how to start, stop, and restart it, where to look or put the pid file, what process conditions to monitor and the options for each of those.
20
+
21
+ The minimum config file looks something like this:
22
+
23
+ ```ruby
24
+ Bluepill.application("app_name") do |app|
25
+ app.process("process_name") do |process|
26
+ process.start_command = "/usr/bin/some_start_command"
27
+ process.pid_file = "/tmp/some_pid_file.pid"
28
+ end
29
+ end
30
+ ```
31
+
32
+ Note that since we specified a PID file and start command, bluepill assumes the process will daemonize itself. If we wanted bluepill to daemonize it for us, we can do (note we still need to specify a PID file):
33
+
34
+ ```ruby
35
+ Bluepill.application("app_name") do |app|
36
+ app.process("process_name") do |process|
37
+ process.start_command = "/usr/bin/some_start_command"
38
+ process.pid_file = "/tmp/some_pid_file.pid"
39
+ process.daemonize = true
40
+ end
41
+ end
42
+ ```
43
+
44
+ If you don't specify a stop command, a TERM signal will be sent by default. Similarly, the default restart action is to issue stop and then start.
45
+
46
+ Now if we want to do something more meaningful, like actually monitor the process, we do:
47
+
48
+ ```ruby
49
+ Bluepill.application("app_name") do |app|
50
+ app.process("process_name") do |process|
51
+ process.start_command = "/usr/bin/some_start_command"
52
+ process.pid_file = "/tmp/some_pid_file.pid"
53
+
54
+ process.checks :cpu_usage, :every => 10.seconds, :below => 5, :times => 3
55
+ end
56
+ end
57
+ ```
58
+
59
+ We added a line that checks every 10 seconds to make sure the cpu usage of this process is below 5 percent; 3 failed checks results in a restart. We can specify a two-element array for the _times_ option to say that it 3 out of 5 failed attempts results in a restart.
60
+
61
+ To watch memory usage, we just add one more line:
62
+
63
+ ```ruby
64
+ Bluepill.application("app_name") do |app|
65
+ app.process("process_name") do |process|
66
+ process.start_command = "/usr/bin/some_start_command"
67
+ process.pid_file = "/tmp/some_pid_file.pid"
68
+
69
+ process.checks :cpu_usage, :every => 10.seconds, :below => 5, :times => 3
70
+ process.checks :mem_usage, :every => 10.seconds, :below => 100.megabytes, :times => [3,5]
71
+ end
72
+ end
73
+ ```
74
+
75
+ We can tell bluepill to give a process some grace time to start/stop/restart before resuming monitoring:
76
+
77
+ ```ruby
78
+ Bluepill.application("app_name") do |app|
79
+ app.process("process_name") do |process|
80
+ process.start_command = "/usr/bin/some_start_command"
81
+ process.pid_file = "/tmp/some_pid_file.pid"
82
+ process.start_grace_time = 3.seconds
83
+ process.stop_grace_time = 5.seconds
84
+ process.restart_grace_time = 8.seconds
85
+
86
+ process.checks :cpu_usage, :every => 10.seconds, :below => 5, :times => 3
87
+ process.checks :mem_usage, :every => 10.seconds, :below => 100.megabytes, :times => [3,5]
88
+ end
89
+ end
90
+ ```
91
+
92
+ We can group processes by name:
93
+
94
+ ```ruby
95
+ Bluepill.application("app_name") do |app|
96
+ 5.times do |i|
97
+ app.process("process_name_#{i}") do |process|
98
+ process.group = "mongrels"
99
+ process.start_command = "/usr/bin/some_start_command"
100
+ process.pid_file = "/tmp/some_pid_file.pid"
101
+ end
102
+ end
103
+ end
104
+ ```
105
+
106
+ If you want to run the process as someone other than root:
107
+
108
+ ```ruby
109
+ Bluepill.application("app_name") do |app|
110
+ app.process("process_name") do |process|
111
+ process.start_command = "/usr/bin/some_start_command"
112
+ process.pid_file = "/tmp/some_pid_file.pid"
113
+ process.uid = "deploy"
114
+ process.gid = "deploy"
115
+
116
+ process.checks :cpu_usage, :every => 10.seconds, :below => 5, :times => 3
117
+ process.checks :mem_usage, :every => 10.seconds, :below => 100.megabytes, :times => [3,5]
118
+ end
119
+ end
120
+ ```
121
+
122
+ If you want to include one or more supplementary groups:
123
+
124
+ ```ruby
125
+ Bluepill.application("app_name") do |app|
126
+ app.process("process_name") do |process|
127
+ process.start_command = "/usr/bin/some_start_command"
128
+ process.pid_file = "/tmp/some_pid_file.pid"
129
+ process.uid = "deploy"
130
+ process.gid = "deploy"
131
+ process.supplementary_groups = ['rvm']
132
+
133
+ process.checks :cpu_usage, :every => 10.seconds, :below => 5, :times => 3
134
+ process.checks :mem_usage, :every => 10.seconds, :below => 100.megabytes, :times => [3,5]
135
+ end
136
+ end
137
+ ```
138
+
139
+ You can also set an app-wide uid/gid:
140
+
141
+ ```ruby
142
+ Bluepill.application("app_name") do |app|
143
+ app.uid = "deploy"
144
+ app.gid = "deploy"
145
+ app.process("process_name") do |process|
146
+ process.start_command = "/usr/bin/some_start_command"
147
+ process.pid_file = "/tmp/some_pid_file.pid"
148
+ end
149
+ end
150
+ ```
151
+
152
+ To check for flapping:
153
+
154
+ ```ruby
155
+ process.checks :flapping, :times => 2, :within => 30.seconds, :retry_in => 7.seconds
156
+ ```
157
+
158
+ To set the working directory to _cd_ into when starting the command:
159
+
160
+ ```ruby
161
+ Bluepill.application("app_name") do |app|
162
+ app.process("process_name") do |process|
163
+ process.start_command = "/usr/bin/some_start_command"
164
+ process.pid_file = "/tmp/some_pid_file.pid"
165
+ process.working_dir = "/path/to/some_directory"
166
+ end
167
+ end
168
+ ```
169
+
170
+ You can also have an app-wide working directory:
171
+
172
+ ```ruby
173
+ Bluepill.application("app_name") do |app|
174
+ app.working_dir = "/path/to/some_directory"
175
+ app.process("process_name") do |process|
176
+ process.start_command = "/usr/bin/some_start_command"
177
+ process.pid_file = "/tmp/some_pid_file.pid"
178
+ end
179
+ end
180
+ ```
181
+
182
+ Note: We also set the PWD in the environment to the working dir you specify. This is useful for when the working dir is a symlink. Unicorn in particular will cd into the environment variable in PWD when it re-execs to deal with a change in the symlink.
183
+
184
+ By default, bluepill will send a SIGTERM to your process when stopping.
185
+ To change the stop command:
186
+
187
+ ```ruby
188
+ Bluepill.application("app_name") do |app|
189
+ app.process("process_name") do |process|
190
+ process.start_command = "/usr/bin/some_start_command"
191
+ process.pid_file = "/tmp/some_pid_file.pid"
192
+ process.stop_command = "/user/bin/some_stop_command"
193
+ end
194
+ end
195
+ ```
196
+
197
+ If you'd like to send a signal or signals to your process to stop it:
198
+
199
+ ```ruby
200
+ Bluepill.application("app_name") do |app|
201
+ app.process("process_name") do |process|
202
+ process.start_command = "/usr/bin/some_start_command"
203
+ process.pid_file = "/tmp/some_pid_file.pid"
204
+ process.stop_signals = [:quit, 30.seconds, :term, 5.seconds, :kill]
205
+ end
206
+ end
207
+ ```
208
+
209
+ We added a line that will send a SIGQUIT, wait 30 seconds and check to
210
+ see if the process is still up, send a SIGTERM, wait 5 seconds and check
211
+ to see if the process is still up, and finally send a SIGKILL.
212
+
213
+ And lastly, to monitor child processes:
214
+
215
+ ```ruby
216
+ process.monitor_children do |child_process|
217
+ child_process.checks :cpu_usage, :every => 10, :below => 5, :times => 3
218
+ child_process.checks :mem_usage, :every => 10, :below => 100.megabytes, :times => [3, 5]
219
+
220
+ child_process.stop_command = "kill -QUIT {{PID}}"
221
+ end
222
+ ```
223
+
224
+ Note {{PID}} will be substituted for the pid of process in both the stop and restart commands.
225
+
226
+ ### A Note About Output Redirection
227
+
228
+ While you can specify shell tricks like the following in the start_command of a process:
229
+
230
+ ```ruby
231
+ Bluepill.application("app_name") do |app|
232
+ app.process("process_name") do |process|
233
+ process.start_command = "cd /tmp/some_dir && SOME_VAR=1 /usr/bin/some_start_command > /tmp/server.log 2>&1"
234
+ process.pid_file = "/tmp/some_pid_file.pid"
235
+ end
236
+ end
237
+ ```
238
+
239
+ We recommend that you _not_ do that and instead use the config options to capture output from your daemons. Like so:
240
+
241
+ ```ruby
242
+ Bluepill.application("app_name") do |app|
243
+ app.process("process_name") do |process|
244
+ process.start_command = "/usr/bin/env SOME_VAR=1 /usr/bin/some_start_command"
245
+
246
+ process.working_dir = "/tmp/some_dir"
247
+ process.stdout = process.stderr = "/tmp/server.log"
248
+
249
+ process.pid_file = "/tmp/some_pid_file.pid"
250
+ end
251
+ end
252
+ ```
253
+
254
+ The main benefit of using the config options is that Bluepill will be able to monitor the correct process instead of just watching the shell that spawned your actual server.
255
+
256
+ ### CLI
257
+ To start a bluepill process and load a config:
258
+
259
+ sudo bluepill load /path/to/production.pill
260
+
261
+ To act on a process or group:
262
+
263
+ sudo bluepill <start|stop|restart|unmonitor> <process_or_group_name>
264
+
265
+ To view process statuses:
266
+
267
+ sudo bluepill status
268
+
269
+ To view the log for a process or group:
270
+
271
+ sudo bluepill log <process_or_group_name>
272
+
273
+ To quit bluepill:
274
+
275
+ sudo bluepill quit
276
+
277
+ ### Logging
278
+ By default, bluepill uses syslog local6 facility as described in the installation section. But if for any reason you don&apos;t want to use syslog, you can use a log file. You can do this by setting the :log\_file option in the config:
279
+
280
+ ```ruby
281
+ Bluepill.application("app_name", :log_file => "/path/to/bluepill.log") do |app|
282
+ # ...
283
+ end
284
+ ```
285
+
286
+ Keep in mind that you still need to set up log rotation (described in the installation section) to keep the log file from growing huge.
287
+
288
+ ### Extra options
289
+ You can run bluepill in the foreground:
290
+
291
+ ```ruby
292
+ Bluepill.application("app_name", :foreground => true) do |app|
293
+ # ...
294
+ end
295
+ ```
296
+
297
+ Note that You must define only one application per config when using foreground mode.
298
+
299
+ ## Links
300
+ Code: [http://github.com/arya/bluepill](http://github.com/arya/bluepill)
301
+ Bugs/Features: [http://github.com/arya/bluepill/issues](http://github.com/arya/bluepill/issues)
302
+ Mailing List: [http://groups.google.com/group/bluepill-rb](http://groups.google.com/group/bluepill-rb)
303
+
304
+
305
+ [rubygems]: http://rubygems.org/gems/bluepill
306
+
data/Rakefile ADDED
@@ -0,0 +1,9 @@
1
+ # encoding: utf-8
2
+
3
+ begin
4
+ require 'bundler'
5
+ Bundler::GemHelper.install_tasks
6
+ rescue LoadError
7
+ $stderr.puts "Bundler not installed. You should install it with: gem install bundler"
8
+ end
9
+
data/bin/bluepill ADDED
@@ -0,0 +1,111 @@
1
+ #! /usr/bin/env ruby
2
+ require 'optparse'
3
+ require 'bluepill'
4
+
5
+ # Default options
6
+ options = {
7
+ :log_file => "/var/log/bluepill.log",
8
+ :base_dir => "/var/bluepill",
9
+ :privileged => true
10
+ }
11
+
12
+ OptionParser.new do |opts|
13
+ opts.banner = "Usage: bluepill [app] cmd [options]"
14
+ opts.on('-l', "--logfile LOGFILE", "Path to logfile, defaults to #{options[:log_file]}") do |file|
15
+ options[:log_file] = file
16
+ end
17
+
18
+ opts.on('-c', "--base-dir DIR", "Directory to store bluepill socket and pid files, defaults to #{options[:base_dir]}") do |base_dir|
19
+ options[:base_dir] = base_dir
20
+ end
21
+
22
+ opts.on("-v", "--version") do
23
+ puts "bluepill, version #{Bluepill::VERSION}"
24
+ exit
25
+ end
26
+
27
+ opts.on("--[no-]privileged", "Allow/disallow to run #{$0} as non-privileged process. disallowed by default") do |v|
28
+ options[:privileged] = v
29
+ end
30
+
31
+ opts.on('-t', '--timeout Seconds', Integer, "Timeout for commands sent to the daemon, in seconds. Defaults to 10.") do |timeout|
32
+ options[:timeout] = timeout
33
+ end
34
+
35
+ help = proc do
36
+ puts opts
37
+ puts
38
+ puts "Commands:"
39
+ puts " load CONFIG_FILE\t\tLoads new instance of bluepill using the specified config file"
40
+ puts " status\t\t\tLists the status of the proceses for the specified app"
41
+ puts " start [TARGET]\t\tIssues the start command for the target process or group, defaults to all processes"
42
+ puts " stop [TARGET]\t\tIssues the stop command for the target process or group, defaults to all processes"
43
+ puts " restart [TARGET]\t\tIssues the restart command for the target process or group, defaults to all processes"
44
+ puts " unmonitor [TARGET]\t\tStop monitoring target process or group, defaults to all processes"
45
+ puts " log [TARGET]\t\tShow the log for the specified process or group, defaults to all for app"
46
+ puts " quit\t\t\tStop bluepill"
47
+ puts
48
+ puts "See http://github.com/arya/bluepill for README"
49
+ exit
50
+ end
51
+
52
+ opts.on_tail('-h','--help', 'Show this message', &help)
53
+ help.call if ARGV.empty?
54
+ end.parse!
55
+
56
+ # Check for root
57
+ if options[:privileged] && ::Process.euid != 0
58
+ $stderr.puts "You must run bluepill as root or use --no-privileged option."
59
+ exit(3)
60
+ end
61
+
62
+ APPLICATION_COMMANDS = %w(status start stop restart unmonitor quit log)
63
+
64
+ controller = Bluepill::Controller.new(options.slice(:base_dir, :log_file))
65
+
66
+ if controller.running_applications.include?(File.basename($0)) && File.symlink?($0)
67
+ # bluepill was called as a symlink with the name of the target application
68
+ options[:application] = File.basename($0)
69
+ elsif controller.running_applications.include?(ARGV.first)
70
+ # the first arg is the application name
71
+ options[:application] = ARGV.shift
72
+ elsif APPLICATION_COMMANDS.include?(ARGV.first)
73
+ if controller.running_applications.length == 1
74
+ # there is only one, let's just use that
75
+ options[:application] = controller.running_applications.first
76
+ elsif controller.running_applications.length > 1
77
+ # There is more than one, tell them the list and exit
78
+ $stderr.puts "You must specify an application name to run that command. Here's the list of running applications:"
79
+ controller.running_applications.each_with_index do |app, index|
80
+ $stderr.puts " #{index + 1}. #{app}"
81
+ end
82
+ $stderr.puts "Usage: bluepill [app] cmd [options]"
83
+ exit(1)
84
+ else
85
+ # There are none running AND they aren't trying to start one
86
+ $stderr.puts "Error: There are no running bluepill daemons.\nTo start a bluepill daemon, use: bluepill load <config file>"
87
+ exit(2)
88
+ end
89
+ end
90
+
91
+ options[:command] = ARGV.shift
92
+
93
+ if options[:command] == "load"
94
+ file = ARGV.shift
95
+ if File.exists?(file)
96
+ # Restart the ruby interpreter for the config file so that anything loaded here
97
+ # does not stay in memory for the daemon
98
+ require 'rbconfig'
99
+ ruby = File.join(Config::CONFIG['bindir'], Config::CONFIG['ruby_install_name'])
100
+ load_path = File.expand_path("#{File.dirname(__FILE__)}/../lib")
101
+ file_path = File.expand_path(file)
102
+
103
+ exec(ruby, "-I#{load_path}", '-rbluepill', file_path)
104
+
105
+ else
106
+ $stderr.puts "Can't find file: #{file}"
107
+ end
108
+ else
109
+ target = ARGV.shift
110
+ controller.handle_command(options[:application], options[:command], target)
111
+ end