skywatch 0.0.1
Sign up to get free protection for your applications and to get access to all the features.
- data/.gitignore +17 -0
- data/Gemfile +4 -0
- data/LICENSE +22 -0
- data/README.md +175 -0
- data/Rakefile +5 -0
- data/bin/skywatch +231 -0
- data/lib/skywatch.rb +3 -0
- data/lib/skywatch/builtin/alerts/email +14 -0
- data/lib/skywatch/builtin/checks/30.example +3 -0
- data/lib/skywatch/builtin/checks/3600.skywatch_watchers +18 -0
- data/lib/skywatch/tool.rb +211 -0
- data/lib/skywatch/version.rb +3 -0
- data/lib/skywatch/watcher/Gemfile +2 -0
- data/lib/skywatch/watcher/Procfile +1 -0
- data/lib/skywatch/watcher/watcher.rb +56 -0
- data/skywatch.gemspec +25 -0
- data/spec/skywatch_spec.rb +8 -0
- metadata +129 -0
data/.gitignore
ADDED
data/Gemfile
ADDED
data/LICENSE
ADDED
@@ -0,0 +1,22 @@
|
|
1
|
+
Copyright (c) 2012 Jeff Lindsay
|
2
|
+
|
3
|
+
MIT License
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
6
|
+
a copy of this software and associated documentation files (the
|
7
|
+
"Software"), to deal in the Software without restriction, including
|
8
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
9
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
10
|
+
permit persons to whom the Software is furnished to do so, subject to
|
11
|
+
the following conditions:
|
12
|
+
|
13
|
+
The above copyright notice and this permission notice shall be
|
14
|
+
included in all copies or substantial portions of the Software.
|
15
|
+
|
16
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
17
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
18
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
19
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
20
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
21
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
22
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,175 @@
|
|
1
|
+
# Skywatch (alpha)
|
2
|
+
|
3
|
+
Simple alerting system that lets you define checks and alerts in any
|
4
|
+
language that are then magically run on Heroku.
|
5
|
+
|
6
|
+
NoOps! Polyglot! Free monitoring of anything!
|
7
|
+
|
8
|
+
## Installation
|
9
|
+
|
10
|
+
$ gem install skywatch
|
11
|
+
|
12
|
+
## Usage
|
13
|
+
|
14
|
+
It's a fairly powerful tool. Run help to see a list of subcommands.
|
15
|
+
|
16
|
+
$ skywatch -h
|
17
|
+
|
18
|
+
## What the hell is this amazing thing??
|
19
|
+
|
20
|
+
Skywatch is a command-line utility that manages checking and alerting
|
21
|
+
scripts used by a builtin watcher service. Skywatch deploys these
|
22
|
+
scripts and service for you on Heroku where they can run and monitor
|
23
|
+
anything from the cloud for free.
|
24
|
+
|
25
|
+
The watcher service runs check scripts that can assert
|
26
|
+
anything at any frequency. If a check script returns a non-zero exit status,
|
27
|
+
it will fire any enabled alert scripts, passing it the output of the
|
28
|
+
check script and other useful data. Alert scripts can then act on this
|
29
|
+
assertion failure, such as send email, SMS, or webhook.
|
30
|
+
|
31
|
+
The check script will continue to run and potentially fail, but the
|
32
|
+
alert script only runs once, if without error. Only until a reset signal
|
33
|
+
is sent will it be ready to fire the alert again for any failed check
|
34
|
+
script. In this way, alerts work like [clip
|
35
|
+
indicators](http://help.adobe.com/en_US/audition/cs/using/WS58a04a822e3e5010548241038980c2c5-7f93.html)
|
36
|
+
in the audio recording world. They turn on once any clipping happens and
|
37
|
+
remain on until you manually reset them.
|
38
|
+
|
39
|
+
You manage your scripts locally with the skywatch command, or by hand
|
40
|
+
since they're just files in directories. When you want to deploy script
|
41
|
+
changes, toggle enabled scripts, or reset the alert state, you can run a
|
42
|
+
skywatch command and it will handle pushing changes to Heroku for you.
|
43
|
+
|
44
|
+
## Getting Started
|
45
|
+
|
46
|
+
The skywatch command manages a directory containing check scripts and
|
47
|
+
alert scripts. You can make a new directory and let skywatch set this up
|
48
|
+
for you:
|
49
|
+
|
50
|
+
$ mkdir my-watcher
|
51
|
+
$ cd my-watcher
|
52
|
+
$ skywatch init
|
53
|
+
|
54
|
+
At this point, it will have you authenticate with your Heroku
|
55
|
+
credentials if you haven't already. Grab a free account if you don't
|
56
|
+
have one.
|
57
|
+
|
58
|
+
When you run `skywatch init` authenticated it will create some example
|
59
|
+
alerts and checks for you, and deploy a Heroku app. None of the checks
|
60
|
+
or alerts are enabled by default. See everything by just running
|
61
|
+
`skywatch` from the directory:
|
62
|
+
|
63
|
+
$ skywatch
|
64
|
+
Checks for fathomless-crag-3169
|
65
|
+
example every 30s disabled
|
66
|
+
skywatch_watchers every 3600s disabled
|
67
|
+
Alerts for fathomless-crag-3169
|
68
|
+
email disabled
|
69
|
+
|
70
|
+
Take a look at all the files in the directory. Checks and alerts are nothing
|
71
|
+
more than scripts. Checks have a naming convention of `<interval>.<name>`,
|
72
|
+
and enabling and disabling is just setting the execute bit on the
|
73
|
+
scripts. There's nothing the `skywatch` command does that you can't
|
74
|
+
easily do by hand. It just happens to be terribly convenient.
|
75
|
+
|
76
|
+
$ skywatch edit alert email
|
77
|
+
|
78
|
+
This will open your editor and you can see the example email alert
|
79
|
+
script is using SendGrid. In fact, when you ran `skywatch init`, you
|
80
|
+
were set up with a free SendGrid starter addon for 600 emails a day. So
|
81
|
+
let's try it by putting your email address in the `TO` variable of the
|
82
|
+
script. Now enable the alert:
|
83
|
+
|
84
|
+
$ skywatch enable alert email
|
85
|
+
|
86
|
+
Let's create a new check script in bash that fails so we can get the
|
87
|
+
alert.
|
88
|
+
|
89
|
+
$ skywatch create check failure_test 30
|
90
|
+
|
91
|
+
The last argument is the interval. Intervals are always in seconds. All
|
92
|
+
this did was create a new file under the `checks` directory with a
|
93
|
+
little bit of boiler plate. Let's replace its contents with this:
|
94
|
+
|
95
|
+
#!/usr/bin/env bash
|
96
|
+
echo "Oh no, a failed check."
|
97
|
+
exit 255
|
98
|
+
|
99
|
+
Enable the check and then we'll deploy:
|
100
|
+
|
101
|
+
$ skywatch enable check failure_test
|
102
|
+
$ skywatch deploy
|
103
|
+
|
104
|
+
It's going to move some files around and then deploy to Heroku. It keeps
|
105
|
+
a staging directory called `.skywatch`, which is a Git repo used to push
|
106
|
+
to Heroku. It automatically adds this to a `.gitignore` file, so you can
|
107
|
+
version your scripts with Git and not worry about this implementation
|
108
|
+
detail.
|
109
|
+
|
110
|
+
Once it's finished, you might want to run monitor to see how it went and
|
111
|
+
what's going on. This is just tailing the Heroku logs of the watcher
|
112
|
+
service:
|
113
|
+
|
114
|
+
$ skywatch monitor
|
115
|
+
|
116
|
+
You can run this whenever to see what it's doing. You'll probably see
|
117
|
+
that it triggered the alert. Go check your email! That will be the only
|
118
|
+
email you get, regardless of whether the check starts to work again and
|
119
|
+
then fail again. No flapping. You have to manually reset:
|
120
|
+
|
121
|
+
$ skywatch reset
|
122
|
+
|
123
|
+
This should cause another alert email within 30 seconds. And of course,
|
124
|
+
you can tear everything down with destroy:
|
125
|
+
|
126
|
+
$ skywatch destroy
|
127
|
+
|
128
|
+
This destroys the Heroku app and the `.skywatch` directory. It doesn't
|
129
|
+
touch your scripts at all. In fact, you can run `skywatch init` again if
|
130
|
+
you'd like.
|
131
|
+
|
132
|
+
The source code to all this is terribly simple. The watcher service is
|
133
|
+
only about 50 lines of Ruby. Everything else is just file operations.
|
134
|
+
In fact, the little state it maintains is kept in file metadata. For how
|
135
|
+
automated it is, skywatch has to be one of the simplest monitoring services
|
136
|
+
ever.
|
137
|
+
|
138
|
+
## Writing Check Scripts
|
139
|
+
|
140
|
+
Check scripts are any executable script using the shebang to define the
|
141
|
+
interpreter. Heroku has most common languages built-in to its Cedar
|
142
|
+
stack, so feel free to use Python, Perl, Ruby, whatever. I like bash.
|
143
|
+
|
144
|
+
The only conventions of check scripts are the interval-in-the-filename and that a non-zero exit status will fire the alerts. Any output of the check script will be piped into STDIN of the alert script, so try be verbose but not too
|
145
|
+
verbose.
|
146
|
+
|
147
|
+
If you're using bash, it's a good idea to use `set -e` so any failed
|
148
|
+
subcommand will bubble up.
|
149
|
+
|
150
|
+
## Writing Alert Scripts
|
151
|
+
|
152
|
+
Like check scripts, alert scripts can be written in any language. Also,
|
153
|
+
like check scripts, the exit status is important. If an alert script
|
154
|
+
exit status is non-zero, it will run again with the next failure of the check
|
155
|
+
script.
|
156
|
+
|
157
|
+
The alert script is given the output of the check script via STDIN. It's
|
158
|
+
also given 2 arguments. The first is the name of the check script. The
|
159
|
+
second is the exit status of the failed check script.
|
160
|
+
|
161
|
+
The output of an alert script is ignored. It might be a good idea to log
|
162
|
+
the output of failed alert scripts. You'd then be able to see it via
|
163
|
+
`skywatch monitor`. Sounds like a contribution idea.
|
164
|
+
|
165
|
+
## Contributing
|
166
|
+
|
167
|
+
1. Fork it
|
168
|
+
2. Create your feature branch (`git checkout -b my-new-feature`)
|
169
|
+
3. Commit your changes (`git commit -am 'Add some feature'`)
|
170
|
+
4. Push to the branch (`git push origin my-new-feature`)
|
171
|
+
5. Create new Pull Request
|
172
|
+
|
173
|
+
## License
|
174
|
+
|
175
|
+
MIT
|
data/Rakefile
ADDED
data/bin/skywatch
ADDED
@@ -0,0 +1,231 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
require 'rubygems'
|
3
|
+
require 'commander/import'
|
4
|
+
require 'highline/import'
|
5
|
+
|
6
|
+
require 'skywatch'
|
7
|
+
|
8
|
+
include Skywatch
|
9
|
+
|
10
|
+
program :name, 'skywatch'
|
11
|
+
program :version, Skywatch::VERSION
|
12
|
+
program :description, 'Simple, Unix-oriented alerting system'
|
13
|
+
default_command :all
|
14
|
+
|
15
|
+
def display_check(check)
|
16
|
+
status = check.enabled? ? "'enabled', :green" : "'disabled', :red"
|
17
|
+
say " %-24s %-16s %s" % [
|
18
|
+
check.name, "every #{check.interval}s", "<%= color #{status} %>"]
|
19
|
+
end
|
20
|
+
|
21
|
+
def display_alert(alert)
|
22
|
+
status = alert.enabled? ? "'enabled', :green" : "'disabled', :red"
|
23
|
+
say " %-16s %s" % [
|
24
|
+
alert.name, "<%= color #{status} %>"]
|
25
|
+
end
|
26
|
+
|
27
|
+
def ensure_logged_in
|
28
|
+
if not logged_in?
|
29
|
+
say_warning "You're not logged into Heroku. Please login then try again"
|
30
|
+
login
|
31
|
+
end
|
32
|
+
end
|
33
|
+
|
34
|
+
command :init do |c|
|
35
|
+
c.syntax = 'skywatch init [name]'
|
36
|
+
c.description = 'Initializes skywatch in the current directory'
|
37
|
+
c.action do |args, options|
|
38
|
+
ensure_logged_in
|
39
|
+
init args.first
|
40
|
+
say_ok "Skywatch initialized as #{name}"
|
41
|
+
end
|
42
|
+
end
|
43
|
+
|
44
|
+
command :checks do |c|
|
45
|
+
c.syntax = 'skywatch checks'
|
46
|
+
c.description = 'Lists checks'
|
47
|
+
c.action do |args, options|
|
48
|
+
puts " Checks for #{name}"
|
49
|
+
checks.each do |check|
|
50
|
+
display_check check
|
51
|
+
end
|
52
|
+
end
|
53
|
+
end
|
54
|
+
|
55
|
+
command :alerts do |c|
|
56
|
+
c.syntax = 'skywatch alerts'
|
57
|
+
c.description = 'Lists alerts'
|
58
|
+
c.action do |args, options|
|
59
|
+
puts " Alerts for #{name}"
|
60
|
+
alerts.each do |alert|
|
61
|
+
display_alert alert
|
62
|
+
end
|
63
|
+
end
|
64
|
+
end
|
65
|
+
|
66
|
+
command :all do |c|
|
67
|
+
c.syntax = 'skywatch all'
|
68
|
+
c.description = 'Lists checks and alerts'
|
69
|
+
c.action do |args, options|
|
70
|
+
puts " Checks for #{name}"
|
71
|
+
checks.each do |check|
|
72
|
+
display_check check
|
73
|
+
end
|
74
|
+
puts " Alerts for #{name}"
|
75
|
+
alerts.each do |alert|
|
76
|
+
display_alert alert
|
77
|
+
end
|
78
|
+
end
|
79
|
+
end
|
80
|
+
|
81
|
+
command :'enable check' do |c|
|
82
|
+
c.syntax = 'skywatch enable check <name>'
|
83
|
+
c.description = 'Enable a check'
|
84
|
+
c.action do |args, options|
|
85
|
+
fail unless args.length > 0
|
86
|
+
check = check(args.first)
|
87
|
+
check.enable
|
88
|
+
display_check check
|
89
|
+
end
|
90
|
+
end
|
91
|
+
|
92
|
+
command :'disable check' do |c|
|
93
|
+
c.syntax = 'skywatch disable check <name>'
|
94
|
+
c.description = 'Disable a check'
|
95
|
+
c.action do |args, options|
|
96
|
+
fail unless args.length > 0
|
97
|
+
check = check(args.first)
|
98
|
+
check.disable
|
99
|
+
display_check check
|
100
|
+
end
|
101
|
+
end
|
102
|
+
|
103
|
+
command :'enable alert' do |c|
|
104
|
+
c.syntax = 'skywatch enable alert <name>'
|
105
|
+
c.description = 'Enable an alert'
|
106
|
+
c.action do |args, options|
|
107
|
+
fail unless args.length > 0
|
108
|
+
alert = alert(args.first)
|
109
|
+
alert.enable
|
110
|
+
display_alert alert
|
111
|
+
end
|
112
|
+
end
|
113
|
+
|
114
|
+
command :'disable alert' do |c|
|
115
|
+
c.syntax = 'skywatch disable alert <name>'
|
116
|
+
c.description = 'Disable an alert'
|
117
|
+
c.action do |args, options|
|
118
|
+
fail unless args.length > 0
|
119
|
+
alert = alert(args.first)
|
120
|
+
alert.disable
|
121
|
+
display_alert alert
|
122
|
+
end
|
123
|
+
end
|
124
|
+
|
125
|
+
command :'create alert' do |c|
|
126
|
+
c.syntax = 'skywatch create alert <name> [interpreter]'
|
127
|
+
c.description = 'Create an alert script. Default interpreter is bash'
|
128
|
+
c.action do |args, options|
|
129
|
+
fail unless args.length > 0
|
130
|
+
interpreter = args.length > 1 ? args[1] : 'bash'
|
131
|
+
alert = create_alert(args.first, interpreter)
|
132
|
+
display_alert alert
|
133
|
+
end
|
134
|
+
end
|
135
|
+
|
136
|
+
command :'create check' do |c|
|
137
|
+
c.syntax = 'skywatch create check <name> <interval> [interpreter]'
|
138
|
+
c.description = 'Create a check script. Default interpreter is bash'
|
139
|
+
c.action do |args, options|
|
140
|
+
fail unless args.length > 1
|
141
|
+
interpreter = args.length > 2 ? args[2] : 'bash'
|
142
|
+
check = create_check(args[0], args[1], interpreter)
|
143
|
+
display_check check
|
144
|
+
end
|
145
|
+
end
|
146
|
+
|
147
|
+
command :'edit alert' do |c|
|
148
|
+
c.syntax = 'skywatch edit alert <name>'
|
149
|
+
c.description = 'Open the alert script in an editor'
|
150
|
+
c.action do |args|
|
151
|
+
fail unless args.length > 0
|
152
|
+
if ENV['EDITOR'].to_s.empty?
|
153
|
+
puts "The EDITOR environment variable is not defined."
|
154
|
+
else
|
155
|
+
alert = alert(args.first)
|
156
|
+
exec "#{ENV['EDITOR']} #{alert.path}"
|
157
|
+
end
|
158
|
+
end
|
159
|
+
end
|
160
|
+
|
161
|
+
command :'edit check' do |c|
|
162
|
+
c.syntax = 'skywatch edit check <name>'
|
163
|
+
c.description = 'Open the check script in an editor'
|
164
|
+
c.action do |args|
|
165
|
+
fail unless args.length > 0
|
166
|
+
if ENV['EDITOR'].to_s.empty?
|
167
|
+
puts "The EDITOR environment variable is not defined."
|
168
|
+
else
|
169
|
+
check = check(args.first)
|
170
|
+
exec "#{ENV['EDITOR']} #{check.path}"
|
171
|
+
end
|
172
|
+
end
|
173
|
+
end
|
174
|
+
|
175
|
+
command :reset do |c|
|
176
|
+
c.syntax = 'skywatch reset'
|
177
|
+
c.description = 'Reset running check states'
|
178
|
+
c.action do
|
179
|
+
ensure_logged_in
|
180
|
+
reset
|
181
|
+
end
|
182
|
+
end
|
183
|
+
|
184
|
+
command :monitor do |c|
|
185
|
+
c.syntax = 'skywatch monitor'
|
186
|
+
c.description = 'Monitor the running checks'
|
187
|
+
c.action do
|
188
|
+
ensure_logged_in
|
189
|
+
monitor
|
190
|
+
end
|
191
|
+
end
|
192
|
+
|
193
|
+
command :deploy do |c|
|
194
|
+
c.syntax = 'skywatch deploy'
|
195
|
+
c.description = 'Deploy this skywatch'
|
196
|
+
c.action do
|
197
|
+
ensure_logged_in
|
198
|
+
deploy
|
199
|
+
end
|
200
|
+
end
|
201
|
+
|
202
|
+
command :destroy do |c|
|
203
|
+
c.syntax = 'skywatch destroy'
|
204
|
+
c.description = 'Destroy skywatch Heroku app and stage directory'
|
205
|
+
c.action do
|
206
|
+
ensure_logged_in
|
207
|
+
destroy
|
208
|
+
end
|
209
|
+
end
|
210
|
+
|
211
|
+
#command :'destroy alert' do |c|
|
212
|
+
# c.syntax = 'skywatch destroy alert <name>'
|
213
|
+
# c.description = 'Delete an alert script'
|
214
|
+
# c.action do |args, options|
|
215
|
+
# fail unless args.length > 0
|
216
|
+
# alert = alert(args.first)
|
217
|
+
# alert.destroy
|
218
|
+
# say " Alert <%= color '#{alert.name}', :bold %> has been destroyed"
|
219
|
+
# end
|
220
|
+
#end
|
221
|
+
|
222
|
+
#command :'destroy check' do |c|
|
223
|
+
# c.syntax = 'skywatch destroy check <name>'
|
224
|
+
# c.description = 'Delete a check script'
|
225
|
+
# c.action do |args, options|
|
226
|
+
# fail unless args.length > 0
|
227
|
+
# check = check(args.first)
|
228
|
+
# check.destroy
|
229
|
+
# say " Check <%= color '#{check.name}', :bold %> has been destroyed"
|
230
|
+
# end
|
231
|
+
#end
|
data/lib/skywatch.rb
ADDED
@@ -0,0 +1,14 @@
|
|
1
|
+
#!/usr/bin/env bash
|
2
|
+
TO=
|
3
|
+
SUBJECT="[skywatch] $1"
|
4
|
+
BODY=`echo -e "Failure with status $2:\n\n$(cat)"`
|
5
|
+
set -e
|
6
|
+
curl \
|
7
|
+
-X 'POST' \
|
8
|
+
-F "api_user=$SENDGRID_USERNAME" \
|
9
|
+
-F "api_key=$SENDGRID_PASSWORD" \
|
10
|
+
-F "to=$TO" \
|
11
|
+
-F "subject=$SUBJECT" \
|
12
|
+
-F "text=$BODY" \
|
13
|
+
-F "from=$TO" \
|
14
|
+
--silent --fail "https://sendgrid.com/api/mail.send.json"
|
@@ -0,0 +1,18 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
require "net/https"
|
3
|
+
require "uri"
|
4
|
+
require "json"
|
5
|
+
|
6
|
+
uri = URI.parse("https://api.github.com/repos/progrium/skywatch")
|
7
|
+
http = Net::HTTP.new(uri.host, uri.port)
|
8
|
+
http.use_ssl = true
|
9
|
+
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
|
10
|
+
|
11
|
+
request = Net::HTTP::Get.new(uri.request_uri)
|
12
|
+
|
13
|
+
response = http.request(request)
|
14
|
+
repo = JSON.parse(response.body)
|
15
|
+
|
16
|
+
if repo['watchers_count'] > 1000
|
17
|
+
raise "#{repo['watchers_count']} watchers!"
|
18
|
+
end
|
@@ -0,0 +1,211 @@
|
|
1
|
+
require 'fileutils'
|
2
|
+
|
3
|
+
module Skywatch
|
4
|
+
include FileUtils
|
5
|
+
|
6
|
+
def init(name="")
|
7
|
+
fail if skywatch?
|
8
|
+
mkdir '.skywatch'
|
9
|
+
begin
|
10
|
+
cd '.skywatch' do
|
11
|
+
system "git init"
|
12
|
+
puts `heroku create #{name}`
|
13
|
+
fail if not $?.exitstatus.zero?
|
14
|
+
puts `heroku addons:add sendgrid:starter`
|
15
|
+
system "cp #{watcher_path}/* ."
|
16
|
+
system "bundle install > /dev/null 2>&1"
|
17
|
+
end
|
18
|
+
mkdir_p 'alerts'
|
19
|
+
if Dir['alerts/*'].empty?
|
20
|
+
system "cp #{builtin_path}/alerts/* alerts"
|
21
|
+
end
|
22
|
+
mkdir_p 'checks'
|
23
|
+
if Dir['checks/*'].empty?
|
24
|
+
system "cp #{builtin_path}/checks/* checks"
|
25
|
+
end
|
26
|
+
system "echo '.skywatch' >> .gitignore"
|
27
|
+
deploy
|
28
|
+
cd '.skywatch' do
|
29
|
+
system "heroku ps:scale watcher=1"
|
30
|
+
end
|
31
|
+
rescue
|
32
|
+
rm_rf '.skywatch'
|
33
|
+
end
|
34
|
+
end
|
35
|
+
|
36
|
+
def logged_in?
|
37
|
+
`heroku auth:token < /dev/null`
|
38
|
+
$?.exitstatus.zero?
|
39
|
+
end
|
40
|
+
|
41
|
+
def login
|
42
|
+
exec 'heroku auth:login'
|
43
|
+
end
|
44
|
+
|
45
|
+
def watcher_path
|
46
|
+
File.dirname(__FILE__)+"/watcher"
|
47
|
+
end
|
48
|
+
|
49
|
+
def builtin_path
|
50
|
+
File.dirname(__FILE__)+"/builtin"
|
51
|
+
end
|
52
|
+
|
53
|
+
def skywatch?
|
54
|
+
not Dir["./.skywatch"].empty?
|
55
|
+
end
|
56
|
+
|
57
|
+
def checks
|
58
|
+
fail unless skywatch?
|
59
|
+
Dir["./checks/*"].collect{|c| Check[c] }
|
60
|
+
end
|
61
|
+
|
62
|
+
def alerts
|
63
|
+
fail unless skywatch?
|
64
|
+
Dir["./alerts/*"].collect{|c| Alert[c] }
|
65
|
+
end
|
66
|
+
|
67
|
+
def check(name)
|
68
|
+
fail unless skywatch?
|
69
|
+
Check[Dir["./checks/*.#{name}"].first]
|
70
|
+
end
|
71
|
+
|
72
|
+
def alert(name)
|
73
|
+
fail unless skywatch?
|
74
|
+
Alert[Dir["./alerts/#{name}"].first]
|
75
|
+
end
|
76
|
+
|
77
|
+
def create_alert(name, interpreter)
|
78
|
+
fail unless skywatch?
|
79
|
+
fail if File.exist? "./alerts/#{name}"
|
80
|
+
File.open("./alerts/#{name}", 'w') do |f|
|
81
|
+
f.write "#!/usr/bin/env #{interpreter}\n"
|
82
|
+
if interpreter == 'bash'
|
83
|
+
f.write "set -e\n"
|
84
|
+
end
|
85
|
+
f.write "# Write your alert here"
|
86
|
+
end
|
87
|
+
alert(name)
|
88
|
+
end
|
89
|
+
|
90
|
+
def create_check(name, interval, interpreter)
|
91
|
+
fail unless skywatch?
|
92
|
+
fail if File.exist? "./checks/#{interval}.#{name}"
|
93
|
+
File.open("./checks/#{interval}.#{name}", 'w') do |f|
|
94
|
+
f.write "#!/usr/bin/env #{interpreter}\n"
|
95
|
+
if interpreter == 'bash'
|
96
|
+
f.write "set -e\n"
|
97
|
+
end
|
98
|
+
f.write "# Write your check here"
|
99
|
+
end
|
100
|
+
check(name)
|
101
|
+
end
|
102
|
+
|
103
|
+
def name
|
104
|
+
fail unless skywatch?
|
105
|
+
cd ".skywatch" do
|
106
|
+
if not File.exist? 'name'
|
107
|
+
name = `eval $(heroku apps:info -s | grep "^name=") && echo "$name"`.strip
|
108
|
+
File.open('name', 'w') {|f| f.write(name) }
|
109
|
+
end
|
110
|
+
@@name ||= File.read('name')
|
111
|
+
end
|
112
|
+
@@name
|
113
|
+
end
|
114
|
+
|
115
|
+
def reset
|
116
|
+
fail unless skywatch?
|
117
|
+
cd ".skywatch" do
|
118
|
+
puts `heroku restart`
|
119
|
+
end
|
120
|
+
end
|
121
|
+
|
122
|
+
def monitor
|
123
|
+
fail unless skywatch?
|
124
|
+
cd ".skywatch" do
|
125
|
+
exec "heroku logs -t -n 10"
|
126
|
+
end
|
127
|
+
end
|
128
|
+
|
129
|
+
def stage
|
130
|
+
fail unless skywatch?
|
131
|
+
cd '.skywatch' do
|
132
|
+
system 'git rm -f alerts/* > /dev/null 2>&1'
|
133
|
+
system 'git rm -f checks/* > /dev/null 2>&1'
|
134
|
+
end
|
135
|
+
mkdir_p '.skywatch/alerts'
|
136
|
+
mkdir_p '.skywatch/checks'
|
137
|
+
system 'cp alerts/* .skywatch/alerts'
|
138
|
+
system 'cp checks/* .skywatch/checks'
|
139
|
+
commit
|
140
|
+
end
|
141
|
+
|
142
|
+
def commit
|
143
|
+
fail unless skywatch?
|
144
|
+
cd '.skywatch' do
|
145
|
+
system 'git add .'
|
146
|
+
if not `git status`.include? "nothing to commit"
|
147
|
+
system 'git commit -m "skywatch commit"'
|
148
|
+
end
|
149
|
+
end
|
150
|
+
end
|
151
|
+
|
152
|
+
def deploy
|
153
|
+
stage
|
154
|
+
cd '.skywatch' do
|
155
|
+
puts `git push heroku master`
|
156
|
+
end
|
157
|
+
end
|
158
|
+
|
159
|
+
def destroy
|
160
|
+
fail unless skywatch?
|
161
|
+
puts `heroku apps:destroy -a #{name} --confirm #{name}`
|
162
|
+
rm_rf '.skywatch'
|
163
|
+
end
|
164
|
+
|
165
|
+
class Script
|
166
|
+
attr :path
|
167
|
+
attr :name
|
168
|
+
|
169
|
+
def self.[](path)
|
170
|
+
self.new(path)
|
171
|
+
end
|
172
|
+
|
173
|
+
def initialize(path)
|
174
|
+
@path = path
|
175
|
+
@name = File.basename(path)
|
176
|
+
end
|
177
|
+
|
178
|
+
def enabled?
|
179
|
+
File.executable? @path
|
180
|
+
end
|
181
|
+
|
182
|
+
def enable
|
183
|
+
chmod File.stat(@path).mode | 0700, @path
|
184
|
+
end
|
185
|
+
|
186
|
+
def disable
|
187
|
+
enable # since this is relative, force 0700
|
188
|
+
chmod File.stat(@path).mode & ~0100, @path
|
189
|
+
end
|
190
|
+
|
191
|
+
def destroy
|
192
|
+
rm @path
|
193
|
+
end
|
194
|
+
end
|
195
|
+
|
196
|
+
class Check < Script
|
197
|
+
attr :interval
|
198
|
+
|
199
|
+
def initialize(path)
|
200
|
+
interval, name = File.basename(path).split('.', 2)
|
201
|
+
@path = path
|
202
|
+
@name = name
|
203
|
+
@interval = interval.to_i
|
204
|
+
end
|
205
|
+
end
|
206
|
+
|
207
|
+
class Alert < Script
|
208
|
+
end
|
209
|
+
|
210
|
+
end
|
211
|
+
|
@@ -0,0 +1 @@
|
|
1
|
+
watcher: bundle exec clockwork watcher.rb
|
@@ -0,0 +1,56 @@
|
|
1
|
+
require 'clockwork'
|
2
|
+
require 'fileutils'
|
3
|
+
|
4
|
+
include Clockwork
|
5
|
+
include FileUtils
|
6
|
+
|
7
|
+
Clockwork.configure do |config|
|
8
|
+
config[:logger] = Logger.new(STDOUT)
|
9
|
+
config[:logger].level = Logger::ERROR
|
10
|
+
end
|
11
|
+
|
12
|
+
def mark_pass(check); chmod File.stat(check).mode & ~0003, check; end
|
13
|
+
def mark_fail(check); chmod File.stat(check).mode | 0007, check; end
|
14
|
+
def mark_alerted(check); chmod File.stat(check).mode | 0070, check; end
|
15
|
+
def marked_alerted?(check)
|
16
|
+
`stat -c %A #{check} | sed 's/......\\(.\\).\\+/\\1/'` == "x\n"
|
17
|
+
end
|
18
|
+
def executables(glob)
|
19
|
+
Dir[glob].select {|path| File.executable? path }
|
20
|
+
end
|
21
|
+
|
22
|
+
mkdir_p 'output'
|
23
|
+
mkdir_p 'checks'
|
24
|
+
mkdir_p 'alerts'
|
25
|
+
|
26
|
+
executables('checks/*').each do |check|
|
27
|
+
interval, name = File.basename(check).split('.', 2)
|
28
|
+
|
29
|
+
puts "loading check #{name} for every #{interval} seconds"
|
30
|
+
every interval.to_i.seconds, name do
|
31
|
+
|
32
|
+
puts "checking #{name} (#{File.stat(check).mode.to_s(8)})"
|
33
|
+
`#{check} > output/#{name} 2>&1`
|
34
|
+
|
35
|
+
status = $?.exitstatus
|
36
|
+
if status.zero?
|
37
|
+
mark_pass check
|
38
|
+
else
|
39
|
+
puts " #{name} check failed with status #{status}"
|
40
|
+
mark_fail check
|
41
|
+
|
42
|
+
if not marked_alerted? check
|
43
|
+
executables('alerts/*').each do |alert|
|
44
|
+
puts " sending #{File.basename(alert)} alert for #{name}"
|
45
|
+
`cat output/#{name} | #{alert} #{name} #{status} > /dev/null 2>&1`
|
46
|
+
|
47
|
+
if $?.exitstatus.zero?
|
48
|
+
mark_alerted check
|
49
|
+
end
|
50
|
+
end
|
51
|
+
end
|
52
|
+
|
53
|
+
end
|
54
|
+
end # every
|
55
|
+
|
56
|
+
end
|
data/skywatch.gemspec
ADDED
@@ -0,0 +1,25 @@
|
|
1
|
+
# -*- encoding: utf-8 -*-
|
2
|
+
lib = File.expand_path('../lib', __FILE__)
|
3
|
+
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
4
|
+
require 'skywatch/version'
|
5
|
+
|
6
|
+
Gem::Specification.new do |gem|
|
7
|
+
gem.name = "skywatch"
|
8
|
+
gem.version = Skywatch::VERSION
|
9
|
+
gem.authors = ["Jeff Lindsay"]
|
10
|
+
gem.email = ["progrium@gmail.com"]
|
11
|
+
gem.description = %q{Simple alerting system that lets you define checks and alerts in any language that are then magically run on Heroku.}
|
12
|
+
gem.summary = %q{Simple, polyglot alerting system}
|
13
|
+
gem.homepage = "http://github.com/progrium/skywatch"
|
14
|
+
|
15
|
+
gem.files = `git ls-files`.split($/)
|
16
|
+
gem.executables = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
|
17
|
+
gem.test_files = gem.files.grep(%r{^(test|spec|features)/})
|
18
|
+
gem.require_paths = ["lib"]
|
19
|
+
|
20
|
+
gem.add_development_dependency "rspec", "~> 2.6"
|
21
|
+
|
22
|
+
gem.add_dependency "commander", "~> 4.1"
|
23
|
+
gem.add_dependency "heroku", "~> 2.33"
|
24
|
+
gem.add_dependency "bundler", "~> 1.2"
|
25
|
+
end
|
metadata
ADDED
@@ -0,0 +1,129 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: skywatch
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.0.1
|
5
|
+
prerelease:
|
6
|
+
platform: ruby
|
7
|
+
authors:
|
8
|
+
- Jeff Lindsay
|
9
|
+
autorequire:
|
10
|
+
bindir: bin
|
11
|
+
cert_chain: []
|
12
|
+
date: 2012-12-20 00:00:00.000000000 Z
|
13
|
+
dependencies:
|
14
|
+
- !ruby/object:Gem::Dependency
|
15
|
+
name: rspec
|
16
|
+
requirement: !ruby/object:Gem::Requirement
|
17
|
+
none: false
|
18
|
+
requirements:
|
19
|
+
- - ~>
|
20
|
+
- !ruby/object:Gem::Version
|
21
|
+
version: '2.6'
|
22
|
+
type: :development
|
23
|
+
prerelease: false
|
24
|
+
version_requirements: !ruby/object:Gem::Requirement
|
25
|
+
none: false
|
26
|
+
requirements:
|
27
|
+
- - ~>
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: '2.6'
|
30
|
+
- !ruby/object:Gem::Dependency
|
31
|
+
name: commander
|
32
|
+
requirement: !ruby/object:Gem::Requirement
|
33
|
+
none: false
|
34
|
+
requirements:
|
35
|
+
- - ~>
|
36
|
+
- !ruby/object:Gem::Version
|
37
|
+
version: '4.1'
|
38
|
+
type: :runtime
|
39
|
+
prerelease: false
|
40
|
+
version_requirements: !ruby/object:Gem::Requirement
|
41
|
+
none: false
|
42
|
+
requirements:
|
43
|
+
- - ~>
|
44
|
+
- !ruby/object:Gem::Version
|
45
|
+
version: '4.1'
|
46
|
+
- !ruby/object:Gem::Dependency
|
47
|
+
name: heroku
|
48
|
+
requirement: !ruby/object:Gem::Requirement
|
49
|
+
none: false
|
50
|
+
requirements:
|
51
|
+
- - ~>
|
52
|
+
- !ruby/object:Gem::Version
|
53
|
+
version: '2.33'
|
54
|
+
type: :runtime
|
55
|
+
prerelease: false
|
56
|
+
version_requirements: !ruby/object:Gem::Requirement
|
57
|
+
none: false
|
58
|
+
requirements:
|
59
|
+
- - ~>
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '2.33'
|
62
|
+
- !ruby/object:Gem::Dependency
|
63
|
+
name: bundler
|
64
|
+
requirement: !ruby/object:Gem::Requirement
|
65
|
+
none: false
|
66
|
+
requirements:
|
67
|
+
- - ~>
|
68
|
+
- !ruby/object:Gem::Version
|
69
|
+
version: '1.2'
|
70
|
+
type: :runtime
|
71
|
+
prerelease: false
|
72
|
+
version_requirements: !ruby/object:Gem::Requirement
|
73
|
+
none: false
|
74
|
+
requirements:
|
75
|
+
- - ~>
|
76
|
+
- !ruby/object:Gem::Version
|
77
|
+
version: '1.2'
|
78
|
+
description: Simple alerting system that lets you define checks and alerts in any
|
79
|
+
language that are then magically run on Heroku.
|
80
|
+
email:
|
81
|
+
- progrium@gmail.com
|
82
|
+
executables:
|
83
|
+
- skywatch
|
84
|
+
extensions: []
|
85
|
+
extra_rdoc_files: []
|
86
|
+
files:
|
87
|
+
- .gitignore
|
88
|
+
- Gemfile
|
89
|
+
- LICENSE
|
90
|
+
- README.md
|
91
|
+
- Rakefile
|
92
|
+
- bin/skywatch
|
93
|
+
- lib/skywatch.rb
|
94
|
+
- lib/skywatch/builtin/alerts/email
|
95
|
+
- lib/skywatch/builtin/checks/30.example
|
96
|
+
- lib/skywatch/builtin/checks/3600.skywatch_watchers
|
97
|
+
- lib/skywatch/tool.rb
|
98
|
+
- lib/skywatch/version.rb
|
99
|
+
- lib/skywatch/watcher/Gemfile
|
100
|
+
- lib/skywatch/watcher/Procfile
|
101
|
+
- lib/skywatch/watcher/watcher.rb
|
102
|
+
- skywatch.gemspec
|
103
|
+
- spec/skywatch_spec.rb
|
104
|
+
homepage: http://github.com/progrium/skywatch
|
105
|
+
licenses: []
|
106
|
+
post_install_message:
|
107
|
+
rdoc_options: []
|
108
|
+
require_paths:
|
109
|
+
- lib
|
110
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
111
|
+
none: false
|
112
|
+
requirements:
|
113
|
+
- - ! '>='
|
114
|
+
- !ruby/object:Gem::Version
|
115
|
+
version: '0'
|
116
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
117
|
+
none: false
|
118
|
+
requirements:
|
119
|
+
- - ! '>='
|
120
|
+
- !ruby/object:Gem::Version
|
121
|
+
version: '0'
|
122
|
+
requirements: []
|
123
|
+
rubyforge_project:
|
124
|
+
rubygems_version: 1.8.23
|
125
|
+
signing_key:
|
126
|
+
specification_version: 3
|
127
|
+
summary: Simple, polyglot alerting system
|
128
|
+
test_files:
|
129
|
+
- spec/skywatch_spec.rb
|