systemd_mon 0.0.1 → 0.0.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +55 -10
- data/lib/systemd_mon/cli.rb +7 -2
- data/lib/systemd_mon/dbus_manager.rb +5 -1
- data/lib/systemd_mon/error.rb +2 -0
- data/lib/systemd_mon/notification.rb +5 -1
- data/lib/systemd_mon/notifiers/email.rb +8 -3
- data/lib/systemd_mon/notifiers/slack.rb +16 -8
- data/lib/systemd_mon/state.rb +1 -1
- data/lib/systemd_mon/state_change.rb +6 -0
- data/lib/systemd_mon/version.rb +1 -1
- data/systemd_mon.gemspec +1 -1
- metadata +6 -6
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: d4c77a8241786cc39475f30bd269143c00e9bd0c
|
4
|
+
data.tar.gz: 6769b609787a604d20c9200126630613cf3ec9c8
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: dd86b32951f1e8886dee2206ccd74e5099c0c749977f8d23a0141ed97ce0bc7856124672d2eefd047d2333c281f0f99bd5384f6633804782aa45d1fc7a879ec9
|
7
|
+
data.tar.gz: 3f16e8e7123370d584eda98181dc19edd219fcff0e3b26f7f52c24e13ee0c4c90916ee7d6f41a66c0be0d9815f4f8fb7ab09f7826acd3289b19b78aa1bd904b9
|
data/README.md
CHANGED
@@ -4,6 +4,16 @@ Monitor systemd units and trigger alerts for failed states. The command line too
|
|
4
4
|
|
5
5
|
Built-in notifications include email and slack, but more can be added via the ruby API.
|
6
6
|
|
7
|
+
It works by subscribing to DBus notifications from Systemd. This means that there is no polling, and no busy-loops. SystemdMon will sit in the background, happily waiting and using minimal processes.
|
8
|
+
|
9
|
+
## Requirements
|
10
|
+
|
11
|
+
* A linux server
|
12
|
+
* Ruby > 1.9.3
|
13
|
+
* Systemd (v204 was used in development)
|
14
|
+
* `mail` gem (if email notifier is used)
|
15
|
+
* `slack-notifier` gem (if slack notifier is used)
|
16
|
+
|
7
17
|
## Installation
|
8
18
|
|
9
19
|
Install the gem using:
|
@@ -16,33 +26,68 @@ To run the command line tool, you will first need to create a YAML configuration
|
|
16
26
|
|
17
27
|
```yaml
|
18
28
|
---
|
29
|
+
verbose: true # Default is off
|
19
30
|
notifiers:
|
20
|
-
# These are options passed to the 'mail' gem
|
21
31
|
email:
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
32
|
+
to: "team@mydomain.com"
|
33
|
+
from: "systemdmon@mydomain.com"
|
34
|
+
# These are options passed to the 'mail' gem
|
35
|
+
smtp:
|
36
|
+
address: smtp.gmail.com
|
37
|
+
port: 587
|
38
|
+
domain: mydomain.com
|
39
|
+
user_name: "user@mydomain.com"
|
40
|
+
password: "supersecr3t"
|
41
|
+
authentication: "plain"
|
42
|
+
enable_starttls_auto: true
|
29
43
|
slack:
|
30
44
|
team: myteam
|
31
45
|
token: supersecr3ttoken
|
32
46
|
channel: mychannel
|
33
47
|
username: doge
|
48
|
+
icon_emoji: ":computer"
|
49
|
+
icon_url: "http://example.com/icon"
|
34
50
|
units:
|
35
51
|
- unicorn.service
|
36
52
|
- nginx.service
|
37
53
|
- sidekiq.service
|
38
54
|
```
|
39
55
|
|
40
|
-
|
56
|
+
Save that somewhere appropriate (e.g. `/etc/systemd_mon.yml`), then start the command line tool with:
|
57
|
+
|
58
|
+
$ systemd_mon /etc/systemd_mon.yml
|
59
|
+
|
60
|
+
You'll probably want to run it via systemd, which you can do with this example service file (change file paths as appropriate):
|
61
|
+
|
62
|
+
```
|
63
|
+
[Unit]
|
64
|
+
Description=SystemdMon
|
65
|
+
After=network.target
|
66
|
+
|
67
|
+
[Service]
|
68
|
+
Type=simple
|
69
|
+
User=deploy
|
70
|
+
StandardInput=null
|
71
|
+
StandardOutput=syslog
|
72
|
+
StandardError=syslog
|
73
|
+
ExecStart=/usr/local/bin/systemd_mon /etc/systemd_mon.yml
|
74
|
+
|
75
|
+
[Install]
|
76
|
+
WantedBy=multi-user.target
|
77
|
+
```
|
78
|
+
|
79
|
+
## Behaviour
|
41
80
|
|
42
|
-
|
81
|
+
Systemd provides information about state changes in very fine detail. For example, if you start a service, it may go through the following states: activating (start-pre), activiating (start) and finally active (running). This will likely happen in less than a second, and you probably don't want 3 notifications. Therefore, SystemdMon queues up states until it comes across one that you think you should know about. In this case, it will notify you when the state reaches active (running), but the notification can show the history of how the state changed so you get the full picture.
|
82
|
+
|
83
|
+
SystemdMon does simple analysis on the history of state changes, so it can summarise with statuses like "recovered", "automatically restarted", "still failed", etc. It will also report with the host name of the server.
|
84
|
+
|
85
|
+
You'll also want to know if SystemdMon itself falls over, and when it starts back up again. It will attempt to send a final notification before it exits, and one to say it's starting. However, be aware that it might not send a notification in some conditions (e.g. in the case of a SIGKILL), or a network failure. The age-old question: who will watch the watcher?
|
43
86
|
|
44
87
|
## Contributing
|
45
88
|
|
89
|
+
I'd love more contributions, particulary new notifiers. Follow the example of the slack and email notifiers and either package as a new gem or submit a pull request if you think it should be part of the main project.
|
90
|
+
|
46
91
|
1. Fork it ( https://github.com/joonty/systemd_mon/fork )
|
47
92
|
2. Create your feature branch (`git checkout -b my-new-feature`)
|
48
93
|
3. Commit your changes (`git commit -am 'Add some feature'`)
|
data/lib/systemd_mon/cli.rb
CHANGED
@@ -22,8 +22,13 @@ module SystemdMon
|
|
22
22
|
rescue SystemdMon::Error => e
|
23
23
|
err_string = e.message
|
24
24
|
if verbose
|
25
|
-
|
26
|
-
|
25
|
+
if e.original
|
26
|
+
err_string << " - #{e.original.message} (#{e.original.class})"
|
27
|
+
err_string << "\n\t#{e.original.backtrace.join("\n\t")}"
|
28
|
+
else
|
29
|
+
err_string << " (#{e.class})"
|
30
|
+
err_string << "\n\t#{e.backtrace.join("\n\t")}"
|
31
|
+
end
|
27
32
|
end
|
28
33
|
fatal_error(err_string)
|
29
34
|
rescue => e
|
@@ -9,7 +9,11 @@ module SystemdMon
|
|
9
9
|
self.systemd_service = dbus.service("org.freedesktop.systemd1")
|
10
10
|
self.systemd_object = systemd_service.object("/org/freedesktop/systemd1")
|
11
11
|
systemd_object.introspect
|
12
|
-
systemd_object.Subscribe
|
12
|
+
if systemd_object.respond_to?("Subscribe")
|
13
|
+
systemd_object.Subscribe
|
14
|
+
else
|
15
|
+
raise SystemdMon::SystemdError, "Systemd is not installed, or is an incompatible version. It must provide the Subscribe dbus method: version 204 is the minimum recommended version."
|
16
|
+
end
|
13
17
|
end
|
14
18
|
|
15
19
|
def fetch_unit(unit_name)
|
data/lib/systemd_mon/error.rb
CHANGED
@@ -10,8 +10,10 @@ module SystemdMon
|
|
10
10
|
end
|
11
11
|
end
|
12
12
|
|
13
|
+
class SystemdError < Error; end
|
13
14
|
class MonitorError < Error; end
|
14
15
|
class UnknownUnitError < Error; end
|
15
16
|
class NotificationError < Error; end
|
17
|
+
class NotifierDependencyError < Error; end
|
16
18
|
class NotifierError < Error; end
|
17
19
|
end
|
@@ -1,8 +1,13 @@
|
|
1
|
-
require '
|
1
|
+
require 'systemd_mon/error'
|
2
2
|
require 'systemd_mon/notifiers/base'
|
3
|
-
require 'systemd_mon/logger'
|
4
3
|
require 'systemd_mon/formatters/state_table_formatter'
|
5
4
|
|
5
|
+
begin
|
6
|
+
require 'mail'
|
7
|
+
rescue LoadError
|
8
|
+
raise SystemdMon::NotifierDependencyError, "The 'mail' gem is required by the email notifier"
|
9
|
+
end
|
10
|
+
|
6
11
|
module SystemdMon::Notifiers
|
7
12
|
class Email < Base
|
8
13
|
def initialize(*)
|
@@ -19,7 +24,7 @@ module SystemdMon::Notifiers
|
|
19
24
|
|
20
25
|
def notify!(notification)
|
21
26
|
unit = notification.unit
|
22
|
-
subject = "#{unit.name} on #{notification.hostname}: #{unit.state_change.status_text}"
|
27
|
+
subject = "#{notification.type_text}: #{unit.name} on #{notification.hostname}: #{unit.state_change.status_text}"
|
23
28
|
message = "Systemd unit #{unit.name} on #{notification.hostname} #{unit.state_change.status_text}: #{unit.state.active} (#{unit.state.sub})\n\n"
|
24
29
|
if unit.state_change.length > 1
|
25
30
|
message << SystemdMon::Formatters::StateTableFormatter.new(unit).as_text
|
@@ -1,6 +1,12 @@
|
|
1
|
-
require '
|
1
|
+
require 'systemd_mon/error'
|
2
2
|
require 'systemd_mon/notifiers/base'
|
3
3
|
|
4
|
+
begin
|
5
|
+
require 'slack-notifier'
|
6
|
+
rescue LoadError
|
7
|
+
raise SystemdMon::NotifierDependencyError, "The 'slack-notifier' gem is required by the slack notifier"
|
8
|
+
end
|
9
|
+
|
4
10
|
module SystemdMon::Notifiers
|
5
11
|
class Slack < Base
|
6
12
|
def initialize(*)
|
@@ -15,32 +21,32 @@ module SystemdMon::Notifiers
|
|
15
21
|
end
|
16
22
|
|
17
23
|
def notify_start!(hostname)
|
18
|
-
message = "
|
24
|
+
message = "SystemdMon is starting on #{hostname}"
|
19
25
|
|
20
26
|
attach = {
|
21
27
|
fallback: message,
|
22
|
-
text:
|
28
|
+
text: message,
|
23
29
|
color: "good"
|
24
30
|
}
|
25
31
|
|
26
|
-
notifier.ping
|
32
|
+
notifier.ping "", attachments: [attach]
|
27
33
|
end
|
28
34
|
|
29
35
|
def notify_stop!(hostname)
|
30
|
-
message = "
|
36
|
+
message = "SystemdMon is stopping on #{hostname}"
|
31
37
|
|
32
38
|
attach = {
|
33
39
|
fallback: message,
|
34
|
-
text:
|
40
|
+
text: message,
|
35
41
|
color: "danger"
|
36
42
|
}
|
37
43
|
|
38
|
-
notifier.ping
|
44
|
+
notifier.ping "", attachments: [attach]
|
39
45
|
end
|
40
46
|
|
41
47
|
def notify!(notification)
|
42
48
|
unit = notification.unit
|
43
|
-
message = "
|
49
|
+
message = "#{notification.type_text}: systemd unit #{unit.name} on #{notification.hostname} #{unit.state_change.status_text}"
|
44
50
|
|
45
51
|
attach = {
|
46
52
|
fallback: "#{message}: #{unit.state.active} (#{unit.state.sub})",
|
@@ -82,6 +88,8 @@ module SystemdMon::Notifiers
|
|
82
88
|
case type
|
83
89
|
when :alert
|
84
90
|
'danger'
|
91
|
+
when :warning
|
92
|
+
'#FF9900'
|
85
93
|
when :info
|
86
94
|
'#0099CC'
|
87
95
|
else
|
data/lib/systemd_mon/state.rb
CHANGED
@@ -8,7 +8,7 @@ module SystemdMon
|
|
8
8
|
|
9
9
|
def initialize(active, sub, loaded, unit_file)
|
10
10
|
timestamp = Time.now
|
11
|
-
@active = StateValue.new("active", active, timestamp, %w(active), %w(inactive))
|
11
|
+
@active = StateValue.new("active", active, timestamp, %w(active), %w(inactive failed))
|
12
12
|
@sub = StateValue.new("status", sub, timestamp)
|
13
13
|
@loaded = StateValue.new("loaded", loaded, timestamp, %w(loaded))
|
14
14
|
@unit_file = StateValue.new("file", unit_file, timestamp, %w(enabled), %w(disabled))
|
@@ -48,6 +48,10 @@ module SystemdMon
|
|
48
48
|
first.ok? && last.ok? && changes.any? { |s| s.active == "deactivating" }
|
49
49
|
end
|
50
50
|
|
51
|
+
def auto_restart?
|
52
|
+
first.ok? && last.ok? && changes.any? { |s| s.sub == "auto-restart" }
|
53
|
+
end
|
54
|
+
|
51
55
|
def reload?
|
52
56
|
first.ok? && last.ok? && changes.any? { |s| s.active == "reloading" }
|
53
57
|
end
|
@@ -59,6 +63,8 @@ module SystemdMon
|
|
59
63
|
def status_text
|
60
64
|
if recovery?
|
61
65
|
"recovered"
|
66
|
+
elsif auto_restart?
|
67
|
+
"automatically restarted"
|
62
68
|
elsif restart?
|
63
69
|
"restarted"
|
64
70
|
elsif reload?
|
data/lib/systemd_mon/version.rb
CHANGED
data/systemd_mon.gemspec
CHANGED
@@ -18,7 +18,7 @@ Gem::Specification.new do |spec|
|
|
18
18
|
spec.test_files = spec.files.grep(%r{^(test|spec|features)/})
|
19
19
|
spec.require_paths = ["lib"]
|
20
20
|
|
21
|
-
spec.add_dependency "ruby-dbus"
|
21
|
+
spec.add_dependency "ruby-dbus", "~> 0.11.0"
|
22
22
|
spec.add_development_dependency "bundler", "~> 1.6"
|
23
23
|
spec.add_development_dependency "rake"
|
24
24
|
end
|
metadata
CHANGED
@@ -1,29 +1,29 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: systemd_mon
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Jon Cairns
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2014-08-
|
11
|
+
date: 2014-08-29 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: ruby-dbus
|
15
15
|
requirement: !ruby/object:Gem::Requirement
|
16
16
|
requirements:
|
17
|
-
- - "
|
17
|
+
- - "~>"
|
18
18
|
- !ruby/object:Gem::Version
|
19
|
-
version:
|
19
|
+
version: 0.11.0
|
20
20
|
type: :runtime
|
21
21
|
prerelease: false
|
22
22
|
version_requirements: !ruby/object:Gem::Requirement
|
23
23
|
requirements:
|
24
|
-
- - "
|
24
|
+
- - "~>"
|
25
25
|
- !ruby/object:Gem::Version
|
26
|
-
version:
|
26
|
+
version: 0.11.0
|
27
27
|
- !ruby/object:Gem::Dependency
|
28
28
|
name: bundler
|
29
29
|
requirement: !ruby/object:Gem::Requirement
|