litmus_paper 1.1.1 → 1.2.0
Sign up to get free protection for your applications and to get access to all the features.
- data/Dockerfile +20 -0
- data/README.md +279 -3
- data/docker/litmus.conf +7 -0
- data/docker/litmus_unicorn.rb +11 -0
- data/lib/litmus_paper/agent_check_handler.rb +14 -1
- data/lib/litmus_paper/cache.rb +24 -6
- data/lib/litmus_paper/configuration_file.rb +78 -1
- data/lib/litmus_paper/util.rb +12 -0
- data/lib/litmus_paper/version.rb +1 -1
- data/lib/litmus_paper.rb +1 -0
- data/spec/litmus_paper/agent_check_handler_spec.rb +38 -0
- data/spec/litmus_paper/cache_spec.rb +8 -0
- data/spec/litmus_paper/configuration_file_spec.rb +59 -0
- data/spec/spec_helper.rb +3 -0
- data/spec/support/config.d.mixed/.passing_test.config.swp +0 -0
- data/spec/support/config.d.mixed/passing_test.config +5 -0
- data/spec/support/config.d.mixed/test.config.yaml +9 -0
- data/spec/support/config.d.yaml/passing_test.config.yaml +6 -0
- data/spec/support/config.d.yaml/test.config.yaml +9 -0
- data/spec/support/test.config.yaml +20 -0
- data/spec/support/test.d.config.mixed +2 -0
- data/spec/support/test.d.config.yaml +2 -0
- metadata +23 -3
data/Dockerfile
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
FROM debian:stretch
|
2
|
+
EXPOSE 9293/TCP
|
3
|
+
EXPOSE 9294/TCP
|
4
|
+
WORKDIR /home/litmus_paper
|
5
|
+
RUN apt-get update && apt-get install -y ruby ruby-dev git curl rsyslog build-essential
|
6
|
+
RUN gem install --no-ri --no-rdoc bundler \
|
7
|
+
&& gem install sinatra --no-ri --no-rdoc --version "~> 1.3.2" \
|
8
|
+
&& gem install remote_syslog_logger --no-ri --no-rdoc --version "~> 1.0.3" \
|
9
|
+
&& gem install unicorn --no-ri --no-rdoc --version "~> 4.6.2" \
|
10
|
+
&& gem install colorize --no-ri --no-rdoc \
|
11
|
+
&& gem install rspec --no-ri --no-rdoc --version "~> 2.9.0" \
|
12
|
+
&& gem install rack-test --no-ri --no-rdoc --version "~> 0.6.1" \
|
13
|
+
&& gem install rake --no-ri --no-rdoc --version "~> 0.9.2.2" \
|
14
|
+
&& gem install rake_commit --no-ri --no-rdoc --version "~> 0.13"
|
15
|
+
ADD . /home/litmus_paper
|
16
|
+
RUN ln -sf /home/litmus_paper/docker/litmus.conf /etc/litmus.conf \
|
17
|
+
&& ln -sf /home/litmus_paper/docker/litmus_unicorn.rb /etc/litmus_unicorn.rb
|
18
|
+
RUN gem build litmus_paper.gemspec && gem install litmus_paper*.gem
|
19
|
+
|
20
|
+
CMD ["bin/litmus", "-p", "9293", "-c", "/etc/litmus_unicorn.rb"]
|
data/README.md
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
# LitmusPaper
|
2
2
|
|
3
|
-
|
3
|
+
LitmusPaper is a backend health tester for Highly Available (HA) services.
|
4
4
|
|
5
5
|
[![Build Status](https://secure.travis-ci.org/braintree/litmus_paper.png)](http://travis-ci.org/braintree/litmus_paper)
|
6
6
|
|
@@ -18,10 +18,251 @@ Or install it yourself as:
|
|
18
18
|
|
19
19
|
$ gem install litmus_paper
|
20
20
|
|
21
|
+
## Overview
|
22
|
+
|
23
|
+
Litmus Paper reports health for each service on a node (like a server, vm, or container) on a 0-100 scale. Health is computed by aggregating the values returned from running various subchecks. Health information can be queried through a REST API for consumption by other services (like load balancers or monitoring systems), or queried on the command line.
|
24
|
+
|
25
|
+
There are two classes of subchecks: Dependencies and Metrics. Dependencies report either 0 or 100 health, and aggregate such that if any dependency is not 100 the whole service is down. Metrics report health on a scale from 0-100 and aggregate as averages based on their weight.
|
26
|
+
|
27
|
+
You can also force a service to report a health value on a host.
|
28
|
+
Forcing a service up makes it report a health of 100, regardless of the measured health.
|
29
|
+
Forcing a service down makes it report a health of 0.
|
30
|
+
Forcing a service's health to a value between 0 and 100 places a ceiling on its health. The service will report the lower of the measured health or the forced health value.
|
31
|
+
|
32
|
+
Force downs take precedence, followed by force ups, and finally force healths. If you specify both a force up and a force down, the service will report 0 health. If you specify a force up and a force health, the force health will be ignored and the service will report 100 health.
|
33
|
+
|
34
|
+
Using litmus-agent-check, Litmus can also output health information in haproxy agent check format.
|
35
|
+
|
21
36
|
## Usage
|
22
37
|
|
23
|
-
|
24
|
-
|
38
|
+
### Running Litmus Paper
|
39
|
+
|
40
|
+
Start the process under unicorn with `/usr/bin/litmus --unicorn-config <path_to_unicorn_conf>`. In the unicorn config file, set the number of worker processes, the pid, and the working directory. See the [unicorn documentation](https://bogomips.org/unicorn/Unicorn/Configurator.html) for the config format. There are a few command line options:
|
41
|
+
|
42
|
+
```
|
43
|
+
Usage: litmus [options]
|
44
|
+
|
45
|
+
-b, --binding=ip Binds Litmus to the specified ip.
|
46
|
+
Default: 0.0.0.0
|
47
|
+
-d, --daemon Make server run as a Daemon.
|
48
|
+
-p, --port=port Listen Port
|
49
|
+
-c, --unicorn-config=config Unicorn Config
|
50
|
+
|
51
|
+
-h, --help Show this help message.
|
52
|
+
```
|
53
|
+
|
54
|
+
For HAProxy agent checks, run `/usr/bin/litmus-agent-check`. See the "HAProxy Agent Check Configuration" section below for a full list of options.
|
55
|
+
|
56
|
+
### Global configuration
|
57
|
+
|
58
|
+
This can be either yaml or ruby.
|
59
|
+
|
60
|
+
Examples:
|
61
|
+
|
62
|
+
```
|
63
|
+
# /etc/litmus.conf
|
64
|
+
|
65
|
+
include_files "litmus.d/*.conf"
|
66
|
+
|
67
|
+
port 80
|
68
|
+
|
69
|
+
data_directory "/litmus"
|
70
|
+
```
|
71
|
+
|
72
|
+
```yaml
|
73
|
+
# /etc/litmus.conf.yaml
|
74
|
+
---
|
75
|
+
include-files: 'litmus.d/*.yaml'
|
76
|
+
port: 80
|
77
|
+
data_directory: '/litmus'
|
78
|
+
```
|
79
|
+
|
80
|
+
Available fields:
|
81
|
+
- include_files: Tells Litmus to load health check configurations from a path.
|
82
|
+
- port: Port Litmus unicorn server will listen on, defaults to 9292.
|
83
|
+
- data_directory: Where to store force down, up, and health files. Defaults to "/etc/litmus".
|
84
|
+
- cache_location: Where to store cached health information, defaults to "/run/shm".
|
85
|
+
- cache_ttl: Time to live in seconds for cached health check values, defaults to -1.
|
86
|
+
|
87
|
+
### Service health check configuration
|
88
|
+
|
89
|
+
To add services and health checks, Litmus Paper loads YAML configurations or ruby code. The examples below show both styles.
|
90
|
+
|
91
|
+
Suppose you're writing a health check for a web application. You might start with a simple check to report if the server is responding at all:
|
92
|
+
|
93
|
+
```yaml
|
94
|
+
# /etc/litmus.d/myapp.yaml
|
95
|
+
---
|
96
|
+
services:
|
97
|
+
myapp:
|
98
|
+
dependencies:
|
99
|
+
- type: http
|
100
|
+
uri: 'https://localhost/heartbeat'
|
101
|
+
method: 'GET'
|
102
|
+
ca_file: '/etc/ssl/certs/ca-certificates.crt'
|
103
|
+
```
|
104
|
+
|
105
|
+
```ruby
|
106
|
+
# /etc/litmus.d/myapp.conf
|
107
|
+
service "myapp" do |s|
|
108
|
+
s.depends Dependency::HTTP, "https://localhost/heartbeat", :method => "GET", :ca_file => "/etc/ssl/certs/ca-certificates.crt"
|
109
|
+
end
|
110
|
+
```
|
111
|
+
|
112
|
+
Maybe you also want to balance traffic based on CPU load:
|
113
|
+
|
114
|
+
```yaml
|
115
|
+
# /etc/litmus.d/myapp.yaml
|
116
|
+
---
|
117
|
+
services:
|
118
|
+
myapp:
|
119
|
+
dependencies:
|
120
|
+
- type: http
|
121
|
+
uri: 'https://localhost/heartbeat'
|
122
|
+
method: 'GET'
|
123
|
+
ca_file: '/etc/ssl/certs/ca-certificates.crt'
|
124
|
+
checks:
|
125
|
+
- type: cpu_load
|
126
|
+
weight: 100
|
127
|
+
```
|
128
|
+
|
129
|
+
```ruby
|
130
|
+
# /etc/litmus.d/myapp.conf
|
131
|
+
service "myapp" do |s|
|
132
|
+
s.depends Dependency::HTTP, "https://localhost/heartbeat", :method => "GET", :ca_file => "/etc/ssl/certs/ca-certificates.crt"
|
133
|
+
s.measure_health Metric::CPULoad, :weight => 100
|
134
|
+
end
|
135
|
+
```
|
136
|
+
|
137
|
+
Once you've finished adding checks, restart litmus paper to pick up the new service.
|
138
|
+
|
139
|
+
Here are all the types of checks currently implemented:
|
140
|
+
|
141
|
+
- `http` (`Dependency::HTTP`): Checks HTTP 200 response from a URI.
|
142
|
+
* uri
|
143
|
+
* method (defaults to get)
|
144
|
+
* ca_file (defaults to nil)
|
145
|
+
* timeout (defaults to 2s)
|
146
|
+
|
147
|
+
- `tcp` (`Dependency::TCP`): Checks successful completion of a TCP handshake with an address.
|
148
|
+
* ip
|
149
|
+
* port
|
150
|
+
* input_data (defaults to nil)
|
151
|
+
* expected_output (defaults to nil)
|
152
|
+
* timeout (defaults to 2s)
|
153
|
+
|
154
|
+
- `file_contents` (`Dependency::FileContents`): Checks whether the contents of a file match a string or regex.
|
155
|
+
* path
|
156
|
+
* regex
|
157
|
+
* timeout (defaults to 5s)
|
158
|
+
|
159
|
+
- `script` (`Dependency::Script`): Checks whether a command exits with 0 (success) or not (failure).
|
160
|
+
* command
|
161
|
+
* timeout (defaults to 5s)
|
162
|
+
|
163
|
+
- `constant_metric` (`Metric::ConstantMetric`): A dummy metric that always reports a constant.
|
164
|
+
* weight (0-100)
|
165
|
+
|
166
|
+
- `cpu_load` (`Metric::CPULoad`): Normalizes CPU load to a value between 1-100 and inverts it, so higher numbers mean less load and lower numbers mean more. Final health is weighted against other checks by `:weight`. The lower bound of 1 ensures that nodes will not leave the cluster solely based on CPU load. An example of how allowing 0 can cause problems: If one node has 4 CPUs and a load of 4 with CPU usage weighted at 100, it will report its health as 0, and all traffic will be shifted towards other nodes. These nodes in turn hit 100% CPU usage and report 0 health, causing a cascade of exiting nodes that shuts down the service.
|
167
|
+
* weight (1-100)
|
168
|
+
|
169
|
+
- `internet_health` (`Metric::InternetHealth`): Checks connectivity across a set of hosts and computes a weight based on how many are reachable. Helpful if you want to check outbound connectivity through multiple ISPs.
|
170
|
+
* weight (0-100)
|
171
|
+
* hosts
|
172
|
+
* timeout (defaults to 5s)
|
173
|
+
|
174
|
+
- `script` (`Metric::Script`): Runs a script to obtain a health from 0-100. This is helpful for customized metrics.
|
175
|
+
* command
|
176
|
+
* weight (0-100)
|
177
|
+
* timeout (defaults to 5s)
|
178
|
+
|
179
|
+
- `big_brother_service` (`Metric::BigBrotherService`): Used in conjunction with [Big Brother](https://github.com/braintree/big_brother), reports health based on the overall health of another load balanced service.
|
180
|
+
* service
|
181
|
+
|
182
|
+
### HAProxy agent check configuration
|
183
|
+
|
184
|
+
Litmus paper can also report health checks in HAProxy agent check format. The agent check functionality takes the health data from a service health check, and exposes it on a different port in the format HAProxy expects.
|
185
|
+
|
186
|
+
There are no additional configuration files for the agent check, since all options are specified on the command line. Services are configured as normal litmus services as described above.
|
187
|
+
|
188
|
+
```
|
189
|
+
Usage: litmus-agent-check [options]
|
190
|
+
-s, --service SERVICE:PORT,... agent-check service to port mappings
|
191
|
+
-c, --config CONFIG Path to litmus paper config file
|
192
|
+
-p, --pid-file PID_FILE Where to write the pid
|
193
|
+
-w, --workers WORKERS Number of worker processes
|
194
|
+
-D, --daemonize Daemonize the process
|
195
|
+
-h, --help Help text
|
196
|
+
```
|
197
|
+
|
198
|
+
The service:port argument means that the server will expose the data from the litmus check for `service` on `port` in HAProxy agent check format. For example, if you wanted to serve status information about `myapp` on port `8080`, and already had a service config for it, you'd pass `-s myapp:8080`.
|
199
|
+
|
200
|
+
On the HAProxy server, add `agent-check agent-port 8080 agent-inter <seconds>s` to the config line for each server listed for that backend. This tells HAProxy to query port 8080 on the backend every `<seconds>` seconds for health information. See the [HAProxy agent check documentation](https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.2-agent-check) for more details.
|
201
|
+
|
202
|
+
### REST API
|
203
|
+
|
204
|
+
The REST API is the main way other services should interact with Litmus. For routes that take parameters, pass them as form parameters in the request body.
|
205
|
+
|
206
|
+
- /
|
207
|
+
* GET: Returns table with status of each service.
|
208
|
+
|
209
|
+
Litmus Paper 1.1.1
|
210
|
+
|
211
|
+
Service │ Reported │ Measured │ Health
|
212
|
+
Name │ Health │ Health │ Forced?
|
213
|
+
────────────┴──────────┴──────────┴─────────
|
214
|
+
myapp 0 100 Yes, Reason: testing
|
215
|
+
myotherapp 72 72 No
|
216
|
+
|
217
|
+
- /down
|
218
|
+
* POST: Creates a global force down. Parameters: reason => reason for the force down.
|
219
|
+
* DELETE: Deletes global force down, if one is in place. Any service-specific force downs remain in effect.
|
220
|
+
|
221
|
+
- /up
|
222
|
+
* POST: Creates a global force up. Parameters: reason => reason for the force up.
|
223
|
+
* DELETE: Deletes global force up, if one is in place. Any service-specific force ups remain in effect.
|
224
|
+
|
225
|
+
- /health
|
226
|
+
* POST: Creates a global force health. Parameters: reason => reason for the force health, health => health to force to.
|
227
|
+
* DELETE: Deletes global force health, if one is in place. Any service-specific force healths remain in effect.
|
228
|
+
|
229
|
+
- /`<service>`/status
|
230
|
+
* GET: Returns a detailed status output for `<service>`, including all the subchecks. This output easier to parse than the output from GET /. Also sets X-Health and X-Health-Forced headers in the response.
|
231
|
+
|
232
|
+
```
|
233
|
+
Health: 82
|
234
|
+
Measured Health: 82
|
235
|
+
Dependency::HTTP(http://localhost/heartbeat): OK
|
236
|
+
Metric::CPULoad(100): 82
|
237
|
+
```
|
238
|
+
|
239
|
+
- /`<service>`/down
|
240
|
+
* POST: Creates a force down just for `<service>`. Parameters: reason => reason for the force down.
|
241
|
+
* DELETE: Deletes force down for `<service>`, if one is in place. Global force downs remain in effect.
|
242
|
+
|
243
|
+
- /`<service>`/up
|
244
|
+
* POST: Creates a force up just for `<service>`. Parameters: reason => reason for the force up.
|
245
|
+
* DELETE: Deletes force up for `<service>`, if one is in place. Global force ups remain in effect.
|
246
|
+
|
247
|
+
- /`<service>`/health
|
248
|
+
* POST: Creates a force health just for `<service>`. Parameters: reason => reason for the force health, health => health to force to.
|
249
|
+
* DELETE: Deletes force health for `<service>`, if one is in place. Global force healths remain in effect.
|
250
|
+
|
251
|
+
### Litmusctl
|
252
|
+
|
253
|
+
There is also a CLI included called `litmusctl`.
|
254
|
+
|
255
|
+
This has the same functionality as the rest API:
|
256
|
+
- `litmusctl list` = `GET /`
|
257
|
+
- `litmusctl status <service>` = `GET /<service>/status`
|
258
|
+
- `litmusctl force (down|up|health) [health] [-r REASON]` = `POST /(down|up|health)`
|
259
|
+
- `litmusctl force (down|up|health) [health] <service> [-r REASON]` = `POST /<service>/(down|up|health)`
|
260
|
+
- `litmusctl force (down|up|health) -d` = `DELETE /(down|up|health)`
|
261
|
+
- `litmusctl force (down|up|health) <service> -d` = `DELETE /<service>/(down|up|health)`
|
262
|
+
|
263
|
+
## Tests
|
264
|
+
|
265
|
+
Run tests using `rake`. The default task runs all the tests.
|
25
266
|
|
26
267
|
## Releasing
|
27
268
|
|
@@ -34,3 +275,38 @@ use it as an agent-check for HAProxy.
|
|
34
275
|
3. Commit your changes (`git commit -am 'Added some feature'`)
|
35
276
|
4. Push to the branch (`git push origin my-new-feature`)
|
36
277
|
5. Create new Pull Request
|
278
|
+
|
279
|
+
## Docker
|
280
|
+
|
281
|
+
We've provided a simple docker image that can be used to try out litmus_paper, run tests, or to assist in feature development.
|
282
|
+
|
283
|
+
To run it, from the top of your copy of the litmus_paper repository:
|
284
|
+
|
285
|
+
```
|
286
|
+
docker build -t litmus_paper .
|
287
|
+
# Run litmus_paper
|
288
|
+
docker run -d -p 9293:9293 litmus_paper
|
289
|
+
```
|
290
|
+
|
291
|
+
Then, you can `curl localhost:9293` from your host machine to interact with the REST API.
|
292
|
+
|
293
|
+
To run interactively and attach to the container:
|
294
|
+
|
295
|
+
```
|
296
|
+
docker run -it litmus_paper /bin/bash
|
297
|
+
# bin/litmus -p 9293 -d
|
298
|
+
# curl localhost:9293
|
299
|
+
```
|
300
|
+
|
301
|
+
To run litmus-agent-check on port 9294:
|
302
|
+
|
303
|
+
```
|
304
|
+
# Run litmus paper with litmus-agent-check using 10 workers
|
305
|
+
docker run -d -p 9294:9294 litmus_paper bin/litmus-agent-check -s test:9294 -c /etc/litmus.conf -w 10
|
306
|
+
```
|
307
|
+
|
308
|
+
## TODO
|
309
|
+
|
310
|
+
1. Accept configuration in either YAML or ruby format.
|
311
|
+
2. Improve concurrency model, with a health-check process and a responder process.
|
312
|
+
3. Improve concurrency of agent-check daemon.
|
data/docker/litmus.conf
ADDED
@@ -0,0 +1,11 @@
|
|
1
|
+
# vim: set ft=ruby
|
2
|
+
require 'remote_syslog_logger'
|
3
|
+
|
4
|
+
APP_ROOT = '/var/lib/gems/2.3.0/gems/litmus_paper-1.1.1'
|
5
|
+
|
6
|
+
worker_processes 5
|
7
|
+
working_directory APP_ROOT
|
8
|
+
|
9
|
+
logger(RemoteSyslogLogger.new('127.0.0.1', 514, :program => 'litmus_paper', :facility => 'daemon'))
|
10
|
+
|
11
|
+
pid "/tmp/unicorn.pid"
|
@@ -1,8 +1,21 @@
|
|
1
1
|
module LitmusPaper
|
2
2
|
class AgentCheckHandler
|
3
3
|
def self.handle(service)
|
4
|
+
@cache ||= LitmusPaper::Cache.new(
|
5
|
+
LitmusPaper.cache_location,
|
6
|
+
"litmus_agent_cache",
|
7
|
+
LitmusPaper.cache_ttl
|
8
|
+
)
|
4
9
|
output = []
|
5
|
-
|
10
|
+
|
11
|
+
health = @cache.get(service)
|
12
|
+
if !health
|
13
|
+
@cache.set(
|
14
|
+
service,
|
15
|
+
health = LitmusPaper.check_service(service)
|
16
|
+
)
|
17
|
+
end
|
18
|
+
|
6
19
|
if health.nil?
|
7
20
|
output << "failed#NOT_FOUND"
|
8
21
|
else
|
data/lib/litmus_paper/cache.rb
CHANGED
@@ -1,3 +1,5 @@
|
|
1
|
+
require 'fileutils'
|
2
|
+
require 'tempfile'
|
1
3
|
require 'yaml'
|
2
4
|
|
3
5
|
module LitmusPaper
|
@@ -11,12 +13,24 @@ module LitmusPaper
|
|
11
13
|
|
12
14
|
def set(key, value)
|
13
15
|
return unless @ttl > 0
|
14
|
-
File.
|
15
|
-
|
16
|
-
f
|
17
|
-
|
18
|
-
|
19
|
-
|
16
|
+
filename = File.join(@path, key)
|
17
|
+
if File.exist?(filename)
|
18
|
+
File.open(filename, "r+") do |f|
|
19
|
+
f.flock(File::LOCK_EX)
|
20
|
+
f.rewind
|
21
|
+
f.write(_entry(@ttl, value))
|
22
|
+
f.flush
|
23
|
+
f.truncate(f.pos)
|
24
|
+
end
|
25
|
+
else
|
26
|
+
temp = Tempfile.new("#{key}_init", @path)
|
27
|
+
begin
|
28
|
+
temp.write(_entry(@ttl, value))
|
29
|
+
temp.flush
|
30
|
+
ensure
|
31
|
+
temp.close
|
32
|
+
end
|
33
|
+
FileUtils.mv(temp.path, filename)
|
20
34
|
end
|
21
35
|
end
|
22
36
|
|
@@ -29,5 +43,9 @@ module LitmusPaper
|
|
29
43
|
expires_at.to_f < Time.now.to_f ? nil : YAML::load(value)
|
30
44
|
end
|
31
45
|
end
|
46
|
+
|
47
|
+
def _entry(ttl, value)
|
48
|
+
"#{Time.now.to_f + ttl} #{YAML::dump(value)}"
|
49
|
+
end
|
32
50
|
end
|
33
51
|
end
|
@@ -1,3 +1,5 @@
|
|
1
|
+
require 'yaml'
|
2
|
+
|
1
3
|
module LitmusPaper
|
2
4
|
class ConfigurationFile
|
3
5
|
def initialize(config_file_path)
|
@@ -12,7 +14,11 @@ module LitmusPaper
|
|
12
14
|
def evaluate(file = @config_file_path)
|
13
15
|
LitmusPaper.logger.info "Loading file #{file}"
|
14
16
|
config_contents = File.read(file)
|
15
|
-
|
17
|
+
if file =~ /\.(yaml|yml)/
|
18
|
+
load_yaml(config_contents)
|
19
|
+
else
|
20
|
+
instance_eval(config_contents)
|
21
|
+
end
|
16
22
|
LitmusPaper::Configuration.new(
|
17
23
|
@port,
|
18
24
|
@data_directory,
|
@@ -45,6 +51,64 @@ module LitmusPaper
|
|
45
51
|
@services[name.to_s] = service
|
46
52
|
end
|
47
53
|
|
54
|
+
def yaml_service(value)
|
55
|
+
value.each do |name, config|
|
56
|
+
config = Util.symbolize_keys(config)
|
57
|
+
dependencies = parse_yaml_dependencies(config.fetch(:dependencies, []))
|
58
|
+
checks = parse_yaml_checks(config.fetch(:checks, []))
|
59
|
+
service = Service.new(name.to_s, dependencies, checks)
|
60
|
+
@services[name.to_s] = service
|
61
|
+
end
|
62
|
+
end
|
63
|
+
|
64
|
+
def parse_yaml_checks(config)
|
65
|
+
config.map do |check|
|
66
|
+
check_config = Util.symbolize_keys(check)
|
67
|
+
case check[:type].to_sym
|
68
|
+
when :big_brother_service
|
69
|
+
Metric::BigBrotherService.new(check_config.delete(:service))
|
70
|
+
when :cpu_load
|
71
|
+
Metric::CPULoad.new(check_config.delete(:weight))
|
72
|
+
when :constant_metric
|
73
|
+
Metric::ConstantMetric.new(check_config.delete(:weight))
|
74
|
+
when :internet_health
|
75
|
+
weight = check_config.delete(:weight)
|
76
|
+
hosts = check_config.delete(:hosts)
|
77
|
+
Metric::InternetHealth.new(weight, hosts, check_config)
|
78
|
+
when :script
|
79
|
+
command = check_config.delete(:command)
|
80
|
+
weight = check_config.delete(:weight)
|
81
|
+
Metric::Script.new(command, weight, check_config)
|
82
|
+
end
|
83
|
+
end
|
84
|
+
end
|
85
|
+
|
86
|
+
def parse_yaml_dependencies(config)
|
87
|
+
config.map do |dep|
|
88
|
+
dep_config = Util.symbolize_keys(dep)
|
89
|
+
case dep[:type].to_sym
|
90
|
+
when :file_contents
|
91
|
+
path = dep_config.delete(:path)
|
92
|
+
regex = dep_config.delete(:regex)
|
93
|
+
Dependency::FileContents.new(path, regex, dep_config)
|
94
|
+
when :haproxy_backends
|
95
|
+
domain_socket = dep_config.delete(:domain_socket)
|
96
|
+
cluster = dep_config.delete(:cluster)
|
97
|
+
Dependency::HaproxyBackends.new(domain_socket, cluster, dep_config)
|
98
|
+
when :http
|
99
|
+
uri = dep_config.delete(:uri)
|
100
|
+
Dependency::HTTP.new(uri, dep_config)
|
101
|
+
when "script"
|
102
|
+
command = dep_config.delete(:command)
|
103
|
+
Dependency::Script.new(command, options)
|
104
|
+
when "tcp"
|
105
|
+
ip = dep_config.delete(:ip)
|
106
|
+
port = dep_config.delete(:port)
|
107
|
+
Dependency::TCP.new(ip, port, dep_config)
|
108
|
+
end
|
109
|
+
end
|
110
|
+
end
|
111
|
+
|
48
112
|
def cache_location(location)
|
49
113
|
@cache_location = location
|
50
114
|
end
|
@@ -52,5 +116,18 @@ module LitmusPaper
|
|
52
116
|
def cache_ttl(ttl)
|
53
117
|
@cache_ttl = ttl
|
54
118
|
end
|
119
|
+
|
120
|
+
def load_yaml(contents)
|
121
|
+
config = Util.symbolize_keys(YAML.load(contents))
|
122
|
+
|
123
|
+
config.each do |key, value|
|
124
|
+
case key
|
125
|
+
when :include_files, :data_directory, :cache_location, :cache_ttl, :port
|
126
|
+
send(key, value)
|
127
|
+
when :services
|
128
|
+
yaml_service(value)
|
129
|
+
end
|
130
|
+
end
|
131
|
+
end
|
55
132
|
end
|
56
133
|
end
|
data/lib/litmus_paper/version.rb
CHANGED
data/lib/litmus_paper.rb
CHANGED
@@ -2,6 +2,10 @@ require 'spec_helper'
|
|
2
2
|
require 'litmus_paper/agent_check_handler'
|
3
3
|
|
4
4
|
describe LitmusPaper::AgentCheckHandler do
|
5
|
+
def app
|
6
|
+
LitmusPaper::App
|
7
|
+
end
|
8
|
+
|
5
9
|
before :each do
|
6
10
|
LitmusPaper.configure(TEST_CONFIG)
|
7
11
|
end
|
@@ -73,5 +77,39 @@ describe LitmusPaper::AgentCheckHandler do
|
|
73
77
|
LitmusPaper::StatusFile.global_up_file.create("Up for testing")
|
74
78
|
LitmusPaper::AgentCheckHandler.handle("test").should == "ready\tup\t100%"
|
75
79
|
end
|
80
|
+
|
81
|
+
it "retrieves a cached value during the cache_ttl" do
|
82
|
+
begin
|
83
|
+
cache = LitmusPaper::AgentCheckHandler.instance_variable_get(:@cache)
|
84
|
+
cache.instance_variable_set(:@ttl, 0.25)
|
85
|
+
LitmusPaper::AgentCheckHandler.instance_variable_set(:@cache, cache)
|
86
|
+
|
87
|
+
test_service = LitmusPaper::Service.new(
|
88
|
+
'test',
|
89
|
+
[AlwaysAvailableDependency.new],
|
90
|
+
[LitmusPaper::Metric::ConstantMetric.new(100)]
|
91
|
+
)
|
92
|
+
LitmusPaper.services['test'] = test_service
|
93
|
+
|
94
|
+
post "/test/health", :reason => "health for testing", :health => 88
|
95
|
+
last_response.status.should == 201
|
96
|
+
|
97
|
+
LitmusPaper::AgentCheckHandler.handle('test').should == "ready\tup\t88%"
|
98
|
+
|
99
|
+
delete "/test/health"
|
100
|
+
last_response.status.should == 200
|
101
|
+
|
102
|
+
LitmusPaper::AgentCheckHandler.handle('test').should == "ready\tup\t88%"
|
103
|
+
|
104
|
+
sleep 0.25
|
105
|
+
|
106
|
+
LitmusPaper::AgentCheckHandler.handle('test').should_not == "ready\tup\t88%"
|
107
|
+
ensure
|
108
|
+
FileUtils.rm_rf(LitmusPaper.cache_location)
|
109
|
+
cache = LitmusPaper::AgentCheckHandler.instance_variable_get(:@cache)
|
110
|
+
cache.instance_variable_set(:@ttl, -1)
|
111
|
+
LitmusPaper::AgentCheckHandler.instance_variable_set(:@cache, cache)
|
112
|
+
end
|
113
|
+
end
|
76
114
|
end
|
77
115
|
end
|
@@ -70,5 +70,13 @@ describe LitmusPaper::Cache do
|
|
70
70
|
sleep ttl
|
71
71
|
cache.get(key).should be_nil
|
72
72
|
end
|
73
|
+
|
74
|
+
it "works when setting multiple entries" do
|
75
|
+
key = "key"
|
76
|
+
cache = LitmusPaper::Cache.new(@location, @namespace, 1)
|
77
|
+
cache.set(key, "some value")
|
78
|
+
cache.set(key, "other value")
|
79
|
+
cache.get(key).should == "other value"
|
80
|
+
end
|
73
81
|
end
|
74
82
|
end
|
@@ -8,29 +8,59 @@ describe LitmusPaper::ConfigurationFile do
|
|
8
8
|
config.services.has_key?('test').should == true
|
9
9
|
end
|
10
10
|
|
11
|
+
it "configures a service via yaml" do
|
12
|
+
config_file = LitmusPaper::ConfigurationFile.new(TEST_YAML_CONFIG)
|
13
|
+
config = config_file.evaluate
|
14
|
+
config.services.has_key?('test').should == true
|
15
|
+
end
|
16
|
+
|
11
17
|
it "configures the port to listen on" do
|
12
18
|
config_file = LitmusPaper::ConfigurationFile.new(TEST_CONFIG)
|
13
19
|
config = config_file.evaluate
|
14
20
|
config.port.should == 9293
|
15
21
|
end
|
16
22
|
|
23
|
+
it "configures the port to listen on via yaml" do
|
24
|
+
config_file = LitmusPaper::ConfigurationFile.new(TEST_YAML_CONFIG)
|
25
|
+
config = config_file.evaluate
|
26
|
+
config.port.should == 9293
|
27
|
+
end
|
28
|
+
|
17
29
|
it "configures the data directory" do
|
18
30
|
config_file = LitmusPaper::ConfigurationFile.new(TEST_CONFIG)
|
19
31
|
config = config_file.evaluate
|
20
32
|
config.data_directory.should == "/tmp/litmus_paper"
|
21
33
|
end
|
22
34
|
|
35
|
+
it "configures the data directory via yaml" do
|
36
|
+
config_file = LitmusPaper::ConfigurationFile.new(TEST_YAML_CONFIG)
|
37
|
+
config = config_file.evaluate
|
38
|
+
config.data_directory.should == "/tmp/litmus_paper"
|
39
|
+
end
|
40
|
+
|
23
41
|
it "configures the cache_location" do
|
24
42
|
config_file = LitmusPaper::ConfigurationFile.new(TEST_CONFIG)
|
25
43
|
config = config_file.evaluate
|
26
44
|
config.cache_location.should == "/tmp/litmus_paper_cache"
|
27
45
|
end
|
28
46
|
|
47
|
+
it "configures the cache_location via yaml" do
|
48
|
+
config_file = LitmusPaper::ConfigurationFile.new(TEST_YAML_CONFIG)
|
49
|
+
config = config_file.evaluate
|
50
|
+
config.cache_location.should == "/tmp/litmus_paper_cache"
|
51
|
+
end
|
52
|
+
|
29
53
|
it "configures the cache_ttl" do
|
30
54
|
config_file = LitmusPaper::ConfigurationFile.new(TEST_CONFIG)
|
31
55
|
config = config_file.evaluate
|
32
56
|
config.cache_ttl.should == -1
|
33
57
|
end
|
58
|
+
|
59
|
+
it "configures the cache_ttl via yaml" do
|
60
|
+
config_file = LitmusPaper::ConfigurationFile.new(TEST_YAML_CONFIG)
|
61
|
+
config = config_file.evaluate
|
62
|
+
config.cache_ttl.should == -1
|
63
|
+
end
|
34
64
|
end
|
35
65
|
|
36
66
|
describe "include_files" do
|
@@ -40,6 +70,19 @@ describe LitmusPaper::ConfigurationFile do
|
|
40
70
|
config.services.has_key?('test').should == true
|
41
71
|
end
|
42
72
|
|
73
|
+
it "configures a dir glob of services in yaml" do
|
74
|
+
config_file = LitmusPaper::ConfigurationFile.new(TEST_YAML_D_CONFIG)
|
75
|
+
config = config_file.evaluate
|
76
|
+
config.services.has_key?('test').should == true
|
77
|
+
end
|
78
|
+
|
79
|
+
it "configures a dir glob of mixed services" do
|
80
|
+
config_file = LitmusPaper::ConfigurationFile.new(TEST_MIXED_D_CONFIG)
|
81
|
+
config = config_file.evaluate
|
82
|
+
config.services.has_key?('test').should == true
|
83
|
+
config.services.has_key?('passing_test').should == true
|
84
|
+
end
|
85
|
+
|
43
86
|
it "defaults configuration options" do
|
44
87
|
config_file = LitmusPaper::ConfigurationFile.new(TEST_D_CONFIG)
|
45
88
|
config = config_file.evaluate
|
@@ -47,5 +90,21 @@ describe LitmusPaper::ConfigurationFile do
|
|
47
90
|
config.port.should == 9292
|
48
91
|
config.data_directory.should == "/etc/litmus"
|
49
92
|
end
|
93
|
+
|
94
|
+
it "defaults configuration options in yaml" do
|
95
|
+
config_file = LitmusPaper::ConfigurationFile.new(TEST_YAML_D_CONFIG)
|
96
|
+
config = config_file.evaluate
|
97
|
+
config.services.has_key?('test').should == true
|
98
|
+
config.port.should == 9292
|
99
|
+
config.data_directory.should == "/etc/litmus"
|
100
|
+
end
|
101
|
+
|
102
|
+
it "defaults configuration options in mixed dir" do
|
103
|
+
config_file = LitmusPaper::ConfigurationFile.new(TEST_MIXED_D_CONFIG)
|
104
|
+
config = config_file.evaluate
|
105
|
+
config.services.has_key?('test').should == true
|
106
|
+
config.port.should == 9292
|
107
|
+
config.data_directory.should == "/etc/litmus"
|
108
|
+
end
|
50
109
|
end
|
51
110
|
end
|
data/spec/spec_helper.rb
CHANGED
@@ -8,9 +8,12 @@ require 'tempfile'
|
|
8
8
|
|
9
9
|
TEST_CONFIG_DIR = "/tmp/litmus_paper"
|
10
10
|
TEST_CONFIG = File.expand_path('support/test.config', File.dirname(__FILE__))
|
11
|
+
TEST_YAML_CONFIG = File.expand_path('support/test.config.yaml', File.dirname(__FILE__))
|
11
12
|
TEST_RELOAD_CONFIG = File.expand_path('support/test.reload.config', File.dirname(__FILE__))
|
12
13
|
TEST_UNICORN_CONFIG = File.expand_path('support/test.unicorn.config', File.dirname(__FILE__))
|
13
14
|
TEST_D_CONFIG = File.expand_path('support/test.d.config', File.dirname(__FILE__))
|
15
|
+
TEST_YAML_D_CONFIG = File.expand_path('support/test.d.config.yaml', File.dirname(__FILE__))
|
16
|
+
TEST_MIXED_D_CONFIG = File.expand_path('support/test.d.config.mixed', File.dirname(__FILE__))
|
14
17
|
TEST_CA_CERT = File.expand_path('ssl/server.crt', File.dirname(__FILE__))
|
15
18
|
|
16
19
|
Dir.glob("#{File.expand_path('support', File.dirname(__FILE__))}/**/*.rb").each { |f| require f }
|
Binary file
|
@@ -0,0 +1,20 @@
|
|
1
|
+
# vim: set ft=ruby
|
2
|
+
port: 9293
|
3
|
+
data_directory: "/tmp/litmus_paper"
|
4
|
+
cache_location: "/tmp/litmus_paper_cache"
|
5
|
+
cache_ttl: -1
|
6
|
+
services:
|
7
|
+
test:
|
8
|
+
dependencies:
|
9
|
+
- type: http
|
10
|
+
uri: 'http://localhost/heartbeat'
|
11
|
+
checks:
|
12
|
+
- type: cpu_load
|
13
|
+
weight: 50
|
14
|
+
passing_test:
|
15
|
+
dependencies:
|
16
|
+
- type: script
|
17
|
+
command: '/bin/true'
|
18
|
+
checks:
|
19
|
+
- type: cpu_load
|
20
|
+
weight: 50
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: litmus_paper
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.2.0
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2018-
|
12
|
+
date: 2018-04-24 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: sinatra
|
@@ -153,6 +153,7 @@ files:
|
|
153
153
|
- .rake_commit
|
154
154
|
- .ruby-version
|
155
155
|
- .travis.yml
|
156
|
+
- Dockerfile
|
156
157
|
- Gemfile
|
157
158
|
- LICENSE
|
158
159
|
- README.md
|
@@ -161,6 +162,8 @@ files:
|
|
161
162
|
- bin/litmus-agent-check
|
162
163
|
- bin/litmusctl
|
163
164
|
- config.ru
|
165
|
+
- docker/litmus.conf
|
166
|
+
- docker/litmus_unicorn.rb
|
164
167
|
- lib/litmus_paper.rb
|
165
168
|
- lib/litmus_paper/agent_check_handler.rb
|
166
169
|
- lib/litmus_paper/agent_check_server.rb
|
@@ -190,6 +193,7 @@ files:
|
|
190
193
|
- lib/litmus_paper/service.rb
|
191
194
|
- lib/litmus_paper/status_file.rb
|
192
195
|
- lib/litmus_paper/terminal_output.rb
|
196
|
+
- lib/litmus_paper/util.rb
|
193
197
|
- lib/litmus_paper/version.rb
|
194
198
|
- litmus-agent-check.init.sh
|
195
199
|
- litmus_paper.gemspec
|
@@ -219,6 +223,11 @@ files:
|
|
219
223
|
- spec/ssl/server.crt
|
220
224
|
- spec/ssl/server.key
|
221
225
|
- spec/support/always_available_dependency.rb
|
226
|
+
- spec/support/config.d.mixed/.passing_test.config.swp
|
227
|
+
- spec/support/config.d.mixed/passing_test.config
|
228
|
+
- spec/support/config.d.mixed/test.config.yaml
|
229
|
+
- spec/support/config.d.yaml/passing_test.config.yaml
|
230
|
+
- spec/support/config.d.yaml/test.config.yaml
|
222
231
|
- spec/support/config.d/passing_test.config
|
223
232
|
- spec/support/config.d/test.config
|
224
233
|
- spec/support/haproxy_test_socket
|
@@ -227,7 +236,10 @@ files:
|
|
227
236
|
- spec/support/never_available_dependency.rb
|
228
237
|
- spec/support/stdout_logger.rb
|
229
238
|
- spec/support/test.config
|
239
|
+
- spec/support/test.config.yaml
|
230
240
|
- spec/support/test.d.config
|
241
|
+
- spec/support/test.d.config.mixed
|
242
|
+
- spec/support/test.d.config.yaml
|
231
243
|
- spec/support/test.reload.config
|
232
244
|
- spec/support/test.unicorn.config
|
233
245
|
homepage: https://github.com/braintree/litmus_paper
|
@@ -250,7 +262,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
250
262
|
version: '0'
|
251
263
|
requirements: []
|
252
264
|
rubyforge_project:
|
253
|
-
rubygems_version: 1.8.
|
265
|
+
rubygems_version: 1.8.23
|
254
266
|
signing_key:
|
255
267
|
specification_version: 3
|
256
268
|
summary: Backend health tester for HA Services, partner project of big_brother
|
@@ -280,6 +292,11 @@ test_files:
|
|
280
292
|
- spec/ssl/server.crt
|
281
293
|
- spec/ssl/server.key
|
282
294
|
- spec/support/always_available_dependency.rb
|
295
|
+
- spec/support/config.d.mixed/.passing_test.config.swp
|
296
|
+
- spec/support/config.d.mixed/passing_test.config
|
297
|
+
- spec/support/config.d.mixed/test.config.yaml
|
298
|
+
- spec/support/config.d.yaml/passing_test.config.yaml
|
299
|
+
- spec/support/config.d.yaml/test.config.yaml
|
283
300
|
- spec/support/config.d/passing_test.config
|
284
301
|
- spec/support/config.d/test.config
|
285
302
|
- spec/support/haproxy_test_socket
|
@@ -288,6 +305,9 @@ test_files:
|
|
288
305
|
- spec/support/never_available_dependency.rb
|
289
306
|
- spec/support/stdout_logger.rb
|
290
307
|
- spec/support/test.config
|
308
|
+
- spec/support/test.config.yaml
|
291
309
|
- spec/support/test.d.config
|
310
|
+
- spec/support/test.d.config.mixed
|
311
|
+
- spec/support/test.d.config.yaml
|
292
312
|
- spec/support/test.reload.config
|
293
313
|
- spec/support/test.unicorn.config
|