service_skeleton 0.0.0.44.g75d07d7 → 0.0.0.48.g4a40599
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +259 -144
- data/lib/service_skeleton.rb +20 -186
- data/lib/service_skeleton/config.rb +40 -22
- data/lib/service_skeleton/config_class.rb +16 -0
- data/lib/service_skeleton/config_variable.rb +1 -1
- data/lib/service_skeleton/config_variable/yaml_file.rb +42 -0
- data/lib/service_skeleton/config_variables.rb +28 -23
- data/lib/service_skeleton/error.rb +3 -3
- data/lib/service_skeleton/generator.rb +165 -0
- data/lib/service_skeleton/logging_helpers.rb +3 -3
- data/lib/service_skeleton/metric_method_name.rb +9 -0
- data/lib/service_skeleton/metrics_methods.rb +26 -13
- data/lib/service_skeleton/runner.rb +46 -0
- data/lib/service_skeleton/service_name.rb +20 -0
- data/lib/service_skeleton/signal_manager.rb +202 -0
- data/lib/service_skeleton/signals_methods.rb +15 -0
- data/lib/service_skeleton/ultravisor_children.rb +17 -0
- data/lib/service_skeleton/ultravisor_loggerstash.rb +11 -0
- data/service_skeleton.gemspec +6 -7
- metadata +57 -15
- data/lib/service_skeleton/background_worker.rb +0 -96
- data/lib/service_skeleton/signal_handler.rb +0 -197
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 79f6fcf6e15dbc20ca839d72de774a57a45322af64bdf218198e130239ea03f0
|
4
|
+
data.tar.gz: 3ffcdfb56114cf8cf3ba52eed3e9ceffdd8798e3c35f98d34d9e6ed73f4a8f9c
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: da29c0aa008026bc92249ed58105037a98e71291335cfa21292847fdeda0dd4c96644ba30ec613df630a0014abf15997badd61c064c5db1e09583dcad4ad9c87
|
7
|
+
data.tar.gz: 94afc66af6e14f50e0485335dc7f245947df5df9aed970e50fd94da9d46be803bc5479904f578ba6e8cc3e92bf4dee5e95153956605e9a2618f7a97eb2d0f4a7
|
data/README.md
CHANGED
@@ -6,10 +6,11 @@ other parts of a larger system. It provides:
|
|
6
6
|
* Prometheus-based metrics registry;
|
7
7
|
* Signal handling;
|
8
8
|
* Configuration extraction from the process environment;
|
9
|
+
* Supervision and automated restarting of your service code;
|
9
10
|
* and more.
|
10
11
|
|
11
12
|
The general philosophy of `ServiceSkeleton` is to provide features which have
|
12
|
-
been found to be almost universally necessary
|
13
|
+
been found to be almost universally necessary in modern deployment
|
13
14
|
configurations, to prefer convenience over configuration, and to always be
|
14
15
|
secure by default.
|
15
16
|
|
@@ -40,7 +41,9 @@ like this:
|
|
40
41
|
|
41
42
|
require "service_skeleton"
|
42
43
|
|
43
|
-
class HelloService
|
44
|
+
class HelloService
|
45
|
+
include ServiceSkeleton
|
46
|
+
|
44
47
|
def run
|
45
48
|
loop do
|
46
49
|
puts "Hello, Service!"
|
@@ -49,71 +52,132 @@ like this:
|
|
49
52
|
end
|
50
53
|
end
|
51
54
|
|
52
|
-
|
55
|
+
ServiceSkeleton::Runner.new(HelloService, ENV).run if __FILE__ == $0
|
53
56
|
|
54
57
|
First, we require the `"service_skeleton"` library, which is a pre-requisite
|
55
|
-
for the `ServiceSkeleton`
|
56
|
-
|
57
|
-
|
58
|
-
|
58
|
+
for the `ServiceSkeleton` module to be available. Your code is placed in
|
59
|
+
its own class in the `run` method, where you put your service's logic. The
|
60
|
+
`ServiceSkeleton` module provides helper methods and initializers, which will
|
61
|
+
be introduced as we go along.
|
62
|
+
|
63
|
+
The `run` method is typically an infinite loop, because services are long-running,
|
64
|
+
persistent processes. If you `run` method exits, or raises an unhandled exception,
|
65
|
+
the supervisor will restart it.
|
59
66
|
|
60
|
-
Finally, the last line
|
61
|
-
|
62
|
-
|
63
|
-
|
67
|
+
Finally, the last line uses the `ServiceSkeleton::Runner` class to actually run
|
68
|
+
your service. This ensures that all of the scaffolding services, like the
|
69
|
+
signal handler and metrics server, are up and running alongside your service
|
70
|
+
code.
|
64
71
|
|
65
72
|
|
66
73
|
## The `#run` loop
|
67
74
|
|
68
75
|
The core of a service is usually some sort of infinite loop, which waits for a
|
69
|
-
reason to do something, and then does it. A lot of services are
|
70
|
-
|
71
|
-
|
72
|
-
a periodic timer, a filesystem event, or anything else that takes
|
76
|
+
reason to do something, and then does it. A lot of services are network
|
77
|
+
accessible, and so the "reason to do something" is "because someone made a
|
78
|
+
connection to a port on which I'm listening". Other times it could be because
|
79
|
+
of a periodic timer firing, a filesystem event, or anything else that takes
|
80
|
+
your fancy.
|
73
81
|
|
74
82
|
Whatever it is, `ServiceSkeleton` doesn't discriminate. All you have to do is
|
75
|
-
write it in your
|
76
|
-
of the rest.
|
77
|
-
|
78
|
-
If your `#run` method exits, the service will terminate.
|
83
|
+
write it in your service class' `#run` method, and we'll take care of the rest.
|
79
84
|
|
80
85
|
|
81
86
|
### STAHP!
|
82
87
|
|
83
|
-
|
84
|
-
|
85
|
-
|
86
|
-
|
87
|
-
|
88
|
-
|
89
|
-
|
90
|
-
|
88
|
+
When your service needs to be stopped for one reason or another, `ServiceSkeleton`
|
89
|
+
needs to be able to tell your code to stop. By default, the thread that is
|
90
|
+
running your service will just be killed, which might be fine if your service
|
91
|
+
holds no state or persistent resources, but often that isn't the case.
|
92
|
+
|
93
|
+
If your code needs to stop gracefully, you should define a (thread-safe)
|
94
|
+
instance method, `#shutdown`, which does whatever is required to signal to
|
95
|
+
your service worker code that it is time to return from the `#run` method.
|
96
|
+
What that does, exactly, is up to you.
|
97
|
+
|
98
|
+
```
|
99
|
+
class CustomShutdownService
|
100
|
+
include ServiceSkeleton
|
101
|
+
|
102
|
+
def run
|
103
|
+
until @shutdown do
|
104
|
+
puts "Hello, Service!"
|
105
|
+
sleep 1
|
106
|
+
end
|
107
|
+
|
108
|
+
puts "Shutting down gracefully..."
|
109
|
+
end
|
110
|
+
|
111
|
+
def shutdown
|
112
|
+
@shutdown = true
|
113
|
+
end
|
114
|
+
end
|
115
|
+
```
|
116
|
+
|
117
|
+
To avoid the unpleasantness of a hung service, there is a limit on the amount
|
118
|
+
of time that `ServiceSkeleton` will wait for your service code to terminate.
|
119
|
+
This is, by default, five seconds, but you can modify that by defining a
|
120
|
+
`#shutdown_timeout` method, which returns a `Numeric`, to specify the number of
|
121
|
+
seconds that `ServiceSkeleton` should wait for termination.
|
122
|
+
|
123
|
+
```
|
124
|
+
class SlowShutdownService
|
125
|
+
include ServiceSkeleton
|
126
|
+
|
127
|
+
def run
|
128
|
+
until @shutdown do
|
129
|
+
puts "Hello, Service!"
|
130
|
+
sleep 60
|
131
|
+
end
|
132
|
+
end
|
133
|
+
|
134
|
+
def shutdown
|
135
|
+
@shutdown = true
|
136
|
+
end
|
137
|
+
|
138
|
+
def shutdown_timeout
|
139
|
+
# We need an unusually long shutdown timeout for this service because
|
140
|
+
# the shutdown flag is only checked once a minute, which is much longer
|
141
|
+
# than the default shutdown period.
|
142
|
+
90
|
143
|
+
end
|
144
|
+
end
|
145
|
+
```
|
146
|
+
|
147
|
+
If your service code does not terminate before the timeout, the thread will be,
|
148
|
+
once again, unceremoniously killed.
|
91
149
|
|
92
150
|
|
93
151
|
### Exceptional Behaviour
|
94
152
|
|
95
153
|
If your `#run` loop happens to raise an unhandled exception, it will be caught,
|
96
|
-
logged, and
|
97
|
-
|
98
|
-
|
99
|
-
|
100
|
-
|
154
|
+
logged, and your service will be restarted. This involves instantiating a new
|
155
|
+
instance of your service class, and calling `#run` again.
|
156
|
+
|
157
|
+
In the event that the problem that caused the exception isn't transient, and
|
158
|
+
your service code keeps exiting (either by raising an exception, or the `#run`
|
159
|
+
method returning), the supervisor will, after a couple of retries, terminate
|
160
|
+
the whole process.
|
161
|
+
|
162
|
+
This allows for a *really* clean slate restart, by starting a whole new
|
163
|
+
process. Your process manager should handle automatically restarting the
|
164
|
+
process in a sensible manner.
|
101
165
|
|
102
166
|
|
103
167
|
## The Service Name
|
104
168
|
|
105
169
|
Several aspects of a `ServiceSkeleton` service, including environment variable
|
106
|
-
and metric names,
|
107
|
-
service name is derived from the name of the class that
|
108
|
-
`ServiceSkeleton`, by converting the `CamelCase` class name into a
|
109
|
-
service name. If the class name is in a namespace, that is
|
110
|
-
the `::` turned into `_`.
|
170
|
+
and metric names, can incorporate the service's name, usually as a prefix. The
|
171
|
+
service name is derived from the name of the class that you provide to
|
172
|
+
`ServiceSkeleton::Runner.new`, by converting the `CamelCase` class name into a
|
173
|
+
`snake_case` service name. If the class name is in a namespace, that is
|
174
|
+
included also, with the `::` turned into `_`.
|
111
175
|
|
112
176
|
|
113
177
|
## Configuration
|
114
178
|
|
115
179
|
Almost every service has a need for some amount of configuration. In keeping
|
116
|
-
with the general principles of the [12 factor app](https://12factor.net),
|
180
|
+
with the general principles of the [12 factor app](https://12factor.net),
|
117
181
|
`ServiceSkeleton` takes configuration from the environment. However, we try to
|
118
182
|
minimise the amount of manual effort you need to expend to make that happen,
|
119
183
|
and provide configuration management as a first-class operation.
|
@@ -121,15 +185,17 @@ and provide configuration management as a first-class operation.
|
|
121
185
|
|
122
186
|
### Basic Configuration
|
123
187
|
|
124
|
-
|
125
|
-
returns an instance of
|
188
|
+
The `ServiceSkeleton` module defines an instance method, called `#config`, which
|
189
|
+
returns an instance of {ServiceSkeleton::Config} (or some other class you
|
126
190
|
specify; more on that below), which provides access to the environment that was
|
127
191
|
passed into the service object at instantiation time (ie the `ENV` in
|
128
|
-
`
|
192
|
+
`ServiceSkeleton.new(MyService, ENV)`) via the `#[]` method. So, in a very simple
|
129
193
|
application where you want to get the name of the thing to say hello to, it
|
130
194
|
might look like this:
|
131
195
|
|
132
|
-
class GenericHelloService
|
196
|
+
class GenericHelloService
|
197
|
+
include ServiceSkeleton
|
198
|
+
|
133
199
|
def run
|
134
200
|
loop do
|
135
201
|
puts "Hello, #{config["RECIPIENT"]}!"
|
@@ -138,17 +204,21 @@ might look like this:
|
|
138
204
|
end
|
139
205
|
end
|
140
206
|
|
207
|
+
ServiceSkeleton::Runner.new(GenericHelloService, "RECIPIENT" => "Bob").start
|
141
208
|
|
142
|
-
|
209
|
+
This will print "Hello, Bob!" every second.
|
210
|
+
|
211
|
+
|
212
|
+
### Declaring Configuration Variables
|
143
213
|
|
144
214
|
If your application has very minimal needs, it's possible that directly
|
145
|
-
accessing the environment will be sufficient. However, you can (and
|
215
|
+
accessing the environment will be sufficient. However, you can (and usually
|
146
216
|
should) declare your configuration variables in your service class, because
|
147
|
-
that way you can get coerced values (numbers
|
148
|
-
|
149
|
-
ten", or "the string must match this regex"), default
|
150
|
-
reporting. You also get direct access to the configuration
|
151
|
-
call on the `config` object.
|
217
|
+
that way you can get coerced values (numbers, booleans, lists, etc, rather than
|
218
|
+
just plain strings), range and format checking (say "the number must be an
|
219
|
+
integer between one and ten", or "the string must match this regex"), default
|
220
|
+
values, and error reporting. You also get direct access to the configuration
|
221
|
+
value as a method call on the `config` object.
|
152
222
|
|
153
223
|
To declare configuration variables, simply call one of the "config declaration
|
154
224
|
methods" (as listed in the `ServiceSkeleton::ConfigVariables` module) in your
|
@@ -156,20 +226,24 @@ class definition, and pass it an environment variable name (as a string or
|
|
156
226
|
symbol) and any relevant configuration parameters (like a default, or a
|
157
227
|
validity range, or whatever).
|
158
228
|
|
159
|
-
When your service
|
160
|
-
the configuration
|
161
|
-
|
162
|
-
|
229
|
+
When you run your service (via {ServiceSkeleton::Runner#new}), the environment
|
230
|
+
you pass in will be examined and the configuration initialised. If any values
|
231
|
+
are invalid (number out of range, etc) or missing (for any configuration
|
232
|
+
variable that doesn't have a default), then a
|
233
|
+
{ServiceSkeleton::InvalidEnvironmentError} exception will be raised and the
|
234
|
+
service will not start.
|
163
235
|
|
164
236
|
During your service's execution, any time you need to access a configuration
|
165
237
|
value, just call the matching method name (the all-lowercase version of the
|
166
|
-
environment variable name) on `config`, and
|
167
|
-
lap.
|
238
|
+
environment variable name, without the service name prefix) on `config`, and
|
239
|
+
you'll get the value in your lap.
|
168
240
|
|
169
241
|
Here's a version of our generic greeter service, using declared configuration
|
170
242
|
variables:
|
171
243
|
|
172
|
-
class GenericHelloService
|
244
|
+
class GenericHelloService
|
245
|
+
include ServiceSkeleton
|
246
|
+
|
173
247
|
string :RECIPIENT, matches: /\A\w+\z/
|
174
248
|
|
175
249
|
def run
|
@@ -180,27 +254,51 @@ variables:
|
|
180
254
|
end
|
181
255
|
end
|
182
256
|
|
257
|
+
begin
|
258
|
+
ServiceSkeleton::Runner.new(GenericHelloService, ENV).run
|
259
|
+
rescue ServiceSkeleton::InvalidEnvironmentError => ex
|
260
|
+
$stderr.puts "Configuration error found: #{ex.message}"
|
261
|
+
exit 1
|
262
|
+
end
|
263
|
+
|
264
|
+
This service, if run without a `RECIPIENT` environment variable being available,
|
265
|
+
will exit with an error. If that isn't what you want, you can declare a
|
266
|
+
default for a config variable, like so:
|
267
|
+
|
268
|
+
class GenericHelloService
|
269
|
+
include ServiceSkeleton
|
270
|
+
|
271
|
+
string :RECIPIENT, matches: /\a\w+\z/, default: "Anonymous Coward"
|
272
|
+
|
273
|
+
# ...
|
274
|
+
|
275
|
+
*This* version will print "Hello, Anonymous Coward!" if no `RECIPIENT`
|
276
|
+
environment variable is available.
|
277
|
+
|
183
278
|
|
184
279
|
### Environment Variable Prefixes
|
185
280
|
|
186
281
|
It's common for all (or almost all) of your environment variables to have a
|
187
|
-
common prefix, usually named for your service,
|
188
|
-
|
189
|
-
don't want to have to use that prefix when accessing your
|
282
|
+
common prefix, usually named for your service, to distinguish your service's
|
283
|
+
configuration from any other environment variables lying around. However, to
|
284
|
+
save on typing, you don't want to have to use that prefix when accessing your
|
285
|
+
`config` methods.
|
190
286
|
|
191
287
|
Enter: the service name prefix. Any of your environment variables whose name
|
192
288
|
starts with [your service's name](#the-service-name) (matched
|
193
289
|
case-insensitively) followed by an underscore will have that part of the
|
194
290
|
environment variable name removed to determine the method name on `config`.
|
195
291
|
The *original* environment variable name is still matched to a variable
|
196
|
-
declaration, so, you need to declare the variable *with* the prefix,
|
197
|
-
method name won't have
|
292
|
+
declaration, so, you need to declare the variable *with* the prefix, it is only
|
293
|
+
the method name on the `config` object that won't have the prefix.
|
198
294
|
|
199
295
|
Using this environment variable prefix support, the `GenericHelloService` would
|
200
296
|
have a (case-insensitive) prefix of `generic_hello_service_`. In that case,
|
201
297
|
extending the above example a little more, you could do something like this:
|
202
298
|
|
203
|
-
class GenericHelloService
|
299
|
+
class GenericHelloService
|
300
|
+
include ServiceSkeleton
|
301
|
+
|
204
302
|
string :GENERIC_HELLO_SERVICE_RECIPIENT, matches: /\A\w+\z/
|
205
303
|
|
206
304
|
def run
|
@@ -221,12 +319,14 @@ Sometimes your service will take configuration data that really, *really*
|
|
221
319
|
shouldn't be available to subprocesses or anyone who manages to catch a
|
222
320
|
sneak-peek at your service's environment. In that case, you can declare an
|
223
321
|
environment variable as "sensitive", and after the configuration is parsed,
|
224
|
-
that environment variable will be
|
322
|
+
that environment variable will be redacted from the environment.
|
225
323
|
|
226
324
|
To declare an environment variable as "sensitive", simply pass the `sensitive`
|
227
325
|
parameter, with a trueish value, to the variable declaration in your class:
|
228
326
|
|
229
|
-
class DatabaseManager
|
327
|
+
class DatabaseManager
|
328
|
+
include ServiceSkeleton
|
329
|
+
|
230
330
|
string :DB_PASSWORD, sensitive: true
|
231
331
|
|
232
332
|
...
|
@@ -234,17 +334,18 @@ parameter, with a trueish value, to the variable declaration in your class:
|
|
234
334
|
|
235
335
|
> **NOTE**: The process environment can only be modified if you pass the real,
|
236
336
|
> honest-to-goodness `ENV` object into `MyServiceClass.new(ENV)`. If you
|
237
|
-
> provide a copy
|
238
|
-
>
|
239
|
-
>
|
240
|
-
>
|
241
|
-
>
|
242
|
-
> you
|
337
|
+
> provide a copy of `ENV`, or some other hash entirely, that'll work if you
|
338
|
+
> don't have any sensitive variables declared, but the moment you declare a
|
339
|
+
> sensitive variable, passing in any hash other than `ENV` will cause the
|
340
|
+
> service to log an error and refuse to start. This avoids the problems of
|
341
|
+
> accidentally modifying global state if that would be potentially bad (we
|
342
|
+
> assume you copied `ENV` for a reason) without leaving a gaping security hole
|
343
|
+
> (sensitive data blindly passed into subprocesses that you didn't expect).
|
243
344
|
|
244
345
|
|
245
|
-
### Custom Configuration Class
|
346
|
+
### Using a Custom Configuration Class
|
246
347
|
|
247
|
-
Whilst we hope that
|
348
|
+
Whilst we hope that {ServiceSkeleton::Config} will be useful in most
|
248
349
|
situations, there are undoubtedly cases where the config management we provide
|
249
350
|
won't be enough. In that case, you are encouraged to subclass
|
250
351
|
`ServiceSkeleton::Config` and augment the standard interface with your own
|
@@ -260,7 +361,9 @@ class method in your service's class definition, like this:
|
|
260
361
|
end
|
261
362
|
end
|
262
363
|
|
263
|
-
class MyService
|
364
|
+
class MyService
|
365
|
+
include ServiceSkeleton
|
366
|
+
|
264
367
|
config_class MyServiceConfig
|
265
368
|
|
266
369
|
def run
|
@@ -281,11 +384,11 @@ for you to use.
|
|
281
384
|
|
282
385
|
### What You Get
|
283
386
|
|
284
|
-
Every instance of your service class
|
285
|
-
|
286
|
-
|
287
|
-
|
288
|
-
|
387
|
+
Every instance of your service class has a method named, uncreatively,
|
388
|
+
`logger`. It is a (more-or-less) straight-up instance of the Ruby stdlib
|
389
|
+
`Logger`, on which you can call all the usual methods (`#debug`, `#info`,
|
390
|
+
`#warn`, `#error`, etc). By default, it sends all log messages to standard
|
391
|
+
error.
|
289
392
|
|
290
393
|
When calling the logger, you really, *really* want to use the
|
291
394
|
"progname+message-in-a-block" style of recording log messages, which looks like
|
@@ -301,22 +404,22 @@ wish to actively debug, based on log messages that are tagged with a specified
|
|
301
404
|
progname. No more grovelling through thousands of lines of debug logging to
|
302
405
|
find the One Useful Message.
|
303
406
|
|
304
|
-
|
407
|
+
You also get, as part of this package, built-in dynamic log level adjustment;
|
305
408
|
using Unix signals or the admin HTTP interface (if enabled), you can tell the
|
306
409
|
logger to increase or decrease logging verbosity *without interrupting
|
307
410
|
service*. We are truly living in the future.
|
308
411
|
|
309
|
-
Finally, if you're a devotee of the ELK stack,
|
310
|
-
entries straight into logstash, rather than you having to do it in
|
311
|
-
more roundabout fashion.
|
412
|
+
Finally, if you're a devotee of the ELK stack, the logger can automagically
|
413
|
+
send log entries straight into logstash, rather than you having to do it in
|
414
|
+
some more roundabout fashion.
|
312
415
|
|
313
416
|
|
314
417
|
### Logging Configuration
|
315
418
|
|
316
419
|
The logger automatically sets its configuration from, you guessed it, the
|
317
420
|
environment. The following environment variables are recognised by the logger.
|
318
|
-
All are all-uppercase, and the `<SERVICENAME>_`
|
319
|
-
[service name](#the-service-name).
|
421
|
+
All environment variable names are all-uppercase, and the `<SERVICENAME>_`
|
422
|
+
portion is the all-uppercase [service name](#the-service-name).
|
320
423
|
|
321
424
|
* **`<SERVICENAME>_LOG_LEVEL`** (default: `"INFO"`) -- the minimum severity of
|
322
425
|
log messages which will be emitted by the logger.
|
@@ -327,21 +430,21 @@ All are all-uppercase, and the `
|
|
327
430
|
|
328
431
|
If you wish to change the severity level for a single progname, you can
|
329
432
|
override the default log level for messages with a specific progname, by
|
330
|
-
specifying one or more "progname
|
331
|
-
severity looks like this:
|
433
|
+
specifying one or more "progname/severity" pairs, separated by commas. A
|
434
|
+
progname/severity pair looks like this:
|
332
435
|
|
333
436
|
<progname>=<severity>
|
334
437
|
|
335
438
|
To make things even more fun, if `<progname>` looks like a regular expression
|
336
439
|
(starts with `/` or `%r{`, and ends with `/` or `}` plus optional flag
|
337
440
|
characters), then all log messages with prognames *matching* the specified
|
338
|
-
|
441
|
+
regex will have that severity applied. First match wins. The default is
|
339
442
|
still specified as a bare severity name, and the default can only be set
|
340
443
|
once.
|
341
444
|
|
342
|
-
That's a lot to take in, so here's an example which sets the default to
|
343
|
-
debugs the `buggy` progname, and only emits errors for messages with
|
344
|
-
(case-insensitive) string `noisy` in their progname:
|
445
|
+
That's a lot to take in, so here's an example which sets the default to
|
446
|
+
`INFO`, debugs the `buggy` progname, and only emits errors for messages with
|
447
|
+
the (case-insensitive) string `noisy` in their progname:
|
345
448
|
|
346
449
|
INFO,buggy=DEBUG,/noisy/i=ERROR
|
347
450
|
|
@@ -367,8 +470,8 @@ All are all-uppercase, and the `
|
|
367
470
|
where log messages aren't automatically timestamped, then you can use this to
|
368
471
|
get them back.
|
369
472
|
|
370
|
-
* **`<SERVICENAME>_LOG_FILE`** (string; default: `"/dev/stderr`) -- the file
|
371
|
-
which log messages are written. The default, to send messages to standard
|
473
|
+
* **`<SERVICENAME>_LOG_FILE`** (string; default: `"/dev/stderr"`) -- the file
|
474
|
+
to which log messages are written. The default, to send messages to standard
|
372
475
|
error, is a good choice if you are using a supervisor system which captures
|
373
476
|
service output to its own logging system, however if you are stuck without
|
374
477
|
such niceties, you can specify a file on disk to log to instead.
|
@@ -403,7 +506,7 @@ All are all-uppercase, and the `
|
|
403
506
|
## Metrics
|
404
507
|
|
405
508
|
Running a service without metrics is like trying to fly a fighter jet whilst
|
406
|
-
blindfolded
|
509
|
+
blindfolded: everything seems to be going OK until you slam into the side of a
|
407
510
|
mountain you never saw coming. For that reason, `ServiceSkeleton` provides a
|
408
511
|
Prometheus-based metrics registry, a bunch of default process-level metrics, an
|
409
512
|
optional HTTP metrics server, and simple integration with [the Prometheus ruby
|
@@ -415,45 +518,43 @@ easy as possible to instrument the heck out of your service.
|
|
415
518
|
### Defining and Using Metrics
|
416
519
|
|
417
520
|
All the metrics you want to use within your service need to be registered
|
418
|
-
before use. This is
|
419
|
-
|
521
|
+
before use. This is done via class methods, similar to declaring environment
|
522
|
+
variables.
|
420
523
|
|
421
524
|
To register a metric, use one of the standard metric registration methods from
|
422
525
|
[Prometheus::Client::Registry](https://www.rubydoc.info/gems/prometheus-client/0.8.0/Prometheus/Client/Registry)
|
423
|
-
(
|
424
|
-
|
526
|
+
(`counter`, `gauge`, `histogram`, `summary`) or `metric` (equivalent
|
527
|
+
to the `register` method of `Prometheus::Client::Registry) in your class
|
528
|
+
definition to register the metric for use.
|
425
529
|
|
426
530
|
In our generic greeter service we've been using as an example so far, you might
|
427
531
|
like to define a metric to count how many greetings have been sent. You'd define
|
428
532
|
such a metric like this:
|
429
533
|
|
430
|
-
class GenericHelloService
|
534
|
+
class GenericHelloService
|
535
|
+
include ServiceSkeleton
|
536
|
+
|
431
537
|
string :GENERIC_HELLO_SERVICE_RECIPIENT, matches: /\A\w+\z/
|
432
538
|
|
433
|
-
|
434
|
-
metrics.counter(:greetings_total, "How many greetings we have sent")
|
539
|
+
counter :greetings_total, docstring: "How many greetings we have sent", labels: %i{recipient}
|
435
540
|
|
436
|
-
|
437
|
-
puts "Hello, #{config.recipient}!"
|
438
|
-
sleep 1
|
439
|
-
end
|
440
|
-
end
|
441
|
-
end
|
541
|
+
# ...
|
442
542
|
|
443
|
-
When it comes time to actually *use* the metrics you have created,
|
444
|
-
|
445
|
-
|
446
|
-
|
543
|
+
When it comes time to actually *use* the metrics you have created, you access
|
544
|
+
them as methods on the `metrics` method in your service worker instance. Thus,
|
545
|
+
to increment our greeting counter, you simply do:
|
546
|
+
|
547
|
+
class GenericHelloService
|
548
|
+
include ServiceSkeleton
|
447
549
|
|
448
|
-
class GenericHelloService < ServiceSkeleton
|
449
550
|
string :GENERIC_HELLO_SERVICE_RECIPIENT, matches: /\A\w+\z/
|
450
551
|
|
451
|
-
|
452
|
-
metrics.counter(:greetings_total, "How many greetings we have sent")
|
552
|
+
counter :greetings_total, docstring: "How many greetings we have sent", labels: %i{recipient}
|
453
553
|
|
554
|
+
def run
|
454
555
|
loop do
|
455
556
|
puts "Hello, #{config.recipient}!"
|
456
|
-
metrics.greetings_total.increment(recipient: config.recipient)
|
557
|
+
metrics.greetings_total.increment(labels: { recipient: config.recipient })
|
457
558
|
sleep 1
|
458
559
|
end
|
459
560
|
end
|
@@ -464,15 +565,17 @@ any metrics you define which have the [service name](#the-service-name) as a
|
|
464
565
|
prefix will have that prefix (and the immediately-subsequent underscore) removed
|
465
566
|
before defining the metric accessor method, which keeps typing to a minimum:
|
466
567
|
|
467
|
-
class GenericHelloService
|
568
|
+
class GenericHelloService
|
569
|
+
include ServiceSkeleton
|
570
|
+
|
468
571
|
string :GENERIC_HELLO_SERVICE_RECIPIENT, matches: /\A\w+\z/
|
469
572
|
|
470
|
-
|
471
|
-
metrics.counter(:generic_hello_service_greetings_total, "How many greetings we have sent")
|
573
|
+
counter :generic_hello_service_greetings_total, docstring: "How many greetings we have sent", labels: %i{recipient}
|
472
574
|
|
575
|
+
def run
|
473
576
|
loop do
|
474
577
|
puts "Hello, #{config.recipient}!"
|
475
|
-
metrics.greetings_total.increment(recipient: config.recipient)
|
578
|
+
metrics.greetings_total.increment(labels: { recipient: config.recipient })
|
476
579
|
sleep 1
|
477
580
|
end
|
478
581
|
end
|
@@ -501,7 +604,7 @@ all-uppercase, and the `
|
|
501
604
|
of [the service name](#the-service-name).
|
502
605
|
|
503
606
|
* **`<SERVICENAME>_METRICS_PORT`** (integer; range 1..65535; default: `""`) --
|
504
|
-
if set to
|
607
|
+
if set to an integer which is a valid port number (`1` to `65535`,
|
505
608
|
inclusive), an HTTP server will be started which will respond to a request to
|
506
609
|
`/metrics` with a Prometheus-compatible dump of time series data.
|
507
610
|
|
@@ -519,13 +622,15 @@ behaviours for common signals.
|
|
519
622
|
|
520
623
|
### Default Signals
|
521
624
|
|
522
|
-
When the `#run` method on
|
523
|
-
signals will be hooked
|
625
|
+
When the `#run` method on a `ServiceSkeleton::Runner` instance is called, the
|
626
|
+
following signals will be hooked, and will perform the described action when
|
627
|
+
that signal is received:
|
524
628
|
|
525
629
|
* **`SIGUSR1`** -- increase the default minimum severity for messages which
|
526
|
-
will be emitted by the logger
|
527
|
-
|
528
|
-
|
630
|
+
will be emitted by the logger (`FATAL` -> `ERROR` -> `WARN` -> `INFO` ->
|
631
|
+
`DEBUG`). The default severity only applies to log messages whose progname
|
632
|
+
does not match a "progname/severity" pair (see [Logging
|
633
|
+
Configuration](#logging-configuration)).
|
529
634
|
|
530
635
|
* **`SIGUSR2`** -- decrease the default minimum severity for messages which
|
531
636
|
will be emitted by the logger.
|
@@ -543,38 +648,43 @@ signals will be hooked with the following behaviour:
|
|
543
648
|
|
544
649
|
* **`SIGINT`** / **`SIGTERM`** -- ask the service to gracefully stop running.
|
545
650
|
It will call your service's `#shutdown` method to ask it to stop what it's
|
546
|
-
doing and exit. If the signal is sent
|
547
|
-
summarily terminated
|
548
|
-
|
549
|
-
`SIGKILL` is what you
|
651
|
+
doing and exit. If the signal is sent a second time, the service will be
|
652
|
+
summarily terminated as soon as practical, without being given the
|
653
|
+
opportunity to gracefully release resources. As usual, if a service process
|
654
|
+
needs to be whacked completely and utterly *right now*, `SIGKILL` is what you
|
655
|
+
want to use.
|
550
656
|
|
551
657
|
|
552
658
|
### Hooking Signals
|
553
659
|
|
554
660
|
In addition to the above default signal dispositions, you can also hook signals
|
555
661
|
yourself for whatever purpose you desire. This is typically done in your
|
556
|
-
`#run` method, before entering the
|
662
|
+
`#run` method, before entering the main service loop.
|
557
663
|
|
558
664
|
To hook a signal, just call `hook_signal` with a signal specification and a
|
559
|
-
block of code to execute when the signal fires
|
560
|
-
signal more than once, because the signal handlers that
|
561
|
-
chain to other signal handlers. As an example, if you
|
562
|
-
every time the `SIGCONT` signal is received, you'd do
|
665
|
+
block of code to execute when the signal fires in your class definition. You
|
666
|
+
can even hook the same signal more than once, because the signal handlers that
|
667
|
+
`ServiceSkeleton` uses chain to other signal handlers. As an example, if you
|
668
|
+
want to print "oof!" every time the `SIGCONT` signal is received, you'd do
|
669
|
+
something like this:
|
563
670
|
|
564
|
-
class MyService
|
565
|
-
|
566
|
-
|
671
|
+
class MyService
|
672
|
+
include ServiceSkeleton
|
673
|
+
|
674
|
+
hook_signal("CONT") { puts "oof!" }
|
567
675
|
|
676
|
+
def run
|
568
677
|
loop { sleep }
|
569
678
|
end
|
570
679
|
end
|
571
680
|
|
681
|
+
The code in the block will be executed in the context of the service worker
|
682
|
+
instance that is running at the time the signal is received. You are
|
683
|
+
responsible for ensuring that whatever your handler does is concurrency-safe.
|
684
|
+
|
572
685
|
When the service is shutdown, all signal handlers will be automatically
|
573
686
|
unhooked, which saves you having to do it yourself.
|
574
687
|
|
575
|
-
> **NOTE**: You can define a maximum of 256 signal hooks in a single service,
|
576
|
-
> including the default signal hooks.
|
577
|
-
|
578
688
|
|
579
689
|
## HTTP Admin Interface
|
580
690
|
|
@@ -586,9 +696,9 @@ different?
|
|
586
696
|
### HTTP Admin Configuration
|
587
697
|
|
588
698
|
In the spirit of "secure by default", you must explicitly enable the HTTP admin
|
589
|
-
interface, and
|
590
|
-
environment variables, where `<SERVICENAME>_` is the all-uppercase
|
591
|
-
[the service name](#the-service-name).
|
699
|
+
interface, and configure an authentication method. To do that, use the
|
700
|
+
following environment variables, where `<SERVICENAME>_` is the all-uppercase
|
701
|
+
version of [the service name](#the-service-name).
|
592
702
|
|
593
703
|
* **`<SERVICENAME>_HTTP_ADMIN_PORT`** (integer; range 1..65535; default: `""`)
|
594
704
|
-- if set to a valid port number (`1` to `65535` inclusive), the HTTP admin
|
@@ -612,7 +722,11 @@ environment variables, where `
|
|
612
722
|
interface to be enabled.
|
613
723
|
|
614
724
|
|
615
|
-
### HTTP Admin
|
725
|
+
### HTTP Admin Usage
|
726
|
+
|
727
|
+
The HTTP admin interface provides both an interactive, browser-based mode,
|
728
|
+
as well as a RESTful interface, which should, in general, provide equivalent
|
729
|
+
functionality.
|
616
730
|
|
617
731
|
* Visiting the service's `IP address:port` in a web browser will bring up an HTML
|
618
732
|
interface showing all the features that are available. Usage should
|
@@ -637,7 +751,8 @@ conduct](CODE_OF_CONDUCT.md).
|
|
637
751
|
Unless otherwise stated, everything in this repo is covered by the following
|
638
752
|
copyright notice:
|
639
753
|
|
640
|
-
Copyright (C) 2018 Civilized Discourse Construction Kit, Inc.
|
754
|
+
Copyright (C) 2018, 2019 Civilized Discourse Construction Kit, Inc.
|
755
|
+
Copyright (C) 2019, 2020 Matt Palmer
|
641
756
|
|
642
757
|
This program is free software: you can redistribute it and/or modify it
|
643
758
|
under the terms of the GNU General Public License version 3, as
|