kicks_liveness 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +21 -0
- data/LICENSE.txt +21 -0
- data/README.md +319 -0
- data/docs/DESIGN.md +325 -0
- data/docs/KUBERNETES.md +426 -0
- data/docs/LIMITATIONS.md +141 -0
- data/docs/SETUP.md +181 -0
- data/docs/VERIFYING.md +330 -0
- data/exe/kicks-liveness +3 -0
- data/lib/kicks_liveness/attempts.rb +86 -0
- data/lib/kicks_liveness/configuration.rb +83 -0
- data/lib/kicks_liveness/heartbeat.rb +178 -0
- data/lib/kicks_liveness/hooks.rb +93 -0
- data/lib/kicks_liveness/monitor.rb +178 -0
- data/lib/kicks_liveness/probe.rb +16 -0
- data/lib/kicks_liveness/railtie.rb +18 -0
- data/lib/kicks_liveness/registry.rb +102 -0
- data/lib/kicks_liveness/version.rb +4 -0
- data/lib/kicks_liveness.rb +86 -0
- metadata +72 -0
data/docs/SETUP.md
ADDED
|
@@ -0,0 +1,181 @@
|
|
|
1
|
+
# Setup
|
|
2
|
+
|
|
3
|
+
## Requirements
|
|
4
|
+
|
|
5
|
+
You need **either** `kicks` (>= 3.0) **or** `sneakers` (>= 2.11). Neither is
|
|
6
|
+
declared as a dependency of this gem: at runtime it needs only the `Sneakers`
|
|
7
|
+
namespace, and both provide it. Install exactly one — see
|
|
8
|
+
[LIMITATIONS.md](LIMITATIONS.md#do-not-install-both-kicks-and-sneakers) for why
|
|
9
|
+
having both is worse than it looks.
|
|
10
|
+
|
|
11
|
+
Those two floors are exact, not aspirational: CI runs the suite against
|
|
12
|
+
`kicks 3.0.0` and `sneakers 2.11.0` pinned, alongside the matrix that tracks the
|
|
13
|
+
newest release of each. A `~>` matrix on its own would only ever prove that the
|
|
14
|
+
latest version works.
|
|
15
|
+
|
|
16
|
+
```ruby
|
|
17
|
+
gem 'kicks_liveness'
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
If neither is present, `install!` raises a `LoadError` naming both with their
|
|
21
|
+
required versions.
|
|
22
|
+
|
|
23
|
+
## Installing the hooks
|
|
24
|
+
|
|
25
|
+
The gem works by prepending two modules — one to `Sneakers::Worker`, one to
|
|
26
|
+
`Sneakers::WorkerGroup`. `KicksLiveness.install!` does that, and it must run
|
|
27
|
+
before the runner starts. It is idempotent, so calling it twice is harmless.
|
|
28
|
+
|
|
29
|
+
### Rails
|
|
30
|
+
|
|
31
|
+
Nothing to do. A Railtie calls `install!` from `to_prepare`.
|
|
32
|
+
|
|
33
|
+
That is the one path where this gem acts on its own rather than when told, so it
|
|
34
|
+
is worth saying how far it has been checked: a production Rails application whose
|
|
35
|
+
only kicks_liveness code is the `configure` block below — no `install!` anywhere —
|
|
36
|
+
boots its workers with both hooks in place and the probe green. See
|
|
37
|
+
[VERIFYING.md](VERIFYING.md#the-scenarios).
|
|
38
|
+
|
|
39
|
+
The Railtie file is loaded only when `Rails::Railtie` is already defined at the
|
|
40
|
+
time this gem is required, which is what `Bundler.require` in
|
|
41
|
+
`config/application.rb` gives you. If your application requires this gem
|
|
42
|
+
*before* Rails itself is loaded, the Railtie never loads and the hooks are never
|
|
43
|
+
installed — call `install!` manually in that case.
|
|
44
|
+
|
|
45
|
+
### Sinatra, Hanami, Roda, or no framework at all
|
|
46
|
+
|
|
47
|
+
Call `install!` yourself, before the runner starts:
|
|
48
|
+
|
|
49
|
+
```ruby
|
|
50
|
+
require 'kicks_liveness'
|
|
51
|
+
|
|
52
|
+
KicksLiveness.install!
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
There is no Rails coupling to work around: outside Rails the Railtie file is
|
|
56
|
+
never loaded, and the gem pulls in no Rails code. The web framework is not
|
|
57
|
+
involved at all — workers run in their own process, started by
|
|
58
|
+
`rake sneakers:run` or your own runner script, and that process usually contains
|
|
59
|
+
no web framework.
|
|
60
|
+
|
|
61
|
+
### Getting the application loaded under `rake sneakers:run`
|
|
62
|
+
|
|
63
|
+
The line above is the easy half. The half that actually costs people time is
|
|
64
|
+
how the application gets into the rake process at all, because outside Rails
|
|
65
|
+
nothing loads it for you.
|
|
66
|
+
|
|
67
|
+
`sneakers/tasks` declares `task :environment` **empty**, precisely so that you
|
|
68
|
+
can fill it in. Rake accumulates blocks for the same task rather than replacing
|
|
69
|
+
them, so adding your own is the supported way:
|
|
70
|
+
|
|
71
|
+
```ruby
|
|
72
|
+
# Rakefile
|
|
73
|
+
require 'sneakers/tasks'
|
|
74
|
+
|
|
75
|
+
task(:environment) { require_relative 'app' }
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
where `app.rb` is whatever configures Sneakers, calls `install!` and defines (or
|
|
79
|
+
requires) the worker classes. In Rails this is already handled — its own
|
|
80
|
+
`:environment` task loads the application — which is why the problem only shows
|
|
81
|
+
up outside it, usually as `sneakers:run` starting with no workers at all.
|
|
82
|
+
|
|
83
|
+
This is exactly the wiring the gem's own integration fixture uses
|
|
84
|
+
(`spec/integration/fixture/Rakefile`), so it is checked against a real broker on
|
|
85
|
+
every run of [VERIFYING.md](VERIFYING.md).
|
|
86
|
+
|
|
87
|
+
**Hanami 2** differs only in what that block contains: load the app the way
|
|
88
|
+
Hanami wants (`require 'hanami/prepare'` for a prepared, not fully booted,
|
|
89
|
+
application) and call `install!` after it. There is nothing Hanami-specific in
|
|
90
|
+
the gem — as above, the framework is not in the picture, only the question of
|
|
91
|
+
which file brings your worker classes into the process.
|
|
92
|
+
|
|
93
|
+
A minimal standalone runner:
|
|
94
|
+
|
|
95
|
+
```ruby
|
|
96
|
+
require 'sneakers' # kicks installs this same file
|
|
97
|
+
require 'kicks_liveness'
|
|
98
|
+
require_relative 'workers/orders_worker'
|
|
99
|
+
|
|
100
|
+
Sneakers.configure(amqp: ENV.fetch('AMQP_URL'), workers: 2, threads: 10)
|
|
101
|
+
KicksLiveness.install!
|
|
102
|
+
|
|
103
|
+
Sneakers::Runner.new([OrdersWorker]).run
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
## Workers must be started through `Sneakers::Runner`
|
|
107
|
+
|
|
108
|
+
This is the one real constraint, and it has nothing to do with frameworks.
|
|
109
|
+
|
|
110
|
+
The monitor thread is started from the hook on
|
|
111
|
+
`Sneakers::WorkerGroup#after_fork`, and `WorkerGroup` enters the picture through
|
|
112
|
+
`Sneakers::Runner`, which builds a ServerEngine supervisor. If you boot workers
|
|
113
|
+
by hand — instantiating worker classes and calling `worker.run` in your own
|
|
114
|
+
loop, without ServerEngine — that hook never fires. The registry will fill up,
|
|
115
|
+
but nothing writes the mark files, so the probe stays red and the pod is killed.
|
|
116
|
+
|
|
117
|
+
Use `Sneakers::Runner` (that is what `rake sneakers:run` does), or do not use
|
|
118
|
+
this gem.
|
|
119
|
+
|
|
120
|
+
## Configuration
|
|
121
|
+
|
|
122
|
+
Everything has a sensible default; in most applications the block is one line.
|
|
123
|
+
|
|
124
|
+
```ruby
|
|
125
|
+
KicksLiveness.configure do |config|
|
|
126
|
+
config.enabled = ENV['RACK_ENV'] != 'test'
|
|
127
|
+
end
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
**Do not set `config.logger = Sneakers.logger`.** That is already the default,
|
|
131
|
+
and writing it out is worse than leaving it alone, because the default is
|
|
132
|
+
resolved *lazily* and the assignment is not. `Sneakers.logger` is `nil` until
|
|
133
|
+
`Sneakers.configure` runs, so whether the assignment captures a logger or a
|
|
134
|
+
`nil` depends on which file Rails loads first — and initializers load in
|
|
135
|
+
alphabetical order, which is not something you chose or want to depend on. An
|
|
136
|
+
initializer named `kicks_liveness.rb` sorts *before* `sneakers.rb`; one named
|
|
137
|
+
`worker_liveness.rb` sorts after. Same code, different outcome.
|
|
138
|
+
|
|
139
|
+
Set it only to point somewhere **other** than `Sneakers.logger`:
|
|
140
|
+
|
|
141
|
+
```ruby
|
|
142
|
+
config.logger = Rails.logger
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
| Option | Default | |
|
|
146
|
+
|---|---|---|
|
|
147
|
+
| `logger` | `Sneakers.logger`, resolved lazily | where transitions are logged |
|
|
148
|
+
| `enabled` | `true` | set to `false` in tests, so no thread is started |
|
|
149
|
+
| `tick` | `10` | seconds between checks |
|
|
150
|
+
| `startup_grace_ticks` | `6` | unhealthy ticks tolerated at startup before one ERROR |
|
|
151
|
+
|
|
152
|
+
`dir` and `max_age` are **not** here — they come from environment variables
|
|
153
|
+
only. See
|
|
154
|
+
[DESIGN.md](DESIGN.md#what-is-configurable-and-where) for why, and
|
|
155
|
+
[KUBERNETES.md](KUBERNETES.md#environment-variables) for the variables.
|
|
156
|
+
|
|
157
|
+
## Checking that the hooks are in place
|
|
158
|
+
|
|
159
|
+
In a console of the process that boots your workers:
|
|
160
|
+
|
|
161
|
+
```ruby
|
|
162
|
+
Sneakers::Worker.ancestors.include?(KicksLiveness::Hooks::Worker) # => true
|
|
163
|
+
Sneakers::WorkerGroup.ancestors.include?(KicksLiveness::Hooks::WorkerGroup) # => true
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
Both must be `true` before the runner starts. If either is `false`, either
|
|
167
|
+
`install!` has not run, or it ran before `kicks`/`sneakers` was loaded.
|
|
168
|
+
|
|
169
|
+
Once workers are running, the marks directory is the other half of the answer:
|
|
170
|
+
|
|
171
|
+
```
|
|
172
|
+
$ ls -l /opt/app/tmp/health/
|
|
173
|
+
expected
|
|
174
|
+
worker-0
|
|
175
|
+
worker-1
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
One `worker-<slot>` file per fork, plus `expected`. If `expected` is there and
|
|
179
|
+
the slot files are not, the workers have not finished subscribing. A slot that
|
|
180
|
+
keeps restarting without ever subscribing also leaves an `attempt-<slot>` file,
|
|
181
|
+
which is removed as soon as that slot becomes healthy.
|
data/docs/VERIFYING.md
ADDED
|
@@ -0,0 +1,330 @@
|
|
|
1
|
+
# Verifying the probe against a real broker
|
|
2
|
+
|
|
3
|
+
[KUBERNETES.md](KUBERNETES.md#verifying-on-a-live-pod) shows how to confirm the
|
|
4
|
+
probe on a pod you already run. This document is the other half: a disposable
|
|
5
|
+
local cluster where the **failures themselves** can be caused on purpose, so
|
|
6
|
+
that every claim the gem makes is watched rather than reasoned about.
|
|
7
|
+
|
|
8
|
+
Two of those claims cannot be checked any other way, because both are properties
|
|
9
|
+
of Bunny rather than of this gem:
|
|
10
|
+
|
|
11
|
+
- that `channel.any_consumers?` turns false when the broker takes a consumer
|
|
12
|
+
away, which is what the probe fails on;
|
|
13
|
+
- that `recovering_from_network_failure?` stays true for the whole of a broker
|
|
14
|
+
outage, which is what keeps a hiccup from restarting every replica.
|
|
15
|
+
|
|
16
|
+
And one is a property of the kubelet: that a red probe actually restarts the
|
|
17
|
+
container. The `touch -d` recipe in KUBERNETES.md deliberately repairs itself
|
|
18
|
+
within one tick, so on its own it never reaches `failureThreshold`.
|
|
19
|
+
|
|
20
|
+
The fixture lives in `spec/integration/` in the repository. It is not part of the
|
|
21
|
+
published gem, and it is not part of `rake` — nothing under it is a `_spec.rb`
|
|
22
|
+
file, so `rspec` does not collect it.
|
|
23
|
+
|
|
24
|
+
## What you need
|
|
25
|
+
|
|
26
|
+
A local single-node cluster. Docker Desktop's built-in Kubernetes is the
|
|
27
|
+
cheapest option, because an image built locally is visible to the cluster
|
|
28
|
+
immediately, with no registry and no load step — `imagePullPolicy: Never` is all
|
|
29
|
+
it takes.
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
spec/integration/verify.sh build # image, into the engine the cluster uses
|
|
33
|
+
spec/integration/verify.sh up # namespace, broker, worker
|
|
34
|
+
spec/integration/verify.sh down # deletes the namespace
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
Every `kubectl` call in that script names its context and namespace explicitly.
|
|
38
|
+
That is not politeness: a liveness experiment deletes queues and kills pods, and
|
|
39
|
+
it must not be able to reach a real cluster because someone's current-context
|
|
40
|
+
happened to point at one.
|
|
41
|
+
|
|
42
|
+
Three things that cost time if you meet them the hard way:
|
|
43
|
+
|
|
44
|
+
- **Build into the right engine.** If the machine also runs OrbStack, Colima or
|
|
45
|
+
Rancher Desktop, the default docker context is not the one Docker Desktop's
|
|
46
|
+
Kubernetes reads images from, and the pod fails with `ErrImageNeverPull`. The
|
|
47
|
+
script passes `--context desktop-linux` for this reason.
|
|
48
|
+
- **A `path:` gem is not in `GEM_HOME`.** Bundler adds it to the load path and
|
|
49
|
+
nothing else. The standard command, `bundle exec kicks-liveness`, does not care
|
|
50
|
+
— Bundler put it there and Bundler finds it. The **optimised** command,
|
|
51
|
+
plain `ruby -e` with no Bundler, raises `LoadError`. It is one instance of
|
|
52
|
+
[where your image puts its gems](KUBERNETES.md#where-your-image-puts-its-gems),
|
|
53
|
+
and worth meeting here rather than in a rollout. The fixture image therefore
|
|
54
|
+
also `gem install`s the gem it builds, so that both commands are exercised the
|
|
55
|
+
way a published gem would be.
|
|
56
|
+
- **`ruby:*-slim` has no compiler.** `bunny` pulls `sorted_set`, which pulls
|
|
57
|
+
`rbtree`, which is a native extension.
|
|
58
|
+
|
|
59
|
+
## The scenarios
|
|
60
|
+
|
|
61
|
+
Each one is an action and a single observable value. The values below are the
|
|
62
|
+
ones actually recorded on Kubernetes v1.25 with kicks 3.4.0, bunny 3.3.0 and
|
|
63
|
+
ruby 3.4, using the probe settings printed in KUBERNETES.md
|
|
64
|
+
(`tick` 10 s, `max_age` 45 s, liveness `periodSeconds` 30 × `failureThreshold` 3).
|
|
65
|
+
|
|
66
|
+
The same scenarios have since been replayed against a production Rails
|
|
67
|
+
application — five consumers, two runners, one of them an ActiveJob adapter
|
|
68
|
+
whose worker set is a callable registry rather than an array — on the same kind
|
|
69
|
+
of throwaway cluster. Every outcome matched the fixture's, which is the point of
|
|
70
|
+
keeping the fixture small: if a two-queue toy and a real application disagree,
|
|
71
|
+
the disagreement is the finding.
|
|
72
|
+
|
|
73
|
+
The **startup** budget never binds anything here, so no number below depends on
|
|
74
|
+
it: this fixture subscribes in under a second, where a real application takes
|
|
75
|
+
tens of seconds. That is precisely why the cold-start measurement in
|
|
76
|
+
[KUBERNETES.md](KUBERNETES.md#measure-the-cold-start-not-the-restart) has to be
|
|
77
|
+
made against your own application — a fixture cannot do it for you.
|
|
78
|
+
|
|
79
|
+
### 1. It comes up
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
spec/integration/verify.sh logs | grep liveness
|
|
83
|
+
spec/integration/verify.sh marks
|
|
84
|
+
spec/integration/verify.sh probe
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
```
|
|
88
|
+
INFO: [liveness] slot 0: started: dir=/opt/app/tmp/health max_age=45s tick=10s processes=1 consumers=2
|
|
89
|
+
INFO: [liveness] slot 0: waiting for 2 consumers
|
|
90
|
+
INFO: [liveness] slot 0: healthy # one tick later
|
|
91
|
+
expected worker-0 # in the marks directory
|
|
92
|
+
1 process(es) healthy # probe, exit 0
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
The hooks fired inside a real ServerEngine fork, which no double can show.
|
|
96
|
+
|
|
97
|
+
### 2. A stale mark
|
|
98
|
+
|
|
99
|
+
The recipe from KUBERNETES.md. Proves the probe's threshold and the
|
|
100
|
+
self-repair, and nothing else — the mark is fixed on the next tick, well inside
|
|
101
|
+
`failureThreshold`, so the container is never killed.
|
|
102
|
+
|
|
103
|
+
Worth running through **both** commands here, since this is the one scenario
|
|
104
|
+
where the probe is supposed to answer `1`. `spec/integration/verify.sh probe`
|
|
105
|
+
does exactly that: the standard `bundle exec kicks-liveness` and the optimised
|
|
106
|
+
`ruby -e`, which in this image are both available and must agree.
|
|
107
|
+
|
|
108
|
+
```
|
|
109
|
+
worker-0 stale 120s > 45s # exit 1
|
|
110
|
+
1 process(es) healthy # exit 0, twelve seconds later
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
### 3. The broker takes the consumer away
|
|
114
|
+
|
|
115
|
+
The one that matters. Delete a queue the worker consumes from:
|
|
116
|
+
|
|
117
|
+
```bash
|
|
118
|
+
kubectl exec deploy/rabbitmq -- rabbitmqctl delete_queue kicks_liveness.alpha
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
Observed, from the moment of deletion:
|
|
122
|
+
|
|
123
|
+
| after the deletion | what |
|
|
124
|
+
|---|---|
|
|
125
|
+
| 9 s | `ERROR: [liveness] slot 0: became unhealthy (expected 2 consumers)` |
|
|
126
|
+
| 59 s | `ERROR: [liveness] slot 0: not healthy 60s, expected 2 consumers` |
|
|
127
|
+
| 61 s | first kubelet failure: `Liveness probe failed: worker-0 stale 69s > 45s` |
|
|
128
|
+
| 91 s, 121 s | the second and third, `stale 99s` and `stale 129s` |
|
|
129
|
+
| 132 s | `Killing: Container worker failed liveness probe, will be restarted` |
|
|
130
|
+
|
|
131
|
+
The two clocks in that table are different, which is worth reading carefully
|
|
132
|
+
if you are checking the arithmetic. The left column counts from the deletion.
|
|
133
|
+
The `stale Ns` in the probe's own text is the age of the mark, and the mark
|
|
134
|
+
stopped being refreshed one tick *before* the transition was logged — the tick
|
|
135
|
+
that found the process unhealthy is also the first one that did not touch the
|
|
136
|
+
file. So the probe's number runs about eight seconds ahead of the left column,
|
|
137
|
+
and the kill lands at `max_age + periodSeconds × failureThreshold` counted
|
|
138
|
+
from the last refresh, not from the deletion.
|
|
139
|
+
|
|
140
|
+
So `any_consumers?` does turn false, the transition is reported within one tick,
|
|
141
|
+
and the pod is restarted after `max_age + periodSeconds × failureThreshold`
|
|
142
|
+
— 132 s against the 135 s the arithmetic in KUBERNETES.md predicts. The worker
|
|
143
|
+
re-declares the queue on boot, so the pod recovers by itself.
|
|
144
|
+
|
|
145
|
+
Note where the probe's own text appears: in the kubelet's `Unhealthy` event,
|
|
146
|
+
not in the pod log. An exec probe's stdout goes nowhere else, which is why
|
|
147
|
+
`kubectl describe pod` is the place to read it.
|
|
148
|
+
|
|
149
|
+
### 4. The broker dies
|
|
150
|
+
|
|
151
|
+
```bash
|
|
152
|
+
kubectl delete pod -l app=rabbitmq --grace-period=0 --force
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
The pod must **not** restart. "It survived" is not enough on its own — a broker
|
|
156
|
+
that never actually went away would look the same — so check the log for
|
|
157
|
+
Bunny's own account of the outage:
|
|
158
|
+
|
|
159
|
+
```
|
|
160
|
+
WARN: Recovering from connection.close (CONNECTION_FORCED - broker forced connection closure with reason 'shutdown')
|
|
161
|
+
WARN: Will recover from a network failure (no retry limit)...
|
|
162
|
+
WARN: Could not establish TCP connection to rabbitmq:5672 ... TCP connection failed
|
|
163
|
+
WARN: Reconnecting in 5.0 seconds
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
Against that: `restartCount` stayed 0, the probe exited 0 at every check, and
|
|
167
|
+
no `became unhealthy` was logged. Bunny's topology recovery brought both
|
|
168
|
+
consumers back on its own.
|
|
169
|
+
|
|
170
|
+
### 5. The broker stays dead
|
|
171
|
+
|
|
172
|
+
`kubectl scale deploy/rabbitmq --replicas=0`, then wait. Observed over 319 s
|
|
173
|
+
— seven times `max_age`, more than twice the 135 s steady-state budget — the
|
|
174
|
+
mark's age never exceeded 9 s and `restartCount` stayed 0. The monitor kept
|
|
175
|
+
writing, because `recovering_from_network_failure?` never stops being true:
|
|
176
|
+
Bunny has no retry limit, which its own log line above says out loud.
|
|
177
|
+
|
|
178
|
+
This is
|
|
179
|
+
[Broker unavailability is invisible by design](LIMITATIONS.md#broker-unavailability-is-invisible-by-design),
|
|
180
|
+
seen rather than deduced. It is the reason the
|
|
181
|
+
[consumer alert](KUBERNETES.md#required-companion-an-alert-on-consumers) is part
|
|
182
|
+
of the design and not a suggestion.
|
|
183
|
+
|
|
184
|
+
Scaling the broker back up: the worker resumed consuming **in the same
|
|
185
|
+
container**, no restart.
|
|
186
|
+
|
|
187
|
+
### 6. An ordinary deploy
|
|
188
|
+
|
|
189
|
+
`kubectl delete pod` on the worker, with the log followed. Not one ERROR line.
|
|
190
|
+
Bunny cancels each consumer and the process exits.
|
|
191
|
+
|
|
192
|
+
If the shutdown finishes inside a single tick — an idle thread pool cancels
|
|
193
|
+
quickly — the monitor never gets to run again, so
|
|
194
|
+
`shutting down: consumers are no longer checked` may not appear at all. The
|
|
195
|
+
absence of ERROR is the observable, not the presence of that line.
|
|
196
|
+
|
|
197
|
+
### 7. Bad broker credentials
|
|
198
|
+
|
|
199
|
+
Point `AMQP_URL` at a wrong password. The fork cannot subscribe, so the
|
|
200
|
+
supervisor respawns it every `start_worker_delay`, forever. Over 120 s:
|
|
201
|
+
|
|
202
|
+
| | lines |
|
|
203
|
+
|---|---|
|
|
204
|
+
| fork starts | 426 |
|
|
205
|
+
| `started: dir=... consumers=2` | 1 |
|
|
206
|
+
| `waiting for 2 consumers` | 1 |
|
|
207
|
+
| liveness ERROR | 2 |
|
|
208
|
+
|
|
209
|
+
```
|
|
210
|
+
INFO: [liveness] slot 0: started: dir=/opt/app/tmp/health max_age=45s tick=10s processes=1 consumers=2
|
|
211
|
+
INFO: [liveness] slot 0: waiting for 2 consumers
|
|
212
|
+
ERROR: [liveness] slot 0: not healthy 60s over 224 starts, expected 2 consumers
|
|
213
|
+
ERROR: [liveness] slot 0: not healthy 60s over 426 starts, expected 2 consumers
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
Four lines from the gem across 426 respawns: the first incarnation announces
|
|
217
|
+
itself and reports waiting, every one after that is silent, and the escalation
|
|
218
|
+
speaks once per grace window carrying the number of starts, which is the
|
|
219
|
+
diagnosis. This is the behaviour
|
|
220
|
+
[the respawn section](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn)
|
|
221
|
+
describes, against a real storm rather than a simulated one.
|
|
222
|
+
|
|
223
|
+
Worth knowing what the pod log looks like anyway: 27 000 lines in those same
|
|
224
|
+
120 s, an authentication error and a full backtrace printed by the worker gem on
|
|
225
|
+
every respawn. The gem's four lines are in there — `grep liveness` is how you
|
|
226
|
+
find them.
|
|
227
|
+
|
|
228
|
+
### 8. More than one fork
|
|
229
|
+
|
|
230
|
+
`WORKER_COUNT=2` needs no rebuild; the rake task turns it into
|
|
231
|
+
`config[:workers]`.
|
|
232
|
+
|
|
233
|
+
```
|
|
234
|
+
INFO: [liveness] slot 0: started: ... processes=2 consumers=2
|
|
235
|
+
INFO: [liveness] slot 1: started: ... processes=2 consumers=2
|
|
236
|
+
expected worker-0 worker-1 # expected contains "2"
|
|
237
|
+
2 process(es) healthy # exit 0
|
|
238
|
+
worker-1 stale 120s > 45s # exit 1 — staling either mark is enough
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
Both marks are required, not just the first.
|
|
242
|
+
|
|
243
|
+
One trap while checking this by hand: `kubectl exec deploy/worker` picks any pod
|
|
244
|
+
matching the deployment, and during a rollout that can still be the old one.
|
|
245
|
+
Read the marks on a pod named explicitly.
|
|
246
|
+
|
|
247
|
+
### 9. A rolling deploy keeps the queue covered
|
|
248
|
+
|
|
249
|
+
Scenario 6 kills a pod outright. A deploy is the more interesting case,
|
|
250
|
+
because KUBERNETES.md claims the startup probe *incidentally fixes rollouts
|
|
251
|
+
where the old pod was removed before the new one had subscribed* — and that
|
|
252
|
+
claim is about ordering, which needs two pods to be watched at once.
|
|
253
|
+
|
|
254
|
+
`kubectl rollout restart deploy/worker`, with `WORKER_COUNT=2`, so a fully
|
|
255
|
+
subscribed pod is worth two consumers per queue. Counting consumers on the
|
|
256
|
+
broker is what makes the ordering visible:
|
|
257
|
+
|
|
258
|
+
| after | consumers on the queue | pods |
|
|
259
|
+
|---|---|---|
|
|
260
|
+
| 1 s | 2 | old Ready, new Pending |
|
|
261
|
+
| 10 s | **4** | old Ready, new Running but **not** Ready |
|
|
262
|
+
| 16 s | — | the old pod cancels its consumers |
|
|
263
|
+
| 18 s | 2 | new pod only, Ready |
|
|
264
|
+
|
|
265
|
+
Coverage went 2 → 4 → 2 and never dipped. The new pod had subscribed before
|
|
266
|
+
the old one was asked to stop, and the old one logged no ERROR on its way out.
|
|
267
|
+
|
|
268
|
+
The mechanism is worth being explicit about, because it is the probe doing it:
|
|
269
|
+
a container running its startup probe is not Ready, and with one replica
|
|
270
|
+
`maxUnavailable: 25%` rounds down to zero, so Kubernetes may not remove the
|
|
271
|
+
old pod until the new one is Ready. Readiness here means *the mark is on
|
|
272
|
+
tmpfs*, which means the consumers are subscribed. A probe that answered on
|
|
273
|
+
process liveness instead would have released the old pod while the new one was
|
|
274
|
+
still booting.
|
|
275
|
+
|
|
276
|
+
### 10. The alert's metric is the one the docs name
|
|
277
|
+
|
|
278
|
+
The [consumer alert](KUBERNETES.md#required-companion-an-alert-on-consumers) is
|
|
279
|
+
part of the design, not a suggestion — so its expression deserves the same
|
|
280
|
+
treatment as everything else here. The broker in this fixture is enough to check
|
|
281
|
+
it; the plugin is not on by default:
|
|
282
|
+
|
|
283
|
+
```bash
|
|
284
|
+
kubectl exec deploy/rabbitmq -- rabbitmq-plugins enable rabbitmq_prometheus
|
|
285
|
+
```
|
|
286
|
+
|
|
287
|
+
Then read all three endpoints from the worker pod, which has a Ruby but no
|
|
288
|
+
`curl`:
|
|
289
|
+
|
|
290
|
+
```bash
|
|
291
|
+
kubectl exec deploy/worker -- ruby -rnet/http -e \
|
|
292
|
+
'puts Net::HTTP.get(URI("http://rabbitmq:15692/metrics/detailed?family=queue_consumer_count"))'
|
|
293
|
+
```
|
|
294
|
+
|
|
295
|
+
Recorded on RabbitMQ 3.13.7:
|
|
296
|
+
|
|
297
|
+
| endpoint | what comes back |
|
|
298
|
+
|---|---|
|
|
299
|
+
| `/metrics` | `rabbitmq_queue_consumers 4` — one label-less cluster total |
|
|
300
|
+
| `/metrics/detailed` | telemetry and build info only, no queue metrics |
|
|
301
|
+
| `/metrics/detailed?family=queue_consumer_count` | `rabbitmq_detailed_queue_consumers{vhost="/",queue="kicks_liveness.alpha"} 2` |
|
|
302
|
+
|
|
303
|
+
Three things follow, and all three are now in the manifest documentation: the
|
|
304
|
+
metric is a *different* name from the aggregate, the detailed endpoint needs the
|
|
305
|
+
`family` parameter, and the series carries a `vhost` label.
|
|
306
|
+
|
|
307
|
+
Then check the two halves of the alert against a throwaway queue — declare one
|
|
308
|
+
with no consumers, scrape, delete it, scrape again:
|
|
309
|
+
|
|
310
|
+
```
|
|
311
|
+
-- queue exists, zero consumers
|
|
312
|
+
rabbitmq_detailed_queue_consumers{vhost="/",queue="kicks_liveness.tmpcheck"} 0
|
|
313
|
+
-- queue deleted
|
|
314
|
+
(no such series)
|
|
315
|
+
```
|
|
316
|
+
|
|
317
|
+
Both halves matter and they are different failures. An **idle** queue reports
|
|
318
|
+
`0`, which is what `== 0` is for. A **deleted** queue reports nothing at all,
|
|
319
|
+
and neither does a stopped broker or a stopped exporter — so `== 0` cannot fire
|
|
320
|
+
on the very failure scenario 3 demonstrates, where the queue goes away and the
|
|
321
|
+
pod is restarted. That is why the documented rule set pairs it with
|
|
322
|
+
`absent_over_time`, and why an `up == 0` on the scrape job is worth having
|
|
323
|
+
beside both.
|
|
324
|
+
|
|
325
|
+
## What to do with a disagreement
|
|
326
|
+
|
|
327
|
+
Record it, then fix whichever is wrong — the code or the document. The numbers
|
|
328
|
+
above are a baseline: the arithmetic in KUBERNETES.md is meant to predict them,
|
|
329
|
+
and a scenario that lands far from its prediction means one of the two is out of
|
|
330
|
+
date.
|
data/exe/kicks-liveness
ADDED
|
@@ -0,0 +1,86 @@
|
|
|
1
|
+
require 'fileutils'
|
|
2
|
+
|
|
3
|
+
module KicksLiveness
|
|
4
|
+
# Counts starts of a supervisor slot that have not reached healthy yet.
|
|
5
|
+
#
|
|
6
|
+
# Deliberately separate from {Heartbeat}. Heartbeat is the contract shared
|
|
7
|
+
# with the probe and is loaded by it alone, so nothing the probe does not need
|
|
8
|
+
# belongs there — including the +fileutils+ this class requires freely.
|
|
9
|
+
#
|
|
10
|
+
# The count lives in the marks directory rather than in memory because a
|
|
11
|
+
# respawn storm destroys process state every few hundred milliseconds: the
|
|
12
|
+
# supervisor brings a failing fork back after +start_worker_delay+, and every
|
|
13
|
+
# incarnation of the monitor believes it is the first. That is how an outage
|
|
14
|
+
# can produce thousands of identical INFO lines and not one ERROR. The
|
|
15
|
+
# directory is the only state that survives the fork.
|
|
16
|
+
#
|
|
17
|
+
# @see file:docs/DESIGN.md#logging-events-not-the-pulse
|
|
18
|
+
# @api private
|
|
19
|
+
class Attempts
|
|
20
|
+
# @param dir [String] marks directory
|
|
21
|
+
def initialize(dir)
|
|
22
|
+
@dir = dir
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
# Records a start of this slot.
|
|
26
|
+
#
|
|
27
|
+
# @param slot [Integer] supervisor slot of this fork
|
|
28
|
+
# @param now [Time] injected in specs
|
|
29
|
+
# @return [Array(Integer, Float)] this start's number, and seconds since the
|
|
30
|
+
# first start of the current unhealthy run
|
|
31
|
+
def record!(slot, now: Time.now.utc)
|
|
32
|
+
count, first = read(slot)
|
|
33
|
+
first ||= now
|
|
34
|
+
write(slot, count + 1, first)
|
|
35
|
+
[count + 1, now - first]
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
# Resets the elapsed-time window while keeping the count, so a respawn loop
|
|
39
|
+
# reports once per window instead of once per respawn.
|
|
40
|
+
#
|
|
41
|
+
# @param slot [Integer]
|
|
42
|
+
# @param now [Time]
|
|
43
|
+
# @return [void]
|
|
44
|
+
def restart_window!(slot, now: Time.now.utc)
|
|
45
|
+
count, = read(slot)
|
|
46
|
+
write(slot, count, now)
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
# Called once the slot is healthy: the next start of it is a fresh one.
|
|
50
|
+
#
|
|
51
|
+
# @param slot [Integer]
|
|
52
|
+
# @return [void]
|
|
53
|
+
def clear!(slot)
|
|
54
|
+
File.unlink(path(slot))
|
|
55
|
+
rescue StandardError
|
|
56
|
+
nil
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
private
|
|
60
|
+
|
|
61
|
+
def path(slot)
|
|
62
|
+
File.join(@dir, "attempt-#{slot}")
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
# A missing or unreadable counter means "no previous start", never an
|
|
66
|
+
# exception: this is instrumentation, and it may not break a worker.
|
|
67
|
+
def read(slot)
|
|
68
|
+
count, epoch = File.read(path(slot)).split
|
|
69
|
+
[Integer(count), Time.at(Integer(epoch)).utc]
|
|
70
|
+
rescue StandardError
|
|
71
|
+
[0, nil]
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
# Failing to write degrades the reporting back to one line per respawn,
|
|
75
|
+
# which is noisy but harmless — so a read-only directory is swallowed here
|
|
76
|
+
# rather than escalated.
|
|
77
|
+
def write(slot, count, first)
|
|
78
|
+
FileUtils.mkdir_p(@dir)
|
|
79
|
+
tmp = "#{path(slot)}.#{Process.pid}"
|
|
80
|
+
File.write(tmp, "#{count} #{first.to_i}\n")
|
|
81
|
+
File.rename(tmp, path(slot))
|
|
82
|
+
rescue StandardError
|
|
83
|
+
nil
|
|
84
|
+
end
|
|
85
|
+
end
|
|
86
|
+
end
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
require_relative 'heartbeat'
|
|
2
|
+
|
|
3
|
+
module KicksLiveness
|
|
4
|
+
# What the application may configure, and what comes from the environment only.
|
|
5
|
+
#
|
|
6
|
+
# +dir+ and +max_age+ are deliberately read-only here: the probe runs as a
|
|
7
|
+
# separate process launched by the container runtime and cannot read the
|
|
8
|
+
# application's initializer. Were they settable in both places, a mismatch
|
|
9
|
+
# between what the worker writes and what the probe expects would pass
|
|
10
|
+
# unnoticed.
|
|
11
|
+
#
|
|
12
|
+
# @see file:docs/SETUP.md#configuration
|
|
13
|
+
# @see file:docs/DESIGN.md#what-is-configurable-and-where
|
|
14
|
+
class Configuration
|
|
15
|
+
# @return [Integer] default seconds between checks
|
|
16
|
+
DEFAULT_TICK = 10
|
|
17
|
+
# @return [Integer] unhealthy ticks tolerated at startup before one ERROR
|
|
18
|
+
DEFAULT_STARTUP_GRACE_TICKS = 6
|
|
19
|
+
|
|
20
|
+
# @return [Integer] seconds between checks
|
|
21
|
+
attr_reader :tick
|
|
22
|
+
# @return [Integer] unhealthy ticks tolerated at startup before one ERROR
|
|
23
|
+
attr_accessor :startup_grace_ticks
|
|
24
|
+
# @return [Logger, nil] explicit logger; defaults to +Sneakers.logger+
|
|
25
|
+
attr_accessor :logger
|
|
26
|
+
# @param value [Boolean] set false to start no monitor thread, e.g. in tests
|
|
27
|
+
attr_writer :enabled
|
|
28
|
+
|
|
29
|
+
def initialize
|
|
30
|
+
@tick = Heartbeat.env_int(:tick, DEFAULT_TICK)
|
|
31
|
+
@startup_grace_ticks = DEFAULT_STARTUP_GRACE_TICKS
|
|
32
|
+
@enabled = true
|
|
33
|
+
@logger = nil
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
# @return [Boolean]
|
|
37
|
+
def enabled?
|
|
38
|
+
@enabled
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
# A tick is what the monitor thread sleeps on, so a non-positive value is
|
|
42
|
+
# not a setting but a broken monitor. The environment gets a silent fallback
|
|
43
|
+
# instead (see {Heartbeat.env_int}) because a ConfigMap typo must not bring
|
|
44
|
+
# a worker down; an initializer is code, and code should say so at boot,
|
|
45
|
+
# where the developer is looking.
|
|
46
|
+
#
|
|
47
|
+
# @param seconds [Integer]
|
|
48
|
+
# @raise [ArgumentError] if not a positive number
|
|
49
|
+
# @return [Integer]
|
|
50
|
+
def tick=(seconds)
|
|
51
|
+
raise ArgumentError, "tick must be a positive number, got #{seconds.inspect}" unless positive_number?(seconds)
|
|
52
|
+
|
|
53
|
+
@tick = seconds
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
# Read-only, sourced from the environment so that it matches what the probe
|
|
57
|
+
# sees.
|
|
58
|
+
# @return [String] marks directory
|
|
59
|
+
def dir
|
|
60
|
+
Heartbeat.env_dir
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
# Read-only, sourced from the environment so that it matches what the probe
|
|
64
|
+
# sees.
|
|
65
|
+
# @return [Integer] seconds after which a mark is considered stale
|
|
66
|
+
def max_age
|
|
67
|
+
Heartbeat.env_max_age
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
# Resolved lazily: at the time this object is built, +Sneakers.logger+ may
|
|
71
|
+
# not be configured yet.
|
|
72
|
+
# @return [Logger, false, nil]
|
|
73
|
+
def resolved_logger
|
|
74
|
+
@logger || (defined?(::Sneakers) && ::Sneakers.logger)
|
|
75
|
+
end
|
|
76
|
+
|
|
77
|
+
private
|
|
78
|
+
|
|
79
|
+
def positive_number?(value)
|
|
80
|
+
value.is_a?(Numeric) && value.positive?
|
|
81
|
+
end
|
|
82
|
+
end
|
|
83
|
+
end
|