kicks_liveness 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/docs/SETUP.md ADDED
@@ -0,0 +1,181 @@
1
+ # Setup
2
+
3
+ ## Requirements
4
+
5
+ You need **either** `kicks` (>= 3.0) **or** `sneakers` (>= 2.11). Neither is
6
+ declared as a dependency of this gem: at runtime it needs only the `Sneakers`
7
+ namespace, and both provide it. Install exactly one — see
8
+ [LIMITATIONS.md](LIMITATIONS.md#do-not-install-both-kicks-and-sneakers) for why
9
+ having both is worse than it looks.
10
+
11
+ Those two floors are exact, not aspirational: CI runs the suite against
12
+ `kicks 3.0.0` and `sneakers 2.11.0` pinned, alongside the matrix that tracks the
13
+ newest release of each. A `~>` matrix on its own would only ever prove that the
14
+ latest version works.
15
+
16
+ ```ruby
17
+ gem 'kicks_liveness'
18
+ ```
19
+
20
+ If neither is present, `install!` raises a `LoadError` naming both with their
21
+ required versions.
22
+
23
+ ## Installing the hooks
24
+
25
+ The gem works by prepending two modules — one to `Sneakers::Worker`, one to
26
+ `Sneakers::WorkerGroup`. `KicksLiveness.install!` does that, and it must run
27
+ before the runner starts. It is idempotent, so calling it twice is harmless.
28
+
29
+ ### Rails
30
+
31
+ Nothing to do. A Railtie calls `install!` from `to_prepare`.
32
+
33
+ That is the one path where this gem acts on its own rather than when told, so it
34
+ is worth saying how far it has been checked: a production Rails application whose
35
+ only kicks_liveness code is the `configure` block below — no `install!` anywhere —
36
+ boots its workers with both hooks in place and the probe green. See
37
+ [VERIFYING.md](VERIFYING.md#the-scenarios).
38
+
39
+ The Railtie file is loaded only when `Rails::Railtie` is already defined at the
40
+ time this gem is required, which is what `Bundler.require` in
41
+ `config/application.rb` gives you. If your application requires this gem
42
+ *before* Rails itself is loaded, the Railtie never loads and the hooks are never
43
+ installed — call `install!` manually in that case.
44
+
45
+ ### Sinatra, Hanami, Roda, or no framework at all
46
+
47
+ Call `install!` yourself, before the runner starts:
48
+
49
+ ```ruby
50
+ require 'kicks_liveness'
51
+
52
+ KicksLiveness.install!
53
+ ```
54
+
55
+ There is no Rails coupling to work around: outside Rails the Railtie file is
56
+ never loaded, and the gem pulls in no Rails code. The web framework is not
57
+ involved at all — workers run in their own process, started by
58
+ `rake sneakers:run` or your own runner script, and that process usually contains
59
+ no web framework.
60
+
61
+ ### Getting the application loaded under `rake sneakers:run`
62
+
63
+ The line above is the easy half. The half that actually costs people time is
64
+ how the application gets into the rake process at all, because outside Rails
65
+ nothing loads it for you.
66
+
67
+ `sneakers/tasks` declares `task :environment` **empty**, precisely so that you
68
+ can fill it in. Rake accumulates blocks for the same task rather than replacing
69
+ them, so adding your own is the supported way:
70
+
71
+ ```ruby
72
+ # Rakefile
73
+ require 'sneakers/tasks'
74
+
75
+ task(:environment) { require_relative 'app' }
76
+ ```
77
+
78
+ where `app.rb` is whatever configures Sneakers, calls `install!` and defines (or
79
+ requires) the worker classes. In Rails this is already handled — its own
80
+ `:environment` task loads the application — which is why the problem only shows
81
+ up outside it, usually as `sneakers:run` starting with no workers at all.
82
+
83
+ This is exactly the wiring the gem's own integration fixture uses
84
+ (`spec/integration/fixture/Rakefile`), so it is checked against a real broker on
85
+ every run of [VERIFYING.md](VERIFYING.md).
86
+
87
+ **Hanami 2** differs only in what that block contains: load the app the way
88
+ Hanami wants (`require 'hanami/prepare'` for a prepared, not fully booted,
89
+ application) and call `install!` after it. There is nothing Hanami-specific in
90
+ the gem — as above, the framework is not in the picture, only the question of
91
+ which file brings your worker classes into the process.
92
+
93
+ A minimal standalone runner:
94
+
95
+ ```ruby
96
+ require 'sneakers' # kicks installs this same file
97
+ require 'kicks_liveness'
98
+ require_relative 'workers/orders_worker'
99
+
100
+ Sneakers.configure(amqp: ENV.fetch('AMQP_URL'), workers: 2, threads: 10)
101
+ KicksLiveness.install!
102
+
103
+ Sneakers::Runner.new([OrdersWorker]).run
104
+ ```
105
+
106
+ ## Workers must be started through `Sneakers::Runner`
107
+
108
+ This is the one real constraint, and it has nothing to do with frameworks.
109
+
110
+ The monitor thread is started from the hook on
111
+ `Sneakers::WorkerGroup#after_fork`, and `WorkerGroup` enters the picture through
112
+ `Sneakers::Runner`, which builds a ServerEngine supervisor. If you boot workers
113
+ by hand — instantiating worker classes and calling `worker.run` in your own
114
+ loop, without ServerEngine — that hook never fires. The registry will fill up,
115
+ but nothing writes the mark files, so the probe stays red and the pod is killed.
116
+
117
+ Use `Sneakers::Runner` (that is what `rake sneakers:run` does), or do not use
118
+ this gem.
119
+
120
+ ## Configuration
121
+
122
+ Everything has a sensible default; in most applications the block is one line.
123
+
124
+ ```ruby
125
+ KicksLiveness.configure do |config|
126
+ config.enabled = ENV['RACK_ENV'] != 'test'
127
+ end
128
+ ```
129
+
130
+ **Do not set `config.logger = Sneakers.logger`.** That is already the default,
131
+ and writing it out is worse than leaving it alone, because the default is
132
+ resolved *lazily* and the assignment is not. `Sneakers.logger` is `nil` until
133
+ `Sneakers.configure` runs, so whether the assignment captures a logger or a
134
+ `nil` depends on which file Rails loads first — and initializers load in
135
+ alphabetical order, which is not something you chose or want to depend on. An
136
+ initializer named `kicks_liveness.rb` sorts *before* `sneakers.rb`; one named
137
+ `worker_liveness.rb` sorts after. Same code, different outcome.
138
+
139
+ Set it only to point somewhere **other** than `Sneakers.logger`:
140
+
141
+ ```ruby
142
+ config.logger = Rails.logger
143
+ ```
144
+
145
+ | Option | Default | |
146
+ |---|---|---|
147
+ | `logger` | `Sneakers.logger`, resolved lazily | where transitions are logged |
148
+ | `enabled` | `true` | set to `false` in tests, so no thread is started |
149
+ | `tick` | `10` | seconds between checks |
150
+ | `startup_grace_ticks` | `6` | unhealthy ticks tolerated at startup before one ERROR |
151
+
152
+ `dir` and `max_age` are **not** here — they come from environment variables
153
+ only. See
154
+ [DESIGN.md](DESIGN.md#what-is-configurable-and-where) for why, and
155
+ [KUBERNETES.md](KUBERNETES.md#environment-variables) for the variables.
156
+
157
+ ## Checking that the hooks are in place
158
+
159
+ In a console of the process that boots your workers:
160
+
161
+ ```ruby
162
+ Sneakers::Worker.ancestors.include?(KicksLiveness::Hooks::Worker) # => true
163
+ Sneakers::WorkerGroup.ancestors.include?(KicksLiveness::Hooks::WorkerGroup) # => true
164
+ ```
165
+
166
+ Both must be `true` before the runner starts. If either is `false`, either
167
+ `install!` has not run, or it ran before `kicks`/`sneakers` was loaded.
168
+
169
+ Once workers are running, the marks directory is the other half of the answer:
170
+
171
+ ```
172
+ $ ls -l /opt/app/tmp/health/
173
+ expected
174
+ worker-0
175
+ worker-1
176
+ ```
177
+
178
+ One `worker-<slot>` file per fork, plus `expected`. If `expected` is there and
179
+ the slot files are not, the workers have not finished subscribing. A slot that
180
+ keeps restarting without ever subscribing also leaves an `attempt-<slot>` file,
181
+ which is removed as soon as that slot becomes healthy.
data/docs/VERIFYING.md ADDED
@@ -0,0 +1,330 @@
1
+ # Verifying the probe against a real broker
2
+
3
+ [KUBERNETES.md](KUBERNETES.md#verifying-on-a-live-pod) shows how to confirm the
4
+ probe on a pod you already run. This document is the other half: a disposable
5
+ local cluster where the **failures themselves** can be caused on purpose, so
6
+ that every claim the gem makes is watched rather than reasoned about.
7
+
8
+ Two of those claims cannot be checked any other way, because both are properties
9
+ of Bunny rather than of this gem:
10
+
11
+ - that `channel.any_consumers?` turns false when the broker takes a consumer
12
+ away, which is what the probe fails on;
13
+ - that `recovering_from_network_failure?` stays true for the whole of a broker
14
+ outage, which is what keeps a hiccup from restarting every replica.
15
+
16
+ And one is a property of the kubelet: that a red probe actually restarts the
17
+ container. The `touch -d` recipe in KUBERNETES.md deliberately repairs itself
18
+ within one tick, so on its own it never reaches `failureThreshold`.
19
+
20
+ The fixture lives in `spec/integration/` in the repository. It is not part of the
21
+ published gem, and it is not part of `rake` — nothing under it is a `_spec.rb`
22
+ file, so `rspec` does not collect it.
23
+
24
+ ## What you need
25
+
26
+ A local single-node cluster. Docker Desktop's built-in Kubernetes is the
27
+ cheapest option, because an image built locally is visible to the cluster
28
+ immediately, with no registry and no load step — `imagePullPolicy: Never` is all
29
+ it takes.
30
+
31
+ ```bash
32
+ spec/integration/verify.sh build # image, into the engine the cluster uses
33
+ spec/integration/verify.sh up # namespace, broker, worker
34
+ spec/integration/verify.sh down # deletes the namespace
35
+ ```
36
+
37
+ Every `kubectl` call in that script names its context and namespace explicitly.
38
+ That is not politeness: a liveness experiment deletes queues and kills pods, and
39
+ it must not be able to reach a real cluster because someone's current-context
40
+ happened to point at one.
41
+
42
+ Three things that cost time if you meet them the hard way:
43
+
44
+ - **Build into the right engine.** If the machine also runs OrbStack, Colima or
45
+ Rancher Desktop, the default docker context is not the one Docker Desktop's
46
+ Kubernetes reads images from, and the pod fails with `ErrImageNeverPull`. The
47
+ script passes `--context desktop-linux` for this reason.
48
+ - **A `path:` gem is not in `GEM_HOME`.** Bundler adds it to the load path and
49
+ nothing else. The standard command, `bundle exec kicks-liveness`, does not care
50
+ — Bundler put it there and Bundler finds it. The **optimised** command,
51
+ plain `ruby -e` with no Bundler, raises `LoadError`. It is one instance of
52
+ [where your image puts its gems](KUBERNETES.md#where-your-image-puts-its-gems),
53
+ and worth meeting here rather than in a rollout. The fixture image therefore
54
+ also `gem install`s the gem it builds, so that both commands are exercised the
55
+ way a published gem would be.
56
+ - **`ruby:*-slim` has no compiler.** `bunny` pulls `sorted_set`, which pulls
57
+ `rbtree`, which is a native extension.
58
+
59
+ ## The scenarios
60
+
61
+ Each one is an action and a single observable value. The values below are the
62
+ ones actually recorded on Kubernetes v1.25 with kicks 3.4.0, bunny 3.3.0 and
63
+ ruby 3.4, using the probe settings printed in KUBERNETES.md
64
+ (`tick` 10 s, `max_age` 45 s, liveness `periodSeconds` 30 × `failureThreshold` 3).
65
+
66
+ The same scenarios have since been replayed against a production Rails
67
+ application — five consumers, two runners, one of them an ActiveJob adapter
68
+ whose worker set is a callable registry rather than an array — on the same kind
69
+ of throwaway cluster. Every outcome matched the fixture's, which is the point of
70
+ keeping the fixture small: if a two-queue toy and a real application disagree,
71
+ the disagreement is the finding.
72
+
73
+ The **startup** budget never binds anything here, so no number below depends on
74
+ it: this fixture subscribes in under a second, where a real application takes
75
+ tens of seconds. That is precisely why the cold-start measurement in
76
+ [KUBERNETES.md](KUBERNETES.md#measure-the-cold-start-not-the-restart) has to be
77
+ made against your own application — a fixture cannot do it for you.
78
+
79
+ ### 1. It comes up
80
+
81
+ ```bash
82
+ spec/integration/verify.sh logs | grep liveness
83
+ spec/integration/verify.sh marks
84
+ spec/integration/verify.sh probe
85
+ ```
86
+
87
+ ```
88
+ INFO: [liveness] slot 0: started: dir=/opt/app/tmp/health max_age=45s tick=10s processes=1 consumers=2
89
+ INFO: [liveness] slot 0: waiting for 2 consumers
90
+ INFO: [liveness] slot 0: healthy # one tick later
91
+ expected worker-0 # in the marks directory
92
+ 1 process(es) healthy # probe, exit 0
93
+ ```
94
+
95
+ The hooks fired inside a real ServerEngine fork, which no double can show.
96
+
97
+ ### 2. A stale mark
98
+
99
+ The recipe from KUBERNETES.md. Proves the probe's threshold and the
100
+ self-repair, and nothing else — the mark is fixed on the next tick, well inside
101
+ `failureThreshold`, so the container is never killed.
102
+
103
+ Worth running through **both** commands here, since this is the one scenario
104
+ where the probe is supposed to answer `1`. `spec/integration/verify.sh probe`
105
+ does exactly that: the standard `bundle exec kicks-liveness` and the optimised
106
+ `ruby -e`, which in this image are both available and must agree.
107
+
108
+ ```
109
+ worker-0 stale 120s > 45s # exit 1
110
+ 1 process(es) healthy # exit 0, twelve seconds later
111
+ ```
112
+
113
+ ### 3. The broker takes the consumer away
114
+
115
+ The one that matters. Delete a queue the worker consumes from:
116
+
117
+ ```bash
118
+ kubectl exec deploy/rabbitmq -- rabbitmqctl delete_queue kicks_liveness.alpha
119
+ ```
120
+
121
+ Observed, from the moment of deletion:
122
+
123
+ | after the deletion | what |
124
+ |---|---|
125
+ | 9 s | `ERROR: [liveness] slot 0: became unhealthy (expected 2 consumers)` |
126
+ | 59 s | `ERROR: [liveness] slot 0: not healthy 60s, expected 2 consumers` |
127
+ | 61 s | first kubelet failure: `Liveness probe failed: worker-0 stale 69s > 45s` |
128
+ | 91 s, 121 s | the second and third, `stale 99s` and `stale 129s` |
129
+ | 132 s | `Killing: Container worker failed liveness probe, will be restarted` |
130
+
131
+ The two clocks in that table are different, which is worth reading carefully
132
+ if you are checking the arithmetic. The left column counts from the deletion.
133
+ The `stale Ns` in the probe's own text is the age of the mark, and the mark
134
+ stopped being refreshed one tick *before* the transition was logged — the tick
135
+ that found the process unhealthy is also the first one that did not touch the
136
+ file. So the probe's number runs about eight seconds ahead of the left column,
137
+ and the kill lands at `max_age + periodSeconds × failureThreshold` counted
138
+ from the last refresh, not from the deletion.
139
+
140
+ So `any_consumers?` does turn false, the transition is reported within one tick,
141
+ and the pod is restarted after `max_age + periodSeconds × failureThreshold`
142
+ — 132 s against the 135 s the arithmetic in KUBERNETES.md predicts. The worker
143
+ re-declares the queue on boot, so the pod recovers by itself.
144
+
145
+ Note where the probe's own text appears: in the kubelet's `Unhealthy` event,
146
+ not in the pod log. An exec probe's stdout goes nowhere else, which is why
147
+ `kubectl describe pod` is the place to read it.
148
+
149
+ ### 4. The broker dies
150
+
151
+ ```bash
152
+ kubectl delete pod -l app=rabbitmq --grace-period=0 --force
153
+ ```
154
+
155
+ The pod must **not** restart. "It survived" is not enough on its own — a broker
156
+ that never actually went away would look the same — so check the log for
157
+ Bunny's own account of the outage:
158
+
159
+ ```
160
+ WARN: Recovering from connection.close (CONNECTION_FORCED - broker forced connection closure with reason 'shutdown')
161
+ WARN: Will recover from a network failure (no retry limit)...
162
+ WARN: Could not establish TCP connection to rabbitmq:5672 ... TCP connection failed
163
+ WARN: Reconnecting in 5.0 seconds
164
+ ```
165
+
166
+ Against that: `restartCount` stayed 0, the probe exited 0 at every check, and
167
+ no `became unhealthy` was logged. Bunny's topology recovery brought both
168
+ consumers back on its own.
169
+
170
+ ### 5. The broker stays dead
171
+
172
+ `kubectl scale deploy/rabbitmq --replicas=0`, then wait. Observed over 319 s
173
+ — seven times `max_age`, more than twice the 135 s steady-state budget — the
174
+ mark's age never exceeded 9 s and `restartCount` stayed 0. The monitor kept
175
+ writing, because `recovering_from_network_failure?` never stops being true:
176
+ Bunny has no retry limit, which its own log line above says out loud.
177
+
178
+ This is
179
+ [Broker unavailability is invisible by design](LIMITATIONS.md#broker-unavailability-is-invisible-by-design),
180
+ seen rather than deduced. It is the reason the
181
+ [consumer alert](KUBERNETES.md#required-companion-an-alert-on-consumers) is part
182
+ of the design and not a suggestion.
183
+
184
+ Scaling the broker back up: the worker resumed consuming **in the same
185
+ container**, no restart.
186
+
187
+ ### 6. An ordinary deploy
188
+
189
+ `kubectl delete pod` on the worker, with the log followed. Not one ERROR line.
190
+ Bunny cancels each consumer and the process exits.
191
+
192
+ If the shutdown finishes inside a single tick — an idle thread pool cancels
193
+ quickly — the monitor never gets to run again, so
194
+ `shutting down: consumers are no longer checked` may not appear at all. The
195
+ absence of ERROR is the observable, not the presence of that line.
196
+
197
+ ### 7. Bad broker credentials
198
+
199
+ Point `AMQP_URL` at a wrong password. The fork cannot subscribe, so the
200
+ supervisor respawns it every `start_worker_delay`, forever. Over 120 s:
201
+
202
+ | | lines |
203
+ |---|---|
204
+ | fork starts | 426 |
205
+ | `started: dir=... consumers=2` | 1 |
206
+ | `waiting for 2 consumers` | 1 |
207
+ | liveness ERROR | 2 |
208
+
209
+ ```
210
+ INFO: [liveness] slot 0: started: dir=/opt/app/tmp/health max_age=45s tick=10s processes=1 consumers=2
211
+ INFO: [liveness] slot 0: waiting for 2 consumers
212
+ ERROR: [liveness] slot 0: not healthy 60s over 224 starts, expected 2 consumers
213
+ ERROR: [liveness] slot 0: not healthy 60s over 426 starts, expected 2 consumers
214
+ ```
215
+
216
+ Four lines from the gem across 426 respawns: the first incarnation announces
217
+ itself and reports waiting, every one after that is silent, and the escalation
218
+ speaks once per grace window carrying the number of starts, which is the
219
+ diagnosis. This is the behaviour
220
+ [the respawn section](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn)
221
+ describes, against a real storm rather than a simulated one.
222
+
223
+ Worth knowing what the pod log looks like anyway: 27 000 lines in those same
224
+ 120 s, an authentication error and a full backtrace printed by the worker gem on
225
+ every respawn. The gem's four lines are in there — `grep liveness` is how you
226
+ find them.
227
+
228
+ ### 8. More than one fork
229
+
230
+ `WORKER_COUNT=2` needs no rebuild; the rake task turns it into
231
+ `config[:workers]`.
232
+
233
+ ```
234
+ INFO: [liveness] slot 0: started: ... processes=2 consumers=2
235
+ INFO: [liveness] slot 1: started: ... processes=2 consumers=2
236
+ expected worker-0 worker-1 # expected contains "2"
237
+ 2 process(es) healthy # exit 0
238
+ worker-1 stale 120s > 45s # exit 1 — staling either mark is enough
239
+ ```
240
+
241
+ Both marks are required, not just the first.
242
+
243
+ One trap while checking this by hand: `kubectl exec deploy/worker` picks any pod
244
+ matching the deployment, and during a rollout that can still be the old one.
245
+ Read the marks on a pod named explicitly.
246
+
247
+ ### 9. A rolling deploy keeps the queue covered
248
+
249
+ Scenario 6 kills a pod outright. A deploy is the more interesting case,
250
+ because KUBERNETES.md claims the startup probe *incidentally fixes rollouts
251
+ where the old pod was removed before the new one had subscribed* — and that
252
+ claim is about ordering, which needs two pods to be watched at once.
253
+
254
+ `kubectl rollout restart deploy/worker`, with `WORKER_COUNT=2`, so a fully
255
+ subscribed pod is worth two consumers per queue. Counting consumers on the
256
+ broker is what makes the ordering visible:
257
+
258
+ | after | consumers on the queue | pods |
259
+ |---|---|---|
260
+ | 1 s | 2 | old Ready, new Pending |
261
+ | 10 s | **4** | old Ready, new Running but **not** Ready |
262
+ | 16 s | — | the old pod cancels its consumers |
263
+ | 18 s | 2 | new pod only, Ready |
264
+
265
+ Coverage went 2 → 4 → 2 and never dipped. The new pod had subscribed before
266
+ the old one was asked to stop, and the old one logged no ERROR on its way out.
267
+
268
+ The mechanism is worth being explicit about, because it is the probe doing it:
269
+ a container running its startup probe is not Ready, and with one replica
270
+ `maxUnavailable: 25%` rounds down to zero, so Kubernetes may not remove the
271
+ old pod until the new one is Ready. Readiness here means *the mark is on
272
+ tmpfs*, which means the consumers are subscribed. A probe that answered on
273
+ process liveness instead would have released the old pod while the new one was
274
+ still booting.
275
+
276
+ ### 10. The alert's metric is the one the docs name
277
+
278
+ The [consumer alert](KUBERNETES.md#required-companion-an-alert-on-consumers) is
279
+ part of the design, not a suggestion — so its expression deserves the same
280
+ treatment as everything else here. The broker in this fixture is enough to check
281
+ it; the plugin is not on by default:
282
+
283
+ ```bash
284
+ kubectl exec deploy/rabbitmq -- rabbitmq-plugins enable rabbitmq_prometheus
285
+ ```
286
+
287
+ Then read all three endpoints from the worker pod, which has a Ruby but no
288
+ `curl`:
289
+
290
+ ```bash
291
+ kubectl exec deploy/worker -- ruby -rnet/http -e \
292
+ 'puts Net::HTTP.get(URI("http://rabbitmq:15692/metrics/detailed?family=queue_consumer_count"))'
293
+ ```
294
+
295
+ Recorded on RabbitMQ 3.13.7:
296
+
297
+ | endpoint | what comes back |
298
+ |---|---|
299
+ | `/metrics` | `rabbitmq_queue_consumers 4` — one label-less cluster total |
300
+ | `/metrics/detailed` | telemetry and build info only, no queue metrics |
301
+ | `/metrics/detailed?family=queue_consumer_count` | `rabbitmq_detailed_queue_consumers{vhost="/",queue="kicks_liveness.alpha"} 2` |
302
+
303
+ Three things follow, and all three are now in the manifest documentation: the
304
+ metric is a *different* name from the aggregate, the detailed endpoint needs the
305
+ `family` parameter, and the series carries a `vhost` label.
306
+
307
+ Then check the two halves of the alert against a throwaway queue — declare one
308
+ with no consumers, scrape, delete it, scrape again:
309
+
310
+ ```
311
+ -- queue exists, zero consumers
312
+ rabbitmq_detailed_queue_consumers{vhost="/",queue="kicks_liveness.tmpcheck"} 0
313
+ -- queue deleted
314
+ (no such series)
315
+ ```
316
+
317
+ Both halves matter and they are different failures. An **idle** queue reports
318
+ `0`, which is what `== 0` is for. A **deleted** queue reports nothing at all,
319
+ and neither does a stopped broker or a stopped exporter — so `== 0` cannot fire
320
+ on the very failure scenario 3 demonstrates, where the queue goes away and the
321
+ pod is restarted. That is why the documented rule set pairs it with
322
+ `absent_over_time`, and why an `up == 0` on the scrape job is worth having
323
+ beside both.
324
+
325
+ ## What to do with a disagreement
326
+
327
+ Record it, then fix whichever is wrong — the code or the document. The numbers
328
+ above are a baseline: the arithmetic in KUBERNETES.md is meant to predict them,
329
+ and a scenario that lands far from its prediction means one of the two is out of
330
+ date.
@@ -0,0 +1,3 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require 'kicks_liveness/probe'
@@ -0,0 +1,86 @@
1
+ require 'fileutils'
2
+
3
+ module KicksLiveness
4
+ # Counts starts of a supervisor slot that have not reached healthy yet.
5
+ #
6
+ # Deliberately separate from {Heartbeat}. Heartbeat is the contract shared
7
+ # with the probe and is loaded by it alone, so nothing the probe does not need
8
+ # belongs there — including the +fileutils+ this class requires freely.
9
+ #
10
+ # The count lives in the marks directory rather than in memory because a
11
+ # respawn storm destroys process state every few hundred milliseconds: the
12
+ # supervisor brings a failing fork back after +start_worker_delay+, and every
13
+ # incarnation of the monitor believes it is the first. That is how an outage
14
+ # can produce thousands of identical INFO lines and not one ERROR. The
15
+ # directory is the only state that survives the fork.
16
+ #
17
+ # @see file:docs/DESIGN.md#logging-events-not-the-pulse
18
+ # @api private
19
+ class Attempts
20
+ # @param dir [String] marks directory
21
+ def initialize(dir)
22
+ @dir = dir
23
+ end
24
+
25
+ # Records a start of this slot.
26
+ #
27
+ # @param slot [Integer] supervisor slot of this fork
28
+ # @param now [Time] injected in specs
29
+ # @return [Array(Integer, Float)] this start's number, and seconds since the
30
+ # first start of the current unhealthy run
31
+ def record!(slot, now: Time.now.utc)
32
+ count, first = read(slot)
33
+ first ||= now
34
+ write(slot, count + 1, first)
35
+ [count + 1, now - first]
36
+ end
37
+
38
+ # Resets the elapsed-time window while keeping the count, so a respawn loop
39
+ # reports once per window instead of once per respawn.
40
+ #
41
+ # @param slot [Integer]
42
+ # @param now [Time]
43
+ # @return [void]
44
+ def restart_window!(slot, now: Time.now.utc)
45
+ count, = read(slot)
46
+ write(slot, count, now)
47
+ end
48
+
49
+ # Called once the slot is healthy: the next start of it is a fresh one.
50
+ #
51
+ # @param slot [Integer]
52
+ # @return [void]
53
+ def clear!(slot)
54
+ File.unlink(path(slot))
55
+ rescue StandardError
56
+ nil
57
+ end
58
+
59
+ private
60
+
61
+ def path(slot)
62
+ File.join(@dir, "attempt-#{slot}")
63
+ end
64
+
65
+ # A missing or unreadable counter means "no previous start", never an
66
+ # exception: this is instrumentation, and it may not break a worker.
67
+ def read(slot)
68
+ count, epoch = File.read(path(slot)).split
69
+ [Integer(count), Time.at(Integer(epoch)).utc]
70
+ rescue StandardError
71
+ [0, nil]
72
+ end
73
+
74
+ # Failing to write degrades the reporting back to one line per respawn,
75
+ # which is noisy but harmless — so a read-only directory is swallowed here
76
+ # rather than escalated.
77
+ def write(slot, count, first)
78
+ FileUtils.mkdir_p(@dir)
79
+ tmp = "#{path(slot)}.#{Process.pid}"
80
+ File.write(tmp, "#{count} #{first.to_i}\n")
81
+ File.rename(tmp, path(slot))
82
+ rescue StandardError
83
+ nil
84
+ end
85
+ end
86
+ end
@@ -0,0 +1,83 @@
1
+ require_relative 'heartbeat'
2
+
3
+ module KicksLiveness
4
+ # What the application may configure, and what comes from the environment only.
5
+ #
6
+ # +dir+ and +max_age+ are deliberately read-only here: the probe runs as a
7
+ # separate process launched by the container runtime and cannot read the
8
+ # application's initializer. Were they settable in both places, a mismatch
9
+ # between what the worker writes and what the probe expects would pass
10
+ # unnoticed.
11
+ #
12
+ # @see file:docs/SETUP.md#configuration
13
+ # @see file:docs/DESIGN.md#what-is-configurable-and-where
14
+ class Configuration
15
+ # @return [Integer] default seconds between checks
16
+ DEFAULT_TICK = 10
17
+ # @return [Integer] unhealthy ticks tolerated at startup before one ERROR
18
+ DEFAULT_STARTUP_GRACE_TICKS = 6
19
+
20
+ # @return [Integer] seconds between checks
21
+ attr_reader :tick
22
+ # @return [Integer] unhealthy ticks tolerated at startup before one ERROR
23
+ attr_accessor :startup_grace_ticks
24
+ # @return [Logger, nil] explicit logger; defaults to +Sneakers.logger+
25
+ attr_accessor :logger
26
+ # @param value [Boolean] set false to start no monitor thread, e.g. in tests
27
+ attr_writer :enabled
28
+
29
+ def initialize
30
+ @tick = Heartbeat.env_int(:tick, DEFAULT_TICK)
31
+ @startup_grace_ticks = DEFAULT_STARTUP_GRACE_TICKS
32
+ @enabled = true
33
+ @logger = nil
34
+ end
35
+
36
+ # @return [Boolean]
37
+ def enabled?
38
+ @enabled
39
+ end
40
+
41
+ # A tick is what the monitor thread sleeps on, so a non-positive value is
42
+ # not a setting but a broken monitor. The environment gets a silent fallback
43
+ # instead (see {Heartbeat.env_int}) because a ConfigMap typo must not bring
44
+ # a worker down; an initializer is code, and code should say so at boot,
45
+ # where the developer is looking.
46
+ #
47
+ # @param seconds [Integer]
48
+ # @raise [ArgumentError] if not a positive number
49
+ # @return [Integer]
50
+ def tick=(seconds)
51
+ raise ArgumentError, "tick must be a positive number, got #{seconds.inspect}" unless positive_number?(seconds)
52
+
53
+ @tick = seconds
54
+ end
55
+
56
+ # Read-only, sourced from the environment so that it matches what the probe
57
+ # sees.
58
+ # @return [String] marks directory
59
+ def dir
60
+ Heartbeat.env_dir
61
+ end
62
+
63
+ # Read-only, sourced from the environment so that it matches what the probe
64
+ # sees.
65
+ # @return [Integer] seconds after which a mark is considered stale
66
+ def max_age
67
+ Heartbeat.env_max_age
68
+ end
69
+
70
+ # Resolved lazily: at the time this object is built, +Sneakers.logger+ may
71
+ # not be configured yet.
72
+ # @return [Logger, false, nil]
73
+ def resolved_logger
74
+ @logger || (defined?(::Sneakers) && ::Sneakers.logger)
75
+ end
76
+
77
+ private
78
+
79
+ def positive_number?(value)
80
+ value.is_a?(Numeric) && value.positive?
81
+ end
82
+ end
83
+ end