kicks_liveness 0.1.0 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.yardopts +13 -0
- data/CHANGELOG.md +58 -0
- data/README.md +133 -249
- data/docs/DESIGN.md +51 -16
- data/docs/KUBERNETES.md +32 -9
- data/docs/LIMITATIONS.md +54 -15
- data/docs/SETUP.md +16 -11
- data/docs/VERIFYING.md +54 -9
- data/lib/kicks_liveness/configuration.rb +73 -10
- data/lib/kicks_liveness/heartbeat.rb +102 -15
- data/lib/kicks_liveness/hooks.rb +12 -1
- data/lib/kicks_liveness/monitor.rb +6 -0
- data/lib/kicks_liveness/version.rb +1 -1
- data/lib/kicks_liveness.rb +15 -3
- metadata +5 -8
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: df61d00933097bb96e6d752e024e349ab631cd7c9d571cd9bff9ff63b8adda40
|
|
4
|
+
data.tar.gz: 78864373ca06cd998cdee192d064cd00ed69a4ecd73eb1b657474d77ab593066
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: '007091e18fea1b216ba25c634bab1a4770d25cc0f062ee30ba2885563075bc94e7662e58eebbac9507b448217508725612f172189b692e3a4d616e5f81fdd46b'
|
|
7
|
+
data.tar.gz: f460271ca68fdf912559b3fa2c41b02074d5eb457e946d2f985e3e2b0ac3da66dc94ce90c9d5639212a0fc7bb6537046654f31921cf6766fe15c903bd7dc04ef
|
data/.yardopts
ADDED
data/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,61 @@
|
|
|
3
3
|
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
4
4
|
and this project adheres to [Semantic Versioning](https://semver.org/).
|
|
5
5
|
|
|
6
|
+
## [Unreleased]
|
|
7
|
+
|
|
8
|
+
## [0.1.2] - 2026-09-11
|
|
9
|
+
|
|
10
|
+
### Fixed
|
|
11
|
+
|
|
12
|
+
- Reject heartbeat files inherited from a previous container incarnation. A
|
|
13
|
+
Kubernetes `emptyDir` survives container restarts, so a fresh mark left by a
|
|
14
|
+
formerly healthy worker could make the next container pass `startupProbe`
|
|
15
|
+
before its consumers had subscribed. Cache files elsewhere in the same
|
|
16
|
+
`emptyDir` remain untouched and available to warm the next application boot.
|
|
17
|
+
This protection assumes the default container-private PID namespace; pods
|
|
18
|
+
using `shareProcessNamespace` or `hostPID` retain the 0.1.1 freshness-only
|
|
19
|
+
behavior (see `docs/LIMITATIONS.md`).
|
|
20
|
+
The marks directory now contains `<dir>/generation`, and each
|
|
21
|
+
`worker-<slot>` mark carries the same value; tooling that enumerates the
|
|
22
|
+
directory should allow for this new metadata file.
|
|
23
|
+
|
|
24
|
+
## [0.1.1] - 2026-09-09
|
|
25
|
+
|
|
26
|
+
### Added
|
|
27
|
+
|
|
28
|
+
- Add a repository-hosted social-preview asset for the project documentation.
|
|
29
|
+
|
|
30
|
+
### Changed
|
|
31
|
+
|
|
32
|
+
- Rework the README into a shorter entry point while retaining the operational
|
|
33
|
+
detail in the focused guides under `docs/`.
|
|
34
|
+
- Reject initializer values where `tick >= max_age`, and require
|
|
35
|
+
`startup_grace_ticks` to be a positive integer. Initializers that previously
|
|
36
|
+
supplied either invalid value now stop application boot with `ArgumentError`.
|
|
37
|
+
- Reject bundles that activate both `kicks` and `sneakers` with `LoadError`
|
|
38
|
+
instead of letting their shared `lib/sneakers.rb` resolve silently by
|
|
39
|
+
load-path order. Applications that still bundle both worker gems must remove
|
|
40
|
+
one before upgrading.
|
|
41
|
+
- Replace the contributor-only `AMQP_WORKER_GEM` and
|
|
42
|
+
`AMQP_WORKER_GEM_VERSION` test selection with Appraisal-generated dependency
|
|
43
|
+
sets, including current and supported-floor targets for both worker gems.
|
|
44
|
+
|
|
45
|
+
### Fixed
|
|
46
|
+
|
|
47
|
+
- Recover environment values where `tick >= max_age` with a warning and a safe
|
|
48
|
+
interval, preventing a healthy heartbeat from becoming stale between normal
|
|
49
|
+
monitor ticks.
|
|
50
|
+
- Keep worker startup, heartbeat publication, and the monitor loop alive when a
|
|
51
|
+
custom logger raises while handling an INFO or ERROR event.
|
|
52
|
+
- Correct the runtime-scaling documentation: live changes to `workers` are not
|
|
53
|
+
supported in either direction. During scale-up, old and new forks otherwise
|
|
54
|
+
race to publish different expected counts; during scale-down, retired slots
|
|
55
|
+
remain expected.
|
|
56
|
+
- Strengthen the release gate with floor-version coverage, style checks, tag
|
|
57
|
+
ancestry validation, and a packaged-executable smoke test.
|
|
58
|
+
- Include `.yardopts` in the gem so RubyDoc builds the intended README and
|
|
59
|
+
public guide set from the packaged release.
|
|
60
|
+
|
|
6
61
|
## [0.1.0] - 2026-09-08
|
|
7
62
|
|
|
8
63
|
### Added
|
|
@@ -18,4 +73,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
|
|
|
18
73
|
[docs/](https://github.com/PoroshkinaVV/kicks_liveness/tree/main/docs); start
|
|
19
74
|
with `SETUP.md`, and read `LIMITATIONS.md` before relying on it.
|
|
20
75
|
|
|
76
|
+
[Unreleased]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.2...HEAD
|
|
77
|
+
[0.1.2]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.1...v0.1.2
|
|
78
|
+
[0.1.1]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.0...v0.1.1
|
|
21
79
|
[0.1.0]: https://github.com/PoroshkinaVV/kicks_liveness/releases/tag/v0.1.0
|
data/README.md
CHANGED
|
@@ -1,119 +1,75 @@
|
|
|
1
1
|
# kicks_liveness
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
tmpfs, and the probe only reads it.
|
|
3
|
+
[](https://rubygems.org/gems/kicks_liveness)
|
|
4
|
+
[](https://github.com/PoroshkinaVV/kicks_liveness/actions/workflows/ci.yml)
|
|
6
5
|
|
|
7
|
-
|
|
6
|
+

|
|
8
7
|
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
- [docs/DESIGN.md](docs/DESIGN.md) — why the gem is built this way: the health
|
|
13
|
-
predicate, the mark files, slots instead of PIDs, where the hooks attach, what
|
|
14
|
-
is configurable and why `dir` and `max_age` live in the environment only.
|
|
15
|
-
- [docs/KUBERNETES.md](docs/KUBERNETES.md) — the manifest, the budget
|
|
16
|
-
arithmetic, verifying on a live pod, the alert on consumers.
|
|
17
|
-
- [docs/LIMITATIONS.md](docs/LIMITATIONS.md) — where the in-memory predicate
|
|
18
|
-
stops being truthful.
|
|
19
|
-
- [docs/VERIFYING.md](docs/VERIFYING.md) — the runbook that causes the real failures on a local cluster: what was observed, and how long each took.
|
|
20
|
-
|
|
21
|
-
## Why
|
|
22
|
-
|
|
23
|
-
The usual implementation is a rake task that boots Rails and asks RabbitMQ for
|
|
24
|
-
`consumer_count`. It has three defects that cannot be fixed in place:
|
|
25
|
-
|
|
26
|
-
1. **It is expensive.** Booting Rails every few seconds inside the worker's own
|
|
27
|
-
cgroup. On one real service this cost 2.7 s per invocation on a 15 s
|
|
28
|
-
period — 38% of the container's CPU request, around the clock.
|
|
29
|
-
2. **It depends on the disk.** A disk that stalls for a few seconds stretches
|
|
30
|
-
the boot past `timeoutSeconds`, and the kubelet kills a healthy pod.
|
|
31
|
-
3. **It cascades.** A liveness probe that checks an external dependency turns a
|
|
32
|
-
broker hiccup into every replica restarting at once, finishing the broker off.
|
|
33
|
-
|
|
34
|
-
On top of that, `consumer_count` is a metric of the **queue**, not of the pod:
|
|
35
|
-
with two replicas the live one covers for the stalled one, and the probe cannot
|
|
36
|
-
tell.
|
|
37
|
-
|
|
38
|
-
This gem inverts that. Every 10 seconds the worker checks, against the Bunny
|
|
39
|
-
objects **in its own memory**, that its consumers are in place, and touches a
|
|
40
|
-
file. The standard probe invocation took 156 ms in the fixture container, makes
|
|
41
|
-
no network call, boots no framework, and reads the state it judges from tmpfs,
|
|
42
|
-
which is RAM. It does still load Bundler, the interpreter and the gem's probe
|
|
43
|
-
files from the image filesystem, so the disk is not out of the picture
|
|
44
|
-
altogether — it is reduced to a small process startup instead of a full
|
|
45
|
-
application boot on every probe.
|
|
46
|
-
|
|
47
|
-
## Installation
|
|
8
|
+
**Know when a [Kicks](https://github.com/ruby-amqp/kicks) or
|
|
9
|
+
[Sneakers](https://github.com/jondot/sneakers) worker is stuck — without booting
|
|
10
|
+
Rails or querying RabbitMQ from the probe.**
|
|
48
11
|
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
12
|
+
The worker checks its own Bunny consumers and publishes a heartbeat to tmpfs.
|
|
13
|
+
The probe only reads that heartbeat, scoped to the current container
|
|
14
|
+
incarnation. No application boot, no network call, and no healthy replica
|
|
15
|
+
hiding a stalled one.
|
|
52
16
|
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
versions, not just through a `~>` that resolves to the newest release.
|
|
17
|
+
That restart scoping assumes Kubernetes' default container-private PID
|
|
18
|
+
namespace; see
|
|
19
|
+
[Limitations](docs/LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1)
|
|
20
|
+
for pods that set `shareProcessNamespace` or `hostPID`.
|
|
58
21
|
|
|
59
|
-
|
|
60
|
-
that: not Bundler, not a crash at boot. Both own the file `lib/sneakers.rb`, so
|
|
61
|
-
one silently wins the load path. An application that has moved to `kicks` would
|
|
62
|
-
keep executing `sneakers` 2.12 code while its `Gemfile.lock` claims otherwise.
|
|
63
|
-
`sneakers` also caps the `kicks` and `bunny` versions.
|
|
22
|
+
## Install: one line in Rails
|
|
64
23
|
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
started through `Sneakers::Runner` — are in [docs/SETUP.md](docs/SETUP.md).
|
|
24
|
+
```ruby
|
|
25
|
+
# Gemfile — keep exactly one of kicks or sneakers alongside it
|
|
26
|
+
gem 'kicks_liveness'
|
|
27
|
+
```
|
|
70
28
|
|
|
71
|
-
|
|
29
|
+
```bash
|
|
30
|
+
bundle install
|
|
31
|
+
```
|
|
72
32
|
|
|
73
|
-
|
|
33
|
+
In Rails, that is all the application-side setup: the Railtie installs the hooks
|
|
34
|
+
automatically. Without Rails, add two lines before `Sneakers::Runner` starts:
|
|
74
35
|
|
|
75
36
|
```ruby
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
end
|
|
37
|
+
require 'kicks_liveness'
|
|
38
|
+
KicksLiveness.install!
|
|
79
39
|
```
|
|
80
40
|
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
so
|
|
41
|
+
Requirements: Ruby >= 3.1 and exactly one of `kicks` >= 3.0 or `sneakers` >=
|
|
42
|
+
2.11. CI pins both floors and tracks the current release of both worker lines.
|
|
43
|
+
Do not put both worker gems in the same bundle: both provide
|
|
44
|
+
`lib/sneakers.rb`, so `install!` rejects that ambiguous process.
|
|
45
|
+
Framework-specific examples are in [Setup](docs/SETUP.md).
|
|
46
|
+
|
|
47
|
+
## Why replace the usual probe?
|
|
85
48
|
|
|
86
|
-
|
|
49
|
+
The common alternative is a rake task that boots the application and asks
|
|
50
|
+
RabbitMQ for `consumer_count`.
|
|
87
51
|
|
|
88
|
-
|
|
|
52
|
+
| | Rails + `consumer_count` probe | `kicks_liveness` |
|
|
89
53
|
|---|---|---|
|
|
90
|
-
|
|
|
91
|
-
|
|
|
92
|
-
|
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
Garbage in a value does not bring the worker down — it falls back to the default.
|
|
111
|
-
An empty string counts as unset: in a ConfigMap that is what you get by
|
|
112
|
-
declaring a key and leaving it blank. `max_age` and `tick` are durations, so
|
|
113
|
-
zero and negative values fall back too: they parse perfectly well, and a
|
|
114
|
-
negative `max_age` would make every mark stale on arrival.
|
|
115
|
-
|
|
116
|
-
## Manifest
|
|
54
|
+
| Code started on every probe | Rails and the application | Ruby, Bundler, and the probe files |
|
|
55
|
+
| Network call | RabbitMQ request | none |
|
|
56
|
+
| Health scope | a queue shared by every replica | this worker process |
|
|
57
|
+
| Decision path | application files plus RabbitMQ | process memory plus a tmpfs heartbeat |
|
|
58
|
+
| Measured invocation | **2.7 s of CPU** on a real service | **156 ms** in the fixture container |
|
|
59
|
+
| Continuous cost in the measured setups | **~0.18 core** at a 15 s period | **~0.005 core** at a 30 s period |
|
|
60
|
+
| During Bunny network recovery | may restart every replica together | stays healthy while Bunny reconnects |
|
|
61
|
+
|
|
62
|
+
Absolute timings depend on the image and hardware; the two columns are measured
|
|
63
|
+
examples, not a same-host microbenchmark. The calculation and faster 52 ms and
|
|
64
|
+
~10 ms probe forms are documented in [Running under Kubernetes](docs/KUBERNETES.md#why-the-command-looks-like-that).
|
|
65
|
+
|
|
66
|
+
Every 10 seconds by default, each worker verifies that all expected consumers
|
|
67
|
+
are subscribed and touches its own mark. `bundle exec kicks-liveness` exits 0
|
|
68
|
+
only when every expected worker mark is present and fresh.
|
|
69
|
+
|
|
70
|
+
## Kubernetes
|
|
71
|
+
|
|
72
|
+
Use a startup probe for the boot budget and a liveness probe for steady state:
|
|
117
73
|
|
|
118
74
|
```yaml
|
|
119
75
|
startupProbe:
|
|
@@ -131,7 +87,7 @@ negative `max_age` would make every mark stale on arrival.
|
|
|
131
87
|
terminationGracePeriodSeconds: 60
|
|
132
88
|
```
|
|
133
89
|
|
|
134
|
-
|
|
90
|
+
Put the heartbeat directory on tmpfs:
|
|
135
91
|
|
|
136
92
|
```yaml
|
|
137
93
|
volumeMounts:
|
|
@@ -140,180 +96,108 @@ The marks directory has to be on tmpfs:
|
|
|
140
96
|
volumes:
|
|
141
97
|
- name: app-tmp
|
|
142
98
|
emptyDir:
|
|
143
|
-
medium: Memory
|
|
99
|
+
medium: Memory
|
|
144
100
|
```
|
|
145
101
|
|
|
146
|
-
The
|
|
147
|
-
`
|
|
148
|
-
|
|
149
|
-
Without this mount, a container using `readOnlyRootFilesystem: true` cannot
|
|
150
|
-
publish heartbeat files and never passes the startup probe.
|
|
151
|
-
|
|
152
|
-
Why it looks like this:
|
|
153
|
-
|
|
154
|
-
- **`bundle exec kicks-liveness` is the standard command**, and it works
|
|
155
|
-
regardless of where Bundler installed the gems. There is a faster form —
|
|
156
|
-
`ruby -e "require 'kicks_liveness/probe'"`, 52 ms against
|
|
157
|
-
156 ms — but it needs the gems to be in `GEM_HOME`, and **setting
|
|
158
|
-
`BUNDLE_PATH` at all moves them**, even to the directory `GEM_HOME` already
|
|
159
|
-
points at. Check before you optimise:
|
|
160
|
-
|
|
161
|
-
```bash
|
|
162
|
-
docker run --rm your-image bundle exec kicks-liveness
|
|
163
|
-
docker run --rm your-image ruby -e "require 'kicks_liveness/probe'"
|
|
164
|
-
```
|
|
165
|
-
|
|
166
|
-
Both should print `no .../expected` and exit 1. If the second raises
|
|
167
|
-
`LoadError`, keep the standard command — at `periodSeconds: 30` it costs about
|
|
168
|
-
0.005 of a core, against roughly 0.18 for a probe that boots the application.
|
|
169
|
-
The trade-offs are in
|
|
170
|
-
[docs/KUBERNETES.md](docs/KUBERNETES.md#where-your-image-puts-its-gems).
|
|
171
|
-
|
|
172
|
-
- **`startupProbe` is mandatory.** Startup and steady state have different time
|
|
173
|
-
budgets, and one probe cannot serve both. While it runs, liveness is disabled
|
|
174
|
-
and the container is not Ready. `failureThreshold: 60` buys 300 s, which is
|
|
175
|
-
deliberately generous — measure a *freshly created* pod before trimming it, as
|
|
176
|
-
a restarted container inherits a warm compile cache and a new pod does not:
|
|
177
|
-
[measure the cold start](docs/KUBERNETES.md#measure-the-cold-start-not-the-restart).
|
|
178
|
-
- **It also protects rollouts — given `maxUnavailable: 0`.** Because the
|
|
179
|
-
container is not Ready until the mark exists, a `RollingUpdate` that is not
|
|
180
|
-
allowed to drop below full capacity cannot remove the old pod before the new
|
|
181
|
-
one has subscribed. That is a property of the strategy as much as of the probe:
|
|
182
|
-
with a non-zero `maxUnavailable`, or with `Recreate`, the old pod may go first
|
|
183
|
-
and leave the queue uncovered.
|
|
184
|
-
- **`initialDelaySeconds` on liveness is unnecessary**; the startupProbe plays
|
|
185
|
-
that role.
|
|
186
|
-
- Before a kill there is `max_age + periodSeconds × failureThreshold` = 45 + 90 =
|
|
187
|
-
**135 s** of confirmed silence. Slow on purpose: a worker is not
|
|
188
|
-
latency-critical, and a false restart costs more than two minutes of stalling.
|
|
189
|
-
|
|
190
|
-
## Required companion: an alert on consumers
|
|
191
|
-
|
|
192
|
-
The probe deliberately reports healthy while Bunny is reconnecting — otherwise a
|
|
193
|
-
broker hiccup would restart every replica at once. The price is that **broker
|
|
194
|
-
unavailability stops being visible automatically**. A pod can be consuming
|
|
195
|
-
nothing and look perfectly healthy.
|
|
196
|
-
|
|
197
|
-
So the probe needs an alert alongside it, and that alert should be in place
|
|
198
|
-
**before** the manifest is switched over:
|
|
102
|
+
The default heartbeat directory is `/opt/app/tmp/health`; the gem creates the
|
|
103
|
+
`health` subdirectory. The mount is also required with
|
|
104
|
+
`readOnlyRootFilesystem: true`.
|
|
199
105
|
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
# Not optional: the expression above cannot fire when the series is gone.
|
|
206
|
-
- alert: QueueConsumerMetricMissing
|
|
207
|
-
expr: absent_over_time(rabbitmq_detailed_queue_consumers{queue=~"myapp\..+"}[10m])
|
|
208
|
-
for: 5m
|
|
209
|
-
```
|
|
106
|
+
With the values above, confirmed silence lasts at most `45 + 30 × 3 = 135 s`
|
|
107
|
+
before Kubernetes kills the container. Size the startup budget from fresh-pod
|
|
108
|
+
measurements, not warm restarts. See [the full manifest and budget
|
|
109
|
+
arithmetic](docs/KUBERNETES.md).
|
|
210
110
|
|
|
211
|
-
|
|
212
|
-
without the exclusion the alert fires permanently until someone silences it.
|
|
213
|
-
Match them as a **substring**: under ActiveJob the full name looks like
|
|
214
|
-
`myapp.active_job.myapp.delayed_active_job:60`, so an anchored pattern never
|
|
215
|
-
matches.
|
|
111
|
+
## Configuration
|
|
216
112
|
|
|
217
|
-
|
|
218
|
-
recalled. The per-queue series is `rabbitmq_detailed_queue_consumers`, and it
|
|
219
|
-
only exists on `/metrics/detailed` **scraped with `family=queue_consumer_count`**
|
|
220
|
-
— bare, that endpoint serves no queue metrics at all, while plain `/metrics`
|
|
221
|
-
offers a same-sounding `rabbitmq_queue_consumers` that is a single label-less
|
|
222
|
-
cluster total, so a `{queue=~...}` matcher against it is quietly never true. And
|
|
223
|
-
the series **disappears** rather than going to zero when the queue is deleted or
|
|
224
|
-
the broker stops, which is why the second alert is there.
|
|
225
|
-
[docs/KUBERNETES.md](docs/KUBERNETES.md#getting-that-metric-at-all) has the
|
|
226
|
-
scrape config.
|
|
113
|
+
Everything is optional. Application settings belong in an initializer:
|
|
227
114
|
|
|
228
|
-
|
|
115
|
+
```ruby
|
|
116
|
+
KicksLiveness.configure do |config|
|
|
117
|
+
config.enabled = !Rails.env.local?
|
|
118
|
+
end
|
|
119
|
+
```
|
|
229
120
|
|
|
230
|
-
|
|
121
|
+
| Ruby setting | Default | Purpose | Valid values |
|
|
122
|
+
|---|---|---|---|
|
|
123
|
+
| `logger` | `Sneakers.logger`, resolved lazily | transition and error logs | logger-compatible object |
|
|
124
|
+
| `enabled` | `true` | start the monitor thread | `true` or `false` |
|
|
125
|
+
| `tick` | `10` | seconds between worker checks | positive number smaller than `max_age` |
|
|
126
|
+
| `startup_grace_ticks` | `6` | unhealthy startup ticks before one ERROR | positive integer |
|
|
231
127
|
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
| Bunny is recovering the connection | **healthy** — the broker heals itself |
|
|
236
|
-
| connection open, no consumers | unhealthy, a restart is the cure |
|
|
237
|
-
| not the whole set subscribed | unhealthy |
|
|
128
|
+
The separate probe cannot read an application initializer. The shared path and
|
|
129
|
+
freshness threshold therefore come from the environment; `tick` may use the
|
|
130
|
+
environment too:
|
|
238
131
|
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
132
|
+
| Environment variable | Default | Purpose | Invalid or blank value |
|
|
133
|
+
|---|---|---|---|
|
|
134
|
+
| `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | heartbeat directory; must be on tmpfs | uses the default |
|
|
135
|
+
| `KICKS_LIVENESS_MAX_AGE` | `45` | seconds before a mark is stale | uses the default |
|
|
136
|
+
| `KICKS_LIVENESS_TICK` | `10` | seconds between worker checks | uses the default, then the safety fallback below if needed |
|
|
242
137
|
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
138
|
+
If the effective environment tick is not smaller than `max_age`, the worker
|
|
139
|
+
writes a WARN to stderr and uses `10` seconds when that is safe, or half of
|
|
140
|
+
`max_age` otherwise. This keeps a ConfigMap mistake from stopping the worker
|
|
141
|
+
while guaranteeing that a healthy heartbeat cannot become stale between
|
|
142
|
+
ordinary ticks.
|
|
247
143
|
|
|
248
|
-
|
|
144
|
+
An initializer value for `tick` wins over `KICKS_LIVENESS_TICK`. `dir` and
|
|
145
|
+
`max_age` deliberately have no initializer setters: allowing two configuration
|
|
146
|
+
sources could make the worker and probe silently disagree.
|
|
249
147
|
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
148
|
+
If two runners share one pod — for example `sneakers:run` and
|
|
149
|
+
`sneakers:active_job` — give each a different `KICKS_LIVENESS_DIR`. Separate
|
|
150
|
+
pods already have separate `emptyDir` volumes.
|
|
253
151
|
|
|
254
|
-
|
|
255
|
-
broker cancels a consumer (on `consumer_timeout`, for example), it sends
|
|
256
|
-
`basic.cancel`, and Bunny removes the entry from its list — but it does so
|
|
257
|
-
through `@work_pool.submit`, that is, on the same pool that processes jobs
|
|
258
|
-
(`threads: 10` by default). Until a thread frees up, `any_consumers?` still
|
|
259
|
-
answers yes.
|
|
152
|
+
## What is healthy?
|
|
260
153
|
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
154
|
+
| Worker state | Probe verdict | Reason |
|
|
155
|
+
|---|---|---|
|
|
156
|
+
| every consumer subscribed, channel open | healthy | the process can consume work |
|
|
157
|
+
| Bunny recovering its connection | **healthy** | Bunny reconnects itself; restarting adds a reconnect storm |
|
|
158
|
+
| connection open, no consumers | unhealthy | a restart is the cure |
|
|
159
|
+
| only part of the expected worker set subscribed | unhealthy | one healthy worker must not hide a missing one |
|
|
160
|
+
|
|
161
|
+
The check reads Bunny objects already in the worker's memory. It never asks the
|
|
162
|
+
broker for its view.
|
|
264
163
|
|
|
265
|
-
|
|
266
|
-
consumer.** `Bunny::Channel#recover_cancelled_consumers!` is an opt-in method
|
|
267
|
-
that neither Kicks nor Sneakers enables. With it on, Bunny re-subscribes the
|
|
268
|
-
consumer itself on `basic.cancel` and **keeps** the entry in its list — so the
|
|
269
|
-
"consumer was cancelled" case stops being detectable at all. Do not enable it
|
|
270
|
-
together with this probe.
|
|
164
|
+
## Operational contract
|
|
271
165
|
|
|
272
|
-
|
|
273
|
-
|
|
166
|
+
- Start workers through `Sneakers::Runner`; that is where the monitor hook is
|
|
167
|
+
attached. `rake sneakers:run` already does this.
|
|
168
|
+
- Keep the `startupProbe`. With `RollingUpdate` and `maxUnavailable: 0`, it also
|
|
169
|
+
keeps the old pod until the new worker has subscribed.
|
|
170
|
+
- Add an external alert for queues with zero or missing consumers **before**
|
|
171
|
+
switching probes. Broker unavailability is invisible here by design; the
|
|
172
|
+
exact Prometheus rules are in [Running under Kubernetes](docs/KUBERNETES.md#required-companion-an-alert-on-consumers).
|
|
173
|
+
- Restart the runner after changing its process count; changing `workers` at
|
|
174
|
+
runtime is not supported.
|
|
274
175
|
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
catches exactly what an in-memory predicate cannot see.
|
|
176
|
+
The other boundaries — blocked worker pools, cancelled-consumer recovery, and
|
|
177
|
+
per-pod versus per-queue scope — are collected in [Limitations](docs/LIMITATIONS.md).
|
|
278
178
|
|
|
279
|
-
##
|
|
179
|
+
## Verify it
|
|
280
180
|
|
|
281
181
|
```bash
|
|
282
182
|
kubectl exec deploy/myapp -- ls -l /opt/app/tmp/health/
|
|
283
183
|
kubectl exec deploy/myapp -- sh -c 'bundle exec kicks-liveness; echo "exit=$?"'
|
|
284
|
-
kubectl logs deploy/myapp | grep liveness
|
|
285
184
|
```
|
|
286
185
|
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
```bash
|
|
291
|
-
kubectl exec deploy/myapp -- sh -c 'touch -d "2 minutes ago" /opt/app/tmp/health/worker-0; bundle exec kicks-liveness; echo "exit=$?"'
|
|
292
|
-
```
|
|
186
|
+
The healthy result is `N process(es) healthy` with exit code 0. Also test the
|
|
187
|
+
negative path before rollout; the commands and expected output are in
|
|
188
|
+
[Verifying the probe](docs/VERIFYING.md#2-a-stale-mark).
|
|
293
189
|
|
|
294
|
-
|
|
295
|
-
itself.
|
|
190
|
+
## Documentation
|
|
296
191
|
|
|
297
|
-
|
|
192
|
+
| Guide | Use it for |
|
|
193
|
+
|---|---|
|
|
194
|
+
| [Setup](docs/SETUP.md) | Rails, Sinatra, Hanami, Roda, standalone runners, and hook verification |
|
|
195
|
+
| [Design](docs/DESIGN.md) | health predicate, heartbeat files, hooks, configuration choices, and measurements |
|
|
196
|
+
| [Running under Kubernetes](docs/KUBERNETES.md) | full manifest, CPU/startup budgets, alerts, and live-pod verification |
|
|
197
|
+
| [Limitations](docs/LIMITATIONS.md) | cases the in-memory predicate intentionally cannot cover |
|
|
198
|
+
| [Verification scenarios](docs/VERIFYING.md) | eleven deliberately induced failures and their observed outcomes |
|
|
298
199
|
|
|
299
|
-
|
|
200
|
+
## Contributing
|
|
300
201
|
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
| started, with the effective settings | INFO |
|
|
304
|
-
| waiting for consumers | INFO |
|
|
305
|
-
| became healthy | INFO |
|
|
306
|
-
| shutting down | INFO |
|
|
307
|
-
| still not healthy after `startup_grace_ticks` ticks | ERROR |
|
|
308
|
-
| was healthy, became unhealthy | ERROR |
|
|
309
|
-
| a slot respawning without ever becoming healthy, once per grace window | ERROR |
|
|
310
|
-
| the monitor thread caught an exception | ERROR |
|
|
311
|
-
| the monitor could not be started at all | ERROR |
|
|
312
|
-
|
|
313
|
-
ERROR is reserved for genuine failure: a logger usually comes up with
|
|
314
|
-
`LOG_LEVEL` defaulting to `error`, so anything that matters must survive that
|
|
315
|
-
filter, while an ordinary deploy must not make noise. That is also why shutdown
|
|
316
|
-
is an event of its own: the workers unsubscribe while the thread pool drains,
|
|
317
|
-
and judged by the consumer count alone a normal deploy would look exactly like
|
|
318
|
-
a fault. From the moment shutdown begins the probe keeps the mark fresh
|
|
319
|
-
instead, so a slow drain gets the full `terminationGracePeriodSeconds`.
|
|
202
|
+
Setup, running the suite against both worker gems, and the release procedure are
|
|
203
|
+
in [CONTRIBUTING.md](https://github.com/PoroshkinaVV/kicks_liveness/blob/main/CONTRIBUTING.md).
|