kicks_liveness 0.1.1 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +18 -1
- data/README.md +9 -3
- data/docs/DESIGN.md +33 -12
- data/docs/KUBERNETES.md +21 -0
- data/docs/LIMITATIONS.md +40 -0
- data/docs/SETUP.md +6 -4
- data/docs/VERIFYING.md +41 -5
- data/lib/kicks_liveness/heartbeat.rb +102 -15
- data/lib/kicks_liveness/version.rb +1 -1
- data/lib/kicks_liveness.rb +2 -2
- metadata +4 -8
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: df61d00933097bb96e6d752e024e349ab631cd7c9d571cd9bff9ff63b8adda40
|
|
4
|
+
data.tar.gz: 78864373ca06cd998cdee192d064cd00ed69a4ecd73eb1b657474d77ab593066
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: '007091e18fea1b216ba25c634bab1a4770d25cc0f062ee30ba2885563075bc94e7662e58eebbac9507b448217508725612f172189b692e3a4d616e5f81fdd46b'
|
|
7
|
+
data.tar.gz: f460271ca68fdf912559b3fa2c41b02074d5eb457e946d2f985e3e2b0ac3da66dc94ce90c9d5639212a0fc7bb6537046654f31921cf6766fe15c903bd7dc04ef
|
data/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
|
|
|
5
5
|
|
|
6
6
|
## [Unreleased]
|
|
7
7
|
|
|
8
|
+
## [0.1.2] - 2026-09-11
|
|
9
|
+
|
|
10
|
+
### Fixed
|
|
11
|
+
|
|
12
|
+
- Reject heartbeat files inherited from a previous container incarnation. A
|
|
13
|
+
Kubernetes `emptyDir` survives container restarts, so a fresh mark left by a
|
|
14
|
+
formerly healthy worker could make the next container pass `startupProbe`
|
|
15
|
+
before its consumers had subscribed. Cache files elsewhere in the same
|
|
16
|
+
`emptyDir` remain untouched and available to warm the next application boot.
|
|
17
|
+
This protection assumes the default container-private PID namespace; pods
|
|
18
|
+
using `shareProcessNamespace` or `hostPID` retain the 0.1.1 freshness-only
|
|
19
|
+
behavior (see `docs/LIMITATIONS.md`).
|
|
20
|
+
The marks directory now contains `<dir>/generation`, and each
|
|
21
|
+
`worker-<slot>` mark carries the same value; tooling that enumerates the
|
|
22
|
+
directory should allow for this new metadata file.
|
|
23
|
+
|
|
8
24
|
## [0.1.1] - 2026-09-09
|
|
9
25
|
|
|
10
26
|
### Added
|
|
@@ -57,6 +73,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
|
|
|
57
73
|
[docs/](https://github.com/PoroshkinaVV/kicks_liveness/tree/main/docs); start
|
|
58
74
|
with `SETUP.md`, and read `LIMITATIONS.md` before relying on it.
|
|
59
75
|
|
|
60
|
-
[Unreleased]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.
|
|
76
|
+
[Unreleased]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.2...HEAD
|
|
77
|
+
[0.1.2]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.1...v0.1.2
|
|
61
78
|
[0.1.1]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.0...v0.1.1
|
|
62
79
|
[0.1.0]: https://github.com/PoroshkinaVV/kicks_liveness/releases/tag/v0.1.0
|
data/README.md
CHANGED
|
@@ -10,8 +10,14 @@
|
|
|
10
10
|
Rails or querying RabbitMQ from the probe.**
|
|
11
11
|
|
|
12
12
|
The worker checks its own Bunny consumers and publishes a heartbeat to tmpfs.
|
|
13
|
-
The probe only reads that heartbeat
|
|
14
|
-
no
|
|
13
|
+
The probe only reads that heartbeat, scoped to the current container
|
|
14
|
+
incarnation. No application boot, no network call, and no healthy replica
|
|
15
|
+
hiding a stalled one.
|
|
16
|
+
|
|
17
|
+
That restart scoping assumes Kubernetes' default container-private PID
|
|
18
|
+
namespace; see
|
|
19
|
+
[Limitations](docs/LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1)
|
|
20
|
+
for pods that set `shareProcessNamespace` or `hostPID`.
|
|
15
21
|
|
|
16
22
|
## Install: one line in Rails
|
|
17
23
|
|
|
@@ -189,7 +195,7 @@ negative path before rollout; the commands and expected output are in
|
|
|
189
195
|
| [Design](docs/DESIGN.md) | health predicate, heartbeat files, hooks, configuration choices, and measurements |
|
|
190
196
|
| [Running under Kubernetes](docs/KUBERNETES.md) | full manifest, CPU/startup budgets, alerts, and live-pod verification |
|
|
191
197
|
| [Limitations](docs/LIMITATIONS.md) | cases the in-memory predicate intentionally cannot cover |
|
|
192
|
-
| [Verification scenarios](docs/VERIFYING.md) |
|
|
198
|
+
| [Verification scenarios](docs/VERIFYING.md) | eleven deliberately induced failures and their observed outcomes |
|
|
193
199
|
|
|
194
200
|
## Contributing
|
|
195
201
|
|
data/docs/DESIGN.md
CHANGED
|
@@ -40,16 +40,17 @@ outside it can, so let the worker publish and let the probe read.
|
|
|
40
40
|
|
|
41
41
|
Every `tick` seconds (10 by default) the worker checks, **in its own memory**,
|
|
42
42
|
that its consumers are still subscribed, and touches a file on tmpfs. The probe
|
|
43
|
-
loads one dependency-free Ruby file,
|
|
44
|
-
1.
|
|
43
|
+
loads one dependency-free Ruby file, checks the marks' container generation and
|
|
44
|
+
mtimes, and exits with 0 or 1.
|
|
45
45
|
|
|
46
46
|
The probe therefore performs no network I/O and boots no framework, and the
|
|
47
|
-
state it reads — the mark
|
|
48
|
-
about the disk, though: starting the probe still loads the Ruby
|
|
49
|
-
two files of this gem from the image filesystem, and those reads
|
|
50
|
-
filesystem reads (usually served from page cache, but not
|
|
51
|
-
honest claim is not "no disk" but *no application boot,
|
|
52
|
-
that decides the answer* — which is what removes the
|
|
47
|
+
state it reads — the generation and mark mtimes — comes from procfs and tmpfs.
|
|
48
|
+
Be precise about the disk, though: starting the probe still loads the Ruby
|
|
49
|
+
interpreter and two files of this gem from the image filesystem, and those reads
|
|
50
|
+
are ordinary filesystem reads (usually served from page cache, but not
|
|
51
|
+
guaranteed to be). The honest claim is not "no disk" but *no application boot,
|
|
52
|
+
and no disk on the path that decides the answer* — which is what removes the
|
|
53
|
+
defects above, not tuning.
|
|
53
54
|
|
|
54
55
|
## The health predicate
|
|
55
56
|
|
|
@@ -112,6 +113,7 @@ silently. That is the thing to re-check when upgrading Bunny.
|
|
|
112
113
|
|
|
113
114
|
```
|
|
114
115
|
<dir>/expected how many forks the probe must wait for
|
|
116
|
+
<dir>/generation which container incarnation wrote this heartbeat
|
|
115
117
|
<dir>/worker-<slot> one per fork, refreshed every tick the fork is healthy
|
|
116
118
|
<dir>/attempt-<slot> starts of a slot that has not become healthy yet
|
|
117
119
|
```
|
|
@@ -127,6 +129,24 @@ it: if the directory is wiped, `touch!` brings the slot marks back while
|
|
|
127
129
|
rest of the pod's life. Rewriting it is also what lets a respawned set of forks
|
|
128
130
|
correct a count that has been lowered.
|
|
129
131
|
|
|
132
|
+
**`generation` closes a container-restart hole when the container owns PID 1.**
|
|
133
|
+
Kubernetes preserves an `emptyDir` when it restarts a container inside the same
|
|
134
|
+
pod. That is useful for an application cache elsewhere in the volume, but a
|
|
135
|
+
fresh heartbeat from the dead process must not let the new container pass its
|
|
136
|
+
one-shot `startupProbe`. On Linux, the worker and exec probe independently
|
|
137
|
+
derive the same incarnation from the container's mount namespace and PID 1
|
|
138
|
+
start time. Both `generation` and every slot mark carry it. Until the new
|
|
139
|
+
container declares itself and every current fork publishes its own mark, files
|
|
140
|
+
inherited from the previous container are rejected. Nothing outside the marks
|
|
141
|
+
directory is removed.
|
|
142
|
+
|
|
143
|
+
This guarantee assumes Kubernetes' default container-private PID namespace.
|
|
144
|
+
With `shareProcessNamespace: true` PID 1 belongs to the pod sandbox, and with
|
|
145
|
+
`hostPID: true` it is the node init process; neither restarts with the worker
|
|
146
|
+
container. Under either setting the guard can accept an inherited fresh mark.
|
|
147
|
+
Do not enable them on a pod whose startup probe relies on this guarantee; see
|
|
148
|
+
[LIMITATIONS.md](LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1).
|
|
149
|
+
|
|
130
150
|
**Files are named by supervisor slot, not by PID.** A fork killed with SIGKILL
|
|
131
151
|
is respawned into the same slot and overwrites its own file. Had the name
|
|
132
152
|
contained a PID, that file would sit there stale forever and the probe would
|
|
@@ -143,12 +163,13 @@ probe sees either the old value or the new one.
|
|
|
143
163
|
the single reason that the directory outlives the fork: a monitor caught in a
|
|
144
164
|
respawn loop is a brand-new object every few hundred milliseconds and can hold
|
|
145
165
|
no counter of its own. The file is removed once the slot becomes healthy, so in
|
|
146
|
-
steady state the directory holds
|
|
147
|
-
|
|
166
|
+
steady state the directory holds `expected`, `generation`, and the
|
|
167
|
+
`worker-<slot>` marks. What the counter is for is in
|
|
148
168
|
[LIMITATIONS.md](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn).
|
|
149
169
|
|
|
150
|
-
The
|
|
151
|
-
|
|
170
|
+
The timestamp, pid, and slot in `worker-<slot>` exist for a human running
|
|
171
|
+
`kubectl exec ... cat`. Its generation is part of the probe contract; freshness
|
|
172
|
+
still comes from mtime.
|
|
152
173
|
|
|
153
174
|
The directory must be on tmpfs — in Kubernetes, an `emptyDir` with
|
|
154
175
|
`medium: Memory`. Put it on a real disk and the probe starts depending on the
|
data/docs/KUBERNETES.md
CHANGED
|
@@ -46,6 +46,21 @@ Without `medium: Memory` an `emptyDir` is backed by the node's disk, and the
|
|
|
46
46
|
probe starts depending on the disk again — which is one of the things it exists
|
|
47
47
|
to avoid.
|
|
48
48
|
|
|
49
|
+
An `emptyDir` deliberately survives a restart of the container inside its pod.
|
|
50
|
+
That keeps a Bootsnap or other application cache under `/opt/app/tmp` warm for
|
|
51
|
+
the next attempt, but it also leaves the previous process's heartbeat files in
|
|
52
|
+
`health/`. The gem does not clear the volume or that directory. Instead it
|
|
53
|
+
records the current Linux container generation separately and in every slot
|
|
54
|
+
mark. A fresh mark from the previous container is rejected until every fork in
|
|
55
|
+
the current one has subscribed and published its own mark, so preserving the
|
|
56
|
+
cache cannot make `startupProbe` succeed early.
|
|
57
|
+
|
|
58
|
+
That guarantee assumes the default container-private PID namespace. Do not set
|
|
59
|
+
`shareProcessNamespace: true` or `hostPID: true` on a pod that relies on it: in
|
|
60
|
+
either topology PID 1 survives a worker-container restart, and the generation
|
|
61
|
+
guard can accept an inherited fresh mark. See
|
|
62
|
+
[LIMITATIONS.md](LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1).
|
|
63
|
+
|
|
49
64
|
## Why the command looks like that
|
|
50
65
|
|
|
51
66
|
**`bundle exec kicks-liveness` is the standard command.** It works regardless
|
|
@@ -104,6 +119,12 @@ executable is found regardless of `BUNDLE_PATH`, which is why it is the standard
|
|
|
104
119
|
command. Run bare, it depends on the gem's `bin` directory being on `PATH`,
|
|
105
120
|
which is not something to rely on in a manifest.
|
|
106
121
|
|
|
122
|
+
Keep the worker and probe on the same gem version. Since 0.1.2 the heartbeat
|
|
123
|
+
protocol includes a container generation; a newer probe correctly rejects the
|
|
124
|
+
generation-less files written by an older worker. `bundle exec` guarantees that
|
|
125
|
+
both sides resolve from the same bundle. If the faster form below installs a
|
|
126
|
+
second copy in `GEM_HOME`, rebuild that copy on every gem upgrade too.
|
|
127
|
+
|
|
107
128
|
### Where your image puts its gems
|
|
108
129
|
|
|
109
130
|
Before replacing the standard command with plain Ruby, run this against your
|
data/docs/LIMITATIONS.md
CHANGED
|
@@ -92,6 +92,37 @@ count and the retired slots eventually go stale. Neither direction is safe.
|
|
|
92
92
|
|
|
93
93
|
If you change `workers`, restart the runner. Runtime scaling is not supported.
|
|
94
94
|
|
|
95
|
+
## Container generations require Linux procfs and container-owned PID 1
|
|
96
|
+
|
|
97
|
+
Kubernetes keeps an `emptyDir` across restarts of a container in the same pod.
|
|
98
|
+
To prevent the next container from inheriting a fresh heartbeat, the worker and
|
|
99
|
+
exec probe independently identify their shared incarnation from Linux procfs:
|
|
100
|
+
the mount namespace plus PID 1 start time. No application cache is removed.
|
|
101
|
+
|
|
102
|
+
The identifier also assumes that the container owns PID 1. With
|
|
103
|
+
`shareProcessNamespace: true`, PID 1 is the pod sandbox, and with
|
|
104
|
+
`hostPID: true` it is the node's init process; neither restarts when the worker
|
|
105
|
+
container does, so the start-time half of the identifier stays constant. The
|
|
106
|
+
mount namespace inode is then the only remaining signal, and the kernel
|
|
107
|
+
normally hands the just-released inode back to the replacement container in an
|
|
108
|
+
otherwise quiet pod. Under either setting the guard silently degrades to the
|
|
109
|
+
0.1.1 freshness-only behavior and can accept a fresh mark from the previous
|
|
110
|
+
container. Do not enable either setting on a pod whose `startupProbe` relies on
|
|
111
|
+
this guarantee.
|
|
112
|
+
|
|
113
|
+
If procfs is unavailable — for example, when using the gem outside a Linux
|
|
114
|
+
container — generation detection falls back to the original freshness-only
|
|
115
|
+
check. The worker and probe still function, but they cannot distinguish a fresh
|
|
116
|
+
mark left by a previous process from one written by the current process. Linux
|
|
117
|
+
Kubernetes, Docker, and Nomad containers expose the required procfs entries
|
|
118
|
+
under their normal configuration.
|
|
119
|
+
|
|
120
|
+
The worker and its exec probe must run in the **same container**. A neighbouring
|
|
121
|
+
sidecar can mount the same `emptyDir`, but it has a different mount namespace
|
|
122
|
+
and therefore treats the worker's marks as belonging to another container.
|
|
123
|
+
Kubernetes exec probes already run inside the container they check; do not move
|
|
124
|
+
`kicks-liveness` into a separate health sidecar.
|
|
125
|
+
|
|
95
126
|
## A respawn loop is reported once per grace window, not once per respawn
|
|
96
127
|
|
|
97
128
|
When a fork cannot subscribe at all, the supervisor brings it back after
|
|
@@ -112,6 +143,15 @@ reports the elapsed time and the number of starts:
|
|
|
112
143
|
The number of starts is the diagnosis: it separates a slow start from a respawn
|
|
113
144
|
loop at a glance.
|
|
114
145
|
|
|
146
|
+
This unhealthy-run counter is scoped to the marks volume, not to the container
|
|
147
|
+
generation. If a container is restarted before its slot has ever become
|
|
148
|
+
healthy, the replacement continues the same count: its repeated `started:` line
|
|
149
|
+
is suppressed, and an expired grace window may immediately report an ERROR that
|
|
150
|
+
includes starts from the previous container. This is diagnostic state only; it
|
|
151
|
+
does not participate in the probe result, and the first healthy tick removes
|
|
152
|
+
it. Treat the count as "starts since this slot was last healthy in this pod",
|
|
153
|
+
not "starts in this container".
|
|
154
|
+
|
|
115
155
|
## Do not install both `kicks` and `sneakers`
|
|
116
156
|
|
|
117
157
|
The gem declares neither as a dependency, because at runtime it needs only the
|
data/docs/SETUP.md
CHANGED
|
@@ -174,11 +174,13 @@ Once workers are running, the marks directory is the other half of the answer:
|
|
|
174
174
|
```
|
|
175
175
|
$ ls -l /opt/app/tmp/health/
|
|
176
176
|
expected
|
|
177
|
+
generation
|
|
177
178
|
worker-0
|
|
178
179
|
worker-1
|
|
179
180
|
```
|
|
180
181
|
|
|
181
|
-
One `worker-<slot>` file per fork, plus `expected
|
|
182
|
-
the slot files are not, the workers
|
|
183
|
-
|
|
184
|
-
which is removed as soon as
|
|
182
|
+
One `worker-<slot>` file per fork, plus `expected` and the current container
|
|
183
|
+
`generation`. If `expected` is there and the slot files are not, the workers
|
|
184
|
+
have not finished subscribing. A slot that keeps restarting without ever
|
|
185
|
+
subscribing also leaves an `attempt-<slot>` file, which is removed as soon as
|
|
186
|
+
that slot becomes healthy.
|
data/docs/VERIFYING.md
CHANGED
|
@@ -97,11 +97,15 @@ spec/integration/verify.sh probe
|
|
|
97
97
|
INFO: [liveness] slot 0: started: dir=/opt/app/tmp/health max_age=45s tick=10s processes=1 consumers=2
|
|
98
98
|
INFO: [liveness] slot 0: waiting for 2 consumers
|
|
99
99
|
INFO: [liveness] slot 0: healthy # one tick later
|
|
100
|
-
expected worker-0
|
|
100
|
+
expected generation worker-0 # in the marks directory
|
|
101
101
|
1 process(es) healthy # probe, exit 0
|
|
102
102
|
```
|
|
103
103
|
|
|
104
|
-
The hooks fired inside a real ServerEngine fork, which no double can show.
|
|
104
|
+
The hooks fired inside a real ServerEngine fork, which no double can show. On a
|
|
105
|
+
fresh fixture the first attempt also prints `started:`. Do not use that line as
|
|
106
|
+
the sole proof after a same-pod container restart: an inherited unhealthy-run
|
|
107
|
+
counter deliberately suppresses repeated start lines, as described in
|
|
108
|
+
[LIMITATIONS.md](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn).
|
|
105
109
|
|
|
106
110
|
### 2. A stale mark
|
|
107
111
|
|
|
@@ -242,9 +246,9 @@ find them.
|
|
|
242
246
|
```
|
|
243
247
|
INFO: [liveness] slot 0: started: ... processes=2 consumers=2
|
|
244
248
|
INFO: [liveness] slot 1: started: ... processes=2 consumers=2
|
|
245
|
-
expected worker-0 worker-1
|
|
246
|
-
2 process(es) healthy
|
|
247
|
-
worker-1 stale 120s > 45s
|
|
249
|
+
expected generation worker-0 worker-1 # expected contains "2"
|
|
250
|
+
2 process(es) healthy # exit 0
|
|
251
|
+
worker-1 stale 120s > 45s # exit 1 — staling either mark is enough
|
|
248
252
|
```
|
|
249
253
|
|
|
250
254
|
Both marks are required, not just the first.
|
|
@@ -331,6 +335,38 @@ pod is restarted. That is why the documented rule set pairs it with
|
|
|
331
335
|
`absent_over_time`, and why an `up == 0` on the scrape job is worth having
|
|
332
336
|
beside both.
|
|
333
337
|
|
|
338
|
+
### 11. A private-PID container restart cannot inherit a healthy mark
|
|
339
|
+
|
|
340
|
+
This scenario needs the fixture image but not Kubernetes or RabbitMQ. It starts
|
|
341
|
+
a container that publishes a healthy mark into a named volume, restarts that
|
|
342
|
+
same container, and delays the new writer for 15 seconds:
|
|
343
|
+
|
|
344
|
+
```bash
|
|
345
|
+
spec/integration/verify.sh build
|
|
346
|
+
spec/integration/verify.sh generation-restart
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
During the delay the inherited mark is still fresh, but the probe must reject
|
|
350
|
+
its old generation. Once the replacement writer publishes its own mark, the
|
|
351
|
+
same probe must become healthy again. The command checks both transitions and
|
|
352
|
+
prints output in this form:
|
|
353
|
+
|
|
354
|
+
```
|
|
355
|
+
before=mnt:[4026532686]:123456
|
|
356
|
+
after=mnt:[4026532686]:123789
|
|
357
|
+
inherited=heartbeat belongs to a previous container: worker has not started yet
|
|
358
|
+
recovered=1 process(es) healthy
|
|
359
|
+
```
|
|
360
|
+
|
|
361
|
+
The namespace inode is allowed to be reused, as in the example above; the PID 1
|
|
362
|
+
start time still distinguishes the two incarnations. The test removes its
|
|
363
|
+
container and named volume on exit.
|
|
364
|
+
|
|
365
|
+
This scenario covers the default container-private PID namespace. With
|
|
366
|
+
`shareProcessNamespace: true` or `hostPID: true`, PID 1 survives the worker
|
|
367
|
+
container restart and this guarantee does not hold; see
|
|
368
|
+
[LIMITATIONS.md](LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1).
|
|
369
|
+
|
|
334
370
|
## What to do with a disagreement
|
|
335
371
|
|
|
336
372
|
Record it, then fix whichever is wrong — the code or the document. The numbers
|
|
@@ -1,5 +1,57 @@
|
|
|
1
1
|
module KicksLiveness
|
|
2
|
-
#
|
|
2
|
+
# Identifies an ordinary Linux container incarnation shared by the worker and
|
|
3
|
+
# exec probes without writing anything to the marks directory. This assumes
|
|
4
|
+
# the container owns PID 1; shared and host PID namespaces are documented as
|
|
5
|
+
# unsupported in +docs/LIMITATIONS.md+.
|
|
6
|
+
# @api private
|
|
7
|
+
module ContainerGeneration
|
|
8
|
+
module_function
|
|
9
|
+
|
|
10
|
+
# @return [String, nil] current private-PID Linux container incarnation, or
|
|
11
|
+
# nil when procfs does not expose one
|
|
12
|
+
# @api private
|
|
13
|
+
def current
|
|
14
|
+
mount_namespace = File.readlink('/proc/self/ns/mnt')
|
|
15
|
+
stat = File.read('/proc/1/stat')
|
|
16
|
+
closing_parenthesis = stat.rindex(') ')
|
|
17
|
+
return unless closing_parenthesis
|
|
18
|
+
|
|
19
|
+
# After the command in parentheses, field 3 (`state`) is index 0. Process
|
|
20
|
+
# start time is field 22, therefore index 19 in this tail.
|
|
21
|
+
started_at = stat[(closing_parenthesis + 2)..].split[19]
|
|
22
|
+
return unless started_at && Integer(started_at).positive?
|
|
23
|
+
|
|
24
|
+
"#{mount_namespace}:#{started_at}"
|
|
25
|
+
rescue StandardError
|
|
26
|
+
nil
|
|
27
|
+
end
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
# Generation-specific parts of the filesystem contract.
|
|
31
|
+
# @api private
|
|
32
|
+
module GenerationGuard
|
|
33
|
+
private
|
|
34
|
+
|
|
35
|
+
def generation_path
|
|
36
|
+
File.join(@dir, 'generation')
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
def declared_generation
|
|
40
|
+
File.read(generation_path)
|
|
41
|
+
rescue StandardError
|
|
42
|
+
nil
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
def current_generation?
|
|
46
|
+
!@generation || declared_generation == @generation
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
def previous_generation?(path)
|
|
50
|
+
@generation && File.read(path)[/\bgeneration=(\S+)/, 1] != @generation
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
# The liveness marks on the filesystem: written by the worker, read by the
|
|
3
55
|
# probe.
|
|
4
56
|
#
|
|
5
57
|
# The directory must live on tmpfs — in Kubernetes, an emptyDir with
|
|
@@ -17,6 +69,8 @@ module KicksLiveness
|
|
|
17
69
|
#
|
|
18
70
|
# @see file:docs/DESIGN.md#why-the-heartbeat-file-has-no-require-of-its-own
|
|
19
71
|
class Heartbeat
|
|
72
|
+
include GenerationGuard
|
|
73
|
+
|
|
20
74
|
# @return [String] marks directory used when the environment says nothing
|
|
21
75
|
DEFAULT_DIR = '/opt/app/tmp/health'.freeze
|
|
22
76
|
# @return [Integer] seconds after which a mark is stale, by default
|
|
@@ -29,6 +83,16 @@ module KicksLiveness
|
|
|
29
83
|
tick: 'KICKS_LIVENESS_TICK'
|
|
30
84
|
}.freeze
|
|
31
85
|
|
|
86
|
+
# Linux exposes a stable identifier shared by a private-PID container and
|
|
87
|
+
# its exec probes. A restarted container gets a new identifier even though
|
|
88
|
+
# its Kubernetes emptyDir survives.
|
|
89
|
+
# @return [String, nil] current private-PID container incarnation, or nil
|
|
90
|
+
# off Linux
|
|
91
|
+
# @api private
|
|
92
|
+
def self.container_generation
|
|
93
|
+
ContainerGeneration.current
|
|
94
|
+
end
|
|
95
|
+
|
|
32
96
|
# An empty string counts as unset: in a ConfigMap that is what you get by
|
|
33
97
|
# declaring a key and leaving it blank.
|
|
34
98
|
#
|
|
@@ -72,9 +136,15 @@ module KicksLiveness
|
|
|
72
136
|
|
|
73
137
|
# @param dir [String] marks directory
|
|
74
138
|
# @param max_age [Integer] seconds after which a mark is considered stale
|
|
75
|
-
|
|
139
|
+
# @param generation [String, nil] container incarnation; injected in specs
|
|
140
|
+
def initialize(
|
|
141
|
+
dir: Heartbeat.env_dir,
|
|
142
|
+
max_age: Heartbeat.env_max_age,
|
|
143
|
+
generation: Heartbeat.container_generation
|
|
144
|
+
)
|
|
76
145
|
@dir = dir
|
|
77
146
|
@max_age = max_age
|
|
147
|
+
@generation = generation
|
|
78
148
|
end
|
|
79
149
|
|
|
80
150
|
attr_reader :dir, :max_age
|
|
@@ -93,10 +163,8 @@ module KicksLiveness
|
|
|
93
163
|
# @return [void]
|
|
94
164
|
def declare!(processes)
|
|
95
165
|
make_dir
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
File.write(tmp, processes)
|
|
99
|
-
File.rename(tmp, expected_path)
|
|
166
|
+
atomic_write(generation_path, @generation) if @generation
|
|
167
|
+
atomic_write(expected_path, processes)
|
|
100
168
|
end
|
|
101
169
|
|
|
102
170
|
# Refreshes this fork's mark.
|
|
@@ -106,14 +174,18 @@ module KicksLiveness
|
|
|
106
174
|
# PID in the name that file would stay stale forever and the probe would fail
|
|
107
175
|
# permanently.
|
|
108
176
|
#
|
|
109
|
-
# The
|
|
110
|
-
#
|
|
177
|
+
# The timestamp, pid and slot exist for a human running
|
|
178
|
+
# <tt>kubectl exec ... cat</tt>. The generation is also checked by the probe:
|
|
179
|
+
# a Kubernetes emptyDir survives a container restart, so freshness alone
|
|
180
|
+
# cannot distinguish this process from the one that just exited.
|
|
111
181
|
#
|
|
112
182
|
# @param slot [Integer] supervisor slot of this fork
|
|
113
183
|
# @return [Integer] bytes written
|
|
114
184
|
def touch!(slot)
|
|
115
185
|
make_dir
|
|
116
|
-
|
|
186
|
+
contents = "#{Time.now.utc.strftime('%FT%TZ')} pid=#{Process.pid} slot=#{slot}"
|
|
187
|
+
contents = "#{contents} generation=#{@generation}" if @generation
|
|
188
|
+
atomic_write(slot_path(slot), "#{contents}\n")
|
|
117
189
|
end
|
|
118
190
|
|
|
119
191
|
# The probe side: is every declared fork's mark present and fresh?
|
|
@@ -126,13 +198,9 @@ module KicksLiveness
|
|
|
126
198
|
def check(now: Time.now.utc)
|
|
127
199
|
processes = expected
|
|
128
200
|
return [false, "no #{expected_path}: worker has not started yet"] unless processes&.positive?
|
|
201
|
+
return [false, 'heartbeat belongs to a previous container: worker has not started yet'] unless current_generation?
|
|
129
202
|
|
|
130
|
-
problems = (0...processes).filter_map
|
|
131
|
-
age = age_of(slot_path(slot), now)
|
|
132
|
-
next "worker-#{slot} missing" if age.nil?
|
|
133
|
-
|
|
134
|
-
"worker-#{slot} stale #{age.round}s > #{@max_age}s" if age > @max_age
|
|
135
|
-
end
|
|
203
|
+
problems = (0...processes).filter_map { |slot| problem_for(slot, now) }
|
|
136
204
|
|
|
137
205
|
problems.empty? ? [true, "#{processes} process(es) healthy"] : [false, problems.join('; ')]
|
|
138
206
|
end
|
|
@@ -169,6 +237,25 @@ module KicksLiveness
|
|
|
169
237
|
nil
|
|
170
238
|
end
|
|
171
239
|
|
|
240
|
+
def problem_for(slot, now)
|
|
241
|
+
path = slot_path(slot)
|
|
242
|
+
age = age_of(path, now)
|
|
243
|
+
return "worker-#{slot} missing" if age.nil?
|
|
244
|
+
return "worker-#{slot} belongs to a previous container" if previous_generation?(path)
|
|
245
|
+
|
|
246
|
+
"worker-#{slot} stale #{age.round}s > #{@max_age}s" if age > @max_age
|
|
247
|
+
rescue StandardError
|
|
248
|
+
"worker-#{slot} unreadable"
|
|
249
|
+
end
|
|
250
|
+
|
|
251
|
+
def atomic_write(path, contents)
|
|
252
|
+
# The pid keeps concurrent forks from sharing the temporary file.
|
|
253
|
+
tmp = "#{path}.#{Process.pid}"
|
|
254
|
+
bytes = File.write(tmp, contents)
|
|
255
|
+
File.rename(tmp, path)
|
|
256
|
+
bytes
|
|
257
|
+
end
|
|
258
|
+
|
|
172
259
|
def age_of(path, now)
|
|
173
260
|
now - File.mtime(path)
|
|
174
261
|
rescue StandardError
|
data/lib/kicks_liveness.rb
CHANGED
|
@@ -9,8 +9,8 @@ require_relative 'kicks_liveness/hooks'
|
|
|
9
9
|
# Liveness probe for Kicks and Sneakers workers, backed by a tmpfs heartbeat.
|
|
10
10
|
#
|
|
11
11
|
# The worker publishes a mark from inside its own process, checking its Bunny
|
|
12
|
-
# consumers in memory; the probe reads only the
|
|
13
|
-
# to the broker.
|
|
12
|
+
# consumers in memory; the probe reads only the container generation and mark
|
|
13
|
+
# mtimes. No Rails, no call to the broker.
|
|
14
14
|
#
|
|
15
15
|
# @see file:docs/SETUP.md
|
|
16
16
|
# @see file:docs/DESIGN.md
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: kicks_liveness
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.2
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- PoroshkinaVV
|
|
@@ -11,12 +11,8 @@ date: 1980-01-02 00:00:00.000000000 Z
|
|
|
11
11
|
dependencies: []
|
|
12
12
|
description: |
|
|
13
13
|
A liveness probe for RabbitMQ worker pods that loads no Rails and never talks
|
|
14
|
-
to the broker.
|
|
15
|
-
|
|
16
|
-
file's mtime and runs as `bundle exec kicks-liveness`. It loads no application
|
|
17
|
-
code; aside from Bundler and the interpreter it loads only the gem's small
|
|
18
|
-
probe files, while the state itself comes from tmpfs. A broker hiccup cannot
|
|
19
|
-
restart every replica at once.
|
|
14
|
+
to the broker. Workers check their Bunny consumers in memory and publish a
|
|
15
|
+
heartbeat to tmpfs; the probe only reads that heartbeat.
|
|
20
16
|
email:
|
|
21
17
|
- lera.poroshkina@mail.ru
|
|
22
18
|
executables:
|
|
@@ -49,7 +45,7 @@ licenses:
|
|
|
49
45
|
- MIT
|
|
50
46
|
metadata:
|
|
51
47
|
source_code_uri: https://github.com/PoroshkinaVV/kicks_liveness
|
|
52
|
-
documentation_uri: https://rubydoc.info/gems/kicks_liveness/0.1.
|
|
48
|
+
documentation_uri: https://rubydoc.info/gems/kicks_liveness/0.1.2
|
|
53
49
|
changelog_uri: https://github.com/PoroshkinaVV/kicks_liveness/blob/main/CHANGELOG.md
|
|
54
50
|
bug_tracker_uri: https://github.com/PoroshkinaVV/kicks_liveness/issues
|
|
55
51
|
rubygems_mfa_required: 'true'
|