kicks_liveness 0.1.0 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.yardopts +13 -0
- data/CHANGELOG.md +58 -0
- data/README.md +133 -249
- data/docs/DESIGN.md +51 -16
- data/docs/KUBERNETES.md +32 -9
- data/docs/LIMITATIONS.md +54 -15
- data/docs/SETUP.md +16 -11
- data/docs/VERIFYING.md +54 -9
- data/lib/kicks_liveness/configuration.rb +73 -10
- data/lib/kicks_liveness/heartbeat.rb +102 -15
- data/lib/kicks_liveness/hooks.rb +12 -1
- data/lib/kicks_liveness/monitor.rb +6 -0
- data/lib/kicks_liveness/version.rb +1 -1
- data/lib/kicks_liveness.rb +15 -3
- metadata +5 -8
data/docs/DESIGN.md
CHANGED
|
@@ -40,16 +40,17 @@ outside it can, so let the worker publish and let the probe read.
|
|
|
40
40
|
|
|
41
41
|
Every `tick` seconds (10 by default) the worker checks, **in its own memory**,
|
|
42
42
|
that its consumers are still subscribed, and touches a file on tmpfs. The probe
|
|
43
|
-
loads one dependency-free Ruby file,
|
|
44
|
-
1.
|
|
43
|
+
loads one dependency-free Ruby file, checks the marks' container generation and
|
|
44
|
+
mtimes, and exits with 0 or 1.
|
|
45
45
|
|
|
46
46
|
The probe therefore performs no network I/O and boots no framework, and the
|
|
47
|
-
state it reads — the mark
|
|
48
|
-
about the disk, though: starting the probe still loads the Ruby
|
|
49
|
-
two files of this gem from the image filesystem, and those reads
|
|
50
|
-
filesystem reads (usually served from page cache, but not
|
|
51
|
-
honest claim is not "no disk" but *no application boot,
|
|
52
|
-
that decides the answer* — which is what removes the
|
|
47
|
+
state it reads — the generation and mark mtimes — comes from procfs and tmpfs.
|
|
48
|
+
Be precise about the disk, though: starting the probe still loads the Ruby
|
|
49
|
+
interpreter and two files of this gem from the image filesystem, and those reads
|
|
50
|
+
are ordinary filesystem reads (usually served from page cache, but not
|
|
51
|
+
guaranteed to be). The honest claim is not "no disk" but *no application boot,
|
|
52
|
+
and no disk on the path that decides the answer* — which is what removes the
|
|
53
|
+
defects above, not tuning.
|
|
53
54
|
|
|
54
55
|
## The health predicate
|
|
55
56
|
|
|
@@ -112,6 +113,7 @@ silently. That is the thing to re-check when upgrading Bunny.
|
|
|
112
113
|
|
|
113
114
|
```
|
|
114
115
|
<dir>/expected how many forks the probe must wait for
|
|
116
|
+
<dir>/generation which container incarnation wrote this heartbeat
|
|
115
117
|
<dir>/worker-<slot> one per fork, refreshed every tick the fork is healthy
|
|
116
118
|
<dir>/attempt-<slot> starts of a slot that has not become healthy yet
|
|
117
119
|
```
|
|
@@ -127,6 +129,24 @@ it: if the directory is wiped, `touch!` brings the slot marks back while
|
|
|
127
129
|
rest of the pod's life. Rewriting it is also what lets a respawned set of forks
|
|
128
130
|
correct a count that has been lowered.
|
|
129
131
|
|
|
132
|
+
**`generation` closes a container-restart hole when the container owns PID 1.**
|
|
133
|
+
Kubernetes preserves an `emptyDir` when it restarts a container inside the same
|
|
134
|
+
pod. That is useful for an application cache elsewhere in the volume, but a
|
|
135
|
+
fresh heartbeat from the dead process must not let the new container pass its
|
|
136
|
+
one-shot `startupProbe`. On Linux, the worker and exec probe independently
|
|
137
|
+
derive the same incarnation from the container's mount namespace and PID 1
|
|
138
|
+
start time. Both `generation` and every slot mark carry it. Until the new
|
|
139
|
+
container declares itself and every current fork publishes its own mark, files
|
|
140
|
+
inherited from the previous container are rejected. Nothing outside the marks
|
|
141
|
+
directory is removed.
|
|
142
|
+
|
|
143
|
+
This guarantee assumes Kubernetes' default container-private PID namespace.
|
|
144
|
+
With `shareProcessNamespace: true` PID 1 belongs to the pod sandbox, and with
|
|
145
|
+
`hostPID: true` it is the node init process; neither restarts with the worker
|
|
146
|
+
container. Under either setting the guard can accept an inherited fresh mark.
|
|
147
|
+
Do not enable them on a pod whose startup probe relies on this guarantee; see
|
|
148
|
+
[LIMITATIONS.md](LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1).
|
|
149
|
+
|
|
130
150
|
**Files are named by supervisor slot, not by PID.** A fork killed with SIGKILL
|
|
131
151
|
is respawned into the same slot and overwrites its own file. Had the name
|
|
132
152
|
contained a PID, that file would sit there stale forever and the probe would
|
|
@@ -143,12 +163,13 @@ probe sees either the old value or the new one.
|
|
|
143
163
|
the single reason that the directory outlives the fork: a monitor caught in a
|
|
144
164
|
respawn loop is a brand-new object every few hundred milliseconds and can hold
|
|
145
165
|
no counter of its own. The file is removed once the slot becomes healthy, so in
|
|
146
|
-
steady state the directory holds
|
|
147
|
-
|
|
166
|
+
steady state the directory holds `expected`, `generation`, and the
|
|
167
|
+
`worker-<slot>` marks. What the counter is for is in
|
|
148
168
|
[LIMITATIONS.md](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn).
|
|
149
169
|
|
|
150
|
-
The
|
|
151
|
-
|
|
170
|
+
The timestamp, pid, and slot in `worker-<slot>` exist for a human running
|
|
171
|
+
`kubectl exec ... cat`. Its generation is part of the probe contract; freshness
|
|
172
|
+
still comes from mtime.
|
|
152
173
|
|
|
153
174
|
The directory must be on tmpfs — in Kubernetes, an `emptyDir` with
|
|
154
175
|
`medium: Memory`. Put it on a real disk and the probe starts depending on the
|
|
@@ -249,7 +270,7 @@ check.
|
|
|
249
270
|
|
|
250
271
|
## What is configurable, and where
|
|
251
272
|
|
|
252
|
-
| | Where | Why |
|
|
273
|
+
| Setting | Where | Why |
|
|
253
274
|
|---|---|---|
|
|
254
275
|
| `logger`, `enabled`, `startup_grace_ticks` | application config block | only the worker needs them |
|
|
255
276
|
| `tick` | either, and the config block wins | only the worker reads it, so two sources cannot contradict each other |
|
|
@@ -281,13 +302,27 @@ makes every mark stale on arrival, so the probe can never pass again. They fall
|
|
|
281
302
|
back to the default too.
|
|
282
303
|
|
|
283
304
|
Setting `tick` from the application is held to a stricter standard: a
|
|
284
|
-
non-positive value raises `ArgumentError
|
|
285
|
-
|
|
286
|
-
|
|
305
|
+
non-positive value raises `ArgumentError`, as does a value greater than or equal
|
|
306
|
+
to `max_age`: such a monitor would inevitably let a healthy mark go stale. The
|
|
307
|
+
environment gets a silent fallback for values that cannot be parsed. If its
|
|
308
|
+
effective tick is greater than or equal to `max_age`, configuration emits a
|
|
309
|
+
warning on stderr and uses the default tick when it is safe, or half of `max_age`
|
|
310
|
+
otherwise. A ConfigMap typo must not bring a worker down, whereas an initializer
|
|
311
|
+
is code and should fail loudly at boot, where the developer is looking.
|
|
312
|
+
|
|
313
|
+
`startup_grace_ticks` must be a positive integer. It is compared directly with
|
|
314
|
+
an integer counter; accepting zero, a float, or a string would silently disable
|
|
315
|
+
the escalation that is supposed to diagnose a worker that never subscribes.
|
|
287
316
|
|
|
288
317
|
The logger defaults to `Sneakers.logger` but is resolved lazily, because at the
|
|
289
318
|
time the configuration object is built it may not be set up yet.
|
|
290
319
|
|
|
320
|
+
Logging is diagnostic; the heartbeat is the liveness contract. An exception
|
|
321
|
+
raised by a custom logger is therefore swallowed at the logging boundary. It
|
|
322
|
+
cannot prevent a healthy tick from writing its mark, abort monitor startup, or
|
|
323
|
+
escape the loop's error handler and kill the monitor thread. No fallback message
|
|
324
|
+
is attempted through the same broken logger.
|
|
325
|
+
|
|
291
326
|
## Logging: events, not the pulse
|
|
292
327
|
|
|
293
328
|
The pulse lives in the mtime of a file; writing a log line every tick would only
|
data/docs/KUBERNETES.md
CHANGED
|
@@ -46,6 +46,21 @@ Without `medium: Memory` an `emptyDir` is backed by the node's disk, and the
|
|
|
46
46
|
probe starts depending on the disk again — which is one of the things it exists
|
|
47
47
|
to avoid.
|
|
48
48
|
|
|
49
|
+
An `emptyDir` deliberately survives a restart of the container inside its pod.
|
|
50
|
+
That keeps a Bootsnap or other application cache under `/opt/app/tmp` warm for
|
|
51
|
+
the next attempt, but it also leaves the previous process's heartbeat files in
|
|
52
|
+
`health/`. The gem does not clear the volume or that directory. Instead it
|
|
53
|
+
records the current Linux container generation separately and in every slot
|
|
54
|
+
mark. A fresh mark from the previous container is rejected until every fork in
|
|
55
|
+
the current one has subscribed and published its own mark, so preserving the
|
|
56
|
+
cache cannot make `startupProbe` succeed early.
|
|
57
|
+
|
|
58
|
+
That guarantee assumes the default container-private PID namespace. Do not set
|
|
59
|
+
`shareProcessNamespace: true` or `hostPID: true` on a pod that relies on it: in
|
|
60
|
+
either topology PID 1 survives a worker-container restart, and the generation
|
|
61
|
+
guard can accept an inherited fresh mark. See
|
|
62
|
+
[LIMITATIONS.md](LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1).
|
|
63
|
+
|
|
49
64
|
## Why the command looks like that
|
|
50
65
|
|
|
51
66
|
**`bundle exec kicks-liveness` is the standard command.** It works regardless
|
|
@@ -104,6 +119,12 @@ executable is found regardless of `BUNDLE_PATH`, which is why it is the standard
|
|
|
104
119
|
command. Run bare, it depends on the gem's `bin` directory being on `PATH`,
|
|
105
120
|
which is not something to rely on in a manifest.
|
|
106
121
|
|
|
122
|
+
Keep the worker and probe on the same gem version. Since 0.1.2 the heartbeat
|
|
123
|
+
protocol includes a container generation; a newer probe correctly rejects the
|
|
124
|
+
generation-less files written by an older worker. `bundle exec` guarantees that
|
|
125
|
+
both sides resolve from the same bundle. If the faster form below installs a
|
|
126
|
+
second copy in `GEM_HOME`, rebuild that copy on every gem upgrade too.
|
|
127
|
+
|
|
107
128
|
### Where your image puts its gems
|
|
108
129
|
|
|
109
130
|
Before replacing the standard command with plain Ruby, run this against your
|
|
@@ -286,15 +307,18 @@ redelivered.
|
|
|
286
307
|
|
|
287
308
|
## Environment variables
|
|
288
309
|
|
|
289
|
-
| Variable | Default | |
|
|
290
|
-
|
|
291
|
-
| `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | marks directory, **must be on tmpfs** |
|
|
292
|
-
| `KICKS_LIVENESS_MAX_AGE` | `45` | seconds after which a mark is stale |
|
|
293
|
-
| `KICKS_LIVENESS_TICK` | `10` | interval between ticks |
|
|
310
|
+
| Variable | Default | Purpose | Invalid or blank value |
|
|
311
|
+
|---|---|---|---|
|
|
312
|
+
| `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | marks directory, **must be on tmpfs** | uses the default |
|
|
313
|
+
| `KICKS_LIVENESS_MAX_AGE` | `45` | seconds after which a mark is stale | uses the default |
|
|
314
|
+
| `KICKS_LIVENESS_TICK` | `10` | interval between ticks | uses the default, then the safety fallback below if needed |
|
|
294
315
|
|
|
295
316
|
Keep `tick` well below `max_age`. A tick longer than half of `max_age` leaves no
|
|
296
317
|
room for a single missed write, and a tick longer than `max_age` guarantees a
|
|
297
|
-
restart loop.
|
|
318
|
+
restart loop. If environment values produce `tick >= max_age`, the worker writes
|
|
319
|
+
a WARN to stderr and uses the default tick when that is safe, or half of
|
|
320
|
+
`max_age` otherwise. An initializer value with the same mismatch raises
|
|
321
|
+
`ArgumentError`. The half-threshold remains the recommended operational margin.
|
|
298
322
|
|
|
299
323
|
`KICKS_LIVENESS_DIR` is also what separates two runners that share a pod. An
|
|
300
324
|
application running both `rake sneakers:run` and `rake sneakers:active_job` has
|
|
@@ -315,9 +339,8 @@ probe command pointed at its own directory — the probe reads
|
|
|
315
339
|
`KICKS_LIVENESS_DIR` from its own environment, which the kubelet takes from the
|
|
316
340
|
container it runs in.
|
|
317
341
|
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
declared and left blank.
|
|
342
|
+
An empty string counts as unset, which is what a ConfigMap gives you when a key
|
|
343
|
+
is declared and left blank.
|
|
321
344
|
|
|
322
345
|
## Verifying on a live pod
|
|
323
346
|
|
data/docs/LIMITATIONS.md
CHANGED
|
@@ -80,20 +80,48 @@ see the respawn escalation below.
|
|
|
80
80
|
|
|
81
81
|
## Changing the worker count at runtime is not supported
|
|
82
82
|
|
|
83
|
-
`ServerEngine` re-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
83
|
+
`ServerEngine` can re-read its configuration on SIGHUP and scale the fork set,
|
|
84
|
+
but a fork holds its own copy of that configuration from the moment it was
|
|
85
|
+
created. Every monitor re-declares that captured count on every tick.
|
|
86
|
+
|
|
87
|
+
After a scale-up, old forks keep declaring the old count while new forks declare
|
|
88
|
+
the new one. The shared `expected` file therefore depends on which fork wrote
|
|
89
|
+
last; it can temporarily require the new slots, or incorrectly report the old
|
|
90
|
+
set as complete. After a scale-down, the surviving forks keep declaring the old
|
|
91
|
+
count and the retired slots eventually go stale. Neither direction is safe.
|
|
92
|
+
|
|
93
|
+
If you change `workers`, restart the runner. Runtime scaling is not supported.
|
|
94
|
+
|
|
95
|
+
## Container generations require Linux procfs and container-owned PID 1
|
|
96
|
+
|
|
97
|
+
Kubernetes keeps an `emptyDir` across restarts of a container in the same pod.
|
|
98
|
+
To prevent the next container from inheriting a fresh heartbeat, the worker and
|
|
99
|
+
exec probe independently identify their shared incarnation from Linux procfs:
|
|
100
|
+
the mount namespace plus PID 1 start time. No application cache is removed.
|
|
101
|
+
|
|
102
|
+
The identifier also assumes that the container owns PID 1. With
|
|
103
|
+
`shareProcessNamespace: true`, PID 1 is the pod sandbox, and with
|
|
104
|
+
`hostPID: true` it is the node's init process; neither restarts when the worker
|
|
105
|
+
container does, so the start-time half of the identifier stays constant. The
|
|
106
|
+
mount namespace inode is then the only remaining signal, and the kernel
|
|
107
|
+
normally hands the just-released inode back to the replacement container in an
|
|
108
|
+
otherwise quiet pod. Under either setting the guard silently degrades to the
|
|
109
|
+
0.1.1 freshness-only behavior and can accept a fresh mark from the previous
|
|
110
|
+
container. Do not enable either setting on a pod whose `startupProbe` relies on
|
|
111
|
+
this guarantee.
|
|
112
|
+
|
|
113
|
+
If procfs is unavailable — for example, when using the gem outside a Linux
|
|
114
|
+
container — generation detection falls back to the original freshness-only
|
|
115
|
+
check. The worker and probe still function, but they cannot distinguish a fresh
|
|
116
|
+
mark left by a previous process from one written by the current process. Linux
|
|
117
|
+
Kubernetes, Docker, and Nomad containers expose the required procfs entries
|
|
118
|
+
under their normal configuration.
|
|
119
|
+
|
|
120
|
+
The worker and its exec probe must run in the **same container**. A neighbouring
|
|
121
|
+
sidecar can mount the same `emptyDir`, but it has a different mount namespace
|
|
122
|
+
and therefore treats the worker's marks as belonging to another container.
|
|
123
|
+
Kubernetes exec probes already run inside the container they check; do not move
|
|
124
|
+
`kicks-liveness` into a separate health sidecar.
|
|
97
125
|
|
|
98
126
|
## A respawn loop is reported once per grace window, not once per respawn
|
|
99
127
|
|
|
@@ -115,6 +143,15 @@ reports the elapsed time and the number of starts:
|
|
|
115
143
|
The number of starts is the diagnosis: it separates a slow start from a respawn
|
|
116
144
|
loop at a glance.
|
|
117
145
|
|
|
146
|
+
This unhealthy-run counter is scoped to the marks volume, not to the container
|
|
147
|
+
generation. If a container is restarted before its slot has ever become
|
|
148
|
+
healthy, the replacement continues the same count: its repeated `started:` line
|
|
149
|
+
is suppressed, and an expired grace window may immediately report an ERROR that
|
|
150
|
+
includes starts from the previous container. This is diagnostic state only; it
|
|
151
|
+
does not participate in the probe result, and the first healthy tick removes
|
|
152
|
+
it. Treat the count as "starts since this slot was last healthy in this pod",
|
|
153
|
+
not "starts in this container".
|
|
154
|
+
|
|
118
155
|
## Do not install both `kicks` and `sneakers`
|
|
119
156
|
|
|
120
157
|
The gem declares neither as a dependency, because at runtime it needs only the
|
|
@@ -126,7 +163,9 @@ executing the `sneakers` code while its lockfile says otherwise, and pulling in
|
|
|
126
163
|
resolve.
|
|
127
164
|
|
|
128
165
|
A loud failure gets fixed; a silent substitution does not. Keep exactly one of
|
|
129
|
-
the two.
|
|
166
|
+
the two. Bundler itself still accepts the combination, but `install!` now
|
|
167
|
+
rejects a process in which both gems are activated before either set of hooks is
|
|
168
|
+
installed.
|
|
130
169
|
|
|
131
170
|
If neither is present, `install!` raises a `LoadError` naming both with their
|
|
132
171
|
required versions.
|
data/docs/SETUP.md
CHANGED
|
@@ -10,7 +10,7 @@ having both is worse than it looks.
|
|
|
10
10
|
|
|
11
11
|
Those two floors are exact, not aspirational: CI runs the suite against
|
|
12
12
|
`kicks 3.0.0` and `sneakers 2.11.0` pinned, alongside the matrix that tracks the
|
|
13
|
-
|
|
13
|
+
current release of each. A `~>` matrix on its own would only ever prove that the
|
|
14
14
|
latest version works.
|
|
15
15
|
|
|
16
16
|
```ruby
|
|
@@ -142,12 +142,15 @@ Set it only to point somewhere **other** than `Sneakers.logger`:
|
|
|
142
142
|
config.logger = Rails.logger
|
|
143
143
|
```
|
|
144
144
|
|
|
145
|
-
| Option | Default | |
|
|
146
|
-
|
|
147
|
-
| `logger` | `Sneakers.logger`, resolved lazily |
|
|
148
|
-
| `enabled` | `true` |
|
|
149
|
-
| `tick` | `10` | seconds between checks |
|
|
150
|
-
| `startup_grace_ticks` | `6` | unhealthy ticks
|
|
145
|
+
| Option | Default | Purpose | Valid values |
|
|
146
|
+
|---|---|---|---|
|
|
147
|
+
| `logger` | `Sneakers.logger`, resolved lazily | transition and error logs | logger-compatible object |
|
|
148
|
+
| `enabled` | `true` | start the monitor thread | `true` or `false` |
|
|
149
|
+
| `tick` | `10` | seconds between checks | positive number smaller than `max_age` |
|
|
150
|
+
| `startup_grace_ticks` | `6` | unhealthy startup ticks before one ERROR | positive integer |
|
|
151
|
+
|
|
152
|
+
Invalid combinations are rejected before the monitor starts instead of running
|
|
153
|
+
one that is guaranteed to publish stale marks or never report a stalled startup.
|
|
151
154
|
|
|
152
155
|
`dir` and `max_age` are **not** here — they come from environment variables
|
|
153
156
|
only. See
|
|
@@ -171,11 +174,13 @@ Once workers are running, the marks directory is the other half of the answer:
|
|
|
171
174
|
```
|
|
172
175
|
$ ls -l /opt/app/tmp/health/
|
|
173
176
|
expected
|
|
177
|
+
generation
|
|
174
178
|
worker-0
|
|
175
179
|
worker-1
|
|
176
180
|
```
|
|
177
181
|
|
|
178
|
-
One `worker-<slot>` file per fork, plus `expected
|
|
179
|
-
the slot files are not, the workers
|
|
180
|
-
|
|
181
|
-
which is removed as soon as
|
|
182
|
+
One `worker-<slot>` file per fork, plus `expected` and the current container
|
|
183
|
+
`generation`. If `expected` is there and the slot files are not, the workers
|
|
184
|
+
have not finished subscribing. A slot that keeps restarting without ever
|
|
185
|
+
subscribing also leaves an `attempt-<slot>` file, which is removed as soon as
|
|
186
|
+
that slot becomes healthy.
|
data/docs/VERIFYING.md
CHANGED
|
@@ -30,14 +30,23 @@ it takes.
|
|
|
30
30
|
|
|
31
31
|
```bash
|
|
32
32
|
spec/integration/verify.sh build # image, into the engine the cluster uses
|
|
33
|
+
spec/integration/verify.sh status # print and verify the exact target
|
|
33
34
|
spec/integration/verify.sh up # namespace, broker, worker
|
|
34
35
|
spec/integration/verify.sh down # deletes the namespace
|
|
35
36
|
```
|
|
36
37
|
|
|
37
|
-
Every `kubectl` call
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
38
|
+
Every `kubectl` call names its context and namespace explicitly, so the script
|
|
39
|
+
never falls back to `current-context`. It will still operate on whichever target
|
|
40
|
+
you configure: run `status` and inspect both values before `up` or `down`. The
|
|
41
|
+
script labels namespaces it creates and refuses to apply to or delete an
|
|
42
|
+
existing namespace without that ownership label.
|
|
43
|
+
|
|
44
|
+
The default image uses Kicks. Build it with Sneakers instead without editing the
|
|
45
|
+
fixture:
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
AMQP_WORKER_GEM=sneakers spec/integration/verify.sh build
|
|
49
|
+
```
|
|
41
50
|
|
|
42
51
|
Three things that cost time if you meet them the hard way:
|
|
43
52
|
|
|
@@ -88,11 +97,15 @@ spec/integration/verify.sh probe
|
|
|
88
97
|
INFO: [liveness] slot 0: started: dir=/opt/app/tmp/health max_age=45s tick=10s processes=1 consumers=2
|
|
89
98
|
INFO: [liveness] slot 0: waiting for 2 consumers
|
|
90
99
|
INFO: [liveness] slot 0: healthy # one tick later
|
|
91
|
-
expected worker-0
|
|
100
|
+
expected generation worker-0 # in the marks directory
|
|
92
101
|
1 process(es) healthy # probe, exit 0
|
|
93
102
|
```
|
|
94
103
|
|
|
95
|
-
The hooks fired inside a real ServerEngine fork, which no double can show.
|
|
104
|
+
The hooks fired inside a real ServerEngine fork, which no double can show. On a
|
|
105
|
+
fresh fixture the first attempt also prints `started:`. Do not use that line as
|
|
106
|
+
the sole proof after a same-pod container restart: an inherited unhealthy-run
|
|
107
|
+
counter deliberately suppresses repeated start lines, as described in
|
|
108
|
+
[LIMITATIONS.md](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn).
|
|
96
109
|
|
|
97
110
|
### 2. A stale mark
|
|
98
111
|
|
|
@@ -233,9 +246,9 @@ find them.
|
|
|
233
246
|
```
|
|
234
247
|
INFO: [liveness] slot 0: started: ... processes=2 consumers=2
|
|
235
248
|
INFO: [liveness] slot 1: started: ... processes=2 consumers=2
|
|
236
|
-
expected worker-0 worker-1
|
|
237
|
-
2 process(es) healthy
|
|
238
|
-
worker-1 stale 120s > 45s
|
|
249
|
+
expected generation worker-0 worker-1 # expected contains "2"
|
|
250
|
+
2 process(es) healthy # exit 0
|
|
251
|
+
worker-1 stale 120s > 45s # exit 1 — staling either mark is enough
|
|
239
252
|
```
|
|
240
253
|
|
|
241
254
|
Both marks are required, not just the first.
|
|
@@ -322,6 +335,38 @@ pod is restarted. That is why the documented rule set pairs it with
|
|
|
322
335
|
`absent_over_time`, and why an `up == 0` on the scrape job is worth having
|
|
323
336
|
beside both.
|
|
324
337
|
|
|
338
|
+
### 11. A private-PID container restart cannot inherit a healthy mark
|
|
339
|
+
|
|
340
|
+
This scenario needs the fixture image but not Kubernetes or RabbitMQ. It starts
|
|
341
|
+
a container that publishes a healthy mark into a named volume, restarts that
|
|
342
|
+
same container, and delays the new writer for 15 seconds:
|
|
343
|
+
|
|
344
|
+
```bash
|
|
345
|
+
spec/integration/verify.sh build
|
|
346
|
+
spec/integration/verify.sh generation-restart
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
During the delay the inherited mark is still fresh, but the probe must reject
|
|
350
|
+
its old generation. Once the replacement writer publishes its own mark, the
|
|
351
|
+
same probe must become healthy again. The command checks both transitions and
|
|
352
|
+
prints output in this form:
|
|
353
|
+
|
|
354
|
+
```
|
|
355
|
+
before=mnt:[4026532686]:123456
|
|
356
|
+
after=mnt:[4026532686]:123789
|
|
357
|
+
inherited=heartbeat belongs to a previous container: worker has not started yet
|
|
358
|
+
recovered=1 process(es) healthy
|
|
359
|
+
```
|
|
360
|
+
|
|
361
|
+
The namespace inode is allowed to be reused, as in the example above; the PID 1
|
|
362
|
+
start time still distinguishes the two incarnations. The test removes its
|
|
363
|
+
container and named volume on exit.
|
|
364
|
+
|
|
365
|
+
This scenario covers the default container-private PID namespace. With
|
|
366
|
+
`shareProcessNamespace: true` or `hostPID: true`, PID 1 survives the worker
|
|
367
|
+
container restart and this guarantee does not hold; see
|
|
368
|
+
[LIMITATIONS.md](LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1).
|
|
369
|
+
|
|
325
370
|
## What to do with a disagreement
|
|
326
371
|
|
|
327
372
|
Record it, then fix whichever is wrong — the code or the document. The numbers
|
|
@@ -17,20 +17,26 @@ module KicksLiveness
|
|
|
17
17
|
# @return [Integer] unhealthy ticks tolerated at startup before one ERROR
|
|
18
18
|
DEFAULT_STARTUP_GRACE_TICKS = 6
|
|
19
19
|
|
|
20
|
-
#
|
|
21
|
-
|
|
20
|
+
# Seconds between checks. An incompatibility warning for an environment
|
|
21
|
+
# fallback is delayed until this value is actually used, so an initializer
|
|
22
|
+
# can override it or disable the monitor without a misleading warning.
|
|
23
|
+
# @return [Numeric]
|
|
24
|
+
def tick
|
|
25
|
+
warn_incompatible_environment_tick_once
|
|
26
|
+
@tick
|
|
27
|
+
end
|
|
22
28
|
# @return [Integer] unhealthy ticks tolerated at startup before one ERROR
|
|
23
|
-
|
|
29
|
+
attr_reader :startup_grace_ticks
|
|
24
30
|
# @return [Logger, nil] explicit logger; defaults to +Sneakers.logger+
|
|
25
31
|
attr_accessor :logger
|
|
26
32
|
# @param value [Boolean] set false to start no monitor thread, e.g. in tests
|
|
27
33
|
attr_writer :enabled
|
|
28
34
|
|
|
29
35
|
def initialize
|
|
30
|
-
@tick = Heartbeat.env_int(:tick, DEFAULT_TICK)
|
|
31
36
|
@startup_grace_ticks = DEFAULT_STARTUP_GRACE_TICKS
|
|
32
37
|
@enabled = true
|
|
33
38
|
@logger = nil
|
|
39
|
+
@tick = environment_tick
|
|
34
40
|
end
|
|
35
41
|
|
|
36
42
|
# @return [Boolean]
|
|
@@ -39,20 +45,36 @@ module KicksLiveness
|
|
|
39
45
|
end
|
|
40
46
|
|
|
41
47
|
# A tick is what the monitor thread sleeps on, so a non-positive value is
|
|
42
|
-
# not a setting but a broken monitor.
|
|
43
|
-
#
|
|
44
|
-
#
|
|
45
|
-
# where the developer is looking.
|
|
48
|
+
# not a setting but a broken monitor. An initializer is code, and code
|
|
49
|
+
# should fail loudly at boot, where the developer is looking. Environment
|
|
50
|
+
# input is resolved separately by {#environment_tick}, with a safe fallback.
|
|
46
51
|
#
|
|
47
|
-
# @param seconds [
|
|
52
|
+
# @param seconds [Numeric]
|
|
48
53
|
# @raise [ArgumentError] if not a positive number
|
|
49
|
-
# @return [
|
|
54
|
+
# @return [Numeric]
|
|
50
55
|
def tick=(seconds)
|
|
51
56
|
raise ArgumentError, "tick must be a positive number, got #{seconds.inspect}" unless positive_number?(seconds)
|
|
57
|
+
raise ArgumentError, "tick must be less than max_age (#{max_age}s), got #{seconds.inspect}" if seconds >= max_age
|
|
52
58
|
|
|
59
|
+
@incompatible_environment_tick = nil
|
|
53
60
|
@tick = seconds
|
|
54
61
|
end
|
|
55
62
|
|
|
63
|
+
# The monitor compares an integer tick counter with this value. Accepting
|
|
64
|
+
# zero, a float, or a string would silently prevent the startup escalation
|
|
65
|
+
# from ever firing.
|
|
66
|
+
#
|
|
67
|
+
# @param ticks [Integer]
|
|
68
|
+
# @raise [ArgumentError] unless +ticks+ is a positive integer
|
|
69
|
+
# @return [Integer]
|
|
70
|
+
def startup_grace_ticks=(ticks)
|
|
71
|
+
unless ticks.is_a?(Integer) && ticks.positive?
|
|
72
|
+
raise ArgumentError, "startup_grace_ticks must be a positive integer, got #{ticks.inspect}"
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
@startup_grace_ticks = ticks
|
|
76
|
+
end
|
|
77
|
+
|
|
56
78
|
# Read-only, sourced from the environment so that it matches what the probe
|
|
57
79
|
# sees.
|
|
58
80
|
# @return [String] marks directory
|
|
@@ -76,6 +98,47 @@ module KicksLiveness
|
|
|
76
98
|
|
|
77
99
|
private
|
|
78
100
|
|
|
101
|
+
def environment_tick
|
|
102
|
+
raw = Heartbeat.env_raw(:tick)
|
|
103
|
+
seconds = Heartbeat.env_int(:tick, DEFAULT_TICK)
|
|
104
|
+
threshold = max_age
|
|
105
|
+
return seconds if seconds < threshold
|
|
106
|
+
|
|
107
|
+
fallback = [DEFAULT_TICK, threshold / 2.0].min
|
|
108
|
+
@incompatible_environment_tick = [seconds, threshold, fallback, raw]
|
|
109
|
+
fallback
|
|
110
|
+
end
|
|
111
|
+
|
|
112
|
+
def warn_incompatible_environment_tick_once
|
|
113
|
+
warning = @incompatible_environment_tick
|
|
114
|
+
return unless warning
|
|
115
|
+
|
|
116
|
+
@incompatible_environment_tick = nil
|
|
117
|
+
warn_incompatible_environment_tick(*warning)
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
def warn_incompatible_environment_tick(seconds, threshold, fallback, raw)
|
|
121
|
+
Kernel.warn(
|
|
122
|
+
"WARN [liveness] effective tick=#{seconds}s (#{environment_tick_source(raw)}) must be less than " \
|
|
123
|
+
"KICKS_LIVENESS_MAX_AGE=#{threshold}s; using #{fallback}s"
|
|
124
|
+
)
|
|
125
|
+
rescue StandardError
|
|
126
|
+
# Configuration recovery must not become a worker boot failure merely
|
|
127
|
+
# because stderr is unavailable or warning output has been overridden.
|
|
128
|
+
nil
|
|
129
|
+
end
|
|
130
|
+
|
|
131
|
+
def environment_tick_source(raw)
|
|
132
|
+
return 'default' if raw.nil?
|
|
133
|
+
|
|
134
|
+
value = Integer(raw)
|
|
135
|
+
return "KICKS_LIVENESS_TICK=#{raw}" if value.positive?
|
|
136
|
+
|
|
137
|
+
"default after invalid KICKS_LIVENESS_TICK=#{raw.inspect}"
|
|
138
|
+
rescue ArgumentError, TypeError
|
|
139
|
+
"default after invalid KICKS_LIVENESS_TICK=#{raw.inspect}"
|
|
140
|
+
end
|
|
141
|
+
|
|
79
142
|
def positive_number?(value)
|
|
80
143
|
value.is_a?(Numeric) && value.positive?
|
|
81
144
|
end
|