kicks_liveness 0.1.2 → 0.1.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +74 -1
- data/README.md +21 -6
- data/docs/DESIGN.md +133 -188
- data/docs/KUBERNETES.md +40 -13
- data/docs/LIMITATIONS.md +154 -61
- data/docs/SETUP.md +30 -0
- data/docs/VERIFYING.md +269 -13
- data/lib/kicks_liveness/attempts.rb +21 -16
- data/lib/kicks_liveness/configuration.rb +24 -9
- data/lib/kicks_liveness/heartbeat.rb +241 -111
- data/lib/kicks_liveness/hooks.rb +11 -29
- data/lib/kicks_liveness/monitor.rb +50 -51
- data/lib/kicks_liveness/registry.rb +17 -19
- data/lib/kicks_liveness/topology_report.rb +110 -0
- data/lib/kicks_liveness/version.rb +1 -1
- data/lib/kicks_liveness.rb +3 -1
- metadata +3 -2
data/docs/VERIFYING.md
CHANGED
|
@@ -35,12 +35,32 @@ spec/integration/verify.sh up # namespace, broker, worker
|
|
|
35
35
|
spec/integration/verify.sh down # deletes the namespace
|
|
36
36
|
```
|
|
37
37
|
|
|
38
|
+
The Docker form of scenario 11 and scenario 12 need no cluster or broker.
|
|
39
|
+
|
|
38
40
|
Every `kubectl` call names its context and namespace explicitly, so the script
|
|
39
41
|
never falls back to `current-context`. It will still operate on whichever target
|
|
40
42
|
you configure: run `status` and inspect both values before `up` or `down`. The
|
|
41
43
|
script labels namespaces it creates and refuses to apply to or delete an
|
|
42
44
|
existing namespace without that ownership label.
|
|
43
45
|
|
|
46
|
+
For another local cluster, export `KICKS_LIVENESS_K8S_CONTEXT` and
|
|
47
|
+
`KICKS_LIVENESS_DOCKER_CONTEXT` before building. `KICKS_LIVENESS_K8S_NAMESPACE`
|
|
48
|
+
selects the disposable namespace; `KICKS_LIVENESS_IMAGE` and
|
|
49
|
+
`KICKS_LIVENESS_RABBITMQ_IMAGE` select local worker and broker images.
|
|
50
|
+
Run the direct recipes below from the repository root with this guarded helper:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
export KICKS_LIVENESS_K8S_CONTEXT="${KICKS_LIVENESS_K8S_CONTEXT:-docker-desktop}"
|
|
54
|
+
export KICKS_LIVENESS_K8S_NAMESPACE="${KICKS_LIVENESS_K8S_NAMESPACE:-kicks-liveness}"
|
|
55
|
+
k() {
|
|
56
|
+
local owner
|
|
57
|
+
owner="$(kubectl --context "$KICKS_LIVENESS_K8S_CONTEXT" get namespace \
|
|
58
|
+
"$KICKS_LIVENESS_K8S_NAMESPACE" -o 'jsonpath={.metadata.labels.app\.kubernetes\.io/managed-by}')" || return
|
|
59
|
+
[ "$owner" = kicks-liveness-fixture ] || { echo 'not a fixture namespace' >&2; return 1; }
|
|
60
|
+
kubectl --context "$KICKS_LIVENESS_K8S_CONTEXT" --namespace "$KICKS_LIVENESS_K8S_NAMESPACE" "$@"
|
|
61
|
+
}
|
|
62
|
+
```
|
|
63
|
+
|
|
44
64
|
The default image uses Kicks. Build it with Sneakers instead without editing the
|
|
45
65
|
fixture:
|
|
46
66
|
|
|
@@ -101,11 +121,11 @@ expected generation worker-0 # in the marks directory
|
|
|
101
121
|
1 process(es) healthy # probe, exit 0
|
|
102
122
|
```
|
|
103
123
|
|
|
104
|
-
The hooks fired inside a real ServerEngine fork, which no double can show.
|
|
105
|
-
|
|
106
|
-
the
|
|
107
|
-
|
|
108
|
-
|
|
124
|
+
The hooks fired inside a real ServerEngine fork, which no double can show. The
|
|
125
|
+
`started:` line is also valid proof after a same-pod container restart: since
|
|
126
|
+
0.1.3 the respawn counter carries the container generation, so a replacement
|
|
127
|
+
container reports its own first start rather than continuing the previous
|
|
128
|
+
container's unhealthy run.
|
|
109
129
|
|
|
110
130
|
### 2. A stale mark
|
|
111
131
|
|
|
@@ -113,6 +133,14 @@ The recipe from KUBERNETES.md. Proves the probe's threshold and the
|
|
|
113
133
|
self-repair, and nothing else — the mark is fixed on the next tick, well inside
|
|
114
134
|
`failureThreshold`, so the container is never killed.
|
|
115
135
|
|
|
136
|
+
```bash
|
|
137
|
+
pod="$(k get pods -l app=worker -o jsonpath='{.items[0].metadata.name}')"
|
|
138
|
+
k exec "$pod" -- touch -d '120 seconds ago' /opt/app/tmp/health/worker-0
|
|
139
|
+
spec/integration/verify.sh probe # both commands print stale and exit=1
|
|
140
|
+
sleep 12
|
|
141
|
+
spec/integration/verify.sh probe # both commands print healthy and exit=0
|
|
142
|
+
```
|
|
143
|
+
|
|
116
144
|
Worth running through **both** commands here, since this is the one scenario
|
|
117
145
|
where the probe is supposed to answer `1`. `spec/integration/verify.sh probe`
|
|
118
146
|
does exactly that: the standard `bundle exec kicks-liveness` and the optimised
|
|
@@ -128,7 +156,9 @@ worker-0 stale 120s > 45s # exit 1
|
|
|
128
156
|
The one that matters. Delete a queue the worker consumes from:
|
|
129
157
|
|
|
130
158
|
```bash
|
|
131
|
-
|
|
159
|
+
k exec deploy/rabbitmq -- rabbitmqctl delete_queue kicks_liveness.alpha
|
|
160
|
+
k get pods -l app=worker -w # watch RESTARTS increase; Ctrl-C after recovery
|
|
161
|
+
spec/integration/verify.sh probe # exit=0 after the worker re-declares the queue
|
|
132
162
|
```
|
|
133
163
|
|
|
134
164
|
Observed, from the moment of deletion:
|
|
@@ -162,7 +192,10 @@ not in the pod log. An exec probe's stdout goes nowhere else, which is why
|
|
|
162
192
|
### 4. The broker dies
|
|
163
193
|
|
|
164
194
|
```bash
|
|
165
|
-
|
|
195
|
+
k delete pod -l app=rabbitmq --grace-period=0 --force
|
|
196
|
+
k rollout status deploy/rabbitmq --timeout=300s # the Deployment recreates it
|
|
197
|
+
spec/integration/verify.sh logs 100
|
|
198
|
+
spec/integration/verify.sh probe
|
|
166
199
|
```
|
|
167
200
|
|
|
168
201
|
The pod must **not** restart. "It survived" is not enough on its own — a broker
|
|
@@ -182,7 +215,16 @@ consumers back on its own.
|
|
|
182
215
|
|
|
183
216
|
### 5. The broker stays dead
|
|
184
217
|
|
|
185
|
-
|
|
218
|
+
```bash
|
|
219
|
+
k scale deploy/rabbitmq --replicas=0
|
|
220
|
+
sleep 150 # longer than max_age + periodSeconds * failureThreshold
|
|
221
|
+
spec/integration/verify.sh probe # still exit=0 during Bunny recovery
|
|
222
|
+
k get pods -l app=worker # RESTARTS must not increase
|
|
223
|
+
k scale deploy/rabbitmq --replicas=1 # restore, including after an interrupted check
|
|
224
|
+
k rollout status deploy/rabbitmq --timeout=300s
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
With the broker scaled to zero, observed over 319 s
|
|
186
228
|
— seven times `max_age`, more than twice the 135 s steady-state budget — the
|
|
187
229
|
mark's age never exceeded 9 s and `restartCount` stayed 0. The monitor kept
|
|
188
230
|
writing, because `recovering_from_network_failure?` never stops being true:
|
|
@@ -207,10 +249,82 @@ quickly — the monitor never gets to run again, so
|
|
|
207
249
|
`shutting down: consumers are no longer checked` may not appear at all. The
|
|
208
250
|
absence of ERROR is the observable, not the presence of that line.
|
|
209
251
|
|
|
252
|
+
The shutdown exemption is now bounded by `shutdown_grace` (60 seconds by
|
|
253
|
+
default). Keep `KICKS_LIVENESS_SHUTDOWN_GRACE` at least as long as the fixture's
|
|
254
|
+
`terminationGracePeriodSeconds`. An out-of-band SIGTERM does not start a kubelet
|
|
255
|
+
termination deadline: if the process remains stuck draining, the expected
|
|
256
|
+
monitor diagnostic after its grace is:
|
|
257
|
+
|
|
258
|
+
```
|
|
259
|
+
[liveness] slot 0: shutdown grace expired: heartbeat is no longer refreshed
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
The probe can still pass until the last mark exceeds `max_age`; afterwards it
|
|
263
|
+
must exit 1 with `worker-0 stale ...`, and the monitor must not refresh that
|
|
264
|
+
mark on later ticks. The deadline, repeated callbacks, wall-clock changes, and
|
|
265
|
+
a sibling fork's independent lifetime are covered without Kubernetes by:
|
|
266
|
+
|
|
267
|
+
```bash
|
|
268
|
+
bundle exec rspec spec/kicks_liveness/configuration_spec.rb \
|
|
269
|
+
spec/kicks_liveness/monitor_spec.rb spec/probe_spec.rb
|
|
270
|
+
```
|
|
271
|
+
|
|
272
|
+
The probe spec launches a separate Ruby process against the monitor's marks.
|
|
273
|
+
For the real worker and broker, run:
|
|
274
|
+
|
|
275
|
+
```bash
|
|
276
|
+
spec/integration/verify.sh shutdown
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
This temporarily uses `tick=1`, `max_age=3`, `shutdown_grace=10` and pod
|
|
280
|
+
termination grace 10 seconds, then restores the original deployment even on
|
|
281
|
+
failure. First it holds a job, rolls out a worker, releases the job only after
|
|
282
|
+
the old pod's first shutdown tick, and requires the job to finish and no
|
|
283
|
+
liveness ERROR; how long the replacement takes to become Ready does not matter.
|
|
284
|
+
Then it blocks a real pool drain, sends SIGTERM without deleting the pod,
|
|
285
|
+
removes the remaining idle consumer, and checks zero consumers, an initially
|
|
286
|
+
healthy probe, one expiry ERROR, an unchanged mark, a stale probe, and a kubelet
|
|
287
|
+
restart restoring both consumers. It releases the blocking job during recovery.
|
|
288
|
+
|
|
289
|
+
Recorded on 2026-10-21 with 0.1.4: both paths passed on OrbStack Kubernetes
|
|
290
|
+
v1.30.7+orb1, Linux arm64 6.12.10-orbstack-00297-gf8f6e015b993, Docker 27.4.1,
|
|
291
|
+
Ruby 3.4.11, Kicks 3.4.0, Bunny 3.4.0 and the openssl gem 4.0.2, using an image
|
|
292
|
+
built with `--no-cache --pull`. The stuck pod's `restartCount` changed from 0
|
|
293
|
+
to 1.
|
|
294
|
+
|
|
295
|
+
Replayed on 2026-10-21–22 on the same cluster with Sneakers 2.11.0 /
|
|
296
|
+
ServerEngine 2.0.7 and Sneakers 2.12.0 / ServerEngine 2.1.1, both on Ruby
|
|
297
|
+
3.4.11 and Bunny 2.24.0: both drain paths passed, including the kubelet restart
|
|
298
|
+
and restored consumers. These images used cached layers and local gem packages,
|
|
299
|
+
not another clean online build.
|
|
300
|
+
|
|
301
|
+
Separate manual pod-deletion checks held the job beyond `shutdown_grace=3s`,
|
|
302
|
+
with `tick=1s`, `max_age=3s` and Kubernetes termination grace 20 seconds. On
|
|
303
|
+
both Sneakers versions and Kicks 3.4.0, an expired mark stayed unchanged and
|
|
304
|
+
both probe commands failed, but the terminating container was not restarted.
|
|
305
|
+
Releasing the job after expiry let it finish with exit 0 in 10.4–11.6 seconds
|
|
306
|
+
from deletion; leaving it blocked caused kubelet to kill it with exit 137 in
|
|
307
|
+
20.8–20.9 seconds. Each run logged expiry once and recovered a healthy
|
|
308
|
+
replacement. The expiry ERROR is expected with this deliberately shorter
|
|
309
|
+
exemption; normal deployments should keep it at least as long as pod grace.
|
|
310
|
+
|
|
210
311
|
### 7. Bad broker credentials
|
|
211
312
|
|
|
212
313
|
Point `AMQP_URL` at a wrong password. The fork cannot subscribe, so the
|
|
213
|
-
supervisor respawns it every `start_worker_delay`, forever.
|
|
314
|
+
supervisor respawns it every `start_worker_delay`, forever. The new pod never
|
|
315
|
+
passes its startup probe, so the rollout keeps the old healthy pod running and
|
|
316
|
+
`deploy/worker` may resolve to either one; read the newest pod by name. Over
|
|
317
|
+
120 s:
|
|
318
|
+
|
|
319
|
+
```bash
|
|
320
|
+
amqp="$(k get deploy worker -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="AMQP_URL")].value}')"
|
|
321
|
+
k set env deploy/worker AMQP_URL=amqp://guest:wrong@rabbitmq:5672
|
|
322
|
+
sleep 120
|
|
323
|
+
new="$(k get pods -l app=worker --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1:].metadata.name}')"
|
|
324
|
+
k logs "$new" --tail=30000 | grep liveness
|
|
325
|
+
k set env deploy/worker "AMQP_URL=$amqp" # restore even if the check fails
|
|
326
|
+
k rollout status deploy/worker --timeout=300s
|
|
327
|
+
```
|
|
214
328
|
|
|
215
329
|
| | lines |
|
|
216
330
|
|---|---|
|
|
@@ -243,6 +357,18 @@ find them.
|
|
|
243
357
|
`WORKER_COUNT=2` needs no rebuild; the rake task turns it into
|
|
244
358
|
`config[:workers]`.
|
|
245
359
|
|
|
360
|
+
```bash
|
|
361
|
+
k set env deploy/worker WORKER_COUNT=2
|
|
362
|
+
k rollout status deploy/worker --timeout=300s
|
|
363
|
+
spec/integration/verify.sh probe # 2 process(es) healthy
|
|
364
|
+
pod="$(k get pods -l app=worker -o jsonpath='{.items[0].metadata.name}')"
|
|
365
|
+
k exec "$pod" -- touch -d '120 seconds ago' /opt/app/tmp/health/worker-1
|
|
366
|
+
spec/integration/verify.sh probe # worker-1 alone makes it fail
|
|
367
|
+
sleep 12 # the monitor repairs the mark
|
|
368
|
+
k set env deploy/worker WORKER_COUNT=1
|
|
369
|
+
k rollout status deploy/worker --timeout=300s
|
|
370
|
+
```
|
|
371
|
+
|
|
246
372
|
```
|
|
247
373
|
INFO: [liveness] slot 0: started: ... processes=2 consumers=2
|
|
248
374
|
INFO: [liveness] slot 1: started: ... processes=2 consumers=2
|
|
@@ -264,6 +390,17 @@ because KUBERNETES.md claims the startup probe *incidentally fixes rollouts
|
|
|
264
390
|
where the old pod was removed before the new one had subscribed* — and that
|
|
265
391
|
claim is about ordering, which needs two pods to be watched at once.
|
|
266
392
|
|
|
393
|
+
```bash
|
|
394
|
+
k set env deploy/worker WORKER_COUNT=2
|
|
395
|
+
k rollout status deploy/worker --timeout=300s
|
|
396
|
+
# In a second shell with the same exports and helper, watch until rollout completes:
|
|
397
|
+
# while true; do k exec deploy/rabbitmq -- rabbitmqctl -q list_queues name consumers; sleep 1; done
|
|
398
|
+
k rollout restart deploy/worker
|
|
399
|
+
k rollout status deploy/worker --timeout=300s
|
|
400
|
+
k set env deploy/worker WORKER_COUNT=1 # restore; stop the watcher with Ctrl-C
|
|
401
|
+
k rollout status deploy/worker --timeout=300s
|
|
402
|
+
```
|
|
403
|
+
|
|
267
404
|
`kubectl rollout restart deploy/worker`, with `WORKER_COUNT=2`, so a fully
|
|
268
405
|
subscribed pod is worth two consumers per queue. Counting consumers on the
|
|
269
406
|
broker is what makes the ordering visible:
|
|
@@ -294,14 +431,14 @@ treatment as everything else here. The broker in this fixture is enough to check
|
|
|
294
431
|
it; the plugin is not on by default:
|
|
295
432
|
|
|
296
433
|
```bash
|
|
297
|
-
|
|
434
|
+
k exec deploy/rabbitmq -- rabbitmq-plugins enable rabbitmq_prometheus
|
|
298
435
|
```
|
|
299
436
|
|
|
300
437
|
Then read all three endpoints from the worker pod, which has a Ruby but no
|
|
301
438
|
`curl`:
|
|
302
439
|
|
|
303
440
|
```bash
|
|
304
|
-
|
|
441
|
+
k exec deploy/worker -- ruby -rnet/http -e \
|
|
305
442
|
'puts Net::HTTP.get(URI("http://rabbitmq:15692/metrics/detailed?family=queue_consumer_count"))'
|
|
306
443
|
```
|
|
307
444
|
|
|
@@ -320,6 +457,15 @@ metric is a *different* name from the aggregate, the detailed endpoint needs the
|
|
|
320
457
|
Then check the two halves of the alert against a throwaway queue — declare one
|
|
321
458
|
with no consumers, scrape, delete it, scrape again:
|
|
322
459
|
|
|
460
|
+
```bash
|
|
461
|
+
k exec deploy/worker -- bundle exec ruby -rbunny -e \
|
|
462
|
+
'c = Bunny.new(ENV.fetch("AMQP_URL")).start; c.create_channel.queue("kicks_liveness.tmpcheck"); c.close'
|
|
463
|
+
# Repeat the scrape above: tmpcheck must have a zero-consumer series.
|
|
464
|
+
k exec deploy/rabbitmq -- rabbitmqctl delete_queue kicks_liveness.tmpcheck
|
|
465
|
+
# Repeat the scrape: tmpcheck must have no series, not a zero.
|
|
466
|
+
k exec deploy/rabbitmq -- rabbitmq-plugins disable rabbitmq_prometheus # restore
|
|
467
|
+
```
|
|
468
|
+
|
|
323
469
|
```
|
|
324
470
|
-- queue exists, zero consumers
|
|
325
471
|
rabbitmq_detailed_queue_consumers{vhost="/",queue="kicks_liveness.tmpcheck"} 0
|
|
@@ -362,10 +508,120 @@ The namespace inode is allowed to be reused, as in the example above; the PID 1
|
|
|
362
508
|
start time still distinguishes the two incarnations. The test removes its
|
|
363
509
|
container and named volume on exit.
|
|
364
510
|
|
|
511
|
+
The command also prints the topology it ran under, which for an ordinary
|
|
512
|
+
container is the fully verified one:
|
|
513
|
+
|
|
514
|
+
```
|
|
515
|
+
generation-restart: topology=verified
|
|
516
|
+
```
|
|
517
|
+
|
|
365
518
|
This scenario covers the default container-private PID namespace. With
|
|
366
519
|
`shareProcessNamespace: true` or `hostPID: true`, PID 1 survives the worker
|
|
367
|
-
container restart and this guarantee
|
|
368
|
-
|
|
520
|
+
container restart and this guarantee cannot hold — scenario 12 is that case,
|
|
521
|
+
showing both fail-closed rejection when the topology can be proved and the
|
|
522
|
+
degraded outcomes when procfs access cannot prove it.
|
|
523
|
+
|
|
524
|
+
The Kubernetes counterpart also leaves `attempt-0` in the pod's `emptyDir`:
|
|
525
|
+
|
|
526
|
+
```bash
|
|
527
|
+
spec/integration/verify.sh container-restart
|
|
528
|
+
```
|
|
529
|
+
|
|
530
|
+
It makes PID 1 exit itself through a fixture-only WINCH handler, not an ignored
|
|
531
|
+
in-namespace SIGKILL or ServerEngine's USR2 reload. It requires the same pod,
|
|
532
|
+
an increased `restartCount`, exit 137, a surviving volume witness, a changed
|
|
533
|
+
generation, one replacement `started:` line without inherited escalation, and
|
|
534
|
+
a healthy probe; the original deployment is restored afterwards. On the
|
|
535
|
+
environment recorded in scenario 6, all checks passed with `restartCount=0->1`.
|
|
536
|
+
The same check also passed for both Sneakers versions recorded there.
|
|
537
|
+
The Docker form was also rerun: PID 1 start time changed from 1515589 to 1515784
|
|
538
|
+
with mount namespace inode 4026534862 unchanged; inherited marks were rejected
|
|
539
|
+
until the delayed replacement writer published.
|
|
540
|
+
|
|
541
|
+
### 12. Shared and host PID topology outcomes
|
|
542
|
+
|
|
543
|
+
The four PID topologies, in one command. It needs the fixture image but neither
|
|
544
|
+
Kubernetes nor RabbitMQ, because everything here turns on which PID namespace
|
|
545
|
+
the worker container is in:
|
|
546
|
+
|
|
547
|
+
```bash
|
|
548
|
+
spec/integration/verify.sh build
|
|
549
|
+
spec/integration/verify.sh topology
|
|
550
|
+
```
|
|
551
|
+
|
|
552
|
+
Recorded on Linux 6.12 (arm64), Docker 27.4.1, ruby 3.4.10:
|
|
553
|
+
|
|
554
|
+
```
|
|
555
|
+
shared-pid: topology=unsafe
|
|
556
|
+
shared-pid: 0.1.2 generation before=mnt:[4026532693]:17207878 after=mnt:[4026532693]:17207878
|
|
557
|
+
shared-pid inherited heartbeat: exit=1 unsupported PID topology: pod must not set shareProcessNamespace or hostPID
|
|
558
|
+
shared-pid: marks directory holds expected generation topology-unsafe worker-0
|
|
559
|
+
shared-pid after the worker published: exit=1 unsupported PID topology: pod must not set shareProcessNamespace or hostPID
|
|
560
|
+
host-pid: topology=unsafe
|
|
561
|
+
host-pid: 0.1.2 generation before=mnt:[4026532699]:105 after=mnt:[4026532699]:105
|
|
562
|
+
host-pid inherited heartbeat: exit=1 unsupported PID topology: pod must not set shareProcessNamespace or hostPID
|
|
563
|
+
host-pid: marks directory holds expected generation topology-unsafe worker-0
|
|
564
|
+
host-pid after the worker published: exit=1 unsupported PID topology: pod must not set shareProcessNamespace or hostPID
|
|
565
|
+
private-pid: topology=verified
|
|
566
|
+
private-pid: exit=0 1 process(es) healthy
|
|
567
|
+
unverifiable: topology=unverifiable
|
|
568
|
+
unverifiable: warnings across three fork respawns=1
|
|
569
|
+
unverifiable inherited heartbeat: exit=1 heartbeat belongs to a previous container: worker has not started yet
|
|
570
|
+
unverifiable own heartbeat: exit=0 1 process(es) healthy
|
|
571
|
+
```
|
|
572
|
+
|
|
573
|
+
Four cases, and each line is the point of one of them.
|
|
574
|
+
|
|
575
|
+
A separate run on a real Kubernetes v1.25.0 node on Docker Desktop used
|
|
576
|
+
containerd, no `securityContext`, a root worker and the default capability set.
|
|
577
|
+
It produced a different and operationally important table:
|
|
578
|
+
|
|
579
|
+
| pod topology | PID 1 | PID 1 UID | `/proc/1/ns/mnt` | start time | 0.1.3 state |
|
|
580
|
+
|---|---|---:|---|---:|---|
|
|
581
|
+
| `hostPID: true` | node init | 0 | denied | 0 | `unavailable` |
|
|
582
|
+
| `shareProcessNamespace: true` | pod sandbox | 65535 | denied | 19031096 | `unverifiable` |
|
|
583
|
+
| private PID | container process | 0 | readable | 19031243 | `verified` |
|
|
584
|
+
| shared PID plus `CAP_SYS_PTRACE` | pod sandbox | 65535 | readable | — | `unsafe` |
|
|
585
|
+
|
|
586
|
+
The Docker matrix proves the fail-closed branch by granting the access it needs;
|
|
587
|
+
the Kubernetes measurement shows that this is not the default result for either
|
|
588
|
+
unsupported setting. The degraded states still warn, but they retain the weaker
|
|
589
|
+
generation or freshness-only guarantees described in LIMITATIONS.md.
|
|
590
|
+
|
|
591
|
+
**`shared-pid`** joins another container's PID namespace, which is what
|
|
592
|
+
Kubernetes `shareProcessNamespace: true` does — there PID 1 is the pod sandbox.
|
|
593
|
+
**`host-pid`** uses the node's PID namespace, which is `hostPID: true`. In both,
|
|
594
|
+
the run publishes a complete, fresh heartbeat in the **0.1.2 format**, restarts
|
|
595
|
+
only the worker container, and recomputes the 0.1.2 identifier by the 0.1.2
|
|
596
|
+
rule. The `before`/`after` pair is the defect being reproduced rather than
|
|
597
|
+
argued about: both halves repeat, including the mount namespace inode, which the
|
|
598
|
+
kernel handed straight back to the replacement container. A 0.1.2 probe would
|
|
599
|
+
have accepted those inherited marks. The revised probe rejects the topology
|
|
600
|
+
before it ever looks at them, and the worker then adds `topology-unsafe` beside
|
|
601
|
+
the inherited files rather than a heartbeat of its own.
|
|
602
|
+
|
|
603
|
+
Reading PID 1's namespace link is a ptrace-level operation, so the `host-pid`
|
|
604
|
+
case runs with `CAP_SYS_PTRACE`; without it the link is unreadable and the
|
|
605
|
+
outcome depends on whether a generation can still be derived. Node init on the
|
|
606
|
+
measured Kubernetes node reported start time zero, so that pod was
|
|
607
|
+
`unavailable`. A readable positive start time would instead leave the topology
|
|
608
|
+
`unverifiable`. The fourth case below constructs that latter branch explicitly;
|
|
609
|
+
both are honest limits on what a container can prove about a topology it is not
|
|
610
|
+
allowed to inspect.
|
|
611
|
+
|
|
612
|
+
**`private-pid`** is the fully verified positive control: an ordinary container,
|
|
613
|
+
owning PID 1, probe green.
|
|
614
|
+
|
|
615
|
+
**`unverifiable`** is the second positive control, and the one that has to stay
|
|
616
|
+
green: a private-PID container whose checker cannot read `/proc/1/ns/mnt`. The
|
|
617
|
+
0.1.2 generation check is kept in full — a heartbeat from another generation is
|
|
618
|
+
still rejected, its own is still accepted — and the cost is exactly one worker
|
|
619
|
+
warning across three fork respawns, not a probe that is permanently red.
|
|
620
|
+
|
|
621
|
+
That case is constructed by running the checker as a user other than PID 1's.
|
|
622
|
+
It is **not** a model of a Kubernetes exec probe, which has no user setting of
|
|
623
|
+
its own and normally inherits the container process credentials; it is simply
|
|
624
|
+
the one way to make a namespace link unreadable without a custom procfs mount.
|
|
369
625
|
|
|
370
626
|
## What to do with a disagreement
|
|
371
627
|
|
|
@@ -3,23 +3,18 @@ require 'fileutils'
|
|
|
3
3
|
module KicksLiveness
|
|
4
4
|
# Counts starts of a supervisor slot that have not reached healthy yet.
|
|
5
5
|
#
|
|
6
|
-
#
|
|
7
|
-
#
|
|
8
|
-
#
|
|
6
|
+
# Persistent state survives fork respawns; its generation separates container
|
|
7
|
+
# incarnations when available. Kept outside {Heartbeat} because the probe does
|
|
8
|
+
# not read these counters.
|
|
9
9
|
#
|
|
10
|
-
#
|
|
11
|
-
# respawn storm destroys process state every few hundred milliseconds: the
|
|
12
|
-
# supervisor brings a failing fork back after +start_worker_delay+, and every
|
|
13
|
-
# incarnation of the monitor believes it is the first. That is how an outage
|
|
14
|
-
# can produce thousands of identical INFO lines and not one ERROR. The
|
|
15
|
-
# directory is the only state that survives the fork.
|
|
16
|
-
#
|
|
17
|
-
# @see file:docs/DESIGN.md#logging-events-not-the-pulse
|
|
10
|
+
# @see file:docs/LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn
|
|
18
11
|
# @api private
|
|
19
12
|
class Attempts
|
|
20
13
|
# @param dir [String] marks directory
|
|
21
|
-
|
|
14
|
+
# @param generation [String, nil] container incarnation the count belongs to
|
|
15
|
+
def initialize(dir, generation: nil)
|
|
22
16
|
@dir = dir
|
|
17
|
+
@generation = generation
|
|
23
18
|
end
|
|
24
19
|
|
|
25
20
|
# Records a start of this slot.
|
|
@@ -62,22 +57,32 @@ module KicksLiveness
|
|
|
62
57
|
File.join(@dir, "attempt-#{slot}")
|
|
63
58
|
end
|
|
64
59
|
|
|
65
|
-
#
|
|
66
|
-
#
|
|
60
|
+
# Missing, unreadable, or foreign-generation counts start a new run.
|
|
61
|
+
# Counter failures must not prevent worker startup.
|
|
67
62
|
def read(slot)
|
|
68
|
-
count, epoch = File.read(path(slot)).split
|
|
63
|
+
count, epoch, generation = File.read(path(slot)).split
|
|
64
|
+
return [0, nil] unless same_generation?(generation)
|
|
65
|
+
|
|
69
66
|
[Integer(count), Time.at(Integer(epoch)).utc]
|
|
70
67
|
rescue StandardError
|
|
71
68
|
[0, nil]
|
|
72
69
|
end
|
|
73
70
|
|
|
71
|
+
def same_generation?(declared)
|
|
72
|
+
return true unless @generation
|
|
73
|
+
|
|
74
|
+
declared == "generation=#{@generation}"
|
|
75
|
+
end
|
|
76
|
+
|
|
74
77
|
# Failing to write degrades the reporting back to one line per respawn,
|
|
75
78
|
# which is noisy but harmless — so a read-only directory is swallowed here
|
|
76
79
|
# rather than escalated.
|
|
77
80
|
def write(slot, count, first)
|
|
78
81
|
FileUtils.mkdir_p(@dir)
|
|
82
|
+
contents = "#{count} #{first.to_i}"
|
|
83
|
+
contents = "#{contents} generation=#{@generation}" if @generation
|
|
79
84
|
tmp = "#{path(slot)}.#{Process.pid}"
|
|
80
|
-
File.write(tmp, "#{
|
|
85
|
+
File.write(tmp, "#{contents}\n")
|
|
81
86
|
File.rename(tmp, path(slot))
|
|
82
87
|
rescue StandardError
|
|
83
88
|
nil
|
|
@@ -3,11 +3,8 @@ require_relative 'heartbeat'
|
|
|
3
3
|
module KicksLiveness
|
|
4
4
|
# What the application may configure, and what comes from the environment only.
|
|
5
5
|
#
|
|
6
|
-
#
|
|
7
|
-
#
|
|
8
|
-
# application's initializer. Were they settable in both places, a mismatch
|
|
9
|
-
# between what the worker writes and what the probe expects would pass
|
|
10
|
-
# unnoticed.
|
|
6
|
+
# dir and max_age are environment-only: the separate exec probe cannot read
|
|
7
|
+
# an application initializer and must resolve the same values as the worker.
|
|
11
8
|
#
|
|
12
9
|
# @see file:docs/SETUP.md#configuration
|
|
13
10
|
# @see file:docs/DESIGN.md#what-is-configurable-and-where
|
|
@@ -16,6 +13,8 @@ module KicksLiveness
|
|
|
16
13
|
DEFAULT_TICK = 10
|
|
17
14
|
# @return [Integer] unhealthy ticks tolerated at startup before one ERROR
|
|
18
15
|
DEFAULT_STARTUP_GRACE_TICKS = 6
|
|
16
|
+
# @return [Integer] seconds to keep publishing while the workers drain
|
|
17
|
+
DEFAULT_SHUTDOWN_GRACE = 60
|
|
19
18
|
|
|
20
19
|
# Seconds between checks. An incompatibility warning for an environment
|
|
21
20
|
# fallback is delayed until this value is actually used, so an initializer
|
|
@@ -27,6 +26,8 @@ module KicksLiveness
|
|
|
27
26
|
end
|
|
28
27
|
# @return [Integer] unhealthy ticks tolerated at startup before one ERROR
|
|
29
28
|
attr_reader :startup_grace_ticks
|
|
29
|
+
# @return [Numeric] seconds allowed for graceful shutdown, measured monotonically
|
|
30
|
+
attr_reader :shutdown_grace
|
|
30
31
|
# @return [Logger, nil] explicit logger; defaults to +Sneakers.logger+
|
|
31
32
|
attr_accessor :logger
|
|
32
33
|
# @param value [Boolean] set false to start no monitor thread, e.g. in tests
|
|
@@ -37,6 +38,7 @@ module KicksLiveness
|
|
|
37
38
|
@enabled = true
|
|
38
39
|
@logger = nil
|
|
39
40
|
@tick = environment_tick
|
|
41
|
+
@shutdown_grace = Heartbeat.env_int(:shutdown_grace, DEFAULT_SHUTDOWN_GRACE)
|
|
40
42
|
end
|
|
41
43
|
|
|
42
44
|
# @return [Boolean]
|
|
@@ -44,10 +46,8 @@ module KicksLiveness
|
|
|
44
46
|
@enabled
|
|
45
47
|
end
|
|
46
48
|
|
|
47
|
-
#
|
|
48
|
-
#
|
|
49
|
-
# should fail loudly at boot, where the developer is looking. Environment
|
|
50
|
-
# input is resolved separately by {#environment_tick}, with a safe fallback.
|
|
49
|
+
# Initializer values fail at boot; environment input uses a safe fallback
|
|
50
|
+
# through {#environment_tick}.
|
|
51
51
|
#
|
|
52
52
|
# @param seconds [Numeric]
|
|
53
53
|
# @raise [ArgumentError] if not a positive number
|
|
@@ -75,6 +75,21 @@ module KicksLiveness
|
|
|
75
75
|
@startup_grace_ticks = ticks
|
|
76
76
|
end
|
|
77
77
|
|
|
78
|
+
# Only the monitor reads this value, so an initializer may override the
|
|
79
|
+
# environment. Keep it at least as long as terminationGracePeriodSeconds.
|
|
80
|
+
# An infinite value would restore the unbounded shutdown exemption.
|
|
81
|
+
#
|
|
82
|
+
# @param seconds [Numeric] finite, positive duration
|
|
83
|
+
# @raise [ArgumentError] unless +seconds+ is a positive finite real number
|
|
84
|
+
# @return [Numeric]
|
|
85
|
+
def shutdown_grace=(seconds)
|
|
86
|
+
unless seconds.is_a?(Numeric) && seconds.real? && seconds.finite? && seconds.positive?
|
|
87
|
+
raise ArgumentError, "shutdown_grace must be a positive finite number, got #{seconds.inspect}"
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
@shutdown_grace = seconds
|
|
91
|
+
end
|
|
92
|
+
|
|
78
93
|
# Read-only, sourced from the environment so that it matches what the probe
|
|
79
94
|
# sees.
|
|
80
95
|
# @return [String] marks directory
|