kicks_liveness 0.1.2 → 0.1.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/docs/VERIFYING.md CHANGED
@@ -35,12 +35,32 @@ spec/integration/verify.sh up # namespace, broker, worker
35
35
  spec/integration/verify.sh down # deletes the namespace
36
36
  ```
37
37
 
38
+ The Docker form of scenario 11 and scenario 12 need no cluster or broker.
39
+
38
40
  Every `kubectl` call names its context and namespace explicitly, so the script
39
41
  never falls back to `current-context`. It will still operate on whichever target
40
42
  you configure: run `status` and inspect both values before `up` or `down`. The
41
43
  script labels namespaces it creates and refuses to apply to or delete an
42
44
  existing namespace without that ownership label.
43
45
 
46
+ For another local cluster, export `KICKS_LIVENESS_K8S_CONTEXT` and
47
+ `KICKS_LIVENESS_DOCKER_CONTEXT` before building. `KICKS_LIVENESS_K8S_NAMESPACE`
48
+ selects the disposable namespace; `KICKS_LIVENESS_IMAGE` and
49
+ `KICKS_LIVENESS_RABBITMQ_IMAGE` select local worker and broker images.
50
+ Run the direct recipes below from the repository root with this guarded helper:
51
+
52
+ ```bash
53
+ export KICKS_LIVENESS_K8S_CONTEXT="${KICKS_LIVENESS_K8S_CONTEXT:-docker-desktop}"
54
+ export KICKS_LIVENESS_K8S_NAMESPACE="${KICKS_LIVENESS_K8S_NAMESPACE:-kicks-liveness}"
55
+ k() {
56
+ local owner
57
+ owner="$(kubectl --context "$KICKS_LIVENESS_K8S_CONTEXT" get namespace \
58
+ "$KICKS_LIVENESS_K8S_NAMESPACE" -o 'jsonpath={.metadata.labels.app\.kubernetes\.io/managed-by}')" || return
59
+ [ "$owner" = kicks-liveness-fixture ] || { echo 'not a fixture namespace' >&2; return 1; }
60
+ kubectl --context "$KICKS_LIVENESS_K8S_CONTEXT" --namespace "$KICKS_LIVENESS_K8S_NAMESPACE" "$@"
61
+ }
62
+ ```
63
+
44
64
  The default image uses Kicks. Build it with Sneakers instead without editing the
45
65
  fixture:
46
66
 
@@ -101,11 +121,11 @@ expected generation worker-0 # in the marks directory
101
121
  1 process(es) healthy # probe, exit 0
102
122
  ```
103
123
 
104
- The hooks fired inside a real ServerEngine fork, which no double can show. On a
105
- fresh fixture the first attempt also prints `started:`. Do not use that line as
106
- the sole proof after a same-pod container restart: an inherited unhealthy-run
107
- counter deliberately suppresses repeated start lines, as described in
108
- [LIMITATIONS.md](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn).
124
+ The hooks fired inside a real ServerEngine fork, which no double can show. The
125
+ `started:` line is also valid proof after a same-pod container restart: since
126
+ 0.1.3 the respawn counter carries the container generation, so a replacement
127
+ container reports its own first start rather than continuing the previous
128
+ container's unhealthy run.
109
129
 
110
130
  ### 2. A stale mark
111
131
 
@@ -113,6 +133,14 @@ The recipe from KUBERNETES.md. Proves the probe's threshold and the
113
133
  self-repair, and nothing else — the mark is fixed on the next tick, well inside
114
134
  `failureThreshold`, so the container is never killed.
115
135
 
136
+ ```bash
137
+ pod="$(k get pods -l app=worker -o jsonpath='{.items[0].metadata.name}')"
138
+ k exec "$pod" -- touch -d '120 seconds ago' /opt/app/tmp/health/worker-0
139
+ spec/integration/verify.sh probe # both commands print stale and exit=1
140
+ sleep 12
141
+ spec/integration/verify.sh probe # both commands print healthy and exit=0
142
+ ```
143
+
116
144
  Worth running through **both** commands here, since this is the one scenario
117
145
  where the probe is supposed to answer `1`. `spec/integration/verify.sh probe`
118
146
  does exactly that: the standard `bundle exec kicks-liveness` and the optimised
@@ -128,7 +156,9 @@ worker-0 stale 120s > 45s # exit 1
128
156
  The one that matters. Delete a queue the worker consumes from:
129
157
 
130
158
  ```bash
131
- kubectl exec deploy/rabbitmq -- rabbitmqctl delete_queue kicks_liveness.alpha
159
+ k exec deploy/rabbitmq -- rabbitmqctl delete_queue kicks_liveness.alpha
160
+ k get pods -l app=worker -w # watch RESTARTS increase; Ctrl-C after recovery
161
+ spec/integration/verify.sh probe # exit=0 after the worker re-declares the queue
132
162
  ```
133
163
 
134
164
  Observed, from the moment of deletion:
@@ -162,7 +192,10 @@ not in the pod log. An exec probe's stdout goes nowhere else, which is why
162
192
  ### 4. The broker dies
163
193
 
164
194
  ```bash
165
- kubectl delete pod -l app=rabbitmq --grace-period=0 --force
195
+ k delete pod -l app=rabbitmq --grace-period=0 --force
196
+ k rollout status deploy/rabbitmq --timeout=300s # the Deployment recreates it
197
+ spec/integration/verify.sh logs 100
198
+ spec/integration/verify.sh probe
166
199
  ```
167
200
 
168
201
  The pod must **not** restart. "It survived" is not enough on its own — a broker
@@ -182,7 +215,16 @@ consumers back on its own.
182
215
 
183
216
  ### 5. The broker stays dead
184
217
 
185
- `kubectl scale deploy/rabbitmq --replicas=0`, then wait. Observed over 319 s
218
+ ```bash
219
+ k scale deploy/rabbitmq --replicas=0
220
+ sleep 150 # longer than max_age + periodSeconds * failureThreshold
221
+ spec/integration/verify.sh probe # still exit=0 during Bunny recovery
222
+ k get pods -l app=worker # RESTARTS must not increase
223
+ k scale deploy/rabbitmq --replicas=1 # restore, including after an interrupted check
224
+ k rollout status deploy/rabbitmq --timeout=300s
225
+ ```
226
+
227
+ With the broker scaled to zero, observed over 319 s
186
228
  — seven times `max_age`, more than twice the 135 s steady-state budget — the
187
229
  mark's age never exceeded 9 s and `restartCount` stayed 0. The monitor kept
188
230
  writing, because `recovering_from_network_failure?` never stops being true:
@@ -207,10 +249,82 @@ quickly — the monitor never gets to run again, so
207
249
  `shutting down: consumers are no longer checked` may not appear at all. The
208
250
  absence of ERROR is the observable, not the presence of that line.
209
251
 
252
+ The shutdown exemption is now bounded by `shutdown_grace` (60 seconds by
253
+ default). Keep `KICKS_LIVENESS_SHUTDOWN_GRACE` at least as long as the fixture's
254
+ `terminationGracePeriodSeconds`. An out-of-band SIGTERM does not start a kubelet
255
+ termination deadline: if the process remains stuck draining, the expected
256
+ monitor diagnostic after its grace is:
257
+
258
+ ```
259
+ [liveness] slot 0: shutdown grace expired: heartbeat is no longer refreshed
260
+ ```
261
+
262
+ The probe can still pass until the last mark exceeds `max_age`; afterwards it
263
+ must exit 1 with `worker-0 stale ...`, and the monitor must not refresh that
264
+ mark on later ticks. The deadline, repeated callbacks, wall-clock changes, and
265
+ a sibling fork's independent lifetime are covered without Kubernetes by:
266
+
267
+ ```bash
268
+ bundle exec rspec spec/kicks_liveness/configuration_spec.rb \
269
+ spec/kicks_liveness/monitor_spec.rb spec/probe_spec.rb
270
+ ```
271
+
272
+ The probe spec launches a separate Ruby process against the monitor's marks.
273
+ For the real worker and broker, run:
274
+
275
+ ```bash
276
+ spec/integration/verify.sh shutdown
277
+ ```
278
+
279
+ This temporarily uses `tick=1`, `max_age=3`, `shutdown_grace=10` and pod
280
+ termination grace 10 seconds, then restores the original deployment even on
281
+ failure. First it holds a job, rolls out a worker, releases the job only after
282
+ the old pod's first shutdown tick, and requires the job to finish and no
283
+ liveness ERROR; how long the replacement takes to become Ready does not matter.
284
+ Then it blocks a real pool drain, sends SIGTERM without deleting the pod,
285
+ removes the remaining idle consumer, and checks zero consumers, an initially
286
+ healthy probe, one expiry ERROR, an unchanged mark, a stale probe, and a kubelet
287
+ restart restoring both consumers. It releases the blocking job during recovery.
288
+
289
+ Recorded on 2026-10-21 with 0.1.4: both paths passed on OrbStack Kubernetes
290
+ v1.30.7+orb1, Linux arm64 6.12.10-orbstack-00297-gf8f6e015b993, Docker 27.4.1,
291
+ Ruby 3.4.11, Kicks 3.4.0, Bunny 3.4.0 and the openssl gem 4.0.2, using an image
292
+ built with `--no-cache --pull`. The stuck pod's `restartCount` changed from 0
293
+ to 1.
294
+
295
+ Replayed on 2026-10-21–22 on the same cluster with Sneakers 2.11.0 /
296
+ ServerEngine 2.0.7 and Sneakers 2.12.0 / ServerEngine 2.1.1, both on Ruby
297
+ 3.4.11 and Bunny 2.24.0: both drain paths passed, including the kubelet restart
298
+ and restored consumers. These images used cached layers and local gem packages,
299
+ not another clean online build.
300
+
301
+ Separate manual pod-deletion checks held the job beyond `shutdown_grace=3s`,
302
+ with `tick=1s`, `max_age=3s` and Kubernetes termination grace 20 seconds. On
303
+ both Sneakers versions and Kicks 3.4.0, an expired mark stayed unchanged and
304
+ both probe commands failed, but the terminating container was not restarted.
305
+ Releasing the job after expiry let it finish with exit 0 in 10.4–11.6 seconds
306
+ from deletion; leaving it blocked caused kubelet to kill it with exit 137 in
307
+ 20.8–20.9 seconds. Each run logged expiry once and recovered a healthy
308
+ replacement. The expiry ERROR is expected with this deliberately shorter
309
+ exemption; normal deployments should keep it at least as long as pod grace.
310
+
210
311
  ### 7. Bad broker credentials
211
312
 
212
313
  Point `AMQP_URL` at a wrong password. The fork cannot subscribe, so the
213
- supervisor respawns it every `start_worker_delay`, forever. Over 120 s:
314
+ supervisor respawns it every `start_worker_delay`, forever. The new pod never
315
+ passes its startup probe, so the rollout keeps the old healthy pod running and
316
+ `deploy/worker` may resolve to either one; read the newest pod by name. Over
317
+ 120 s:
318
+
319
+ ```bash
320
+ amqp="$(k get deploy worker -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="AMQP_URL")].value}')"
321
+ k set env deploy/worker AMQP_URL=amqp://guest:wrong@rabbitmq:5672
322
+ sleep 120
323
+ new="$(k get pods -l app=worker --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1:].metadata.name}')"
324
+ k logs "$new" --tail=30000 | grep liveness
325
+ k set env deploy/worker "AMQP_URL=$amqp" # restore even if the check fails
326
+ k rollout status deploy/worker --timeout=300s
327
+ ```
214
328
 
215
329
  | | lines |
216
330
  |---|---|
@@ -243,6 +357,18 @@ find them.
243
357
  `WORKER_COUNT=2` needs no rebuild; the rake task turns it into
244
358
  `config[:workers]`.
245
359
 
360
+ ```bash
361
+ k set env deploy/worker WORKER_COUNT=2
362
+ k rollout status deploy/worker --timeout=300s
363
+ spec/integration/verify.sh probe # 2 process(es) healthy
364
+ pod="$(k get pods -l app=worker -o jsonpath='{.items[0].metadata.name}')"
365
+ k exec "$pod" -- touch -d '120 seconds ago' /opt/app/tmp/health/worker-1
366
+ spec/integration/verify.sh probe # worker-1 alone makes it fail
367
+ sleep 12 # the monitor repairs the mark
368
+ k set env deploy/worker WORKER_COUNT=1
369
+ k rollout status deploy/worker --timeout=300s
370
+ ```
371
+
246
372
  ```
247
373
  INFO: [liveness] slot 0: started: ... processes=2 consumers=2
248
374
  INFO: [liveness] slot 1: started: ... processes=2 consumers=2
@@ -264,6 +390,17 @@ because KUBERNETES.md claims the startup probe *incidentally fixes rollouts
264
390
  where the old pod was removed before the new one had subscribed* — and that
265
391
  claim is about ordering, which needs two pods to be watched at once.
266
392
 
393
+ ```bash
394
+ k set env deploy/worker WORKER_COUNT=2
395
+ k rollout status deploy/worker --timeout=300s
396
+ # In a second shell with the same exports and helper, watch until rollout completes:
397
+ # while true; do k exec deploy/rabbitmq -- rabbitmqctl -q list_queues name consumers; sleep 1; done
398
+ k rollout restart deploy/worker
399
+ k rollout status deploy/worker --timeout=300s
400
+ k set env deploy/worker WORKER_COUNT=1 # restore; stop the watcher with Ctrl-C
401
+ k rollout status deploy/worker --timeout=300s
402
+ ```
403
+
267
404
  `kubectl rollout restart deploy/worker`, with `WORKER_COUNT=2`, so a fully
268
405
  subscribed pod is worth two consumers per queue. Counting consumers on the
269
406
  broker is what makes the ordering visible:
@@ -294,14 +431,14 @@ treatment as everything else here. The broker in this fixture is enough to check
294
431
  it; the plugin is not on by default:
295
432
 
296
433
  ```bash
297
- kubectl exec deploy/rabbitmq -- rabbitmq-plugins enable rabbitmq_prometheus
434
+ k exec deploy/rabbitmq -- rabbitmq-plugins enable rabbitmq_prometheus
298
435
  ```
299
436
 
300
437
  Then read all three endpoints from the worker pod, which has a Ruby but no
301
438
  `curl`:
302
439
 
303
440
  ```bash
304
- kubectl exec deploy/worker -- ruby -rnet/http -e \
441
+ k exec deploy/worker -- ruby -rnet/http -e \
305
442
  'puts Net::HTTP.get(URI("http://rabbitmq:15692/metrics/detailed?family=queue_consumer_count"))'
306
443
  ```
307
444
 
@@ -320,6 +457,15 @@ metric is a *different* name from the aggregate, the detailed endpoint needs the
320
457
  Then check the two halves of the alert against a throwaway queue — declare one
321
458
  with no consumers, scrape, delete it, scrape again:
322
459
 
460
+ ```bash
461
+ k exec deploy/worker -- bundle exec ruby -rbunny -e \
462
+ 'c = Bunny.new(ENV.fetch("AMQP_URL")).start; c.create_channel.queue("kicks_liveness.tmpcheck"); c.close'
463
+ # Repeat the scrape above: tmpcheck must have a zero-consumer series.
464
+ k exec deploy/rabbitmq -- rabbitmqctl delete_queue kicks_liveness.tmpcheck
465
+ # Repeat the scrape: tmpcheck must have no series, not a zero.
466
+ k exec deploy/rabbitmq -- rabbitmq-plugins disable rabbitmq_prometheus # restore
467
+ ```
468
+
323
469
  ```
324
470
  -- queue exists, zero consumers
325
471
  rabbitmq_detailed_queue_consumers{vhost="/",queue="kicks_liveness.tmpcheck"} 0
@@ -362,10 +508,120 @@ The namespace inode is allowed to be reused, as in the example above; the PID 1
362
508
  start time still distinguishes the two incarnations. The test removes its
363
509
  container and named volume on exit.
364
510
 
511
+ The command also prints the topology it ran under, which for an ordinary
512
+ container is the fully verified one:
513
+
514
+ ```
515
+ generation-restart: topology=verified
516
+ ```
517
+
365
518
  This scenario covers the default container-private PID namespace. With
366
519
  `shareProcessNamespace: true` or `hostPID: true`, PID 1 survives the worker
367
- container restart and this guarantee does not hold; see
368
- [LIMITATIONS.md](LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1).
520
+ container restart and this guarantee cannot hold — scenario 12 is that case,
521
+ showing both fail-closed rejection when the topology can be proved and the
522
+ degraded outcomes when procfs access cannot prove it.
523
+
524
+ The Kubernetes counterpart also leaves `attempt-0` in the pod's `emptyDir`:
525
+
526
+ ```bash
527
+ spec/integration/verify.sh container-restart
528
+ ```
529
+
530
+ It makes PID 1 exit itself through a fixture-only WINCH handler, not an ignored
531
+ in-namespace SIGKILL or ServerEngine's USR2 reload. It requires the same pod,
532
+ an increased `restartCount`, exit 137, a surviving volume witness, a changed
533
+ generation, one replacement `started:` line without inherited escalation, and
534
+ a healthy probe; the original deployment is restored afterwards. On the
535
+ environment recorded in scenario 6, all checks passed with `restartCount=0->1`.
536
+ The same check also passed for both Sneakers versions recorded there.
537
+ The Docker form was also rerun: PID 1 start time changed from 1515589 to 1515784
538
+ with mount namespace inode 4026534862 unchanged; inherited marks were rejected
539
+ until the delayed replacement writer published.
540
+
541
+ ### 12. Shared and host PID topology outcomes
542
+
543
+ The four PID topologies, in one command. It needs the fixture image but neither
544
+ Kubernetes nor RabbitMQ, because everything here turns on which PID namespace
545
+ the worker container is in:
546
+
547
+ ```bash
548
+ spec/integration/verify.sh build
549
+ spec/integration/verify.sh topology
550
+ ```
551
+
552
+ Recorded on Linux 6.12 (arm64), Docker 27.4.1, ruby 3.4.10:
553
+
554
+ ```
555
+ shared-pid: topology=unsafe
556
+ shared-pid: 0.1.2 generation before=mnt:[4026532693]:17207878 after=mnt:[4026532693]:17207878
557
+ shared-pid inherited heartbeat: exit=1 unsupported PID topology: pod must not set shareProcessNamespace or hostPID
558
+ shared-pid: marks directory holds expected generation topology-unsafe worker-0
559
+ shared-pid after the worker published: exit=1 unsupported PID topology: pod must not set shareProcessNamespace or hostPID
560
+ host-pid: topology=unsafe
561
+ host-pid: 0.1.2 generation before=mnt:[4026532699]:105 after=mnt:[4026532699]:105
562
+ host-pid inherited heartbeat: exit=1 unsupported PID topology: pod must not set shareProcessNamespace or hostPID
563
+ host-pid: marks directory holds expected generation topology-unsafe worker-0
564
+ host-pid after the worker published: exit=1 unsupported PID topology: pod must not set shareProcessNamespace or hostPID
565
+ private-pid: topology=verified
566
+ private-pid: exit=0 1 process(es) healthy
567
+ unverifiable: topology=unverifiable
568
+ unverifiable: warnings across three fork respawns=1
569
+ unverifiable inherited heartbeat: exit=1 heartbeat belongs to a previous container: worker has not started yet
570
+ unverifiable own heartbeat: exit=0 1 process(es) healthy
571
+ ```
572
+
573
+ Four cases, and each line is the point of one of them.
574
+
575
+ A separate run on a real Kubernetes v1.25.0 node on Docker Desktop used
576
+ containerd, no `securityContext`, a root worker and the default capability set.
577
+ It produced a different and operationally important table:
578
+
579
+ | pod topology | PID 1 | PID 1 UID | `/proc/1/ns/mnt` | start time | 0.1.3 state |
580
+ |---|---|---:|---|---:|---|
581
+ | `hostPID: true` | node init | 0 | denied | 0 | `unavailable` |
582
+ | `shareProcessNamespace: true` | pod sandbox | 65535 | denied | 19031096 | `unverifiable` |
583
+ | private PID | container process | 0 | readable | 19031243 | `verified` |
584
+ | shared PID plus `CAP_SYS_PTRACE` | pod sandbox | 65535 | readable | — | `unsafe` |
585
+
586
+ The Docker matrix proves the fail-closed branch by granting the access it needs;
587
+ the Kubernetes measurement shows that this is not the default result for either
588
+ unsupported setting. The degraded states still warn, but they retain the weaker
589
+ generation or freshness-only guarantees described in LIMITATIONS.md.
590
+
591
+ **`shared-pid`** joins another container's PID namespace, which is what
592
+ Kubernetes `shareProcessNamespace: true` does — there PID 1 is the pod sandbox.
593
+ **`host-pid`** uses the node's PID namespace, which is `hostPID: true`. In both,
594
+ the run publishes a complete, fresh heartbeat in the **0.1.2 format**, restarts
595
+ only the worker container, and recomputes the 0.1.2 identifier by the 0.1.2
596
+ rule. The `before`/`after` pair is the defect being reproduced rather than
597
+ argued about: both halves repeat, including the mount namespace inode, which the
598
+ kernel handed straight back to the replacement container. A 0.1.2 probe would
599
+ have accepted those inherited marks. The revised probe rejects the topology
600
+ before it ever looks at them, and the worker then adds `topology-unsafe` beside
601
+ the inherited files rather than a heartbeat of its own.
602
+
603
+ Reading PID 1's namespace link is a ptrace-level operation, so the `host-pid`
604
+ case runs with `CAP_SYS_PTRACE`; without it the link is unreadable and the
605
+ outcome depends on whether a generation can still be derived. Node init on the
606
+ measured Kubernetes node reported start time zero, so that pod was
607
+ `unavailable`. A readable positive start time would instead leave the topology
608
+ `unverifiable`. The fourth case below constructs that latter branch explicitly;
609
+ both are honest limits on what a container can prove about a topology it is not
610
+ allowed to inspect.
611
+
612
+ **`private-pid`** is the fully verified positive control: an ordinary container,
613
+ owning PID 1, probe green.
614
+
615
+ **`unverifiable`** is the second positive control, and the one that has to stay
616
+ green: a private-PID container whose checker cannot read `/proc/1/ns/mnt`. The
617
+ 0.1.2 generation check is kept in full — a heartbeat from another generation is
618
+ still rejected, its own is still accepted — and the cost is exactly one worker
619
+ warning across three fork respawns, not a probe that is permanently red.
620
+
621
+ That case is constructed by running the checker as a user other than PID 1's.
622
+ It is **not** a model of a Kubernetes exec probe, which has no user setting of
623
+ its own and normally inherits the container process credentials; it is simply
624
+ the one way to make a namespace link unreadable without a custom procfs mount.
369
625
 
370
626
  ## What to do with a disagreement
371
627
 
@@ -3,23 +3,18 @@ require 'fileutils'
3
3
  module KicksLiveness
4
4
  # Counts starts of a supervisor slot that have not reached healthy yet.
5
5
  #
6
- # Deliberately separate from {Heartbeat}. Heartbeat is the contract shared
7
- # with the probe and is loaded by it alone, so nothing the probe does not need
8
- # belongs there — including the +fileutils+ this class requires freely.
6
+ # Persistent state survives fork respawns; its generation separates container
7
+ # incarnations when available. Kept outside {Heartbeat} because the probe does
8
+ # not read these counters.
9
9
  #
10
- # The count lives in the marks directory rather than in memory because a
11
- # respawn storm destroys process state every few hundred milliseconds: the
12
- # supervisor brings a failing fork back after +start_worker_delay+, and every
13
- # incarnation of the monitor believes it is the first. That is how an outage
14
- # can produce thousands of identical INFO lines and not one ERROR. The
15
- # directory is the only state that survives the fork.
16
- #
17
- # @see file:docs/DESIGN.md#logging-events-not-the-pulse
10
+ # @see file:docs/LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn
18
11
  # @api private
19
12
  class Attempts
20
13
  # @param dir [String] marks directory
21
- def initialize(dir)
14
+ # @param generation [String, nil] container incarnation the count belongs to
15
+ def initialize(dir, generation: nil)
22
16
  @dir = dir
17
+ @generation = generation
23
18
  end
24
19
 
25
20
  # Records a start of this slot.
@@ -62,22 +57,32 @@ module KicksLiveness
62
57
  File.join(@dir, "attempt-#{slot}")
63
58
  end
64
59
 
65
- # A missing or unreadable counter means "no previous start", never an
66
- # exception: this is instrumentation, and it may not break a worker.
60
+ # Missing, unreadable, or foreign-generation counts start a new run.
61
+ # Counter failures must not prevent worker startup.
67
62
  def read(slot)
68
- count, epoch = File.read(path(slot)).split
63
+ count, epoch, generation = File.read(path(slot)).split
64
+ return [0, nil] unless same_generation?(generation)
65
+
69
66
  [Integer(count), Time.at(Integer(epoch)).utc]
70
67
  rescue StandardError
71
68
  [0, nil]
72
69
  end
73
70
 
71
+ def same_generation?(declared)
72
+ return true unless @generation
73
+
74
+ declared == "generation=#{@generation}"
75
+ end
76
+
74
77
  # Failing to write degrades the reporting back to one line per respawn,
75
78
  # which is noisy but harmless — so a read-only directory is swallowed here
76
79
  # rather than escalated.
77
80
  def write(slot, count, first)
78
81
  FileUtils.mkdir_p(@dir)
82
+ contents = "#{count} #{first.to_i}"
83
+ contents = "#{contents} generation=#{@generation}" if @generation
79
84
  tmp = "#{path(slot)}.#{Process.pid}"
80
- File.write(tmp, "#{count} #{first.to_i}\n")
85
+ File.write(tmp, "#{contents}\n")
81
86
  File.rename(tmp, path(slot))
82
87
  rescue StandardError
83
88
  nil
@@ -3,11 +3,8 @@ require_relative 'heartbeat'
3
3
  module KicksLiveness
4
4
  # What the application may configure, and what comes from the environment only.
5
5
  #
6
- # +dir+ and +max_age+ are deliberately read-only here: the probe runs as a
7
- # separate process launched by the container runtime and cannot read the
8
- # application's initializer. Were they settable in both places, a mismatch
9
- # between what the worker writes and what the probe expects would pass
10
- # unnoticed.
6
+ # dir and max_age are environment-only: the separate exec probe cannot read
7
+ # an application initializer and must resolve the same values as the worker.
11
8
  #
12
9
  # @see file:docs/SETUP.md#configuration
13
10
  # @see file:docs/DESIGN.md#what-is-configurable-and-where
@@ -16,6 +13,8 @@ module KicksLiveness
16
13
  DEFAULT_TICK = 10
17
14
  # @return [Integer] unhealthy ticks tolerated at startup before one ERROR
18
15
  DEFAULT_STARTUP_GRACE_TICKS = 6
16
+ # @return [Integer] seconds to keep publishing while the workers drain
17
+ DEFAULT_SHUTDOWN_GRACE = 60
19
18
 
20
19
  # Seconds between checks. An incompatibility warning for an environment
21
20
  # fallback is delayed until this value is actually used, so an initializer
@@ -27,6 +26,8 @@ module KicksLiveness
27
26
  end
28
27
  # @return [Integer] unhealthy ticks tolerated at startup before one ERROR
29
28
  attr_reader :startup_grace_ticks
29
+ # @return [Numeric] seconds allowed for graceful shutdown, measured monotonically
30
+ attr_reader :shutdown_grace
30
31
  # @return [Logger, nil] explicit logger; defaults to +Sneakers.logger+
31
32
  attr_accessor :logger
32
33
  # @param value [Boolean] set false to start no monitor thread, e.g. in tests
@@ -37,6 +38,7 @@ module KicksLiveness
37
38
  @enabled = true
38
39
  @logger = nil
39
40
  @tick = environment_tick
41
+ @shutdown_grace = Heartbeat.env_int(:shutdown_grace, DEFAULT_SHUTDOWN_GRACE)
40
42
  end
41
43
 
42
44
  # @return [Boolean]
@@ -44,10 +46,8 @@ module KicksLiveness
44
46
  @enabled
45
47
  end
46
48
 
47
- # A tick is what the monitor thread sleeps on, so a non-positive value is
48
- # not a setting but a broken monitor. An initializer is code, and code
49
- # should fail loudly at boot, where the developer is looking. Environment
50
- # input is resolved separately by {#environment_tick}, with a safe fallback.
49
+ # Initializer values fail at boot; environment input uses a safe fallback
50
+ # through {#environment_tick}.
51
51
  #
52
52
  # @param seconds [Numeric]
53
53
  # @raise [ArgumentError] if not a positive number
@@ -75,6 +75,21 @@ module KicksLiveness
75
75
  @startup_grace_ticks = ticks
76
76
  end
77
77
 
78
+ # Only the monitor reads this value, so an initializer may override the
79
+ # environment. Keep it at least as long as terminationGracePeriodSeconds.
80
+ # An infinite value would restore the unbounded shutdown exemption.
81
+ #
82
+ # @param seconds [Numeric] finite, positive duration
83
+ # @raise [ArgumentError] unless +seconds+ is a positive finite real number
84
+ # @return [Numeric]
85
+ def shutdown_grace=(seconds)
86
+ unless seconds.is_a?(Numeric) && seconds.real? && seconds.finite? && seconds.positive?
87
+ raise ArgumentError, "shutdown_grace must be a positive finite number, got #{seconds.inspect}"
88
+ end
89
+
90
+ @shutdown_grace = seconds
91
+ end
92
+
78
93
  # Read-only, sourced from the environment so that it matches what the probe
79
94
  # sees.
80
95
  # @return [String] marks directory