kicks_liveness 0.1.0 → 0.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/docs/DESIGN.md CHANGED
@@ -40,16 +40,17 @@ outside it can, so let the worker publish and let the probe read.
40
40
 
41
41
  Every `tick` seconds (10 by default) the worker checks, **in its own memory**,
42
42
  that its consumers are still subscribed, and touches a file on tmpfs. The probe
43
- loads one dependency-free Ruby file, reads the file's mtime, and exits with 0 or
44
- 1.
43
+ loads one dependency-free Ruby file, checks the marks' container generation and
44
+ mtimes, and exits with 0 or 1.
45
45
 
46
46
  The probe therefore performs no network I/O and boots no framework, and the
47
- state it reads — the mark's mtime — comes from tmpfs, which is RAM. Be precise
48
- about the disk, though: starting the probe still loads the Ruby interpreter and
49
- two files of this gem from the image filesystem, and those reads are ordinary
50
- filesystem reads (usually served from page cache, but not guaranteed to be). The
51
- honest claim is not "no disk" but *no application boot, and no disk on the path
52
- that decides the answer* — which is what removes the defects above, not tuning.
47
+ state it reads — the generation and mark mtimes — comes from procfs and tmpfs.
48
+ Be precise about the disk, though: starting the probe still loads the Ruby
49
+ interpreter and two files of this gem from the image filesystem, and those reads
50
+ are ordinary filesystem reads (usually served from page cache, but not
51
+ guaranteed to be). The honest claim is not "no disk" but *no application boot,
52
+ and no disk on the path that decides the answer* — which is what removes the
53
+ defects above, not tuning.
53
54
 
54
55
  ## The health predicate
55
56
 
@@ -112,6 +113,7 @@ silently. That is the thing to re-check when upgrading Bunny.
112
113
 
113
114
  ```
114
115
  <dir>/expected how many forks the probe must wait for
116
+ <dir>/generation which container incarnation wrote this heartbeat
115
117
  <dir>/worker-<slot> one per fork, refreshed every tick the fork is healthy
116
118
  <dir>/attempt-<slot> starts of a slot that has not become healthy yet
117
119
  ```
@@ -127,6 +129,24 @@ it: if the directory is wiped, `touch!` brings the slot marks back while
127
129
  rest of the pod's life. Rewriting it is also what lets a respawned set of forks
128
130
  correct a count that has been lowered.
129
131
 
132
+ **`generation` closes a container-restart hole when the container owns PID 1.**
133
+ Kubernetes preserves an `emptyDir` when it restarts a container inside the same
134
+ pod. That is useful for an application cache elsewhere in the volume, but a
135
+ fresh heartbeat from the dead process must not let the new container pass its
136
+ one-shot `startupProbe`. On Linux, the worker and exec probe independently
137
+ derive the same incarnation from the container's mount namespace and PID 1
138
+ start time. Both `generation` and every slot mark carry it. Until the new
139
+ container declares itself and every current fork publishes its own mark, files
140
+ inherited from the previous container are rejected. Nothing outside the marks
141
+ directory is removed.
142
+
143
+ This guarantee assumes Kubernetes' default container-private PID namespace.
144
+ With `shareProcessNamespace: true` PID 1 belongs to the pod sandbox, and with
145
+ `hostPID: true` it is the node init process; neither restarts with the worker
146
+ container. Under either setting the guard can accept an inherited fresh mark.
147
+ Do not enable them on a pod whose startup probe relies on this guarantee; see
148
+ [LIMITATIONS.md](LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1).
149
+
130
150
  **Files are named by supervisor slot, not by PID.** A fork killed with SIGKILL
131
151
  is respawned into the same slot and overwrites its own file. Had the name
132
152
  contained a PID, that file would sit there stale forever and the probe would
@@ -143,12 +163,13 @@ probe sees either the old value or the new one.
143
163
  the single reason that the directory outlives the fork: a monitor caught in a
144
164
  respawn loop is a brand-new object every few hundred milliseconds and can hold
145
165
  no counter of its own. The file is removed once the slot becomes healthy, so in
146
- steady state the directory holds only `expected` and the `worker-<slot>` marks;
147
- what the counter is for is in
166
+ steady state the directory holds `expected`, `generation`, and the
167
+ `worker-<slot>` marks. What the counter is for is in
148
168
  [LIMITATIONS.md](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn).
149
169
 
150
- The *contents* of `worker-<slot>` (timestamp, pid, slot) exist only for a human
151
- running `kubectl exec ... cat`. The probe decides on mtime alone.
170
+ The timestamp, pid, and slot in `worker-<slot>` exist for a human running
171
+ `kubectl exec ... cat`. Its generation is part of the probe contract; freshness
172
+ still comes from mtime.
152
173
 
153
174
  The directory must be on tmpfs — in Kubernetes, an `emptyDir` with
154
175
  `medium: Memory`. Put it on a real disk and the probe starts depending on the
@@ -249,7 +270,7 @@ check.
249
270
 
250
271
  ## What is configurable, and where
251
272
 
252
- | | Where | Why |
273
+ | Setting | Where | Why |
253
274
  |---|---|---|
254
275
  | `logger`, `enabled`, `startup_grace_ticks` | application config block | only the worker needs them |
255
276
  | `tick` | either, and the config block wins | only the worker reads it, so two sources cannot contradict each other |
@@ -281,13 +302,27 @@ makes every mark stale on arrival, so the probe can never pass again. They fall
281
302
  back to the default too.
282
303
 
283
304
  Setting `tick` from the application is held to a stricter standard: a
284
- non-positive value raises `ArgumentError`. The environment gets a silent fallback
285
- because a ConfigMap typo must not bring a worker down, whereas an initializer is
286
- code, and code should fail loudly at boot, where the developer is looking.
305
+ non-positive value raises `ArgumentError`, as does a value greater than or equal
306
+ to `max_age`: such a monitor would inevitably let a healthy mark go stale. The
307
+ environment gets a silent fallback for values that cannot be parsed. If its
308
+ effective tick is greater than or equal to `max_age`, configuration emits a
309
+ warning on stderr and uses the default tick when it is safe, or half of `max_age`
310
+ otherwise. A ConfigMap typo must not bring a worker down, whereas an initializer
311
+ is code and should fail loudly at boot, where the developer is looking.
312
+
313
+ `startup_grace_ticks` must be a positive integer. It is compared directly with
314
+ an integer counter; accepting zero, a float, or a string would silently disable
315
+ the escalation that is supposed to diagnose a worker that never subscribes.
287
316
 
288
317
  The logger defaults to `Sneakers.logger` but is resolved lazily, because at the
289
318
  time the configuration object is built it may not be set up yet.
290
319
 
320
+ Logging is diagnostic; the heartbeat is the liveness contract. An exception
321
+ raised by a custom logger is therefore swallowed at the logging boundary. It
322
+ cannot prevent a healthy tick from writing its mark, abort monitor startup, or
323
+ escape the loop's error handler and kill the monitor thread. No fallback message
324
+ is attempted through the same broken logger.
325
+
291
326
  ## Logging: events, not the pulse
292
327
 
293
328
  The pulse lives in the mtime of a file; writing a log line every tick would only
data/docs/KUBERNETES.md CHANGED
@@ -46,6 +46,21 @@ Without `medium: Memory` an `emptyDir` is backed by the node's disk, and the
46
46
  probe starts depending on the disk again — which is one of the things it exists
47
47
  to avoid.
48
48
 
49
+ An `emptyDir` deliberately survives a restart of the container inside its pod.
50
+ That keeps a Bootsnap or other application cache under `/opt/app/tmp` warm for
51
+ the next attempt, but it also leaves the previous process's heartbeat files in
52
+ `health/`. The gem does not clear the volume or that directory. Instead it
53
+ records the current Linux container generation separately and in every slot
54
+ mark. A fresh mark from the previous container is rejected until every fork in
55
+ the current one has subscribed and published its own mark, so preserving the
56
+ cache cannot make `startupProbe` succeed early.
57
+
58
+ That guarantee assumes the default container-private PID namespace. Do not set
59
+ `shareProcessNamespace: true` or `hostPID: true` on a pod that relies on it: in
60
+ either topology PID 1 survives a worker-container restart, and the generation
61
+ guard can accept an inherited fresh mark. See
62
+ [LIMITATIONS.md](LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1).
63
+
49
64
  ## Why the command looks like that
50
65
 
51
66
  **`bundle exec kicks-liveness` is the standard command.** It works regardless
@@ -104,6 +119,12 @@ executable is found regardless of `BUNDLE_PATH`, which is why it is the standard
104
119
  command. Run bare, it depends on the gem's `bin` directory being on `PATH`,
105
120
  which is not something to rely on in a manifest.
106
121
 
122
+ Keep the worker and probe on the same gem version. Since 0.1.2 the heartbeat
123
+ protocol includes a container generation; a newer probe correctly rejects the
124
+ generation-less files written by an older worker. `bundle exec` guarantees that
125
+ both sides resolve from the same bundle. If the faster form below installs a
126
+ second copy in `GEM_HOME`, rebuild that copy on every gem upgrade too.
127
+
107
128
  ### Where your image puts its gems
108
129
 
109
130
  Before replacing the standard command with plain Ruby, run this against your
@@ -286,15 +307,18 @@ redelivered.
286
307
 
287
308
  ## Environment variables
288
309
 
289
- | Variable | Default | |
290
- |---|---|---|
291
- | `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | marks directory, **must be on tmpfs** |
292
- | `KICKS_LIVENESS_MAX_AGE` | `45` | seconds after which a mark is stale |
293
- | `KICKS_LIVENESS_TICK` | `10` | interval between ticks |
310
+ | Variable | Default | Purpose | Invalid or blank value |
311
+ |---|---|---|---|
312
+ | `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | marks directory, **must be on tmpfs** | uses the default |
313
+ | `KICKS_LIVENESS_MAX_AGE` | `45` | seconds after which a mark is stale | uses the default |
314
+ | `KICKS_LIVENESS_TICK` | `10` | interval between ticks | uses the default, then the safety fallback below if needed |
294
315
 
295
316
  Keep `tick` well below `max_age`. A tick longer than half of `max_age` leaves no
296
317
  room for a single missed write, and a tick longer than `max_age` guarantees a
297
- restart loop.
318
+ restart loop. If environment values produce `tick >= max_age`, the worker writes
319
+ a WARN to stderr and uses the default tick when that is safe, or half of
320
+ `max_age` otherwise. An initializer value with the same mismatch raises
321
+ `ArgumentError`. The half-threshold remains the recommended operational margin.
298
322
 
299
323
  `KICKS_LIVENESS_DIR` is also what separates two runners that share a pod. An
300
324
  application running both `rake sneakers:run` and `rake sneakers:active_job` has
@@ -315,9 +339,8 @@ probe command pointed at its own directory — the probe reads
315
339
  `KICKS_LIVENESS_DIR` from its own environment, which the kubelet takes from the
316
340
  container it runs in.
317
341
 
318
- Garbage in a value does not crash the worker it falls back to the default. An
319
- empty string counts as unset, which is what a ConfigMap gives you when a key is
320
- declared and left blank.
342
+ An empty string counts as unset, which is what a ConfigMap gives you when a key
343
+ is declared and left blank.
321
344
 
322
345
  ## Verifying on a live pod
323
346
 
data/docs/LIMITATIONS.md CHANGED
@@ -80,20 +80,48 @@ see the respawn escalation below.
80
80
 
81
81
  ## Changing the worker count at runtime is not supported
82
82
 
83
- `ServerEngine` re-reads its configuration on SIGHUP, and a changed `workers`
84
- value scales the fork set. Scaling **up** is fine. Scaling **down** is only
85
- handled when the supervisor also restarts the forks, which is what the restart
86
- path does: the surviving monitors re-declare the current count on their next
87
- tick and the probe follows.
88
-
89
- What cannot be handled is a reload that lowers `workers` while leaving existing
90
- forks running. `reload_config` executes in the supervisor, and a fork holds its
91
- own copy of the configuration from the moment it was forked so a fork cannot
92
- see the new number however often it looks. The declared count then stays too
93
- high, the marks for the retired slots go stale, and the probe fails until the
94
- pod restarts.
95
-
96
- If you scale workers, restart them.
83
+ `ServerEngine` can re-read its configuration on SIGHUP and scale the fork set,
84
+ but a fork holds its own copy of that configuration from the moment it was
85
+ created. Every monitor re-declares that captured count on every tick.
86
+
87
+ After a scale-up, old forks keep declaring the old count while new forks declare
88
+ the new one. The shared `expected` file therefore depends on which fork wrote
89
+ last; it can temporarily require the new slots, or incorrectly report the old
90
+ set as complete. After a scale-down, the surviving forks keep declaring the old
91
+ count and the retired slots eventually go stale. Neither direction is safe.
92
+
93
+ If you change `workers`, restart the runner. Runtime scaling is not supported.
94
+
95
+ ## Container generations require Linux procfs and container-owned PID 1
96
+
97
+ Kubernetes keeps an `emptyDir` across restarts of a container in the same pod.
98
+ To prevent the next container from inheriting a fresh heartbeat, the worker and
99
+ exec probe independently identify their shared incarnation from Linux procfs:
100
+ the mount namespace plus PID 1 start time. No application cache is removed.
101
+
102
+ The identifier also assumes that the container owns PID 1. With
103
+ `shareProcessNamespace: true`, PID 1 is the pod sandbox, and with
104
+ `hostPID: true` it is the node's init process; neither restarts when the worker
105
+ container does, so the start-time half of the identifier stays constant. The
106
+ mount namespace inode is then the only remaining signal, and the kernel
107
+ normally hands the just-released inode back to the replacement container in an
108
+ otherwise quiet pod. Under either setting the guard silently degrades to the
109
+ 0.1.1 freshness-only behavior and can accept a fresh mark from the previous
110
+ container. Do not enable either setting on a pod whose `startupProbe` relies on
111
+ this guarantee.
112
+
113
+ If procfs is unavailable — for example, when using the gem outside a Linux
114
+ container — generation detection falls back to the original freshness-only
115
+ check. The worker and probe still function, but they cannot distinguish a fresh
116
+ mark left by a previous process from one written by the current process. Linux
117
+ Kubernetes, Docker, and Nomad containers expose the required procfs entries
118
+ under their normal configuration.
119
+
120
+ The worker and its exec probe must run in the **same container**. A neighbouring
121
+ sidecar can mount the same `emptyDir`, but it has a different mount namespace
122
+ and therefore treats the worker's marks as belonging to another container.
123
+ Kubernetes exec probes already run inside the container they check; do not move
124
+ `kicks-liveness` into a separate health sidecar.
97
125
 
98
126
  ## A respawn loop is reported once per grace window, not once per respawn
99
127
 
@@ -115,6 +143,15 @@ reports the elapsed time and the number of starts:
115
143
  The number of starts is the diagnosis: it separates a slow start from a respawn
116
144
  loop at a glance.
117
145
 
146
+ This unhealthy-run counter is scoped to the marks volume, not to the container
147
+ generation. If a container is restarted before its slot has ever become
148
+ healthy, the replacement continues the same count: its repeated `started:` line
149
+ is suppressed, and an expired grace window may immediately report an ERROR that
150
+ includes starts from the previous container. This is diagnostic state only; it
151
+ does not participate in the probe result, and the first healthy tick removes
152
+ it. Treat the count as "starts since this slot was last healthy in this pod",
153
+ not "starts in this container".
154
+
118
155
  ## Do not install both `kicks` and `sneakers`
119
156
 
120
157
  The gem declares neither as a dependency, because at runtime it needs only the
@@ -126,7 +163,9 @@ executing the `sneakers` code while its lockfile says otherwise, and pulling in
126
163
  resolve.
127
164
 
128
165
  A loud failure gets fixed; a silent substitution does not. Keep exactly one of
129
- the two.
166
+ the two. Bundler itself still accepts the combination, but `install!` now
167
+ rejects a process in which both gems are activated before either set of hooks is
168
+ installed.
130
169
 
131
170
  If neither is present, `install!` raises a `LoadError` naming both with their
132
171
  required versions.
data/docs/SETUP.md CHANGED
@@ -10,7 +10,7 @@ having both is worse than it looks.
10
10
 
11
11
  Those two floors are exact, not aspirational: CI runs the suite against
12
12
  `kicks 3.0.0` and `sneakers 2.11.0` pinned, alongside the matrix that tracks the
13
- newest release of each. A `~>` matrix on its own would only ever prove that the
13
+ current release of each. A `~>` matrix on its own would only ever prove that the
14
14
  latest version works.
15
15
 
16
16
  ```ruby
@@ -142,12 +142,15 @@ Set it only to point somewhere **other** than `Sneakers.logger`:
142
142
  config.logger = Rails.logger
143
143
  ```
144
144
 
145
- | Option | Default | |
146
- |---|---|---|
147
- | `logger` | `Sneakers.logger`, resolved lazily | where transitions are logged |
148
- | `enabled` | `true` | set to `false` in tests, so no thread is started |
149
- | `tick` | `10` | seconds between checks |
150
- | `startup_grace_ticks` | `6` | unhealthy ticks tolerated at startup before one ERROR |
145
+ | Option | Default | Purpose | Valid values |
146
+ |---|---|---|---|
147
+ | `logger` | `Sneakers.logger`, resolved lazily | transition and error logs | logger-compatible object |
148
+ | `enabled` | `true` | start the monitor thread | `true` or `false` |
149
+ | `tick` | `10` | seconds between checks | positive number smaller than `max_age` |
150
+ | `startup_grace_ticks` | `6` | unhealthy startup ticks before one ERROR | positive integer |
151
+
152
+ Invalid combinations are rejected before the monitor starts instead of running
153
+ one that is guaranteed to publish stale marks or never report a stalled startup.
151
154
 
152
155
  `dir` and `max_age` are **not** here — they come from environment variables
153
156
  only. See
@@ -171,11 +174,13 @@ Once workers are running, the marks directory is the other half of the answer:
171
174
  ```
172
175
  $ ls -l /opt/app/tmp/health/
173
176
  expected
177
+ generation
174
178
  worker-0
175
179
  worker-1
176
180
  ```
177
181
 
178
- One `worker-<slot>` file per fork, plus `expected`. If `expected` is there and
179
- the slot files are not, the workers have not finished subscribing. A slot that
180
- keeps restarting without ever subscribing also leaves an `attempt-<slot>` file,
181
- which is removed as soon as that slot becomes healthy.
182
+ One `worker-<slot>` file per fork, plus `expected` and the current container
183
+ `generation`. If `expected` is there and the slot files are not, the workers
184
+ have not finished subscribing. A slot that keeps restarting without ever
185
+ subscribing also leaves an `attempt-<slot>` file, which is removed as soon as
186
+ that slot becomes healthy.
data/docs/VERIFYING.md CHANGED
@@ -30,14 +30,23 @@ it takes.
30
30
 
31
31
  ```bash
32
32
  spec/integration/verify.sh build # image, into the engine the cluster uses
33
+ spec/integration/verify.sh status # print and verify the exact target
33
34
  spec/integration/verify.sh up # namespace, broker, worker
34
35
  spec/integration/verify.sh down # deletes the namespace
35
36
  ```
36
37
 
37
- Every `kubectl` call in that script names its context and namespace explicitly.
38
- That is not politeness: a liveness experiment deletes queues and kills pods, and
39
- it must not be able to reach a real cluster because someone's current-context
40
- happened to point at one.
38
+ Every `kubectl` call names its context and namespace explicitly, so the script
39
+ never falls back to `current-context`. It will still operate on whichever target
40
+ you configure: run `status` and inspect both values before `up` or `down`. The
41
+ script labels namespaces it creates and refuses to apply to or delete an
42
+ existing namespace without that ownership label.
43
+
44
+ The default image uses Kicks. Build it with Sneakers instead without editing the
45
+ fixture:
46
+
47
+ ```bash
48
+ AMQP_WORKER_GEM=sneakers spec/integration/verify.sh build
49
+ ```
41
50
 
42
51
  Three things that cost time if you meet them the hard way:
43
52
 
@@ -88,11 +97,15 @@ spec/integration/verify.sh probe
88
97
  INFO: [liveness] slot 0: started: dir=/opt/app/tmp/health max_age=45s tick=10s processes=1 consumers=2
89
98
  INFO: [liveness] slot 0: waiting for 2 consumers
90
99
  INFO: [liveness] slot 0: healthy # one tick later
91
- expected worker-0 # in the marks directory
100
+ expected generation worker-0 # in the marks directory
92
101
  1 process(es) healthy # probe, exit 0
93
102
  ```
94
103
 
95
- The hooks fired inside a real ServerEngine fork, which no double can show.
104
+ The hooks fired inside a real ServerEngine fork, which no double can show. On a
105
+ fresh fixture the first attempt also prints `started:`. Do not use that line as
106
+ the sole proof after a same-pod container restart: an inherited unhealthy-run
107
+ counter deliberately suppresses repeated start lines, as described in
108
+ [LIMITATIONS.md](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn).
96
109
 
97
110
  ### 2. A stale mark
98
111
 
@@ -233,9 +246,9 @@ find them.
233
246
  ```
234
247
  INFO: [liveness] slot 0: started: ... processes=2 consumers=2
235
248
  INFO: [liveness] slot 1: started: ... processes=2 consumers=2
236
- expected worker-0 worker-1 # expected contains "2"
237
- 2 process(es) healthy # exit 0
238
- worker-1 stale 120s > 45s # exit 1 — staling either mark is enough
249
+ expected generation worker-0 worker-1 # expected contains "2"
250
+ 2 process(es) healthy # exit 0
251
+ worker-1 stale 120s > 45s # exit 1 — staling either mark is enough
239
252
  ```
240
253
 
241
254
  Both marks are required, not just the first.
@@ -322,6 +335,38 @@ pod is restarted. That is why the documented rule set pairs it with
322
335
  `absent_over_time`, and why an `up == 0` on the scrape job is worth having
323
336
  beside both.
324
337
 
338
+ ### 11. A private-PID container restart cannot inherit a healthy mark
339
+
340
+ This scenario needs the fixture image but not Kubernetes or RabbitMQ. It starts
341
+ a container that publishes a healthy mark into a named volume, restarts that
342
+ same container, and delays the new writer for 15 seconds:
343
+
344
+ ```bash
345
+ spec/integration/verify.sh build
346
+ spec/integration/verify.sh generation-restart
347
+ ```
348
+
349
+ During the delay the inherited mark is still fresh, but the probe must reject
350
+ its old generation. Once the replacement writer publishes its own mark, the
351
+ same probe must become healthy again. The command checks both transitions and
352
+ prints output in this form:
353
+
354
+ ```
355
+ before=mnt:[4026532686]:123456
356
+ after=mnt:[4026532686]:123789
357
+ inherited=heartbeat belongs to a previous container: worker has not started yet
358
+ recovered=1 process(es) healthy
359
+ ```
360
+
361
+ The namespace inode is allowed to be reused, as in the example above; the PID 1
362
+ start time still distinguishes the two incarnations. The test removes its
363
+ container and named volume on exit.
364
+
365
+ This scenario covers the default container-private PID namespace. With
366
+ `shareProcessNamespace: true` or `hostPID: true`, PID 1 survives the worker
367
+ container restart and this guarantee does not hold; see
368
+ [LIMITATIONS.md](LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1).
369
+
325
370
  ## What to do with a disagreement
326
371
 
327
372
  Record it, then fix whichever is wrong — the code or the document. The numbers
@@ -17,20 +17,26 @@ module KicksLiveness
17
17
  # @return [Integer] unhealthy ticks tolerated at startup before one ERROR
18
18
  DEFAULT_STARTUP_GRACE_TICKS = 6
19
19
 
20
- # @return [Integer] seconds between checks
21
- attr_reader :tick
20
+ # Seconds between checks. An incompatibility warning for an environment
21
+ # fallback is delayed until this value is actually used, so an initializer
22
+ # can override it or disable the monitor without a misleading warning.
23
+ # @return [Numeric]
24
+ def tick
25
+ warn_incompatible_environment_tick_once
26
+ @tick
27
+ end
22
28
  # @return [Integer] unhealthy ticks tolerated at startup before one ERROR
23
- attr_accessor :startup_grace_ticks
29
+ attr_reader :startup_grace_ticks
24
30
  # @return [Logger, nil] explicit logger; defaults to +Sneakers.logger+
25
31
  attr_accessor :logger
26
32
  # @param value [Boolean] set false to start no monitor thread, e.g. in tests
27
33
  attr_writer :enabled
28
34
 
29
35
  def initialize
30
- @tick = Heartbeat.env_int(:tick, DEFAULT_TICK)
31
36
  @startup_grace_ticks = DEFAULT_STARTUP_GRACE_TICKS
32
37
  @enabled = true
33
38
  @logger = nil
39
+ @tick = environment_tick
34
40
  end
35
41
 
36
42
  # @return [Boolean]
@@ -39,20 +45,36 @@ module KicksLiveness
39
45
  end
40
46
 
41
47
  # A tick is what the monitor thread sleeps on, so a non-positive value is
42
- # not a setting but a broken monitor. The environment gets a silent fallback
43
- # instead (see {Heartbeat.env_int}) because a ConfigMap typo must not bring
44
- # a worker down; an initializer is code, and code should say so at boot,
45
- # where the developer is looking.
48
+ # not a setting but a broken monitor. An initializer is code, and code
49
+ # should fail loudly at boot, where the developer is looking. Environment
50
+ # input is resolved separately by {#environment_tick}, with a safe fallback.
46
51
  #
47
- # @param seconds [Integer]
52
+ # @param seconds [Numeric]
48
53
  # @raise [ArgumentError] if not a positive number
49
- # @return [Integer]
54
+ # @return [Numeric]
50
55
  def tick=(seconds)
51
56
  raise ArgumentError, "tick must be a positive number, got #{seconds.inspect}" unless positive_number?(seconds)
57
+ raise ArgumentError, "tick must be less than max_age (#{max_age}s), got #{seconds.inspect}" if seconds >= max_age
52
58
 
59
+ @incompatible_environment_tick = nil
53
60
  @tick = seconds
54
61
  end
55
62
 
63
+ # The monitor compares an integer tick counter with this value. Accepting
64
+ # zero, a float, or a string would silently prevent the startup escalation
65
+ # from ever firing.
66
+ #
67
+ # @param ticks [Integer]
68
+ # @raise [ArgumentError] unless +ticks+ is a positive integer
69
+ # @return [Integer]
70
+ def startup_grace_ticks=(ticks)
71
+ unless ticks.is_a?(Integer) && ticks.positive?
72
+ raise ArgumentError, "startup_grace_ticks must be a positive integer, got #{ticks.inspect}"
73
+ end
74
+
75
+ @startup_grace_ticks = ticks
76
+ end
77
+
56
78
  # Read-only, sourced from the environment so that it matches what the probe
57
79
  # sees.
58
80
  # @return [String] marks directory
@@ -76,6 +98,47 @@ module KicksLiveness
76
98
 
77
99
  private
78
100
 
101
+ def environment_tick
102
+ raw = Heartbeat.env_raw(:tick)
103
+ seconds = Heartbeat.env_int(:tick, DEFAULT_TICK)
104
+ threshold = max_age
105
+ return seconds if seconds < threshold
106
+
107
+ fallback = [DEFAULT_TICK, threshold / 2.0].min
108
+ @incompatible_environment_tick = [seconds, threshold, fallback, raw]
109
+ fallback
110
+ end
111
+
112
+ def warn_incompatible_environment_tick_once
113
+ warning = @incompatible_environment_tick
114
+ return unless warning
115
+
116
+ @incompatible_environment_tick = nil
117
+ warn_incompatible_environment_tick(*warning)
118
+ end
119
+
120
+ def warn_incompatible_environment_tick(seconds, threshold, fallback, raw)
121
+ Kernel.warn(
122
+ "WARN [liveness] effective tick=#{seconds}s (#{environment_tick_source(raw)}) must be less than " \
123
+ "KICKS_LIVENESS_MAX_AGE=#{threshold}s; using #{fallback}s"
124
+ )
125
+ rescue StandardError
126
+ # Configuration recovery must not become a worker boot failure merely
127
+ # because stderr is unavailable or warning output has been overridden.
128
+ nil
129
+ end
130
+
131
+ def environment_tick_source(raw)
132
+ return 'default' if raw.nil?
133
+
134
+ value = Integer(raw)
135
+ return "KICKS_LIVENESS_TICK=#{raw}" if value.positive?
136
+
137
+ "default after invalid KICKS_LIVENESS_TICK=#{raw.inspect}"
138
+ rescue ArgumentError, TypeError
139
+ "default after invalid KICKS_LIVENESS_TICK=#{raw.inspect}"
140
+ end
141
+
79
142
  def positive_number?(value)
80
143
  value.is_a?(Numeric) && value.positive?
81
144
  end