kicks_liveness 0.1.1 → 0.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 3c784de1ca564182e9e36f8fb2956bb61a854c50995bdfe656c68805df671aed
4
- data.tar.gz: e40f05ab3385e7b6dd680e040888120693751bda5ff5b2ff2625e107c4ff7cf9
3
+ metadata.gz: df61d00933097bb96e6d752e024e349ab631cd7c9d571cd9bff9ff63b8adda40
4
+ data.tar.gz: 78864373ca06cd998cdee192d064cd00ed69a4ecd73eb1b657474d77ab593066
5
5
  SHA512:
6
- metadata.gz: e4e1088cc18b3a51493f330e2f37b3eba223dd6b21be77adcaf50833362a29b805d35f7d851ad31a2f84f2d7a9e5b69792cf443fd5cac2e8b2a44e53d70b9f29
7
- data.tar.gz: 99345f3a010f4f6c72936e80a81fdff2a6ebf0ea371596bc90850b47cfe5abacaa93694b629fd9121de386e04a5272d44b46b3425f88229ae66c2e3fc0c555f7
6
+ metadata.gz: '007091e18fea1b216ba25c634bab1a4770d25cc0f062ee30ba2885563075bc94e7662e58eebbac9507b448217508725612f172189b692e3a4d616e5f81fdd46b'
7
+ data.tar.gz: f460271ca68fdf912559b3fa2c41b02074d5eb457e946d2f985e3e2b0ac3da66dc94ce90c9d5639212a0fc7bb6537046654f31921cf6766fe15c903bd7dc04ef
data/CHANGELOG.md CHANGED
@@ -5,6 +5,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
5
5
 
6
6
  ## [Unreleased]
7
7
 
8
+ ## [0.1.2] - 2026-09-11
9
+
10
+ ### Fixed
11
+
12
+ - Reject heartbeat files inherited from a previous container incarnation. A
13
+ Kubernetes `emptyDir` survives container restarts, so a fresh mark left by a
14
+ formerly healthy worker could make the next container pass `startupProbe`
15
+ before its consumers had subscribed. Cache files elsewhere in the same
16
+ `emptyDir` remain untouched and available to warm the next application boot.
17
+ This protection assumes the default container-private PID namespace; pods
18
+ using `shareProcessNamespace` or `hostPID` retain the 0.1.1 freshness-only
19
+ behavior (see `docs/LIMITATIONS.md`).
20
+ The marks directory now contains `<dir>/generation`, and each
21
+ `worker-<slot>` mark carries the same value; tooling that enumerates the
22
+ directory should allow for this new metadata file.
23
+
8
24
  ## [0.1.1] - 2026-09-09
9
25
 
10
26
  ### Added
@@ -57,6 +73,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
57
73
  [docs/](https://github.com/PoroshkinaVV/kicks_liveness/tree/main/docs); start
58
74
  with `SETUP.md`, and read `LIMITATIONS.md` before relying on it.
59
75
 
60
- [Unreleased]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.1...HEAD
76
+ [Unreleased]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.2...HEAD
77
+ [0.1.2]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.1...v0.1.2
61
78
  [0.1.1]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.0...v0.1.1
62
79
  [0.1.0]: https://github.com/PoroshkinaVV/kicks_liveness/releases/tag/v0.1.0
data/README.md CHANGED
@@ -10,8 +10,14 @@
10
10
  Rails or querying RabbitMQ from the probe.**
11
11
 
12
12
  The worker checks its own Bunny consumers and publishes a heartbeat to tmpfs.
13
- The probe only reads that heartbeat. No application boot, no network call, and
14
- no healthy replica hiding a stalled one.
13
+ The probe only reads that heartbeat, scoped to the current container
14
+ incarnation. No application boot, no network call, and no healthy replica
15
+ hiding a stalled one.
16
+
17
+ That restart scoping assumes Kubernetes' default container-private PID
18
+ namespace; see
19
+ [Limitations](docs/LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1)
20
+ for pods that set `shareProcessNamespace` or `hostPID`.
15
21
 
16
22
  ## Install: one line in Rails
17
23
 
@@ -189,7 +195,7 @@ negative path before rollout; the commands and expected output are in
189
195
  | [Design](docs/DESIGN.md) | health predicate, heartbeat files, hooks, configuration choices, and measurements |
190
196
  | [Running under Kubernetes](docs/KUBERNETES.md) | full manifest, CPU/startup budgets, alerts, and live-pod verification |
191
197
  | [Limitations](docs/LIMITATIONS.md) | cases the in-memory predicate intentionally cannot cover |
192
- | [Verification scenarios](docs/VERIFYING.md) | ten deliberately induced failures and their observed outcomes |
198
+ | [Verification scenarios](docs/VERIFYING.md) | eleven deliberately induced failures and their observed outcomes |
193
199
 
194
200
  ## Contributing
195
201
 
data/docs/DESIGN.md CHANGED
@@ -40,16 +40,17 @@ outside it can, so let the worker publish and let the probe read.
40
40
 
41
41
  Every `tick` seconds (10 by default) the worker checks, **in its own memory**,
42
42
  that its consumers are still subscribed, and touches a file on tmpfs. The probe
43
- loads one dependency-free Ruby file, reads the file's mtime, and exits with 0 or
44
- 1.
43
+ loads one dependency-free Ruby file, checks the marks' container generation and
44
+ mtimes, and exits with 0 or 1.
45
45
 
46
46
  The probe therefore performs no network I/O and boots no framework, and the
47
- state it reads — the mark's mtime — comes from tmpfs, which is RAM. Be precise
48
- about the disk, though: starting the probe still loads the Ruby interpreter and
49
- two files of this gem from the image filesystem, and those reads are ordinary
50
- filesystem reads (usually served from page cache, but not guaranteed to be). The
51
- honest claim is not "no disk" but *no application boot, and no disk on the path
52
- that decides the answer* — which is what removes the defects above, not tuning.
47
+ state it reads — the generation and mark mtimes — comes from procfs and tmpfs.
48
+ Be precise about the disk, though: starting the probe still loads the Ruby
49
+ interpreter and two files of this gem from the image filesystem, and those reads
50
+ are ordinary filesystem reads (usually served from page cache, but not
51
+ guaranteed to be). The honest claim is not "no disk" but *no application boot,
52
+ and no disk on the path that decides the answer* — which is what removes the
53
+ defects above, not tuning.
53
54
 
54
55
  ## The health predicate
55
56
 
@@ -112,6 +113,7 @@ silently. That is the thing to re-check when upgrading Bunny.
112
113
 
113
114
  ```
114
115
  <dir>/expected how many forks the probe must wait for
116
+ <dir>/generation which container incarnation wrote this heartbeat
115
117
  <dir>/worker-<slot> one per fork, refreshed every tick the fork is healthy
116
118
  <dir>/attempt-<slot> starts of a slot that has not become healthy yet
117
119
  ```
@@ -127,6 +129,24 @@ it: if the directory is wiped, `touch!` brings the slot marks back while
127
129
  rest of the pod's life. Rewriting it is also what lets a respawned set of forks
128
130
  correct a count that has been lowered.
129
131
 
132
+ **`generation` closes a container-restart hole when the container owns PID 1.**
133
+ Kubernetes preserves an `emptyDir` when it restarts a container inside the same
134
+ pod. That is useful for an application cache elsewhere in the volume, but a
135
+ fresh heartbeat from the dead process must not let the new container pass its
136
+ one-shot `startupProbe`. On Linux, the worker and exec probe independently
137
+ derive the same incarnation from the container's mount namespace and PID 1
138
+ start time. Both `generation` and every slot mark carry it. Until the new
139
+ container declares itself and every current fork publishes its own mark, files
140
+ inherited from the previous container are rejected. Nothing outside the marks
141
+ directory is removed.
142
+
143
+ This guarantee assumes Kubernetes' default container-private PID namespace.
144
+ With `shareProcessNamespace: true` PID 1 belongs to the pod sandbox, and with
145
+ `hostPID: true` it is the node init process; neither restarts with the worker
146
+ container. Under either setting the guard can accept an inherited fresh mark.
147
+ Do not enable them on a pod whose startup probe relies on this guarantee; see
148
+ [LIMITATIONS.md](LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1).
149
+
130
150
  **Files are named by supervisor slot, not by PID.** A fork killed with SIGKILL
131
151
  is respawned into the same slot and overwrites its own file. Had the name
132
152
  contained a PID, that file would sit there stale forever and the probe would
@@ -143,12 +163,13 @@ probe sees either the old value or the new one.
143
163
  the single reason that the directory outlives the fork: a monitor caught in a
144
164
  respawn loop is a brand-new object every few hundred milliseconds and can hold
145
165
  no counter of its own. The file is removed once the slot becomes healthy, so in
146
- steady state the directory holds only `expected` and the `worker-<slot>` marks;
147
- what the counter is for is in
166
+ steady state the directory holds `expected`, `generation`, and the
167
+ `worker-<slot>` marks. What the counter is for is in
148
168
  [LIMITATIONS.md](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn).
149
169
 
150
- The *contents* of `worker-<slot>` (timestamp, pid, slot) exist only for a human
151
- running `kubectl exec ... cat`. The probe decides on mtime alone.
170
+ The timestamp, pid, and slot in `worker-<slot>` exist for a human running
171
+ `kubectl exec ... cat`. Its generation is part of the probe contract; freshness
172
+ still comes from mtime.
152
173
 
153
174
  The directory must be on tmpfs — in Kubernetes, an `emptyDir` with
154
175
  `medium: Memory`. Put it on a real disk and the probe starts depending on the
data/docs/KUBERNETES.md CHANGED
@@ -46,6 +46,21 @@ Without `medium: Memory` an `emptyDir` is backed by the node's disk, and the
46
46
  probe starts depending on the disk again — which is one of the things it exists
47
47
  to avoid.
48
48
 
49
+ An `emptyDir` deliberately survives a restart of the container inside its pod.
50
+ That keeps a Bootsnap or other application cache under `/opt/app/tmp` warm for
51
+ the next attempt, but it also leaves the previous process's heartbeat files in
52
+ `health/`. The gem does not clear the volume or that directory. Instead it
53
+ records the current Linux container generation separately and in every slot
54
+ mark. A fresh mark from the previous container is rejected until every fork in
55
+ the current one has subscribed and published its own mark, so preserving the
56
+ cache cannot make `startupProbe` succeed early.
57
+
58
+ That guarantee assumes the default container-private PID namespace. Do not set
59
+ `shareProcessNamespace: true` or `hostPID: true` on a pod that relies on it: in
60
+ either topology PID 1 survives a worker-container restart, and the generation
61
+ guard can accept an inherited fresh mark. See
62
+ [LIMITATIONS.md](LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1).
63
+
49
64
  ## Why the command looks like that
50
65
 
51
66
  **`bundle exec kicks-liveness` is the standard command.** It works regardless
@@ -104,6 +119,12 @@ executable is found regardless of `BUNDLE_PATH`, which is why it is the standard
104
119
  command. Run bare, it depends on the gem's `bin` directory being on `PATH`,
105
120
  which is not something to rely on in a manifest.
106
121
 
122
+ Keep the worker and probe on the same gem version. Since 0.1.2 the heartbeat
123
+ protocol includes a container generation; a newer probe correctly rejects the
124
+ generation-less files written by an older worker. `bundle exec` guarantees that
125
+ both sides resolve from the same bundle. If the faster form below installs a
126
+ second copy in `GEM_HOME`, rebuild that copy on every gem upgrade too.
127
+
107
128
  ### Where your image puts its gems
108
129
 
109
130
  Before replacing the standard command with plain Ruby, run this against your
data/docs/LIMITATIONS.md CHANGED
@@ -92,6 +92,37 @@ count and the retired slots eventually go stale. Neither direction is safe.
92
92
 
93
93
  If you change `workers`, restart the runner. Runtime scaling is not supported.
94
94
 
95
+ ## Container generations require Linux procfs and container-owned PID 1
96
+
97
+ Kubernetes keeps an `emptyDir` across restarts of a container in the same pod.
98
+ To prevent the next container from inheriting a fresh heartbeat, the worker and
99
+ exec probe independently identify their shared incarnation from Linux procfs:
100
+ the mount namespace plus PID 1 start time. No application cache is removed.
101
+
102
+ The identifier also assumes that the container owns PID 1. With
103
+ `shareProcessNamespace: true`, PID 1 is the pod sandbox, and with
104
+ `hostPID: true` it is the node's init process; neither restarts when the worker
105
+ container does, so the start-time half of the identifier stays constant. The
106
+ mount namespace inode is then the only remaining signal, and the kernel
107
+ normally hands the just-released inode back to the replacement container in an
108
+ otherwise quiet pod. Under either setting the guard silently degrades to the
109
+ 0.1.1 freshness-only behavior and can accept a fresh mark from the previous
110
+ container. Do not enable either setting on a pod whose `startupProbe` relies on
111
+ this guarantee.
112
+
113
+ If procfs is unavailable — for example, when using the gem outside a Linux
114
+ container — generation detection falls back to the original freshness-only
115
+ check. The worker and probe still function, but they cannot distinguish a fresh
116
+ mark left by a previous process from one written by the current process. Linux
117
+ Kubernetes, Docker, and Nomad containers expose the required procfs entries
118
+ under their normal configuration.
119
+
120
+ The worker and its exec probe must run in the **same container**. A neighbouring
121
+ sidecar can mount the same `emptyDir`, but it has a different mount namespace
122
+ and therefore treats the worker's marks as belonging to another container.
123
+ Kubernetes exec probes already run inside the container they check; do not move
124
+ `kicks-liveness` into a separate health sidecar.
125
+
95
126
  ## A respawn loop is reported once per grace window, not once per respawn
96
127
 
97
128
  When a fork cannot subscribe at all, the supervisor brings it back after
@@ -112,6 +143,15 @@ reports the elapsed time and the number of starts:
112
143
  The number of starts is the diagnosis: it separates a slow start from a respawn
113
144
  loop at a glance.
114
145
 
146
+ This unhealthy-run counter is scoped to the marks volume, not to the container
147
+ generation. If a container is restarted before its slot has ever become
148
+ healthy, the replacement continues the same count: its repeated `started:` line
149
+ is suppressed, and an expired grace window may immediately report an ERROR that
150
+ includes starts from the previous container. This is diagnostic state only; it
151
+ does not participate in the probe result, and the first healthy tick removes
152
+ it. Treat the count as "starts since this slot was last healthy in this pod",
153
+ not "starts in this container".
154
+
115
155
  ## Do not install both `kicks` and `sneakers`
116
156
 
117
157
  The gem declares neither as a dependency, because at runtime it needs only the
data/docs/SETUP.md CHANGED
@@ -174,11 +174,13 @@ Once workers are running, the marks directory is the other half of the answer:
174
174
  ```
175
175
  $ ls -l /opt/app/tmp/health/
176
176
  expected
177
+ generation
177
178
  worker-0
178
179
  worker-1
179
180
  ```
180
181
 
181
- One `worker-<slot>` file per fork, plus `expected`. If `expected` is there and
182
- the slot files are not, the workers have not finished subscribing. A slot that
183
- keeps restarting without ever subscribing also leaves an `attempt-<slot>` file,
184
- which is removed as soon as that slot becomes healthy.
182
+ One `worker-<slot>` file per fork, plus `expected` and the current container
183
+ `generation`. If `expected` is there and the slot files are not, the workers
184
+ have not finished subscribing. A slot that keeps restarting without ever
185
+ subscribing also leaves an `attempt-<slot>` file, which is removed as soon as
186
+ that slot becomes healthy.
data/docs/VERIFYING.md CHANGED
@@ -97,11 +97,15 @@ spec/integration/verify.sh probe
97
97
  INFO: [liveness] slot 0: started: dir=/opt/app/tmp/health max_age=45s tick=10s processes=1 consumers=2
98
98
  INFO: [liveness] slot 0: waiting for 2 consumers
99
99
  INFO: [liveness] slot 0: healthy # one tick later
100
- expected worker-0 # in the marks directory
100
+ expected generation worker-0 # in the marks directory
101
101
  1 process(es) healthy # probe, exit 0
102
102
  ```
103
103
 
104
- The hooks fired inside a real ServerEngine fork, which no double can show.
104
+ The hooks fired inside a real ServerEngine fork, which no double can show. On a
105
+ fresh fixture the first attempt also prints `started:`. Do not use that line as
106
+ the sole proof after a same-pod container restart: an inherited unhealthy-run
107
+ counter deliberately suppresses repeated start lines, as described in
108
+ [LIMITATIONS.md](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn).
105
109
 
106
110
  ### 2. A stale mark
107
111
 
@@ -242,9 +246,9 @@ find them.
242
246
  ```
243
247
  INFO: [liveness] slot 0: started: ... processes=2 consumers=2
244
248
  INFO: [liveness] slot 1: started: ... processes=2 consumers=2
245
- expected worker-0 worker-1 # expected contains "2"
246
- 2 process(es) healthy # exit 0
247
- worker-1 stale 120s > 45s # exit 1 — staling either mark is enough
249
+ expected generation worker-0 worker-1 # expected contains "2"
250
+ 2 process(es) healthy # exit 0
251
+ worker-1 stale 120s > 45s # exit 1 — staling either mark is enough
248
252
  ```
249
253
 
250
254
  Both marks are required, not just the first.
@@ -331,6 +335,38 @@ pod is restarted. That is why the documented rule set pairs it with
331
335
  `absent_over_time`, and why an `up == 0` on the scrape job is worth having
332
336
  beside both.
333
337
 
338
+ ### 11. A private-PID container restart cannot inherit a healthy mark
339
+
340
+ This scenario needs the fixture image but not Kubernetes or RabbitMQ. It starts
341
+ a container that publishes a healthy mark into a named volume, restarts that
342
+ same container, and delays the new writer for 15 seconds:
343
+
344
+ ```bash
345
+ spec/integration/verify.sh build
346
+ spec/integration/verify.sh generation-restart
347
+ ```
348
+
349
+ During the delay the inherited mark is still fresh, but the probe must reject
350
+ its old generation. Once the replacement writer publishes its own mark, the
351
+ same probe must become healthy again. The command checks both transitions and
352
+ prints output in this form:
353
+
354
+ ```
355
+ before=mnt:[4026532686]:123456
356
+ after=mnt:[4026532686]:123789
357
+ inherited=heartbeat belongs to a previous container: worker has not started yet
358
+ recovered=1 process(es) healthy
359
+ ```
360
+
361
+ The namespace inode is allowed to be reused, as in the example above; the PID 1
362
+ start time still distinguishes the two incarnations. The test removes its
363
+ container and named volume on exit.
364
+
365
+ This scenario covers the default container-private PID namespace. With
366
+ `shareProcessNamespace: true` or `hostPID: true`, PID 1 survives the worker
367
+ container restart and this guarantee does not hold; see
368
+ [LIMITATIONS.md](LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1).
369
+
334
370
  ## What to do with a disagreement
335
371
 
336
372
  Record it, then fix whichever is wrong — the code or the document. The numbers
@@ -1,5 +1,57 @@
1
1
  module KicksLiveness
2
- # The liveness mark on the filesystem: written by the worker, read by the
2
+ # Identifies an ordinary Linux container incarnation shared by the worker and
3
+ # exec probes without writing anything to the marks directory. This assumes
4
+ # the container owns PID 1; shared and host PID namespaces are documented as
5
+ # unsupported in +docs/LIMITATIONS.md+.
6
+ # @api private
7
+ module ContainerGeneration
8
+ module_function
9
+
10
+ # @return [String, nil] current private-PID Linux container incarnation, or
11
+ # nil when procfs does not expose one
12
+ # @api private
13
+ def current
14
+ mount_namespace = File.readlink('/proc/self/ns/mnt')
15
+ stat = File.read('/proc/1/stat')
16
+ closing_parenthesis = stat.rindex(') ')
17
+ return unless closing_parenthesis
18
+
19
+ # After the command in parentheses, field 3 (`state`) is index 0. Process
20
+ # start time is field 22, therefore index 19 in this tail.
21
+ started_at = stat[(closing_parenthesis + 2)..].split[19]
22
+ return unless started_at && Integer(started_at).positive?
23
+
24
+ "#{mount_namespace}:#{started_at}"
25
+ rescue StandardError
26
+ nil
27
+ end
28
+ end
29
+
30
+ # Generation-specific parts of the filesystem contract.
31
+ # @api private
32
+ module GenerationGuard
33
+ private
34
+
35
+ def generation_path
36
+ File.join(@dir, 'generation')
37
+ end
38
+
39
+ def declared_generation
40
+ File.read(generation_path)
41
+ rescue StandardError
42
+ nil
43
+ end
44
+
45
+ def current_generation?
46
+ !@generation || declared_generation == @generation
47
+ end
48
+
49
+ def previous_generation?(path)
50
+ @generation && File.read(path)[/\bgeneration=(\S+)/, 1] != @generation
51
+ end
52
+ end
53
+
54
+ # The liveness marks on the filesystem: written by the worker, read by the
3
55
  # probe.
4
56
  #
5
57
  # The directory must live on tmpfs — in Kubernetes, an emptyDir with
@@ -17,6 +69,8 @@ module KicksLiveness
17
69
  #
18
70
  # @see file:docs/DESIGN.md#why-the-heartbeat-file-has-no-require-of-its-own
19
71
  class Heartbeat
72
+ include GenerationGuard
73
+
20
74
  # @return [String] marks directory used when the environment says nothing
21
75
  DEFAULT_DIR = '/opt/app/tmp/health'.freeze
22
76
  # @return [Integer] seconds after which a mark is stale, by default
@@ -29,6 +83,16 @@ module KicksLiveness
29
83
  tick: 'KICKS_LIVENESS_TICK'
30
84
  }.freeze
31
85
 
86
+ # Linux exposes a stable identifier shared by a private-PID container and
87
+ # its exec probes. A restarted container gets a new identifier even though
88
+ # its Kubernetes emptyDir survives.
89
+ # @return [String, nil] current private-PID container incarnation, or nil
90
+ # off Linux
91
+ # @api private
92
+ def self.container_generation
93
+ ContainerGeneration.current
94
+ end
95
+
32
96
  # An empty string counts as unset: in a ConfigMap that is what you get by
33
97
  # declaring a key and leaving it blank.
34
98
  #
@@ -72,9 +136,15 @@ module KicksLiveness
72
136
 
73
137
  # @param dir [String] marks directory
74
138
  # @param max_age [Integer] seconds after which a mark is considered stale
75
- def initialize(dir: Heartbeat.env_dir, max_age: Heartbeat.env_max_age)
139
+ # @param generation [String, nil] container incarnation; injected in specs
140
+ def initialize(
141
+ dir: Heartbeat.env_dir,
142
+ max_age: Heartbeat.env_max_age,
143
+ generation: Heartbeat.container_generation
144
+ )
76
145
  @dir = dir
77
146
  @max_age = max_age
147
+ @generation = generation
78
148
  end
79
149
 
80
150
  attr_reader :dir, :max_age
@@ -93,10 +163,8 @@ module KicksLiveness
93
163
  # @return [void]
94
164
  def declare!(processes)
95
165
  make_dir
96
- # The pid keeps concurrent forks from sharing the temporary file.
97
- tmp = "#{expected_path}.#{Process.pid}"
98
- File.write(tmp, processes)
99
- File.rename(tmp, expected_path)
166
+ atomic_write(generation_path, @generation) if @generation
167
+ atomic_write(expected_path, processes)
100
168
  end
101
169
 
102
170
  # Refreshes this fork's mark.
@@ -106,14 +174,18 @@ module KicksLiveness
106
174
  # PID in the name that file would stay stale forever and the probe would fail
107
175
  # permanently.
108
176
  #
109
- # The contents exist only for a human running <tt>kubectl exec ... cat</tt>;
110
- # the probe decides on mtime alone.
177
+ # The timestamp, pid and slot exist for a human running
178
+ # <tt>kubectl exec ... cat</tt>. The generation is also checked by the probe:
179
+ # a Kubernetes emptyDir survives a container restart, so freshness alone
180
+ # cannot distinguish this process from the one that just exited.
111
181
  #
112
182
  # @param slot [Integer] supervisor slot of this fork
113
183
  # @return [Integer] bytes written
114
184
  def touch!(slot)
115
185
  make_dir
116
- File.write(slot_path(slot), "#{Time.now.utc.strftime('%FT%TZ')} pid=#{Process.pid} slot=#{slot}\n")
186
+ contents = "#{Time.now.utc.strftime('%FT%TZ')} pid=#{Process.pid} slot=#{slot}"
187
+ contents = "#{contents} generation=#{@generation}" if @generation
188
+ atomic_write(slot_path(slot), "#{contents}\n")
117
189
  end
118
190
 
119
191
  # The probe side: is every declared fork's mark present and fresh?
@@ -126,13 +198,9 @@ module KicksLiveness
126
198
  def check(now: Time.now.utc)
127
199
  processes = expected
128
200
  return [false, "no #{expected_path}: worker has not started yet"] unless processes&.positive?
201
+ return [false, 'heartbeat belongs to a previous container: worker has not started yet'] unless current_generation?
129
202
 
130
- problems = (0...processes).filter_map do |slot|
131
- age = age_of(slot_path(slot), now)
132
- next "worker-#{slot} missing" if age.nil?
133
-
134
- "worker-#{slot} stale #{age.round}s > #{@max_age}s" if age > @max_age
135
- end
203
+ problems = (0...processes).filter_map { |slot| problem_for(slot, now) }
136
204
 
137
205
  problems.empty? ? [true, "#{processes} process(es) healthy"] : [false, problems.join('; ')]
138
206
  end
@@ -169,6 +237,25 @@ module KicksLiveness
169
237
  nil
170
238
  end
171
239
 
240
+ def problem_for(slot, now)
241
+ path = slot_path(slot)
242
+ age = age_of(path, now)
243
+ return "worker-#{slot} missing" if age.nil?
244
+ return "worker-#{slot} belongs to a previous container" if previous_generation?(path)
245
+
246
+ "worker-#{slot} stale #{age.round}s > #{@max_age}s" if age > @max_age
247
+ rescue StandardError
248
+ "worker-#{slot} unreadable"
249
+ end
250
+
251
+ def atomic_write(path, contents)
252
+ # The pid keeps concurrent forks from sharing the temporary file.
253
+ tmp = "#{path}.#{Process.pid}"
254
+ bytes = File.write(tmp, contents)
255
+ File.rename(tmp, path)
256
+ bytes
257
+ end
258
+
172
259
  def age_of(path, now)
173
260
  now - File.mtime(path)
174
261
  rescue StandardError
@@ -1,4 +1,4 @@
1
1
  module KicksLiveness
2
2
  # @return [String] gem version
3
- VERSION = '0.1.1'.freeze
3
+ VERSION = '0.1.2'.freeze
4
4
  end
@@ -9,8 +9,8 @@ require_relative 'kicks_liveness/hooks'
9
9
  # Liveness probe for Kicks and Sneakers workers, backed by a tmpfs heartbeat.
10
10
  #
11
11
  # The worker publishes a mark from inside its own process, checking its Bunny
12
- # consumers in memory; the probe reads only the mark's mtime. No Rails, no call
13
- # to the broker.
12
+ # consumers in memory; the probe reads only the container generation and mark
13
+ # mtimes. No Rails, no call to the broker.
14
14
  #
15
15
  # @see file:docs/SETUP.md
16
16
  # @see file:docs/DESIGN.md
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: kicks_liveness
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.1
4
+ version: 0.1.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - PoroshkinaVV
@@ -11,12 +11,8 @@ date: 1980-01-02 00:00:00.000000000 Z
11
11
  dependencies: []
12
12
  description: |
13
13
  A liveness probe for RabbitMQ worker pods that loads no Rails and never talks
14
- to the broker. The worker publishes a heartbeat to tmpfs from inside its own
15
- process, checking its Bunny consumers in memory; the probe only reads the
16
- file's mtime and runs as `bundle exec kicks-liveness`. It loads no application
17
- code; aside from Bundler and the interpreter it loads only the gem's small
18
- probe files, while the state itself comes from tmpfs. A broker hiccup cannot
19
- restart every replica at once.
14
+ to the broker. Workers check their Bunny consumers in memory and publish a
15
+ heartbeat to tmpfs; the probe only reads that heartbeat.
20
16
  email:
21
17
  - lera.poroshkina@mail.ru
22
18
  executables:
@@ -49,7 +45,7 @@ licenses:
49
45
  - MIT
50
46
  metadata:
51
47
  source_code_uri: https://github.com/PoroshkinaVV/kicks_liveness
52
- documentation_uri: https://rubydoc.info/gems/kicks_liveness/0.1.1
48
+ documentation_uri: https://rubydoc.info/gems/kicks_liveness/0.1.2
53
49
  changelog_uri: https://github.com/PoroshkinaVV/kicks_liveness/blob/main/CHANGELOG.md
54
50
  bug_tracker_uri: https://github.com/PoroshkinaVV/kicks_liveness/issues
55
51
  rubygems_mfa_required: 'true'