kicks_liveness 0.1.2 → 0.1.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: df61d00933097bb96e6d752e024e349ab631cd7c9d571cd9bff9ff63b8adda40
4
- data.tar.gz: 78864373ca06cd998cdee192d064cd00ed69a4ecd73eb1b657474d77ab593066
3
+ metadata.gz: faf817806eb5142b8271959b7856a16db10a99db8eed583bb5594887198b6dd8
4
+ data.tar.gz: 9c5f3f845c78a56e5da97204769e5c2b10161fbf56e5c40a731b904650ca469a
5
5
  SHA512:
6
- metadata.gz: '007091e18fea1b216ba25c634bab1a4770d25cc0f062ee30ba2885563075bc94e7662e58eebbac9507b448217508725612f172189b692e3a4d616e5f81fdd46b'
7
- data.tar.gz: f460271ca68fdf912559b3fa2c41b02074d5eb457e946d2f985e3e2b0ac3da66dc94ce90c9d5639212a0fc7bb6537046654f31921cf6766fe15c903bd7dc04ef
6
+ metadata.gz: a077fc085bed4c460ed9820940271fedb44e69a2778ca87997b23c172fe94d0622bbf334d9b5456827d1e231813ec5e903de6ebe6c655804f8b8c359ca26832c
7
+ data.tar.gz: 1d2efdab9aa59777cc14075703c9c75bfad76dc511d94659a8f2f40d2cc8a3a957779e4795f304beabbed04c4854f88e2413fd044216497df7338d6a7a32a864
data/CHANGELOG.md CHANGED
@@ -5,6 +5,77 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
5
5
 
6
6
  ## [Unreleased]
7
7
 
8
+ ## [0.1.4] - 2026-11-03
9
+
10
+ ### Fixed
11
+
12
+ - Bound the graceful-shutdown heartbeat exemption by a monotonic deadline
13
+ starting at the first stop notification; repeated callbacks cannot extend it.
14
+ At expiry the monitor logs one ERROR and stops refreshing its mark, but still
15
+ declares the fork count. A stuck drain then fails the ordinary freshness
16
+ check; the monitor does not kill the process.
17
+
18
+ ### Added
19
+
20
+ - Add `shutdown_grace` / `KICKS_LIVENESS_SHUTDOWN_GRACE` (default 60 seconds;
21
+ initializer wins). Set it to at least `terminationGracePeriodSeconds`.
22
+ Restart still waits for mark staleness and the kubelet's probe failure budget.
23
+ - Add `verify.sh shutdown` and `container-restart` to check a real drain and
24
+ same-pod container replacement, with deployment restoration afterwards.
25
+ - Add an `install!` setup hint when the probe finds no declared worker count.
26
+ It remains a failing check, not proof that the hooks were never installed.
27
+
28
+ ## [0.1.3] - 2026-10-21
29
+
30
+ ### Fixed
31
+
32
+ - Detect and reject PID topologies in which the container generation cannot
33
+ identify an incarnation. `shareProcessNamespace: true` and `hostPID: true`
34
+ leave PID 1 alive across a restart of the worker container, so the 0.1.2
35
+ identifier can repeat and an inherited fresh heartbeat could satisfy
36
+ `startupProbe`. Rejection is conditional on a readable `/proc/1/ns/mnt`.
37
+ Where the comparison can be made, the worker publishes `<dir>/topology-unsafe`
38
+ instead of a heartbeat and every probe exits 1 with
39
+ `unsupported PID topology: pod must not set shareProcessNamespace or hostPID`,
40
+ including one that cannot read the link itself; the state is terminal for the
41
+ marks volume, and recovery is recreating the pod without those settings.
42
+ Where it cannot, the previous behavior remains and the worker warns once. The
43
+ measured outcomes are in `docs/LIMITATIONS.md`.
44
+ - Distinguish a valid zero in PID 1's stat start-time field from a stat line
45
+ that cannot be parsed: the generation stays unavailable, and the warning
46
+ names `hostPID` or a virtualized procfs instead of a false parse error.
47
+ - Scope the `attempt-<slot>` respawn counter to the container generation. A
48
+ replacement container starts its own unhealthy run and cannot escalate over
49
+ starts it did not take, while respawns within one generation go on sharing a
50
+ count. Where no generation is available the counter keeps its previous
51
+ volume-wide behavior.
52
+
53
+ ### Added
54
+
55
+ - Report a degraded container generation once from the worker, at WARN — once
56
+ per container incarnation where a generation exists, once per error state
57
+ where it does not. Neither degraded state adds anything to the probe's
58
+ output, and the intentional non-Linux fallback stays silent.
59
+ - The marks directory gains two files that appear only when something is
60
+ wrong: `topology-unsafe`, the terminal fail-closed state, and
61
+ `topology-report`, the worker's own flag for saying each thing once. Tooling
62
+ that enumerates the directory should allow for both.
63
+ - Document the boundary of the graceful-shutdown exemption in
64
+ `docs/LIMITATIONS.md`: a SIGTERM that does not come from a pod deletion can
65
+ leave a pod Ready with no consumers. Existing behavior, newly documented.
66
+ - Add `spec/integration/verify.sh topology`, a Linux matrix covering a shared
67
+ PID namespace, the host PID namespace, a verified private-PID container, and
68
+ a private-PID container whose checker cannot read `/proc/1/ns/mnt`.
69
+
70
+ ### Changed
71
+
72
+ - `Heartbeat#declare!` documents its actual `Integer` return value.
73
+ - Replace `Heartbeat.container_generation` with `Heartbeat.topology`, returning
74
+ a structured result instead of a string or `nil`; replace the `Heartbeat.new`
75
+ keyword `generation:` with `topology:`, accepting that result; and remove
76
+ `KicksLiveness::GenerationGuard`. All are `@api private`, but callers need
77
+ updating.
78
+
8
79
  ## [0.1.2] - 2026-09-11
9
80
 
10
81
  ### Fixed
@@ -73,7 +144,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
73
144
  [docs/](https://github.com/PoroshkinaVV/kicks_liveness/tree/main/docs); start
74
145
  with `SETUP.md`, and read `LIMITATIONS.md` before relying on it.
75
146
 
76
- [Unreleased]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.2...HEAD
147
+ [Unreleased]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.4...HEAD
148
+ [0.1.4]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.3...v0.1.4
149
+ [0.1.3]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.2...v0.1.3
77
150
  [0.1.2]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.1...v0.1.2
78
151
  [0.1.1]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.0...v0.1.1
79
152
  [0.1.0]: https://github.com/PoroshkinaVV/kicks_liveness/releases/tag/v0.1.0
data/README.md CHANGED
@@ -14,10 +14,15 @@ The probe only reads that heartbeat, scoped to the current container
14
14
  incarnation. No application boot, no network call, and no healthy replica
15
15
  hiding a stalled one.
16
16
 
17
- That restart scoping assumes Kubernetes' default container-private PID
18
- namespace; see
19
- [Limitations](docs/LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1)
20
- for pods that set `shareProcessNamespace` or `hostPID`.
17
+ That restart scoping needs Kubernetes' default container-private PID
18
+ namespace. Where the namespace links are readable, a pod that sets
19
+ `shareProcessNamespace` or `hostPID` is rejected outright. With the default
20
+ capability set they were not readable on the measured Kubernetes node: the
21
+ shared-PID pod was `unverifiable`, while the host-PID pod had no usable
22
+ generation because node init reported a zero start time. Both degradations warn
23
+ instead of pretending the restart guarantee still holds; see
24
+ [Limitations](docs/LIMITATIONS.md#container-generations-need-linux-procfs-and-a-private-pid-namespace)
25
+ for what each side can prove and what it does when it cannot.
21
26
 
22
27
  ## Install: one line in Rails
23
28
 
@@ -72,6 +77,9 @@ only when every expected worker mark is present and fresh.
72
77
  Use a startup probe for the boot budget and a liveness probe for steady state:
73
78
 
74
79
  ```yaml
80
+ env:
81
+ - name: KICKS_LIVENESS_SHUTDOWN_GRACE
82
+ value: "60" # at least terminationGracePeriodSeconds
75
83
  startupProbe:
76
84
  exec:
77
85
  command: ["bundle", "exec", "kicks-liveness"]
@@ -134,6 +142,7 @@ environment too:
134
142
  | `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | heartbeat directory; must be on tmpfs | uses the default |
135
143
  | `KICKS_LIVENESS_MAX_AGE` | `45` | seconds before a mark is stale | uses the default |
136
144
  | `KICKS_LIVENESS_TICK` | `10` | seconds between worker checks | uses the default, then the safety fallback below if needed |
145
+ | `KICKS_LIVENESS_SHUTDOWN_GRACE` | `60` | seconds to keep publishing during shutdown | uses the default |
137
146
 
138
147
  If the effective environment tick is not smaller than `max_age`, the worker
139
148
  writes a WARN to stderr and uses `10` seconds when that is safe, or half of
@@ -141,10 +150,16 @@ writes a WARN to stderr and uses `10` seconds when that is safe, or half of
141
150
  while guaranteeing that a healthy heartbeat cannot become stale between
142
151
  ordinary ticks.
143
152
 
144
- An initializer value for `tick` wins over `KICKS_LIVENESS_TICK`. `dir` and
153
+ Initializer values for `tick` and `shutdown_grace` override their environment
154
+ variables. `dir` and
145
155
  `max_age` deliberately have no initializer setters: allowing two configuration
146
156
  sources could make the worker and probe silently disagree.
147
157
 
158
+ Set `shutdown_grace` to at least `terminationGracePeriodSeconds`. A drain that
159
+ exceeds this deadline stops refreshing its heartbeat and logs one ERROR; the
160
+ probe fails once the last mark becomes stale. See
161
+ [shutdown timing](docs/LIMITATIONS.md#graceful-shutdown-has-a-finite-heartbeat-exemption).
162
+
148
163
  If two runners share one pod — for example `sneakers:run` and
149
164
  `sneakers:active_job` — give each a different `KICKS_LIVENESS_DIR`. Separate
150
165
  pods already have separate `emptyDir` volumes.
@@ -195,7 +210,7 @@ negative path before rollout; the commands and expected output are in
195
210
  | [Design](docs/DESIGN.md) | health predicate, heartbeat files, hooks, configuration choices, and measurements |
196
211
  | [Running under Kubernetes](docs/KUBERNETES.md) | full manifest, CPU/startup budgets, alerts, and live-pod verification |
197
212
  | [Limitations](docs/LIMITATIONS.md) | cases the in-memory predicate intentionally cannot cover |
198
- | [Verification scenarios](docs/VERIFYING.md) | eleven deliberately induced failures and their observed outcomes |
213
+ | [Verification scenarios](docs/VERIFYING.md) | twelve deliberately exercised scenarios and their observed outcomes |
199
214
 
200
215
  ## Contributing
201
216
 
data/docs/DESIGN.md CHANGED
@@ -74,9 +74,8 @@ channel.open? && connection.open? && channel.any_consumers?
74
74
  | connection open, no consumers | unhealthy | the worker silently stopped consuming — a restart is the only cure |
75
75
  | not the whole set subscribed | unhealthy | a worker that quietly dropped out would otherwise leave the probe green |
76
76
 
77
- The last row is the reason the registry compares its size against an expected
78
- count rather than just checking that whatever registered is alive. Four healthy
79
- workers out of five look perfectly fine one object at a time.
77
+ The expected count catches partial subscription: four healthy workers out of
78
+ five must still make the process unhealthy.
80
79
 
81
80
  ### Nothing here talks to the broker
82
81
 
@@ -101,84 +100,76 @@ The predicate learns that a consumer was cancelled because RabbitMQ sends
101
100
  push, not a poll — which is what makes an in-memory predicate possible at all,
102
101
  and also where its limits come from. See [LIMITATIONS.md](LIMITATIONS.md).
103
102
 
104
- `recovering_from_network_failure?` is marked `@private` in Bunny's own
105
- documentation, so an upgrade may remove it. It is therefore called through
106
- `respond_to?`: without that guard the call would raise `NoMethodError`, `alive?`
107
- would catch it and report *unhealthy*, and every pod would enter a restart loop
108
- on the next Bunny upgrade. What the guard cannot preserve is the exemption
109
- itself — if the method disappears, so does "healthy while reconnecting",
110
- silently. That is the thing to re-check when upgrading Bunny.
103
+ `recovering_from_network_failure?` is private Bunny API. The `respond_to?` guard
104
+ avoids turning its removal into a rescued `NoMethodError` and an unhealthy
105
+ verdict on every tick. Removal would still disable the recovery exemption
106
+ silently; re-check this API when upgrading Bunny.
111
107
 
112
108
  ## The mark files
113
109
 
114
110
  ```
115
- <dir>/expected how many forks the probe must wait for
116
- <dir>/generation which container incarnation wrote this heartbeat
117
- <dir>/worker-<slot> one per fork, refreshed every tick the fork is healthy
118
- <dir>/attempt-<slot> starts of a slot that has not become healthy yet
111
+ <dir>/expected how many forks the probe must wait for
112
+ <dir>/generation which container incarnation wrote this heartbeat
113
+ <dir>/worker-<slot> one per fork, refreshed every tick the fork is healthy
114
+ <dir>/attempt-<slot> starts of a slot that has not become healthy yet
115
+ <dir>/topology-unsafe the PID namespace is not this container's: fail closed
116
+ <dir>/topology-report what the worker has already reported about it, once
119
117
  ```
120
118
 
121
- **`expected` closes a startup hole.** Without it the probe would pass as soon as
122
- any single file was fresh: fork 0 has written, fork 1 has not, and the pod looks
123
- ready while half of it is not subscribed. Every fork writes the same value, so
124
- they cannot disagree.
125
-
126
- Every fork rewrites it on every tick, not only at startup. Nothing else restores
127
- it: if the directory is wiped, `touch!` brings the slot marks back while
128
- `expected` stays missing, and the probe reports "worker has not started" for the
129
- rest of the pod's life. Rewriting it is also what lets a respawned set of forks
130
- correct a count that has been lowered.
131
-
132
- **`generation` closes a container-restart hole when the container owns PID 1.**
133
- Kubernetes preserves an `emptyDir` when it restarts a container inside the same
134
- pod. That is useful for an application cache elsewhere in the volume, but a
135
- fresh heartbeat from the dead process must not let the new container pass its
136
- one-shot `startupProbe`. On Linux, the worker and exec probe independently
137
- derive the same incarnation from the container's mount namespace and PID 1
138
- start time. Both `generation` and every slot mark carry it. Until the new
139
- container declares itself and every current fork publishes its own mark, files
140
- inherited from the previous container are rejected. Nothing outside the marks
141
- directory is removed.
142
-
143
- This guarantee assumes Kubernetes' default container-private PID namespace.
144
- With `shareProcessNamespace: true` PID 1 belongs to the pod sandbox, and with
145
- `hostPID: true` it is the node init process; neither restarts with the worker
146
- container. Under either setting the guard can accept an inherited fresh mark.
147
- Do not enable them on a pod whose startup probe relies on this guarantee; see
148
- [LIMITATIONS.md](LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1).
149
-
150
- **Files are named by supervisor slot, not by PID.** A fork killed with SIGKILL
151
- is respawned into the same slot and overwrites its own file. Had the name
152
- contained a PID, that file would sit there stale forever and the probe would
153
- fail permanently, turning one dead fork into a pod that can never come back.
154
-
155
- `expected` is written through a temporary file and a `rename`, not with a plain
156
- write. A plain write truncates first, and a probe reading in that window finds
157
- the file empty and reports that the worker has not started. The window is real
158
- and recurring: every fork rewrites this file on every tick, while the liveness
159
- probe reads it on a schedule of its own. `rename` is atomic on tmpfs, so the
160
- probe sees either the old value or the new one.
161
-
162
- **`attempt-<slot>` is the respawn counter**, and it lives in this directory for
163
- the single reason that the directory outlives the fork: a monitor caught in a
164
- respawn loop is a brand-new object every few hundred milliseconds and can hold
165
- no counter of its own. The file is removed once the slot becomes healthy, so in
166
- steady state the directory holds `expected`, `generation`, and the
167
- `worker-<slot>` marks. What the counter is for is in
119
+ **`expected` prevents partial startup from passing.** Checking only existing
120
+ marks could accept fork 0's fresh mark before fork 1 has subscribed. Declaring
121
+ the count tells the probe which marks must exist, not just which are fresh.
122
+ With a fixed runner configuration, every fork declares the same count.
123
+ Re-declaring on each tick also restores `expected` if the directory is
124
+ wiped; refreshing slot marks alone would leave the probe permanently unhealthy.
125
+ Live worker-count changes remain
126
+ [unsupported](LIMITATIONS.md#changing-the-worker-count-at-runtime-is-not-supported).
127
+
128
+ **`generation` rejects inherited heartbeats.** With usable Linux procfs and a
129
+ container-private PID namespace, the worker and exec probe derive it from the
130
+ mount namespace and PID 1 start time. Both `generation` and every slot mark must
131
+ match the current incarnation, so an `emptyDir` preserved across a container
132
+ restart cannot pass `startupProbe` until all replacement forks publish. No
133
+ application cache is removed. Procfs and PID-topology limits are described in
134
+ [LIMITATIONS.md](LIMITATIONS.md#container-generations-need-linux-procfs-and-a-private-pid-namespace).
135
+
136
+ **`topology-unsafe` carries terminal evidence from the worker.** Each participant
137
+ compares `/proc/1/ns/mnt` with `/proc/self/ns/mnt` independently. A worker that
138
+ proves them different writes this file instead of a heartbeat; the probe checks
139
+ it before `expected`, generation, or freshness. Omitting a new generation would
140
+ leave inherited marks eligible for acceptance. Weaker evidence never clears
141
+ this file.
142
+
143
+ **`topology-report` deduplicates worker diagnostics across forks.** The probe
144
+ ignores it. Claims lock the complete read-compare-write operation; atomic rename
145
+ alone cannot prevent concurrent claims. Clearing by truncation keeps every fork
146
+ locking the same inode.
147
+
148
+ **Slot names survive fork replacement.** A respawn overwrites its predecessor's
149
+ mark. PID-based names would leave stale files that permanently fail the probe.
150
+
151
+ `expected` and slot marks are published through temporary files and `rename`.
152
+ Concurrent probes therefore see complete old or new contents, never a truncated
153
+ count or a mark missing its generation. Temporary filenames include the writer's
154
+ PID so concurrent forks cannot overwrite one another's temporary files.
155
+
156
+ **`attempt-<slot>` persists an unhealthy run across fork respawns.** Its
157
+ generation rejects counts from a previous container when one can be derived.
158
+ The first healthy tick removes it. Reporting windows and generation-less scope
159
+ are explained in
168
160
  [LIMITATIONS.md](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn).
169
161
 
170
162
  The timestamp, pid, and slot in `worker-<slot>` exist for a human running
171
163
  `kubectl exec ... cat`. Its generation is part of the probe contract; freshness
172
164
  still comes from mtime.
173
165
 
174
- The directory must be on tmpfs — in Kubernetes, an `emptyDir` with
175
- `medium: Memory`. Put it on a real disk and the probe starts depending on the
176
- disk again, which was defect number two.
166
+ Use tmpfs — an `emptyDir` with `medium: Memory` in Kubernetes — to keep mark
167
+ reads independent of disk stalls.
177
168
 
178
169
  ## Why the heartbeat file has no `require` of its own
179
170
 
180
- The probe loads exactly one file and nothing else. Measured over 30 invocations
181
- each; absolute values depend on the hardware, but the ratios do not:
171
+ The probe entry point loads only `heartbeat.rb`. Recorded over 30 invocations
172
+ each; timings vary with the runtime and hardware:
182
173
 
183
174
  | | per invocation |
184
175
  |---|---|
@@ -188,22 +179,13 @@ each; absolute values depend on the hardware, but the ratios do not:
188
179
  | the probe, found through RubyGems | 43 ms |
189
180
  | **`bundle exec ruby -e ''`** | **300 ms** |
190
181
 
191
- Subtract the baselines and the probe's own work is a **few milliseconds**.
192
- Everything else is interpreter startup, and 28 ms of that is RubyGems
193
- initializing.
194
-
195
- Three consequences follow. `bundle exec kicks-liveness` is the portable
196
- default; most of its runtime is Bundler startup rather than the check itself.
197
- When the gem is installed in `GEM_HOME`, RubyGems can find it with a plain
198
- `require` for under 50 ms. Skipping RubyGems entirely is faster still, but it
199
- requires naming the path to the gem's `lib` directory; see
200
- [KUBERNETES.md](KUBERNETES.md) for when that trade is worth making.
201
-
202
- Keeping the probe's path through the heartbeat file free of `require`
203
- statements — and the file free of references to the rest of the gem — is what
204
- makes all three possible. The single `require` the file does contain,
205
- `fileutils`, is lazy and sits in the directory-creation branch that only the
206
- worker ever reaches.
182
+ Interpreter and dependency startup dominate these measurements. `bundle exec`
183
+ is the portable default; direct loading depends on where the image installs
184
+ the gem, as described in [KUBERNETES.md](KUBERNETES.md#where-your-image-puts-its-gems).
185
+ `heartbeat.rb` therefore references no other gem code and requires `fileutils`
186
+ only in the worker's directory-creation branch. Worker-only counters and report
187
+ deduplication live in `Attempts` and `TopologyReport` so their code and requires
188
+ stay outside the probe's load path.
207
189
 
208
190
  ## Where the hooks attach
209
191
 
@@ -217,111 +199,71 @@ Railtie in Rails), covering four methods:
217
199
  - `Sneakers::WorkerGroup#stop` — record that the shutdown is deliberate, *before*
218
200
  `super` starts unsubscribing.
219
201
 
220
- `after_fork` rather than `run`, even though `run` is where the monitor must
221
- ultimately live: `Sneakers::WorkerGroup#run` calls `after_fork` as its very first
222
- statement and only then resolves `worker_classes`, so hooking `after_fork` is the
223
- earliest point in the fork at which the application's own `after_fork` hook has
224
- already run. That matters because resolving the expected consumer count may run
225
- application code — under `sneakers:active_job` the set is a callable registry —
226
- and the standard use of `after_fork` is re-establishing the connections a fork
227
- inherited. Resolving the set first would run that code against a fork that is not
228
- ready yet, and the `rescue` around the start would then leave the pod with no
229
- monitor at all. A spec asserts this ordering against the real worker gem, so an
230
- upgrade that reorders `run` fails in CI rather than in production.
231
-
232
- `ServerEngine::Server#create_worker` calls `w.extend(@worker_module)`: the
233
- `WorkerGroup` module lands in the singleton class of the *instance*, not as an
234
- `include` in a class. A `prepend` to the module still sits ahead of it in that
235
- lookup chain, which is what makes the hook work; `worker_id` and `config` come
236
- from `ServerEngine::Worker`. Verified on serverengine 2.4.0, and covered by a
237
- spec that asserts the order in `ancestors` — this is the most fragile assumption
238
- in the gem, so it fails in CI rather than in production.
202
+ `WorkerGroup#run` calls `after_fork` before resolving `worker_classes`. Starting
203
+ after that hook lets the application restore inherited connections before the
204
+ consumer count runs application code. This matters for the callable registry
205
+ used by `sneakers:active_job`; resolving it too early can prevent monitor startup.
206
+ A spec pins this ordering against the worker gem.
207
+
208
+ ServerEngine 2.4.0 uses `w.extend(@worker_module)`, placing `WorkerGroup` in the
209
+ instance's singleton-class chain. Prepending to the module still intercepts its
210
+ methods; an `ancestors` spec verifies that assumption. `worker_id` and `config`
211
+ come from `ServerEngine::Worker`.
239
212
 
240
213
  Slot numbers come from `worker_id`, which is the index into ServerEngine's
241
214
  `@monitors` array, and both `start_new_worker(wid)` and `restart_worker(wid)`
242
215
  reuse the same index. That is what the slot-not-PID scheme relies on.
243
216
 
244
- The monitor thread is created inside the fork, because threads do not survive
245
- `fork`. Starting it earlier would accomplish nothing. For the same reason the
246
- registry initializes its array and mutex at file-load time: that happens once,
247
- single-threaded, before any fork, so no lazy initialization is needed and none
248
- can race.
217
+ The monitor thread is created inside the fork because threads do not survive
218
+ `fork`. The registry's array and mutex are initialized at load time, before
219
+ forking, avoiding races during lazy initialization.
249
220
 
250
- In Rails the hooks are installed from a Railtie, and from `to_prepare` rather
251
- than an `initializer`. An application's `lib` is normally managed by Zeitwerk and
252
- reloadable, and reloadable constants must not be referenced while the
253
- application is initializing.
254
-
255
- Starting the monitor is wrapped in a `rescue`: instrumentation has no right to
256
- prevent a worker from starting. `Dir.mkdir` raises `Errno::EROFS` if the marks
257
- directory was never mounted, and resolving the worker set runs application code.
258
- The worst outcome should be a missing mark and a pod restart, never a pod that
259
- stays up with silent queues.
221
+ The Rails Railtie registers installation through `to_prepare`, allowing access
222
+ to reloadable worker constants after initialization. Startup exceptions, from
223
+ either application code or an unwritable marks directory, are logged without
224
+ preventing the worker from starting. The probe remains responsible for detecting
225
+ a missing or stale heartbeat.
260
226
 
261
227
  ### The expected consumer count is not configured
262
228
 
263
- It is derived from `config[:worker_classes]` — the very set from which the
264
- worker gem builds its workers. The queue list is therefore never duplicated and
265
- cannot drift from the application's configuration.
266
-
267
- That value is an array of classes under `sneakers:run` and a callable registry
268
- under `sneakers:active_job`, so it is resolved with a `respond_to?(:call)`
269
- check.
229
+ `config[:worker_classes]` is the set the worker gem itself consumes: an array
230
+ under `sneakers:run`, a callable registry under `sneakers:active_job`. Resolve it
231
+ with `respond_to?(:call)` rather than maintaining a separate queue list.
270
232
 
271
233
  ## What is configurable, and where
272
234
 
273
235
  | Setting | Where | Why |
274
236
  |---|---|---|
275
237
  | `logger`, `enabled`, `startup_grace_ticks` | application config block | only the worker needs them |
276
- | `tick` | either, and the config block wins | only the worker reads it, so two sources cannot contradict each other |
238
+ | `tick`, `shutdown_grace` | either, and the config block wins | only the worker reads them, so two sources cannot contradict each other |
277
239
  | `dir`, `max_age` | environment variables **only** | the probe is a separate process |
278
240
 
279
- `dir` and `max_age` are deliberately *not* settable from the application. The
280
- probe is launched by the kubelet as its own process; it never reads, and cannot
281
- read, the application's initializer. If those two values were configurable in
282
- both places, a mismatch between what the worker writes and what the probe
283
- expects would pass unnoticed — and that mismatch is silent by nature: the worker
284
- would look fine and the probe would look broken. Making them read-only from the
285
- application removes the failure mode instead of documenting it. A spec asserts
286
- that `Configuration` does not respond to `dir=` or `max_age=`.
287
-
288
- `tick` is the opposite case and may be set from either side, because only the
289
- worker ever reads it: with a single reader there is nothing for a mismatch to
290
- happen between. `KICKS_LIVENESS_TICK` supplies the initial value and the
291
- configuration block overrides it, so `config.tick = 5` wins over the variable.
292
-
293
- Values arriving from a ConfigMap are parsed defensively: garbage falls back to
294
- the default rather than crashing the worker, and an empty string counts as
295
- "unset" — in a ConfigMap that is what you get by declaring a key and leaving it
296
- blank.
297
-
298
- `max_age` and `tick` are durations, so a value must be parseable *and* positive.
299
- Zero and negative numbers are the more dangerous half: they parse perfectly
300
- well, a tick of zero turns the monitor into a hot loop, and a negative `max_age`
301
- makes every mark stale on arrival, so the probe can never pass again. They fall
302
- back to the default too.
303
-
304
- Setting `tick` from the application is held to a stricter standard: a
305
- non-positive value raises `ArgumentError`, as does a value greater than or equal
306
- to `max_age`: such a monitor would inevitably let a healthy mark go stale. The
307
- environment gets a silent fallback for values that cannot be parsed. If its
308
- effective tick is greater than or equal to `max_age`, configuration emits a
309
- warning on stderr and uses the default tick when it is safe, or half of `max_age`
310
- otherwise. A ConfigMap typo must not bring a worker down, whereas an initializer
311
- is code and should fail loudly at boot, where the developer is looking.
312
-
313
- `startup_grace_ticks` must be a positive integer. It is compared directly with
314
- an integer counter; accepting zero, a float, or a string would silently disable
315
- the escalation that is supposed to diagnose a worker that never subscribes.
316
-
317
- The logger defaults to `Sneakers.logger` but is resolved lazily, because at the
318
- time the configuration object is built it may not be set up yet.
319
-
320
- Logging is diagnostic; the heartbeat is the liveness contract. An exception
321
- raised by a custom logger is therefore swallowed at the logging boundary. It
322
- cannot prevent a healthy tick from writing its mark, abort monitor startup, or
323
- escape the loop's error handler and kill the monitor thread. No fallback message
324
- is attempted through the same broken logger.
241
+ The exec probe cannot read an application initializer, so `dir` and `max_age`
242
+ have one source: the environment. Specs reject application setters for both.
243
+ Only the worker reads `tick` and `shutdown_grace`, so initializer values may
244
+ override their environment values without contradicting the probe.
245
+
246
+ Environment durations must be positive integers; missing, empty, unparseable,
247
+ zero, and negative values use defaults. If the effective tick is at least
248
+ `max_age`, it warns on stderr and uses the default tick when safe, or half of
249
+ `max_age` otherwise. The warning is deferred until the tick is read, so an
250
+ initializer override or a disabled monitor does not produce a misleading WARN.
251
+
252
+ `KICKS_LIVENESS_SHUTDOWN_GRACE` defaults to 60 seconds. Its initializer override
253
+ accepts positive finite real numbers. Set it to at least the deployment's
254
+ `terminationGracePeriodSeconds`, which the gem cannot discover from a signal.
255
+
256
+ Invalid ConfigMap input falls back so a deployment typo does not prevent worker
257
+ startup. Initializer values are application code: invalid values raise
258
+ `ArgumentError` at boot, where the developer can fix the error, rather than
259
+ silently substituting a default. `tick` must be a positive number smaller than
260
+ `max_age`, and `startup_grace_ticks` a positive integer. The latter is compared
261
+ with an integer counter; accepting a float or zero would disable escalation.
262
+
263
+ The default logger is resolved lazily from `Sneakers.logger`, which may be unset
264
+ during configuration. Logger exceptions are swallowed at the logging boundary
265
+ so they cannot prevent heartbeat publication or stop the monitor. No fallback
266
+ uses the same broken logger.
325
267
 
326
268
  ## Logging: events, not the pulse
327
269
 
@@ -331,30 +273,33 @@ add noise. What gets logged is transitions.
331
273
  | Event | Level |
332
274
  |---|---|
333
275
  | started, with the effective settings | INFO |
276
+ | the PID topology could not be verified, or yielded no generation | WARN |
334
277
  | waiting for consumers | INFO |
335
278
  | became healthy | INFO |
336
279
  | shutting down | INFO |
337
280
  | still not healthy after `startup_grace_ticks` ticks | ERROR |
338
281
  | was healthy, became unhealthy | ERROR |
339
282
  | a slot respawning without ever becoming healthy, once per grace window | ERROR |
283
+ | the PID topology is proven unsupported | ERROR |
340
284
  | the monitor thread caught an exception | ERROR |
341
285
  | the monitor could not be started at all | ERROR |
342
286
 
343
- ERROR is reserved for genuine failure. A logger typically comes up with
344
- `LOG_LEVEL` defaulting to `error`, so anything important must survive that
345
- filter — while an ordinary pod start must not emit ERROR on every deploy, or an
346
- alert on ERROR logs fires on every rollout and is promptly muted.
347
-
348
- A graceful shutdown is the other half of that rule. `Sneakers::Worker#stop`
349
- unsubscribes and then waits for the thread pool to drain, which takes as long as
350
- the longest in-flight job — and the registry empties at the start of it. Judged
351
- by the consumer count that is indistinguishable from a fault, so every single
352
- deploy would emit an ERROR. From `WorkerGroup#stop` onwards the predicate is
353
- therefore suspended: the monitor says so once at INFO and keeps the mark fresh,
354
- so a slow drain gets the full `terminationGracePeriodSeconds` instead of having
355
- the probe cut it short at `max_age`.
356
-
357
- The startup grace exists because a slow start and a start that will never finish
358
- look identical for the first few ticks. Once the grace expires, staying silent
359
- would mean a pod that never comes up never says so — hence exactly one ERROR,
360
- not one per tick.
287
+ `topology-report` deduplicates only the three topology results: `unverifiable`,
288
+ `unavailable`, and `unsafe`. Their warning frequency, probe output, and recovery
289
+ are described in
290
+ [LIMITATIONS.md](LIMITATIONS.md#container-generations-need-linux-procfs-and-a-private-pid-namespace).
291
+ Loop and startup exceptions do not use that flag.
292
+
293
+ ERROR must survive the usual error-level logger filter, while normal startup
294
+ and deliberate shutdown remain INFO. On shutdown the monitor suspends its
295
+ predicate until a monotonic deadline: the first `Registry.stopping!` records the
296
+ stop time, and the monitor adds `shutdown_grace`. A delayed tick or repeated
297
+ callback cannot renew it. At expiry it logs one ERROR and stops `touch!` while
298
+ continuing `declare!`. Nothing is persisted, so a replacement fork does not
299
+ inherit a sibling's shutdown. Mark freshness still uses wall time; see the
300
+ [timing boundary](LIMITATIONS.md#graceful-shutdown-has-a-finite-heartbeat-exemption).
301
+
302
+ Startup grace postpones the error for a slow initial subscription; a surviving
303
+ monitor reports once when it expires. Fork-respawn loops use the persistent
304
+ counter and reporting windows described in
305
+ [LIMITATIONS.md](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn).