kicks_liveness 0.1.2 → 0.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: df61d00933097bb96e6d752e024e349ab631cd7c9d571cd9bff9ff63b8adda40
4
- data.tar.gz: 78864373ca06cd998cdee192d064cd00ed69a4ecd73eb1b657474d77ab593066
3
+ metadata.gz: db97cf34081f6a62735363603c3139ff3ddea09c23d81305d2e3a2e2e84ff126
4
+ data.tar.gz: ca1ee94ae412b24615b4df2eabe3d1344375721f4b50a33ca6af2590c6b60ae3
5
5
  SHA512:
6
- metadata.gz: '007091e18fea1b216ba25c634bab1a4770d25cc0f062ee30ba2885563075bc94e7662e58eebbac9507b448217508725612f172189b692e3a4d616e5f81fdd46b'
7
- data.tar.gz: f460271ca68fdf912559b3fa2c41b02074d5eb457e946d2f985e3e2b0ac3da66dc94ce90c9d5639212a0fc7bb6537046654f31921cf6766fe15c903bd7dc04ef
6
+ metadata.gz: 11b5aaed04e4e67c3607a25291d7f1443f5ca00885b0fec3f9931ebc967c02f11deb150697a78f0a936c832334ef895783ec532801d72e61d92e2f2b715a2c67
7
+ data.tar.gz: dd47406061ce002cd058bd4d89577a383d0eafcc7808688f4d147a660ccd635b6141b75ffa97a6e7778ce661d51cde8b99f0d31e9ce7bf731e4633d84dc9438c
data/CHANGELOG.md CHANGED
@@ -5,6 +5,57 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
5
5
 
6
6
  ## [Unreleased]
7
7
 
8
+ ## [0.1.3] - 2026-10-21
9
+
10
+ ### Fixed
11
+
12
+ - Detect and reject PID topologies in which the container generation cannot
13
+ identify an incarnation. `shareProcessNamespace: true` and `hostPID: true`
14
+ leave PID 1 alive across a restart of the worker container, so the 0.1.2
15
+ identifier can repeat and an inherited fresh heartbeat could satisfy
16
+ `startupProbe`. Rejection is conditional on a readable `/proc/1/ns/mnt`.
17
+ Where the comparison can be made, the worker publishes `<dir>/topology-unsafe`
18
+ instead of a heartbeat and every probe exits 1 with
19
+ `unsupported PID topology: pod must not set shareProcessNamespace or hostPID`,
20
+ including one that cannot read the link itself; the state is terminal for the
21
+ marks volume, and recovery is recreating the pod without those settings.
22
+ Where it cannot, the previous behavior remains and the worker warns once. The
23
+ measured outcomes are in `docs/LIMITATIONS.md`.
24
+ - Distinguish a valid zero in PID 1's stat start-time field from a stat line
25
+ that cannot be parsed: the generation stays unavailable, and the warning
26
+ names `hostPID` or a virtualized procfs instead of a false parse error.
27
+ - Scope the `attempt-<slot>` respawn counter to the container generation. A
28
+ replacement container starts its own unhealthy run and cannot escalate over
29
+ starts it did not take, while respawns within one generation go on sharing a
30
+ count. Where no generation is available the counter keeps its previous
31
+ volume-wide behavior.
32
+
33
+ ### Added
34
+
35
+ - Report a degraded container generation once from the worker, at WARN — once
36
+ per container incarnation where a generation exists, once per error state
37
+ where it does not. Neither degraded state adds anything to the probe's
38
+ output, and the intentional non-Linux fallback stays silent.
39
+ - The marks directory gains two files that appear only when something is
40
+ wrong: `topology-unsafe`, the terminal fail-closed state, and
41
+ `topology-report`, the worker's own flag for saying each thing once. Tooling
42
+ that enumerates the directory should allow for both.
43
+ - Document the boundary of the graceful-shutdown exemption in
44
+ `docs/LIMITATIONS.md`: a SIGTERM that does not come from a pod deletion can
45
+ leave a pod Ready with no consumers. Existing behavior, newly documented.
46
+ - Add `spec/integration/verify.sh topology`, a Linux matrix covering a shared
47
+ PID namespace, the host PID namespace, a verified private-PID container, and
48
+ a private-PID container whose checker cannot read `/proc/1/ns/mnt`.
49
+
50
+ ### Changed
51
+
52
+ - `Heartbeat#declare!` documents its actual `Integer` return value.
53
+ - Replace `Heartbeat.container_generation` with `Heartbeat.topology`, returning
54
+ a structured result instead of a string or `nil`; replace the `Heartbeat.new`
55
+ keyword `generation:` with `topology:`, accepting that result; and remove
56
+ `KicksLiveness::GenerationGuard`. All are `@api private`, but callers need
57
+ updating.
58
+
8
59
  ## [0.1.2] - 2026-09-11
9
60
 
10
61
  ### Fixed
@@ -73,7 +124,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
73
124
  [docs/](https://github.com/PoroshkinaVV/kicks_liveness/tree/main/docs); start
74
125
  with `SETUP.md`, and read `LIMITATIONS.md` before relying on it.
75
126
 
76
- [Unreleased]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.2...HEAD
127
+ [Unreleased]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.3...HEAD
128
+ [0.1.3]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.2...v0.1.3
77
129
  [0.1.2]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.1...v0.1.2
78
130
  [0.1.1]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.0...v0.1.1
79
131
  [0.1.0]: https://github.com/PoroshkinaVV/kicks_liveness/releases/tag/v0.1.0
data/README.md CHANGED
@@ -14,10 +14,15 @@ The probe only reads that heartbeat, scoped to the current container
14
14
  incarnation. No application boot, no network call, and no healthy replica
15
15
  hiding a stalled one.
16
16
 
17
- That restart scoping assumes Kubernetes' default container-private PID
18
- namespace; see
19
- [Limitations](docs/LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1)
20
- for pods that set `shareProcessNamespace` or `hostPID`.
17
+ That restart scoping needs Kubernetes' default container-private PID
18
+ namespace. Where the namespace links are readable, a pod that sets
19
+ `shareProcessNamespace` or `hostPID` is rejected outright. With the default
20
+ capability set they were not readable on the measured Kubernetes node: the
21
+ shared-PID pod was `unverifiable`, while the host-PID pod had no usable
22
+ generation because node init reported a zero start time. Both degradations warn
23
+ instead of pretending the restart guarantee still holds; see
24
+ [Limitations](docs/LIMITATIONS.md#container-generations-need-linux-procfs-and-a-private-pid-namespace)
25
+ for what each side can prove and what it does when it cannot.
21
26
 
22
27
  ## Install: one line in Rails
23
28
 
@@ -195,7 +200,7 @@ negative path before rollout; the commands and expected output are in
195
200
  | [Design](docs/DESIGN.md) | health predicate, heartbeat files, hooks, configuration choices, and measurements |
196
201
  | [Running under Kubernetes](docs/KUBERNETES.md) | full manifest, CPU/startup budgets, alerts, and live-pod verification |
197
202
  | [Limitations](docs/LIMITATIONS.md) | cases the in-memory predicate intentionally cannot cover |
198
- | [Verification scenarios](docs/VERIFYING.md) | eleven deliberately induced failures and their observed outcomes |
203
+ | [Verification scenarios](docs/VERIFYING.md) | twelve deliberately exercised scenarios and their observed outcomes |
199
204
 
200
205
  ## Contributing
201
206
 
data/docs/DESIGN.md CHANGED
@@ -74,9 +74,8 @@ channel.open? && connection.open? && channel.any_consumers?
74
74
  | connection open, no consumers | unhealthy | the worker silently stopped consuming — a restart is the only cure |
75
75
  | not the whole set subscribed | unhealthy | a worker that quietly dropped out would otherwise leave the probe green |
76
76
 
77
- The last row is the reason the registry compares its size against an expected
78
- count rather than just checking that whatever registered is alive. Four healthy
79
- workers out of five look perfectly fine one object at a time.
77
+ The expected count catches partial subscription: four healthy workers out of
78
+ five must still make the process unhealthy.
80
79
 
81
80
  ### Nothing here talks to the broker
82
81
 
@@ -101,84 +100,76 @@ The predicate learns that a consumer was cancelled because RabbitMQ sends
101
100
  push, not a poll — which is what makes an in-memory predicate possible at all,
102
101
  and also where its limits come from. See [LIMITATIONS.md](LIMITATIONS.md).
103
102
 
104
- `recovering_from_network_failure?` is marked `@private` in Bunny's own
105
- documentation, so an upgrade may remove it. It is therefore called through
106
- `respond_to?`: without that guard the call would raise `NoMethodError`, `alive?`
107
- would catch it and report *unhealthy*, and every pod would enter a restart loop
108
- on the next Bunny upgrade. What the guard cannot preserve is the exemption
109
- itself — if the method disappears, so does "healthy while reconnecting",
110
- silently. That is the thing to re-check when upgrading Bunny.
103
+ `recovering_from_network_failure?` is private Bunny API. The `respond_to?` guard
104
+ avoids turning its removal into a rescued `NoMethodError` and an unhealthy
105
+ verdict on every tick. Removal would still disable the recovery exemption
106
+ silently; re-check this API when upgrading Bunny.
111
107
 
112
108
  ## The mark files
113
109
 
114
110
  ```
115
- <dir>/expected how many forks the probe must wait for
116
- <dir>/generation which container incarnation wrote this heartbeat
117
- <dir>/worker-<slot> one per fork, refreshed every tick the fork is healthy
118
- <dir>/attempt-<slot> starts of a slot that has not become healthy yet
111
+ <dir>/expected how many forks the probe must wait for
112
+ <dir>/generation which container incarnation wrote this heartbeat
113
+ <dir>/worker-<slot> one per fork, refreshed every tick the fork is healthy
114
+ <dir>/attempt-<slot> starts of a slot that has not become healthy yet
115
+ <dir>/topology-unsafe the PID namespace is not this container's: fail closed
116
+ <dir>/topology-report what the worker has already reported about it, once
119
117
  ```
120
118
 
121
- **`expected` closes a startup hole.** Without it the probe would pass as soon as
122
- any single file was fresh: fork 0 has written, fork 1 has not, and the pod looks
123
- ready while half of it is not subscribed. Every fork writes the same value, so
124
- they cannot disagree.
125
-
126
- Every fork rewrites it on every tick, not only at startup. Nothing else restores
127
- it: if the directory is wiped, `touch!` brings the slot marks back while
128
- `expected` stays missing, and the probe reports "worker has not started" for the
129
- rest of the pod's life. Rewriting it is also what lets a respawned set of forks
130
- correct a count that has been lowered.
131
-
132
- **`generation` closes a container-restart hole when the container owns PID 1.**
133
- Kubernetes preserves an `emptyDir` when it restarts a container inside the same
134
- pod. That is useful for an application cache elsewhere in the volume, but a
135
- fresh heartbeat from the dead process must not let the new container pass its
136
- one-shot `startupProbe`. On Linux, the worker and exec probe independently
137
- derive the same incarnation from the container's mount namespace and PID 1
138
- start time. Both `generation` and every slot mark carry it. Until the new
139
- container declares itself and every current fork publishes its own mark, files
140
- inherited from the previous container are rejected. Nothing outside the marks
141
- directory is removed.
142
-
143
- This guarantee assumes Kubernetes' default container-private PID namespace.
144
- With `shareProcessNamespace: true` PID 1 belongs to the pod sandbox, and with
145
- `hostPID: true` it is the node init process; neither restarts with the worker
146
- container. Under either setting the guard can accept an inherited fresh mark.
147
- Do not enable them on a pod whose startup probe relies on this guarantee; see
148
- [LIMITATIONS.md](LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1).
149
-
150
- **Files are named by supervisor slot, not by PID.** A fork killed with SIGKILL
151
- is respawned into the same slot and overwrites its own file. Had the name
152
- contained a PID, that file would sit there stale forever and the probe would
153
- fail permanently, turning one dead fork into a pod that can never come back.
154
-
155
- `expected` is written through a temporary file and a `rename`, not with a plain
156
- write. A plain write truncates first, and a probe reading in that window finds
157
- the file empty and reports that the worker has not started. The window is real
158
- and recurring: every fork rewrites this file on every tick, while the liveness
159
- probe reads it on a schedule of its own. `rename` is atomic on tmpfs, so the
160
- probe sees either the old value or the new one.
161
-
162
- **`attempt-<slot>` is the respawn counter**, and it lives in this directory for
163
- the single reason that the directory outlives the fork: a monitor caught in a
164
- respawn loop is a brand-new object every few hundred milliseconds and can hold
165
- no counter of its own. The file is removed once the slot becomes healthy, so in
166
- steady state the directory holds `expected`, `generation`, and the
167
- `worker-<slot>` marks. What the counter is for is in
119
+ **`expected` prevents partial startup from passing.** Checking only existing
120
+ marks could accept fork 0's fresh mark before fork 1 has subscribed. Declaring
121
+ the count tells the probe which marks must exist, not just which are fresh.
122
+ With a fixed runner configuration, every fork declares the same count.
123
+ Re-declaring on each tick also restores `expected` if the directory is
124
+ wiped; refreshing slot marks alone would leave the probe permanently unhealthy.
125
+ Live worker-count changes remain
126
+ [unsupported](LIMITATIONS.md#changing-the-worker-count-at-runtime-is-not-supported).
127
+
128
+ **`generation` rejects inherited heartbeats.** With usable Linux procfs and a
129
+ container-private PID namespace, the worker and exec probe derive it from the
130
+ mount namespace and PID 1 start time. Both `generation` and every slot mark must
131
+ match the current incarnation, so an `emptyDir` preserved across a container
132
+ restart cannot pass `startupProbe` until all replacement forks publish. No
133
+ application cache is removed. Procfs and PID-topology limits are described in
134
+ [LIMITATIONS.md](LIMITATIONS.md#container-generations-need-linux-procfs-and-a-private-pid-namespace).
135
+
136
+ **`topology-unsafe` carries terminal evidence from the worker.** Each participant
137
+ compares `/proc/1/ns/mnt` with `/proc/self/ns/mnt` independently. A worker that
138
+ proves them different writes this file instead of a heartbeat; the probe checks
139
+ it before `expected`, generation, or freshness. Omitting a new generation would
140
+ leave inherited marks eligible for acceptance. Weaker evidence never clears
141
+ this file.
142
+
143
+ **`topology-report` deduplicates worker diagnostics across forks.** The probe
144
+ ignores it. Claims lock the complete read-compare-write operation; atomic rename
145
+ alone cannot prevent concurrent claims. Clearing by truncation keeps every fork
146
+ locking the same inode.
147
+
148
+ **Slot names survive fork replacement.** A respawn overwrites its predecessor's
149
+ mark. PID-based names would leave stale files that permanently fail the probe.
150
+
151
+ `expected` and slot marks are published through temporary files and `rename`.
152
+ Concurrent probes therefore see complete old or new contents, never a truncated
153
+ count or a mark missing its generation. Temporary filenames include the writer's
154
+ PID so concurrent forks cannot overwrite one another's temporary files.
155
+
156
+ **`attempt-<slot>` persists an unhealthy run across fork respawns.** Its
157
+ generation rejects counts from a previous container when one can be derived.
158
+ The first healthy tick removes it. Reporting windows and generation-less scope
159
+ are explained in
168
160
  [LIMITATIONS.md](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn).
169
161
 
170
162
  The timestamp, pid, and slot in `worker-<slot>` exist for a human running
171
163
  `kubectl exec ... cat`. Its generation is part of the probe contract; freshness
172
164
  still comes from mtime.
173
165
 
174
- The directory must be on tmpfs — in Kubernetes, an `emptyDir` with
175
- `medium: Memory`. Put it on a real disk and the probe starts depending on the
176
- disk again, which was defect number two.
166
+ Use tmpfs — an `emptyDir` with `medium: Memory` in Kubernetes — to keep mark
167
+ reads independent of disk stalls.
177
168
 
178
169
  ## Why the heartbeat file has no `require` of its own
179
170
 
180
- The probe loads exactly one file and nothing else. Measured over 30 invocations
181
- each; absolute values depend on the hardware, but the ratios do not:
171
+ The probe entry point loads only `heartbeat.rb`. Recorded over 30 invocations
172
+ each; timings vary with the runtime and hardware:
182
173
 
183
174
  | | per invocation |
184
175
  |---|---|
@@ -188,22 +179,13 @@ each; absolute values depend on the hardware, but the ratios do not:
188
179
  | the probe, found through RubyGems | 43 ms |
189
180
  | **`bundle exec ruby -e ''`** | **300 ms** |
190
181
 
191
- Subtract the baselines and the probe's own work is a **few milliseconds**.
192
- Everything else is interpreter startup, and 28 ms of that is RubyGems
193
- initializing.
194
-
195
- Three consequences follow. `bundle exec kicks-liveness` is the portable
196
- default; most of its runtime is Bundler startup rather than the check itself.
197
- When the gem is installed in `GEM_HOME`, RubyGems can find it with a plain
198
- `require` for under 50 ms. Skipping RubyGems entirely is faster still, but it
199
- requires naming the path to the gem's `lib` directory; see
200
- [KUBERNETES.md](KUBERNETES.md) for when that trade is worth making.
201
-
202
- Keeping the probe's path through the heartbeat file free of `require`
203
- statements — and the file free of references to the rest of the gem — is what
204
- makes all three possible. The single `require` the file does contain,
205
- `fileutils`, is lazy and sits in the directory-creation branch that only the
206
- worker ever reaches.
182
+ Interpreter and dependency startup dominate these measurements. `bundle exec`
183
+ is the portable default; direct loading depends on where the image installs
184
+ the gem, as described in [KUBERNETES.md](KUBERNETES.md#where-your-image-puts-its-gems).
185
+ `heartbeat.rb` therefore references no other gem code and requires `fileutils`
186
+ only in the worker's directory-creation branch. Worker-only counters and report
187
+ deduplication live in `Attempts` and `TopologyReport` so their code and requires
188
+ stay outside the probe's load path.
207
189
 
208
190
  ## Where the hooks attach
209
191
 
@@ -217,56 +199,36 @@ Railtie in Rails), covering four methods:
217
199
  - `Sneakers::WorkerGroup#stop` — record that the shutdown is deliberate, *before*
218
200
  `super` starts unsubscribing.
219
201
 
220
- `after_fork` rather than `run`, even though `run` is where the monitor must
221
- ultimately live: `Sneakers::WorkerGroup#run` calls `after_fork` as its very first
222
- statement and only then resolves `worker_classes`, so hooking `after_fork` is the
223
- earliest point in the fork at which the application's own `after_fork` hook has
224
- already run. That matters because resolving the expected consumer count may run
225
- application code — under `sneakers:active_job` the set is a callable registry —
226
- and the standard use of `after_fork` is re-establishing the connections a fork
227
- inherited. Resolving the set first would run that code against a fork that is not
228
- ready yet, and the `rescue` around the start would then leave the pod with no
229
- monitor at all. A spec asserts this ordering against the real worker gem, so an
230
- upgrade that reorders `run` fails in CI rather than in production.
231
-
232
- `ServerEngine::Server#create_worker` calls `w.extend(@worker_module)`: the
233
- `WorkerGroup` module lands in the singleton class of the *instance*, not as an
234
- `include` in a class. A `prepend` to the module still sits ahead of it in that
235
- lookup chain, which is what makes the hook work; `worker_id` and `config` come
236
- from `ServerEngine::Worker`. Verified on serverengine 2.4.0, and covered by a
237
- spec that asserts the order in `ancestors` — this is the most fragile assumption
238
- in the gem, so it fails in CI rather than in production.
202
+ `WorkerGroup#run` calls `after_fork` before resolving `worker_classes`. Starting
203
+ after that hook lets the application restore inherited connections before the
204
+ consumer count runs application code. This matters for the callable registry
205
+ used by `sneakers:active_job`; resolving it too early can prevent monitor startup.
206
+ A spec pins this ordering against the worker gem.
207
+
208
+ ServerEngine 2.4.0 uses `w.extend(@worker_module)`, placing `WorkerGroup` in the
209
+ instance's singleton-class chain. Prepending to the module still intercepts its
210
+ methods; an `ancestors` spec verifies that assumption. `worker_id` and `config`
211
+ come from `ServerEngine::Worker`.
239
212
 
240
213
  Slot numbers come from `worker_id`, which is the index into ServerEngine's
241
214
  `@monitors` array, and both `start_new_worker(wid)` and `restart_worker(wid)`
242
215
  reuse the same index. That is what the slot-not-PID scheme relies on.
243
216
 
244
- The monitor thread is created inside the fork, because threads do not survive
245
- `fork`. Starting it earlier would accomplish nothing. For the same reason the
246
- registry initializes its array and mutex at file-load time: that happens once,
247
- single-threaded, before any fork, so no lazy initialization is needed and none
248
- can race.
249
-
250
- In Rails the hooks are installed from a Railtie, and from `to_prepare` rather
251
- than an `initializer`. An application's `lib` is normally managed by Zeitwerk and
252
- reloadable, and reloadable constants must not be referenced while the
253
- application is initializing.
217
+ The monitor thread is created inside the fork because threads do not survive
218
+ `fork`. The registry's array and mutex are initialized at load time, before
219
+ forking, avoiding races during lazy initialization.
254
220
 
255
- Starting the monitor is wrapped in a `rescue`: instrumentation has no right to
256
- prevent a worker from starting. `Dir.mkdir` raises `Errno::EROFS` if the marks
257
- directory was never mounted, and resolving the worker set runs application code.
258
- The worst outcome should be a missing mark and a pod restart, never a pod that
259
- stays up with silent queues.
221
+ The Rails Railtie registers installation through `to_prepare`, allowing access
222
+ to reloadable worker constants after initialization. Startup exceptions, from
223
+ either application code or an unwritable marks directory, are logged without
224
+ preventing the worker from starting. The probe remains responsible for detecting
225
+ a missing or stale heartbeat.
260
226
 
261
227
  ### The expected consumer count is not configured
262
228
 
263
- It is derived from `config[:worker_classes]` — the very set from which the
264
- worker gem builds its workers. The queue list is therefore never duplicated and
265
- cannot drift from the application's configuration.
266
-
267
- That value is an array of classes under `sneakers:run` and a callable registry
268
- under `sneakers:active_job`, so it is resolved with a `respond_to?(:call)`
269
- check.
229
+ `config[:worker_classes]` is the set the worker gem itself consumes: an array
230
+ under `sneakers:run`, a callable registry under `sneakers:active_job`. Resolve it
231
+ with `respond_to?(:call)` rather than maintaining a separate queue list.
270
232
 
271
233
  ## What is configurable, and where
272
234
 
@@ -276,52 +238,28 @@ check.
276
238
  | `tick` | either, and the config block wins | only the worker reads it, so two sources cannot contradict each other |
277
239
  | `dir`, `max_age` | environment variables **only** | the probe is a separate process |
278
240
 
279
- `dir` and `max_age` are deliberately *not* settable from the application. The
280
- probe is launched by the kubelet as its own process; it never reads, and cannot
281
- read, the application's initializer. If those two values were configurable in
282
- both places, a mismatch between what the worker writes and what the probe
283
- expects would pass unnoticed — and that mismatch is silent by nature: the worker
284
- would look fine and the probe would look broken. Making them read-only from the
285
- application removes the failure mode instead of documenting it. A spec asserts
286
- that `Configuration` does not respond to `dir=` or `max_age=`.
287
-
288
- `tick` is the opposite case and may be set from either side, because only the
289
- worker ever reads it: with a single reader there is nothing for a mismatch to
290
- happen between. `KICKS_LIVENESS_TICK` supplies the initial value and the
291
- configuration block overrides it, so `config.tick = 5` wins over the variable.
292
-
293
- Values arriving from a ConfigMap are parsed defensively: garbage falls back to
294
- the default rather than crashing the worker, and an empty string counts as
295
- "unset" — in a ConfigMap that is what you get by declaring a key and leaving it
296
- blank.
297
-
298
- `max_age` and `tick` are durations, so a value must be parseable *and* positive.
299
- Zero and negative numbers are the more dangerous half: they parse perfectly
300
- well, a tick of zero turns the monitor into a hot loop, and a negative `max_age`
301
- makes every mark stale on arrival, so the probe can never pass again. They fall
302
- back to the default too.
303
-
304
- Setting `tick` from the application is held to a stricter standard: a
305
- non-positive value raises `ArgumentError`, as does a value greater than or equal
306
- to `max_age`: such a monitor would inevitably let a healthy mark go stale. The
307
- environment gets a silent fallback for values that cannot be parsed. If its
308
- effective tick is greater than or equal to `max_age`, configuration emits a
309
- warning on stderr and uses the default tick when it is safe, or half of `max_age`
310
- otherwise. A ConfigMap typo must not bring a worker down, whereas an initializer
311
- is code and should fail loudly at boot, where the developer is looking.
312
-
313
- `startup_grace_ticks` must be a positive integer. It is compared directly with
314
- an integer counter; accepting zero, a float, or a string would silently disable
315
- the escalation that is supposed to diagnose a worker that never subscribes.
316
-
317
- The logger defaults to `Sneakers.logger` but is resolved lazily, because at the
318
- time the configuration object is built it may not be set up yet.
319
-
320
- Logging is diagnostic; the heartbeat is the liveness contract. An exception
321
- raised by a custom logger is therefore swallowed at the logging boundary. It
322
- cannot prevent a healthy tick from writing its mark, abort monitor startup, or
323
- escape the loop's error handler and kill the monitor thread. No fallback message
324
- is attempted through the same broken logger.
241
+ The exec probe cannot read an application initializer, so `dir` and `max_age`
242
+ have one source: the environment. Specs reject application setters for both.
243
+ Only the worker reads `tick`, so `config.tick` may override
244
+ `KICKS_LIVENESS_TICK` without contradicting the probe.
245
+
246
+ Environment durations must be positive integers; missing, empty, unparseable,
247
+ zero, and negative values use defaults. If the effective tick is at least
248
+ `max_age`, it warns on stderr and uses the default tick when safe, or half of
249
+ `max_age` otherwise. The warning is deferred until the tick is read, so an
250
+ initializer override or a disabled monitor does not produce a misleading WARN.
251
+
252
+ Invalid ConfigMap input falls back so a deployment typo does not prevent worker
253
+ startup. Initializer values are application code: invalid values raise
254
+ `ArgumentError` at boot, where the developer can fix the error, rather than
255
+ silently substituting a default. `tick` must be a positive number smaller than
256
+ `max_age`, and `startup_grace_ticks` a positive integer. The latter is compared
257
+ with an integer counter; accepting a float or zero would disable escalation.
258
+
259
+ The default logger is resolved lazily from `Sneakers.logger`, which may be unset
260
+ during configuration. Logger exceptions are swallowed at the logging boundary
261
+ so they cannot prevent heartbeat publication or stop the monitor. No fallback
262
+ uses the same broken logger.
325
263
 
326
264
  ## Logging: events, not the pulse
327
265
 
@@ -331,30 +269,29 @@ add noise. What gets logged is transitions.
331
269
  | Event | Level |
332
270
  |---|---|
333
271
  | started, with the effective settings | INFO |
272
+ | the PID topology could not be verified, or yielded no generation | WARN |
334
273
  | waiting for consumers | INFO |
335
274
  | became healthy | INFO |
336
275
  | shutting down | INFO |
337
276
  | still not healthy after `startup_grace_ticks` ticks | ERROR |
338
277
  | was healthy, became unhealthy | ERROR |
339
278
  | a slot respawning without ever becoming healthy, once per grace window | ERROR |
279
+ | the PID topology is proven unsupported | ERROR |
340
280
  | the monitor thread caught an exception | ERROR |
341
281
  | the monitor could not be started at all | ERROR |
342
282
 
343
- ERROR is reserved for genuine failure. A logger typically comes up with
344
- `LOG_LEVEL` defaulting to `error`, so anything important must survive that
345
- filter — while an ordinary pod start must not emit ERROR on every deploy, or an
346
- alert on ERROR logs fires on every rollout and is promptly muted.
347
-
348
- A graceful shutdown is the other half of that rule. `Sneakers::Worker#stop`
349
- unsubscribes and then waits for the thread pool to drain, which takes as long as
350
- the longest in-flight job — and the registry empties at the start of it. Judged
351
- by the consumer count that is indistinguishable from a fault, so every single
352
- deploy would emit an ERROR. From `WorkerGroup#stop` onwards the predicate is
353
- therefore suspended: the monitor says so once at INFO and keeps the mark fresh,
354
- so a slow drain gets the full `terminationGracePeriodSeconds` instead of having
355
- the probe cut it short at `max_age`.
356
-
357
- The startup grace exists because a slow start and a start that will never finish
358
- look identical for the first few ticks. Once the grace expires, staying silent
359
- would mean a pod that never comes up never says so — hence exactly one ERROR,
360
- not one per tick.
283
+ `topology-report` deduplicates only the three topology results: `unverifiable`,
284
+ `unavailable`, and `unsafe`. Their warning frequency, probe output, and recovery
285
+ are described in
286
+ [LIMITATIONS.md](LIMITATIONS.md#container-generations-need-linux-procfs-and-a-private-pid-namespace).
287
+ Loop and startup exceptions do not use that flag.
288
+
289
+ ERROR must survive the usual error-level logger filter, while normal startup
290
+ and deliberate shutdown remain INFO. On shutdown the monitor suspends its
291
+ predicate and refreshes the mark for the rest of the process, assuming an
292
+ [external deadline](LIMITATIONS.md#graceful-shutdown-is-safe-only-when-something-else-owns-the-deadline).
293
+
294
+ Startup grace postpones the error for a slow initial subscription; a surviving
295
+ monitor reports once when it expires. Fork-respawn loops use the persistent
296
+ counter and reporting windows described in
297
+ [LIMITATIONS.md](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn).
data/docs/KUBERNETES.md CHANGED
@@ -55,11 +55,16 @@ mark. A fresh mark from the previous container is rejected until every fork in
55
55
  the current one has subscribed and published its own mark, so preserving the
56
56
  cache cannot make `startupProbe` succeed early.
57
57
 
58
- That guarantee assumes the default container-private PID namespace. Do not set
59
- `shareProcessNamespace: true` or `hostPID: true` on a pod that relies on it: in
60
- either topology PID 1 survives a worker-container restart, and the generation
61
- guard can accept an inherited fresh mark. See
62
- [LIMITATIONS.md](LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1).
58
+ That guarantee needs the default container-private PID namespace. Do not set
59
+ `shareProcessNamespace: true` or `hostPID: true`: in either topology PID 1
60
+ survives a worker-container restart, so the generation repeats and an inherited
61
+ mark would be accepted. Since 0.1.3 the gem rejects that topology when either
62
+ the worker or probe can read and compare both namespace links. With Kubernetes'
63
+ default capability set those reads failed on the measured node: the shared-PID
64
+ pod was `unverifiable`, and the host-PID pod was `unavailable` because node init
65
+ reported a zero start time. Those states warn and retain the documented degraded
66
+ checks; they do not claim the topology is safe. See
67
+ [LIMITATIONS.md](LIMITATIONS.md#container-generations-need-linux-procfs-and-a-private-pid-namespace).
63
68
 
64
69
  ## Why the command looks like that
65
70
 
@@ -121,9 +126,12 @@ which is not something to rely on in a manifest.
121
126
 
122
127
  Keep the worker and probe on the same gem version. Since 0.1.2 the heartbeat
123
128
  protocol includes a container generation; a newer probe correctly rejects the
124
- generation-less files written by an older worker. `bundle exec` guarantees that
125
- both sides resolve from the same bundle. If the faster form below installs a
126
- second copy in `GEM_HOME`, rebuild that copy on every gem upgrade too.
129
+ generation-less files written by an older worker. Since 0.1.3 it also includes
130
+ the fail-closed topology state, which only a 0.1.3 probe reads: an older probe
131
+ beside a 0.1.3 worker would ignore a proven unsupported PID topology instead of
132
+ failing on it. `bundle exec` guarantees that both sides resolve from the same
133
+ bundle. If the faster form below installs a second copy in `GEM_HOME`, rebuild
134
+ that copy on every gem upgrade too.
127
135
 
128
136
  ### Where your image puts its gems
129
137