kicks_liveness 0.1.2 → 0.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +53 -1
- data/README.md +10 -5
- data/docs/DESIGN.md +124 -187
- data/docs/KUBERNETES.md +16 -8
- data/docs/LIMITATIONS.md +145 -61
- data/docs/SETUP.md +11 -0
- data/docs/VERIFYING.md +104 -7
- data/lib/kicks_liveness/attempts.rb +21 -16
- data/lib/kicks_liveness/configuration.rb +4 -9
- data/lib/kicks_liveness/heartbeat.rb +232 -109
- data/lib/kicks_liveness/hooks.rb +11 -29
- data/lib/kicks_liveness/monitor.rb +36 -46
- data/lib/kicks_liveness/registry.rb +3 -15
- data/lib/kicks_liveness/topology_report.rb +110 -0
- data/lib/kicks_liveness/version.rb +1 -1
- data/lib/kicks_liveness.rb +3 -1
- metadata +3 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: db97cf34081f6a62735363603c3139ff3ddea09c23d81305d2e3a2e2e84ff126
|
|
4
|
+
data.tar.gz: ca1ee94ae412b24615b4df2eabe3d1344375721f4b50a33ca6af2590c6b60ae3
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 11b5aaed04e4e67c3607a25291d7f1443f5ca00885b0fec3f9931ebc967c02f11deb150697a78f0a936c832334ef895783ec532801d72e61d92e2f2b715a2c67
|
|
7
|
+
data.tar.gz: dd47406061ce002cd058bd4d89577a383d0eafcc7808688f4d147a660ccd635b6141b75ffa97a6e7778ce661d51cde8b99f0d31e9ce7bf731e4633d84dc9438c
|
data/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,57 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
|
|
|
5
5
|
|
|
6
6
|
## [Unreleased]
|
|
7
7
|
|
|
8
|
+
## [0.1.3] - 2026-10-21
|
|
9
|
+
|
|
10
|
+
### Fixed
|
|
11
|
+
|
|
12
|
+
- Detect and reject PID topologies in which the container generation cannot
|
|
13
|
+
identify an incarnation. `shareProcessNamespace: true` and `hostPID: true`
|
|
14
|
+
leave PID 1 alive across a restart of the worker container, so the 0.1.2
|
|
15
|
+
identifier can repeat and an inherited fresh heartbeat could satisfy
|
|
16
|
+
`startupProbe`. Rejection is conditional on a readable `/proc/1/ns/mnt`.
|
|
17
|
+
Where the comparison can be made, the worker publishes `<dir>/topology-unsafe`
|
|
18
|
+
instead of a heartbeat and every probe exits 1 with
|
|
19
|
+
`unsupported PID topology: pod must not set shareProcessNamespace or hostPID`,
|
|
20
|
+
including one that cannot read the link itself; the state is terminal for the
|
|
21
|
+
marks volume, and recovery is recreating the pod without those settings.
|
|
22
|
+
Where it cannot, the previous behavior remains and the worker warns once. The
|
|
23
|
+
measured outcomes are in `docs/LIMITATIONS.md`.
|
|
24
|
+
- Distinguish a valid zero in PID 1's stat start-time field from a stat line
|
|
25
|
+
that cannot be parsed: the generation stays unavailable, and the warning
|
|
26
|
+
names `hostPID` or a virtualized procfs instead of a false parse error.
|
|
27
|
+
- Scope the `attempt-<slot>` respawn counter to the container generation. A
|
|
28
|
+
replacement container starts its own unhealthy run and cannot escalate over
|
|
29
|
+
starts it did not take, while respawns within one generation go on sharing a
|
|
30
|
+
count. Where no generation is available the counter keeps its previous
|
|
31
|
+
volume-wide behavior.
|
|
32
|
+
|
|
33
|
+
### Added
|
|
34
|
+
|
|
35
|
+
- Report a degraded container generation once from the worker, at WARN — once
|
|
36
|
+
per container incarnation where a generation exists, once per error state
|
|
37
|
+
where it does not. Neither degraded state adds anything to the probe's
|
|
38
|
+
output, and the intentional non-Linux fallback stays silent.
|
|
39
|
+
- The marks directory gains two files that appear only when something is
|
|
40
|
+
wrong: `topology-unsafe`, the terminal fail-closed state, and
|
|
41
|
+
`topology-report`, the worker's own flag for saying each thing once. Tooling
|
|
42
|
+
that enumerates the directory should allow for both.
|
|
43
|
+
- Document the boundary of the graceful-shutdown exemption in
|
|
44
|
+
`docs/LIMITATIONS.md`: a SIGTERM that does not come from a pod deletion can
|
|
45
|
+
leave a pod Ready with no consumers. Existing behavior, newly documented.
|
|
46
|
+
- Add `spec/integration/verify.sh topology`, a Linux matrix covering a shared
|
|
47
|
+
PID namespace, the host PID namespace, a verified private-PID container, and
|
|
48
|
+
a private-PID container whose checker cannot read `/proc/1/ns/mnt`.
|
|
49
|
+
|
|
50
|
+
### Changed
|
|
51
|
+
|
|
52
|
+
- `Heartbeat#declare!` documents its actual `Integer` return value.
|
|
53
|
+
- Replace `Heartbeat.container_generation` with `Heartbeat.topology`, returning
|
|
54
|
+
a structured result instead of a string or `nil`; replace the `Heartbeat.new`
|
|
55
|
+
keyword `generation:` with `topology:`, accepting that result; and remove
|
|
56
|
+
`KicksLiveness::GenerationGuard`. All are `@api private`, but callers need
|
|
57
|
+
updating.
|
|
58
|
+
|
|
8
59
|
## [0.1.2] - 2026-09-11
|
|
9
60
|
|
|
10
61
|
### Fixed
|
|
@@ -73,7 +124,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
|
|
|
73
124
|
[docs/](https://github.com/PoroshkinaVV/kicks_liveness/tree/main/docs); start
|
|
74
125
|
with `SETUP.md`, and read `LIMITATIONS.md` before relying on it.
|
|
75
126
|
|
|
76
|
-
[Unreleased]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.
|
|
127
|
+
[Unreleased]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.3...HEAD
|
|
128
|
+
[0.1.3]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.2...v0.1.3
|
|
77
129
|
[0.1.2]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.1...v0.1.2
|
|
78
130
|
[0.1.1]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.0...v0.1.1
|
|
79
131
|
[0.1.0]: https://github.com/PoroshkinaVV/kicks_liveness/releases/tag/v0.1.0
|
data/README.md
CHANGED
|
@@ -14,10 +14,15 @@ The probe only reads that heartbeat, scoped to the current container
|
|
|
14
14
|
incarnation. No application boot, no network call, and no healthy replica
|
|
15
15
|
hiding a stalled one.
|
|
16
16
|
|
|
17
|
-
That restart scoping
|
|
18
|
-
namespace
|
|
19
|
-
|
|
20
|
-
|
|
17
|
+
That restart scoping needs Kubernetes' default container-private PID
|
|
18
|
+
namespace. Where the namespace links are readable, a pod that sets
|
|
19
|
+
`shareProcessNamespace` or `hostPID` is rejected outright. With the default
|
|
20
|
+
capability set they were not readable on the measured Kubernetes node: the
|
|
21
|
+
shared-PID pod was `unverifiable`, while the host-PID pod had no usable
|
|
22
|
+
generation because node init reported a zero start time. Both degradations warn
|
|
23
|
+
instead of pretending the restart guarantee still holds; see
|
|
24
|
+
[Limitations](docs/LIMITATIONS.md#container-generations-need-linux-procfs-and-a-private-pid-namespace)
|
|
25
|
+
for what each side can prove and what it does when it cannot.
|
|
21
26
|
|
|
22
27
|
## Install: one line in Rails
|
|
23
28
|
|
|
@@ -195,7 +200,7 @@ negative path before rollout; the commands and expected output are in
|
|
|
195
200
|
| [Design](docs/DESIGN.md) | health predicate, heartbeat files, hooks, configuration choices, and measurements |
|
|
196
201
|
| [Running under Kubernetes](docs/KUBERNETES.md) | full manifest, CPU/startup budgets, alerts, and live-pod verification |
|
|
197
202
|
| [Limitations](docs/LIMITATIONS.md) | cases the in-memory predicate intentionally cannot cover |
|
|
198
|
-
| [Verification scenarios](docs/VERIFYING.md) |
|
|
203
|
+
| [Verification scenarios](docs/VERIFYING.md) | twelve deliberately exercised scenarios and their observed outcomes |
|
|
199
204
|
|
|
200
205
|
## Contributing
|
|
201
206
|
|
data/docs/DESIGN.md
CHANGED
|
@@ -74,9 +74,8 @@ channel.open? && connection.open? && channel.any_consumers?
|
|
|
74
74
|
| connection open, no consumers | unhealthy | the worker silently stopped consuming — a restart is the only cure |
|
|
75
75
|
| not the whole set subscribed | unhealthy | a worker that quietly dropped out would otherwise leave the probe green |
|
|
76
76
|
|
|
77
|
-
The
|
|
78
|
-
|
|
79
|
-
workers out of five look perfectly fine one object at a time.
|
|
77
|
+
The expected count catches partial subscription: four healthy workers out of
|
|
78
|
+
five must still make the process unhealthy.
|
|
80
79
|
|
|
81
80
|
### Nothing here talks to the broker
|
|
82
81
|
|
|
@@ -101,84 +100,76 @@ The predicate learns that a consumer was cancelled because RabbitMQ sends
|
|
|
101
100
|
push, not a poll — which is what makes an in-memory predicate possible at all,
|
|
102
101
|
and also where its limits come from. See [LIMITATIONS.md](LIMITATIONS.md).
|
|
103
102
|
|
|
104
|
-
`recovering_from_network_failure?` is
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
on the next Bunny upgrade. What the guard cannot preserve is the exemption
|
|
109
|
-
itself — if the method disappears, so does "healthy while reconnecting",
|
|
110
|
-
silently. That is the thing to re-check when upgrading Bunny.
|
|
103
|
+
`recovering_from_network_failure?` is private Bunny API. The `respond_to?` guard
|
|
104
|
+
avoids turning its removal into a rescued `NoMethodError` and an unhealthy
|
|
105
|
+
verdict on every tick. Removal would still disable the recovery exemption
|
|
106
|
+
silently; re-check this API when upgrading Bunny.
|
|
111
107
|
|
|
112
108
|
## The mark files
|
|
113
109
|
|
|
114
110
|
```
|
|
115
|
-
<dir>/expected
|
|
116
|
-
<dir>/generation
|
|
117
|
-
<dir>/worker-<slot>
|
|
118
|
-
<dir>/attempt-<slot>
|
|
111
|
+
<dir>/expected how many forks the probe must wait for
|
|
112
|
+
<dir>/generation which container incarnation wrote this heartbeat
|
|
113
|
+
<dir>/worker-<slot> one per fork, refreshed every tick the fork is healthy
|
|
114
|
+
<dir>/attempt-<slot> starts of a slot that has not become healthy yet
|
|
115
|
+
<dir>/topology-unsafe the PID namespace is not this container's: fail closed
|
|
116
|
+
<dir>/topology-report what the worker has already reported about it, once
|
|
119
117
|
```
|
|
120
118
|
|
|
121
|
-
**`expected`
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
`
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
**
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
**`attempt-<slot>` is the respawn counter**, and it lives in this directory for
|
|
163
|
-
the single reason that the directory outlives the fork: a monitor caught in a
|
|
164
|
-
respawn loop is a brand-new object every few hundred milliseconds and can hold
|
|
165
|
-
no counter of its own. The file is removed once the slot becomes healthy, so in
|
|
166
|
-
steady state the directory holds `expected`, `generation`, and the
|
|
167
|
-
`worker-<slot>` marks. What the counter is for is in
|
|
119
|
+
**`expected` prevents partial startup from passing.** Checking only existing
|
|
120
|
+
marks could accept fork 0's fresh mark before fork 1 has subscribed. Declaring
|
|
121
|
+
the count tells the probe which marks must exist, not just which are fresh.
|
|
122
|
+
With a fixed runner configuration, every fork declares the same count.
|
|
123
|
+
Re-declaring on each tick also restores `expected` if the directory is
|
|
124
|
+
wiped; refreshing slot marks alone would leave the probe permanently unhealthy.
|
|
125
|
+
Live worker-count changes remain
|
|
126
|
+
[unsupported](LIMITATIONS.md#changing-the-worker-count-at-runtime-is-not-supported).
|
|
127
|
+
|
|
128
|
+
**`generation` rejects inherited heartbeats.** With usable Linux procfs and a
|
|
129
|
+
container-private PID namespace, the worker and exec probe derive it from the
|
|
130
|
+
mount namespace and PID 1 start time. Both `generation` and every slot mark must
|
|
131
|
+
match the current incarnation, so an `emptyDir` preserved across a container
|
|
132
|
+
restart cannot pass `startupProbe` until all replacement forks publish. No
|
|
133
|
+
application cache is removed. Procfs and PID-topology limits are described in
|
|
134
|
+
[LIMITATIONS.md](LIMITATIONS.md#container-generations-need-linux-procfs-and-a-private-pid-namespace).
|
|
135
|
+
|
|
136
|
+
**`topology-unsafe` carries terminal evidence from the worker.** Each participant
|
|
137
|
+
compares `/proc/1/ns/mnt` with `/proc/self/ns/mnt` independently. A worker that
|
|
138
|
+
proves them different writes this file instead of a heartbeat; the probe checks
|
|
139
|
+
it before `expected`, generation, or freshness. Omitting a new generation would
|
|
140
|
+
leave inherited marks eligible for acceptance. Weaker evidence never clears
|
|
141
|
+
this file.
|
|
142
|
+
|
|
143
|
+
**`topology-report` deduplicates worker diagnostics across forks.** The probe
|
|
144
|
+
ignores it. Claims lock the complete read-compare-write operation; atomic rename
|
|
145
|
+
alone cannot prevent concurrent claims. Clearing by truncation keeps every fork
|
|
146
|
+
locking the same inode.
|
|
147
|
+
|
|
148
|
+
**Slot names survive fork replacement.** A respawn overwrites its predecessor's
|
|
149
|
+
mark. PID-based names would leave stale files that permanently fail the probe.
|
|
150
|
+
|
|
151
|
+
`expected` and slot marks are published through temporary files and `rename`.
|
|
152
|
+
Concurrent probes therefore see complete old or new contents, never a truncated
|
|
153
|
+
count or a mark missing its generation. Temporary filenames include the writer's
|
|
154
|
+
PID so concurrent forks cannot overwrite one another's temporary files.
|
|
155
|
+
|
|
156
|
+
**`attempt-<slot>` persists an unhealthy run across fork respawns.** Its
|
|
157
|
+
generation rejects counts from a previous container when one can be derived.
|
|
158
|
+
The first healthy tick removes it. Reporting windows and generation-less scope
|
|
159
|
+
are explained in
|
|
168
160
|
[LIMITATIONS.md](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn).
|
|
169
161
|
|
|
170
162
|
The timestamp, pid, and slot in `worker-<slot>` exist for a human running
|
|
171
163
|
`kubectl exec ... cat`. Its generation is part of the probe contract; freshness
|
|
172
164
|
still comes from mtime.
|
|
173
165
|
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
disk again, which was defect number two.
|
|
166
|
+
Use tmpfs — an `emptyDir` with `medium: Memory` in Kubernetes — to keep mark
|
|
167
|
+
reads independent of disk stalls.
|
|
177
168
|
|
|
178
169
|
## Why the heartbeat file has no `require` of its own
|
|
179
170
|
|
|
180
|
-
The probe
|
|
181
|
-
each;
|
|
171
|
+
The probe entry point loads only `heartbeat.rb`. Recorded over 30 invocations
|
|
172
|
+
each; timings vary with the runtime and hardware:
|
|
182
173
|
|
|
183
174
|
| | per invocation |
|
|
184
175
|
|---|---|
|
|
@@ -188,22 +179,13 @@ each; absolute values depend on the hardware, but the ratios do not:
|
|
|
188
179
|
| the probe, found through RubyGems | 43 ms |
|
|
189
180
|
| **`bundle exec ruby -e ''`** | **300 ms** |
|
|
190
181
|
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
`require` for under 50 ms. Skipping RubyGems entirely is faster still, but it
|
|
199
|
-
requires naming the path to the gem's `lib` directory; see
|
|
200
|
-
[KUBERNETES.md](KUBERNETES.md) for when that trade is worth making.
|
|
201
|
-
|
|
202
|
-
Keeping the probe's path through the heartbeat file free of `require`
|
|
203
|
-
statements — and the file free of references to the rest of the gem — is what
|
|
204
|
-
makes all three possible. The single `require` the file does contain,
|
|
205
|
-
`fileutils`, is lazy and sits in the directory-creation branch that only the
|
|
206
|
-
worker ever reaches.
|
|
182
|
+
Interpreter and dependency startup dominate these measurements. `bundle exec`
|
|
183
|
+
is the portable default; direct loading depends on where the image installs
|
|
184
|
+
the gem, as described in [KUBERNETES.md](KUBERNETES.md#where-your-image-puts-its-gems).
|
|
185
|
+
`heartbeat.rb` therefore references no other gem code and requires `fileutils`
|
|
186
|
+
only in the worker's directory-creation branch. Worker-only counters and report
|
|
187
|
+
deduplication live in `Attempts` and `TopologyReport` so their code and requires
|
|
188
|
+
stay outside the probe's load path.
|
|
207
189
|
|
|
208
190
|
## Where the hooks attach
|
|
209
191
|
|
|
@@ -217,56 +199,36 @@ Railtie in Rails), covering four methods:
|
|
|
217
199
|
- `Sneakers::WorkerGroup#stop` — record that the shutdown is deliberate, *before*
|
|
218
200
|
`super` starts unsubscribing.
|
|
219
201
|
|
|
220
|
-
`
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
upgrade that reorders `run` fails in CI rather than in production.
|
|
231
|
-
|
|
232
|
-
`ServerEngine::Server#create_worker` calls `w.extend(@worker_module)`: the
|
|
233
|
-
`WorkerGroup` module lands in the singleton class of the *instance*, not as an
|
|
234
|
-
`include` in a class. A `prepend` to the module still sits ahead of it in that
|
|
235
|
-
lookup chain, which is what makes the hook work; `worker_id` and `config` come
|
|
236
|
-
from `ServerEngine::Worker`. Verified on serverengine 2.4.0, and covered by a
|
|
237
|
-
spec that asserts the order in `ancestors` — this is the most fragile assumption
|
|
238
|
-
in the gem, so it fails in CI rather than in production.
|
|
202
|
+
`WorkerGroup#run` calls `after_fork` before resolving `worker_classes`. Starting
|
|
203
|
+
after that hook lets the application restore inherited connections before the
|
|
204
|
+
consumer count runs application code. This matters for the callable registry
|
|
205
|
+
used by `sneakers:active_job`; resolving it too early can prevent monitor startup.
|
|
206
|
+
A spec pins this ordering against the worker gem.
|
|
207
|
+
|
|
208
|
+
ServerEngine 2.4.0 uses `w.extend(@worker_module)`, placing `WorkerGroup` in the
|
|
209
|
+
instance's singleton-class chain. Prepending to the module still intercepts its
|
|
210
|
+
methods; an `ancestors` spec verifies that assumption. `worker_id` and `config`
|
|
211
|
+
come from `ServerEngine::Worker`.
|
|
239
212
|
|
|
240
213
|
Slot numbers come from `worker_id`, which is the index into ServerEngine's
|
|
241
214
|
`@monitors` array, and both `start_new_worker(wid)` and `restart_worker(wid)`
|
|
242
215
|
reuse the same index. That is what the slot-not-PID scheme relies on.
|
|
243
216
|
|
|
244
|
-
The monitor thread is created inside the fork
|
|
245
|
-
`fork`.
|
|
246
|
-
|
|
247
|
-
single-threaded, before any fork, so no lazy initialization is needed and none
|
|
248
|
-
can race.
|
|
249
|
-
|
|
250
|
-
In Rails the hooks are installed from a Railtie, and from `to_prepare` rather
|
|
251
|
-
than an `initializer`. An application's `lib` is normally managed by Zeitwerk and
|
|
252
|
-
reloadable, and reloadable constants must not be referenced while the
|
|
253
|
-
application is initializing.
|
|
217
|
+
The monitor thread is created inside the fork because threads do not survive
|
|
218
|
+
`fork`. The registry's array and mutex are initialized at load time, before
|
|
219
|
+
forking, avoiding races during lazy initialization.
|
|
254
220
|
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
221
|
+
The Rails Railtie registers installation through `to_prepare`, allowing access
|
|
222
|
+
to reloadable worker constants after initialization. Startup exceptions, from
|
|
223
|
+
either application code or an unwritable marks directory, are logged without
|
|
224
|
+
preventing the worker from starting. The probe remains responsible for detecting
|
|
225
|
+
a missing or stale heartbeat.
|
|
260
226
|
|
|
261
227
|
### The expected consumer count is not configured
|
|
262
228
|
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
That value is an array of classes under `sneakers:run` and a callable registry
|
|
268
|
-
under `sneakers:active_job`, so it is resolved with a `respond_to?(:call)`
|
|
269
|
-
check.
|
|
229
|
+
`config[:worker_classes]` is the set the worker gem itself consumes: an array
|
|
230
|
+
under `sneakers:run`, a callable registry under `sneakers:active_job`. Resolve it
|
|
231
|
+
with `respond_to?(:call)` rather than maintaining a separate queue list.
|
|
270
232
|
|
|
271
233
|
## What is configurable, and where
|
|
272
234
|
|
|
@@ -276,52 +238,28 @@ check.
|
|
|
276
238
|
| `tick` | either, and the config block wins | only the worker reads it, so two sources cannot contradict each other |
|
|
277
239
|
| `dir`, `max_age` | environment variables **only** | the probe is a separate process |
|
|
278
240
|
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
makes every mark stale on arrival, so the probe can never pass again. They fall
|
|
302
|
-
back to the default too.
|
|
303
|
-
|
|
304
|
-
Setting `tick` from the application is held to a stricter standard: a
|
|
305
|
-
non-positive value raises `ArgumentError`, as does a value greater than or equal
|
|
306
|
-
to `max_age`: such a monitor would inevitably let a healthy mark go stale. The
|
|
307
|
-
environment gets a silent fallback for values that cannot be parsed. If its
|
|
308
|
-
effective tick is greater than or equal to `max_age`, configuration emits a
|
|
309
|
-
warning on stderr and uses the default tick when it is safe, or half of `max_age`
|
|
310
|
-
otherwise. A ConfigMap typo must not bring a worker down, whereas an initializer
|
|
311
|
-
is code and should fail loudly at boot, where the developer is looking.
|
|
312
|
-
|
|
313
|
-
`startup_grace_ticks` must be a positive integer. It is compared directly with
|
|
314
|
-
an integer counter; accepting zero, a float, or a string would silently disable
|
|
315
|
-
the escalation that is supposed to diagnose a worker that never subscribes.
|
|
316
|
-
|
|
317
|
-
The logger defaults to `Sneakers.logger` but is resolved lazily, because at the
|
|
318
|
-
time the configuration object is built it may not be set up yet.
|
|
319
|
-
|
|
320
|
-
Logging is diagnostic; the heartbeat is the liveness contract. An exception
|
|
321
|
-
raised by a custom logger is therefore swallowed at the logging boundary. It
|
|
322
|
-
cannot prevent a healthy tick from writing its mark, abort monitor startup, or
|
|
323
|
-
escape the loop's error handler and kill the monitor thread. No fallback message
|
|
324
|
-
is attempted through the same broken logger.
|
|
241
|
+
The exec probe cannot read an application initializer, so `dir` and `max_age`
|
|
242
|
+
have one source: the environment. Specs reject application setters for both.
|
|
243
|
+
Only the worker reads `tick`, so `config.tick` may override
|
|
244
|
+
`KICKS_LIVENESS_TICK` without contradicting the probe.
|
|
245
|
+
|
|
246
|
+
Environment durations must be positive integers; missing, empty, unparseable,
|
|
247
|
+
zero, and negative values use defaults. If the effective tick is at least
|
|
248
|
+
`max_age`, it warns on stderr and uses the default tick when safe, or half of
|
|
249
|
+
`max_age` otherwise. The warning is deferred until the tick is read, so an
|
|
250
|
+
initializer override or a disabled monitor does not produce a misleading WARN.
|
|
251
|
+
|
|
252
|
+
Invalid ConfigMap input falls back so a deployment typo does not prevent worker
|
|
253
|
+
startup. Initializer values are application code: invalid values raise
|
|
254
|
+
`ArgumentError` at boot, where the developer can fix the error, rather than
|
|
255
|
+
silently substituting a default. `tick` must be a positive number smaller than
|
|
256
|
+
`max_age`, and `startup_grace_ticks` a positive integer. The latter is compared
|
|
257
|
+
with an integer counter; accepting a float or zero would disable escalation.
|
|
258
|
+
|
|
259
|
+
The default logger is resolved lazily from `Sneakers.logger`, which may be unset
|
|
260
|
+
during configuration. Logger exceptions are swallowed at the logging boundary
|
|
261
|
+
so they cannot prevent heartbeat publication or stop the monitor. No fallback
|
|
262
|
+
uses the same broken logger.
|
|
325
263
|
|
|
326
264
|
## Logging: events, not the pulse
|
|
327
265
|
|
|
@@ -331,30 +269,29 @@ add noise. What gets logged is transitions.
|
|
|
331
269
|
| Event | Level |
|
|
332
270
|
|---|---|
|
|
333
271
|
| started, with the effective settings | INFO |
|
|
272
|
+
| the PID topology could not be verified, or yielded no generation | WARN |
|
|
334
273
|
| waiting for consumers | INFO |
|
|
335
274
|
| became healthy | INFO |
|
|
336
275
|
| shutting down | INFO |
|
|
337
276
|
| still not healthy after `startup_grace_ticks` ticks | ERROR |
|
|
338
277
|
| was healthy, became unhealthy | ERROR |
|
|
339
278
|
| a slot respawning without ever becoming healthy, once per grace window | ERROR |
|
|
279
|
+
| the PID topology is proven unsupported | ERROR |
|
|
340
280
|
| the monitor thread caught an exception | ERROR |
|
|
341
281
|
| the monitor could not be started at all | ERROR |
|
|
342
282
|
|
|
343
|
-
|
|
344
|
-
`
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
look identical for the first few ticks. Once the grace expires, staying silent
|
|
359
|
-
would mean a pod that never comes up never says so — hence exactly one ERROR,
|
|
360
|
-
not one per tick.
|
|
283
|
+
`topology-report` deduplicates only the three topology results: `unverifiable`,
|
|
284
|
+
`unavailable`, and `unsafe`. Their warning frequency, probe output, and recovery
|
|
285
|
+
are described in
|
|
286
|
+
[LIMITATIONS.md](LIMITATIONS.md#container-generations-need-linux-procfs-and-a-private-pid-namespace).
|
|
287
|
+
Loop and startup exceptions do not use that flag.
|
|
288
|
+
|
|
289
|
+
ERROR must survive the usual error-level logger filter, while normal startup
|
|
290
|
+
and deliberate shutdown remain INFO. On shutdown the monitor suspends its
|
|
291
|
+
predicate and refreshes the mark for the rest of the process, assuming an
|
|
292
|
+
[external deadline](LIMITATIONS.md#graceful-shutdown-is-safe-only-when-something-else-owns-the-deadline).
|
|
293
|
+
|
|
294
|
+
Startup grace postpones the error for a slow initial subscription; a surviving
|
|
295
|
+
monitor reports once when it expires. Fork-respawn loops use the persistent
|
|
296
|
+
counter and reporting windows described in
|
|
297
|
+
[LIMITATIONS.md](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn).
|
data/docs/KUBERNETES.md
CHANGED
|
@@ -55,11 +55,16 @@ mark. A fresh mark from the previous container is rejected until every fork in
|
|
|
55
55
|
the current one has subscribed and published its own mark, so preserving the
|
|
56
56
|
cache cannot make `startupProbe` succeed early.
|
|
57
57
|
|
|
58
|
-
That guarantee
|
|
59
|
-
`shareProcessNamespace: true` or `hostPID: true
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
58
|
+
That guarantee needs the default container-private PID namespace. Do not set
|
|
59
|
+
`shareProcessNamespace: true` or `hostPID: true`: in either topology PID 1
|
|
60
|
+
survives a worker-container restart, so the generation repeats and an inherited
|
|
61
|
+
mark would be accepted. Since 0.1.3 the gem rejects that topology when either
|
|
62
|
+
the worker or probe can read and compare both namespace links. With Kubernetes'
|
|
63
|
+
default capability set those reads failed on the measured node: the shared-PID
|
|
64
|
+
pod was `unverifiable`, and the host-PID pod was `unavailable` because node init
|
|
65
|
+
reported a zero start time. Those states warn and retain the documented degraded
|
|
66
|
+
checks; they do not claim the topology is safe. See
|
|
67
|
+
[LIMITATIONS.md](LIMITATIONS.md#container-generations-need-linux-procfs-and-a-private-pid-namespace).
|
|
63
68
|
|
|
64
69
|
## Why the command looks like that
|
|
65
70
|
|
|
@@ -121,9 +126,12 @@ which is not something to rely on in a manifest.
|
|
|
121
126
|
|
|
122
127
|
Keep the worker and probe on the same gem version. Since 0.1.2 the heartbeat
|
|
123
128
|
protocol includes a container generation; a newer probe correctly rejects the
|
|
124
|
-
generation-less files written by an older worker.
|
|
125
|
-
|
|
126
|
-
|
|
129
|
+
generation-less files written by an older worker. Since 0.1.3 it also includes
|
|
130
|
+
the fail-closed topology state, which only a 0.1.3 probe reads: an older probe
|
|
131
|
+
beside a 0.1.3 worker would ignore a proven unsupported PID topology instead of
|
|
132
|
+
failing on it. `bundle exec` guarantees that both sides resolve from the same
|
|
133
|
+
bundle. If the faster form below installs a second copy in `GEM_HOME`, rebuild
|
|
134
|
+
that copy on every gem upgrade too.
|
|
127
135
|
|
|
128
136
|
### Where your image puts its gems
|
|
129
137
|
|