kicks_liveness 0.1.2 → 0.1.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +74 -1
- data/README.md +21 -6
- data/docs/DESIGN.md +133 -188
- data/docs/KUBERNETES.md +40 -13
- data/docs/LIMITATIONS.md +154 -61
- data/docs/SETUP.md +30 -0
- data/docs/VERIFYING.md +269 -13
- data/lib/kicks_liveness/attempts.rb +21 -16
- data/lib/kicks_liveness/configuration.rb +24 -9
- data/lib/kicks_liveness/heartbeat.rb +241 -111
- data/lib/kicks_liveness/hooks.rb +11 -29
- data/lib/kicks_liveness/monitor.rb +50 -51
- data/lib/kicks_liveness/registry.rb +17 -19
- data/lib/kicks_liveness/topology_report.rb +110 -0
- data/lib/kicks_liveness/version.rb +1 -1
- data/lib/kicks_liveness.rb +3 -1
- metadata +3 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: faf817806eb5142b8271959b7856a16db10a99db8eed583bb5594887198b6dd8
|
|
4
|
+
data.tar.gz: 9c5f3f845c78a56e5da97204769e5c2b10161fbf56e5c40a731b904650ca469a
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: a077fc085bed4c460ed9820940271fedb44e69a2778ca87997b23c172fe94d0622bbf334d9b5456827d1e231813ec5e903de6ebe6c655804f8b8c359ca26832c
|
|
7
|
+
data.tar.gz: 1d2efdab9aa59777cc14075703c9c75bfad76dc511d94659a8f2f40d2cc8a3a957779e4795f304beabbed04c4854f88e2413fd044216497df7338d6a7a32a864
|
data/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,77 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
|
|
|
5
5
|
|
|
6
6
|
## [Unreleased]
|
|
7
7
|
|
|
8
|
+
## [0.1.4] - 2026-11-03
|
|
9
|
+
|
|
10
|
+
### Fixed
|
|
11
|
+
|
|
12
|
+
- Bound the graceful-shutdown heartbeat exemption by a monotonic deadline
|
|
13
|
+
starting at the first stop notification; repeated callbacks cannot extend it.
|
|
14
|
+
At expiry the monitor logs one ERROR and stops refreshing its mark, but still
|
|
15
|
+
declares the fork count. A stuck drain then fails the ordinary freshness
|
|
16
|
+
check; the monitor does not kill the process.
|
|
17
|
+
|
|
18
|
+
### Added
|
|
19
|
+
|
|
20
|
+
- Add `shutdown_grace` / `KICKS_LIVENESS_SHUTDOWN_GRACE` (default 60 seconds;
|
|
21
|
+
initializer wins). Set it to at least `terminationGracePeriodSeconds`.
|
|
22
|
+
Restart still waits for mark staleness and the kubelet's probe failure budget.
|
|
23
|
+
- Add `verify.sh shutdown` and `container-restart` to check a real drain and
|
|
24
|
+
same-pod container replacement, with deployment restoration afterwards.
|
|
25
|
+
- Add an `install!` setup hint when the probe finds no declared worker count.
|
|
26
|
+
It remains a failing check, not proof that the hooks were never installed.
|
|
27
|
+
|
|
28
|
+
## [0.1.3] - 2026-10-21
|
|
29
|
+
|
|
30
|
+
### Fixed
|
|
31
|
+
|
|
32
|
+
- Detect and reject PID topologies in which the container generation cannot
|
|
33
|
+
identify an incarnation. `shareProcessNamespace: true` and `hostPID: true`
|
|
34
|
+
leave PID 1 alive across a restart of the worker container, so the 0.1.2
|
|
35
|
+
identifier can repeat and an inherited fresh heartbeat could satisfy
|
|
36
|
+
`startupProbe`. Rejection is conditional on a readable `/proc/1/ns/mnt`.
|
|
37
|
+
Where the comparison can be made, the worker publishes `<dir>/topology-unsafe`
|
|
38
|
+
instead of a heartbeat and every probe exits 1 with
|
|
39
|
+
`unsupported PID topology: pod must not set shareProcessNamespace or hostPID`,
|
|
40
|
+
including one that cannot read the link itself; the state is terminal for the
|
|
41
|
+
marks volume, and recovery is recreating the pod without those settings.
|
|
42
|
+
Where it cannot, the previous behavior remains and the worker warns once. The
|
|
43
|
+
measured outcomes are in `docs/LIMITATIONS.md`.
|
|
44
|
+
- Distinguish a valid zero in PID 1's stat start-time field from a stat line
|
|
45
|
+
that cannot be parsed: the generation stays unavailable, and the warning
|
|
46
|
+
names `hostPID` or a virtualized procfs instead of a false parse error.
|
|
47
|
+
- Scope the `attempt-<slot>` respawn counter to the container generation. A
|
|
48
|
+
replacement container starts its own unhealthy run and cannot escalate over
|
|
49
|
+
starts it did not take, while respawns within one generation go on sharing a
|
|
50
|
+
count. Where no generation is available the counter keeps its previous
|
|
51
|
+
volume-wide behavior.
|
|
52
|
+
|
|
53
|
+
### Added
|
|
54
|
+
|
|
55
|
+
- Report a degraded container generation once from the worker, at WARN — once
|
|
56
|
+
per container incarnation where a generation exists, once per error state
|
|
57
|
+
where it does not. Neither degraded state adds anything to the probe's
|
|
58
|
+
output, and the intentional non-Linux fallback stays silent.
|
|
59
|
+
- The marks directory gains two files that appear only when something is
|
|
60
|
+
wrong: `topology-unsafe`, the terminal fail-closed state, and
|
|
61
|
+
`topology-report`, the worker's own flag for saying each thing once. Tooling
|
|
62
|
+
that enumerates the directory should allow for both.
|
|
63
|
+
- Document the boundary of the graceful-shutdown exemption in
|
|
64
|
+
`docs/LIMITATIONS.md`: a SIGTERM that does not come from a pod deletion can
|
|
65
|
+
leave a pod Ready with no consumers. Existing behavior, newly documented.
|
|
66
|
+
- Add `spec/integration/verify.sh topology`, a Linux matrix covering a shared
|
|
67
|
+
PID namespace, the host PID namespace, a verified private-PID container, and
|
|
68
|
+
a private-PID container whose checker cannot read `/proc/1/ns/mnt`.
|
|
69
|
+
|
|
70
|
+
### Changed
|
|
71
|
+
|
|
72
|
+
- `Heartbeat#declare!` documents its actual `Integer` return value.
|
|
73
|
+
- Replace `Heartbeat.container_generation` with `Heartbeat.topology`, returning
|
|
74
|
+
a structured result instead of a string or `nil`; replace the `Heartbeat.new`
|
|
75
|
+
keyword `generation:` with `topology:`, accepting that result; and remove
|
|
76
|
+
`KicksLiveness::GenerationGuard`. All are `@api private`, but callers need
|
|
77
|
+
updating.
|
|
78
|
+
|
|
8
79
|
## [0.1.2] - 2026-09-11
|
|
9
80
|
|
|
10
81
|
### Fixed
|
|
@@ -73,7 +144,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
|
|
|
73
144
|
[docs/](https://github.com/PoroshkinaVV/kicks_liveness/tree/main/docs); start
|
|
74
145
|
with `SETUP.md`, and read `LIMITATIONS.md` before relying on it.
|
|
75
146
|
|
|
76
|
-
[Unreleased]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.
|
|
147
|
+
[Unreleased]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.4...HEAD
|
|
148
|
+
[0.1.4]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.3...v0.1.4
|
|
149
|
+
[0.1.3]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.2...v0.1.3
|
|
77
150
|
[0.1.2]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.1...v0.1.2
|
|
78
151
|
[0.1.1]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.0...v0.1.1
|
|
79
152
|
[0.1.0]: https://github.com/PoroshkinaVV/kicks_liveness/releases/tag/v0.1.0
|
data/README.md
CHANGED
|
@@ -14,10 +14,15 @@ The probe only reads that heartbeat, scoped to the current container
|
|
|
14
14
|
incarnation. No application boot, no network call, and no healthy replica
|
|
15
15
|
hiding a stalled one.
|
|
16
16
|
|
|
17
|
-
That restart scoping
|
|
18
|
-
namespace
|
|
19
|
-
|
|
20
|
-
|
|
17
|
+
That restart scoping needs Kubernetes' default container-private PID
|
|
18
|
+
namespace. Where the namespace links are readable, a pod that sets
|
|
19
|
+
`shareProcessNamespace` or `hostPID` is rejected outright. With the default
|
|
20
|
+
capability set they were not readable on the measured Kubernetes node: the
|
|
21
|
+
shared-PID pod was `unverifiable`, while the host-PID pod had no usable
|
|
22
|
+
generation because node init reported a zero start time. Both degradations warn
|
|
23
|
+
instead of pretending the restart guarantee still holds; see
|
|
24
|
+
[Limitations](docs/LIMITATIONS.md#container-generations-need-linux-procfs-and-a-private-pid-namespace)
|
|
25
|
+
for what each side can prove and what it does when it cannot.
|
|
21
26
|
|
|
22
27
|
## Install: one line in Rails
|
|
23
28
|
|
|
@@ -72,6 +77,9 @@ only when every expected worker mark is present and fresh.
|
|
|
72
77
|
Use a startup probe for the boot budget and a liveness probe for steady state:
|
|
73
78
|
|
|
74
79
|
```yaml
|
|
80
|
+
env:
|
|
81
|
+
- name: KICKS_LIVENESS_SHUTDOWN_GRACE
|
|
82
|
+
value: "60" # at least terminationGracePeriodSeconds
|
|
75
83
|
startupProbe:
|
|
76
84
|
exec:
|
|
77
85
|
command: ["bundle", "exec", "kicks-liveness"]
|
|
@@ -134,6 +142,7 @@ environment too:
|
|
|
134
142
|
| `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | heartbeat directory; must be on tmpfs | uses the default |
|
|
135
143
|
| `KICKS_LIVENESS_MAX_AGE` | `45` | seconds before a mark is stale | uses the default |
|
|
136
144
|
| `KICKS_LIVENESS_TICK` | `10` | seconds between worker checks | uses the default, then the safety fallback below if needed |
|
|
145
|
+
| `KICKS_LIVENESS_SHUTDOWN_GRACE` | `60` | seconds to keep publishing during shutdown | uses the default |
|
|
137
146
|
|
|
138
147
|
If the effective environment tick is not smaller than `max_age`, the worker
|
|
139
148
|
writes a WARN to stderr and uses `10` seconds when that is safe, or half of
|
|
@@ -141,10 +150,16 @@ writes a WARN to stderr and uses `10` seconds when that is safe, or half of
|
|
|
141
150
|
while guaranteeing that a healthy heartbeat cannot become stale between
|
|
142
151
|
ordinary ticks.
|
|
143
152
|
|
|
144
|
-
|
|
153
|
+
Initializer values for `tick` and `shutdown_grace` override their environment
|
|
154
|
+
variables. `dir` and
|
|
145
155
|
`max_age` deliberately have no initializer setters: allowing two configuration
|
|
146
156
|
sources could make the worker and probe silently disagree.
|
|
147
157
|
|
|
158
|
+
Set `shutdown_grace` to at least `terminationGracePeriodSeconds`. A drain that
|
|
159
|
+
exceeds this deadline stops refreshing its heartbeat and logs one ERROR; the
|
|
160
|
+
probe fails once the last mark becomes stale. See
|
|
161
|
+
[shutdown timing](docs/LIMITATIONS.md#graceful-shutdown-has-a-finite-heartbeat-exemption).
|
|
162
|
+
|
|
148
163
|
If two runners share one pod — for example `sneakers:run` and
|
|
149
164
|
`sneakers:active_job` — give each a different `KICKS_LIVENESS_DIR`. Separate
|
|
150
165
|
pods already have separate `emptyDir` volumes.
|
|
@@ -195,7 +210,7 @@ negative path before rollout; the commands and expected output are in
|
|
|
195
210
|
| [Design](docs/DESIGN.md) | health predicate, heartbeat files, hooks, configuration choices, and measurements |
|
|
196
211
|
| [Running under Kubernetes](docs/KUBERNETES.md) | full manifest, CPU/startup budgets, alerts, and live-pod verification |
|
|
197
212
|
| [Limitations](docs/LIMITATIONS.md) | cases the in-memory predicate intentionally cannot cover |
|
|
198
|
-
| [Verification scenarios](docs/VERIFYING.md) |
|
|
213
|
+
| [Verification scenarios](docs/VERIFYING.md) | twelve deliberately exercised scenarios and their observed outcomes |
|
|
199
214
|
|
|
200
215
|
## Contributing
|
|
201
216
|
|
data/docs/DESIGN.md
CHANGED
|
@@ -74,9 +74,8 @@ channel.open? && connection.open? && channel.any_consumers?
|
|
|
74
74
|
| connection open, no consumers | unhealthy | the worker silently stopped consuming — a restart is the only cure |
|
|
75
75
|
| not the whole set subscribed | unhealthy | a worker that quietly dropped out would otherwise leave the probe green |
|
|
76
76
|
|
|
77
|
-
The
|
|
78
|
-
|
|
79
|
-
workers out of five look perfectly fine one object at a time.
|
|
77
|
+
The expected count catches partial subscription: four healthy workers out of
|
|
78
|
+
five must still make the process unhealthy.
|
|
80
79
|
|
|
81
80
|
### Nothing here talks to the broker
|
|
82
81
|
|
|
@@ -101,84 +100,76 @@ The predicate learns that a consumer was cancelled because RabbitMQ sends
|
|
|
101
100
|
push, not a poll — which is what makes an in-memory predicate possible at all,
|
|
102
101
|
and also where its limits come from. See [LIMITATIONS.md](LIMITATIONS.md).
|
|
103
102
|
|
|
104
|
-
`recovering_from_network_failure?` is
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
on the next Bunny upgrade. What the guard cannot preserve is the exemption
|
|
109
|
-
itself — if the method disappears, so does "healthy while reconnecting",
|
|
110
|
-
silently. That is the thing to re-check when upgrading Bunny.
|
|
103
|
+
`recovering_from_network_failure?` is private Bunny API. The `respond_to?` guard
|
|
104
|
+
avoids turning its removal into a rescued `NoMethodError` and an unhealthy
|
|
105
|
+
verdict on every tick. Removal would still disable the recovery exemption
|
|
106
|
+
silently; re-check this API when upgrading Bunny.
|
|
111
107
|
|
|
112
108
|
## The mark files
|
|
113
109
|
|
|
114
110
|
```
|
|
115
|
-
<dir>/expected
|
|
116
|
-
<dir>/generation
|
|
117
|
-
<dir>/worker-<slot>
|
|
118
|
-
<dir>/attempt-<slot>
|
|
111
|
+
<dir>/expected how many forks the probe must wait for
|
|
112
|
+
<dir>/generation which container incarnation wrote this heartbeat
|
|
113
|
+
<dir>/worker-<slot> one per fork, refreshed every tick the fork is healthy
|
|
114
|
+
<dir>/attempt-<slot> starts of a slot that has not become healthy yet
|
|
115
|
+
<dir>/topology-unsafe the PID namespace is not this container's: fail closed
|
|
116
|
+
<dir>/topology-report what the worker has already reported about it, once
|
|
119
117
|
```
|
|
120
118
|
|
|
121
|
-
**`expected`
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
`
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
**
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
**`attempt-<slot>` is the respawn counter**, and it lives in this directory for
|
|
163
|
-
the single reason that the directory outlives the fork: a monitor caught in a
|
|
164
|
-
respawn loop is a brand-new object every few hundred milliseconds and can hold
|
|
165
|
-
no counter of its own. The file is removed once the slot becomes healthy, so in
|
|
166
|
-
steady state the directory holds `expected`, `generation`, and the
|
|
167
|
-
`worker-<slot>` marks. What the counter is for is in
|
|
119
|
+
**`expected` prevents partial startup from passing.** Checking only existing
|
|
120
|
+
marks could accept fork 0's fresh mark before fork 1 has subscribed. Declaring
|
|
121
|
+
the count tells the probe which marks must exist, not just which are fresh.
|
|
122
|
+
With a fixed runner configuration, every fork declares the same count.
|
|
123
|
+
Re-declaring on each tick also restores `expected` if the directory is
|
|
124
|
+
wiped; refreshing slot marks alone would leave the probe permanently unhealthy.
|
|
125
|
+
Live worker-count changes remain
|
|
126
|
+
[unsupported](LIMITATIONS.md#changing-the-worker-count-at-runtime-is-not-supported).
|
|
127
|
+
|
|
128
|
+
**`generation` rejects inherited heartbeats.** With usable Linux procfs and a
|
|
129
|
+
container-private PID namespace, the worker and exec probe derive it from the
|
|
130
|
+
mount namespace and PID 1 start time. Both `generation` and every slot mark must
|
|
131
|
+
match the current incarnation, so an `emptyDir` preserved across a container
|
|
132
|
+
restart cannot pass `startupProbe` until all replacement forks publish. No
|
|
133
|
+
application cache is removed. Procfs and PID-topology limits are described in
|
|
134
|
+
[LIMITATIONS.md](LIMITATIONS.md#container-generations-need-linux-procfs-and-a-private-pid-namespace).
|
|
135
|
+
|
|
136
|
+
**`topology-unsafe` carries terminal evidence from the worker.** Each participant
|
|
137
|
+
compares `/proc/1/ns/mnt` with `/proc/self/ns/mnt` independently. A worker that
|
|
138
|
+
proves them different writes this file instead of a heartbeat; the probe checks
|
|
139
|
+
it before `expected`, generation, or freshness. Omitting a new generation would
|
|
140
|
+
leave inherited marks eligible for acceptance. Weaker evidence never clears
|
|
141
|
+
this file.
|
|
142
|
+
|
|
143
|
+
**`topology-report` deduplicates worker diagnostics across forks.** The probe
|
|
144
|
+
ignores it. Claims lock the complete read-compare-write operation; atomic rename
|
|
145
|
+
alone cannot prevent concurrent claims. Clearing by truncation keeps every fork
|
|
146
|
+
locking the same inode.
|
|
147
|
+
|
|
148
|
+
**Slot names survive fork replacement.** A respawn overwrites its predecessor's
|
|
149
|
+
mark. PID-based names would leave stale files that permanently fail the probe.
|
|
150
|
+
|
|
151
|
+
`expected` and slot marks are published through temporary files and `rename`.
|
|
152
|
+
Concurrent probes therefore see complete old or new contents, never a truncated
|
|
153
|
+
count or a mark missing its generation. Temporary filenames include the writer's
|
|
154
|
+
PID so concurrent forks cannot overwrite one another's temporary files.
|
|
155
|
+
|
|
156
|
+
**`attempt-<slot>` persists an unhealthy run across fork respawns.** Its
|
|
157
|
+
generation rejects counts from a previous container when one can be derived.
|
|
158
|
+
The first healthy tick removes it. Reporting windows and generation-less scope
|
|
159
|
+
are explained in
|
|
168
160
|
[LIMITATIONS.md](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn).
|
|
169
161
|
|
|
170
162
|
The timestamp, pid, and slot in `worker-<slot>` exist for a human running
|
|
171
163
|
`kubectl exec ... cat`. Its generation is part of the probe contract; freshness
|
|
172
164
|
still comes from mtime.
|
|
173
165
|
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
disk again, which was defect number two.
|
|
166
|
+
Use tmpfs — an `emptyDir` with `medium: Memory` in Kubernetes — to keep mark
|
|
167
|
+
reads independent of disk stalls.
|
|
177
168
|
|
|
178
169
|
## Why the heartbeat file has no `require` of its own
|
|
179
170
|
|
|
180
|
-
The probe
|
|
181
|
-
each;
|
|
171
|
+
The probe entry point loads only `heartbeat.rb`. Recorded over 30 invocations
|
|
172
|
+
each; timings vary with the runtime and hardware:
|
|
182
173
|
|
|
183
174
|
| | per invocation |
|
|
184
175
|
|---|---|
|
|
@@ -188,22 +179,13 @@ each; absolute values depend on the hardware, but the ratios do not:
|
|
|
188
179
|
| the probe, found through RubyGems | 43 ms |
|
|
189
180
|
| **`bundle exec ruby -e ''`** | **300 ms** |
|
|
190
181
|
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
`require` for under 50 ms. Skipping RubyGems entirely is faster still, but it
|
|
199
|
-
requires naming the path to the gem's `lib` directory; see
|
|
200
|
-
[KUBERNETES.md](KUBERNETES.md) for when that trade is worth making.
|
|
201
|
-
|
|
202
|
-
Keeping the probe's path through the heartbeat file free of `require`
|
|
203
|
-
statements — and the file free of references to the rest of the gem — is what
|
|
204
|
-
makes all three possible. The single `require` the file does contain,
|
|
205
|
-
`fileutils`, is lazy and sits in the directory-creation branch that only the
|
|
206
|
-
worker ever reaches.
|
|
182
|
+
Interpreter and dependency startup dominate these measurements. `bundle exec`
|
|
183
|
+
is the portable default; direct loading depends on where the image installs
|
|
184
|
+
the gem, as described in [KUBERNETES.md](KUBERNETES.md#where-your-image-puts-its-gems).
|
|
185
|
+
`heartbeat.rb` therefore references no other gem code and requires `fileutils`
|
|
186
|
+
only in the worker's directory-creation branch. Worker-only counters and report
|
|
187
|
+
deduplication live in `Attempts` and `TopologyReport` so their code and requires
|
|
188
|
+
stay outside the probe's load path.
|
|
207
189
|
|
|
208
190
|
## Where the hooks attach
|
|
209
191
|
|
|
@@ -217,111 +199,71 @@ Railtie in Rails), covering four methods:
|
|
|
217
199
|
- `Sneakers::WorkerGroup#stop` — record that the shutdown is deliberate, *before*
|
|
218
200
|
`super` starts unsubscribing.
|
|
219
201
|
|
|
220
|
-
`
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
upgrade that reorders `run` fails in CI rather than in production.
|
|
231
|
-
|
|
232
|
-
`ServerEngine::Server#create_worker` calls `w.extend(@worker_module)`: the
|
|
233
|
-
`WorkerGroup` module lands in the singleton class of the *instance*, not as an
|
|
234
|
-
`include` in a class. A `prepend` to the module still sits ahead of it in that
|
|
235
|
-
lookup chain, which is what makes the hook work; `worker_id` and `config` come
|
|
236
|
-
from `ServerEngine::Worker`. Verified on serverengine 2.4.0, and covered by a
|
|
237
|
-
spec that asserts the order in `ancestors` — this is the most fragile assumption
|
|
238
|
-
in the gem, so it fails in CI rather than in production.
|
|
202
|
+
`WorkerGroup#run` calls `after_fork` before resolving `worker_classes`. Starting
|
|
203
|
+
after that hook lets the application restore inherited connections before the
|
|
204
|
+
consumer count runs application code. This matters for the callable registry
|
|
205
|
+
used by `sneakers:active_job`; resolving it too early can prevent monitor startup.
|
|
206
|
+
A spec pins this ordering against the worker gem.
|
|
207
|
+
|
|
208
|
+
ServerEngine 2.4.0 uses `w.extend(@worker_module)`, placing `WorkerGroup` in the
|
|
209
|
+
instance's singleton-class chain. Prepending to the module still intercepts its
|
|
210
|
+
methods; an `ancestors` spec verifies that assumption. `worker_id` and `config`
|
|
211
|
+
come from `ServerEngine::Worker`.
|
|
239
212
|
|
|
240
213
|
Slot numbers come from `worker_id`, which is the index into ServerEngine's
|
|
241
214
|
`@monitors` array, and both `start_new_worker(wid)` and `restart_worker(wid)`
|
|
242
215
|
reuse the same index. That is what the slot-not-PID scheme relies on.
|
|
243
216
|
|
|
244
|
-
The monitor thread is created inside the fork
|
|
245
|
-
`fork`.
|
|
246
|
-
|
|
247
|
-
single-threaded, before any fork, so no lazy initialization is needed and none
|
|
248
|
-
can race.
|
|
217
|
+
The monitor thread is created inside the fork because threads do not survive
|
|
218
|
+
`fork`. The registry's array and mutex are initialized at load time, before
|
|
219
|
+
forking, avoiding races during lazy initialization.
|
|
249
220
|
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
Starting the monitor is wrapped in a `rescue`: instrumentation has no right to
|
|
256
|
-
prevent a worker from starting. `Dir.mkdir` raises `Errno::EROFS` if the marks
|
|
257
|
-
directory was never mounted, and resolving the worker set runs application code.
|
|
258
|
-
The worst outcome should be a missing mark and a pod restart, never a pod that
|
|
259
|
-
stays up with silent queues.
|
|
221
|
+
The Rails Railtie registers installation through `to_prepare`, allowing access
|
|
222
|
+
to reloadable worker constants after initialization. Startup exceptions, from
|
|
223
|
+
either application code or an unwritable marks directory, are logged without
|
|
224
|
+
preventing the worker from starting. The probe remains responsible for detecting
|
|
225
|
+
a missing or stale heartbeat.
|
|
260
226
|
|
|
261
227
|
### The expected consumer count is not configured
|
|
262
228
|
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
That value is an array of classes under `sneakers:run` and a callable registry
|
|
268
|
-
under `sneakers:active_job`, so it is resolved with a `respond_to?(:call)`
|
|
269
|
-
check.
|
|
229
|
+
`config[:worker_classes]` is the set the worker gem itself consumes: an array
|
|
230
|
+
under `sneakers:run`, a callable registry under `sneakers:active_job`. Resolve it
|
|
231
|
+
with `respond_to?(:call)` rather than maintaining a separate queue list.
|
|
270
232
|
|
|
271
233
|
## What is configurable, and where
|
|
272
234
|
|
|
273
235
|
| Setting | Where | Why |
|
|
274
236
|
|---|---|---|
|
|
275
237
|
| `logger`, `enabled`, `startup_grace_ticks` | application config block | only the worker needs them |
|
|
276
|
-
| `tick` | either, and the config block wins | only the worker reads
|
|
238
|
+
| `tick`, `shutdown_grace` | either, and the config block wins | only the worker reads them, so two sources cannot contradict each other |
|
|
277
239
|
| `dir`, `max_age` | environment variables **only** | the probe is a separate process |
|
|
278
240
|
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
`max_age
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
non-positive value raises `ArgumentError`, as does a value greater than or equal
|
|
306
|
-
to `max_age`: such a monitor would inevitably let a healthy mark go stale. The
|
|
307
|
-
environment gets a silent fallback for values that cannot be parsed. If its
|
|
308
|
-
effective tick is greater than or equal to `max_age`, configuration emits a
|
|
309
|
-
warning on stderr and uses the default tick when it is safe, or half of `max_age`
|
|
310
|
-
otherwise. A ConfigMap typo must not bring a worker down, whereas an initializer
|
|
311
|
-
is code and should fail loudly at boot, where the developer is looking.
|
|
312
|
-
|
|
313
|
-
`startup_grace_ticks` must be a positive integer. It is compared directly with
|
|
314
|
-
an integer counter; accepting zero, a float, or a string would silently disable
|
|
315
|
-
the escalation that is supposed to diagnose a worker that never subscribes.
|
|
316
|
-
|
|
317
|
-
The logger defaults to `Sneakers.logger` but is resolved lazily, because at the
|
|
318
|
-
time the configuration object is built it may not be set up yet.
|
|
319
|
-
|
|
320
|
-
Logging is diagnostic; the heartbeat is the liveness contract. An exception
|
|
321
|
-
raised by a custom logger is therefore swallowed at the logging boundary. It
|
|
322
|
-
cannot prevent a healthy tick from writing its mark, abort monitor startup, or
|
|
323
|
-
escape the loop's error handler and kill the monitor thread. No fallback message
|
|
324
|
-
is attempted through the same broken logger.
|
|
241
|
+
The exec probe cannot read an application initializer, so `dir` and `max_age`
|
|
242
|
+
have one source: the environment. Specs reject application setters for both.
|
|
243
|
+
Only the worker reads `tick` and `shutdown_grace`, so initializer values may
|
|
244
|
+
override their environment values without contradicting the probe.
|
|
245
|
+
|
|
246
|
+
Environment durations must be positive integers; missing, empty, unparseable,
|
|
247
|
+
zero, and negative values use defaults. If the effective tick is at least
|
|
248
|
+
`max_age`, it warns on stderr and uses the default tick when safe, or half of
|
|
249
|
+
`max_age` otherwise. The warning is deferred until the tick is read, so an
|
|
250
|
+
initializer override or a disabled monitor does not produce a misleading WARN.
|
|
251
|
+
|
|
252
|
+
`KICKS_LIVENESS_SHUTDOWN_GRACE` defaults to 60 seconds. Its initializer override
|
|
253
|
+
accepts positive finite real numbers. Set it to at least the deployment's
|
|
254
|
+
`terminationGracePeriodSeconds`, which the gem cannot discover from a signal.
|
|
255
|
+
|
|
256
|
+
Invalid ConfigMap input falls back so a deployment typo does not prevent worker
|
|
257
|
+
startup. Initializer values are application code: invalid values raise
|
|
258
|
+
`ArgumentError` at boot, where the developer can fix the error, rather than
|
|
259
|
+
silently substituting a default. `tick` must be a positive number smaller than
|
|
260
|
+
`max_age`, and `startup_grace_ticks` a positive integer. The latter is compared
|
|
261
|
+
with an integer counter; accepting a float or zero would disable escalation.
|
|
262
|
+
|
|
263
|
+
The default logger is resolved lazily from `Sneakers.logger`, which may be unset
|
|
264
|
+
during configuration. Logger exceptions are swallowed at the logging boundary
|
|
265
|
+
so they cannot prevent heartbeat publication or stop the monitor. No fallback
|
|
266
|
+
uses the same broken logger.
|
|
325
267
|
|
|
326
268
|
## Logging: events, not the pulse
|
|
327
269
|
|
|
@@ -331,30 +273,33 @@ add noise. What gets logged is transitions.
|
|
|
331
273
|
| Event | Level |
|
|
332
274
|
|---|---|
|
|
333
275
|
| started, with the effective settings | INFO |
|
|
276
|
+
| the PID topology could not be verified, or yielded no generation | WARN |
|
|
334
277
|
| waiting for consumers | INFO |
|
|
335
278
|
| became healthy | INFO |
|
|
336
279
|
| shutting down | INFO |
|
|
337
280
|
| still not healthy after `startup_grace_ticks` ticks | ERROR |
|
|
338
281
|
| was healthy, became unhealthy | ERROR |
|
|
339
282
|
| a slot respawning without ever becoming healthy, once per grace window | ERROR |
|
|
283
|
+
| the PID topology is proven unsupported | ERROR |
|
|
340
284
|
| the monitor thread caught an exception | ERROR |
|
|
341
285
|
| the monitor could not be started at all | ERROR |
|
|
342
286
|
|
|
343
|
-
|
|
344
|
-
`
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
|
|
287
|
+
`topology-report` deduplicates only the three topology results: `unverifiable`,
|
|
288
|
+
`unavailable`, and `unsafe`. Their warning frequency, probe output, and recovery
|
|
289
|
+
are described in
|
|
290
|
+
[LIMITATIONS.md](LIMITATIONS.md#container-generations-need-linux-procfs-and-a-private-pid-namespace).
|
|
291
|
+
Loop and startup exceptions do not use that flag.
|
|
292
|
+
|
|
293
|
+
ERROR must survive the usual error-level logger filter, while normal startup
|
|
294
|
+
and deliberate shutdown remain INFO. On shutdown the monitor suspends its
|
|
295
|
+
predicate until a monotonic deadline: the first `Registry.stopping!` records the
|
|
296
|
+
stop time, and the monitor adds `shutdown_grace`. A delayed tick or repeated
|
|
297
|
+
callback cannot renew it. At expiry it logs one ERROR and stops `touch!` while
|
|
298
|
+
continuing `declare!`. Nothing is persisted, so a replacement fork does not
|
|
299
|
+
inherit a sibling's shutdown. Mark freshness still uses wall time; see the
|
|
300
|
+
[timing boundary](LIMITATIONS.md#graceful-shutdown-has-a-finite-heartbeat-exemption).
|
|
301
|
+
|
|
302
|
+
Startup grace postpones the error for a slow initial subscription; a surviving
|
|
303
|
+
monitor reports once when it expires. Fork-respawn loops use the persistent
|
|
304
|
+
counter and reporting windows described in
|
|
305
|
+
[LIMITATIONS.md](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn).
|