kicks_liveness 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.yardopts +13 -0
- data/CHANGELOG.md +41 -0
- data/README.md +127 -249
- data/docs/DESIGN.md +18 -4
- data/docs/KUBERNETES.md +11 -9
- data/docs/LIMITATIONS.md +12 -13
- data/docs/SETUP.md +10 -7
- data/docs/VERIFYING.md +13 -4
- data/lib/kicks_liveness/configuration.rb +73 -10
- data/lib/kicks_liveness/hooks.rb +12 -1
- data/lib/kicks_liveness/monitor.rb +6 -0
- data/lib/kicks_liveness/version.rb +1 -1
- data/lib/kicks_liveness.rb +13 -1
- metadata +3 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 3c784de1ca564182e9e36f8fb2956bb61a854c50995bdfe656c68805df671aed
|
|
4
|
+
data.tar.gz: e40f05ab3385e7b6dd680e040888120693751bda5ff5b2ff2625e107c4ff7cf9
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: e4e1088cc18b3a51493f330e2f37b3eba223dd6b21be77adcaf50833362a29b805d35f7d851ad31a2f84f2d7a9e5b69792cf443fd5cac2e8b2a44e53d70b9f29
|
|
7
|
+
data.tar.gz: 99345f3a010f4f6c72936e80a81fdff2a6ebf0ea371596bc90850b47cfe5abacaa93694b629fd9121de386e04a5272d44b46b3425f88229ae66c2e3fc0c555f7
|
data/.yardopts
ADDED
data/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,45 @@
|
|
|
3
3
|
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
4
4
|
and this project adheres to [Semantic Versioning](https://semver.org/).
|
|
5
5
|
|
|
6
|
+
## [Unreleased]
|
|
7
|
+
|
|
8
|
+
## [0.1.1] - 2026-09-09
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
|
|
12
|
+
- Add a repository-hosted social-preview asset for the project documentation.
|
|
13
|
+
|
|
14
|
+
### Changed
|
|
15
|
+
|
|
16
|
+
- Rework the README into a shorter entry point while retaining the operational
|
|
17
|
+
detail in the focused guides under `docs/`.
|
|
18
|
+
- Reject initializer values where `tick >= max_age`, and require
|
|
19
|
+
`startup_grace_ticks` to be a positive integer. Initializers that previously
|
|
20
|
+
supplied either invalid value now stop application boot with `ArgumentError`.
|
|
21
|
+
- Reject bundles that activate both `kicks` and `sneakers` with `LoadError`
|
|
22
|
+
instead of letting their shared `lib/sneakers.rb` resolve silently by
|
|
23
|
+
load-path order. Applications that still bundle both worker gems must remove
|
|
24
|
+
one before upgrading.
|
|
25
|
+
- Replace the contributor-only `AMQP_WORKER_GEM` and
|
|
26
|
+
`AMQP_WORKER_GEM_VERSION` test selection with Appraisal-generated dependency
|
|
27
|
+
sets, including current and supported-floor targets for both worker gems.
|
|
28
|
+
|
|
29
|
+
### Fixed
|
|
30
|
+
|
|
31
|
+
- Recover environment values where `tick >= max_age` with a warning and a safe
|
|
32
|
+
interval, preventing a healthy heartbeat from becoming stale between normal
|
|
33
|
+
monitor ticks.
|
|
34
|
+
- Keep worker startup, heartbeat publication, and the monitor loop alive when a
|
|
35
|
+
custom logger raises while handling an INFO or ERROR event.
|
|
36
|
+
- Correct the runtime-scaling documentation: live changes to `workers` are not
|
|
37
|
+
supported in either direction. During scale-up, old and new forks otherwise
|
|
38
|
+
race to publish different expected counts; during scale-down, retired slots
|
|
39
|
+
remain expected.
|
|
40
|
+
- Strengthen the release gate with floor-version coverage, style checks, tag
|
|
41
|
+
ancestry validation, and a packaged-executable smoke test.
|
|
42
|
+
- Include `.yardopts` in the gem so RubyDoc builds the intended README and
|
|
43
|
+
public guide set from the packaged release.
|
|
44
|
+
|
|
6
45
|
## [0.1.0] - 2026-09-08
|
|
7
46
|
|
|
8
47
|
### Added
|
|
@@ -18,4 +57,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
|
|
|
18
57
|
[docs/](https://github.com/PoroshkinaVV/kicks_liveness/tree/main/docs); start
|
|
19
58
|
with `SETUP.md`, and read `LIMITATIONS.md` before relying on it.
|
|
20
59
|
|
|
60
|
+
[Unreleased]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.1...HEAD
|
|
61
|
+
[0.1.1]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.0...v0.1.1
|
|
21
62
|
[0.1.0]: https://github.com/PoroshkinaVV/kicks_liveness/releases/tag/v0.1.0
|
data/README.md
CHANGED
|
@@ -1,119 +1,69 @@
|
|
|
1
1
|
# kicks_liveness
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
tmpfs, and the probe only reads it.
|
|
3
|
+
[](https://rubygems.org/gems/kicks_liveness)
|
|
4
|
+
[](https://github.com/PoroshkinaVV/kicks_liveness/actions/workflows/ci.yml)
|
|
6
5
|
|
|
7
|
-
|
|
6
|
+

|
|
7
|
+
|
|
8
|
+
**Know when a [Kicks](https://github.com/ruby-amqp/kicks) or
|
|
9
|
+
[Sneakers](https://github.com/jondot/sneakers) worker is stuck — without booting
|
|
10
|
+
Rails or querying RabbitMQ from the probe.**
|
|
8
11
|
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
is configurable and why `dir` and `max_age` live in the environment only.
|
|
15
|
-
- [docs/KUBERNETES.md](docs/KUBERNETES.md) — the manifest, the budget
|
|
16
|
-
arithmetic, verifying on a live pod, the alert on consumers.
|
|
17
|
-
- [docs/LIMITATIONS.md](docs/LIMITATIONS.md) — where the in-memory predicate
|
|
18
|
-
stops being truthful.
|
|
19
|
-
- [docs/VERIFYING.md](docs/VERIFYING.md) — the runbook that causes the real failures on a local cluster: what was observed, and how long each took.
|
|
20
|
-
|
|
21
|
-
## Why
|
|
22
|
-
|
|
23
|
-
The usual implementation is a rake task that boots Rails and asks RabbitMQ for
|
|
24
|
-
`consumer_count`. It has three defects that cannot be fixed in place:
|
|
25
|
-
|
|
26
|
-
1. **It is expensive.** Booting Rails every few seconds inside the worker's own
|
|
27
|
-
cgroup. On one real service this cost 2.7 s per invocation on a 15 s
|
|
28
|
-
period — 38% of the container's CPU request, around the clock.
|
|
29
|
-
2. **It depends on the disk.** A disk that stalls for a few seconds stretches
|
|
30
|
-
the boot past `timeoutSeconds`, and the kubelet kills a healthy pod.
|
|
31
|
-
3. **It cascades.** A liveness probe that checks an external dependency turns a
|
|
32
|
-
broker hiccup into every replica restarting at once, finishing the broker off.
|
|
33
|
-
|
|
34
|
-
On top of that, `consumer_count` is a metric of the **queue**, not of the pod:
|
|
35
|
-
with two replicas the live one covers for the stalled one, and the probe cannot
|
|
36
|
-
tell.
|
|
37
|
-
|
|
38
|
-
This gem inverts that. Every 10 seconds the worker checks, against the Bunny
|
|
39
|
-
objects **in its own memory**, that its consumers are in place, and touches a
|
|
40
|
-
file. The standard probe invocation took 156 ms in the fixture container, makes
|
|
41
|
-
no network call, boots no framework, and reads the state it judges from tmpfs,
|
|
42
|
-
which is RAM. It does still load Bundler, the interpreter and the gem's probe
|
|
43
|
-
files from the image filesystem, so the disk is not out of the picture
|
|
44
|
-
altogether — it is reduced to a small process startup instead of a full
|
|
45
|
-
application boot on every probe.
|
|
46
|
-
|
|
47
|
-
## Installation
|
|
12
|
+
The worker checks its own Bunny consumers and publishes a heartbeat to tmpfs.
|
|
13
|
+
The probe only reads that heartbeat. No application boot, no network call, and
|
|
14
|
+
no healthy replica hiding a stalled one.
|
|
15
|
+
|
|
16
|
+
## Install: one line in Rails
|
|
48
17
|
|
|
49
18
|
```ruby
|
|
19
|
+
# Gemfile — keep exactly one of kicks or sneakers alongside it
|
|
50
20
|
gem 'kicks_liveness'
|
|
51
21
|
```
|
|
52
22
|
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
`LoadError` naming both. Both floors are exercised in CI at their exact
|
|
57
|
-
versions, not just through a `~>` that resolves to the newest release.
|
|
58
|
-
|
|
59
|
-
Keeping both gems in one `Gemfile` is **not** allowed, and nothing enforces
|
|
60
|
-
that: not Bundler, not a crash at boot. Both own the file `lib/sneakers.rb`, so
|
|
61
|
-
one silently wins the load path. An application that has moved to `kicks` would
|
|
62
|
-
keep executing `sneakers` 2.12 code while its `Gemfile.lock` claims otherwise.
|
|
63
|
-
`sneakers` also caps the `kicks` and `bunny` versions.
|
|
64
|
-
|
|
65
|
-
The gem installs its hooks itself through a Railtie. Outside Rails, call
|
|
66
|
-
`KicksLiveness.install!` before the runner starts: the tie to Rails is a single
|
|
67
|
-
conditionally required file; it is not loaded outside Rails, and no Rails code is
|
|
68
|
-
pulled in. The details, including the one genuine restriction — workers must be
|
|
69
|
-
started through `Sneakers::Runner` — are in [docs/SETUP.md](docs/SETUP.md).
|
|
70
|
-
|
|
71
|
-
### Initializer
|
|
23
|
+
```bash
|
|
24
|
+
bundle install
|
|
25
|
+
```
|
|
72
26
|
|
|
73
|
-
|
|
27
|
+
In Rails, that is all the application-side setup: the Railtie installs the hooks
|
|
28
|
+
automatically. Without Rails, add two lines before `Sneakers::Runner` starts:
|
|
74
29
|
|
|
75
30
|
```ruby
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
end
|
|
31
|
+
require 'kicks_liveness'
|
|
32
|
+
KicksLiveness.install!
|
|
79
33
|
```
|
|
80
34
|
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
so
|
|
35
|
+
Requirements: Ruby >= 3.1 and exactly one of `kicks` >= 3.0 or `sneakers` >=
|
|
36
|
+
2.11. CI pins both floors and tracks the current release of both worker lines.
|
|
37
|
+
Do not put both worker gems in the same bundle: both provide
|
|
38
|
+
`lib/sneakers.rb`, so `install!` rejects that ambiguous process.
|
|
39
|
+
Framework-specific examples are in [Setup](docs/SETUP.md).
|
|
40
|
+
|
|
41
|
+
## Why replace the usual probe?
|
|
85
42
|
|
|
86
|
-
|
|
43
|
+
The common alternative is a rake task that boots the application and asks
|
|
44
|
+
RabbitMQ for `consumer_count`.
|
|
87
45
|
|
|
88
|
-
|
|
|
46
|
+
| | Rails + `consumer_count` probe | `kicks_liveness` |
|
|
89
47
|
|---|---|---|
|
|
90
|
-
|
|
|
91
|
-
|
|
|
92
|
-
|
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
Garbage in a value does not bring the worker down — it falls back to the default.
|
|
111
|
-
An empty string counts as unset: in a ConfigMap that is what you get by
|
|
112
|
-
declaring a key and leaving it blank. `max_age` and `tick` are durations, so
|
|
113
|
-
zero and negative values fall back too: they parse perfectly well, and a
|
|
114
|
-
negative `max_age` would make every mark stale on arrival.
|
|
115
|
-
|
|
116
|
-
## Manifest
|
|
48
|
+
| Code started on every probe | Rails and the application | Ruby, Bundler, and the probe files |
|
|
49
|
+
| Network call | RabbitMQ request | none |
|
|
50
|
+
| Health scope | a queue shared by every replica | this worker process |
|
|
51
|
+
| Decision path | application files plus RabbitMQ | process memory plus a tmpfs heartbeat |
|
|
52
|
+
| Measured invocation | **2.7 s of CPU** on a real service | **156 ms** in the fixture container |
|
|
53
|
+
| Continuous cost in the measured setups | **~0.18 core** at a 15 s period | **~0.005 core** at a 30 s period |
|
|
54
|
+
| During Bunny network recovery | may restart every replica together | stays healthy while Bunny reconnects |
|
|
55
|
+
|
|
56
|
+
Absolute timings depend on the image and hardware; the two columns are measured
|
|
57
|
+
examples, not a same-host microbenchmark. The calculation and faster 52 ms and
|
|
58
|
+
~10 ms probe forms are documented in [Running under Kubernetes](docs/KUBERNETES.md#why-the-command-looks-like-that).
|
|
59
|
+
|
|
60
|
+
Every 10 seconds by default, each worker verifies that all expected consumers
|
|
61
|
+
are subscribed and touches its own mark. `bundle exec kicks-liveness` exits 0
|
|
62
|
+
only when every expected worker mark is present and fresh.
|
|
63
|
+
|
|
64
|
+
## Kubernetes
|
|
65
|
+
|
|
66
|
+
Use a startup probe for the boot budget and a liveness probe for steady state:
|
|
117
67
|
|
|
118
68
|
```yaml
|
|
119
69
|
startupProbe:
|
|
@@ -131,7 +81,7 @@ negative `max_age` would make every mark stale on arrival.
|
|
|
131
81
|
terminationGracePeriodSeconds: 60
|
|
132
82
|
```
|
|
133
83
|
|
|
134
|
-
|
|
84
|
+
Put the heartbeat directory on tmpfs:
|
|
135
85
|
|
|
136
86
|
```yaml
|
|
137
87
|
volumeMounts:
|
|
@@ -140,180 +90,108 @@ The marks directory has to be on tmpfs:
|
|
|
140
90
|
volumes:
|
|
141
91
|
- name: app-tmp
|
|
142
92
|
emptyDir:
|
|
143
|
-
medium: Memory
|
|
93
|
+
medium: Memory
|
|
144
94
|
```
|
|
145
95
|
|
|
146
|
-
The
|
|
147
|
-
`
|
|
148
|
-
|
|
149
|
-
Without this mount, a container using `readOnlyRootFilesystem: true` cannot
|
|
150
|
-
publish heartbeat files and never passes the startup probe.
|
|
151
|
-
|
|
152
|
-
Why it looks like this:
|
|
153
|
-
|
|
154
|
-
- **`bundle exec kicks-liveness` is the standard command**, and it works
|
|
155
|
-
regardless of where Bundler installed the gems. There is a faster form —
|
|
156
|
-
`ruby -e "require 'kicks_liveness/probe'"`, 52 ms against
|
|
157
|
-
156 ms — but it needs the gems to be in `GEM_HOME`, and **setting
|
|
158
|
-
`BUNDLE_PATH` at all moves them**, even to the directory `GEM_HOME` already
|
|
159
|
-
points at. Check before you optimise:
|
|
160
|
-
|
|
161
|
-
```bash
|
|
162
|
-
docker run --rm your-image bundle exec kicks-liveness
|
|
163
|
-
docker run --rm your-image ruby -e "require 'kicks_liveness/probe'"
|
|
164
|
-
```
|
|
165
|
-
|
|
166
|
-
Both should print `no .../expected` and exit 1. If the second raises
|
|
167
|
-
`LoadError`, keep the standard command — at `periodSeconds: 30` it costs about
|
|
168
|
-
0.005 of a core, against roughly 0.18 for a probe that boots the application.
|
|
169
|
-
The trade-offs are in
|
|
170
|
-
[docs/KUBERNETES.md](docs/KUBERNETES.md#where-your-image-puts-its-gems).
|
|
171
|
-
|
|
172
|
-
- **`startupProbe` is mandatory.** Startup and steady state have different time
|
|
173
|
-
budgets, and one probe cannot serve both. While it runs, liveness is disabled
|
|
174
|
-
and the container is not Ready. `failureThreshold: 60` buys 300 s, which is
|
|
175
|
-
deliberately generous — measure a *freshly created* pod before trimming it, as
|
|
176
|
-
a restarted container inherits a warm compile cache and a new pod does not:
|
|
177
|
-
[measure the cold start](docs/KUBERNETES.md#measure-the-cold-start-not-the-restart).
|
|
178
|
-
- **It also protects rollouts — given `maxUnavailable: 0`.** Because the
|
|
179
|
-
container is not Ready until the mark exists, a `RollingUpdate` that is not
|
|
180
|
-
allowed to drop below full capacity cannot remove the old pod before the new
|
|
181
|
-
one has subscribed. That is a property of the strategy as much as of the probe:
|
|
182
|
-
with a non-zero `maxUnavailable`, or with `Recreate`, the old pod may go first
|
|
183
|
-
and leave the queue uncovered.
|
|
184
|
-
- **`initialDelaySeconds` on liveness is unnecessary**; the startupProbe plays
|
|
185
|
-
that role.
|
|
186
|
-
- Before a kill there is `max_age + periodSeconds × failureThreshold` = 45 + 90 =
|
|
187
|
-
**135 s** of confirmed silence. Slow on purpose: a worker is not
|
|
188
|
-
latency-critical, and a false restart costs more than two minutes of stalling.
|
|
189
|
-
|
|
190
|
-
## Required companion: an alert on consumers
|
|
191
|
-
|
|
192
|
-
The probe deliberately reports healthy while Bunny is reconnecting — otherwise a
|
|
193
|
-
broker hiccup would restart every replica at once. The price is that **broker
|
|
194
|
-
unavailability stops being visible automatically**. A pod can be consuming
|
|
195
|
-
nothing and look perfectly healthy.
|
|
196
|
-
|
|
197
|
-
So the probe needs an alert alongside it, and that alert should be in place
|
|
198
|
-
**before** the manifest is switched over:
|
|
96
|
+
The default heartbeat directory is `/opt/app/tmp/health`; the gem creates the
|
|
97
|
+
`health` subdirectory. The mount is also required with
|
|
98
|
+
`readOnlyRootFilesystem: true`.
|
|
199
99
|
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
# Not optional: the expression above cannot fire when the series is gone.
|
|
206
|
-
- alert: QueueConsumerMetricMissing
|
|
207
|
-
expr: absent_over_time(rabbitmq_detailed_queue_consumers{queue=~"myapp\..+"}[10m])
|
|
208
|
-
for: 5m
|
|
209
|
-
```
|
|
100
|
+
With the values above, confirmed silence lasts at most `45 + 30 × 3 = 135 s`
|
|
101
|
+
before Kubernetes kills the container. Size the startup budget from fresh-pod
|
|
102
|
+
measurements, not warm restarts. See [the full manifest and budget
|
|
103
|
+
arithmetic](docs/KUBERNETES.md).
|
|
210
104
|
|
|
211
|
-
|
|
212
|
-
without the exclusion the alert fires permanently until someone silences it.
|
|
213
|
-
Match them as a **substring**: under ActiveJob the full name looks like
|
|
214
|
-
`myapp.active_job.myapp.delayed_active_job:60`, so an anchored pattern never
|
|
215
|
-
matches.
|
|
105
|
+
## Configuration
|
|
216
106
|
|
|
217
|
-
|
|
218
|
-
recalled. The per-queue series is `rabbitmq_detailed_queue_consumers`, and it
|
|
219
|
-
only exists on `/metrics/detailed` **scraped with `family=queue_consumer_count`**
|
|
220
|
-
— bare, that endpoint serves no queue metrics at all, while plain `/metrics`
|
|
221
|
-
offers a same-sounding `rabbitmq_queue_consumers` that is a single label-less
|
|
222
|
-
cluster total, so a `{queue=~...}` matcher against it is quietly never true. And
|
|
223
|
-
the series **disappears** rather than going to zero when the queue is deleted or
|
|
224
|
-
the broker stops, which is why the second alert is there.
|
|
225
|
-
[docs/KUBERNETES.md](docs/KUBERNETES.md#getting-that-metric-at-all) has the
|
|
226
|
-
scrape config.
|
|
107
|
+
Everything is optional. Application settings belong in an initializer:
|
|
227
108
|
|
|
228
|
-
|
|
109
|
+
```ruby
|
|
110
|
+
KicksLiveness.configure do |config|
|
|
111
|
+
config.enabled = !Rails.env.local?
|
|
112
|
+
end
|
|
113
|
+
```
|
|
229
114
|
|
|
230
|
-
|
|
115
|
+
| Ruby setting | Default | Purpose | Valid values |
|
|
116
|
+
|---|---|---|---|
|
|
117
|
+
| `logger` | `Sneakers.logger`, resolved lazily | transition and error logs | logger-compatible object |
|
|
118
|
+
| `enabled` | `true` | start the monitor thread | `true` or `false` |
|
|
119
|
+
| `tick` | `10` | seconds between worker checks | positive number smaller than `max_age` |
|
|
120
|
+
| `startup_grace_ticks` | `6` | unhealthy startup ticks before one ERROR | positive integer |
|
|
231
121
|
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
| Bunny is recovering the connection | **healthy** — the broker heals itself |
|
|
236
|
-
| connection open, no consumers | unhealthy, a restart is the cure |
|
|
237
|
-
| not the whole set subscribed | unhealthy |
|
|
122
|
+
The separate probe cannot read an application initializer. The shared path and
|
|
123
|
+
freshness threshold therefore come from the environment; `tick` may use the
|
|
124
|
+
environment too:
|
|
238
125
|
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
126
|
+
| Environment variable | Default | Purpose | Invalid or blank value |
|
|
127
|
+
|---|---|---|---|
|
|
128
|
+
| `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | heartbeat directory; must be on tmpfs | uses the default |
|
|
129
|
+
| `KICKS_LIVENESS_MAX_AGE` | `45` | seconds before a mark is stale | uses the default |
|
|
130
|
+
| `KICKS_LIVENESS_TICK` | `10` | seconds between worker checks | uses the default, then the safety fallback below if needed |
|
|
242
131
|
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
132
|
+
If the effective environment tick is not smaller than `max_age`, the worker
|
|
133
|
+
writes a WARN to stderr and uses `10` seconds when that is safe, or half of
|
|
134
|
+
`max_age` otherwise. This keeps a ConfigMap mistake from stopping the worker
|
|
135
|
+
while guaranteeing that a healthy heartbeat cannot become stale between
|
|
136
|
+
ordinary ticks.
|
|
247
137
|
|
|
248
|
-
|
|
138
|
+
An initializer value for `tick` wins over `KICKS_LIVENESS_TICK`. `dir` and
|
|
139
|
+
`max_age` deliberately have no initializer setters: allowing two configuration
|
|
140
|
+
sources could make the worker and probe silently disagree.
|
|
249
141
|
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
142
|
+
If two runners share one pod — for example `sneakers:run` and
|
|
143
|
+
`sneakers:active_job` — give each a different `KICKS_LIVENESS_DIR`. Separate
|
|
144
|
+
pods already have separate `emptyDir` volumes.
|
|
253
145
|
|
|
254
|
-
|
|
255
|
-
broker cancels a consumer (on `consumer_timeout`, for example), it sends
|
|
256
|
-
`basic.cancel`, and Bunny removes the entry from its list — but it does so
|
|
257
|
-
through `@work_pool.submit`, that is, on the same pool that processes jobs
|
|
258
|
-
(`threads: 10` by default). Until a thread frees up, `any_consumers?` still
|
|
259
|
-
answers yes.
|
|
146
|
+
## What is healthy?
|
|
260
147
|
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
148
|
+
| Worker state | Probe verdict | Reason |
|
|
149
|
+
|---|---|---|
|
|
150
|
+
| every consumer subscribed, channel open | healthy | the process can consume work |
|
|
151
|
+
| Bunny recovering its connection | **healthy** | Bunny reconnects itself; restarting adds a reconnect storm |
|
|
152
|
+
| connection open, no consumers | unhealthy | a restart is the cure |
|
|
153
|
+
| only part of the expected worker set subscribed | unhealthy | one healthy worker must not hide a missing one |
|
|
154
|
+
|
|
155
|
+
The check reads Bunny objects already in the worker's memory. It never asks the
|
|
156
|
+
broker for its view.
|
|
264
157
|
|
|
265
|
-
|
|
266
|
-
consumer.** `Bunny::Channel#recover_cancelled_consumers!` is an opt-in method
|
|
267
|
-
that neither Kicks nor Sneakers enables. With it on, Bunny re-subscribes the
|
|
268
|
-
consumer itself on `basic.cancel` and **keeps** the entry in its list — so the
|
|
269
|
-
"consumer was cancelled" case stops being detectable at all. Do not enable it
|
|
270
|
-
together with this probe.
|
|
158
|
+
## Operational contract
|
|
271
159
|
|
|
272
|
-
|
|
273
|
-
|
|
160
|
+
- Start workers through `Sneakers::Runner`; that is where the monitor hook is
|
|
161
|
+
attached. `rake sneakers:run` already does this.
|
|
162
|
+
- Keep the `startupProbe`. With `RollingUpdate` and `maxUnavailable: 0`, it also
|
|
163
|
+
keeps the old pod until the new worker has subscribed.
|
|
164
|
+
- Add an external alert for queues with zero or missing consumers **before**
|
|
165
|
+
switching probes. Broker unavailability is invisible here by design; the
|
|
166
|
+
exact Prometheus rules are in [Running under Kubernetes](docs/KUBERNETES.md#required-companion-an-alert-on-consumers).
|
|
167
|
+
- Restart the runner after changing its process count; changing `workers` at
|
|
168
|
+
runtime is not supported.
|
|
274
169
|
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
catches exactly what an in-memory predicate cannot see.
|
|
170
|
+
The other boundaries — blocked worker pools, cancelled-consumer recovery, and
|
|
171
|
+
per-pod versus per-queue scope — are collected in [Limitations](docs/LIMITATIONS.md).
|
|
278
172
|
|
|
279
|
-
##
|
|
173
|
+
## Verify it
|
|
280
174
|
|
|
281
175
|
```bash
|
|
282
176
|
kubectl exec deploy/myapp -- ls -l /opt/app/tmp/health/
|
|
283
177
|
kubectl exec deploy/myapp -- sh -c 'bundle exec kicks-liveness; echo "exit=$?"'
|
|
284
|
-
kubectl logs deploy/myapp | grep liveness
|
|
285
178
|
```
|
|
286
179
|
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
```bash
|
|
291
|
-
kubectl exec deploy/myapp -- sh -c 'touch -d "2 minutes ago" /opt/app/tmp/health/worker-0; bundle exec kicks-liveness; echo "exit=$?"'
|
|
292
|
-
```
|
|
180
|
+
The healthy result is `N process(es) healthy` with exit code 0. Also test the
|
|
181
|
+
negative path before rollout; the commands and expected output are in
|
|
182
|
+
[Verifying the probe](docs/VERIFYING.md#2-a-stale-mark).
|
|
293
183
|
|
|
294
|
-
|
|
295
|
-
itself.
|
|
184
|
+
## Documentation
|
|
296
185
|
|
|
297
|
-
|
|
186
|
+
| Guide | Use it for |
|
|
187
|
+
|---|---|
|
|
188
|
+
| [Setup](docs/SETUP.md) | Rails, Sinatra, Hanami, Roda, standalone runners, and hook verification |
|
|
189
|
+
| [Design](docs/DESIGN.md) | health predicate, heartbeat files, hooks, configuration choices, and measurements |
|
|
190
|
+
| [Running under Kubernetes](docs/KUBERNETES.md) | full manifest, CPU/startup budgets, alerts, and live-pod verification |
|
|
191
|
+
| [Limitations](docs/LIMITATIONS.md) | cases the in-memory predicate intentionally cannot cover |
|
|
192
|
+
| [Verification scenarios](docs/VERIFYING.md) | ten deliberately induced failures and their observed outcomes |
|
|
298
193
|
|
|
299
|
-
|
|
194
|
+
## Contributing
|
|
300
195
|
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
| started, with the effective settings | INFO |
|
|
304
|
-
| waiting for consumers | INFO |
|
|
305
|
-
| became healthy | INFO |
|
|
306
|
-
| shutting down | INFO |
|
|
307
|
-
| still not healthy after `startup_grace_ticks` ticks | ERROR |
|
|
308
|
-
| was healthy, became unhealthy | ERROR |
|
|
309
|
-
| a slot respawning without ever becoming healthy, once per grace window | ERROR |
|
|
310
|
-
| the monitor thread caught an exception | ERROR |
|
|
311
|
-
| the monitor could not be started at all | ERROR |
|
|
312
|
-
|
|
313
|
-
ERROR is reserved for genuine failure: a logger usually comes up with
|
|
314
|
-
`LOG_LEVEL` defaulting to `error`, so anything that matters must survive that
|
|
315
|
-
filter, while an ordinary deploy must not make noise. That is also why shutdown
|
|
316
|
-
is an event of its own: the workers unsubscribe while the thread pool drains,
|
|
317
|
-
and judged by the consumer count alone a normal deploy would look exactly like
|
|
318
|
-
a fault. From the moment shutdown begins the probe keeps the mark fresh
|
|
319
|
-
instead, so a slow drain gets the full `terminationGracePeriodSeconds`.
|
|
196
|
+
Setup, running the suite against both worker gems, and the release procedure are
|
|
197
|
+
in [CONTRIBUTING.md](https://github.com/PoroshkinaVV/kicks_liveness/blob/main/CONTRIBUTING.md).
|
data/docs/DESIGN.md
CHANGED
|
@@ -249,7 +249,7 @@ check.
|
|
|
249
249
|
|
|
250
250
|
## What is configurable, and where
|
|
251
251
|
|
|
252
|
-
| | Where | Why |
|
|
252
|
+
| Setting | Where | Why |
|
|
253
253
|
|---|---|---|
|
|
254
254
|
| `logger`, `enabled`, `startup_grace_ticks` | application config block | only the worker needs them |
|
|
255
255
|
| `tick` | either, and the config block wins | only the worker reads it, so two sources cannot contradict each other |
|
|
@@ -281,13 +281,27 @@ makes every mark stale on arrival, so the probe can never pass again. They fall
|
|
|
281
281
|
back to the default too.
|
|
282
282
|
|
|
283
283
|
Setting `tick` from the application is held to a stricter standard: a
|
|
284
|
-
non-positive value raises `ArgumentError
|
|
285
|
-
|
|
286
|
-
|
|
284
|
+
non-positive value raises `ArgumentError`, as does a value greater than or equal
|
|
285
|
+
to `max_age`: such a monitor would inevitably let a healthy mark go stale. The
|
|
286
|
+
environment gets a silent fallback for values that cannot be parsed. If its
|
|
287
|
+
effective tick is greater than or equal to `max_age`, configuration emits a
|
|
288
|
+
warning on stderr and uses the default tick when it is safe, or half of `max_age`
|
|
289
|
+
otherwise. A ConfigMap typo must not bring a worker down, whereas an initializer
|
|
290
|
+
is code and should fail loudly at boot, where the developer is looking.
|
|
291
|
+
|
|
292
|
+
`startup_grace_ticks` must be a positive integer. It is compared directly with
|
|
293
|
+
an integer counter; accepting zero, a float, or a string would silently disable
|
|
294
|
+
the escalation that is supposed to diagnose a worker that never subscribes.
|
|
287
295
|
|
|
288
296
|
The logger defaults to `Sneakers.logger` but is resolved lazily, because at the
|
|
289
297
|
time the configuration object is built it may not be set up yet.
|
|
290
298
|
|
|
299
|
+
Logging is diagnostic; the heartbeat is the liveness contract. An exception
|
|
300
|
+
raised by a custom logger is therefore swallowed at the logging boundary. It
|
|
301
|
+
cannot prevent a healthy tick from writing its mark, abort monitor startup, or
|
|
302
|
+
escape the loop's error handler and kill the monitor thread. No fallback message
|
|
303
|
+
is attempted through the same broken logger.
|
|
304
|
+
|
|
291
305
|
## Logging: events, not the pulse
|
|
292
306
|
|
|
293
307
|
The pulse lives in the mtime of a file; writing a log line every tick would only
|
data/docs/KUBERNETES.md
CHANGED
|
@@ -286,15 +286,18 @@ redelivered.
|
|
|
286
286
|
|
|
287
287
|
## Environment variables
|
|
288
288
|
|
|
289
|
-
| Variable | Default | |
|
|
290
|
-
|
|
291
|
-
| `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | marks directory, **must be on tmpfs** |
|
|
292
|
-
| `KICKS_LIVENESS_MAX_AGE` | `45` | seconds after which a mark is stale |
|
|
293
|
-
| `KICKS_LIVENESS_TICK` | `10` | interval between ticks |
|
|
289
|
+
| Variable | Default | Purpose | Invalid or blank value |
|
|
290
|
+
|---|---|---|---|
|
|
291
|
+
| `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | marks directory, **must be on tmpfs** | uses the default |
|
|
292
|
+
| `KICKS_LIVENESS_MAX_AGE` | `45` | seconds after which a mark is stale | uses the default |
|
|
293
|
+
| `KICKS_LIVENESS_TICK` | `10` | interval between ticks | uses the default, then the safety fallback below if needed |
|
|
294
294
|
|
|
295
295
|
Keep `tick` well below `max_age`. A tick longer than half of `max_age` leaves no
|
|
296
296
|
room for a single missed write, and a tick longer than `max_age` guarantees a
|
|
297
|
-
restart loop.
|
|
297
|
+
restart loop. If environment values produce `tick >= max_age`, the worker writes
|
|
298
|
+
a WARN to stderr and uses the default tick when that is safe, or half of
|
|
299
|
+
`max_age` otherwise. An initializer value with the same mismatch raises
|
|
300
|
+
`ArgumentError`. The half-threshold remains the recommended operational margin.
|
|
298
301
|
|
|
299
302
|
`KICKS_LIVENESS_DIR` is also what separates two runners that share a pod. An
|
|
300
303
|
application running both `rake sneakers:run` and `rake sneakers:active_job` has
|
|
@@ -315,9 +318,8 @@ probe command pointed at its own directory — the probe reads
|
|
|
315
318
|
`KICKS_LIVENESS_DIR` from its own environment, which the kubelet takes from the
|
|
316
319
|
container it runs in.
|
|
317
320
|
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
declared and left blank.
|
|
321
|
+
An empty string counts as unset, which is what a ConfigMap gives you when a key
|
|
322
|
+
is declared and left blank.
|
|
321
323
|
|
|
322
324
|
## Verifying on a live pod
|
|
323
325
|
|
data/docs/LIMITATIONS.md
CHANGED
|
@@ -80,20 +80,17 @@ see the respawn escalation below.
|
|
|
80
80
|
|
|
81
81
|
## Changing the worker count at runtime is not supported
|
|
82
82
|
|
|
83
|
-
`ServerEngine` re-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
path does: the surviving monitors re-declare the current count on their next
|
|
87
|
-
tick and the probe follows.
|
|
83
|
+
`ServerEngine` can re-read its configuration on SIGHUP and scale the fork set,
|
|
84
|
+
but a fork holds its own copy of that configuration from the moment it was
|
|
85
|
+
created. Every monitor re-declares that captured count on every tick.
|
|
88
86
|
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
pod restarts.
|
|
87
|
+
After a scale-up, old forks keep declaring the old count while new forks declare
|
|
88
|
+
the new one. The shared `expected` file therefore depends on which fork wrote
|
|
89
|
+
last; it can temporarily require the new slots, or incorrectly report the old
|
|
90
|
+
set as complete. After a scale-down, the surviving forks keep declaring the old
|
|
91
|
+
count and the retired slots eventually go stale. Neither direction is safe.
|
|
95
92
|
|
|
96
|
-
If you
|
|
93
|
+
If you change `workers`, restart the runner. Runtime scaling is not supported.
|
|
97
94
|
|
|
98
95
|
## A respawn loop is reported once per grace window, not once per respawn
|
|
99
96
|
|
|
@@ -126,7 +123,9 @@ executing the `sneakers` code while its lockfile says otherwise, and pulling in
|
|
|
126
123
|
resolve.
|
|
127
124
|
|
|
128
125
|
A loud failure gets fixed; a silent substitution does not. Keep exactly one of
|
|
129
|
-
the two.
|
|
126
|
+
the two. Bundler itself still accepts the combination, but `install!` now
|
|
127
|
+
rejects a process in which both gems are activated before either set of hooks is
|
|
128
|
+
installed.
|
|
130
129
|
|
|
131
130
|
If neither is present, `install!` raises a `LoadError` naming both with their
|
|
132
131
|
required versions.
|
data/docs/SETUP.md
CHANGED
|
@@ -10,7 +10,7 @@ having both is worse than it looks.
|
|
|
10
10
|
|
|
11
11
|
Those two floors are exact, not aspirational: CI runs the suite against
|
|
12
12
|
`kicks 3.0.0` and `sneakers 2.11.0` pinned, alongside the matrix that tracks the
|
|
13
|
-
|
|
13
|
+
current release of each. A `~>` matrix on its own would only ever prove that the
|
|
14
14
|
latest version works.
|
|
15
15
|
|
|
16
16
|
```ruby
|
|
@@ -142,12 +142,15 @@ Set it only to point somewhere **other** than `Sneakers.logger`:
|
|
|
142
142
|
config.logger = Rails.logger
|
|
143
143
|
```
|
|
144
144
|
|
|
145
|
-
| Option | Default | |
|
|
146
|
-
|
|
147
|
-
| `logger` | `Sneakers.logger`, resolved lazily |
|
|
148
|
-
| `enabled` | `true` |
|
|
149
|
-
| `tick` | `10` | seconds between checks |
|
|
150
|
-
| `startup_grace_ticks` | `6` | unhealthy ticks
|
|
145
|
+
| Option | Default | Purpose | Valid values |
|
|
146
|
+
|---|---|---|---|
|
|
147
|
+
| `logger` | `Sneakers.logger`, resolved lazily | transition and error logs | logger-compatible object |
|
|
148
|
+
| `enabled` | `true` | start the monitor thread | `true` or `false` |
|
|
149
|
+
| `tick` | `10` | seconds between checks | positive number smaller than `max_age` |
|
|
150
|
+
| `startup_grace_ticks` | `6` | unhealthy startup ticks before one ERROR | positive integer |
|
|
151
|
+
|
|
152
|
+
Invalid combinations are rejected before the monitor starts instead of running
|
|
153
|
+
one that is guaranteed to publish stale marks or never report a stalled startup.
|
|
151
154
|
|
|
152
155
|
`dir` and `max_age` are **not** here — they come from environment variables
|
|
153
156
|
only. See
|
data/docs/VERIFYING.md
CHANGED
|
@@ -30,14 +30,23 @@ it takes.
|
|
|
30
30
|
|
|
31
31
|
```bash
|
|
32
32
|
spec/integration/verify.sh build # image, into the engine the cluster uses
|
|
33
|
+
spec/integration/verify.sh status # print and verify the exact target
|
|
33
34
|
spec/integration/verify.sh up # namespace, broker, worker
|
|
34
35
|
spec/integration/verify.sh down # deletes the namespace
|
|
35
36
|
```
|
|
36
37
|
|
|
37
|
-
Every `kubectl` call
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
38
|
+
Every `kubectl` call names its context and namespace explicitly, so the script
|
|
39
|
+
never falls back to `current-context`. It will still operate on whichever target
|
|
40
|
+
you configure: run `status` and inspect both values before `up` or `down`. The
|
|
41
|
+
script labels namespaces it creates and refuses to apply to or delete an
|
|
42
|
+
existing namespace without that ownership label.
|
|
43
|
+
|
|
44
|
+
The default image uses Kicks. Build it with Sneakers instead without editing the
|
|
45
|
+
fixture:
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
AMQP_WORKER_GEM=sneakers spec/integration/verify.sh build
|
|
49
|
+
```
|
|
41
50
|
|
|
42
51
|
Three things that cost time if you meet them the hard way:
|
|
43
52
|
|
|
@@ -17,20 +17,26 @@ module KicksLiveness
|
|
|
17
17
|
# @return [Integer] unhealthy ticks tolerated at startup before one ERROR
|
|
18
18
|
DEFAULT_STARTUP_GRACE_TICKS = 6
|
|
19
19
|
|
|
20
|
-
#
|
|
21
|
-
|
|
20
|
+
# Seconds between checks. An incompatibility warning for an environment
|
|
21
|
+
# fallback is delayed until this value is actually used, so an initializer
|
|
22
|
+
# can override it or disable the monitor without a misleading warning.
|
|
23
|
+
# @return [Numeric]
|
|
24
|
+
def tick
|
|
25
|
+
warn_incompatible_environment_tick_once
|
|
26
|
+
@tick
|
|
27
|
+
end
|
|
22
28
|
# @return [Integer] unhealthy ticks tolerated at startup before one ERROR
|
|
23
|
-
|
|
29
|
+
attr_reader :startup_grace_ticks
|
|
24
30
|
# @return [Logger, nil] explicit logger; defaults to +Sneakers.logger+
|
|
25
31
|
attr_accessor :logger
|
|
26
32
|
# @param value [Boolean] set false to start no monitor thread, e.g. in tests
|
|
27
33
|
attr_writer :enabled
|
|
28
34
|
|
|
29
35
|
def initialize
|
|
30
|
-
@tick = Heartbeat.env_int(:tick, DEFAULT_TICK)
|
|
31
36
|
@startup_grace_ticks = DEFAULT_STARTUP_GRACE_TICKS
|
|
32
37
|
@enabled = true
|
|
33
38
|
@logger = nil
|
|
39
|
+
@tick = environment_tick
|
|
34
40
|
end
|
|
35
41
|
|
|
36
42
|
# @return [Boolean]
|
|
@@ -39,20 +45,36 @@ module KicksLiveness
|
|
|
39
45
|
end
|
|
40
46
|
|
|
41
47
|
# A tick is what the monitor thread sleeps on, so a non-positive value is
|
|
42
|
-
# not a setting but a broken monitor.
|
|
43
|
-
#
|
|
44
|
-
#
|
|
45
|
-
# where the developer is looking.
|
|
48
|
+
# not a setting but a broken monitor. An initializer is code, and code
|
|
49
|
+
# should fail loudly at boot, where the developer is looking. Environment
|
|
50
|
+
# input is resolved separately by {#environment_tick}, with a safe fallback.
|
|
46
51
|
#
|
|
47
|
-
# @param seconds [
|
|
52
|
+
# @param seconds [Numeric]
|
|
48
53
|
# @raise [ArgumentError] if not a positive number
|
|
49
|
-
# @return [
|
|
54
|
+
# @return [Numeric]
|
|
50
55
|
def tick=(seconds)
|
|
51
56
|
raise ArgumentError, "tick must be a positive number, got #{seconds.inspect}" unless positive_number?(seconds)
|
|
57
|
+
raise ArgumentError, "tick must be less than max_age (#{max_age}s), got #{seconds.inspect}" if seconds >= max_age
|
|
52
58
|
|
|
59
|
+
@incompatible_environment_tick = nil
|
|
53
60
|
@tick = seconds
|
|
54
61
|
end
|
|
55
62
|
|
|
63
|
+
# The monitor compares an integer tick counter with this value. Accepting
|
|
64
|
+
# zero, a float, or a string would silently prevent the startup escalation
|
|
65
|
+
# from ever firing.
|
|
66
|
+
#
|
|
67
|
+
# @param ticks [Integer]
|
|
68
|
+
# @raise [ArgumentError] unless +ticks+ is a positive integer
|
|
69
|
+
# @return [Integer]
|
|
70
|
+
def startup_grace_ticks=(ticks)
|
|
71
|
+
unless ticks.is_a?(Integer) && ticks.positive?
|
|
72
|
+
raise ArgumentError, "startup_grace_ticks must be a positive integer, got #{ticks.inspect}"
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
@startup_grace_ticks = ticks
|
|
76
|
+
end
|
|
77
|
+
|
|
56
78
|
# Read-only, sourced from the environment so that it matches what the probe
|
|
57
79
|
# sees.
|
|
58
80
|
# @return [String] marks directory
|
|
@@ -76,6 +98,47 @@ module KicksLiveness
|
|
|
76
98
|
|
|
77
99
|
private
|
|
78
100
|
|
|
101
|
+
def environment_tick
|
|
102
|
+
raw = Heartbeat.env_raw(:tick)
|
|
103
|
+
seconds = Heartbeat.env_int(:tick, DEFAULT_TICK)
|
|
104
|
+
threshold = max_age
|
|
105
|
+
return seconds if seconds < threshold
|
|
106
|
+
|
|
107
|
+
fallback = [DEFAULT_TICK, threshold / 2.0].min
|
|
108
|
+
@incompatible_environment_tick = [seconds, threshold, fallback, raw]
|
|
109
|
+
fallback
|
|
110
|
+
end
|
|
111
|
+
|
|
112
|
+
def warn_incompatible_environment_tick_once
|
|
113
|
+
warning = @incompatible_environment_tick
|
|
114
|
+
return unless warning
|
|
115
|
+
|
|
116
|
+
@incompatible_environment_tick = nil
|
|
117
|
+
warn_incompatible_environment_tick(*warning)
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
def warn_incompatible_environment_tick(seconds, threshold, fallback, raw)
|
|
121
|
+
Kernel.warn(
|
|
122
|
+
"WARN [liveness] effective tick=#{seconds}s (#{environment_tick_source(raw)}) must be less than " \
|
|
123
|
+
"KICKS_LIVENESS_MAX_AGE=#{threshold}s; using #{fallback}s"
|
|
124
|
+
)
|
|
125
|
+
rescue StandardError
|
|
126
|
+
# Configuration recovery must not become a worker boot failure merely
|
|
127
|
+
# because stderr is unavailable or warning output has been overridden.
|
|
128
|
+
nil
|
|
129
|
+
end
|
|
130
|
+
|
|
131
|
+
def environment_tick_source(raw)
|
|
132
|
+
return 'default' if raw.nil?
|
|
133
|
+
|
|
134
|
+
value = Integer(raw)
|
|
135
|
+
return "KICKS_LIVENESS_TICK=#{raw}" if value.positive?
|
|
136
|
+
|
|
137
|
+
"default after invalid KICKS_LIVENESS_TICK=#{raw.inspect}"
|
|
138
|
+
rescue ArgumentError, TypeError
|
|
139
|
+
"default after invalid KICKS_LIVENESS_TICK=#{raw.inspect}"
|
|
140
|
+
end
|
|
141
|
+
|
|
79
142
|
def positive_number?(value)
|
|
80
143
|
value.is_a?(Numeric) && value.positive?
|
|
81
144
|
end
|
data/lib/kicks_liveness/hooks.rb
CHANGED
|
@@ -60,7 +60,7 @@ module KicksLiveness
|
|
|
60
60
|
consumers: kicks_liveness_expected_consumers
|
|
61
61
|
)
|
|
62
62
|
rescue StandardError => e
|
|
63
|
-
|
|
63
|
+
report_start_failure(e)
|
|
64
64
|
end
|
|
65
65
|
end
|
|
66
66
|
|
|
@@ -80,6 +80,17 @@ module KicksLiveness
|
|
|
80
80
|
|
|
81
81
|
private
|
|
82
82
|
|
|
83
|
+
def report_start_failure(error)
|
|
84
|
+
KicksLiveness.config.resolved_logger&.error(
|
|
85
|
+
"[liveness] failed to start: #{error.class}: #{error.message}"
|
|
86
|
+
)
|
|
87
|
+
rescue StandardError
|
|
88
|
+
# This is already the failure path. Neither resolving the configuration
|
|
89
|
+
# again nor a broken logger may let liveness instrumentation stop the
|
|
90
|
+
# worker process it is meant to observe.
|
|
91
|
+
nil
|
|
92
|
+
end
|
|
93
|
+
|
|
83
94
|
# The same set the worker gem itself builds its workers from, so the
|
|
84
95
|
# queue list is never duplicated and cannot drift. An array of classes
|
|
85
96
|
# under sneakers:run, a callable registry under sneakers:active_job.
|
|
@@ -173,6 +173,12 @@ module KicksLiveness
|
|
|
173
173
|
|
|
174
174
|
def log(level, message)
|
|
175
175
|
@config.resolved_logger&.public_send(level, "[liveness] slot #{@slot}: #{message}")
|
|
176
|
+
rescue StandardError
|
|
177
|
+
# Logging is diagnostic, while the heartbeat is the liveness contract. A
|
|
178
|
+
# broken custom logger must not prevent a mark from being written or kill
|
|
179
|
+
# the only thread that can refresh it. There is deliberately no fallback
|
|
180
|
+
# log here: calling the same logger again would only repeat the failure.
|
|
181
|
+
nil
|
|
176
182
|
end
|
|
177
183
|
end
|
|
178
184
|
end
|
data/lib/kicks_liveness.rb
CHANGED
|
@@ -46,10 +46,13 @@ module KicksLiveness
|
|
|
46
46
|
# <tt>gem 'kicks', require: false</tt> the to_prepare hook runs earlier and
|
|
47
47
|
# would fail on NameError.
|
|
48
48
|
#
|
|
49
|
-
# @raise [LoadError] if neither
|
|
49
|
+
# @raise [LoadError] if neither worker gem is available, or if both +kicks+
|
|
50
|
+
# and +sneakers+ are activated
|
|
50
51
|
# @return [Module]
|
|
51
52
|
# @see file:docs/SETUP.md#installing-the-hooks
|
|
52
53
|
def install!
|
|
54
|
+
reject_ambiguous_worker_gems!
|
|
55
|
+
|
|
53
56
|
begin
|
|
54
57
|
require 'sneakers'
|
|
55
58
|
require 'sneakers/workergroup'
|
|
@@ -80,6 +83,15 @@ module KicksLiveness
|
|
|
80
83
|
|
|
81
84
|
Monitor.new(slot: slot, processes: processes, consumers: consumers, config: config).start!
|
|
82
85
|
end
|
|
86
|
+
|
|
87
|
+
private
|
|
88
|
+
|
|
89
|
+
def reject_ambiguous_worker_gems!
|
|
90
|
+
return unless defined?(Gem.loaded_specs)
|
|
91
|
+
return unless Gem.loaded_specs.key?('kicks') && Gem.loaded_specs.key?('sneakers')
|
|
92
|
+
|
|
93
|
+
raise LoadError, 'kicks_liveness cannot run with both kicks and sneakers activated; install exactly one'
|
|
94
|
+
end
|
|
83
95
|
end
|
|
84
96
|
end
|
|
85
97
|
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: kicks_liveness
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.1
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- PoroshkinaVV
|
|
@@ -24,6 +24,7 @@ executables:
|
|
|
24
24
|
extensions: []
|
|
25
25
|
extra_rdoc_files: []
|
|
26
26
|
files:
|
|
27
|
+
- ".yardopts"
|
|
27
28
|
- CHANGELOG.md
|
|
28
29
|
- LICENSE.txt
|
|
29
30
|
- README.md
|
|
@@ -48,7 +49,7 @@ licenses:
|
|
|
48
49
|
- MIT
|
|
49
50
|
metadata:
|
|
50
51
|
source_code_uri: https://github.com/PoroshkinaVV/kicks_liveness
|
|
51
|
-
documentation_uri: https://rubydoc.info/gems/kicks_liveness/0.1.
|
|
52
|
+
documentation_uri: https://rubydoc.info/gems/kicks_liveness/0.1.1
|
|
52
53
|
changelog_uri: https://github.com/PoroshkinaVV/kicks_liveness/blob/main/CHANGELOG.md
|
|
53
54
|
bug_tracker_uri: https://github.com/PoroshkinaVV/kicks_liveness/issues
|
|
54
55
|
rubygems_mfa_required: 'true'
|