kicks_liveness 0.1.0 → 0.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 206f62be48c616553a219c499e1106beb68a49fd3c91ca30a1d7f3ee812c15f7
4
- data.tar.gz: b2edcb9d1a6dd351df5b7a25d27e358582def6b6d8cdc4f7ad70b055ad1c25a9
3
+ metadata.gz: 3c784de1ca564182e9e36f8fb2956bb61a854c50995bdfe656c68805df671aed
4
+ data.tar.gz: e40f05ab3385e7b6dd680e040888120693751bda5ff5b2ff2625e107c4ff7cf9
5
5
  SHA512:
6
- metadata.gz: 06caaf5bfe502f5a83eebe11d03a8122f91ad03672fe2b11f046143ed107d6b81d126344cc293afc2ee311c6fb2f1a1f42739e82ff37b3e737cad7d3b71d97fb
7
- data.tar.gz: 6cf539fef1a39318702015363035bc8f1538fa9cabaaa6543eca038a62059db98a21b9b82a2a53be62a0315e6872adc7b6891c586df52d1bff3700b2071eb700
6
+ metadata.gz: e4e1088cc18b3a51493f330e2f37b3eba223dd6b21be77adcaf50833362a29b805d35f7d851ad31a2f84f2d7a9e5b69792cf443fd5cac2e8b2a44e53d70b9f29
7
+ data.tar.gz: 99345f3a010f4f6c72936e80a81fdff2a6ebf0ea371596bc90850b47cfe5abacaa93694b629fd9121de386e04a5272d44b46b3425f88229ae66c2e3fc0c555f7
data/.yardopts ADDED
@@ -0,0 +1,13 @@
1
+ --markup markdown
2
+ --readme README.md
3
+ --title "kicks_liveness"
4
+ --no-private
5
+ lib/**/*.rb
6
+ -
7
+ docs/SETUP.md
8
+ docs/DESIGN.md
9
+ docs/KUBERNETES.md
10
+ docs/LIMITATIONS.md
11
+ docs/VERIFYING.md
12
+ CHANGELOG.md
13
+ LICENSE.txt
data/CHANGELOG.md CHANGED
@@ -3,6 +3,45 @@
3
3
  The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
4
4
  and this project adheres to [Semantic Versioning](https://semver.org/).
5
5
 
6
+ ## [Unreleased]
7
+
8
+ ## [0.1.1] - 2026-09-09
9
+
10
+ ### Added
11
+
12
+ - Add a repository-hosted social-preview asset for the project documentation.
13
+
14
+ ### Changed
15
+
16
+ - Rework the README into a shorter entry point while retaining the operational
17
+ detail in the focused guides under `docs/`.
18
+ - Reject initializer values where `tick >= max_age`, and require
19
+ `startup_grace_ticks` to be a positive integer. Initializers that previously
20
+ supplied either invalid value now stop application boot with `ArgumentError`.
21
+ - Reject bundles that activate both `kicks` and `sneakers` with `LoadError`
22
+ instead of letting their shared `lib/sneakers.rb` resolve silently by
23
+ load-path order. Applications that still bundle both worker gems must remove
24
+ one before upgrading.
25
+ - Replace the contributor-only `AMQP_WORKER_GEM` and
26
+ `AMQP_WORKER_GEM_VERSION` test selection with Appraisal-generated dependency
27
+ sets, including current and supported-floor targets for both worker gems.
28
+
29
+ ### Fixed
30
+
31
+ - Recover environment values where `tick >= max_age` with a warning and a safe
32
+ interval, preventing a healthy heartbeat from becoming stale between normal
33
+ monitor ticks.
34
+ - Keep worker startup, heartbeat publication, and the monitor loop alive when a
35
+ custom logger raises while handling an INFO or ERROR event.
36
+ - Correct the runtime-scaling documentation: live changes to `workers` are not
37
+ supported in either direction. During scale-up, old and new forks otherwise
38
+ race to publish different expected counts; during scale-down, retired slots
39
+ remain expected.
40
+ - Strengthen the release gate with floor-version coverage, style checks, tag
41
+ ancestry validation, and a packaged-executable smoke test.
42
+ - Include `.yardopts` in the gem so RubyDoc builds the intended README and
43
+ public guide set from the packaged release.
44
+
6
45
  ## [0.1.0] - 2026-09-08
7
46
 
8
47
  ### Added
@@ -18,4 +57,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
18
57
  [docs/](https://github.com/PoroshkinaVV/kicks_liveness/tree/main/docs); start
19
58
  with `SETUP.md`, and read `LIMITATIONS.md` before relying on it.
20
59
 
60
+ [Unreleased]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.1...HEAD
61
+ [0.1.1]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.0...v0.1.1
21
62
  [0.1.0]: https://github.com/PoroshkinaVV/kicks_liveness/releases/tag/v0.1.0
data/README.md CHANGED
@@ -1,119 +1,69 @@
1
1
  # kicks_liveness
2
2
 
3
- A liveness probe for [Kicks](https://github.com/ruby-amqp/kicks) workers (and
4
- for its predecessor Sneakers): the worker itself publishes a liveness mark to
5
- tmpfs, and the probe only reads it.
3
+ [![Gem Version](https://badge.fury.io/rb/kicks_liveness.svg)](https://rubygems.org/gems/kicks_liveness)
4
+ [![CI](https://github.com/PoroshkinaVV/kicks_liveness/actions/workflows/ci.yml/badge.svg)](https://github.com/PoroshkinaVV/kicks_liveness/actions/workflows/ci.yml)
6
5
 
7
- ## Documentation
6
+ ![kicks_liveness — Kicks and Sneakers worker health](https://raw.githubusercontent.com/PoroshkinaVV/kicks_liveness/main/assets/github-social-preview.png)
7
+
8
+ **Know when a [Kicks](https://github.com/ruby-amqp/kicks) or
9
+ [Sneakers](https://github.com/jondot/sneakers) worker is stuck — without booting
10
+ Rails or querying RabbitMQ from the probe.**
8
11
 
9
- - [docs/SETUP.md](docs/SETUP.md) installation and wiring: Rails, Sinatra,
10
- Hanami, Roda, and no framework at all; what `install!` does and when to call
11
- it.
12
- - [docs/DESIGN.md](docs/DESIGN.md) — why the gem is built this way: the health
13
- predicate, the mark files, slots instead of PIDs, where the hooks attach, what
14
- is configurable and why `dir` and `max_age` live in the environment only.
15
- - [docs/KUBERNETES.md](docs/KUBERNETES.md) — the manifest, the budget
16
- arithmetic, verifying on a live pod, the alert on consumers.
17
- - [docs/LIMITATIONS.md](docs/LIMITATIONS.md) — where the in-memory predicate
18
- stops being truthful.
19
- - [docs/VERIFYING.md](docs/VERIFYING.md) — the runbook that causes the real failures on a local cluster: what was observed, and how long each took.
20
-
21
- ## Why
22
-
23
- The usual implementation is a rake task that boots Rails and asks RabbitMQ for
24
- `consumer_count`. It has three defects that cannot be fixed in place:
25
-
26
- 1. **It is expensive.** Booting Rails every few seconds inside the worker's own
27
- cgroup. On one real service this cost 2.7 s per invocation on a 15 s
28
- period — 38% of the container's CPU request, around the clock.
29
- 2. **It depends on the disk.** A disk that stalls for a few seconds stretches
30
- the boot past `timeoutSeconds`, and the kubelet kills a healthy pod.
31
- 3. **It cascades.** A liveness probe that checks an external dependency turns a
32
- broker hiccup into every replica restarting at once, finishing the broker off.
33
-
34
- On top of that, `consumer_count` is a metric of the **queue**, not of the pod:
35
- with two replicas the live one covers for the stalled one, and the probe cannot
36
- tell.
37
-
38
- This gem inverts that. Every 10 seconds the worker checks, against the Bunny
39
- objects **in its own memory**, that its consumers are in place, and touches a
40
- file. The standard probe invocation took 156 ms in the fixture container, makes
41
- no network call, boots no framework, and reads the state it judges from tmpfs,
42
- which is RAM. It does still load Bundler, the interpreter and the gem's probe
43
- files from the image filesystem, so the disk is not out of the picture
44
- altogether — it is reduced to a small process startup instead of a full
45
- application boot on every probe.
46
-
47
- ## Installation
12
+ The worker checks its own Bunny consumers and publishes a heartbeat to tmpfs.
13
+ The probe only reads that heartbeat. No application boot, no network call, and
14
+ no healthy replica hiding a stalled one.
15
+
16
+ ## Install: one line in Rails
48
17
 
49
18
  ```ruby
19
+ # Gemfile — keep exactly one of kicks or sneakers alongside it
50
20
  gem 'kicks_liveness'
51
21
  ```
52
22
 
53
- You need `kicks` (>= 3.0) **or** `sneakers` (>= 2.11). Neither is declared as a
54
- dependency of this gem, on purpose: at runtime only the `Sneakers` namespace is
55
- required, and both gems provide it. If neither is present, `install!` raises a
56
- `LoadError` naming both. Both floors are exercised in CI at their exact
57
- versions, not just through a `~>` that resolves to the newest release.
58
-
59
- Keeping both gems in one `Gemfile` is **not** allowed, and nothing enforces
60
- that: not Bundler, not a crash at boot. Both own the file `lib/sneakers.rb`, so
61
- one silently wins the load path. An application that has moved to `kicks` would
62
- keep executing `sneakers` 2.12 code while its `Gemfile.lock` claims otherwise.
63
- `sneakers` also caps the `kicks` and `bunny` versions.
64
-
65
- The gem installs its hooks itself through a Railtie. Outside Rails, call
66
- `KicksLiveness.install!` before the runner starts: the tie to Rails is a single
67
- conditionally required file; it is not loaded outside Rails, and no Rails code is
68
- pulled in. The details, including the one genuine restriction — workers must be
69
- started through `Sneakers::Runner` — are in [docs/SETUP.md](docs/SETUP.md).
70
-
71
- ### Initializer
23
+ ```bash
24
+ bundle install
25
+ ```
72
26
 
73
- Optional. For example, a Rails application can disable the monitor locally:
27
+ In Rails, that is all the application-side setup: the Railtie installs the hooks
28
+ automatically. Without Rails, add two lines before `Sneakers::Runner` starts:
74
29
 
75
30
  ```ruby
76
- KicksLiveness.configure do |config|
77
- config.enabled = !Rails.env.local?
78
- end
31
+ require 'kicks_liveness'
32
+ KicksLiveness.install!
79
33
  ```
80
34
 
81
- The full list: `logger` (defaults lazily to `Sneakers.logger`), `enabled`,
82
- `tick` (10 s), `startup_grace_ticks` (6).
83
- A non-positive `tick` raises `ArgumentError`: the monitor thread sleeps on it,
84
- so a non-positive value is not a setting but a broken monitor.
35
+ Requirements: Ruby >= 3.1 and exactly one of `kicks` >= 3.0 or `sneakers` >=
36
+ 2.11. CI pins both floors and tracks the current release of both worker lines.
37
+ Do not put both worker gems in the same bundle: both provide
38
+ `lib/sneakers.rb`, so `install!` rejects that ambiguous process.
39
+ Framework-specific examples are in [Setup](docs/SETUP.md).
40
+
41
+ ## Why replace the usual probe?
85
42
 
86
- ### Environment variables
43
+ The common alternative is a rake task that boots the application and asks
44
+ RabbitMQ for `consumer_count`.
87
45
 
88
- | Variable | Default | |
46
+ | | Rails + `consumer_count` probe | `kicks_liveness` |
89
47
  |---|---|---|
90
- | `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | marks directory, **must live on tmpfs** |
91
- | `KICKS_LIVENESS_MAX_AGE` | `45` | seconds after which a mark is stale |
92
- | `KICKS_LIVENESS_TICK` | `10` | interval between ticks |
93
-
94
- `tick` is the one setting that appears in both places; the initializer wins over
95
- the variable, and only the worker reads it either way.
96
-
97
- **Two runners in one pod need two directories.** An application that runs both
98
- `rake sneakers:run` and `rake sneakers:active_job` has two supervisors, each
99
- numbering its forks from zero so with the default `KICKS_LIVENESS_DIR` they
100
- would overwrite each other's `expected` and `worker-0`, and the probe would
101
- answer for whichever wrote last. In separate pods, which is the usual
102
- arrangement, there is nothing to do: each gets its own `emptyDir`. In one pod,
103
- give each runner its own `KICKS_LIVENESS_DIR`.
104
-
105
- `dir` and `max_age` deliberately **cannot** be set in the initializer. The probe
106
- is launched by the kubelet as a separate process, which does not — and cannot —
107
- read the application's initializer. Were they settable in two places, a
108
- mismatch between the worker and the probe would pass unnoticed.
109
-
110
- Garbage in a value does not bring the worker down — it falls back to the default.
111
- An empty string counts as unset: in a ConfigMap that is what you get by
112
- declaring a key and leaving it blank. `max_age` and `tick` are durations, so
113
- zero and negative values fall back too: they parse perfectly well, and a
114
- negative `max_age` would make every mark stale on arrival.
115
-
116
- ## Manifest
48
+ | Code started on every probe | Rails and the application | Ruby, Bundler, and the probe files |
49
+ | Network call | RabbitMQ request | none |
50
+ | Health scope | a queue shared by every replica | this worker process |
51
+ | Decision path | application files plus RabbitMQ | process memory plus a tmpfs heartbeat |
52
+ | Measured invocation | **2.7 s of CPU** on a real service | **156 ms** in the fixture container |
53
+ | Continuous cost in the measured setups | **~0.18 core** at a 15 s period | **~0.005 core** at a 30 s period |
54
+ | During Bunny network recovery | may restart every replica together | stays healthy while Bunny reconnects |
55
+
56
+ Absolute timings depend on the image and hardware; the two columns are measured
57
+ examples, not a same-host microbenchmark. The calculation and faster 52 ms and
58
+ ~10 ms probe forms are documented in [Running under Kubernetes](docs/KUBERNETES.md#why-the-command-looks-like-that).
59
+
60
+ Every 10 seconds by default, each worker verifies that all expected consumers
61
+ are subscribed and touches its own mark. `bundle exec kicks-liveness` exits 0
62
+ only when every expected worker mark is present and fresh.
63
+
64
+ ## Kubernetes
65
+
66
+ Use a startup probe for the boot budget and a liveness probe for steady state:
117
67
 
118
68
  ```yaml
119
69
  startupProbe:
@@ -131,7 +81,7 @@ negative `max_age` would make every mark stale on arrival.
131
81
  terminationGracePeriodSeconds: 60
132
82
  ```
133
83
 
134
- The marks directory has to be on tmpfs:
84
+ Put the heartbeat directory on tmpfs:
135
85
 
136
86
  ```yaml
137
87
  volumeMounts:
@@ -140,180 +90,108 @@ The marks directory has to be on tmpfs:
140
90
  volumes:
141
91
  - name: app-tmp
142
92
  emptyDir:
143
- medium: Memory # without this the mark lands on disk
93
+ medium: Memory
144
94
  ```
145
95
 
146
- The container needs a writable volume mounted at, or above,
147
- `KICKS_LIVENESS_DIR`. The default mount at `/opt/app/tmp` covers
148
- `/opt/app/tmp/health`, and the gem creates the `health` subdirectory itself.
149
- Without this mount, a container using `readOnlyRootFilesystem: true` cannot
150
- publish heartbeat files and never passes the startup probe.
151
-
152
- Why it looks like this:
153
-
154
- - **`bundle exec kicks-liveness` is the standard command**, and it works
155
- regardless of where Bundler installed the gems. There is a faster form —
156
- `ruby -e "require 'kicks_liveness/probe'"`, 52 ms against
157
- 156 ms — but it needs the gems to be in `GEM_HOME`, and **setting
158
- `BUNDLE_PATH` at all moves them**, even to the directory `GEM_HOME` already
159
- points at. Check before you optimise:
160
-
161
- ```bash
162
- docker run --rm your-image bundle exec kicks-liveness
163
- docker run --rm your-image ruby -e "require 'kicks_liveness/probe'"
164
- ```
165
-
166
- Both should print `no .../expected` and exit 1. If the second raises
167
- `LoadError`, keep the standard command — at `periodSeconds: 30` it costs about
168
- 0.005 of a core, against roughly 0.18 for a probe that boots the application.
169
- The trade-offs are in
170
- [docs/KUBERNETES.md](docs/KUBERNETES.md#where-your-image-puts-its-gems).
171
-
172
- - **`startupProbe` is mandatory.** Startup and steady state have different time
173
- budgets, and one probe cannot serve both. While it runs, liveness is disabled
174
- and the container is not Ready. `failureThreshold: 60` buys 300 s, which is
175
- deliberately generous — measure a *freshly created* pod before trimming it, as
176
- a restarted container inherits a warm compile cache and a new pod does not:
177
- [measure the cold start](docs/KUBERNETES.md#measure-the-cold-start-not-the-restart).
178
- - **It also protects rollouts — given `maxUnavailable: 0`.** Because the
179
- container is not Ready until the mark exists, a `RollingUpdate` that is not
180
- allowed to drop below full capacity cannot remove the old pod before the new
181
- one has subscribed. That is a property of the strategy as much as of the probe:
182
- with a non-zero `maxUnavailable`, or with `Recreate`, the old pod may go first
183
- and leave the queue uncovered.
184
- - **`initialDelaySeconds` on liveness is unnecessary**; the startupProbe plays
185
- that role.
186
- - Before a kill there is `max_age + periodSeconds × failureThreshold` = 45 + 90 =
187
- **135 s** of confirmed silence. Slow on purpose: a worker is not
188
- latency-critical, and a false restart costs more than two minutes of stalling.
189
-
190
- ## Required companion: an alert on consumers
191
-
192
- The probe deliberately reports healthy while Bunny is reconnecting — otherwise a
193
- broker hiccup would restart every replica at once. The price is that **broker
194
- unavailability stops being visible automatically**. A pod can be consuming
195
- nothing and look perfectly healthy.
196
-
197
- So the probe needs an alert alongside it, and that alert should be in place
198
- **before** the manifest is switched over:
96
+ The default heartbeat directory is `/opt/app/tmp/health`; the gem creates the
97
+ `health` subdirectory. The mount is also required with
98
+ `readOnlyRootFilesystem: true`.
199
99
 
200
- ```yaml
201
- - alert: QueueWithoutConsumers
202
- expr: rabbitmq_detailed_queue_consumers{queue=~"myapp\..+", queue!~".*delayed_active_job.*"} == 0
203
- for: 5m
204
-
205
- # Not optional: the expression above cannot fire when the series is gone.
206
- - alert: QueueConsumerMetricMissing
207
- expr: absent_over_time(rabbitmq_detailed_queue_consumers{queue=~"myapp\..+"}[10m])
208
- for: 5m
209
- ```
100
+ With the values above, confirmed silence lasts at most `45 + 30 × 3 = 135 s`
101
+ before Kubernetes kills the container. Size the startup budget from fresh-pod
102
+ measurements, not warm restarts. See [the full manifest and budget
103
+ arithmetic](docs/KUBERNETES.md).
210
104
 
211
- Excluding the delayed queues is mandatory — they have no consumers by design, and
212
- without the exclusion the alert fires permanently until someone silences it.
213
- Match them as a **substring**: under ActiveJob the full name looks like
214
- `myapp.active_job.myapp.delayed_active_job:60`, so an anchored pattern never
215
- matches.
105
+ ## Configuration
216
106
 
217
- Two traps in getting that metric, both measured on RabbitMQ 3.13.7 rather than
218
- recalled. The per-queue series is `rabbitmq_detailed_queue_consumers`, and it
219
- only exists on `/metrics/detailed` **scraped with `family=queue_consumer_count`**
220
- — bare, that endpoint serves no queue metrics at all, while plain `/metrics`
221
- offers a same-sounding `rabbitmq_queue_consumers` that is a single label-less
222
- cluster total, so a `{queue=~...}` matcher against it is quietly never true. And
223
- the series **disappears** rather than going to zero when the queue is deleted or
224
- the broker stops, which is why the second alert is there.
225
- [docs/KUBERNETES.md](docs/KUBERNETES.md#getting-that-metric-at-all) has the
226
- scrape config.
107
+ Everything is optional. Application settings belong in an initializer:
227
108
 
228
- ## What is checked
109
+ ```ruby
110
+ KicksLiveness.configure do |config|
111
+ config.enabled = !Rails.env.local?
112
+ end
113
+ ```
229
114
 
230
- The mark is written only if **every** worker in the process has subscribed:
115
+ | Ruby setting | Default | Purpose | Valid values |
116
+ |---|---|---|---|
117
+ | `logger` | `Sneakers.logger`, resolved lazily | transition and error logs | logger-compatible object |
118
+ | `enabled` | `true` | start the monitor thread | `true` or `false` |
119
+ | `tick` | `10` | seconds between worker checks | positive number smaller than `max_age` |
120
+ | `startup_grace_ticks` | `6` | unhealthy startup ticks before one ERROR | positive integer |
231
121
 
232
- | State | Verdict |
233
- |---|---|
234
- | consumers in place, channel open | healthy |
235
- | Bunny is recovering the connection | **healthy** — the broker heals itself |
236
- | connection open, no consumers | unhealthy, a restart is the cure |
237
- | not the whole set subscribed | unhealthy |
122
+ The separate probe cannot read an application initializer. The shared path and
123
+ freshness threshold therefore come from the environment; `tick` may use the
124
+ environment too:
238
125
 
239
- The expected consumer count comes from `config[:worker_classes]` the same set
240
- kicks itself builds its workers from. That way the queue list is never
241
- duplicated and cannot drift from the configuration.
126
+ | Environment variable | Default | Purpose | Invalid or blank value |
127
+ |---|---|---|---|
128
+ | `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | heartbeat directory; must be on tmpfs | uses the default |
129
+ | `KICKS_LIVENESS_MAX_AGE` | `45` | seconds before a mark is stale | uses the default |
130
+ | `KICKS_LIVENESS_TICK` | `10` | seconds between worker checks | uses the default, then the safety fallback below if needed |
242
131
 
243
- The files are named by supervisor slot rather than by PID: a dead fork is
244
- respawned into the same slot and overwrites its own file. With a PID in the
245
- name, SIGKILL would leave a stale file forever. With `workers > 1` the probe
246
- requires every slot to be fresh.
132
+ If the effective environment tick is not smaller than `max_age`, the worker
133
+ writes a WARN to stderr and uses `10` seconds when that is safe, or half of
134
+ `max_age` otherwise. This keeps a ConfigMap mistake from stopping the worker
135
+ while guaranteeing that a healthy heartbeat cannot become stale between
136
+ ordinary ticks.
247
137
 
248
- ## Limitations
138
+ An initializer value for `tick` wins over `KICKS_LIVENESS_TICK`. `dir` and
139
+ `max_age` deliberately have no initializer setters: allowing two configuration
140
+ sources could make the worker and probe silently disagree.
249
141
 
250
- The predicate is computed over the Bunny objects **in the process's memory** —
251
- that is the whole point, but it has a boundary: it is exactly as truthful as
252
- Bunny's own bookkeeping. The broker is never asked for its opinion.
142
+ If two runners share one pod for example `sneakers:run` and
143
+ `sneakers:active_job` give each a different `KICKS_LIVENESS_DIR`. Separate
144
+ pods already have separate `emptyDir` volumes.
253
145
 
254
- **1. Consumer bookkeeping is updated on the worker's thread pool.** When the
255
- broker cancels a consumer (on `consumer_timeout`, for example), it sends
256
- `basic.cancel`, and Bunny removes the entry from its list — but it does so
257
- through `@work_pool.submit`, that is, on the same pool that processes jobs
258
- (`threads: 10` by default). Until a thread frees up, `any_consumers?` still
259
- answers yes.
146
+ ## What is healthy?
260
147
 
261
- In practice this means: if **all** of the worker's threads are busy with stuck
262
- jobs, the probe stays green even though the pod is consuming nothing. One stuck
263
- job out of ten does not produce that effect.
148
+ | Worker state | Probe verdict | Reason |
149
+ |---|---|---|
150
+ | every consumer subscribed, channel open | healthy | the process can consume work |
151
+ | Bunny recovering its connection | **healthy** | Bunny reconnects itself; restarting adds a reconnect storm |
152
+ | connection open, no consumers | unhealthy | a restart is the cure |
153
+ | only part of the expected worker set subscribed | unhealthy | one healthy worker must not hide a missing one |
154
+
155
+ The check reads Bunny objects already in the worker's memory. It never asks the
156
+ broker for its view.
264
157
 
265
- **2. `recover_cancelled_consumers!` makes the probe blind to a cancelled
266
- consumer.** `Bunny::Channel#recover_cancelled_consumers!` is an opt-in method
267
- that neither Kicks nor Sneakers enables. With it on, Bunny re-subscribes the
268
- consumer itself on `basic.cancel` and **keeps** the entry in its list — so the
269
- "consumer was cancelled" case stops being detectable at all. Do not enable it
270
- together with this probe.
158
+ ## Operational contract
271
159
 
272
- **3. Broker unavailability is invisible by design** see [the alert on
273
- consumers](#required-companion-an-alert-on-consumers) above.
160
+ - Start workers through `Sneakers::Runner`; that is where the monitor hook is
161
+ attached. `rake sneakers:run` already does this.
162
+ - Keep the `startupProbe`. With `RollingUpdate` and `maxUnavailable: 0`, it also
163
+ keeps the old pod until the new worker has subscribed.
164
+ - Add an external alert for queues with zero or missing consumers **before**
165
+ switching probes. Broker unavailability is invisible here by design; the
166
+ exact Prometheus rules are in [Running under Kubernetes](docs/KUBERNETES.md#required-companion-an-alert-on-consumers).
167
+ - Restart the runner after changing its process count; changing `workers` at
168
+ runtime is not supported.
274
169
 
275
- All three limitations point the same way: an alert on consumers, watching the
276
- broker **from the outside**, is not a nice-to-have but part of the design. It
277
- catches exactly what an in-memory predicate cannot see.
170
+ The other boundaries blocked worker pools, cancelled-consumer recovery, and
171
+ per-pod versus per-queue scope are collected in [Limitations](docs/LIMITATIONS.md).
278
172
 
279
- ## Verifying on a live pod
173
+ ## Verify it
280
174
 
281
175
  ```bash
282
176
  kubectl exec deploy/myapp -- ls -l /opt/app/tmp/health/
283
177
  kubectl exec deploy/myapp -- sh -c 'bundle exec kicks-liveness; echo "exit=$?"'
284
- kubectl logs deploy/myapp | grep liveness
285
178
  ```
286
179
 
287
- Check the **negative** path too otherwise a working probe is
288
- indistinguishable from an always-green one:
289
-
290
- ```bash
291
- kubectl exec deploy/myapp -- sh -c 'touch -d "2 minutes ago" /opt/app/tmp/health/worker-0; bundle exec kicks-liveness; echo "exit=$?"'
292
- ```
180
+ The healthy result is `N process(es) healthy` with exit code 0. Also test the
181
+ negative path before rollout; the commands and expected output are in
182
+ [Verifying the probe](docs/VERIFYING.md#2-a-stale-mark).
293
183
 
294
- Expect `worker-0 stale 120s > 45s` and `exit=1`. One tick later the mark repairs
295
- itself.
184
+ ## Documentation
296
185
 
297
- ## Logs
186
+ | Guide | Use it for |
187
+ |---|---|
188
+ | [Setup](docs/SETUP.md) | Rails, Sinatra, Hanami, Roda, standalone runners, and hook verification |
189
+ | [Design](docs/DESIGN.md) | health predicate, heartbeat files, hooks, configuration choices, and measurements |
190
+ | [Running under Kubernetes](docs/KUBERNETES.md) | full manifest, CPU/startup budgets, alerts, and live-pod verification |
191
+ | [Limitations](docs/LIMITATIONS.md) | cases the in-memory predicate intentionally cannot cover |
192
+ | [Verification scenarios](docs/VERIFYING.md) | ten deliberately induced failures and their observed outcomes |
298
193
 
299
- Events are logged, not the pulse: the pulse lives in the mark's mtime.
194
+ ## Contributing
300
195
 
301
- | Event | Level |
302
- |---|---|
303
- | started, with the effective settings | INFO |
304
- | waiting for consumers | INFO |
305
- | became healthy | INFO |
306
- | shutting down | INFO |
307
- | still not healthy after `startup_grace_ticks` ticks | ERROR |
308
- | was healthy, became unhealthy | ERROR |
309
- | a slot respawning without ever becoming healthy, once per grace window | ERROR |
310
- | the monitor thread caught an exception | ERROR |
311
- | the monitor could not be started at all | ERROR |
312
-
313
- ERROR is reserved for genuine failure: a logger usually comes up with
314
- `LOG_LEVEL` defaulting to `error`, so anything that matters must survive that
315
- filter, while an ordinary deploy must not make noise. That is also why shutdown
316
- is an event of its own: the workers unsubscribe while the thread pool drains,
317
- and judged by the consumer count alone a normal deploy would look exactly like
318
- a fault. From the moment shutdown begins the probe keeps the mark fresh
319
- instead, so a slow drain gets the full `terminationGracePeriodSeconds`.
196
+ Setup, running the suite against both worker gems, and the release procedure are
197
+ in [CONTRIBUTING.md](https://github.com/PoroshkinaVV/kicks_liveness/blob/main/CONTRIBUTING.md).
data/docs/DESIGN.md CHANGED
@@ -249,7 +249,7 @@ check.
249
249
 
250
250
  ## What is configurable, and where
251
251
 
252
- | | Where | Why |
252
+ | Setting | Where | Why |
253
253
  |---|---|---|
254
254
  | `logger`, `enabled`, `startup_grace_ticks` | application config block | only the worker needs them |
255
255
  | `tick` | either, and the config block wins | only the worker reads it, so two sources cannot contradict each other |
@@ -281,13 +281,27 @@ makes every mark stale on arrival, so the probe can never pass again. They fall
281
281
  back to the default too.
282
282
 
283
283
  Setting `tick` from the application is held to a stricter standard: a
284
- non-positive value raises `ArgumentError`. The environment gets a silent fallback
285
- because a ConfigMap typo must not bring a worker down, whereas an initializer is
286
- code, and code should fail loudly at boot, where the developer is looking.
284
+ non-positive value raises `ArgumentError`, as does a value greater than or equal
285
+ to `max_age`: such a monitor would inevitably let a healthy mark go stale. The
286
+ environment gets a silent fallback for values that cannot be parsed. If its
287
+ effective tick is greater than or equal to `max_age`, configuration emits a
288
+ warning on stderr and uses the default tick when it is safe, or half of `max_age`
289
+ otherwise. A ConfigMap typo must not bring a worker down, whereas an initializer
290
+ is code and should fail loudly at boot, where the developer is looking.
291
+
292
+ `startup_grace_ticks` must be a positive integer. It is compared directly with
293
+ an integer counter; accepting zero, a float, or a string would silently disable
294
+ the escalation that is supposed to diagnose a worker that never subscribes.
287
295
 
288
296
  The logger defaults to `Sneakers.logger` but is resolved lazily, because at the
289
297
  time the configuration object is built it may not be set up yet.
290
298
 
299
+ Logging is diagnostic; the heartbeat is the liveness contract. An exception
300
+ raised by a custom logger is therefore swallowed at the logging boundary. It
301
+ cannot prevent a healthy tick from writing its mark, abort monitor startup, or
302
+ escape the loop's error handler and kill the monitor thread. No fallback message
303
+ is attempted through the same broken logger.
304
+
291
305
  ## Logging: events, not the pulse
292
306
 
293
307
  The pulse lives in the mtime of a file; writing a log line every tick would only
data/docs/KUBERNETES.md CHANGED
@@ -286,15 +286,18 @@ redelivered.
286
286
 
287
287
  ## Environment variables
288
288
 
289
- | Variable | Default | |
290
- |---|---|---|
291
- | `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | marks directory, **must be on tmpfs** |
292
- | `KICKS_LIVENESS_MAX_AGE` | `45` | seconds after which a mark is stale |
293
- | `KICKS_LIVENESS_TICK` | `10` | interval between ticks |
289
+ | Variable | Default | Purpose | Invalid or blank value |
290
+ |---|---|---|---|
291
+ | `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | marks directory, **must be on tmpfs** | uses the default |
292
+ | `KICKS_LIVENESS_MAX_AGE` | `45` | seconds after which a mark is stale | uses the default |
293
+ | `KICKS_LIVENESS_TICK` | `10` | interval between ticks | uses the default, then the safety fallback below if needed |
294
294
 
295
295
  Keep `tick` well below `max_age`. A tick longer than half of `max_age` leaves no
296
296
  room for a single missed write, and a tick longer than `max_age` guarantees a
297
- restart loop.
297
+ restart loop. If environment values produce `tick >= max_age`, the worker writes
298
+ a WARN to stderr and uses the default tick when that is safe, or half of
299
+ `max_age` otherwise. An initializer value with the same mismatch raises
300
+ `ArgumentError`. The half-threshold remains the recommended operational margin.
298
301
 
299
302
  `KICKS_LIVENESS_DIR` is also what separates two runners that share a pod. An
300
303
  application running both `rake sneakers:run` and `rake sneakers:active_job` has
@@ -315,9 +318,8 @@ probe command pointed at its own directory — the probe reads
315
318
  `KICKS_LIVENESS_DIR` from its own environment, which the kubelet takes from the
316
319
  container it runs in.
317
320
 
318
- Garbage in a value does not crash the worker it falls back to the default. An
319
- empty string counts as unset, which is what a ConfigMap gives you when a key is
320
- declared and left blank.
321
+ An empty string counts as unset, which is what a ConfigMap gives you when a key
322
+ is declared and left blank.
321
323
 
322
324
  ## Verifying on a live pod
323
325
 
data/docs/LIMITATIONS.md CHANGED
@@ -80,20 +80,17 @@ see the respawn escalation below.
80
80
 
81
81
  ## Changing the worker count at runtime is not supported
82
82
 
83
- `ServerEngine` re-reads its configuration on SIGHUP, and a changed `workers`
84
- value scales the fork set. Scaling **up** is fine. Scaling **down** is only
85
- handled when the supervisor also restarts the forks, which is what the restart
86
- path does: the surviving monitors re-declare the current count on their next
87
- tick and the probe follows.
83
+ `ServerEngine` can re-read its configuration on SIGHUP and scale the fork set,
84
+ but a fork holds its own copy of that configuration from the moment it was
85
+ created. Every monitor re-declares that captured count on every tick.
88
86
 
89
- What cannot be handled is a reload that lowers `workers` while leaving existing
90
- forks running. `reload_config` executes in the supervisor, and a fork holds its
91
- own copy of the configuration from the moment it was forked so a fork cannot
92
- see the new number however often it looks. The declared count then stays too
93
- high, the marks for the retired slots go stale, and the probe fails until the
94
- pod restarts.
87
+ After a scale-up, old forks keep declaring the old count while new forks declare
88
+ the new one. The shared `expected` file therefore depends on which fork wrote
89
+ last; it can temporarily require the new slots, or incorrectly report the old
90
+ set as complete. After a scale-down, the surviving forks keep declaring the old
91
+ count and the retired slots eventually go stale. Neither direction is safe.
95
92
 
96
- If you scale workers, restart them.
93
+ If you change `workers`, restart the runner. Runtime scaling is not supported.
97
94
 
98
95
  ## A respawn loop is reported once per grace window, not once per respawn
99
96
 
@@ -126,7 +123,9 @@ executing the `sneakers` code while its lockfile says otherwise, and pulling in
126
123
  resolve.
127
124
 
128
125
  A loud failure gets fixed; a silent substitution does not. Keep exactly one of
129
- the two.
126
+ the two. Bundler itself still accepts the combination, but `install!` now
127
+ rejects a process in which both gems are activated before either set of hooks is
128
+ installed.
130
129
 
131
130
  If neither is present, `install!` raises a `LoadError` naming both with their
132
131
  required versions.
data/docs/SETUP.md CHANGED
@@ -10,7 +10,7 @@ having both is worse than it looks.
10
10
 
11
11
  Those two floors are exact, not aspirational: CI runs the suite against
12
12
  `kicks 3.0.0` and `sneakers 2.11.0` pinned, alongside the matrix that tracks the
13
- newest release of each. A `~>` matrix on its own would only ever prove that the
13
+ current release of each. A `~>` matrix on its own would only ever prove that the
14
14
  latest version works.
15
15
 
16
16
  ```ruby
@@ -142,12 +142,15 @@ Set it only to point somewhere **other** than `Sneakers.logger`:
142
142
  config.logger = Rails.logger
143
143
  ```
144
144
 
145
- | Option | Default | |
146
- |---|---|---|
147
- | `logger` | `Sneakers.logger`, resolved lazily | where transitions are logged |
148
- | `enabled` | `true` | set to `false` in tests, so no thread is started |
149
- | `tick` | `10` | seconds between checks |
150
- | `startup_grace_ticks` | `6` | unhealthy ticks tolerated at startup before one ERROR |
145
+ | Option | Default | Purpose | Valid values |
146
+ |---|---|---|---|
147
+ | `logger` | `Sneakers.logger`, resolved lazily | transition and error logs | logger-compatible object |
148
+ | `enabled` | `true` | start the monitor thread | `true` or `false` |
149
+ | `tick` | `10` | seconds between checks | positive number smaller than `max_age` |
150
+ | `startup_grace_ticks` | `6` | unhealthy startup ticks before one ERROR | positive integer |
151
+
152
+ Invalid combinations are rejected before the monitor starts instead of running
153
+ one that is guaranteed to publish stale marks or never report a stalled startup.
151
154
 
152
155
  `dir` and `max_age` are **not** here — they come from environment variables
153
156
  only. See
data/docs/VERIFYING.md CHANGED
@@ -30,14 +30,23 @@ it takes.
30
30
 
31
31
  ```bash
32
32
  spec/integration/verify.sh build # image, into the engine the cluster uses
33
+ spec/integration/verify.sh status # print and verify the exact target
33
34
  spec/integration/verify.sh up # namespace, broker, worker
34
35
  spec/integration/verify.sh down # deletes the namespace
35
36
  ```
36
37
 
37
- Every `kubectl` call in that script names its context and namespace explicitly.
38
- That is not politeness: a liveness experiment deletes queues and kills pods, and
39
- it must not be able to reach a real cluster because someone's current-context
40
- happened to point at one.
38
+ Every `kubectl` call names its context and namespace explicitly, so the script
39
+ never falls back to `current-context`. It will still operate on whichever target
40
+ you configure: run `status` and inspect both values before `up` or `down`. The
41
+ script labels namespaces it creates and refuses to apply to or delete an
42
+ existing namespace without that ownership label.
43
+
44
+ The default image uses Kicks. Build it with Sneakers instead without editing the
45
+ fixture:
46
+
47
+ ```bash
48
+ AMQP_WORKER_GEM=sneakers spec/integration/verify.sh build
49
+ ```
41
50
 
42
51
  Three things that cost time if you meet them the hard way:
43
52
 
@@ -17,20 +17,26 @@ module KicksLiveness
17
17
  # @return [Integer] unhealthy ticks tolerated at startup before one ERROR
18
18
  DEFAULT_STARTUP_GRACE_TICKS = 6
19
19
 
20
- # @return [Integer] seconds between checks
21
- attr_reader :tick
20
+ # Seconds between checks. An incompatibility warning for an environment
21
+ # fallback is delayed until this value is actually used, so an initializer
22
+ # can override it or disable the monitor without a misleading warning.
23
+ # @return [Numeric]
24
+ def tick
25
+ warn_incompatible_environment_tick_once
26
+ @tick
27
+ end
22
28
  # @return [Integer] unhealthy ticks tolerated at startup before one ERROR
23
- attr_accessor :startup_grace_ticks
29
+ attr_reader :startup_grace_ticks
24
30
  # @return [Logger, nil] explicit logger; defaults to +Sneakers.logger+
25
31
  attr_accessor :logger
26
32
  # @param value [Boolean] set false to start no monitor thread, e.g. in tests
27
33
  attr_writer :enabled
28
34
 
29
35
  def initialize
30
- @tick = Heartbeat.env_int(:tick, DEFAULT_TICK)
31
36
  @startup_grace_ticks = DEFAULT_STARTUP_GRACE_TICKS
32
37
  @enabled = true
33
38
  @logger = nil
39
+ @tick = environment_tick
34
40
  end
35
41
 
36
42
  # @return [Boolean]
@@ -39,20 +45,36 @@ module KicksLiveness
39
45
  end
40
46
 
41
47
  # A tick is what the monitor thread sleeps on, so a non-positive value is
42
- # not a setting but a broken monitor. The environment gets a silent fallback
43
- # instead (see {Heartbeat.env_int}) because a ConfigMap typo must not bring
44
- # a worker down; an initializer is code, and code should say so at boot,
45
- # where the developer is looking.
48
+ # not a setting but a broken monitor. An initializer is code, and code
49
+ # should fail loudly at boot, where the developer is looking. Environment
50
+ # input is resolved separately by {#environment_tick}, with a safe fallback.
46
51
  #
47
- # @param seconds [Integer]
52
+ # @param seconds [Numeric]
48
53
  # @raise [ArgumentError] if not a positive number
49
- # @return [Integer]
54
+ # @return [Numeric]
50
55
  def tick=(seconds)
51
56
  raise ArgumentError, "tick must be a positive number, got #{seconds.inspect}" unless positive_number?(seconds)
57
+ raise ArgumentError, "tick must be less than max_age (#{max_age}s), got #{seconds.inspect}" if seconds >= max_age
52
58
 
59
+ @incompatible_environment_tick = nil
53
60
  @tick = seconds
54
61
  end
55
62
 
63
+ # The monitor compares an integer tick counter with this value. Accepting
64
+ # zero, a float, or a string would silently prevent the startup escalation
65
+ # from ever firing.
66
+ #
67
+ # @param ticks [Integer]
68
+ # @raise [ArgumentError] unless +ticks+ is a positive integer
69
+ # @return [Integer]
70
+ def startup_grace_ticks=(ticks)
71
+ unless ticks.is_a?(Integer) && ticks.positive?
72
+ raise ArgumentError, "startup_grace_ticks must be a positive integer, got #{ticks.inspect}"
73
+ end
74
+
75
+ @startup_grace_ticks = ticks
76
+ end
77
+
56
78
  # Read-only, sourced from the environment so that it matches what the probe
57
79
  # sees.
58
80
  # @return [String] marks directory
@@ -76,6 +98,47 @@ module KicksLiveness
76
98
 
77
99
  private
78
100
 
101
+ def environment_tick
102
+ raw = Heartbeat.env_raw(:tick)
103
+ seconds = Heartbeat.env_int(:tick, DEFAULT_TICK)
104
+ threshold = max_age
105
+ return seconds if seconds < threshold
106
+
107
+ fallback = [DEFAULT_TICK, threshold / 2.0].min
108
+ @incompatible_environment_tick = [seconds, threshold, fallback, raw]
109
+ fallback
110
+ end
111
+
112
+ def warn_incompatible_environment_tick_once
113
+ warning = @incompatible_environment_tick
114
+ return unless warning
115
+
116
+ @incompatible_environment_tick = nil
117
+ warn_incompatible_environment_tick(*warning)
118
+ end
119
+
120
+ def warn_incompatible_environment_tick(seconds, threshold, fallback, raw)
121
+ Kernel.warn(
122
+ "WARN [liveness] effective tick=#{seconds}s (#{environment_tick_source(raw)}) must be less than " \
123
+ "KICKS_LIVENESS_MAX_AGE=#{threshold}s; using #{fallback}s"
124
+ )
125
+ rescue StandardError
126
+ # Configuration recovery must not become a worker boot failure merely
127
+ # because stderr is unavailable or warning output has been overridden.
128
+ nil
129
+ end
130
+
131
+ def environment_tick_source(raw)
132
+ return 'default' if raw.nil?
133
+
134
+ value = Integer(raw)
135
+ return "KICKS_LIVENESS_TICK=#{raw}" if value.positive?
136
+
137
+ "default after invalid KICKS_LIVENESS_TICK=#{raw.inspect}"
138
+ rescue ArgumentError, TypeError
139
+ "default after invalid KICKS_LIVENESS_TICK=#{raw.inspect}"
140
+ end
141
+
79
142
  def positive_number?(value)
80
143
  value.is_a?(Numeric) && value.positive?
81
144
  end
@@ -60,7 +60,7 @@ module KicksLiveness
60
60
  consumers: kicks_liveness_expected_consumers
61
61
  )
62
62
  rescue StandardError => e
63
- KicksLiveness.config.resolved_logger&.error("[liveness] failed to start: #{e.class}: #{e.message}")
63
+ report_start_failure(e)
64
64
  end
65
65
  end
66
66
 
@@ -80,6 +80,17 @@ module KicksLiveness
80
80
 
81
81
  private
82
82
 
83
+ def report_start_failure(error)
84
+ KicksLiveness.config.resolved_logger&.error(
85
+ "[liveness] failed to start: #{error.class}: #{error.message}"
86
+ )
87
+ rescue StandardError
88
+ # This is already the failure path. Neither resolving the configuration
89
+ # again nor a broken logger may let liveness instrumentation stop the
90
+ # worker process it is meant to observe.
91
+ nil
92
+ end
93
+
83
94
  # The same set the worker gem itself builds its workers from, so the
84
95
  # queue list is never duplicated and cannot drift. An array of classes
85
96
  # under sneakers:run, a callable registry under sneakers:active_job.
@@ -173,6 +173,12 @@ module KicksLiveness
173
173
 
174
174
  def log(level, message)
175
175
  @config.resolved_logger&.public_send(level, "[liveness] slot #{@slot}: #{message}")
176
+ rescue StandardError
177
+ # Logging is diagnostic, while the heartbeat is the liveness contract. A
178
+ # broken custom logger must not prevent a mark from being written or kill
179
+ # the only thread that can refresh it. There is deliberately no fallback
180
+ # log here: calling the same logger again would only repeat the failure.
181
+ nil
176
182
  end
177
183
  end
178
184
  end
@@ -1,4 +1,4 @@
1
1
  module KicksLiveness
2
2
  # @return [String] gem version
3
- VERSION = '0.1.0'.freeze
3
+ VERSION = '0.1.1'.freeze
4
4
  end
@@ -46,10 +46,13 @@ module KicksLiveness
46
46
  # <tt>gem 'kicks', require: false</tt> the to_prepare hook runs earlier and
47
47
  # would fail on NameError.
48
48
  #
49
- # @raise [LoadError] if neither +kicks+ nor +sneakers+ is available
49
+ # @raise [LoadError] if neither worker gem is available, or if both +kicks+
50
+ # and +sneakers+ are activated
50
51
  # @return [Module]
51
52
  # @see file:docs/SETUP.md#installing-the-hooks
52
53
  def install!
54
+ reject_ambiguous_worker_gems!
55
+
53
56
  begin
54
57
  require 'sneakers'
55
58
  require 'sneakers/workergroup'
@@ -80,6 +83,15 @@ module KicksLiveness
80
83
 
81
84
  Monitor.new(slot: slot, processes: processes, consumers: consumers, config: config).start!
82
85
  end
86
+
87
+ private
88
+
89
+ def reject_ambiguous_worker_gems!
90
+ return unless defined?(Gem.loaded_specs)
91
+ return unless Gem.loaded_specs.key?('kicks') && Gem.loaded_specs.key?('sneakers')
92
+
93
+ raise LoadError, 'kicks_liveness cannot run with both kicks and sneakers activated; install exactly one'
94
+ end
83
95
  end
84
96
  end
85
97
 
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: kicks_liveness
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 0.1.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - PoroshkinaVV
@@ -24,6 +24,7 @@ executables:
24
24
  extensions: []
25
25
  extra_rdoc_files: []
26
26
  files:
27
+ - ".yardopts"
27
28
  - CHANGELOG.md
28
29
  - LICENSE.txt
29
30
  - README.md
@@ -48,7 +49,7 @@ licenses:
48
49
  - MIT
49
50
  metadata:
50
51
  source_code_uri: https://github.com/PoroshkinaVV/kicks_liveness
51
- documentation_uri: https://rubydoc.info/gems/kicks_liveness/0.1.0
52
+ documentation_uri: https://rubydoc.info/gems/kicks_liveness/0.1.1
52
53
  changelog_uri: https://github.com/PoroshkinaVV/kicks_liveness/blob/main/CHANGELOG.md
53
54
  bug_tracker_uri: https://github.com/PoroshkinaVV/kicks_liveness/issues
54
55
  rubygems_mfa_required: 'true'