kicks_liveness 0.1.0 → 0.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 206f62be48c616553a219c499e1106beb68a49fd3c91ca30a1d7f3ee812c15f7
4
- data.tar.gz: b2edcb9d1a6dd351df5b7a25d27e358582def6b6d8cdc4f7ad70b055ad1c25a9
3
+ metadata.gz: df61d00933097bb96e6d752e024e349ab631cd7c9d571cd9bff9ff63b8adda40
4
+ data.tar.gz: 78864373ca06cd998cdee192d064cd00ed69a4ecd73eb1b657474d77ab593066
5
5
  SHA512:
6
- metadata.gz: 06caaf5bfe502f5a83eebe11d03a8122f91ad03672fe2b11f046143ed107d6b81d126344cc293afc2ee311c6fb2f1a1f42739e82ff37b3e737cad7d3b71d97fb
7
- data.tar.gz: 6cf539fef1a39318702015363035bc8f1538fa9cabaaa6543eca038a62059db98a21b9b82a2a53be62a0315e6872adc7b6891c586df52d1bff3700b2071eb700
6
+ metadata.gz: '007091e18fea1b216ba25c634bab1a4770d25cc0f062ee30ba2885563075bc94e7662e58eebbac9507b448217508725612f172189b692e3a4d616e5f81fdd46b'
7
+ data.tar.gz: f460271ca68fdf912559b3fa2c41b02074d5eb457e946d2f985e3e2b0ac3da66dc94ce90c9d5639212a0fc7bb6537046654f31921cf6766fe15c903bd7dc04ef
data/.yardopts ADDED
@@ -0,0 +1,13 @@
1
+ --markup markdown
2
+ --readme README.md
3
+ --title "kicks_liveness"
4
+ --no-private
5
+ lib/**/*.rb
6
+ -
7
+ docs/SETUP.md
8
+ docs/DESIGN.md
9
+ docs/KUBERNETES.md
10
+ docs/LIMITATIONS.md
11
+ docs/VERIFYING.md
12
+ CHANGELOG.md
13
+ LICENSE.txt
data/CHANGELOG.md CHANGED
@@ -3,6 +3,61 @@
3
3
  The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
4
4
  and this project adheres to [Semantic Versioning](https://semver.org/).
5
5
 
6
+ ## [Unreleased]
7
+
8
+ ## [0.1.2] - 2026-09-11
9
+
10
+ ### Fixed
11
+
12
+ - Reject heartbeat files inherited from a previous container incarnation. A
13
+ Kubernetes `emptyDir` survives container restarts, so a fresh mark left by a
14
+ formerly healthy worker could make the next container pass `startupProbe`
15
+ before its consumers had subscribed. Cache files elsewhere in the same
16
+ `emptyDir` remain untouched and available to warm the next application boot.
17
+ This protection assumes the default container-private PID namespace; pods
18
+ using `shareProcessNamespace` or `hostPID` retain the 0.1.1 freshness-only
19
+ behavior (see `docs/LIMITATIONS.md`).
20
+ The marks directory now contains `<dir>/generation`, and each
21
+ `worker-<slot>` mark carries the same value; tooling that enumerates the
22
+ directory should allow for this new metadata file.
23
+
24
+ ## [0.1.1] - 2026-09-09
25
+
26
+ ### Added
27
+
28
+ - Add a repository-hosted social-preview asset for the project documentation.
29
+
30
+ ### Changed
31
+
32
+ - Rework the README into a shorter entry point while retaining the operational
33
+ detail in the focused guides under `docs/`.
34
+ - Reject initializer values where `tick >= max_age`, and require
35
+ `startup_grace_ticks` to be a positive integer. Initializers that previously
36
+ supplied either invalid value now stop application boot with `ArgumentError`.
37
+ - Reject bundles that activate both `kicks` and `sneakers` with `LoadError`
38
+ instead of letting their shared `lib/sneakers.rb` resolve silently by
39
+ load-path order. Applications that still bundle both worker gems must remove
40
+ one before upgrading.
41
+ - Replace the contributor-only `AMQP_WORKER_GEM` and
42
+ `AMQP_WORKER_GEM_VERSION` test selection with Appraisal-generated dependency
43
+ sets, including current and supported-floor targets for both worker gems.
44
+
45
+ ### Fixed
46
+
47
+ - Recover environment values where `tick >= max_age` with a warning and a safe
48
+ interval, preventing a healthy heartbeat from becoming stale between normal
49
+ monitor ticks.
50
+ - Keep worker startup, heartbeat publication, and the monitor loop alive when a
51
+ custom logger raises while handling an INFO or ERROR event.
52
+ - Correct the runtime-scaling documentation: live changes to `workers` are not
53
+ supported in either direction. During scale-up, old and new forks otherwise
54
+ race to publish different expected counts; during scale-down, retired slots
55
+ remain expected.
56
+ - Strengthen the release gate with floor-version coverage, style checks, tag
57
+ ancestry validation, and a packaged-executable smoke test.
58
+ - Include `.yardopts` in the gem so RubyDoc builds the intended README and
59
+ public guide set from the packaged release.
60
+
6
61
  ## [0.1.0] - 2026-09-08
7
62
 
8
63
  ### Added
@@ -18,4 +73,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
18
73
  [docs/](https://github.com/PoroshkinaVV/kicks_liveness/tree/main/docs); start
19
74
  with `SETUP.md`, and read `LIMITATIONS.md` before relying on it.
20
75
 
76
+ [Unreleased]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.2...HEAD
77
+ [0.1.2]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.1...v0.1.2
78
+ [0.1.1]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.0...v0.1.1
21
79
  [0.1.0]: https://github.com/PoroshkinaVV/kicks_liveness/releases/tag/v0.1.0
data/README.md CHANGED
@@ -1,119 +1,75 @@
1
1
  # kicks_liveness
2
2
 
3
- A liveness probe for [Kicks](https://github.com/ruby-amqp/kicks) workers (and
4
- for its predecessor Sneakers): the worker itself publishes a liveness mark to
5
- tmpfs, and the probe only reads it.
3
+ [![Gem Version](https://badge.fury.io/rb/kicks_liveness.svg)](https://rubygems.org/gems/kicks_liveness)
4
+ [![CI](https://github.com/PoroshkinaVV/kicks_liveness/actions/workflows/ci.yml/badge.svg)](https://github.com/PoroshkinaVV/kicks_liveness/actions/workflows/ci.yml)
6
5
 
7
- ## Documentation
6
+ ![kicks_liveness — Kicks and Sneakers worker health](https://raw.githubusercontent.com/PoroshkinaVV/kicks_liveness/main/assets/github-social-preview.png)
8
7
 
9
- - [docs/SETUP.md](docs/SETUP.md) — installation and wiring: Rails, Sinatra,
10
- Hanami, Roda, and no framework at all; what `install!` does and when to call
11
- it.
12
- - [docs/DESIGN.md](docs/DESIGN.md) — why the gem is built this way: the health
13
- predicate, the mark files, slots instead of PIDs, where the hooks attach, what
14
- is configurable and why `dir` and `max_age` live in the environment only.
15
- - [docs/KUBERNETES.md](docs/KUBERNETES.md) — the manifest, the budget
16
- arithmetic, verifying on a live pod, the alert on consumers.
17
- - [docs/LIMITATIONS.md](docs/LIMITATIONS.md) — where the in-memory predicate
18
- stops being truthful.
19
- - [docs/VERIFYING.md](docs/VERIFYING.md) — the runbook that causes the real failures on a local cluster: what was observed, and how long each took.
20
-
21
- ## Why
22
-
23
- The usual implementation is a rake task that boots Rails and asks RabbitMQ for
24
- `consumer_count`. It has three defects that cannot be fixed in place:
25
-
26
- 1. **It is expensive.** Booting Rails every few seconds inside the worker's own
27
- cgroup. On one real service this cost 2.7 s per invocation on a 15 s
28
- period — 38% of the container's CPU request, around the clock.
29
- 2. **It depends on the disk.** A disk that stalls for a few seconds stretches
30
- the boot past `timeoutSeconds`, and the kubelet kills a healthy pod.
31
- 3. **It cascades.** A liveness probe that checks an external dependency turns a
32
- broker hiccup into every replica restarting at once, finishing the broker off.
33
-
34
- On top of that, `consumer_count` is a metric of the **queue**, not of the pod:
35
- with two replicas the live one covers for the stalled one, and the probe cannot
36
- tell.
37
-
38
- This gem inverts that. Every 10 seconds the worker checks, against the Bunny
39
- objects **in its own memory**, that its consumers are in place, and touches a
40
- file. The standard probe invocation took 156 ms in the fixture container, makes
41
- no network call, boots no framework, and reads the state it judges from tmpfs,
42
- which is RAM. It does still load Bundler, the interpreter and the gem's probe
43
- files from the image filesystem, so the disk is not out of the picture
44
- altogether — it is reduced to a small process startup instead of a full
45
- application boot on every probe.
46
-
47
- ## Installation
8
+ **Know when a [Kicks](https://github.com/ruby-amqp/kicks) or
9
+ [Sneakers](https://github.com/jondot/sneakers) worker is stuck without booting
10
+ Rails or querying RabbitMQ from the probe.**
48
11
 
49
- ```ruby
50
- gem 'kicks_liveness'
51
- ```
12
+ The worker checks its own Bunny consumers and publishes a heartbeat to tmpfs.
13
+ The probe only reads that heartbeat, scoped to the current container
14
+ incarnation. No application boot, no network call, and no healthy replica
15
+ hiding a stalled one.
52
16
 
53
- You need `kicks` (>= 3.0) **or** `sneakers` (>= 2.11). Neither is declared as a
54
- dependency of this gem, on purpose: at runtime only the `Sneakers` namespace is
55
- required, and both gems provide it. If neither is present, `install!` raises a
56
- `LoadError` naming both. Both floors are exercised in CI at their exact
57
- versions, not just through a `~>` that resolves to the newest release.
17
+ That restart scoping assumes Kubernetes' default container-private PID
18
+ namespace; see
19
+ [Limitations](docs/LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1)
20
+ for pods that set `shareProcessNamespace` or `hostPID`.
58
21
 
59
- Keeping both gems in one `Gemfile` is **not** allowed, and nothing enforces
60
- that: not Bundler, not a crash at boot. Both own the file `lib/sneakers.rb`, so
61
- one silently wins the load path. An application that has moved to `kicks` would
62
- keep executing `sneakers` 2.12 code while its `Gemfile.lock` claims otherwise.
63
- `sneakers` also caps the `kicks` and `bunny` versions.
22
+ ## Install: one line in Rails
64
23
 
65
- The gem installs its hooks itself through a Railtie. Outside Rails, call
66
- `KicksLiveness.install!` before the runner starts: the tie to Rails is a single
67
- conditionally required file; it is not loaded outside Rails, and no Rails code is
68
- pulled in. The details, including the one genuine restriction — workers must be
69
- started through `Sneakers::Runner` — are in [docs/SETUP.md](docs/SETUP.md).
24
+ ```ruby
25
+ # Gemfile keep exactly one of kicks or sneakers alongside it
26
+ gem 'kicks_liveness'
27
+ ```
70
28
 
71
- ### Initializer
29
+ ```bash
30
+ bundle install
31
+ ```
72
32
 
73
- Optional. For example, a Rails application can disable the monitor locally:
33
+ In Rails, that is all the application-side setup: the Railtie installs the hooks
34
+ automatically. Without Rails, add two lines before `Sneakers::Runner` starts:
74
35
 
75
36
  ```ruby
76
- KicksLiveness.configure do |config|
77
- config.enabled = !Rails.env.local?
78
- end
37
+ require 'kicks_liveness'
38
+ KicksLiveness.install!
79
39
  ```
80
40
 
81
- The full list: `logger` (defaults lazily to `Sneakers.logger`), `enabled`,
82
- `tick` (10 s), `startup_grace_ticks` (6).
83
- A non-positive `tick` raises `ArgumentError`: the monitor thread sleeps on it,
84
- so a non-positive value is not a setting but a broken monitor.
41
+ Requirements: Ruby >= 3.1 and exactly one of `kicks` >= 3.0 or `sneakers` >=
42
+ 2.11. CI pins both floors and tracks the current release of both worker lines.
43
+ Do not put both worker gems in the same bundle: both provide
44
+ `lib/sneakers.rb`, so `install!` rejects that ambiguous process.
45
+ Framework-specific examples are in [Setup](docs/SETUP.md).
46
+
47
+ ## Why replace the usual probe?
85
48
 
86
- ### Environment variables
49
+ The common alternative is a rake task that boots the application and asks
50
+ RabbitMQ for `consumer_count`.
87
51
 
88
- | Variable | Default | |
52
+ | | Rails + `consumer_count` probe | `kicks_liveness` |
89
53
  |---|---|---|
90
- | `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | marks directory, **must live on tmpfs** |
91
- | `KICKS_LIVENESS_MAX_AGE` | `45` | seconds after which a mark is stale |
92
- | `KICKS_LIVENESS_TICK` | `10` | interval between ticks |
93
-
94
- `tick` is the one setting that appears in both places; the initializer wins over
95
- the variable, and only the worker reads it either way.
96
-
97
- **Two runners in one pod need two directories.** An application that runs both
98
- `rake sneakers:run` and `rake sneakers:active_job` has two supervisors, each
99
- numbering its forks from zero so with the default `KICKS_LIVENESS_DIR` they
100
- would overwrite each other's `expected` and `worker-0`, and the probe would
101
- answer for whichever wrote last. In separate pods, which is the usual
102
- arrangement, there is nothing to do: each gets its own `emptyDir`. In one pod,
103
- give each runner its own `KICKS_LIVENESS_DIR`.
104
-
105
- `dir` and `max_age` deliberately **cannot** be set in the initializer. The probe
106
- is launched by the kubelet as a separate process, which does not — and cannot —
107
- read the application's initializer. Were they settable in two places, a
108
- mismatch between the worker and the probe would pass unnoticed.
109
-
110
- Garbage in a value does not bring the worker down — it falls back to the default.
111
- An empty string counts as unset: in a ConfigMap that is what you get by
112
- declaring a key and leaving it blank. `max_age` and `tick` are durations, so
113
- zero and negative values fall back too: they parse perfectly well, and a
114
- negative `max_age` would make every mark stale on arrival.
115
-
116
- ## Manifest
54
+ | Code started on every probe | Rails and the application | Ruby, Bundler, and the probe files |
55
+ | Network call | RabbitMQ request | none |
56
+ | Health scope | a queue shared by every replica | this worker process |
57
+ | Decision path | application files plus RabbitMQ | process memory plus a tmpfs heartbeat |
58
+ | Measured invocation | **2.7 s of CPU** on a real service | **156 ms** in the fixture container |
59
+ | Continuous cost in the measured setups | **~0.18 core** at a 15 s period | **~0.005 core** at a 30 s period |
60
+ | During Bunny network recovery | may restart every replica together | stays healthy while Bunny reconnects |
61
+
62
+ Absolute timings depend on the image and hardware; the two columns are measured
63
+ examples, not a same-host microbenchmark. The calculation and faster 52 ms and
64
+ ~10 ms probe forms are documented in [Running under Kubernetes](docs/KUBERNETES.md#why-the-command-looks-like-that).
65
+
66
+ Every 10 seconds by default, each worker verifies that all expected consumers
67
+ are subscribed and touches its own mark. `bundle exec kicks-liveness` exits 0
68
+ only when every expected worker mark is present and fresh.
69
+
70
+ ## Kubernetes
71
+
72
+ Use a startup probe for the boot budget and a liveness probe for steady state:
117
73
 
118
74
  ```yaml
119
75
  startupProbe:
@@ -131,7 +87,7 @@ negative `max_age` would make every mark stale on arrival.
131
87
  terminationGracePeriodSeconds: 60
132
88
  ```
133
89
 
134
- The marks directory has to be on tmpfs:
90
+ Put the heartbeat directory on tmpfs:
135
91
 
136
92
  ```yaml
137
93
  volumeMounts:
@@ -140,180 +96,108 @@ The marks directory has to be on tmpfs:
140
96
  volumes:
141
97
  - name: app-tmp
142
98
  emptyDir:
143
- medium: Memory # without this the mark lands on disk
99
+ medium: Memory
144
100
  ```
145
101
 
146
- The container needs a writable volume mounted at, or above,
147
- `KICKS_LIVENESS_DIR`. The default mount at `/opt/app/tmp` covers
148
- `/opt/app/tmp/health`, and the gem creates the `health` subdirectory itself.
149
- Without this mount, a container using `readOnlyRootFilesystem: true` cannot
150
- publish heartbeat files and never passes the startup probe.
151
-
152
- Why it looks like this:
153
-
154
- - **`bundle exec kicks-liveness` is the standard command**, and it works
155
- regardless of where Bundler installed the gems. There is a faster form —
156
- `ruby -e "require 'kicks_liveness/probe'"`, 52 ms against
157
- 156 ms — but it needs the gems to be in `GEM_HOME`, and **setting
158
- `BUNDLE_PATH` at all moves them**, even to the directory `GEM_HOME` already
159
- points at. Check before you optimise:
160
-
161
- ```bash
162
- docker run --rm your-image bundle exec kicks-liveness
163
- docker run --rm your-image ruby -e "require 'kicks_liveness/probe'"
164
- ```
165
-
166
- Both should print `no .../expected` and exit 1. If the second raises
167
- `LoadError`, keep the standard command — at `periodSeconds: 30` it costs about
168
- 0.005 of a core, against roughly 0.18 for a probe that boots the application.
169
- The trade-offs are in
170
- [docs/KUBERNETES.md](docs/KUBERNETES.md#where-your-image-puts-its-gems).
171
-
172
- - **`startupProbe` is mandatory.** Startup and steady state have different time
173
- budgets, and one probe cannot serve both. While it runs, liveness is disabled
174
- and the container is not Ready. `failureThreshold: 60` buys 300 s, which is
175
- deliberately generous — measure a *freshly created* pod before trimming it, as
176
- a restarted container inherits a warm compile cache and a new pod does not:
177
- [measure the cold start](docs/KUBERNETES.md#measure-the-cold-start-not-the-restart).
178
- - **It also protects rollouts — given `maxUnavailable: 0`.** Because the
179
- container is not Ready until the mark exists, a `RollingUpdate` that is not
180
- allowed to drop below full capacity cannot remove the old pod before the new
181
- one has subscribed. That is a property of the strategy as much as of the probe:
182
- with a non-zero `maxUnavailable`, or with `Recreate`, the old pod may go first
183
- and leave the queue uncovered.
184
- - **`initialDelaySeconds` on liveness is unnecessary**; the startupProbe plays
185
- that role.
186
- - Before a kill there is `max_age + periodSeconds × failureThreshold` = 45 + 90 =
187
- **135 s** of confirmed silence. Slow on purpose: a worker is not
188
- latency-critical, and a false restart costs more than two minutes of stalling.
189
-
190
- ## Required companion: an alert on consumers
191
-
192
- The probe deliberately reports healthy while Bunny is reconnecting — otherwise a
193
- broker hiccup would restart every replica at once. The price is that **broker
194
- unavailability stops being visible automatically**. A pod can be consuming
195
- nothing and look perfectly healthy.
196
-
197
- So the probe needs an alert alongside it, and that alert should be in place
198
- **before** the manifest is switched over:
102
+ The default heartbeat directory is `/opt/app/tmp/health`; the gem creates the
103
+ `health` subdirectory. The mount is also required with
104
+ `readOnlyRootFilesystem: true`.
199
105
 
200
- ```yaml
201
- - alert: QueueWithoutConsumers
202
- expr: rabbitmq_detailed_queue_consumers{queue=~"myapp\..+", queue!~".*delayed_active_job.*"} == 0
203
- for: 5m
204
-
205
- # Not optional: the expression above cannot fire when the series is gone.
206
- - alert: QueueConsumerMetricMissing
207
- expr: absent_over_time(rabbitmq_detailed_queue_consumers{queue=~"myapp\..+"}[10m])
208
- for: 5m
209
- ```
106
+ With the values above, confirmed silence lasts at most `45 + 30 × 3 = 135 s`
107
+ before Kubernetes kills the container. Size the startup budget from fresh-pod
108
+ measurements, not warm restarts. See [the full manifest and budget
109
+ arithmetic](docs/KUBERNETES.md).
210
110
 
211
- Excluding the delayed queues is mandatory — they have no consumers by design, and
212
- without the exclusion the alert fires permanently until someone silences it.
213
- Match them as a **substring**: under ActiveJob the full name looks like
214
- `myapp.active_job.myapp.delayed_active_job:60`, so an anchored pattern never
215
- matches.
111
+ ## Configuration
216
112
 
217
- Two traps in getting that metric, both measured on RabbitMQ 3.13.7 rather than
218
- recalled. The per-queue series is `rabbitmq_detailed_queue_consumers`, and it
219
- only exists on `/metrics/detailed` **scraped with `family=queue_consumer_count`**
220
- — bare, that endpoint serves no queue metrics at all, while plain `/metrics`
221
- offers a same-sounding `rabbitmq_queue_consumers` that is a single label-less
222
- cluster total, so a `{queue=~...}` matcher against it is quietly never true. And
223
- the series **disappears** rather than going to zero when the queue is deleted or
224
- the broker stops, which is why the second alert is there.
225
- [docs/KUBERNETES.md](docs/KUBERNETES.md#getting-that-metric-at-all) has the
226
- scrape config.
113
+ Everything is optional. Application settings belong in an initializer:
227
114
 
228
- ## What is checked
115
+ ```ruby
116
+ KicksLiveness.configure do |config|
117
+ config.enabled = !Rails.env.local?
118
+ end
119
+ ```
229
120
 
230
- The mark is written only if **every** worker in the process has subscribed:
121
+ | Ruby setting | Default | Purpose | Valid values |
122
+ |---|---|---|---|
123
+ | `logger` | `Sneakers.logger`, resolved lazily | transition and error logs | logger-compatible object |
124
+ | `enabled` | `true` | start the monitor thread | `true` or `false` |
125
+ | `tick` | `10` | seconds between worker checks | positive number smaller than `max_age` |
126
+ | `startup_grace_ticks` | `6` | unhealthy startup ticks before one ERROR | positive integer |
231
127
 
232
- | State | Verdict |
233
- |---|---|
234
- | consumers in place, channel open | healthy |
235
- | Bunny is recovering the connection | **healthy** — the broker heals itself |
236
- | connection open, no consumers | unhealthy, a restart is the cure |
237
- | not the whole set subscribed | unhealthy |
128
+ The separate probe cannot read an application initializer. The shared path and
129
+ freshness threshold therefore come from the environment; `tick` may use the
130
+ environment too:
238
131
 
239
- The expected consumer count comes from `config[:worker_classes]` the same set
240
- kicks itself builds its workers from. That way the queue list is never
241
- duplicated and cannot drift from the configuration.
132
+ | Environment variable | Default | Purpose | Invalid or blank value |
133
+ |---|---|---|---|
134
+ | `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | heartbeat directory; must be on tmpfs | uses the default |
135
+ | `KICKS_LIVENESS_MAX_AGE` | `45` | seconds before a mark is stale | uses the default |
136
+ | `KICKS_LIVENESS_TICK` | `10` | seconds between worker checks | uses the default, then the safety fallback below if needed |
242
137
 
243
- The files are named by supervisor slot rather than by PID: a dead fork is
244
- respawned into the same slot and overwrites its own file. With a PID in the
245
- name, SIGKILL would leave a stale file forever. With `workers > 1` the probe
246
- requires every slot to be fresh.
138
+ If the effective environment tick is not smaller than `max_age`, the worker
139
+ writes a WARN to stderr and uses `10` seconds when that is safe, or half of
140
+ `max_age` otherwise. This keeps a ConfigMap mistake from stopping the worker
141
+ while guaranteeing that a healthy heartbeat cannot become stale between
142
+ ordinary ticks.
247
143
 
248
- ## Limitations
144
+ An initializer value for `tick` wins over `KICKS_LIVENESS_TICK`. `dir` and
145
+ `max_age` deliberately have no initializer setters: allowing two configuration
146
+ sources could make the worker and probe silently disagree.
249
147
 
250
- The predicate is computed over the Bunny objects **in the process's memory** —
251
- that is the whole point, but it has a boundary: it is exactly as truthful as
252
- Bunny's own bookkeeping. The broker is never asked for its opinion.
148
+ If two runners share one pod for example `sneakers:run` and
149
+ `sneakers:active_job` give each a different `KICKS_LIVENESS_DIR`. Separate
150
+ pods already have separate `emptyDir` volumes.
253
151
 
254
- **1. Consumer bookkeeping is updated on the worker's thread pool.** When the
255
- broker cancels a consumer (on `consumer_timeout`, for example), it sends
256
- `basic.cancel`, and Bunny removes the entry from its list — but it does so
257
- through `@work_pool.submit`, that is, on the same pool that processes jobs
258
- (`threads: 10` by default). Until a thread frees up, `any_consumers?` still
259
- answers yes.
152
+ ## What is healthy?
260
153
 
261
- In practice this means: if **all** of the worker's threads are busy with stuck
262
- jobs, the probe stays green even though the pod is consuming nothing. One stuck
263
- job out of ten does not produce that effect.
154
+ | Worker state | Probe verdict | Reason |
155
+ |---|---|---|
156
+ | every consumer subscribed, channel open | healthy | the process can consume work |
157
+ | Bunny recovering its connection | **healthy** | Bunny reconnects itself; restarting adds a reconnect storm |
158
+ | connection open, no consumers | unhealthy | a restart is the cure |
159
+ | only part of the expected worker set subscribed | unhealthy | one healthy worker must not hide a missing one |
160
+
161
+ The check reads Bunny objects already in the worker's memory. It never asks the
162
+ broker for its view.
264
163
 
265
- **2. `recover_cancelled_consumers!` makes the probe blind to a cancelled
266
- consumer.** `Bunny::Channel#recover_cancelled_consumers!` is an opt-in method
267
- that neither Kicks nor Sneakers enables. With it on, Bunny re-subscribes the
268
- consumer itself on `basic.cancel` and **keeps** the entry in its list — so the
269
- "consumer was cancelled" case stops being detectable at all. Do not enable it
270
- together with this probe.
164
+ ## Operational contract
271
165
 
272
- **3. Broker unavailability is invisible by design** see [the alert on
273
- consumers](#required-companion-an-alert-on-consumers) above.
166
+ - Start workers through `Sneakers::Runner`; that is where the monitor hook is
167
+ attached. `rake sneakers:run` already does this.
168
+ - Keep the `startupProbe`. With `RollingUpdate` and `maxUnavailable: 0`, it also
169
+ keeps the old pod until the new worker has subscribed.
170
+ - Add an external alert for queues with zero or missing consumers **before**
171
+ switching probes. Broker unavailability is invisible here by design; the
172
+ exact Prometheus rules are in [Running under Kubernetes](docs/KUBERNETES.md#required-companion-an-alert-on-consumers).
173
+ - Restart the runner after changing its process count; changing `workers` at
174
+ runtime is not supported.
274
175
 
275
- All three limitations point the same way: an alert on consumers, watching the
276
- broker **from the outside**, is not a nice-to-have but part of the design. It
277
- catches exactly what an in-memory predicate cannot see.
176
+ The other boundaries blocked worker pools, cancelled-consumer recovery, and
177
+ per-pod versus per-queue scope are collected in [Limitations](docs/LIMITATIONS.md).
278
178
 
279
- ## Verifying on a live pod
179
+ ## Verify it
280
180
 
281
181
  ```bash
282
182
  kubectl exec deploy/myapp -- ls -l /opt/app/tmp/health/
283
183
  kubectl exec deploy/myapp -- sh -c 'bundle exec kicks-liveness; echo "exit=$?"'
284
- kubectl logs deploy/myapp | grep liveness
285
184
  ```
286
185
 
287
- Check the **negative** path too otherwise a working probe is
288
- indistinguishable from an always-green one:
289
-
290
- ```bash
291
- kubectl exec deploy/myapp -- sh -c 'touch -d "2 minutes ago" /opt/app/tmp/health/worker-0; bundle exec kicks-liveness; echo "exit=$?"'
292
- ```
186
+ The healthy result is `N process(es) healthy` with exit code 0. Also test the
187
+ negative path before rollout; the commands and expected output are in
188
+ [Verifying the probe](docs/VERIFYING.md#2-a-stale-mark).
293
189
 
294
- Expect `worker-0 stale 120s > 45s` and `exit=1`. One tick later the mark repairs
295
- itself.
190
+ ## Documentation
296
191
 
297
- ## Logs
192
+ | Guide | Use it for |
193
+ |---|---|
194
+ | [Setup](docs/SETUP.md) | Rails, Sinatra, Hanami, Roda, standalone runners, and hook verification |
195
+ | [Design](docs/DESIGN.md) | health predicate, heartbeat files, hooks, configuration choices, and measurements |
196
+ | [Running under Kubernetes](docs/KUBERNETES.md) | full manifest, CPU/startup budgets, alerts, and live-pod verification |
197
+ | [Limitations](docs/LIMITATIONS.md) | cases the in-memory predicate intentionally cannot cover |
198
+ | [Verification scenarios](docs/VERIFYING.md) | eleven deliberately induced failures and their observed outcomes |
298
199
 
299
- Events are logged, not the pulse: the pulse lives in the mark's mtime.
200
+ ## Contributing
300
201
 
301
- | Event | Level |
302
- |---|---|
303
- | started, with the effective settings | INFO |
304
- | waiting for consumers | INFO |
305
- | became healthy | INFO |
306
- | shutting down | INFO |
307
- | still not healthy after `startup_grace_ticks` ticks | ERROR |
308
- | was healthy, became unhealthy | ERROR |
309
- | a slot respawning without ever becoming healthy, once per grace window | ERROR |
310
- | the monitor thread caught an exception | ERROR |
311
- | the monitor could not be started at all | ERROR |
312
-
313
- ERROR is reserved for genuine failure: a logger usually comes up with
314
- `LOG_LEVEL` defaulting to `error`, so anything that matters must survive that
315
- filter, while an ordinary deploy must not make noise. That is also why shutdown
316
- is an event of its own: the workers unsubscribe while the thread pool drains,
317
- and judged by the consumer count alone a normal deploy would look exactly like
318
- a fault. From the moment shutdown begins the probe keeps the mark fresh
319
- instead, so a slow drain gets the full `terminationGracePeriodSeconds`.
202
+ Setup, running the suite against both worker gems, and the release procedure are
203
+ in [CONTRIBUTING.md](https://github.com/PoroshkinaVV/kicks_liveness/blob/main/CONTRIBUTING.md).