kicks_liveness 0.1.3 → 0.1.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: db97cf34081f6a62735363603c3139ff3ddea09c23d81305d2e3a2e2e84ff126
-  data.tar.gz: ca1ee94ae412b24615b4df2eabe3d1344375721f4b50a33ca6af2590c6b60ae3
+  metadata.gz: faf817806eb5142b8271959b7856a16db10a99db8eed583bb5594887198b6dd8
+  data.tar.gz: 9c5f3f845c78a56e5da97204769e5c2b10161fbf56e5c40a731b904650ca469a
 SHA512:
-  metadata.gz: 11b5aaed04e4e67c3607a25291d7f1443f5ca00885b0fec3f9931ebc967c02f11deb150697a78f0a936c832334ef895783ec532801d72e61d92e2f2b715a2c67
-  data.tar.gz: dd47406061ce002cd058bd4d89577a383d0eafcc7808688f4d147a660ccd635b6141b75ffa97a6e7778ce661d51cde8b99f0d31e9ce7bf731e4633d84dc9438c
+  metadata.gz: a077fc085bed4c460ed9820940271fedb44e69a2778ca87997b23c172fe94d0622bbf334d9b5456827d1e231813ec5e903de6ebe6c655804f8b8c359ca26832c
+  data.tar.gz: 1d2efdab9aa59777cc14075703c9c75bfad76dc511d94659a8f2f40d2cc8a3a957779e4795f304beabbed04c4854f88e2413fd044216497df7338d6a7a32a864

data/CHANGELOG.md

@@ -5,6 +5,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 
+## [0.1.4] - 2026-11-03
+
+### Fixed
+
+- Bound the graceful-shutdown heartbeat exemption by a monotonic deadline
+  starting at the first stop notification; repeated callbacks cannot extend it.
+  At expiry the monitor logs one ERROR and stops refreshing its mark, but still
+  declares the fork count. A stuck drain then fails the ordinary freshness
+  check; the monitor does not kill the process.
+
+### Added
+
+- Add `shutdown_grace` / `KICKS_LIVENESS_SHUTDOWN_GRACE` (default 60 seconds;
+  initializer wins). Set it to at least `terminationGracePeriodSeconds`.
+  Restart still waits for mark staleness and the kubelet's probe failure budget.
+- Add `verify.sh shutdown` and `container-restart` to check a real drain and
+  same-pod container replacement, with deployment restoration afterwards.
+- Add an `install!` setup hint when the probe finds no declared worker count.
+  It remains a failing check, not proof that the hooks were never installed.
+
 ## [0.1.3] - 2026-10-21
 
 ### Fixed
@@ -124,7 +144,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
   [docs/](https://github.com/PoroshkinaVV/kicks_liveness/tree/main/docs); start
   with `SETUP.md`, and read `LIMITATIONS.md` before relying on it.
 
-[Unreleased]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.3...HEAD
+[Unreleased]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.4...HEAD
+[0.1.4]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.3...v0.1.4
 [0.1.3]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.2...v0.1.3
 [0.1.2]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.1...v0.1.2
 [0.1.1]: https://github.com/PoroshkinaVV/kicks_liveness/compare/v0.1.0...v0.1.1

data/README.md

@@ -77,6 +77,9 @@ only when every expected worker mark is present and fresh.
 Use a startup probe for the boot budget and a liveness probe for steady state:
 
 ```yaml
+        env:
+          - name: KICKS_LIVENESS_SHUTDOWN_GRACE
+            value: "60"           # at least terminationGracePeriodSeconds
         startupProbe:
           exec:
             command: ["bundle", "exec", "kicks-liveness"]
@@ -139,6 +142,7 @@ environment too:
 | `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | heartbeat directory; must be on tmpfs | uses the default |
 | `KICKS_LIVENESS_MAX_AGE` | `45` | seconds before a mark is stale | uses the default |
 | `KICKS_LIVENESS_TICK` | `10` | seconds between worker checks | uses the default, then the safety fallback below if needed |
+| `KICKS_LIVENESS_SHUTDOWN_GRACE` | `60` | seconds to keep publishing during shutdown | uses the default |
 
 If the effective environment tick is not smaller than `max_age`, the worker
 writes a WARN to stderr and uses `10` seconds when that is safe, or half of
@@ -146,10 +150,16 @@ writes a WARN to stderr and uses `10` seconds when that is safe, or half of
 while guaranteeing that a healthy heartbeat cannot become stale between
 ordinary ticks.
 
-An initializer value for `tick` wins over `KICKS_LIVENESS_TICK`. `dir` and
+Initializer values for `tick` and `shutdown_grace` override their environment
+variables. `dir` and
 `max_age` deliberately have no initializer setters: allowing two configuration
 sources could make the worker and probe silently disagree.
 
+Set `shutdown_grace` to at least `terminationGracePeriodSeconds`. A drain that
+exceeds this deadline stops refreshing its heartbeat and logs one ERROR; the
+probe fails once the last mark becomes stale. See
+[shutdown timing](docs/LIMITATIONS.md#graceful-shutdown-has-a-finite-heartbeat-exemption).
+
 If two runners share one pod — for example `sneakers:run` and
 `sneakers:active_job` — give each a different `KICKS_LIVENESS_DIR`. Separate
 pods already have separate `emptyDir` volumes.

data/docs/DESIGN.md

@@ -235,13 +235,13 @@ with `respond_to?(:call)` rather than maintaining a separate queue list.
 | Setting | Where | Why |
 |---|---|---|
 | `logger`, `enabled`, `startup_grace_ticks` | application config block | only the worker needs them |
-| `tick` | either, and the config block wins | only the worker reads it, so two sources cannot contradict each other |
+| `tick`, `shutdown_grace` | either, and the config block wins | only the worker reads them, so two sources cannot contradict each other |
 | `dir`, `max_age` | environment variables **only** | the probe is a separate process |
 
 The exec probe cannot read an application initializer, so `dir` and `max_age`
 have one source: the environment. Specs reject application setters for both.
-Only the worker reads `tick`, so `config.tick` may override
-`KICKS_LIVENESS_TICK` without contradicting the probe.
+Only the worker reads `tick` and `shutdown_grace`, so initializer values may
+override their environment values without contradicting the probe.
 
 Environment durations must be positive integers; missing, empty, unparseable,
 zero, and negative values use defaults. If the effective tick is at least
@@ -249,6 +249,10 @@ zero, and negative values use defaults. If the effective tick is at least
 `max_age` otherwise. The warning is deferred until the tick is read, so an
 initializer override or a disabled monitor does not produce a misleading WARN.
 
+`KICKS_LIVENESS_SHUTDOWN_GRACE` defaults to 60 seconds. Its initializer override
+accepts positive finite real numbers. Set it to at least the deployment's
+`terminationGracePeriodSeconds`, which the gem cannot discover from a signal.
+
 Invalid ConfigMap input falls back so a deployment typo does not prevent worker
 startup. Initializer values are application code: invalid values raise
 `ArgumentError` at boot, where the developer can fix the error, rather than
@@ -288,8 +292,12 @@ Loop and startup exceptions do not use that flag.
 
 ERROR must survive the usual error-level logger filter, while normal startup
 and deliberate shutdown remain INFO. On shutdown the monitor suspends its
-predicate and refreshes the mark for the rest of the process, assuming an
-[external deadline](LIMITATIONS.md#graceful-shutdown-is-safe-only-when-something-else-owns-the-deadline).
+predicate until a monotonic deadline: the first `Registry.stopping!` records the
+stop time, and the monitor adds `shutdown_grace`. A delayed tick or repeated
+callback cannot renew it. At expiry it logs one ERROR and stops `touch!` while
+continuing `declare!`. Nothing is persisted, so a replacement fork does not
+inherit a sibling's shutdown. Mark freshness still uses wall time; see the
+[timing boundary](LIMITATIONS.md#graceful-shutdown-has-a-finite-heartbeat-exemption).
 
 Startup grace postpones the error for a slow initial subscription; a surviving
 monitor reports once when it expires. Fork-respawn loops use the persistent

data/docs/KUBERNETES.md

@@ -3,6 +3,9 @@
 ## Manifest
 
 ```yaml
+        env:
+          - name: KICKS_LIVENESS_SHUTDOWN_GRACE
+            value: "60"           # at least terminationGracePeriodSeconds
         startupProbe:
           exec:
             command: ["bundle", "exec", "kicks-liveness"]
@@ -142,11 +145,17 @@ image. One line, no cluster needed:
 docker run --rm your-image ruby -e "require 'kicks_liveness/probe'"
 ```
 
-A line like `no /opt/app/tmp/health/expected: worker has not started yet` and
-exit 1 means the plain command works — the probe found itself and correctly
-reported that no worker is running. A `LoadError` means it does not: keep
-`bundle exec kicks-liveness`, or use the explicit-path optimisation below. Do
-not put the plain command in a manifest without this check: its failure mode is
+With no worker running, exit 1 and the following line mean the plain command
+works — the probe loaded and found no declared worker count:
+
+```
+no /opt/app/tmp/health/expected: worker has not started yet (is KicksLiveness.install! called before the runner starts? outside Rails it is not automatic)
+```
+
+The question is a [setup hint](SETUP.md#checking-that-the-hooks-are-in-place),
+not proof of missing installation. A `LoadError` means the command does not
+work: keep `bundle exec kicks-liveness`, or use the explicit-path optimisation
+below. Do not put the plain command in a manifest without this check: its failure mode is
 a pod that never passes `startupProbe` and lands in `CrashLoopBackOff`.
 
 What decides it is whether Bundler installed into `GEM_HOME` or somewhere of
@@ -313,6 +322,15 @@ several consumers per worker these add up. If the grace period is shorter than
 that, the container is SIGKILLed mid-shutdown and in-flight messages are
 redelivered.
 
+Set `KICKS_LIVENESS_SHUTDOWN_GRACE` (or `config.shutdown_grace`) to at least
+`terminationGracePeriodSeconds`; both are 60 seconds in the example. The
+monitor keeps the mark fresh only within that shutdown grace. For a SIGTERM
+sent outside pod deletion, a stuck drain stops publishing at the deadline;
+its last mark then has to exceed `max_age`, followed by the liveness probe's
+period and failure threshold. Under normal clock progression, restart
+eligibility is bounded by `shutdown_grace + max_age`, with the probe budget
+additional. During pod deletion the kubelet still owns the termination deadline.
+
 ## Environment variables
 
 | Variable | Default | Purpose | Invalid or blank value |
@@ -320,6 +338,7 @@ redelivered.
 | `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | marks directory, **must be on tmpfs** | uses the default |
 | `KICKS_LIVENESS_MAX_AGE` | `45` | seconds after which a mark is stale | uses the default |
 | `KICKS_LIVENESS_TICK` | `10` | interval between ticks | uses the default, then the safety fallback below if needed |
+| `KICKS_LIVENESS_SHUTDOWN_GRACE` | `60` | seconds to keep publishing during shutdown; at least `terminationGracePeriodSeconds` | uses the default |
 
 Keep `tick` well below `max_age`. A tick longer than half of `max_age` leaves no
 room for a single missed write, and a tick longer than `max_age` guarantees a

data/docs/LIMITATIONS.md

@@ -70,29 +70,38 @@ flapping pod rather than a storm. And the supervisor respawning failing forks
 inside the container — which is *not* throttled by anything — is now reported:
 see the respawn escalation below.
 
-## Graceful shutdown is safe only when something else owns the deadline
-
-`Registry.stopping!` suspends the consumer predicate before `WorkerGroup#stop`
-unsubscribes workers and drains their pools. The monitor keeps the heartbeat
-fresh for the rest of the process so liveness does not interrupt an in-flight
-job. During pod deletion the kubelet enforces `terminationGracePeriodSeconds`,
-which bounds this exemption externally.
-
-A SIGTERM from an operator, sidecar, or internal supervisor enters the same
-path without deleting the pod. If Sneakers or Bunny then drains or reconnects
-indefinitely, the heartbeat stays fresh with zero consumers and the pod can
-remain Ready forever. The process cannot infer an external deadline from the
-signal.
-
-This is existing shutdown behaviour, not a 0.1.3 regression. Until the exemption
-is bounded — planned for 0.1.4 — send SIGTERM only through a termination path
-that has its own enforced deadline. Use pod deletion for Kubernetes operations.
-An internal supervisor must eventually make PID 1 exit itself, or ask the
-container runtime to stop it, if graceful shutdown does not complete; an
-in-namespace `kill -9 1` is not a reliable escalation even when `kill` reports
-success. Keep the external consumer alert enabled as well: it still sees the
-zero-consumer state that the process-local predicate has intentionally stopped
-evaluating.
+## Graceful shutdown has a finite heartbeat exemption
+
+`Sneakers::WorkerGroup#stop` removes consumers before waiting for their thread
+pools to drain. `Registry.stopping!` therefore suspends the consumer predicate
+before the first worker stops, allowing an orderly drain to keep its heartbeat
+fresh without reporting a consumer failure on every deploy.
+
+The exemption lasts for `shutdown_grace` seconds from the first stop callback
+(default 60). Set `KICKS_LIVENESS_SHUTDOWN_GRACE` or `config.shutdown_grace` to at
+least the deployment's `terminationGracePeriodSeconds`. During pod deletion the
+kubelet still owns the outer termination deadline. The gem cannot determine
+whether a SIGTERM came from the kubelet, an operator, or another process.
+
+At the deadline, the monitor logs one ERROR and stops refreshing the slot mark.
+It keeps declaring the expected fork count, so an existing mark eventually
+fails as `worker-<slot> stale`, rather than claiming that the worker never
+started. The monitor does not kill the process or resume checking consumers.
+This bounds the formerly indefinite healthy result when an out-of-band SIGTERM
+leaves a process stuck draining with no consumers.
+
+Expiry is not an immediate probe failure: the last mark must still exceed
+`max_age`. Under normal clock progression, the bound from stop notification to
+restart eligibility is `shutdown_grace + max_age`; actual restart also depends
+on the probe period and failure threshold. The first stop time and deadline are
+process-local, so repeated stop callbacks do not renew them and a sibling
+replacement fork starts without them.
+
+The deadline uses the monotonic clock. A wall-clock jump cannot shorten or
+extend the exemption, but it can change mark staleness: the probe still compares
+filesystem mtime with wall time. Keep the external consumer alert enabled; the
+shutdown grace and the probe budget deliberately allow a period with no
+consumers before a restart.
 
 ## Changing the worker count at runtime is not supported
 

data/docs/SETUP.md

@@ -148,10 +148,19 @@ config.logger = Rails.logger
 | `enabled` | `true` | start the monitor thread | `true` or `false` |
 | `tick` | `10` | seconds between checks | positive number smaller than `max_age` |
 | `startup_grace_ticks` | `6` | unhealthy startup ticks before one ERROR | positive integer |
+| `shutdown_grace` | `60` | seconds to keep publishing during shutdown | positive finite number |
 
 Invalid combinations are rejected before the monitor starts instead of running
 one that is guaranteed to publish stale marks or never report a stalled startup.
 
+`shutdown_grace` defaults from `KICKS_LIVENESS_SHUTDOWN_GRACE`; the initializer
+wins when both are set. Set it to at least your Kubernetes
+`terminationGracePeriodSeconds`. It bounds the shutdown exemption, not the
+restart time: after expiry the last mark still has to exceed `max_age` and the
+probe's failure budget still applies. The environment accepts positive integer
+seconds and silently uses 60 for invalid or blank values. Invalid initializer
+values raise `ArgumentError`; fractional positive seconds are accepted.
+
 `dir` and `max_age` are **not** here — they come from environment variables
 only. See
 [DESIGN.md](DESIGN.md#what-is-configurable-and-where) for why, and
@@ -169,6 +178,16 @@ Sneakers::WorkerGroup.ancestors.include?(KicksLiveness::Hooks::WorkerGroup) # =>
 Both must be `true` before the runner starts. If either is `false`, either
 `install!` has not run, or it ran before `kicks`/`sneakers` was loaded.
 
+When `expected` is missing, the probe asks whether `KicksLiveness.install!` was
+called before the runner started. This is a hint, not a diagnosis: a slow boot
+or a removed marks directory can produce the same result. It does not install
+hooks or change the failing exit status. The complete output is shown in the
+[image check](KUBERNETES.md#where-your-image-puts-its-gems).
+
+When migrating from another probe, confirm the `[liveness] slot 0: started:`
+line and the `expected` file on dev before removing the old probe gem, its
+initializer and its rake tasks.
+
 Once workers are running, the marks directory is the other half of the answer:
 
 ```

data/docs/VERIFYING.md

@@ -35,9 +35,7 @@ spec/integration/verify.sh up      # namespace, broker, worker
 spec/integration/verify.sh down    # deletes the namespace
 ```
 
-Scenarios 11 and 12 are the exception: they need the image and Docker, but no
-cluster and no broker, because both are about the container itself rather than
-about consuming.
+The Docker form of scenario 11 and scenario 12 need no cluster or broker.
 
 Every `kubectl` call names its context and namespace explicitly, so the script
 never falls back to `current-context`. It will still operate on whichever target
@@ -45,6 +43,24 @@ you configure: run `status` and inspect both values before `up` or `down`. The
 script labels namespaces it creates and refuses to apply to or delete an
 existing namespace without that ownership label.
 
+For another local cluster, export `KICKS_LIVENESS_K8S_CONTEXT` and
+`KICKS_LIVENESS_DOCKER_CONTEXT` before building. `KICKS_LIVENESS_K8S_NAMESPACE`
+selects the disposable namespace; `KICKS_LIVENESS_IMAGE` and
+`KICKS_LIVENESS_RABBITMQ_IMAGE` select local worker and broker images.
+Run the direct recipes below from the repository root with this guarded helper:
+
+```bash
+export KICKS_LIVENESS_K8S_CONTEXT="${KICKS_LIVENESS_K8S_CONTEXT:-docker-desktop}"
+export KICKS_LIVENESS_K8S_NAMESPACE="${KICKS_LIVENESS_K8S_NAMESPACE:-kicks-liveness}"
+k() {
+  local owner
+  owner="$(kubectl --context "$KICKS_LIVENESS_K8S_CONTEXT" get namespace \
+    "$KICKS_LIVENESS_K8S_NAMESPACE" -o 'jsonpath={.metadata.labels.app\.kubernetes\.io/managed-by}')" || return
+  [ "$owner" = kicks-liveness-fixture ] || { echo 'not a fixture namespace' >&2; return 1; }
+  kubectl --context "$KICKS_LIVENESS_K8S_CONTEXT" --namespace "$KICKS_LIVENESS_K8S_NAMESPACE" "$@"
+}
+```
+
 The default image uses Kicks. Build it with Sneakers instead without editing the
 fixture:
 
@@ -117,6 +133,14 @@ The recipe from KUBERNETES.md. Proves the probe's threshold and the
 self-repair, and nothing else — the mark is fixed on the next tick, well inside
 `failureThreshold`, so the container is never killed.
 
+```bash
+pod="$(k get pods -l app=worker -o jsonpath='{.items[0].metadata.name}')"
+k exec "$pod" -- touch -d '120 seconds ago' /opt/app/tmp/health/worker-0
+spec/integration/verify.sh probe  # both commands print stale and exit=1
+sleep 12
+spec/integration/verify.sh probe  # both commands print healthy and exit=0
+```
+
 Worth running through **both** commands here, since this is the one scenario
 where the probe is supposed to answer `1`. `spec/integration/verify.sh probe`
 does exactly that: the standard `bundle exec kicks-liveness` and the optimised
@@ -132,7 +156,9 @@ worker-0 stale 120s > 45s   # exit 1
 The one that matters. Delete a queue the worker consumes from:
 
 ```bash
-kubectl exec deploy/rabbitmq -- rabbitmqctl delete_queue kicks_liveness.alpha
+k exec deploy/rabbitmq -- rabbitmqctl delete_queue kicks_liveness.alpha
+k get pods -l app=worker -w  # watch RESTARTS increase; Ctrl-C after recovery
+spec/integration/verify.sh probe  # exit=0 after the worker re-declares the queue
 ```
 
 Observed, from the moment of deletion:
@@ -166,7 +192,10 @@ not in the pod log. An exec probe's stdout goes nowhere else, which is why
 ### 4. The broker dies
 
 ```bash
-kubectl delete pod -l app=rabbitmq --grace-period=0 --force
+k delete pod -l app=rabbitmq --grace-period=0 --force
+k rollout status deploy/rabbitmq --timeout=300s  # the Deployment recreates it
+spec/integration/verify.sh logs 100
+spec/integration/verify.sh probe
 ```
 
 The pod must **not** restart. "It survived" is not enough on its own — a broker
@@ -186,7 +215,16 @@ consumers back on its own.
 
 ### 5. The broker stays dead
 
-`kubectl scale deploy/rabbitmq --replicas=0`, then wait. Observed over 319 s
+```bash
+k scale deploy/rabbitmq --replicas=0
+sleep 150  # longer than max_age + periodSeconds * failureThreshold
+spec/integration/verify.sh probe  # still exit=0 during Bunny recovery
+k get pods -l app=worker  # RESTARTS must not increase
+k scale deploy/rabbitmq --replicas=1  # restore, including after an interrupted check
+k rollout status deploy/rabbitmq --timeout=300s
+```
+
+With the broker scaled to zero, observed over 319 s
 — seven times `max_age`, more than twice the 135 s steady-state budget — the
 mark's age never exceeded 9 s and `restartCount` stayed 0. The monitor kept
 writing, because `recovering_from_network_failure?` never stops being true:
@@ -211,10 +249,82 @@ quickly — the monitor never gets to run again, so
 `shutting down: consumers are no longer checked` may not appear at all. The
 absence of ERROR is the observable, not the presence of that line.
 
+The shutdown exemption is now bounded by `shutdown_grace` (60 seconds by
+default). Keep `KICKS_LIVENESS_SHUTDOWN_GRACE` at least as long as the fixture's
+`terminationGracePeriodSeconds`. An out-of-band SIGTERM does not start a kubelet
+termination deadline: if the process remains stuck draining, the expected
+monitor diagnostic after its grace is:
+
+```
+[liveness] slot 0: shutdown grace expired: heartbeat is no longer refreshed
+```
+
+The probe can still pass until the last mark exceeds `max_age`; afterwards it
+must exit 1 with `worker-0 stale ...`, and the monitor must not refresh that
+mark on later ticks. The deadline, repeated callbacks, wall-clock changes, and
+a sibling fork's independent lifetime are covered without Kubernetes by:
+
+```bash
+bundle exec rspec spec/kicks_liveness/configuration_spec.rb \
+  spec/kicks_liveness/monitor_spec.rb spec/probe_spec.rb
+```
+
+The probe spec launches a separate Ruby process against the monitor's marks.
+For the real worker and broker, run:
+
+```bash
+spec/integration/verify.sh shutdown
+```
+
+This temporarily uses `tick=1`, `max_age=3`, `shutdown_grace=10` and pod
+termination grace 10 seconds, then restores the original deployment even on
+failure. First it holds a job, rolls out a worker, releases the job only after
+the old pod's first shutdown tick, and requires the job to finish and no
+liveness ERROR; how long the replacement takes to become Ready does not matter.
+Then it blocks a real pool drain, sends SIGTERM without deleting the pod,
+removes the remaining idle consumer, and checks zero consumers, an initially
+healthy probe, one expiry ERROR, an unchanged mark, a stale probe, and a kubelet
+restart restoring both consumers. It releases the blocking job during recovery.
+
+Recorded on 2026-10-21 with 0.1.4: both paths passed on OrbStack Kubernetes
+v1.30.7+orb1, Linux arm64 6.12.10-orbstack-00297-gf8f6e015b993, Docker 27.4.1,
+Ruby 3.4.11, Kicks 3.4.0, Bunny 3.4.0 and the openssl gem 4.0.2, using an image
+built with `--no-cache --pull`. The stuck pod's `restartCount` changed from 0
+to 1.
+
+Replayed on 2026-10-21–22 on the same cluster with Sneakers 2.11.0 /
+ServerEngine 2.0.7 and Sneakers 2.12.0 / ServerEngine 2.1.1, both on Ruby
+3.4.11 and Bunny 2.24.0: both drain paths passed, including the kubelet restart
+and restored consumers. These images used cached layers and local gem packages,
+not another clean online build.
+
+Separate manual pod-deletion checks held the job beyond `shutdown_grace=3s`,
+with `tick=1s`, `max_age=3s` and Kubernetes termination grace 20 seconds. On
+both Sneakers versions and Kicks 3.4.0, an expired mark stayed unchanged and
+both probe commands failed, but the terminating container was not restarted.
+Releasing the job after expiry let it finish with exit 0 in 10.4–11.6 seconds
+from deletion; leaving it blocked caused kubelet to kill it with exit 137 in
+20.8–20.9 seconds. Each run logged expiry once and recovered a healthy
+replacement. The expiry ERROR is expected with this deliberately shorter
+exemption; normal deployments should keep it at least as long as pod grace.
+
 ### 7. Bad broker credentials
 
 Point `AMQP_URL` at a wrong password. The fork cannot subscribe, so the
-supervisor respawns it every `start_worker_delay`, forever. Over 120 s:
+supervisor respawns it every `start_worker_delay`, forever. The new pod never
+passes its startup probe, so the rollout keeps the old healthy pod running and
+`deploy/worker` may resolve to either one; read the newest pod by name. Over
+120 s:
+
+```bash
+amqp="$(k get deploy worker -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="AMQP_URL")].value}')"
+k set env deploy/worker AMQP_URL=amqp://guest:wrong@rabbitmq:5672
+sleep 120
+new="$(k get pods -l app=worker --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1:].metadata.name}')"
+k logs "$new" --tail=30000 | grep liveness
+k set env deploy/worker "AMQP_URL=$amqp"  # restore even if the check fails
+k rollout status deploy/worker --timeout=300s
+```
 
 | | lines |
 |---|---|
@@ -247,6 +357,18 @@ find them.
 `WORKER_COUNT=2` needs no rebuild; the rake task turns it into
 `config[:workers]`.
 
+```bash
+k set env deploy/worker WORKER_COUNT=2
+k rollout status deploy/worker --timeout=300s
+spec/integration/verify.sh probe  # 2 process(es) healthy
+pod="$(k get pods -l app=worker -o jsonpath='{.items[0].metadata.name}')"
+k exec "$pod" -- touch -d '120 seconds ago' /opt/app/tmp/health/worker-1
+spec/integration/verify.sh probe  # worker-1 alone makes it fail
+sleep 12  # the monitor repairs the mark
+k set env deploy/worker WORKER_COUNT=1
+k rollout status deploy/worker --timeout=300s
+```
+
 ```
 INFO: [liveness] slot 0: started: ... processes=2 consumers=2
 INFO: [liveness] slot 1: started: ... processes=2 consumers=2
@@ -268,6 +390,17 @@ because KUBERNETES.md claims the startup probe *incidentally fixes rollouts
 where the old pod was removed before the new one had subscribed* — and that
 claim is about ordering, which needs two pods to be watched at once.
 
+```bash
+k set env deploy/worker WORKER_COUNT=2
+k rollout status deploy/worker --timeout=300s
+# In a second shell with the same exports and helper, watch until rollout completes:
+# while true; do k exec deploy/rabbitmq -- rabbitmqctl -q list_queues name consumers; sleep 1; done
+k rollout restart deploy/worker
+k rollout status deploy/worker --timeout=300s
+k set env deploy/worker WORKER_COUNT=1  # restore; stop the watcher with Ctrl-C
+k rollout status deploy/worker --timeout=300s
+```
+
 `kubectl rollout restart deploy/worker`, with `WORKER_COUNT=2`, so a fully
 subscribed pod is worth two consumers per queue. Counting consumers on the
 broker is what makes the ordering visible:
@@ -298,14 +431,14 @@ treatment as everything else here. The broker in this fixture is enough to check
 it; the plugin is not on by default:
 
 ```bash
-kubectl exec deploy/rabbitmq -- rabbitmq-plugins enable rabbitmq_prometheus
+k exec deploy/rabbitmq -- rabbitmq-plugins enable rabbitmq_prometheus
 ```
 
 Then read all three endpoints from the worker pod, which has a Ruby but no
 `curl`:
 
 ```bash
-kubectl exec deploy/worker -- ruby -rnet/http -e \
+k exec deploy/worker -- ruby -rnet/http -e \
   'puts Net::HTTP.get(URI("http://rabbitmq:15692/metrics/detailed?family=queue_consumer_count"))'
 ```
 
@@ -324,6 +457,15 @@ metric is a *different* name from the aggregate, the detailed endpoint needs the
 Then check the two halves of the alert against a throwaway queue — declare one
 with no consumers, scrape, delete it, scrape again:
 
+```bash
+k exec deploy/worker -- bundle exec ruby -rbunny -e \
+  'c = Bunny.new(ENV.fetch("AMQP_URL")).start; c.create_channel.queue("kicks_liveness.tmpcheck"); c.close'
+# Repeat the scrape above: tmpcheck must have a zero-consumer series.
+k exec deploy/rabbitmq -- rabbitmqctl delete_queue kicks_liveness.tmpcheck
+# Repeat the scrape: tmpcheck must have no series, not a zero.
+k exec deploy/rabbitmq -- rabbitmq-plugins disable rabbitmq_prometheus  # restore
+```
+
 ```
 -- queue exists, zero consumers
    rabbitmq_detailed_queue_consumers{vhost="/",queue="kicks_liveness.tmpcheck"} 0
@@ -379,6 +521,23 @@ container restart and this guarantee cannot hold — scenario 12 is that case,
 showing both fail-closed rejection when the topology can be proved and the
 degraded outcomes when procfs access cannot prove it.
 
+The Kubernetes counterpart also leaves `attempt-0` in the pod's `emptyDir`:
+
+```bash
+spec/integration/verify.sh container-restart
+```
+
+It makes PID 1 exit itself through a fixture-only WINCH handler, not an ignored
+in-namespace SIGKILL or ServerEngine's USR2 reload. It requires the same pod,
+an increased `restartCount`, exit 137, a surviving volume witness, a changed
+generation, one replacement `started:` line without inherited escalation, and
+a healthy probe; the original deployment is restored afterwards. On the
+environment recorded in scenario 6, all checks passed with `restartCount=0->1`.
+The same check also passed for both Sneakers versions recorded there.
+The Docker form was also rerun: PID 1 start time changed from 1515589 to 1515784
+with mount namespace inode 4026534862 unchanged; inherited marks were rejected
+until the delayed replacement writer published.
+
 ### 12. Shared and host PID topology outcomes
 
 The four PID topologies, in one command. It needs the fixture image but neither

data/lib/kicks_liveness/configuration.rb

@@ -13,6 +13,8 @@ module KicksLiveness
     DEFAULT_TICK = 10
     # @return [Integer] unhealthy ticks tolerated at startup before one ERROR
     DEFAULT_STARTUP_GRACE_TICKS = 6
+    # @return [Integer] seconds to keep publishing while the workers drain
+    DEFAULT_SHUTDOWN_GRACE = 60
 
     # Seconds between checks. An incompatibility warning for an environment
     # fallback is delayed until this value is actually used, so an initializer
@@ -24,6 +26,8 @@ module KicksLiveness
     end
     # @return [Integer] unhealthy ticks tolerated at startup before one ERROR
     attr_reader :startup_grace_ticks
+    # @return [Numeric] seconds allowed for graceful shutdown, measured monotonically
+    attr_reader :shutdown_grace
     # @return [Logger, nil] explicit logger; defaults to +Sneakers.logger+
     attr_accessor :logger
     # @param value [Boolean] set false to start no monitor thread, e.g. in tests
@@ -34,6 +38,7 @@ module KicksLiveness
       @enabled = true
       @logger = nil
       @tick = environment_tick
+      @shutdown_grace = Heartbeat.env_int(:shutdown_grace, DEFAULT_SHUTDOWN_GRACE)
     end
 
     # @return [Boolean]
@@ -70,6 +75,21 @@ module KicksLiveness
       @startup_grace_ticks = ticks
     end
 
+    # Only the monitor reads this value, so an initializer may override the
+    # environment. Keep it at least as long as terminationGracePeriodSeconds.
+    # An infinite value would restore the unbounded shutdown exemption.
+    #
+    # @param seconds [Numeric] finite, positive duration
+    # @raise [ArgumentError] unless +seconds+ is a positive finite real number
+    # @return [Numeric]
+    def shutdown_grace=(seconds)
+      unless seconds.is_a?(Numeric) && seconds.real? && seconds.finite? && seconds.positive?
+        raise ArgumentError, "shutdown_grace must be a positive finite number, got #{seconds.inspect}"
+      end
+
+      @shutdown_grace = seconds
+    end
+
     # Read-only, sourced from the environment so that it matches what the probe
     # sees.
     # @return [String] marks directory

data/lib/kicks_liveness/heartbeat.rb

@@ -168,7 +168,8 @@ module KicksLiveness
     ENV_NAMES = {
       dir: 'KICKS_LIVENESS_DIR',
       max_age: 'KICKS_LIVENESS_MAX_AGE',
-      tick: 'KICKS_LIVENESS_TICK'
+      tick: 'KICKS_LIVENESS_TICK',
+      shutdown_grace: 'KICKS_LIVENESS_SHUTDOWN_GRACE'
     }.freeze
 
     # Inspects this process's container generation and PID topology.
@@ -293,7 +294,13 @@ module KicksLiveness
       return [false, UNSUPPORTED_TOPOLOGY] if unsupported_topology?
 
       processes = expected
-      return [false, "no #{expected_path}: worker has not started yet"] unless processes&.positive?
+      unless processes&.positive?
+        return [
+          false,
+          "no #{expected_path}: worker has not started yet " \
+          '(is KicksLiveness.install! called before the runner starts? outside Rails it is not automatic)'
+        ]
+      end
       return [false, 'heartbeat belongs to a previous container: worker has not started yet'] unless current_generation?
 
       problems = (0...processes).filter_map { |slot| problem_for(slot, now) }

data/lib/kicks_liveness/monitor.rb

@@ -45,10 +45,7 @@ module KicksLiveness
       # cannot recover a probe whose expected count is missing.
       @heartbeat.declare!(@processes)
 
-      if Registry.stopping?
-        shutdown_tick!
-        return true
-      end
+      return shutdown_tick! if Registry.stopping?
 
       healthy = Registry.healthy?(@consumers)
       if healthy
@@ -111,12 +108,24 @@ module KicksLiveness
       @unhealthy_ticks ||= 0
     end
 
-    # Deliberate drain suspends the predicate for the rest of the process.
-    # An external deadline must bound it; the monitor does not own that deadline.
-    # @see file:docs/LIMITATIONS.md#graceful-shutdown-is-safe-only-when-something-else-owns-the-deadline
-    def shutdown_tick!
+    # The first stop callback, not a delayed tick, starts the monotonic deadline.
+    # Expiry stops refreshes; ordinary mark staleness then fails the probe.
+    # @see file:docs/LIMITATIONS.md#graceful-shutdown-has-a-finite-heartbeat-exemption
+    def shutdown_tick! # rubocop:disable Naming/PredicateMethod -- Publishes the heartbeat and records transitions.
+      @shutdown_deadline ||= Registry.stopping_at + @config.shutdown_grace
+      report_shutdown_once
+
+      if Process.clock_gettime(Process::CLOCK_MONOTONIC) >= @shutdown_deadline
+        log(:error, 'shutdown grace expired: heartbeat is no longer refreshed') unless @shutdown_expired
+        @shutdown_expired = true
+        return false
+      end
+
       @heartbeat.touch!(@slot)
+      true
+    end
 
+    def report_shutdown_once
       return if @shutdown_reported
 
       log(:info, 'shutting down: consumers are no longer checked')

data/lib/kicks_liveness/registry.rb

@@ -9,7 +9,7 @@ module KicksLiveness
     # Initialised at file-load time: single-threaded, and before any fork.
     @workers = []
     @mutex = Mutex.new
-    @stopping = false
+    @stopping_at = nil
 
     class << self
       # Records a worker as subscribed.
@@ -36,13 +36,23 @@ module KicksLiveness
       # Mark deliberate shutdown before unsubscription changes the predicate.
       #
       # @return [true]
-      def stopping!
-        @stopping = true
+      def stopping! # rubocop:disable Naming/PredicateMethod -- Records the stop notification.
+        @mutex.synchronize { @stopping_at ||= Process.clock_gettime(Process::CLOCK_MONOTONIC) }
+        true
       end
 
       # @return [Boolean] whether shutdown has begun
       def stopping?
-        @stopping
+        !stopping_at.nil?
+      end
+
+      # Captured by the first stop callback, even if the monitor's next tick is
+      # delayed. Repeated callbacks cannot extend the shutdown exemption. This
+      # is process-local: a sibling replacement fork has not been asked to stop.
+      #
+      # @return [Float, nil] monotonic time of the first stop notification
+      def stopping_at
+        @mutex.synchronize { @stopping_at }
       end
 
       # Whether this process is consuming everything it is supposed to consume.

data/lib/kicks_liveness/version.rb

@@ -1,4 +1,4 @@
 module KicksLiveness
   # @return [String] gem version
-  VERSION = '0.1.3'.freeze
+  VERSION = '0.1.4'.freeze
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: kicks_liveness
 version: !ruby/object:Gem::Version
-  version: 0.1.3
+  version: 0.1.4
 platform: ruby
 authors:
 - PoroshkinaVV
@@ -46,7 +46,7 @@ licenses:
 - MIT
 metadata:
   source_code_uri: https://github.com/PoroshkinaVV/kicks_liveness
-  documentation_uri: https://rubydoc.info/gems/kicks_liveness/0.1.3
+  documentation_uri: https://rubydoc.info/gems/kicks_liveness/0.1.4
   changelog_uri: https://github.com/PoroshkinaVV/kicks_liveness/blob/main/CHANGELOG.md
   bug_tracker_uri: https://github.com/PoroshkinaVV/kicks_liveness/issues
   rubygems_mfa_required: 'true'