kicks_liveness 0.1.2 → 0.1.4

data/docs/KUBERNETES.md

@@ -3,6 +3,9 @@
 ## Manifest
 
 ```yaml
+        env:
+          - name: KICKS_LIVENESS_SHUTDOWN_GRACE
+            value: "60"           # at least terminationGracePeriodSeconds
         startupProbe:
           exec:
             command: ["bundle", "exec", "kicks-liveness"]
@@ -55,11 +58,16 @@ mark. A fresh mark from the previous container is rejected until every fork in
 the current one has subscribed and published its own mark, so preserving the
 cache cannot make `startupProbe` succeed early.
 
-That guarantee assumes the default container-private PID namespace. Do not set
-`shareProcessNamespace: true` or `hostPID: true` on a pod that relies on it: in
-either topology PID 1 survives a worker-container restart, and the generation
-guard can accept an inherited fresh mark. See
-[LIMITATIONS.md](LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1).
+That guarantee needs the default container-private PID namespace. Do not set
+`shareProcessNamespace: true` or `hostPID: true`: in either topology PID 1
+survives a worker-container restart, so the generation repeats and an inherited
+mark would be accepted. Since 0.1.3 the gem rejects that topology when either
+the worker or probe can read and compare both namespace links. With Kubernetes'
+default capability set those reads failed on the measured node: the shared-PID
+pod was `unverifiable`, and the host-PID pod was `unavailable` because node init
+reported a zero start time. Those states warn and retain the documented degraded
+checks; they do not claim the topology is safe. See
+[LIMITATIONS.md](LIMITATIONS.md#container-generations-need-linux-procfs-and-a-private-pid-namespace).
 
 ## Why the command looks like that
 
@@ -121,9 +129,12 @@ which is not something to rely on in a manifest.
 
 Keep the worker and probe on the same gem version. Since 0.1.2 the heartbeat
 protocol includes a container generation; a newer probe correctly rejects the
-generation-less files written by an older worker. `bundle exec` guarantees that
-both sides resolve from the same bundle. If the faster form below installs a
-second copy in `GEM_HOME`, rebuild that copy on every gem upgrade too.
+generation-less files written by an older worker. Since 0.1.3 it also includes
+the fail-closed topology state, which only a 0.1.3 probe reads: an older probe
+beside a 0.1.3 worker would ignore a proven unsupported PID topology instead of
+failing on it. `bundle exec` guarantees that both sides resolve from the same
+bundle. If the faster form below installs a second copy in `GEM_HOME`, rebuild
+that copy on every gem upgrade too.
 
 ### Where your image puts its gems
 
@@ -134,11 +145,17 @@ image. One line, no cluster needed:
 docker run --rm your-image ruby -e "require 'kicks_liveness/probe'"
 ```
 
-A line like `no /opt/app/tmp/health/expected: worker has not started yet` and
-exit 1 means the plain command works — the probe found itself and correctly
-reported that no worker is running. A `LoadError` means it does not: keep
-`bundle exec kicks-liveness`, or use the explicit-path optimisation below. Do
-not put the plain command in a manifest without this check: its failure mode is
+With no worker running, exit 1 and the following line mean the plain command
+works — the probe loaded and found no declared worker count:
+
+```
+no /opt/app/tmp/health/expected: worker has not started yet (is KicksLiveness.install! called before the runner starts? outside Rails it is not automatic)
+```
+
+The question is a [setup hint](SETUP.md#checking-that-the-hooks-are-in-place),
+not proof of missing installation. A `LoadError` means the command does not
+work: keep `bundle exec kicks-liveness`, or use the explicit-path optimisation
+below. Do not put the plain command in a manifest without this check: its failure mode is
 a pod that never passes `startupProbe` and lands in `CrashLoopBackOff`.
 
 What decides it is whether Bundler installed into `GEM_HOME` or somewhere of
@@ -305,6 +322,15 @@ several consumers per worker these add up. If the grace period is shorter than
 that, the container is SIGKILLed mid-shutdown and in-flight messages are
 redelivered.
 
+Set `KICKS_LIVENESS_SHUTDOWN_GRACE` (or `config.shutdown_grace`) to at least
+`terminationGracePeriodSeconds`; both are 60 seconds in the example. The
+monitor keeps the mark fresh only within that shutdown grace. For a SIGTERM
+sent outside pod deletion, a stuck drain stops publishing at the deadline;
+its last mark then has to exceed `max_age`, followed by the liveness probe's
+period and failure threshold. Under normal clock progression, restart
+eligibility is bounded by `shutdown_grace + max_age`, with the probe budget
+additional. During pod deletion the kubelet still owns the termination deadline.
+
 ## Environment variables
 
 | Variable | Default | Purpose | Invalid or blank value |
@@ -312,6 +338,7 @@ redelivered.
 | `KICKS_LIVENESS_DIR` | `/opt/app/tmp/health` | marks directory, **must be on tmpfs** | uses the default |
 | `KICKS_LIVENESS_MAX_AGE` | `45` | seconds after which a mark is stale | uses the default |
 | `KICKS_LIVENESS_TICK` | `10` | interval between ticks | uses the default, then the safety fallback below if needed |
+| `KICKS_LIVENESS_SHUTDOWN_GRACE` | `60` | seconds to keep publishing during shutdown; at least `terminationGracePeriodSeconds` | uses the default |
 
 Keep `tick` well below `max_age`. A tick longer than half of `max_age` leaves no
 room for a single missed write, and a tick longer than `max_age` guarantees a

data/docs/LIMITATIONS.md

@@ -39,25 +39,17 @@ aware that this probe no longer covers that failure mode.
 
 ## Broker unavailability is invisible by design
 
-While Bunny is recovering from a network failure, the probe reports healthy. This
-is deliberate: the alternative is that a ten-second broker hiccup restarts every
-replica simultaneously and reconnects them all at once, which is what a
-struggling broker least needs. Bunny reconnects and re-subscribes its consumers
-by itself, so a restart at that moment cures nothing.
-
-The consequence, plainly: a pod whose broker is unreachable looks healthy to
-this probe. Cover it with the alert.
+The probe reports healthy while Bunny recovers an established connection and
+re-subscribes consumers. This avoids restarting every replica during a broker
+outage, but also means an unreachable broker is invisible to this probe. Cover
+that case with the external consumer alert.
 
 ## The recovery exemption covers an established connection, not a failed subscribe
 
-While Bunny is recovering from a network failure the probe reports healthy, so
-that a broker hiccup does not restart every replica at once. That exemption is
-asked of a `connection` object, which has to exist before it can be asked.
-
-If the broker is unreachable at the moment a worker subscribes, there is no
-connection at all: the registry stays empty and no mark is ever written. So the
-cascade the exemption avoids for a live connection is still possible at startup
-— broker down, every replica restarting.
+The exemption requires a connection object. If the initial subscription fails,
+the worker never registers and no mark is written. A broker outage at startup
+can therefore restart every replica even though an established connection would
+be exempt.
 
 The budget here is the **startup** one, not the steady-state one:
 
@@ -78,6 +70,39 @@ flapping pod rather than a storm. And the supervisor respawning failing forks
 inside the container — which is *not* throttled by anything — is now reported:
 see the respawn escalation below.
 
+## Graceful shutdown has a finite heartbeat exemption
+
+`Sneakers::WorkerGroup#stop` removes consumers before waiting for their thread
+pools to drain. `Registry.stopping!` therefore suspends the consumer predicate
+before the first worker stops, allowing an orderly drain to keep its heartbeat
+fresh without reporting a consumer failure on every deploy.
+
+The exemption lasts for `shutdown_grace` seconds from the first stop callback
+(default 60). Set `KICKS_LIVENESS_SHUTDOWN_GRACE` or `config.shutdown_grace` to at
+least the deployment's `terminationGracePeriodSeconds`. During pod deletion the
+kubelet still owns the outer termination deadline. The gem cannot determine
+whether a SIGTERM came from the kubelet, an operator, or another process.
+
+At the deadline, the monitor logs one ERROR and stops refreshing the slot mark.
+It keeps declaring the expected fork count, so an existing mark eventually
+fails as `worker-<slot> stale`, rather than claiming that the worker never
+started. The monitor does not kill the process or resume checking consumers.
+This bounds the formerly indefinite healthy result when an out-of-band SIGTERM
+leaves a process stuck draining with no consumers.
+
+Expiry is not an immediate probe failure: the last mark must still exceed
+`max_age`. Under normal clock progression, the bound from stop notification to
+restart eligibility is `shutdown_grace + max_age`; actual restart also depends
+on the probe period and failure threshold. The first stop time and deadline are
+process-local, so repeated stop callbacks do not renew them and a sibling
+replacement fork starts without them.
+
+The deadline uses the monotonic clock. A wall-clock jump cannot shorten or
+extend the exemption, but it can change mark staleness: the probe still compares
+filesystem mtime with wall time. Keep the external consumer alert enabled; the
+shutdown grace and the probe budget deliberately allow a period with no
+consumers before a restart.
+
 ## Changing the worker count at runtime is not supported
 
 `ServerEngine` can re-read its configuration on SIGHUP and scale the fork set,
@@ -92,65 +117,133 @@ count and the retired slots eventually go stale. Neither direction is safe.
 
 If you change `workers`, restart the runner. Runtime scaling is not supported.
 
-## Container generations require Linux procfs and container-owned PID 1
+## Container generations need Linux procfs and a private PID namespace
 
 Kubernetes keeps an `emptyDir` across restarts of a container in the same pod.
-To prevent the next container from inheriting a fresh heartbeat, the worker and
-exec probe independently identify their shared incarnation from Linux procfs:
-the mount namespace plus PID 1 start time. No application cache is removed.
-
-The identifier also assumes that the container owns PID 1. With
-`shareProcessNamespace: true`, PID 1 is the pod sandbox, and with
-`hostPID: true` it is the node's init process; neither restarts when the worker
-container does, so the start-time half of the identifier stays constant. The
-mount namespace inode is then the only remaining signal, and the kernel
-normally hands the just-released inode back to the replacement container in an
-otherwise quiet pod. Under either setting the guard silently degrades to the
-0.1.1 freshness-only behavior and can accept a fresh mark from the previous
-container. Do not enable either setting on a pod whose `startupProbe` relies on
-this guarantee.
-
-If procfs is unavailable — for example, when using the gem outside a Linux
-container — generation detection falls back to the original freshness-only
-check. The worker and probe still function, but they cannot distinguish a fresh
-mark left by a previous process from one written by the current process. Linux
-Kubernetes, Docker, and Nomad containers expose the required procfs entries
-under their normal configuration.
+To stop the next container inheriting a fresh heartbeat, the worker and its exec
+probe independently identify their shared incarnation from Linux procfs: the
+mount namespace plus PID 1 start time. No application cache is removed.
+
+That is an incarnation only where the container owns PID 1. With
+`shareProcessNamespace: true` PID 1 is the pod sandbox; with `hostPID: true` it
+is the node's init process. Neither restarts when the worker container does.
+If the replacement container also receives the same mount namespace inode,
+the complete identifier repeats and an inherited fresh mark can satisfy the
+0.1.2 startup probe.
+
+Rejection in 0.1.3 depends on procfs access. On the measured Kubernetes node,
+default capabilities left shared PID `unverifiable` and host PID `unavailable`;
+node init reported start time zero. Adding `CAP_SYS_PTRACE` made shared PID
+provably `unsafe`. This is not a recommendation to add the capability.
+The full restart reproduction and Kubernetes measurement are in
+[VERIFYING.md](VERIFYING.md#12-shared-and-host-pid-topology-outcomes).
+
+### Proven unsafe: the probe fails, and keeps failing
+
+Both links readable and different is proof that PID 1 belongs to something
+outside this container. The probe exits 1 with
+
+```
+unsupported PID topology: pod must not set shareProcessNamespace or hostPID
+```
+
+Procfs access can differ between the worker and probe. A worker that proves the
+topology unsafe publishes `topology-unsafe` instead of a heartbeat; a probe
+rejects either that file or its own unsafe result before checking marks.
+Inherited generations and another fork's marks cannot override the verdict.
+
+The worker logs the reason and recovery at ERROR once for the uninterrupted
+unsafe state, across forks and container restarts sharing the volume. Later
+containers in that pod therefore do not repeat the ERROR; read the verdict from
+the kubelet's `Unhealthy` event or `topology-unsafe`.
+
+This state is terminal for the marks volume. Neither PID setting can change on
+a live pod, and its `emptyDir` dies with it. **Recovery is recreating the pod
+without those settings.** Until then probes fail and the container restarts.
+Unreadable namespace links do not establish this verdict; they take a degraded
+branch below.
+
+### Unverifiable: the 0.1.2 guarantee, and one warning
+
+Reading `/proc/1/ns/mnt` is a ptrace-level operation and may be denied, including
+when PID 1 runs as a different user. Denial cannot distinguish a valid private
+PID namespace from an unsupported topology. If the other procfs entries yield a
+generation, the probe retains its 0.1.2 check and output; the worker logs:
+
+```
+WARN [liveness] slot 0: PID topology unverified (cannot read /proc/1/ns/mnt (Errno::EACCES)):
+the container generation still scopes the heartbeat, but shareProcessNamespace and hostPID
+cannot be ruled out
+```
+
+Read this as unverifiable, not safe: an unsupported topology still has the
+0.1.2 inherited-heartbeat weakness. The WARN is once per derived generation,
+across fork respawns. The measured Kubernetes shared-PID case reached this state;
+its credentials and capability comparison are recorded in VERIFYING.md above.
+
+There is a narrower startup window when the worker and probe have asymmetric
+access: the worker can prove the topology unsafe, but the probe cannot. Before
+that worker reaches the monitor and publishes `topology-unsafe`, the probe can
+accept inherited marks only if the volume already contains a fresh heartbeat
+from an older worker, or from one that also could not prove the topology. Both
+conditions are required. A normal Kubernetes exec probe inherits the
+container's credentials and reaches the same procfs verdict itself; after a
+0.1.3 worker has proved the topology, `topology-unsafe` also survives every
+later restart in the same documented `emptyDir`. The residual window therefore
+requires an in-place restart that also changes the image or effective access,
+or a marks volume that outlives the pod contrary to this setup. In that case the
+probe retains the explicitly degraded 0.1.2 guarantee until the current worker
+publishes its verdict.
+
+### No generation at all
+
+If `/proc/self/ns/mnt` or `/proc/1/stat` cannot yield a generation, the probe
+falls back to the 0.1.1 freshness-only check and may accept fresh inherited
+marks. The worker warns once for the uninterrupted unavailable state. A verified
+topology clears the report flag so a later degradation is reported again:
+
+```
+WARN [liveness] slot 0: no container generation (cannot read /proc/1/stat (Errno::EACCES)):
+falling back to freshness only, which cannot reject a heartbeat inherited from a previous
+container
+```
+
+A zero start time uses the detail `PID 1 start time is zero; possible hostPID or
+virtualized procfs` in that warning. The field counts clock ticks after boot,
+so zero is valid, not a parse error. It was observed for node init in a host-PID
+pod, but may also be exposed for a legitimate PID 1 by virtualized procfs.
+Without measurements of those runtimes, zero alone remains `unavailable`.
+
+Off Linux the same freshness-only fallback is **silent**, an intentional
+compatibility mode. Degraded Linux warnings also stay off the probe's stdout:
+they belong in the worker log, not repeated beside unrelated failures in
+`Unhealthy` events. A proven unsafe topology is different: it is the probe's
+verdict.
+
+### The probe must run in the worker's container
 
 The worker and its exec probe must run in the **same container**. A neighbouring
-sidecar can mount the same `emptyDir`, but it has a different mount namespace
-and therefore treats the worker's marks as belonging to another container.
+sidecar can mount the same `emptyDir`, but it has a different mount namespace and
+therefore treats the worker's marks as belonging to another container.
 Kubernetes exec probes already run inside the container they check; do not move
 `kicks-liveness` into a separate health sidecar.
 
 ## A respawn loop is reported once per grace window, not once per respawn
 
-When a fork cannot subscribe at all, the supervisor brings it back after
-`start_worker_delay` — a fraction of a second — and the monitor in each
-incarnation is a brand-new object. Any counter it holds is destroyed before it
-can reach a threshold, which is how a real outage used to produce thousands of
-identical INFO lines and not a single ERROR.
-
-The count is therefore kept in the marks directory, which survives the fork.
-The start line is logged once, repeated starts within the grace window
-(`startup_grace_ticks × tick`) are silent, and after that one ERROR per window
-reports the elapsed time and the number of starts:
+The supervisor respawns a fork that cannot subscribe after `start_worker_delay`,
+potentially before its monitor reaches startup grace. `attempt-<slot>` therefore
+keeps the count and window start outside process memory. The first incarnation
+logs startup, further starts within `startup_grace_ticks × tick` are silent, and
+each expired window produces one ERROR with elapsed time and cumulative starts:
 
 ```
 [liveness] slot 0: not healthy 62s over 305 starts, expected 5 consumers
 ```
 
-The number of starts is the diagnosis: it separates a slow start from a respawn
-loop at a glance.
-
-This unhealthy-run counter is scoped to the marks volume, not to the container
-generation. If a container is restarted before its slot has ever become
-healthy, the replacement continues the same count: its repeated `started:` line
-is suppressed, and an expired grace window may immediately report an ERROR that
-includes starts from the previous container. This is diagnostic state only; it
-does not participate in the probe result, and the first healthy tick removes
-it. Treat the count as "starts since this slot was last healthy in this pod",
-not "starts in this container".
+The generation in the file rejects an earlier container's count. A replacement
+logs its own `started:` and cannot escalate over starts it did not take; the
+first healthy tick clears the counter. Without a generation, the old volume-wide
+scope remains: read it as starts since this slot was last healthy in the pod.
 
 ## Do not install both `kicks` and `sneakers`
 

data/docs/SETUP.md

@@ -148,10 +148,19 @@ config.logger = Rails.logger
 | `enabled` | `true` | start the monitor thread | `true` or `false` |
 | `tick` | `10` | seconds between checks | positive number smaller than `max_age` |
 | `startup_grace_ticks` | `6` | unhealthy startup ticks before one ERROR | positive integer |
+| `shutdown_grace` | `60` | seconds to keep publishing during shutdown | positive finite number |
 
 Invalid combinations are rejected before the monitor starts instead of running
 one that is guaranteed to publish stale marks or never report a stalled startup.
 
+`shutdown_grace` defaults from `KICKS_LIVENESS_SHUTDOWN_GRACE`; the initializer
+wins when both are set. Set it to at least your Kubernetes
+`terminationGracePeriodSeconds`. It bounds the shutdown exemption, not the
+restart time: after expiry the last mark still has to exceed `max_age` and the
+probe's failure budget still applies. The environment accepts positive integer
+seconds and silently uses 60 for invalid or blank values. Invalid initializer
+values raise `ArgumentError`; fractional positive seconds are accepted.
+
 `dir` and `max_age` are **not** here — they come from environment variables
 only. See
 [DESIGN.md](DESIGN.md#what-is-configurable-and-where) for why, and
@@ -169,6 +178,16 @@ Sneakers::WorkerGroup.ancestors.include?(KicksLiveness::Hooks::WorkerGroup) # =>
 Both must be `true` before the runner starts. If either is `false`, either
 `install!` has not run, or it ran before `kicks`/`sneakers` was loaded.
 
+When `expected` is missing, the probe asks whether `KicksLiveness.install!` was
+called before the runner started. This is a hint, not a diagnosis: a slow boot
+or a removed marks directory can produce the same result. It does not install
+hooks or change the failing exit status. The complete output is shown in the
+[image check](KUBERNETES.md#where-your-image-puts-its-gems).
+
+When migrating from another probe, confirm the `[liveness] slot 0: started:`
+line and the `expected` file on dev before removing the old probe gem, its
+initializer and its rake tasks.
+
 Once workers are running, the marks directory is the other half of the answer:
 
 ```
@@ -184,3 +203,14 @@ One `worker-<slot>` file per fork, plus `expected` and the current container
 have not finished subscribing. A slot that keeps restarting without ever
 subscribing also leaves an `attempt-<slot>` file, which is removed as soon as
 that slot becomes healthy.
+
+The `generation` file is authoritative only while procfs generation detection is
+available: where it is not, the worker writes no generation and a file left by
+an earlier container stays visible, so `kubectl exec ... cat <dir>/generation`
+can show a stale incarnation. It is not deleted in that state on purpose —
+removing it could fail a concurrently running probe that does have a valid
+generation, for a reason that has nothing to do with the worker's health. Two
+further files appear only when something is wrong: `topology-unsafe`, which
+fails every probe until the pod is recreated, and `topology-report`, which is
+how the worker keeps what it says about the topology to one line. Both are explained in
+[LIMITATIONS.md](LIMITATIONS.md#container-generations-need-linux-procfs-and-a-private-pid-namespace).