kicks_liveness 0.1.2 → 0.1.3

data/docs/LIMITATIONS.md

@@ -39,25 +39,17 @@ aware that this probe no longer covers that failure mode.
 
 ## Broker unavailability is invisible by design
 
-While Bunny is recovering from a network failure, the probe reports healthy. This
-is deliberate: the alternative is that a ten-second broker hiccup restarts every
-replica simultaneously and reconnects them all at once, which is what a
-struggling broker least needs. Bunny reconnects and re-subscribes its consumers
-by itself, so a restart at that moment cures nothing.
-
-The consequence, plainly: a pod whose broker is unreachable looks healthy to
-this probe. Cover it with the alert.
+The probe reports healthy while Bunny recovers an established connection and
+re-subscribes consumers. This avoids restarting every replica during a broker
+outage, but also means an unreachable broker is invisible to this probe. Cover
+that case with the external consumer alert.
 
 ## The recovery exemption covers an established connection, not a failed subscribe
 
-While Bunny is recovering from a network failure the probe reports healthy, so
-that a broker hiccup does not restart every replica at once. That exemption is
-asked of a `connection` object, which has to exist before it can be asked.
-
-If the broker is unreachable at the moment a worker subscribes, there is no
-connection at all: the registry stays empty and no mark is ever written. So the
-cascade the exemption avoids for a live connection is still possible at startup
-— broker down, every replica restarting.
+The exemption requires a connection object. If the initial subscription fails,
+the worker never registers and no mark is written. A broker outage at startup
+can therefore restart every replica even though an established connection would
+be exempt.
 
 The budget here is the **startup** one, not the steady-state one:
 
@@ -78,6 +70,30 @@ flapping pod rather than a storm. And the supervisor respawning failing forks
 inside the container — which is *not* throttled by anything — is now reported:
 see the respawn escalation below.
 
+## Graceful shutdown is safe only when something else owns the deadline
+
+`Registry.stopping!` suspends the consumer predicate before `WorkerGroup#stop`
+unsubscribes workers and drains their pools. The monitor keeps the heartbeat
+fresh for the rest of the process so liveness does not interrupt an in-flight
+job. During pod deletion the kubelet enforces `terminationGracePeriodSeconds`,
+which bounds this exemption externally.
+
+A SIGTERM from an operator, sidecar, or internal supervisor enters the same
+path without deleting the pod. If Sneakers or Bunny then drains or reconnects
+indefinitely, the heartbeat stays fresh with zero consumers and the pod can
+remain Ready forever. The process cannot infer an external deadline from the
+signal.
+
+This is existing shutdown behaviour, not a 0.1.3 regression. Until the exemption
+is bounded — planned for 0.1.4 — send SIGTERM only through a termination path
+that has its own enforced deadline. Use pod deletion for Kubernetes operations.
+An internal supervisor must eventually make PID 1 exit itself, or ask the
+container runtime to stop it, if graceful shutdown does not complete; an
+in-namespace `kill -9 1` is not a reliable escalation even when `kill` reports
+success. Keep the external consumer alert enabled as well: it still sees the
+zero-consumer state that the process-local predicate has intentionally stopped
+evaluating.
+
 ## Changing the worker count at runtime is not supported
 
 `ServerEngine` can re-read its configuration on SIGHUP and scale the fork set,
@@ -92,65 +108,133 @@ count and the retired slots eventually go stale. Neither direction is safe.
 
 If you change `workers`, restart the runner. Runtime scaling is not supported.
 
-## Container generations require Linux procfs and container-owned PID 1
+## Container generations need Linux procfs and a private PID namespace
 
 Kubernetes keeps an `emptyDir` across restarts of a container in the same pod.
-To prevent the next container from inheriting a fresh heartbeat, the worker and
-exec probe independently identify their shared incarnation from Linux procfs:
-the mount namespace plus PID 1 start time. No application cache is removed.
-
-The identifier also assumes that the container owns PID 1. With
-`shareProcessNamespace: true`, PID 1 is the pod sandbox, and with
-`hostPID: true` it is the node's init process; neither restarts when the worker
-container does, so the start-time half of the identifier stays constant. The
-mount namespace inode is then the only remaining signal, and the kernel
-normally hands the just-released inode back to the replacement container in an
-otherwise quiet pod. Under either setting the guard silently degrades to the
-0.1.1 freshness-only behavior and can accept a fresh mark from the previous
-container. Do not enable either setting on a pod whose `startupProbe` relies on
-this guarantee.
-
-If procfs is unavailable — for example, when using the gem outside a Linux
-container — generation detection falls back to the original freshness-only
-check. The worker and probe still function, but they cannot distinguish a fresh
-mark left by a previous process from one written by the current process. Linux
-Kubernetes, Docker, and Nomad containers expose the required procfs entries
-under their normal configuration.
+To stop the next container inheriting a fresh heartbeat, the worker and its exec
+probe independently identify their shared incarnation from Linux procfs: the
+mount namespace plus PID 1 start time. No application cache is removed.
+
+That is an incarnation only where the container owns PID 1. With
+`shareProcessNamespace: true` PID 1 is the pod sandbox; with `hostPID: true` it
+is the node's init process. Neither restarts when the worker container does.
+If the replacement container also receives the same mount namespace inode,
+the complete identifier repeats and an inherited fresh mark can satisfy the
+0.1.2 startup probe.
+
+Rejection in 0.1.3 depends on procfs access. On the measured Kubernetes node,
+default capabilities left shared PID `unverifiable` and host PID `unavailable`;
+node init reported start time zero. Adding `CAP_SYS_PTRACE` made shared PID
+provably `unsafe`. This is not a recommendation to add the capability.
+The full restart reproduction and Kubernetes measurement are in
+[VERIFYING.md](VERIFYING.md#12-shared-and-host-pid-topology-outcomes).
+
+### Proven unsafe: the probe fails, and keeps failing
+
+Both links readable and different is proof that PID 1 belongs to something
+outside this container. The probe exits 1 with
+
+```
+unsupported PID topology: pod must not set shareProcessNamespace or hostPID
+```
+
+Procfs access can differ between the worker and probe. A worker that proves the
+topology unsafe publishes `topology-unsafe` instead of a heartbeat; a probe
+rejects either that file or its own unsafe result before checking marks.
+Inherited generations and another fork's marks cannot override the verdict.
+
+The worker logs the reason and recovery at ERROR once for the uninterrupted
+unsafe state, across forks and container restarts sharing the volume. Later
+containers in that pod therefore do not repeat the ERROR; read the verdict from
+the kubelet's `Unhealthy` event or `topology-unsafe`.
+
+This state is terminal for the marks volume. Neither PID setting can change on
+a live pod, and its `emptyDir` dies with it. **Recovery is recreating the pod
+without those settings.** Until then probes fail and the container restarts.
+Unreadable namespace links do not establish this verdict; they take a degraded
+branch below.
+
+### Unverifiable: the 0.1.2 guarantee, and one warning
+
+Reading `/proc/1/ns/mnt` is a ptrace-level operation and may be denied, including
+when PID 1 runs as a different user. Denial cannot distinguish a valid private
+PID namespace from an unsupported topology. If the other procfs entries yield a
+generation, the probe retains its 0.1.2 check and output; the worker logs:
+
+```
+WARN [liveness] slot 0: PID topology unverified (cannot read /proc/1/ns/mnt (Errno::EACCES)):
+the container generation still scopes the heartbeat, but shareProcessNamespace and hostPID
+cannot be ruled out
+```
+
+Read this as unverifiable, not safe: an unsupported topology still has the
+0.1.2 inherited-heartbeat weakness. The WARN is once per derived generation,
+across fork respawns. The measured Kubernetes shared-PID case reached this state;
+its credentials and capability comparison are recorded in VERIFYING.md above.
+
+There is a narrower startup window when the worker and probe have asymmetric
+access: the worker can prove the topology unsafe, but the probe cannot. Before
+that worker reaches the monitor and publishes `topology-unsafe`, the probe can
+accept inherited marks only if the volume already contains a fresh heartbeat
+from an older worker, or from one that also could not prove the topology. Both
+conditions are required. A normal Kubernetes exec probe inherits the
+container's credentials and reaches the same procfs verdict itself; after a
+0.1.3 worker has proved the topology, `topology-unsafe` also survives every
+later restart in the same documented `emptyDir`. The residual window therefore
+requires an in-place restart that also changes the image or effective access,
+or a marks volume that outlives the pod contrary to this setup. In that case the
+probe retains the explicitly degraded 0.1.2 guarantee until the current worker
+publishes its verdict.
+
+### No generation at all
+
+If `/proc/self/ns/mnt` or `/proc/1/stat` cannot yield a generation, the probe
+falls back to the 0.1.1 freshness-only check and may accept fresh inherited
+marks. The worker warns once for the uninterrupted unavailable state. A verified
+topology clears the report flag so a later degradation is reported again:
+
+```
+WARN [liveness] slot 0: no container generation (cannot read /proc/1/stat (Errno::EACCES)):
+falling back to freshness only, which cannot reject a heartbeat inherited from a previous
+container
+```
+
+A zero start time uses the detail `PID 1 start time is zero; possible hostPID or
+virtualized procfs` in that warning. The field counts clock ticks after boot,
+so zero is valid, not a parse error. It was observed for node init in a host-PID
+pod, but may also be exposed for a legitimate PID 1 by virtualized procfs.
+Without measurements of those runtimes, zero alone remains `unavailable`.
+
+Off Linux the same freshness-only fallback is **silent**, an intentional
+compatibility mode. Degraded Linux warnings also stay off the probe's stdout:
+they belong in the worker log, not repeated beside unrelated failures in
+`Unhealthy` events. A proven unsafe topology is different: it is the probe's
+verdict.
+
+### The probe must run in the worker's container
 
 The worker and its exec probe must run in the **same container**. A neighbouring
-sidecar can mount the same `emptyDir`, but it has a different mount namespace
-and therefore treats the worker's marks as belonging to another container.
+sidecar can mount the same `emptyDir`, but it has a different mount namespace and
+therefore treats the worker's marks as belonging to another container.
 Kubernetes exec probes already run inside the container they check; do not move
 `kicks-liveness` into a separate health sidecar.
 
 ## A respawn loop is reported once per grace window, not once per respawn
 
-When a fork cannot subscribe at all, the supervisor brings it back after
-`start_worker_delay` — a fraction of a second — and the monitor in each
-incarnation is a brand-new object. Any counter it holds is destroyed before it
-can reach a threshold, which is how a real outage used to produce thousands of
-identical INFO lines and not a single ERROR.
-
-The count is therefore kept in the marks directory, which survives the fork.
-The start line is logged once, repeated starts within the grace window
-(`startup_grace_ticks × tick`) are silent, and after that one ERROR per window
-reports the elapsed time and the number of starts:
+The supervisor respawns a fork that cannot subscribe after `start_worker_delay`,
+potentially before its monitor reaches startup grace. `attempt-<slot>` therefore
+keeps the count and window start outside process memory. The first incarnation
+logs startup, further starts within `startup_grace_ticks × tick` are silent, and
+each expired window produces one ERROR with elapsed time and cumulative starts:
 
 ```
 [liveness] slot 0: not healthy 62s over 305 starts, expected 5 consumers
 ```
 
-The number of starts is the diagnosis: it separates a slow start from a respawn
-loop at a glance.
-
-This unhealthy-run counter is scoped to the marks volume, not to the container
-generation. If a container is restarted before its slot has ever become
-healthy, the replacement continues the same count: its repeated `started:` line
-is suppressed, and an expired grace window may immediately report an ERROR that
-includes starts from the previous container. This is diagnostic state only; it
-does not participate in the probe result, and the first healthy tick removes
-it. Treat the count as "starts since this slot was last healthy in this pod",
-not "starts in this container".
+The generation in the file rejects an earlier container's count. A replacement
+logs its own `started:` and cannot escalate over starts it did not take; the
+first healthy tick clears the counter. Without a generation, the old volume-wide
+scope remains: read it as starts since this slot was last healthy in the pod.
 
 ## Do not install both `kicks` and `sneakers`
 

data/docs/SETUP.md

@@ -184,3 +184,14 @@ One `worker-<slot>` file per fork, plus `expected` and the current container
 have not finished subscribing. A slot that keeps restarting without ever
 subscribing also leaves an `attempt-<slot>` file, which is removed as soon as
 that slot becomes healthy.
+
+The `generation` file is authoritative only while procfs generation detection is
+available: where it is not, the worker writes no generation and a file left by
+an earlier container stays visible, so `kubectl exec ... cat <dir>/generation`
+can show a stale incarnation. It is not deleted in that state on purpose —
+removing it could fail a concurrently running probe that does have a valid
+generation, for a reason that has nothing to do with the worker's health. Two
+further files appear only when something is wrong: `topology-unsafe`, which
+fails every probe until the pod is recreated, and `topology-report`, which is
+how the worker keeps what it says about the topology to one line. Both are explained in
+[LIMITATIONS.md](LIMITATIONS.md#container-generations-need-linux-procfs-and-a-private-pid-namespace).

data/docs/VERIFYING.md

@@ -35,6 +35,10 @@ spec/integration/verify.sh up      # namespace, broker, worker
 spec/integration/verify.sh down    # deletes the namespace
 ```
 
+Scenarios 11 and 12 are the exception: they need the image and Docker, but no
+cluster and no broker, because both are about the container itself rather than
+about consuming.
+
 Every `kubectl` call names its context and namespace explicitly, so the script
 never falls back to `current-context`. It will still operate on whichever target
 you configure: run `status` and inspect both values before `up` or `down`. The
@@ -101,11 +105,11 @@ expected  generation  worker-0            # in the marks directory
 1 process(es) healthy                     # probe, exit 0
 ```
 
-The hooks fired inside a real ServerEngine fork, which no double can show. On a
-fresh fixture the first attempt also prints `started:`. Do not use that line as
-the sole proof after a same-pod container restart: an inherited unhealthy-run
-counter deliberately suppresses repeated start lines, as described in
-[LIMITATIONS.md](LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn).
+The hooks fired inside a real ServerEngine fork, which no double can show. The
+`started:` line is also valid proof after a same-pod container restart: since
+0.1.3 the respawn counter carries the container generation, so a replacement
+container reports its own first start rather than continuing the previous
+container's unhealthy run.
 
 ### 2. A stale mark
 
@@ -362,10 +366,103 @@ The namespace inode is allowed to be reused, as in the example above; the PID 1
 start time still distinguishes the two incarnations. The test removes its
 container and named volume on exit.
 
+The command also prints the topology it ran under, which for an ordinary
+container is the fully verified one:
+
+```
+generation-restart: topology=verified
+```
+
 This scenario covers the default container-private PID namespace. With
 `shareProcessNamespace: true` or `hostPID: true`, PID 1 survives the worker
-container restart and this guarantee does not hold; see
-[LIMITATIONS.md](LIMITATIONS.md#container-generations-require-linux-procfs-and-container-owned-pid-1).
+container restart and this guarantee cannot hold — scenario 12 is that case,
+showing both fail-closed rejection when the topology can be proved and the
+degraded outcomes when procfs access cannot prove it.
+
+### 12. Shared and host PID topology outcomes
+
+The four PID topologies, in one command. It needs the fixture image but neither
+Kubernetes nor RabbitMQ, because everything here turns on which PID namespace
+the worker container is in:
+
+```bash
+spec/integration/verify.sh build
+spec/integration/verify.sh topology
+```
+
+Recorded on Linux 6.12 (arm64), Docker 27.4.1, ruby 3.4.10:
+
+```
+shared-pid: topology=unsafe
+shared-pid: 0.1.2 generation before=mnt:[4026532693]:17207878 after=mnt:[4026532693]:17207878
+shared-pid inherited heartbeat: exit=1 unsupported PID topology: pod must not set shareProcessNamespace or hostPID
+shared-pid: marks directory holds expected generation topology-unsafe worker-0
+shared-pid after the worker published: exit=1 unsupported PID topology: pod must not set shareProcessNamespace or hostPID
+host-pid: topology=unsafe
+host-pid: 0.1.2 generation before=mnt:[4026532699]:105 after=mnt:[4026532699]:105
+host-pid inherited heartbeat: exit=1 unsupported PID topology: pod must not set shareProcessNamespace or hostPID
+host-pid: marks directory holds expected generation topology-unsafe worker-0
+host-pid after the worker published: exit=1 unsupported PID topology: pod must not set shareProcessNamespace or hostPID
+private-pid: topology=verified
+private-pid: exit=0 1 process(es) healthy
+unverifiable: topology=unverifiable
+unverifiable: warnings across three fork respawns=1
+unverifiable inherited heartbeat: exit=1 heartbeat belongs to a previous container: worker has not started yet
+unverifiable own heartbeat: exit=0 1 process(es) healthy
+```
+
+Four cases, and each line is the point of one of them.
+
+A separate run on a real Kubernetes v1.25.0 node on Docker Desktop used
+containerd, no `securityContext`, a root worker and the default capability set.
+It produced a different and operationally important table:
+
+| pod topology | PID 1 | PID 1 UID | `/proc/1/ns/mnt` | start time | 0.1.3 state |
+|---|---|---:|---|---:|---|
+| `hostPID: true` | node init | 0 | denied | 0 | `unavailable` |
+| `shareProcessNamespace: true` | pod sandbox | 65535 | denied | 19031096 | `unverifiable` |
+| private PID | container process | 0 | readable | 19031243 | `verified` |
+| shared PID plus `CAP_SYS_PTRACE` | pod sandbox | 65535 | readable | — | `unsafe` |
+
+The Docker matrix proves the fail-closed branch by granting the access it needs;
+the Kubernetes measurement shows that this is not the default result for either
+unsupported setting. The degraded states still warn, but they retain the weaker
+generation or freshness-only guarantees described in LIMITATIONS.md.
+
+**`shared-pid`** joins another container's PID namespace, which is what
+Kubernetes `shareProcessNamespace: true` does — there PID 1 is the pod sandbox.
+**`host-pid`** uses the node's PID namespace, which is `hostPID: true`. In both,
+the run publishes a complete, fresh heartbeat in the **0.1.2 format**, restarts
+only the worker container, and recomputes the 0.1.2 identifier by the 0.1.2
+rule. The `before`/`after` pair is the defect being reproduced rather than
+argued about: both halves repeat, including the mount namespace inode, which the
+kernel handed straight back to the replacement container. A 0.1.2 probe would
+have accepted those inherited marks. The revised probe rejects the topology
+before it ever looks at them, and the worker then adds `topology-unsafe` beside
+the inherited files rather than a heartbeat of its own.
+
+Reading PID 1's namespace link is a ptrace-level operation, so the `host-pid`
+case runs with `CAP_SYS_PTRACE`; without it the link is unreadable and the
+outcome depends on whether a generation can still be derived. Node init on the
+measured Kubernetes node reported start time zero, so that pod was
+`unavailable`. A readable positive start time would instead leave the topology
+`unverifiable`. The fourth case below constructs that latter branch explicitly;
+both are honest limits on what a container can prove about a topology it is not
+allowed to inspect.
+
+**`private-pid`** is the fully verified positive control: an ordinary container,
+owning PID 1, probe green.
+
+**`unverifiable`** is the second positive control, and the one that has to stay
+green: a private-PID container whose checker cannot read `/proc/1/ns/mnt`. The
+0.1.2 generation check is kept in full — a heartbeat from another generation is
+still rejected, its own is still accepted — and the cost is exactly one worker
+warning across three fork respawns, not a probe that is permanently red.
+
+That case is constructed by running the checker as a user other than PID 1's.
+It is **not** a model of a Kubernetes exec probe, which has no user setting of
+its own and normally inherits the container process credentials; it is simply
+the one way to make a namespace link unreadable without a custom procfs mount.
 
 ## What to do with a disagreement
 

data/lib/kicks_liveness/attempts.rb

@@ -3,23 +3,18 @@ require 'fileutils'
 module KicksLiveness
   # Counts starts of a supervisor slot that have not reached healthy yet.
   #
-  # Deliberately separate from {Heartbeat}. Heartbeat is the contract shared
-  # with the probe and is loaded by it alone, so nothing the probe does not need
-  # belongs there — including the +fileutils+ this class requires freely.
+  # Persistent state survives fork respawns; its generation separates container
+  # incarnations when available. Kept outside {Heartbeat} because the probe does
+  # not read these counters.
   #
-  # The count lives in the marks directory rather than in memory because a
-  # respawn storm destroys process state every few hundred milliseconds: the
-  # supervisor brings a failing fork back after +start_worker_delay+, and every
-  # incarnation of the monitor believes it is the first. That is how an outage
-  # can produce thousands of identical INFO lines and not one ERROR. The
-  # directory is the only state that survives the fork.
-  #
-  # @see file:docs/DESIGN.md#logging-events-not-the-pulse
+  # @see file:docs/LIMITATIONS.md#a-respawn-loop-is-reported-once-per-grace-window-not-once-per-respawn
   # @api private
   class Attempts
     # @param dir [String] marks directory
-    def initialize(dir)
+    # @param generation [String, nil] container incarnation the count belongs to
+    def initialize(dir, generation: nil)
       @dir = dir
+      @generation = generation
     end
 
     # Records a start of this slot.
@@ -62,22 +57,32 @@ module KicksLiveness
       File.join(@dir, "attempt-#{slot}")
     end
 
-    # A missing or unreadable counter means "no previous start", never an
-    # exception: this is instrumentation, and it may not break a worker.
+    # Missing, unreadable, or foreign-generation counts start a new run.
+    # Counter failures must not prevent worker startup.
     def read(slot)
-      count, epoch = File.read(path(slot)).split
+      count, epoch, generation = File.read(path(slot)).split
+      return [0, nil] unless same_generation?(generation)
+
       [Integer(count), Time.at(Integer(epoch)).utc]
     rescue StandardError
       [0, nil]
     end
 
+    def same_generation?(declared)
+      return true unless @generation
+
+      declared == "generation=#{@generation}"
+    end
+
     # Failing to write degrades the reporting back to one line per respawn,
     # which is noisy but harmless — so a read-only directory is swallowed here
     # rather than escalated.
     def write(slot, count, first)
       FileUtils.mkdir_p(@dir)
+      contents = "#{count} #{first.to_i}"
+      contents = "#{contents} generation=#{@generation}" if @generation
       tmp = "#{path(slot)}.#{Process.pid}"
-      File.write(tmp, "#{count} #{first.to_i}\n")
+      File.write(tmp, "#{contents}\n")
       File.rename(tmp, path(slot))
     rescue StandardError
       nil

data/lib/kicks_liveness/configuration.rb

@@ -3,11 +3,8 @@ require_relative 'heartbeat'
 module KicksLiveness
   # What the application may configure, and what comes from the environment only.
   #
-  # +dir+ and +max_age+ are deliberately read-only here: the probe runs as a
-  # separate process launched by the container runtime and cannot read the
-  # application's initializer. Were they settable in both places, a mismatch
-  # between what the worker writes and what the probe expects would pass
-  # unnoticed.
+  # dir and max_age are environment-only: the separate exec probe cannot read
+  # an application initializer and must resolve the same values as the worker.
   #
   # @see file:docs/SETUP.md#configuration
   # @see file:docs/DESIGN.md#what-is-configurable-and-where
@@ -44,10 +41,8 @@ module KicksLiveness
       @enabled
     end
 
-    # A tick is what the monitor thread sleeps on, so a non-positive value is
-    # not a setting but a broken monitor. An initializer is code, and code
-    # should fail loudly at boot, where the developer is looking. Environment
-    # input is resolved separately by {#environment_tick}, with a safe fallback.
+    # Initializer values fail at boot; environment input uses a safe fallback
+    # through {#environment_tick}.
     #
     # @param seconds [Numeric]
     # @raise [ArgumentError] if not a positive number