rails_pod_kit 0.2.0 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 85fea66a44c5b2198b5c3c7bfd05f5f1f4c3655559ed701748661d0e148650a5
-  data.tar.gz: 9c921528f839ea1d5b962035600316ea37affe17cf8fdc910453c30108a248dd
+  metadata.gz: a40b144ace0af8e73c9a44f21ebb7111e726543a343be3c8f8969ee7befb166c
+  data.tar.gz: 5ec92a058a63671e877011591d2a5585a98a9da5c851db86ea5924814cbed32a
 SHA512:
-  metadata.gz: d7fbfee31ce338a6dd7c11d65214dc442cd2cb261848ae2403a415dc4a6e1333fa55b39e2d06441b4e05fcacff2ec3971ee009613a532f384fcc19fbe84f070d
-  data.tar.gz: 894f99f9fba4596aa7d69e143f5fa23d2ebc64ef85f6b7cac310096c902005dd003103cb12c3933e49dfe26263c68f9656cfc3e68c0d9551a8f118e79496fd48
+  metadata.gz: fca2fbe470f2eaa9c8039034d8795ca9006ea10e835d8db907c20aa3a1740eeffa2bc94a2d31bc024130028f15e4340e4c59dcc3fc330cbb34d12b649bd1f4b1
+  data.tar.gz: 04e115d689e42b62975846ca67a5312ba7de26fbec65ca970864dbc844131dda53188839ff05f1a81601aeaa652bd42e1f17682d8d22cc08203322b5072bf1b7

data/README.md

@@ -159,6 +159,8 @@ endpoint every few seconds); pass `silence_controller_log: false` to keep it.
   `sidekiq_queue_latency`, `sidekiq_active_processes`,
   `sidekiq_active_workers_count`, `sidekiq_jobs_retry_count`,
   `sidekiq_jobs_dead_count`, `sidekiq_jobs_scheduled_count`.
+- **Sidekiq (cron scheduler):** `sidekiq_cron_poll_age_seconds`, on the process
+  hosting the poller (see GlobalScheduler).
 - **SolidQueue (DB-wide):** `solid_queue_backlog`,
   `solid_queue_latency_seconds`.
 
@@ -276,6 +278,12 @@ Sidekiq job class.
 | `sidekiq.active_processes` | — |
 | `sidekiq.active_workers_count` | — |
 
+**Sidekiq — cron scheduler** (only the process hosting the poller, `namespace: sidekiq`):
+
+| canonical Datadog metric | functional tags |
+|---|---|
+| `sidekiq.cron_poll_age_seconds` | — |
+
 **Sidekiq — per-process / job** (worker pod, `namespace: sidekiq`; emitted on job activity):
 
 | canonical Datadog metric | type | functional tags |
@@ -403,6 +411,32 @@ silently. The cron poller's own loop swallows StandardError, so a Redis blip
 costs one skipped tick; the supervisor makes anything it does *not* catch a
 skipped tick too.
 
+### The heartbeat
+
+The supervisor covers a poller thread that *dies*. It cannot see one that is
+running and no longer enqueueing — which from the outside is indistinguishable
+from an idle one: every gauge stays fresh, `/metrics` answers 200, the process
+looks healthy. On the only process carrying the schedule that is a silently
+stopped schedule, i.e. the very failure hosting the poller here was meant to
+eliminate, back through another door.
+
+So `start!` also publishes **`sidekiq_cron_poll_age_seconds`**: seconds since the
+poller last completed a tick. It measures the loop turning, so it stays flat on a
+healthy but idle schedule and climbs the moment ticks stop — the one shape an
+alert can be written against:
+
+```
+max:sidekiq.cron_poll_age_seconds{…} > 10 * <poll interval>
+```
+
+Before the first tick it measures from `start!`, so a poller that never manages
+one reads as climbing rather than as no-data; it is `nil` (and the series absent)
+on any process that hosts no poller, and again once `stop!` runs.
+
+Deliberately **not** "time since last enqueue", which would climb on any quiet
+schedule and so alert on nothing happening. Answering *should this job have run
+by now?* needs a per-job check against the cron expression, not a gauge.
+
 > **Every schedule entry must declare `active_job: true`.** This process has no
 > Rails, so it cannot resolve the job classes; sidekiq-cron then falls back to
 > pushing a raw message, and only that flag makes the message an ActiveJob
@@ -577,7 +611,7 @@ serves `solid_queue_*` and nothing else, so the check config needs no filters.
   localhost-only socket (`no_token: true`, never network-exposed).
 - **Rack version.** Under **Rack 3+** the mmap exporter's WEBrick handler also
   needs the `rackup` gem. Under Rack 2.x `webrick` alone is enough, but on
-  Ruby ≥ 3.5 make sure `ostruct` is in the bundle (Rack 2.2 requires it
+  Ruby ≥ 4.0 make sure `ostruct` is in the bundle (Rack 2.2 requires it
   without declaring it, and it's no longer a default gem).
 - **Two kinds of entry point.** The main file (`require 'rails_pod_kit'`,
   what Bundler.require loads in a Rails app) pulls in every integration

data/VERSION

@@ -1 +1 @@
-0.2.0
+0.3.1

data/lib/rails_pod_kit/global_scheduler/heartbeat.rb

@@ -0,0 +1,89 @@
+# frozen_string_literal: true
+
+module RailsPodKit
+  module GlobalScheduler
+    # The one failure the rest of the machinery cannot see.
+    #
+    # A poller thread that dies is restarted by the Supervisor, and a process
+    # that stops serving is caught by the exporter's own probe. But a poller
+    # that is *running and no longer enqueueing* is indistinguishable from an
+    # idle one from the outside: every gauge stays fresh, /metrics answers 200,
+    # the pod is Running and Ready. On the only process carrying the schedule
+    # that is a silently stopped schedule — exactly the failure hosting the
+    # poller here was meant to eliminate, coming back through another door.
+    #
+    # So publish the age of the last completed tick, as
+    # `sidekiq_cron_poll_age_seconds`. It measures the loop turning, which means
+    # it stays flat on a healthy but idle schedule and climbs the moment ticks
+    # stop — the one shape an alert can be written against.
+    #
+    # Deliberately *not* "time since last enqueue": that climbs on any quiet
+    # schedule, so it would alert on nothing happening. Answering "should this
+    # job have run by now?" needs a per-job check against the cron expression,
+    # not a gauge.
+    module Heartbeat
+      module_function
+
+      # Declares the gauge. One-shot, and safe either side of
+      # `Yabeda.configure!` — a metric declared after it is registered with the
+      # adapters immediately.
+      def install!
+        return false if @installed
+
+        require 'yabeda'
+        declare!
+        @installed = true
+      end
+
+      def declare!
+        Yabeda.configure do
+          group :sidekiq do
+            gauge :cron_poll_age,
+                  unit: :seconds,
+                  tags: [],
+                  aggregation: :most_recent,
+                  comment: 'Seconds since the sidekiq-cron poller last completed a tick'
+
+            collect do
+              age = RailsPodKit::GlobalScheduler::Heartbeat.age
+              Yabeda.sidekiq.cron_poll_age.set({}, age) if age
+            end
+          end
+        end
+      end
+
+      # Begins measuring, from before the first tick — so a poller that never
+      # manages one reads as climbing rather than as no-data.
+      def start!
+        @started_at = monotonic_now
+        @last_poll_at = nil
+      end
+
+      # Drops the series with the poller: a stopped scheduler should read as
+      # no-data, not as an age climbing forever.
+      def stop!
+        @started_at = nil
+        @last_poll_at = nil
+      end
+
+      def record!
+        @last_poll_at = monotonic_now
+      end
+
+      # nil until started, which is what keeps the series off any process that
+      # hosts no poller.
+      def age
+        reference = @last_poll_at || @started_at
+        return nil unless reference
+
+        monotonic_now - reference
+      end
+
+      # Monotonic: this is a duration, and a wall-clock step (NTP, a node coming
+      # back from suspend) must not read as the schedule having stalled.
+      def monotonic_now
+        ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
+      end
+    end
+  end
+end

data/lib/rails_pod_kit/global_scheduler.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require 'rails_pod_kit/config'
+require 'rails_pod_kit/global_scheduler/heartbeat'
 require 'rails_pod_kit/supervisor'
 
 module RailsPodKit
@@ -66,6 +67,8 @@ module RailsPodKit
                  reschedule_grace_period: reschedule_grace_period)
       load_schedule!
 
+      Heartbeat.install!
+      Heartbeat.start!
       @supervisor = build_supervisor(supervision_interval).start
     end
 
@@ -74,6 +77,7 @@ module RailsPodKit
     def stop!
       @supervisor&.stop
       @supervisor = nil
+      Heartbeat.stop!
     end
 
     def poller
@@ -132,7 +136,25 @@ module RailsPodKit
       config[:cron_poll_interval] = ::Sidekiq::Cron.configuration.cron_poll_interval.to_i
       config[:cron_poll_process_count] = ::Sidekiq::Cron.configuration.cron_poll_process_count || 1
 
-      ::Sidekiq::Cron::Poller.new(config)
+      poller_class.new(config)
+    end
+
+    # A subclass rather than a prepended module, so the heartbeat hook is
+    # confined to the poller this module builds and never touches sidekiq-cron's
+    # own for a host that also runs a Sidekiq server. Built lazily because the
+    # superclass does not exist until `sidekiq-cron` is required.
+    #
+    # Records only on a normal return: `Poller#enqueue` rescues internally, so
+    # anything that still escapes it means the tick did not complete, and
+    # counting it as a heartbeat would be exactly the lie the gauge exists to
+    # prevent.
+    def poller_class
+      @poller_class ||= Class.new(::Sidekiq::Cron::Poller) do
+        def enqueue
+          super
+          Heartbeat.record!
+        end
+      end
     end
 
     # An entry whose class this process cannot load and which does not declare

data/lib/rails_pod_kit/puma.rb

@@ -43,10 +43,16 @@ module RailsPodKit
       puma_config.plugin :yabeda
       puma_config.plugin :yabeda_prometheus
 
-      # Silence the exporter's per-scrape access log. The `prometheus_silence_logger`
-      # DSL method is defined by the :yabeda_prometheus plugin, so this must run
-      # after the plugin is loaded above. See Config#silence_exporter_access_log.
-      puma_config.prometheus_silence_logger(true) if RailsPodKit.config.silence_exporter_access_log
+      # Drop the exporter's per-scrape access log the same way the WEBrick path
+      # does (see RailsPodKit::Exporter.start!): the log line comes from the
+      # Rack::CommonLogger the exporter's rack app mounts unless this is exactly
+      # 'false'. See Config#silence_exporter_access_log.
+      #
+      # Deliberately not `prometheus_silence_logger(true)`, which is the plugin's
+      # own knob: it swaps Puma's whole log writer for LogWriter.null, and that
+      # writer also carries the exporter's errors — so a /metrics that raises on
+      # every scrape would fail completely silently.
+      ENV['PROMETHEUS_EXPORTER_LOG_REQUESTS'] = 'false' if RailsPodKit.config.silence_exporter_access_log
 
       # `config/puma.rb` is evaluated before Rails is loaded, so requiring this
       # gem here loads yabeda *before* `defined?(Rails)`, and yabeda's Railtie

data/lib/rails_pod_kit/sidekiq.rb

@@ -21,6 +21,8 @@ module RailsPodKit
     def install!(sidekiq_config = nil)
       return unless RailsPodKit.enabled?
 
+      export_policy_env!(collect_cluster_metrics: RailsPodKit.config.sidekiq_global_metrics == :all)
+
       require 'yabeda/sidekiq'
       require 'yabeda/prometheus/mmap'
 
@@ -72,9 +74,10 @@ module RailsPodKit
     # server — i.e. the web (Puma) process under the :web policy. The web has the
     # Sidekiq client configured (Redis access), so yabeda-sidekiq can read the
     # cluster stats there. We force `collect_cluster_metrics` on and keep
-    # `declare_process_metrics` off (the web runs no jobs). Must run before
-    # Yabeda.configure! so the gauges are declared.
+    # `declare_process_metrics` off (the web runs no jobs).
     def enable_global_collection!
+      export_policy_env!(collect_cluster_metrics: true, declare_process_metrics: false)
+
       require 'yabeda/sidekiq'
 
       Yabeda::Sidekiq.config.collect_cluster_metrics = true
@@ -82,6 +85,24 @@ module RailsPodKit
       apply_retries_segmentation!
     end
 
+    # Publishes the policy through yabeda-sidekiq's own env-backed config, and it
+    # has to happen *before* `require 'yabeda/sidekiq'`.
+    #
+    # yabeda-sidekiq declares its cluster gauges inside its own `Yabeda.configure`
+    # block, guarded by `collect_cluster_metrics` — and when Yabeda has already
+    # been configured (any host where its Railtie ran first) requiring the file
+    # evaluates that block immediately. Assigning the flag afterwards is then too
+    # late: the gauges are never declared, while the `collect` block — which reads
+    # the same flag on every scrape — starts referencing them, and /metrics 500s
+    # with a NameError. The env values are read when the config object is built,
+    # so setting them here makes the declaration see the policy whenever the
+    # require lands.
+    def export_policy_env!(collect_cluster_metrics:, declare_process_metrics: nil)
+      ENV['YABEDA_SIDEKIQ_COLLECT_CLUSTER_METRICS'] = collect_cluster_metrics.to_s
+      ENV['YABEDA_SIDEKIQ_DECLARE_PROCESS_METRICS'] = declare_process_metrics.to_s unless declare_process_metrics.nil?
+      ENV['YABEDA_SIDEKIQ_RETRIES_SEGMENTED_BY_QUEUE'] = RailsPodKit.config.retries_segmented_by_queue.to_s
+    end
+
     # Starts the background WEBrick exporter shared with the other non-Puma
     # entry points; guarded there so a re-entrant Sidekiq boot can't double-bind
     # the port.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rails_pod_kit
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.1
 platform: ruby
 authors:
 - Fabio Napoleoni
@@ -159,6 +159,7 @@ files:
 - lib/rails_pod_kit/exporter.rb
 - lib/rails_pod_kit/global_exporter.rb
 - lib/rails_pod_kit/global_scheduler.rb
+- lib/rails_pod_kit/global_scheduler/heartbeat.rb
 - lib/rails_pod_kit/health.rb
 - lib/rails_pod_kit/puma.rb
 - lib/rails_pod_kit/railtie.rb