rails_pod_kit 0.1.1 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 72d44d2c9e2129f3ca4ae021cbc10a227838b5998a80fafc6bbcd8efd18f80d0
-  data.tar.gz: 1c591dcbf8dccf4dcb608483311fbff89363540a25f7a629c6aaa6cda34f6372
+  metadata.gz: 0dda5e5202f1bb5e509fad032de8449c5d8e9eec78c0cedcd0a59d20cc4f0d32
+  data.tar.gz: ebccb551ba5dcfd23c1dc2e668d417f59fb1d76f06522e02c352cea1f5e0dca5
 SHA512:
-  metadata.gz: 989f811cb7ad46f94466b334f4c5405dc46f6bd7df44830645822a14205f9f955451752fba35505488fd4e45a2e62b35ef7306423d6ea0a42432490d0d7c7895
-  data.tar.gz: 22711c7a9d6b4a333bf135133222706e994791ff7648eb8b92f327dfd44d67ae6fa5ce10d797e9ba29d74dd2243ac6563525177458adaeed7b086104fb23c0c0
+  metadata.gz: f6ab1b5432bf79341efabb50d0d3d1862dafa49523bbbc8f70ea718eb520d258b7c216da274f0d184dbeb1737c887ebfb19f04fbcc656e8fd39c6a0e057d6346
+  data.tar.gz: c2e8aa9a3b35a1ea4d557a47be04810c69e79de50a0297593388f90c1be48f8a144a281ad3d26a6c42c4564ba608d6ded064ba53b12f5f49279ddefa940e049c

data/README.md

@@ -11,8 +11,9 @@ packaged behind a single, opinionated entry point:
 - **Health checks** on `/healthz` (database, cache, optionally Redis and Sidekiq),
   wired for Kubernetes startup/liveness/readiness probes — a thin, opinionated
   wrapper around [health-monitor-rails](https://github.com/lbeder/health-monitor-rails).
-- **A supervised SolidQueue scheduler thread**, so a SolidQueue job executor can
-  be autoscaled to zero without stranding its recurring and scheduled jobs.
+- **Scheduler hosting for scale-to-zero**, so a job executor can be autoscaled to
+  zero without stranding its recurring and scheduled jobs: a supervised
+  SolidQueue scheduler thread, and a supervised sidekiq-cron poller for Sidekiq.
 
 The metrics side is a thin wrapper around the
 [yabeda](https://github.com/yabeda-rb) ecosystem:
@@ -81,13 +82,15 @@ On a SolidQueue stack there is no step 3 — see
 config: besides the `RailsPodKit.configure` block (which runs last and wins),
 every setting can come from a `RAILS_POD_KIT_*` env var (e.g.
 `RAILS_POD_KIT_ENABLED=false`, `RAILS_POD_KIT_PORT=9500`,
-`RAILS_POD_KIT_SIDEKIQ_GLOBAL_METRICS=off`) or an optional
+`RAILS_POD_KIT_SIDEKIQ_GLOBAL_METRICS=off`,
+`RAILS_POD_KIT_SCHEDULER_ENABLED=false`) or an optional
 `config/rails_pod_kit.yml` — handy for the Rails-free exporter pod, which runs
 no initializers.
 
 | setting | default | meaning |
 |---------|---------|---------|
-| `enabled` | on, except in `test` | master switch; `false` ⇒ no exporter, no port bound |
+| `enabled` | on, except in `test` | master switch for the **exporter**; `false` ⇒ no exporter, no port bound |
+| `scheduler_enabled` | on, everywhere | kill switch for the **hosted schedulers** (sidekiq-cron poller, SolidQueue scheduler thread). Separate from `enabled` — an app may want one without the other, and this is the one you may need to flip in a hurry, since the scheduler is a single point of failure for the whole schedule. On even in `test`: nothing starts a scheduler implicitly, so there is no port to protect, and a switch that failed closed on a typo would silently stop a schedule. Turning it off logs a warning naming the scheduler that did not start. |
 | `port` | `9394` (env `PROMETHEUS_EXPORTER_PORT`) | exporter bind port for Puma **and** Sidekiq |
 | `sidekiq_global_metrics` | `:web` | who exports the Redis-wide queue metrics: `:web` = only the always-on web process (no per-worker duplication); `:all` = every worker; `:off` = nobody |
 | `puma_control_url` | `tcp://127.0.0.1:9293` (env `PUMA_CONTROL_URL`) | localhost-only Puma control app the stats reader queries |
@@ -156,6 +159,8 @@ endpoint every few seconds); pass `silence_controller_log: false` to keep it.
   `sidekiq_queue_latency`, `sidekiq_active_processes`,
   `sidekiq_active_workers_count`, `sidekiq_jobs_retry_count`,
   `sidekiq_jobs_dead_count`, `sidekiq_jobs_scheduled_count`.
+- **Sidekiq (cron scheduler):** `sidekiq_cron_poll_age_seconds`, on the process
+  hosting the poller (see GlobalScheduler).
 - **SolidQueue (DB-wide):** `solid_queue_backlog`,
   `solid_queue_latency_seconds`.
 
@@ -273,6 +278,12 @@ Sidekiq job class.
 | `sidekiq.active_processes` | — |
 | `sidekiq.active_workers_count` | — |
 
+**Sidekiq — cron scheduler** (only the process hosting the poller, `namespace: sidekiq`):
+
+| canonical Datadog metric | functional tags |
+|---|---|
+| `sidekiq.cron_poll_age_seconds` | — |
+
 **Sidekiq — per-process / job** (worker pod, `namespace: sidekiq`; emitted on job activity):
 
 | canonical Datadog metric | type | functional tags |
@@ -348,6 +359,99 @@ declares the cluster gauges, starts the exporter and blocks until SIGTERM.
 Booting the full host app just to read a handful of Redis counters would cost
 ~300Mi RSS for nothing — this process sits at ~60Mi.
 
+## Sidekiq: scale-to-zero
+
+Being an always-on singleton makes that same pod the right home for the
+**sidekiq-cron poller**, which is what lets the worker fleet scale to zero.
+
+sidekiq-cron installs its poller from inside `Sidekiq.configure_server`, so on
+its own the schedule exists only while a Sidekiq server is alive. At zero
+replicas nothing polls, nothing is enqueued, and nothing ever raises the queue
+depth that would wake a worker back up — a closed loop that forces a permanent
+floor of one replica just to keep a poller alive. Missed runs are not caught up
+afterwards either: `reschedule_grace_period` (60s by default) discards any run
+older than itself.
+
+The poller has no such requirement of its own — `Sidekiq::Cron::Poller` is a
+Redis-polling thread that runs in any process holding a Sidekiq config — so
+`scheduler: true` hosts it here:
+
+```ruby
+RailsPodKit::GlobalExporter.run!(
+  redis: { url: ENV['REDIS_URL'] },
+  scheduler: true,
+  schedule_file: File.expand_path('../config/schedule.yml', __dir__)
+)
+```
+
+`schedule_file:`, `poll_interval:` and `reschedule_grace_period:` override
+sidekiq-cron's defaults (`config/schedule.yml` resolved against the working
+directory, polled every 30s, catching up runs at most 60s late);
+`supervision_interval:` tunes the liveness check. `RailsPodKit::GlobalScheduler`
+is usable on its own (`start!` / `stop!`) if the always-on process is something
+other than the exporter.
+
+> **Size `reschedule_grace_period` over your worst restart.** It is what makes
+> restarting the *only* scheduling process free: below it a missed occurrence is
+> caught up on the next poll, above it the run is skipped silently. sidekiq-cron
+> defaults to 60s, which a node drain or an evicted pod can easily exceed —
+> rolling updates are covered anyway, since a `maxSurge` overlap means there is
+> no gap at all. Catching up is bounded, not repeated: `last_enqueue_time` in
+> Redis still gates each occurrence to exactly one enqueue.
+>
+> This is the reason a singleton scheduler does **not** need to become an HA
+> pair. A second replica is safe for the poller (same `zadd` lock) but doubles
+> every metric series the pod publishes, and a `PodDisruptionBudget` on a
+> single-replica Deployment stalls node drains rather than protecting anything.
+
+The poller runs under `RailsPodKit::Supervisor` — the same supervising timer
+that keeps the SolidQueue scheduler thread alive, since both share the failure
+mode: the thread dies, the host process notices nothing, and the schedule stops
+silently. The cron poller's own loop swallows StandardError, so a Redis blip
+costs one skipped tick; the supervisor makes anything it does *not* catch a
+skipped tick too.
+
+### The heartbeat
+
+The supervisor covers a poller thread that *dies*. It cannot see one that is
+running and no longer enqueueing — which from the outside is indistinguishable
+from an idle one: every gauge stays fresh, `/metrics` answers 200, the process
+looks healthy. On the only process carrying the schedule that is a silently
+stopped schedule, i.e. the very failure hosting the poller here was meant to
+eliminate, back through another door.
+
+So `start!` also publishes **`sidekiq_cron_poll_age_seconds`**: seconds since the
+poller last completed a tick. It measures the loop turning, so it stays flat on a
+healthy but idle schedule and climbs the moment ticks stop — the one shape an
+alert can be written against:
+
+```
+max:sidekiq.cron_poll_age_seconds{…} > 10 * <poll interval>
+```
+
+Before the first tick it measures from `start!`, so a poller that never manages
+one reads as climbing rather than as no-data; it is `nil` (and the series absent)
+on any process that hosts no poller, and again once `stop!` runs.
+
+Deliberately **not** "time since last enqueue", which would climb on any quiet
+schedule and so alert on nothing happening. Answering *should this job have run
+by now?* needs a per-job check against the cron expression, not a gauge.
+
+> **Every schedule entry must declare `active_job: true`.** This process has no
+> Rails, so it cannot resolve the job classes; sidekiq-cron then falls back to
+> pushing a raw message, and only that flag makes the message an ActiveJob
+> wrapper (naming the class as a *string*, which the worker resolves). Without it
+> the job is pushed as a bare Sidekiq job and runs outside ActiveJob entirely.
+> `start!` logs a warning naming any entry in that state. For the same reason the
+> schedule file's ERB must not reach for Rails.
+
+Leaving the workers' own poller in place is fine and costs nothing: enqueueing is
+gated on a Redis `zadd` that exactly one caller wins — the same lock that already
+lets multiple worker replicas coexist without double-firing. Both processes must
+then read the *same* schedule file, though: `load_from_hash!` removes the
+schedule-sourced jobs that are absent from the file it is given, so two processes
+loading different files will delete each other's entries.
+
 ## SolidQueue: scale-to-zero
 
 SolidQueue's executor has nothing to do while the queue is empty, so it is the
@@ -378,10 +482,11 @@ This is deliberately **not** `plugin :solid_queue`. That one runs the full
 supervisor, which forks and whose watchdog takes Puma down when the supervisor
 exits — and a transient Postgres disconnect is enough to cause that
 ([rails/solid_queue#512](https://github.com/rails/solid_queue/issues/512)). Here
-a DB blip at worst kills the scheduler thread; a `Concurrent::TimerTask` (the
-same primitive SolidQueue supervises its own processes with) notices on the next
-tick and starts a fresh one, the process itself never notices, and the scheduler
-re-registers on recovery.
+a DB blip at worst kills the scheduler thread; `RailsPodKit::Supervisor` — a
+`Concurrent::TimerTask`, the same primitive SolidQueue supervises its own
+processes with, and the same one that keeps the sidekiq-cron poller alive —
+notices on the next tick and starts a fresh one, the process itself never
+notices, and the scheduler re-registers on recovery.
 
 Running it on every replica is safe: enqueues stay exactly-once via the unique
 index on `solid_queue_recurring_executions (task_key, run_at)`. Static tasks come
@@ -395,8 +500,10 @@ ones from the DB.
 | `recurring_schedule_file` | `config/recurring.yml` | static task definitions; skipped when absent |
 
 `start_scheduler!` is **not** gated on `enabled` — that switch owns the metrics
-exporter, and an app may well want the scheduler with metrics off. What keeps it
-out of consoles and specs is *where* you call it from.
+exporter, and an app may well want the scheduler with metrics off.
+`scheduler_enabled` is the switch that does own it, shared with the sidekiq-cron
+poller. Beyond that, what keeps it out of consoles and specs is *where* you call
+it from.
 
 ### 2. Queue depth has to be visible
 
@@ -493,6 +600,8 @@ serves `solid_queue_*` and nothing else, so the check config needs no filters.
   a WEBrick thread instead, started at most once per process.
 - **SolidQueue and Sidekiq are the host's.** The gem depends on neither; the
   SolidQueue integration is inert until you call it, exactly like the Sidekiq one.
+  `sidekiq-cron` too: it is required only when `GlobalScheduler.start!` is called,
+  so an app that does not schedule anything need not carry it.
 - **Queue-gauge query cost.** The gauges run four small grouped aggregates per
   scrape. `MIN(created_at)` is not covered by SolidQueue's indexes, so on a
   backlog of many thousands of rows it is a scan — cheap at a normal scrape

data/VERSION

@@ -1 +1 @@
-0.1.1
+0.3.0

data/lib/rails_pod_kit/config.rb

@@ -60,17 +60,29 @@ module RailsPodKit
     #   transports — the Puma plugin's exporter (`prometheus_silence_logger`)
     #   and the WEBrick exporter used by Sidekiq / the dedicated global exporter
     #   (`Rack::CommonLogger`). Flip to false only to debug the exporter itself.
+    # scheduler_enabled: kill switch for the hosted schedulers (the sidekiq-cron
+    #   poller and the SolidQueue scheduler thread). Separate from `enabled`,
+    #   which owns the metrics exporter: an app may well want one without the
+    #   other, and the scheduler is the piece you may need to turn off in a
+    #   hurry — it is a single point of failure for the whole schedule, and
+    #   `RAILS_POD_KIT_SCHEDULER_ENABLED=false` + a restart beats a deploy at
+    #   3am. Defaults to on *including* in the test env, unlike `enabled`:
+    #   nothing starts a scheduler implicitly (it takes an explicit call from an
+    #   entry point), so there is no port to protect, and a flag that fails
+    #   closed on a typo would silently stop a schedule.
     attr_config :enabled,
                 :port,
                 :puma_control_url,
                 sidekiq_global_metrics: :web,
                 retries_segmented_by_queue: false,
-                silence_exporter_access_log: true
+                silence_exporter_access_log: true,
+                scheduler_enabled: true
 
     coerce_types port: :integer,
                  enabled: :boolean,
                  retries_segmented_by_queue: :boolean,
-                 silence_exporter_access_log: :boolean
+                 silence_exporter_access_log: :boolean,
+                 scheduler_enabled: :boolean
 
     def initialize(overrides = nil)
       super
@@ -87,6 +99,10 @@ module RailsPodKit
       !!enabled
     end
 
+    def scheduler_enabled?
+      !!scheduler_enabled
+    end
+
     # True unless we're clearly in a test environment. Kept independent of
     # Rails so the gem's own specs (which don't load the host app) get a safe
     # default and never bind a socket.
@@ -114,5 +130,15 @@ module RailsPodKit
     def enabled?
       config.enabled?
     end
+
+    # Gate shared by both hosted schedulers. Says so out loud when it turns one
+    # off: a process that starts, stays up and quietly schedules nothing is the
+    # one failure this whole feature exists to avoid.
+    def scheduler_enabled?(what = 'scheduler')
+      return true if config.scheduler_enabled?
+
+      warn "[rails_pod_kit] scheduler_enabled=false — #{what} not started"
+      false
+    end
   end
 end

data/lib/rails_pod_kit/global_exporter.rb

@@ -1,6 +1,8 @@
 # frozen_string_literal: true
 
 require 'rails_pod_kit/config'
+require 'rails_pod_kit/global_scheduler'
+require 'rails_pod_kit/shutdown'
 
 module RailsPodKit
   # Standalone, always-on exporter for the Sidekiq global (Redis-wide) queue
@@ -26,6 +28,10 @@ module RailsPodKit
   #   require 'bundler/setup'
   #   require 'rails_pod_kit/global_exporter'
   #   RailsPodKit::GlobalExporter.run!(redis: { url: ENV['REDIS_URL'] })
+  #
+  # Being an always-on singleton also makes it the natural host for the
+  # sidekiq-cron poller (`scheduler: true`, see GlobalScheduler), which is what
+  # lets the workers scale to zero without losing their schedule.
   module GlobalExporter
     module_function
 
@@ -47,22 +53,34 @@ module RailsPodKit
     # `redis:` takes the same options hash the host passes to its own
     # `Sidekiq.configure_*` blocks (`url:`, `ssl_params:`, …), so the connection
     # config stays a host decision with a single source of truth.
-    def run!(redis:)
-      unless RailsPodKit.enabled?
-        warn '[rails_pod_kit] disabled — global exporter not started'
-        return
-      end
+    #
+    # `scheduler: true` additionally runs the sidekiq-cron poller here; any
+    # extra keywords are GlobalScheduler.start!'s (`schedule_file:`,
+    # `poll_interval:`, `supervision_interval:`). The scheduler is not gated on
+    # `RailsPodKit.enabled?` —
+    # that switch owns the metrics exporter, and an app may well want the
+    # singleton scheduler with metrics turned off.
+    def run!(redis:, scheduler: false, **scheduler_options)
+      serve_metrics = RailsPodKit.enabled?
+      warn '[rails_pod_kit] disabled — /metrics not served by the global exporter' unless serve_metrics
+      return unless serve_metrics || scheduler
 
       configure_redis!(redis)
+      start_metrics! if serve_metrics
+      GlobalScheduler.start!(**scheduler_options) if scheduler
+
+      # Both the exporter and the poller serve from background threads; block
+      # the main thread so the process stays up until the kubelet sends SIGTERM.
+      Shutdown.await
+      GlobalScheduler.stop! if scheduler
+    end
+
+    def start_metrics!
       install!
       RailsPodKit::Sidekiq.start_metrics_server!
 
       require 'yabeda'
       Yabeda.configure! unless Yabeda.already_configured?
-
-      # The exporter serves from a background thread; block the main thread so
-      # the process stays up until the kubelet sends SIGTERM.
-      sleep
     end
 
     # Configure the Sidekiq client's Redis connection so this Rails-free process

data/lib/rails_pod_kit/global_scheduler/heartbeat.rb

@@ -0,0 +1,89 @@
+# frozen_string_literal: true
+
+module RailsPodKit
+  module GlobalScheduler
+    # The one failure the rest of the machinery cannot see.
+    #
+    # A poller thread that dies is restarted by the Supervisor, and a process
+    # that stops serving is caught by the exporter's own probe. But a poller
+    # that is *running and no longer enqueueing* is indistinguishable from an
+    # idle one from the outside: every gauge stays fresh, /metrics answers 200,
+    # the pod is Running and Ready. On the only process carrying the schedule
+    # that is a silently stopped schedule — exactly the failure hosting the
+    # poller here was meant to eliminate, coming back through another door.
+    #
+    # So publish the age of the last completed tick, as
+    # `sidekiq_cron_poll_age_seconds`. It measures the loop turning, which means
+    # it stays flat on a healthy but idle schedule and climbs the moment ticks
+    # stop — the one shape an alert can be written against.
+    #
+    # Deliberately *not* "time since last enqueue": that climbs on any quiet
+    # schedule, so it would alert on nothing happening. Answering "should this
+    # job have run by now?" needs a per-job check against the cron expression,
+    # not a gauge.
+    module Heartbeat
+      module_function
+
+      # Declares the gauge. One-shot, and safe either side of
+      # `Yabeda.configure!` — a metric declared after it is registered with the
+      # adapters immediately.
+      def install!
+        return false if @installed
+
+        require 'yabeda'
+        declare!
+        @installed = true
+      end
+
+      def declare!
+        Yabeda.configure do
+          group :sidekiq do
+            gauge :cron_poll_age,
+                  unit: :seconds,
+                  tags: [],
+                  aggregation: :most_recent,
+                  comment: 'Seconds since the sidekiq-cron poller last completed a tick'
+
+            collect do
+              age = RailsPodKit::GlobalScheduler::Heartbeat.age
+              Yabeda.sidekiq.cron_poll_age.set({}, age) if age
+            end
+          end
+        end
+      end
+
+      # Begins measuring, from before the first tick — so a poller that never
+      # manages one reads as climbing rather than as no-data.
+      def start!
+        @started_at = monotonic_now
+        @last_poll_at = nil
+      end
+
+      # Drops the series with the poller: a stopped scheduler should read as
+      # no-data, not as an age climbing forever.
+      def stop!
+        @started_at = nil
+        @last_poll_at = nil
+      end
+
+      def record!
+        @last_poll_at = monotonic_now
+      end
+
+      # nil until started, which is what keeps the series off any process that
+      # hosts no poller.
+      def age
+        reference = @last_poll_at || @started_at
+        return nil unless reference
+
+        monotonic_now - reference
+      end
+
+      # Monotonic: this is a duration, and a wall-clock step (NTP, a node coming
+      # back from suspend) must not read as the schedule having stalled.
+      def monotonic_now
+        ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
+      end
+    end
+  end
+end

data/lib/rails_pod_kit/global_scheduler.rb

@@ -0,0 +1,178 @@
+# frozen_string_literal: true
+
+require 'rails_pod_kit/config'
+require 'rails_pod_kit/global_scheduler/heartbeat'
+require 'rails_pod_kit/supervisor'
+
+module RailsPodKit
+  # Runs the sidekiq-cron poller in a process that is *not* a Sidekiq server, so
+  # a Sidekiq deployment can be autoscaled to zero without losing its schedule.
+  #
+  # sidekiq-cron installs its poller from inside `Sidekiq.configure_server`, so
+  # the schedule exists only while a Sidekiq server is alive. At zero replicas
+  # nothing polls, nothing is enqueued, and nothing ever raises the queue depth
+  # that would wake a worker back up — a closed loop that forces a permanent
+  # floor of one replica just to keep a poller alive. Recurring jobs are not
+  # caught up afterwards either: `reschedule_grace_period` (60s by default)
+  # discards any run older than itself, so a worker started later skips it.
+  #
+  # The poller has no such requirement of its own. `Sidekiq::Cron::Poller` is a
+  # Redis-polling thread and runs in any process holding a Sidekiq config, so
+  # hosting it on an always-on singleton — the dedicated exporter pod, see
+  # GlobalExporter — breaks the cycle and leaves the workers free to scale to
+  # zero.
+  #
+  # Deliberately Rails-free, like the exporter it sits beside: enqueueing does
+  # not need the job classes. When `Job#enqueue!` cannot resolve the class it
+  # pushes a plain Sidekiq message naming the ActiveJob wrapper with the job
+  # class as a *string*, and the worker — which does have Rails — resolves it.
+  # That fallback is only correct for entries declaring `active_job: true`, so
+  # `start!` warns about any that would instead be pushed as bare Sidekiq jobs.
+  #
+  # The poller runs under the shared Supervisor, exactly like SolidQueue's
+  # scheduler thread. Its own loop swallows StandardError (`Poller#enqueue` and
+  # `#wait` both do), so a Redis blip costs one skipped tick — but anything it
+  # does not catch takes the thread down and, since this process is the only
+  # scheduler, the schedule with it, silently.
+  module GlobalScheduler
+    SOURCE = 'rails_pod_kit.global_scheduler'
+
+    module_function
+
+    # Loads the schedule and starts the supervisor, which starts the poller on
+    # its first (immediate) tick, then returns without blocking. Idempotent: a
+    # second call is a no-op rather than a second poller in the same process.
+    #
+    # `schedule_file:`, `poll_interval:` and `reschedule_grace_period:` override
+    # sidekiq-cron's own defaults (`config/schedule.yml`, resolved against the
+    # working directory, polled every 30s, catching up runs at most 60s late);
+    # `supervision_interval:` is the shared Supervisor's.
+    #
+    # Raising the grace period is what makes a restart of the single scheduling
+    # process free: below it a missed occurrence is caught up on the next poll,
+    # above it the run is skipped silently. Size it over the worst restart —
+    # eviction, reschedule, image pull, boot — not over the poll interval.
+    def start!(schedule_file: nil, poll_interval: nil, reschedule_grace_period: nil,
+               supervision_interval: Supervisor::DEFAULT_INTERVAL)
+      return @supervisor if @supervisor
+      return unless RailsPodKit.scheduler_enabled?('sidekiq-cron poller')
+
+      require 'sidekiq'
+      require 'sidekiq-cron'
+      # sidekiq-cron renders the schedule file through ERB without requiring it:
+      # under Rails it is always already loaded, here it is not.
+      require 'erb'
+
+      configure!(schedule_file: schedule_file, poll_interval: poll_interval,
+                 reschedule_grace_period: reschedule_grace_period)
+      load_schedule!
+
+      Heartbeat.install!
+      Heartbeat.start!
+      @supervisor = build_supervisor(supervision_interval).start
+    end
+
+    # Winds the poller down so an in-flight tick finishes before the process
+    # exits, instead of being cut off mid-enqueue by the signal.
+    def stop!
+      @supervisor&.stop
+      @supervisor = nil
+      Heartbeat.stop!
+    end
+
+    def poller
+      @supervisor&.subject
+    end
+
+    def build_supervisor(interval)
+      Supervisor.new(
+        source: SOURCE,
+        interval: interval,
+        start: -> { build_poller.tap(&:start) },
+        alive: method(:poller_alive?),
+        stop: :terminate
+      )
+    end
+
+    # `Sidekiq::Scheduled::Poller` keeps its thread in `@thread` and `start` is a
+    # no-op once that is set, so there is no public way to ask whether the poller
+    # is still running, nor to revive it. Reading the ivar lets the supervisor
+    # replace a dead poller wholesale — schedule state lives in Redis, so a fresh
+    # one picks up exactly where the old one stopped. An unrecognised shape reads
+    # as alive, so an upstream rename costs the supervision, never a restart loop.
+    def poller_alive?(poller)
+      return true unless poller.instance_variable_defined?(:@thread)
+
+      !!poller.instance_variable_get(:@thread)&.alive?
+    end
+
+    def configure!(schedule_file: nil, poll_interval: nil, reschedule_grace_period: nil)
+      ::Sidekiq::Cron.configure do |cron|
+        cron.cron_schedule_file = schedule_file if schedule_file
+        cron.cron_poll_interval = poll_interval if poll_interval
+        cron.reschedule_grace_period = reschedule_grace_period if reschedule_grace_period
+      end
+    end
+
+    # sidekiq-cron reads the schedule file from a Sidekiq server's `:startup`
+    # lifecycle event, which never fires here, so load it explicitly.
+    def load_schedule!
+      return unless ::Sidekiq::Cron.configuration.enabled
+
+      loader = ::Sidekiq::Cron::ScheduleLoader.new
+      return unless loader.has_schedule_file?
+
+      loader.load_schedule
+      warn_unresolvable_entries!
+    end
+
+    # sidekiq-cron's own Launcher publishes these two into the Sidekiq config
+    # before instantiating the poller, which reads them straight back out.
+    # Pinning the process count keeps the poll interval at the configured value:
+    # the inherited default counts *live Sidekiq servers*, which here is a count
+    # of anything but cron pollers — and is zero while the workers are scaled in.
+    def build_poller
+      config = ::Sidekiq.default_configuration
+      config[:cron_poll_interval] = ::Sidekiq::Cron.configuration.cron_poll_interval.to_i
+      config[:cron_poll_process_count] = ::Sidekiq::Cron.configuration.cron_poll_process_count || 1
+
+      poller_class.new(config)
+    end
+
+    # A subclass rather than a prepended module, so the heartbeat hook is
+    # confined to the poller this module builds and never touches sidekiq-cron's
+    # own for a host that also runs a Sidekiq server. Built lazily because the
+    # superclass does not exist until `sidekiq-cron` is required.
+    #
+    # Records only on a normal return: `Poller#enqueue` rescues internally, so
+    # anything that still escapes it means the tick did not complete, and
+    # counting it as a heartbeat would be exactly the lie the gauge exists to
+    # prevent.
+    def poller_class
+      @poller_class ||= Class.new(::Sidekiq::Cron::Poller) do
+        def enqueue
+          super
+          Heartbeat.record!
+        end
+      end
+    end
+
+    # An entry whose class this process cannot load and which does not declare
+    # `active_job: true` is pushed as a bare Sidekiq job message, so the worker
+    # runs `perform` outside ActiveJob — no callbacks, no argument
+    # deserialization, no retry bookkeeping.
+    def warn_unresolvable_entries!
+      names = ::Sidekiq::Cron::Job.all('*').reject { |job| enqueueable_without_class?(job) }.map(&:name)
+      return if names.empty?
+
+      ::Sidekiq.logger.warn do
+        "[rails_pod_kit] cron entries #{names.join(', ')} name a class this Rails-free process cannot load and " \
+          'are not marked `active_job: true`; they would be enqueued as plain Sidekiq jobs.'
+      end
+    end
+
+    def enqueueable_without_class?(job)
+      job.to_hash[:active_job] == '1' || !::Sidekiq::Cron::Support.safe_constantize(job.klass.to_s).nil?
+    end
+  end
+end

data/lib/rails_pod_kit/shutdown.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+module RailsPodKit
+  # Blocks the main thread of an always-on entry point until the orchestrator
+  # signals. A self-pipe rather than a Queue or a Mutex: writing to an IO is one
+  # of the few things safe to do from a trap handler.
+  module Shutdown
+    SIGNALS = %w[INT TERM].freeze
+
+    module_function
+
+    def await(signals: SIGNALS)
+      reader, writer = IO.pipe
+      signals.each { |signal| Signal.trap(signal) { writer.puts(signal) } }
+      reader.gets
+    end
+  end
+end

data/lib/rails_pod_kit/solid_queue/scheduler_runner.rb

@@ -1,11 +1,10 @@
 # frozen_string_literal: true
 
-require 'concurrent'
 require 'active_support/configuration_file'
 require 'active_support/core_ext/hash/keys'
 
 require 'rails_pod_kit/config'
-require 'rails_pod_kit/error_reporter'
+require 'rails_pod_kit/supervisor'
 
 module RailsPodKit
   module SolidQueue
@@ -44,51 +43,40 @@ module RailsPodKit
         @polling_interval = polling_interval
         @supervision_interval = supervision_interval
         @recurring_schedule_file = recurring_schedule_file
-        @stopping = false
       end
 
       # Starts the supervisor, which starts the scheduler on its first (immediate)
       # tick and returns without blocking.
       def start
-        @supervisor = ::Concurrent::TimerTask.new(
-          execution_interval: @supervision_interval,
-          run_now: true
-        ) { supervise }
-        @supervisor.execute
+        @supervisor = build_supervisor.start
         self
       end
 
-      # Graceful stop: drop the supervisor first so it can't resurrect the
-      # scheduler, then wind the scheduler down (unschedule its timers and
-      # deregister the process, instead of leaving a row to expire).
+      # Graceful stop: the supervisor drops its timer before winding the
+      # scheduler down (unschedule its timers and deregister the process,
+      # instead of leaving a row to expire).
       def stop
-        @stopping = true
-        @supervisor&.shutdown
+        @supervisor&.stop
         @supervisor = nil
-        @scheduler&.stop
-        @scheduler = nil
       end
 
       def running?
-        !!@scheduler&.alive?
+        !!@supervisor&.running?
       end
 
       private
 
-      # Concurrent::TimerTask silently drops a raising block, so report here and
-      # let the next tick retry — a scheduler that can't start because Postgres
-      # is down must keep trying.
-      def supervise
-        ensure_scheduler_running
-      rescue StandardError => e
-        ErrorReporter.report(e, source: SOURCE)
+      def build_supervisor
+        Supervisor.new(
+          source: SOURCE,
+          interval: @supervision_interval,
+          start: -> { build_scheduler },
+          alive: :alive?,
+          stop: :stop
+        )
       end
 
-      # Idempotent: starts the scheduler on the first tick, and restarts it only
-      # once its thread has actually died.
-      def ensure_scheduler_running
-        return if @stopping || @scheduler&.alive?
-
+      def build_scheduler
         scheduler = ::SolidQueue::Scheduler.new(
           recurring_tasks: static_recurring_tasks,
           dynamic_tasks_enabled: true,
@@ -96,7 +84,7 @@ module RailsPodKit
         )
         scheduler.mode = :async
         scheduler.start # spawns the scheduler's own thread and returns
-        @scheduler = scheduler
+        scheduler
       end
 
       # The static tasks from config/recurring.yml; the dynamic ones come from

data/lib/rails_pod_kit/solid_queue.rb

@@ -2,6 +2,7 @@
 
 require 'rails_pod_kit/config'
 require 'rails_pod_kit/exporter'
+require 'rails_pod_kit/shutdown'
 require 'rails_pod_kit/solid_queue/metrics'
 require 'rails_pod_kit/solid_queue/scheduler_runner'
 
@@ -49,10 +50,12 @@ module RailsPodKit
     #
     # Not gated on `RailsPodKit.enabled?` — that switch owns the metrics
     # exporter, and an app may well want the scheduler with metrics turned off.
-    # The guard against starting one in a console or in specs is *where* you
+    # `scheduler_enabled` is the switch that does own this (see Config); beyond
+    # it, the guard against starting one in a console or in specs is *where* you
     # call this from (`after_booted`, or the exporter entrypoint).
     def start_scheduler!(**)
       return scheduler_runner if scheduler_runner
+      return unless RailsPodKit.scheduler_enabled?('SolidQueue scheduler')
 
       @scheduler_runner = SchedulerRunner.new(**).start
     end
@@ -97,13 +100,9 @@ module RailsPodKit
     end
 
     # Blocks the main thread (the exporter and the scheduler both run on their
-    # own) until the kubelet signals. A self-pipe rather than a Queue or a
-    # Mutex: writing to an IO is one of the few things safe to do from a trap
-    # handler.
+    # own) until the kubelet signals.
     def await_shutdown
-      reader, writer = IO.pipe
-      %w[INT TERM].each { |signal| Signal.trap(signal) { writer.puts(signal) } }
-      reader.gets
+      Shutdown.await
     end
   end
 end

data/lib/rails_pod_kit/supervisor.rb

@@ -0,0 +1,90 @@
+# frozen_string_literal: true
+
+require 'concurrent'
+
+require 'rails_pod_kit/error_reporter'
+
+module RailsPodKit
+  # Keeps a background worker alive on a process that must not lose it.
+  #
+  # Both schedulers the gem hosts (SolidQueue's, and sidekiq-cron's) run on
+  # their own thread inside an always-on process, and both share the same
+  # failure mode: the thread dies, the host process notices nothing, and the
+  # schedule stops silently. This is that supervision, once — a timer that
+  # rebuilds the worker as soon as it stops being alive.
+  #
+  # The three things that genuinely differ between workers are injected: how to
+  # build and start one, how to ask whether it is still alive, and how to wind
+  # it down. Everything else — the immediate first tick, the stop latch that
+  # keeps a shutdown from being undone by a tick already in flight, reporting
+  # instead of dying — is identical and lives here.
+  #
+  # Deliberately free of Rails: the sidekiq-cron scheduler runs in a Rails-free
+  # process, and ErrorReporter degrades to a warning where there is no Rails
+  # error reporter to hand the failure to.
+  class Supervisor
+    DEFAULT_INTERVAL = 30
+
+    attr_reader :subject
+
+    # `start:` is a callable returning the started worker. `alive:` and `stop:`
+    # take either a method name to send to that worker or a callable receiving
+    # it, so the common case stays declarative and a liveness check the worker
+    # does not expose itself is still expressible. `stop:` is optional — a
+    # worker with no teardown is simply dropped.
+    def initialize(source:, start:, alive:, stop: nil, interval: DEFAULT_INTERVAL)
+      @source = source
+      @start = start
+      @alive = alive
+      @stop = stop
+      @interval = interval
+      @stopping = false
+    end
+
+    # Starts the timer, which starts the worker on its first (immediate) tick,
+    # and returns without blocking.
+    def start
+      @stopping = false
+      @timer = ::Concurrent::TimerTask.new(execution_interval: @interval, run_now: true) { supervise }
+      @timer.execute
+      self
+    end
+
+    # Graceful stop: drop the timer first so it cannot resurrect the worker,
+    # then wind the worker down rather than leaving it to be cut off.
+    def stop
+      @stopping = true
+      @timer&.shutdown
+      @timer = nil
+      invoke(@stop, @subject) if @subject && @stop
+      @subject = nil
+    end
+
+    def running?
+      !!@subject && invoke(@alive, @subject)
+    end
+
+    private
+
+    def invoke(hook, worker)
+      hook.is_a?(::Symbol) ? worker.public_send(hook) : hook.call(worker)
+    end
+
+    # Concurrent::TimerTask silently drops a raising block, so report here and
+    # let the next tick retry — a worker that cannot start because its backing
+    # store is down must keep trying.
+    def supervise
+      ensure_running
+    rescue StandardError => e
+      ErrorReporter.report(e, source: @source)
+    end
+
+    # Idempotent: starts the worker on the first tick, and replaces it only once
+    # it has actually stopped being alive.
+    def ensure_running
+      return if @stopping || running?
+
+      @subject = @start.call
+    end
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rails_pod_kit
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.3.0
 platform: ruby
 authors:
 - Fabio Napoleoni
@@ -140,9 +140,10 @@ description: |
   opinionated kit for running Rails applications on Kubernetes: Puma, Sidekiq
   and SolidQueue runtime metrics in Prometheus text format on an in-process
   /metrics endpoint (default port 9394), plus a /healthz endpoint wired for
-  liveness/readiness/startup probes. No sidecar, no separate collector. For
-  SolidQueue it also ships the supervised scheduler thread that lets the job
-  executor scale to zero.
+  liveness/readiness/startup probes. No sidecar, no separate collector. It
+  also hosts the schedulers that let a job executor scale to zero without
+  stranding its recurring jobs: a supervised SolidQueue scheduler thread, and
+  a supervised sidekiq-cron poller for Sidekiq.
 email:
 - f.napoleoni@gmail.com
 executables: []
@@ -157,13 +158,17 @@ files:
 - lib/rails_pod_kit/error_reporter.rb
 - lib/rails_pod_kit/exporter.rb
 - lib/rails_pod_kit/global_exporter.rb
+- lib/rails_pod_kit/global_scheduler.rb
+- lib/rails_pod_kit/global_scheduler/heartbeat.rb
 - lib/rails_pod_kit/health.rb
 - lib/rails_pod_kit/puma.rb
 - lib/rails_pod_kit/railtie.rb
+- lib/rails_pod_kit/shutdown.rb
 - lib/rails_pod_kit/sidekiq.rb
 - lib/rails_pod_kit/solid_queue.rb
 - lib/rails_pod_kit/solid_queue/metrics.rb
 - lib/rails_pod_kit/solid_queue/scheduler_runner.rb
+- lib/rails_pod_kit/supervisor.rb
 - lib/rails_pod_kit/version.rb
 homepage: https://github.com/fabn/rails_pod_kit
 licenses: