rails_pod_kit 0.0.3 → 0.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 24c0e428faf1a3c46646a1d7a0e693b5cde8373f6aea61f9dc005d6b6dcba192
-  data.tar.gz: 968fc08545c75f5b22f91c2c882068b20069d7bb2bb6f48d1abf2aa09f4bc71b
+  metadata.gz: 72d44d2c9e2129f3ca4ae021cbc10a227838b5998a80fafc6bbcd8efd18f80d0
+  data.tar.gz: 1c591dcbf8dccf4dcb608483311fbff89363540a25f7a629c6aaa6cda34f6372
 SHA512:
-  metadata.gz: 266b27abb0911f031e54522dfcea870d603e1ec143465b4441a206590654087c1604a9aeb4438745bc10600bf3696b7d4938c0c936e0145322b76cb0279a5513
-  data.tar.gz: dedc4665fb51d38b9e713caead650ee875108aed4a24e9f27bb3053f2ec0152bdcc0f3840cc0ac76056858f330b723c502bc5f66d726261db47a7033e6ae2142
+  metadata.gz: 989f811cb7ad46f94466b334f4c5405dc46f6bd7df44830645822a14205f9f955451752fba35505488fd4e45a2e62b35ef7306423d6ea0a42432490d0d7c7895
+  data.tar.gz: 22711c7a9d6b4a333bf135133222706e994791ff7648eb8b92f327dfd44d67ae6fa5ce10d797e9ba29d74dd2243ac6563525177458adaeed7b086104fb23c0c0

data/README.md

@@ -3,14 +3,16 @@
 The operational endpoints a Rails pod needs to be a good Kubernetes citizen,
 packaged behind a single, opinionated entry point:
 
-- **Prometheus metrics** for Puma and Sidekiq, served **in-process** on a
-  single `/metrics` endpoint (default port **9394**) in both the web (Puma) and
-  worker (Sidekiq) processes — no sidecar, no separate collector process. A
-  metrics agent (e.g. the Datadog Agent via OpenMetrics autodiscovery) scrapes
-  the pod directly.
+- **Prometheus metrics** for Puma, Sidekiq and SolidQueue, served **in-process**
+  on a single `/metrics` endpoint (default port **9394**) in the web (Puma) and
+  worker processes — no sidecar, no separate collector process. A metrics agent
+  (e.g. the Datadog Agent via OpenMetrics autodiscovery) scrapes the pod
+  directly.
 - **Health checks** on `/healthz` (database, cache, optionally Redis and Sidekiq),
   wired for Kubernetes startup/liveness/readiness probes — a thin, opinionated
   wrapper around [health-monitor-rails](https://github.com/lbeder/health-monitor-rails).
+- **A supervised SolidQueue scheduler thread**, so a SolidQueue job executor can
+  be autoscaled to zero without stranding its recurring and scheduled jobs.
 
 The metrics side is a thin wrapper around the
 [yabeda](https://github.com/yabeda-rb) ecosystem:
@@ -21,11 +23,14 @@ The metrics side is a thin wrapper around the
 | `yabeda-puma-plugin` | Puma thread-pool / worker stats + the `:yabeda` / `:yabeda_prometheus` Puma plugins |
 | `yabeda-sidekiq` | Sidekiq per-process job metrics + global/Redis-wide queue metrics |
 | `yabeda-prometheus-mmap` | Prometheus text exporter, multiprocess-safe via `prometheus-client-mmap` |
-| `webrick` | HTTP server for the Sidekiq exporter |
+| `webrick` | HTTP server for the non-Puma exporters |
 
-> Scope: **runtime/worker metrics only** (Puma backlog/threads, Sidekiq queues
-> and jobs). HTTP request-level metrics are intentionally out of scope — that is
-> covered by APM.
+The SolidQueue queue gauges are the gem's own: SolidQueue ships no metrics
+endpoint and there is no `yabeda-solid_queue` plugin to wrap.
+
+> Scope: **runtime/worker metrics only** (Puma backlog/threads, Sidekiq and
+> SolidQueue queues and jobs). HTTP request-level metrics are intentionally out
+> of scope — that is covered by APM.
 
 The gem is deliberately **connection-agnostic**: it never reads `REDIS_URL` and
 makes no TLS decisions. Wherever a Redis connection is needed (health checks,
@@ -67,6 +72,9 @@ RailsPodKit::Puma.activate(self)
 RailsPodKit::Sidekiq.install!(config)
 ```
 
+On a SolidQueue stack there is no step 3 — see
+[SolidQueue](#solidqueue-scale-to-zero) instead.
+
 ## Configuration
 
 `RailsPodKit::Config` is an [anyway_config](https://github.com/palkan/anyway_config)
@@ -148,6 +156,8 @@ endpoint every few seconds); pass `silence_controller_log: false` to keep it.
   `sidekiq_queue_latency`, `sidekiq_active_processes`,
   `sidekiq_active_workers_count`, `sidekiq_jobs_retry_count`,
   `sidekiq_jobs_dead_count`, `sidekiq_jobs_scheduled_count`.
+- **SolidQueue (DB-wide):** `solid_queue_backlog`,
+  `solid_queue_latency_seconds`.
 
 Series are intentionally **untagged**: a scraping agent adds
 `service`/`env`/`version` and `kube_*` tags at scrape time, so the gem doesn't
@@ -182,35 +192,53 @@ instances:
     raw_metric_prefix: "sidekiq_"   # sidekiq_queue_latency -> sidekiq.queue_latency
     metrics: [".*"]
     tag_by_endpoint: false
+
+# SolidQueue (the pod publishing the queue gauges) — :9394/metrics
+instances:
+  - openmetrics_endpoint: "http://%%host%%:9394/metrics"
+    namespace: "solid_queue"
+    raw_metric_prefix: "solid_queue_"  # solid_queue_backlog -> solid_queue.backlog
+    metrics: [".*"]
+    tag_by_endpoint: false
 ```
 
+**One prefix per endpoint.** `metrics: [".*"]` ingests *everything* served
+there, so each block above assumes its endpoint carries a single group — which
+is why the SolidQueue gauges are best published from their own pod (below).
+Where one endpoint really must carry two groups (e.g. a web pod exposing both
+`puma_*` and `solid_queue_*`), give **every** instance on it an explicit filter
+— `metrics: ["puma_.*"]` and `metrics: ["solid_queue_.*"]` — or each namespace
+will swallow the other's series un-stripped.
+
 On Kubernetes this is typically wired as pod-annotation autodiscovery. The
 `openmetrics_endpoint` above uses the Datadog Agent's `%%host%%` autodiscovery
 template (resolves to the pod IP) — a **non-k8s adopter** (a plain `conf.yaml`
 check) swaps `%%host%%:9394` for the real `host:port`; everything else
 (`namespace`, `raw_metric_prefix`, `metrics`) is identical.
 
-### Invariant — the endpoint must stay prefix-pure
+### Invariant — every series must carry a group prefix
 
 The Datadog check uses `metrics: [".*"]`, which ingests **every** series on the
 endpoint. `raw_metric_prefix` only *strips* the prefix when present — **it does
 not filter**. So the naming scheme above holds only because the `/metrics`
-endpoint exposes **solely** yabeda-registered series, all sharing the
-`puma_` / `sidekiq_` group prefix:
+endpoint exposes **solely** yabeda-registered series, each carrying its
+`puma_` / `sidekiq_` / `solid_queue_` group prefix:
 
-- the gem registers only the `:puma` and `:sidekiq` yabeda groups (no process,
-  GC, or Ruby-runtime collectors);
+- the gem registers only the `:puma`, `:sidekiq` and `:solid_queue` yabeda
+  groups (no process, GC, or Ruby-runtime collectors);
 - the exposition serves Yabeda's registry only — the Prometheus client's HTTP
   request collector (`http_*`) is **not** mounted on the exporter.
 
-If a series without the `puma_` / `sidekiq_` prefix ever appeared on the
-endpoint, `metrics: [".*"]` would ingest it **un-stripped** under the namespace
-(e.g. `puma.http_requests_total`). **Do not** add cross-cutting metrics (process,
-runtime, HTTP request) to this exporter, and do not mount the Prometheus Rack
-collector on it. If you ever need such metrics, expose them on a separate
-endpoint with its own check rather than polluting this one. Conversely, any new
-metric you *do* add to the `:puma` / `:sidekiq` groups is picked up automatically
-by `[".*"]` — no check change needed.
+If an unprefixed series ever appeared on the endpoint, `metrics: [".*"]` would
+ingest it **un-stripped** under the namespace (e.g. `puma.http_requests_total`).
+**Do not** add cross-cutting metrics (process, runtime, HTTP request) to this
+exporter, and do not mount the Prometheus Rack collector on it. If you ever need
+such metrics, expose them on a separate endpoint with its own check rather than
+polluting this one. Conversely, any new metric you *do* add to an existing group
+is picked up automatically by `[".*"]` — no check change needed.
+
+`spec/rails_pod_kit/metrics_invariant_spec.rb` guards this at both the registry
+and the exposition level.
 
 > ⚠️ Renaming the namespace prefix is a **breaking metric rename** — existing
 > dashboards/monitors built on the old `puma.puma_*` / `sidekiq.sidekiq_*` series
@@ -258,6 +286,13 @@ Sidekiq job class.
 | `sidekiq.job_runtime` | histogram | `queue`, `worker` |
 | `sidekiq.job_latency` | histogram | `queue`, `worker` |
 
+**SolidQueue — DB-wide queue state** (`namespace: solid_queue`):
+
+| canonical Datadog metric | type | functional tags |
+|---|---|---|
+| `solid_queue.backlog` | gauge | `queue` |
+| `solid_queue.latency_seconds` | gauge | `queue` |
+
 ### Where Sidekiq global metrics come from
 
 The global (Redis-wide) queue gauges — `sidekiq_jobs_waiting_count`,
@@ -313,10 +348,155 @@ declares the cluster gauges, starts the exporter and blocks until SIGTERM.
 Booting the full host app just to read a handful of Redis counters would cost
 ~300Mi RSS for nothing — this process sits at ~60Mi.
 
+## SolidQueue: scale-to-zero
+
+SolidQueue's executor has nothing to do while the queue is empty, so it is the
+natural candidate for **scale-to-zero** autoscaling (KEDA, or an HPA). Two things
+stand in the way, and the gem covers both. Everything here is opt-in; requiring
+the gem alone changes nothing.
+
+### 1. The scheduler has to move off the executor
+
+With the executor at zero there is no scheduler, so nothing enqueues the
+recurring and scheduled jobs that would wake one — the queue stays empty because
+it is empty. A k8s CronJob can't take over either: it can't own **dynamic**
+recurring tasks, the ones created and updated at runtime through
+`SolidQueue.schedule_recurring_task`.
+
+The fix is to run the *scheduler alone* on a process that is always on, and let
+the executor be nothing but dispatcher + workers (`bin/jobs` with
+`SOLID_QUEUE_SKIP_RECURRING=true`):
+
+```ruby
+# config/puma.rb — after_booted only runs in the real Puma process, never in a
+# console, a rake task or the test suite.
+after_booted { RailsPodKit::SolidQueue.start_scheduler! }
+at_exit      { RailsPodKit::SolidQueue.stop_scheduler! }
+```
+
+This is deliberately **not** `plugin :solid_queue`. That one runs the full
+supervisor, which forks and whose watchdog takes Puma down when the supervisor
+exits — and a transient Postgres disconnect is enough to cause that
+([rails/solid_queue#512](https://github.com/rails/solid_queue/issues/512)). Here
+a DB blip at worst kills the scheduler thread; a `Concurrent::TimerTask` (the
+same primitive SolidQueue supervises its own processes with) notices on the next
+tick and starts a fresh one, the process itself never notices, and the scheduler
+re-registers on recovery.
+
+Running it on every replica is safe: enqueues stay exactly-once via the unique
+index on `solid_queue_recurring_executions (task_key, run_at)`. Static tasks come
+from `config/recurring.yml` (honouring `SOLID_QUEUE_RECURRING_SCHEDULE`), dynamic
+ones from the DB.
+
+| option | default | meaning |
+|---|---|---|
+| `polling_interval` | `5` | how often the scheduler re-reads the dynamic tasks |
+| `supervision_interval` | `5` | how often we check the scheduler thread is alive |
+| `recurring_schedule_file` | `config/recurring.yml` | static task definitions; skipped when absent |
+
+`start_scheduler!` is **not** gated on `enabled` — that switch owns the metrics
+exporter, and an app may well want the scheduler with metrics off. What keeps it
+out of consoles and specs is *where* you call it from.
+
+### 2. Queue depth has to be visible
+
+SolidQueue publishes no metrics, so the autoscaler and the dashboards have
+nothing to read. `install_metrics!` adds two gauges, computed **at scrape time**
+from the SolidQueue tables (a yabeda `collect` block — no background thread, no
+cached snapshot):
+
+```ruby
+# config/initializers/rails_pod_kit.rb
+RailsPodKit::SolidQueue.install_metrics!
+```
+
+| metric | meaning |
+|---|---|
+| `solid_queue_backlog` | how many jobs could be claimed right now, per `queue` |
+| `solid_queue_latency_seconds` | how long the oldest of them has been waiting, per `queue` |
+
+"Claimable right now" is ready executions **plus** scheduled ones whose time has
+come — the dispatcher has only to move those across. A scheduled job's wait is
+measured from its `scheduled_at`, not its `created_at`: enqueuing a week ahead of
+the slot doesn't make it a week late.
+
+Both matter, and neither alone is enough: backlog misses a small-but-stalled
+queue, latency misses a large-but-moving one.
+
+**An idle system reads 0, not no-data.** A queue that drains is explicitly zeroed
+rather than left pinned at its last reading, and the zeroing starts from a
+baseline of every queue the app is known to use — discovered once per process
+from the jobs table, or pinned by the host:
+
+```ruby
+RailsPodKit::SolidQueue.install_metrics!(queues: %w[default mailers])
+```
+
+Without that baseline a process booting while the queue is empty — the steady
+state of a scale-to-zero deployment — would publish no series at all, since a
+gauge only exists once it has been set. Discovery is best-effort: the jobs table
+is bounded by `clear_finished_jobs_after`, so a queue idle for longer than the
+retention window leaves no trace in it. Pin `queues:` where the zero has to be
+guaranteed.
+
+**A collection error is always reported** (logged, plus handed to
+`Rails.error`), then either swallowed or raised:
+
+| | |
+|---|---|
+| `fail_scrape_on_error: false` (default) | serve the last reading. Right on an endpoint shared with the Puma or Sidekiq series, where failing the response would lose those too — at the cost of a stale gauge a consumer cannot tell apart from a live one. |
+| `fail_scrape_on_error: true` (`run_exporter!`'s default) | fail the scrape. Right on the dedicated pod, where there is nothing else to protect: the gauges go to no-data and the scrape failure shows up in the scraper's own `up` series. |
+
+### The always-on pod
+
+The cleanest home for both is a **1-replica Deployment** that hosts the scheduler
+and publishes the gauges, decoupled from the web and the executor — so the
+signals survive either scaling to zero, and each series has exactly one source.
+`run_exporter!` is that process: it declares the gauges, starts the scheduler,
+serves `/metrics` and blocks until SIGTERM (winding the scheduler down so it
+deregisters rather than expiring).
+
+Unlike the Sidekiq global exporter it is **not** Rails-free — SolidQueue is
+ActiveRecord-backed and reads the app's own tables — so the host's entrypoint
+boots the environment first. The gem ships no executable; e.g.
+`bin/solid-queue-pod`:
+
+```ruby
+#!/usr/bin/env ruby
+require_relative '../config/environment'
+
+RailsPodKit::SolidQueue.run_exporter!
+```
+
+```
+command: ["bin/solid-queue-pod"]
+```
+
+Pass `scheduler: false` to serve the gauges only, on an app whose web process
+already hosts the scheduler, and `metrics:` to override the gauge options (this
+endpoint is the collector's own, so `fail_scrape_on_error` defaults to `true`
+here):
+
+```ruby
+RailsPodKit::SolidQueue.run_exporter!(scheduler: false, metrics: { queues: %w[default mailers] })
+```
+
+Keep the endpoint single-prefix (see
+[One prefix per endpoint](#datadog-naming--the-canonical-metric-set)) — that pod
+serves `solid_queue_*` and nothing else, so the check config needs no filters.
+
 ## Caveats
 
 - **Puma only.** The Puma plugins only activate under Puma; under any other
-  app server the in-process `/metrics` endpoint is **not** exposed.
+  app server the in-process `/metrics` endpoint is **not** exposed. The non-Puma
+  entry points (Sidekiq, the global exporter, the SolidQueue pod) serve it from
+  a WEBrick thread instead, started at most once per process.
+- **SolidQueue and Sidekiq are the host's.** The gem depends on neither; the
+  SolidQueue integration is inert until you call it, exactly like the Sidekiq one.
+- **Queue-gauge query cost.** The gauges run four small grouped aggregates per
+  scrape. `MIN(created_at)` is not covered by SolidQueue's indexes, so on a
+  backlog of many thousands of rows it is a scan — cheap at a normal scrape
+  interval, worth knowing about if you scrape aggressively.
 - **Puma control app.** `yabeda-puma-plugin` reads Puma's thread-pool stats
   through Puma's control app, so `Puma.activate` activates one on a
   localhost-only socket (`no_token: true`, never network-exposed).
@@ -352,6 +532,10 @@ curl -s localhost:9394/metrics | grep '^puma_'
 bundle exec sidekiq
 curl -s localhost:9394/metrics | grep '^sidekiq_'
 
+# SolidQueue: solid_queue_* series (enqueue a job first so the queue isn't empty)
+bin/solid-queue-pod
+curl -s localhost:9394/metrics | grep '^solid_queue_'
+
 # Health endpoint
 curl -s localhost:3000/healthz -H 'Accept: application/json'
 ```

data/VERSION

@@ -1 +1 @@
-0.0.3
+0.1.1

data/lib/rails_pod_kit/error_reporter.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+module RailsPodKit
+  # Where the gem's background work (the SolidQueue scheduler supervisor, the
+  # scrape-time metric collectors) sends the errors it swallows. Those errors
+  # must never propagate — one would kill the supervising timer or fail the whole
+  # /metrics response — but they must not vanish either.
+  #
+  # Always logs, and additionally hands the error to the Rails error reporter
+  # when there is one, so the host's Rollbar / Sentry / Datadog subscriber picks
+  # it up. The log line is not redundant: `Rails.error.report` only fans out to
+  # subscribers, so on an app with none — or with one that is not wired in a
+  # given environment — the failure would otherwise leave no trace at all, and
+  # the only symptom of a broken collector is a gauge quietly serving a stale
+  # value.
+  module ErrorReporter
+    module_function
+
+    def report(error, source:)
+      message = "[#{source}] #{error.class}: #{error.message}"
+      logger ? logger.error(message) : warn(message)
+
+      rails_reporter&.report(error, handled: true, source: source)
+    end
+
+    def logger
+      ::Rails.logger if defined?(::Rails) && ::Rails.respond_to?(:logger)
+    end
+
+    def rails_reporter
+      ::Rails.error if defined?(::Rails) && ::Rails.respond_to?(:error)
+    end
+  end
+end

data/lib/rails_pod_kit/exporter.rb

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+require 'rails_pod_kit/config'
+
+module RailsPodKit
+  # The in-process WEBrick /metrics server, shared by every non-Puma entry
+  # point: the Sidekiq worker, the Rails-free global exporter and the SolidQueue
+  # exporter. Under Puma the exporter comes from the `:yabeda_prometheus` plugin
+  # instead (see RailsPodKit::Puma), so this is never used there.
+  module Exporter
+    module_function
+
+    # Starts the background exporter and returns whether it did. Idempotent: the
+    # latch keeps a re-entrant boot from double-binding the port.
+    def start!
+      return false unless RailsPodKit.enabled?
+      return false if @started
+
+      require 'yabeda/prometheus/mmap'
+
+      # `start_metrics_server!` reads the bind port from the env, so publish the
+      # configured one first.
+      ENV['PROMETHEUS_EXPORTER_PORT'] ||= RailsPodKit.config.port.to_s
+      # Drop the exporter's per-scrape access log (Rack::CommonLogger, which the
+      # mmap exporter mounts unless this is exactly 'false'). See
+      # Config#silence_exporter_access_log.
+      ENV['PROMETHEUS_EXPORTER_LOG_REQUESTS'] = 'false' if RailsPodKit.config.silence_exporter_access_log
+
+      Yabeda::Prometheus::Exporter.start_metrics_server!
+      @started = true
+    end
+
+    def started?
+      !!@started
+    end
+
+    # Test/reset hook — drops the "already started" latch.
+    def reset!
+      @started = false
+    end
+  end
+end

data/lib/rails_pod_kit/sidekiq.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require 'rails_pod_kit/config'
+require 'rails_pod_kit/exporter'
 
 module RailsPodKit
   # Sidekiq integration. Called from inside `Sidekiq.configure_server`:
@@ -81,19 +82,11 @@ module RailsPodKit
       apply_retries_segmentation!
     end
 
-    # Starts the background WEBrick exporter. `start_metrics_server!` reads the
-    # bind port from PROMETHEUS_EXPORTER_PORT, so we set it from config first.
-    # Guarded so a re-entrant Sidekiq boot doesn't try to double-bind the port.
+    # Starts the background WEBrick exporter shared with the other non-Puma
+    # entry points; guarded there so a re-entrant Sidekiq boot can't double-bind
+    # the port.
     def start_metrics_server!
-      return if @server_started
-
-      ENV['PROMETHEUS_EXPORTER_PORT'] ||= RailsPodKit.config.port.to_s
-      # Drop the WEBrick exporter's per-scrape access log (Rack::CommonLogger,
-      # which the mmap exporter mounts unless this is exactly 'false'). See
-      # Config#silence_exporter_access_log.
-      ENV['PROMETHEUS_EXPORTER_LOG_REQUESTS'] = 'false' if RailsPodKit.config.silence_exporter_access_log
-      Yabeda::Prometheus::Exporter.start_metrics_server!
-      @server_started = true
+      RailsPodKit::Exporter.start!
     end
   end
 end

data/lib/rails_pod_kit/solid_queue/metrics.rb

@@ -0,0 +1,186 @@
+# frozen_string_literal: true
+
+require 'rails_pod_kit/config'
+require 'rails_pod_kit/error_reporter'
+
+module RailsPodKit
+  module SolidQueue
+    # DB-backed queue gauges for SolidQueue, in the shape yabeda-sidekiq exposes
+    # for Redis: SolidQueue ships no metrics endpoint and there is no
+    # `yabeda-solid_queue`.
+    #
+    # Two series, both per `queue`, both computed at scrape time from a yabeda
+    # `collect` block — no background thread, no cached snapshot:
+    #
+    #   solid_queue_backlog          how many jobs could be claimed right now
+    #   solid_queue_latency_seconds  how long the oldest of them has been waiting
+    #
+    # "Claimable right now" is ready executions plus scheduled ones whose time
+    # has come (the dispatcher has yet to move them across). Backlog alone misses
+    # a small-but-stalled queue and latency alone misses a large-but-moving one,
+    # so the pair is what dashboards, alerts and an HPA feed actually need.
+    #
+    # `::SolidQueue` is the host's — the gem declares no dependency on it and
+    # nothing here loads until the host calls `install!`.
+    module Metrics
+      SOURCE = 'rails_pod_kit.solid_queue_metrics'
+
+      module_function
+
+      # Declares the gauges and registers the scrape-time collector. Safe before
+      # or after `Yabeda.configure!` (yabeda replays configurators either way).
+      #
+      # `queues:` pins the zero baseline (see #baseline_queues) instead of
+      # discovering it. `fail_scrape_on_error:` makes a collection failure fail
+      # the whole response rather than serve the last reading — right when this
+      # collector owns the endpoint, wrong when it shares one (see #collect!).
+      #
+      # Declaration is one-shot but the options are not: the dedicated pod boots
+      # the host's initializers before `run_exporter!` runs, so by the time the
+      # pod asks for `fail_scrape_on_error` the app's own `install_metrics!` has
+      # normally already declared the gauges. An option given here always wins;
+      # one left out keeps whatever an earlier call set.
+      def install!(queues: nil, fail_scrape_on_error: nil)
+        @baseline_queues = queues unless queues.nil?
+        @fail_scrape_on_error = fail_scrape_on_error unless fail_scrape_on_error.nil?
+
+        return false if @installed
+
+        require 'yabeda'
+        declare!
+        @installed = true
+      end
+
+      def declare!
+        Yabeda.configure do
+          group :solid_queue do
+            gauge :backlog,
+                  tags: %i[queue],
+                  comment: 'Jobs claimable right now: ready executions plus scheduled ones whose time has come'
+            gauge :latency,
+                  unit: :seconds,
+                  tags: %i[queue],
+                  comment: 'How long the oldest claimable job has been waiting'
+
+            collect { RailsPodKit::SolidQueue::Metrics.collect! }
+          end
+        end
+      end
+
+      # Called by yabeda on every scrape.
+      #
+      # An error is always reported, and then either swallowed or re-raised
+      # depending on who owns the endpoint. Swallowing keeps a transient DB
+      # failure from taking the Puma series down with it on a shared endpoint —
+      # at the cost of serving the last reading, which a consumer cannot tell
+      # apart from a live one. On the dedicated pod (`run_exporter!`) there is
+      # nothing else to protect, so failing the scrape is the honest answer: the
+      # gauges go to no-data and the scraper's own `up` series carries the
+      # failure.
+      def collect!
+        now = ::Time.now.utc
+        with_connection { publish_all(claimable_by_queue(now), now) }
+      rescue StandardError => e
+        ErrorReporter.report(e, source: SOURCE)
+        raise if @fail_scrape_on_error
+      end
+
+      # Collection runs on the exporter's HTTP thread, which would otherwise
+      # check out a connection and pin it there for the life of the process.
+      def with_connection(&)
+        ::ActiveRecord::Base.connection_pool.with_connection(&)
+      end
+
+      # => { "default" => { backlog: 12, waiting_since: <Time> }, … }
+      def claimable_by_queue(now)
+        merge(ready_rows, due_rows(now))
+      end
+
+      def ready_rows
+        rows(::SolidQueue::ReadyExecution.all, :created_at)
+      end
+
+      # A scheduled execution whose time has come is claimable too. Its wait
+      # started at `scheduled_at`, not `created_at` — a job enqueued a week ahead
+      # of its slot is not a week late.
+      def due_rows(now)
+        rows(::SolidQueue::ScheduledExecution.where(scheduled_at: ..now), :scheduled_at)
+      end
+
+      # Two grouped aggregates over an indexed, normally-small table. Both go
+      # through ActiveRecord's calculations so the timestamp comes back
+      # type-cast on every adapter.
+      def rows(relation, waiting_since_column)
+        counts = relation.group(:queue_name).count
+        oldest = relation.group(:queue_name).minimum(waiting_since_column)
+
+        counts.map { |queue, count| [queue, count, oldest[queue]] }
+      end
+
+      def merge(*row_sets)
+        row_sets.flatten(1).each_with_object({}) do |(queue, count, waiting_since), acc|
+          entry = acc[queue] ||= { backlog: 0, waiting_since: nil }
+          entry[:backlog] += count
+          entry[:waiting_since] = [entry[:waiting_since], waiting_since].compact.min
+        end
+      end
+
+      def publish_all(queues, now)
+        queues.each do |queue, entry|
+          publish(queue, backlog: entry[:backlog], latency: age_in_seconds(entry[:waiting_since], now))
+        end
+
+        zero_drained_queues(queues.keys)
+      end
+
+      def publish(queue, backlog:, latency:)
+        Yabeda.solid_queue.backlog.set({ queue: queue }, backlog)
+        Yabeda.solid_queue.latency.set({ queue: queue }, latency)
+      end
+
+      # A gauge keeps its last value per label set, so a queue that just drained
+      # would stay pinned at its final backlog forever — the exact reading that
+      # would keep an alert firing on an idle system. Track the label sets this
+      # process has published and zero the ones missing from this round.
+      def zero_drained_queues(current)
+        seen = seen_queues
+        (seen - current).each { |queue| publish(queue, backlog: 0, latency: 0) }
+        @seen_queues = seen | current
+      end
+
+      # Seeded with the baseline, so the zeroing above also covers queues this
+      # process has never seen busy. A gauge only exists once it has been set:
+      # without the seed an exporter that boots while the queue is empty — the
+      # steady state of a scale-to-zero deployment — publishes no series at all,
+      # and every consumer reads no-data where it should read 0.
+      def seen_queues
+        @seen_queues ||= baseline_queues
+      end
+
+      # The queues the app is known to use, pinned by the host or discovered
+      # once per process from the jobs table (one index scan, never repeated).
+      # Discovery is best-effort by construction: that table is bounded by
+      # `clear_finished_jobs_after`, so a queue idle for longer than the
+      # retention window leaves no trace. Pin `queues:` where the zero has to be
+      # guaranteed.
+      def baseline_queues
+        @baseline_queues || ::SolidQueue::Job.distinct.pluck(:queue_name).compact
+      end
+
+      def age_in_seconds(waiting_since, now)
+        return 0 if waiting_since.nil?
+
+        [(now - waiting_since).to_f, 0].max
+      end
+
+      # Test/reset hook — drops the published-label-set memo and the install
+      # options.
+      def reset!
+        @seen_queues = nil
+        @baseline_queues = nil
+        @fail_scrape_on_error = false
+        @installed = false
+      end
+    end
+  end
+end

data/lib/rails_pod_kit/solid_queue/scheduler_runner.rb

@@ -0,0 +1,130 @@
+# frozen_string_literal: true
+
+require 'concurrent'
+require 'active_support/configuration_file'
+require 'active_support/core_ext/hash/keys'
+
+require 'rails_pod_kit/config'
+require 'rails_pod_kit/error_reporter'
+
+module RailsPodKit
+  module SolidQueue
+    # Runs a SolidQueue *scheduler* — and only the scheduler — as a supervised
+    # background thread, so the process running jobs can scale to zero.
+    #
+    # Scaling the executor to zero is otherwise a chicken-and-egg problem: with
+    # no executor there is no scheduler, so nothing enqueues the recurring or
+    # scheduled jobs that would wake one. A k8s CronJob can't take over either,
+    # because it can't own *dynamic* recurring tasks (created and updated at
+    # runtime through `SolidQueue.schedule_recurring_task`). Moving just the
+    # scheduler onto an always-on process — the web, or the exporter pod —
+    # breaks the cycle: it keeps evaluating the crons and enqueueing, and the
+    # queue depth wakes the executor.
+    #
+    # Deliberately *not* the full supervisor (`plugin :solid_queue`): that one
+    # forks, and its Puma watchdog takes the host process down when the
+    # supervisor exits — which a transient Postgres disconnect is enough to
+    # cause (rails/solid_queue#512). Here a DB blip at worst kills the scheduler
+    # thread; the supervising timer notices on its next tick and starts a fresh
+    # one, and the host process never notices.
+    #
+    # Running it on every replica is safe: enqueues stay exactly-once via the
+    # unique index on `solid_queue_recurring_executions (task_key, run_at)`.
+    class SchedulerRunner
+      # How often the scheduler re-reads the dynamic tasks from the DB.
+      DEFAULT_POLLING_INTERVAL = 5
+      # How often we check that the scheduler thread is still alive.
+      DEFAULT_SUPERVISION_INTERVAL = 5
+
+      SOURCE = 'rails_pod_kit.solid_queue_scheduler'
+
+      def initialize(polling_interval: DEFAULT_POLLING_INTERVAL,
+                     supervision_interval: DEFAULT_SUPERVISION_INTERVAL,
+                     recurring_schedule_file: nil)
+        @polling_interval = polling_interval
+        @supervision_interval = supervision_interval
+        @recurring_schedule_file = recurring_schedule_file
+        @stopping = false
+      end
+
+      # Starts the supervisor, which starts the scheduler on its first (immediate)
+      # tick and returns without blocking.
+      def start
+        @supervisor = ::Concurrent::TimerTask.new(
+          execution_interval: @supervision_interval,
+          run_now: true
+        ) { supervise }
+        @supervisor.execute
+        self
+      end
+
+      # Graceful stop: drop the supervisor first so it can't resurrect the
+      # scheduler, then wind the scheduler down (unschedule its timers and
+      # deregister the process, instead of leaving a row to expire).
+      def stop
+        @stopping = true
+        @supervisor&.shutdown
+        @supervisor = nil
+        @scheduler&.stop
+        @scheduler = nil
+      end
+
+      def running?
+        !!@scheduler&.alive?
+      end
+
+      private
+
+      # Concurrent::TimerTask silently drops a raising block, so report here and
+      # let the next tick retry — a scheduler that can't start because Postgres
+      # is down must keep trying.
+      def supervise
+        ensure_scheduler_running
+      rescue StandardError => e
+        ErrorReporter.report(e, source: SOURCE)
+      end
+
+      # Idempotent: starts the scheduler on the first tick, and restarts it only
+      # once its thread has actually died.
+      def ensure_scheduler_running
+        return if @stopping || @scheduler&.alive?
+
+        scheduler = ::SolidQueue::Scheduler.new(
+          recurring_tasks: static_recurring_tasks,
+          dynamic_tasks_enabled: true,
+          polling_interval: @polling_interval
+        )
+        scheduler.mode = :async
+        scheduler.start # spawns the scheduler's own thread and returns
+        @scheduler = scheduler
+      end
+
+      # The static tasks from config/recurring.yml; the dynamic ones come from
+      # the DB via `dynamic_tasks_enabled`. `SolidQueue::Configuration#recurring_tasks`
+      # is private, so parse the file with the same public helper it uses.
+      def static_recurring_tasks
+        path = recurring_schedule_file
+        return [] unless path && ::File.exist?(path)
+
+        config = ::ActiveSupport::ConfigurationFile.parse(path).deep_symbolize_keys
+        config.fetch(environment.to_sym, {}).filter_map do |key, options|
+          ::SolidQueue::RecurringTask.from_configuration(key, **options) if options&.key?(:schedule)
+        end
+      end
+
+      # Same resolution order SolidQueue's own CLI uses.
+      def recurring_schedule_file
+        return @recurring_schedule_file if @recurring_schedule_file
+        return nil unless defined?(::Rails) && ::Rails.respond_to?(:root) && ::Rails.root
+
+        ::Rails.root.join(ENV.fetch('SOLID_QUEUE_RECURRING_SCHEDULE', 'config/recurring.yml'))
+      end
+
+      def environment
+        return ::Rails.env if defined?(::Rails) && ::Rails.respond_to?(:env)
+
+        ENV.fetch('RAILS_ENV', 'development')
+      end
+    end
+  end
+end

data/lib/rails_pod_kit/solid_queue.rb

@@ -0,0 +1,109 @@
+# frozen_string_literal: true
+
+require 'rails_pod_kit/config'
+require 'rails_pod_kit/exporter'
+require 'rails_pod_kit/solid_queue/metrics'
+require 'rails_pod_kit/solid_queue/scheduler_runner'
+
+module RailsPodKit
+  # SolidQueue integration — the pieces a Rails 8 app needs to run its job
+  # executor with scale-to-zero autoscaling (KEDA/HPA). All three are opt-in;
+  # requiring this file only defines them.
+  #
+  #   install_metrics!    the `solid_queue_backlog` / `solid_queue_latency_seconds`
+  #                       gauges on /metrics (see Metrics)
+  #   start_scheduler!    the SolidQueue scheduler as a supervised background
+  #                       thread on an always-on process (see SchedulerRunner)
+  #   run_exporter!       both of the above in a dedicated always-on pod, with
+  #                       the /metrics server, blocking until SIGTERM
+  #
+  # Nothing here loads `solid_queue` itself: it is the host's dependency, the
+  # same way Puma and Sidekiq are.
+  #
+  # Wiring the two halves separately, when the always-on process is the web:
+  #
+  #   # config/initializers/rails_pod_kit.rb
+  #   RailsPodKit::SolidQueue.install_metrics!
+  #
+  #   # config/puma.rb — after_booted only runs in the real Puma process,
+  #   # never in a console, a rake task or the test suite.
+  #   after_booted { RailsPodKit::SolidQueue.start_scheduler! }
+  #   at_exit      { RailsPodKit::SolidQueue.stop_scheduler! }
+  module SolidQueue
+    module_function
+
+    # Declares the queue gauges. Call from an initializer, in whichever process
+    # should publish them — see the README on keeping one source per series.
+    #
+    # Options are Metrics.install!'s: `queues:` to pin the zero baseline,
+    # `fail_scrape_on_error:` to fail the scrape instead of serving the last
+    # reading when the DB is unreachable (the default only holds on an endpoint
+    # this collector shares with another group).
+    def install_metrics!(**)
+      Metrics.install!(**)
+    end
+
+    # Starts the supervised scheduler thread and returns the runner. Idempotent:
+    # a second call returns the running one rather than starting a second
+    # scheduler in the same process.
+    #
+    # Not gated on `RailsPodKit.enabled?` — that switch owns the metrics
+    # exporter, and an app may well want the scheduler with metrics turned off.
+    # The guard against starting one in a console or in specs is *where* you
+    # call this from (`after_booted`, or the exporter entrypoint).
+    def start_scheduler!(**)
+      return scheduler_runner if scheduler_runner
+
+      @scheduler_runner = SchedulerRunner.new(**).start
+    end
+
+    def stop_scheduler!
+      @scheduler_runner&.stop
+      @scheduler_runner = nil
+    end
+
+    def scheduler_runner
+      @scheduler_runner
+    end
+
+    # Runs the dedicated always-on SolidQueue pod and blocks until SIGTERM:
+    # the queue gauges, the scheduler thread and the /metrics server in one
+    # 1-replica Deployment, so both survive the web and the executor scaling to
+    # zero and each series has exactly one source.
+    #
+    # Unlike GlobalExporter this one is *not* Rails-free — SolidQueue is
+    # ActiveRecord-backed and reads the app's own tables — so the host's
+    # entrypoint boots the environment first, e.g. `bin/solid-queue-pod`:
+    #
+    #   #!/usr/bin/env ruby
+    #   require_relative '../config/environment'
+    #   RailsPodKit::SolidQueue.run_exporter!
+    #
+    # Pass `scheduler: false` to serve the gauges only (an app whose web process
+    # already hosts the scheduler), and `metrics:` to override the gauge options
+    # — this endpoint is the collector's own, so a collection failure fails the
+    # scrape here rather than serving the last reading.
+    def run_exporter!(scheduler: true, metrics: {}, **scheduler_options)
+      install_metrics!(fail_scrape_on_error: true, **metrics)
+      start_scheduler!(**scheduler_options) if scheduler
+
+      warn '[rails_pod_kit] disabled — /metrics not served by the SolidQueue exporter' unless Exporter.start!
+
+      require 'yabeda'
+      Yabeda.configure! unless Yabeda.already_configured?
+
+      await_shutdown
+      stop_scheduler!
+    end
+
+    # Blocks the main thread (the exporter and the scheduler both run on their
+    # own) until the kubelet signals. A self-pipe rather than a Queue or a
+    # Mutex: writing to an IO is one of the few things safe to do from a trap
+    # handler.
+    def await_shutdown
+      reader, writer = IO.pipe
+      %w[INT TERM].each { |signal| Signal.trap(signal) { writer.puts(signal) } }
+      reader.gets
+    end
+  end
+end

data/lib/rails_pod_kit.rb

@@ -6,9 +6,10 @@ require 'rails_pod_kit/config'
 # RailsPodKit packages the operational endpoints a Rails pod needs to be a good
 # Kubernetes citizen behind a single, opinionated entry point:
 #
-# - Prometheus metrics: Puma and Sidekiq runtime metrics in Prometheus text
-#   format on an in-process /metrics endpoint (default port 9394) — no sidecar,
-#   no separate collector process. A thin wrapper around the yabeda ecosystem.
+# - Prometheus metrics: Puma, Sidekiq and SolidQueue runtime metrics in
+#   Prometheus text format on an in-process /metrics endpoint (default port
+#   9394) — no sidecar, no separate collector process. A thin wrapper around the
+#   yabeda ecosystem, plus the SolidQueue queue gauges yabeda has no plugin for.
 # - Health checks: an opinionated health-monitor-rails configuration serving
 #   liveness/readiness/startup probes on /healthz (see RailsPodKit::Health),
 #   auto-mounted by the gem's Railtie.
@@ -22,8 +23,9 @@ require 'rails_pod_kit/config'
 #   - config/initializers/sidekiq.rb          -> RailsPodKit::Sidekiq.install!(config)
 #   - config/initializers/rails_pod_kit.rb    -> RailsPodKit.configure { ... }
 #                                                RailsPodKit::Health.install!(redis: ...)
-# plus, only when running the dedicated global exporter, a host-owned
-# entrypoint (e.g. bin/pod-exporter) calling GlobalExporter.run!(redis: ...).
+# plus, only when running a dedicated always-on exporter, a host-owned
+# entrypoint calling GlobalExporter.run!(redis: ...) (Sidekiq, Rails-free) or
+# SolidQueue.run_exporter! (SolidQueue, needs the app's ActiveRecord models).
 #
 # The gem is deliberately connection-agnostic: it never reads REDIS_URL or
 # makes TLS decisions. The host injects its Redis options where needed
@@ -57,5 +59,6 @@ require 'rails_pod_kit/config'
 require 'yabeda/prometheus/mmap'
 
 require 'rails_pod_kit/sidekiq'
+require 'rails_pod_kit/solid_queue'
 require 'rails_pod_kit/health'
 require 'rails_pod_kit/railtie'

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rails_pod_kit
 version: !ruby/object:Gem::Version
-  version: 0.0.3
+  version: 0.1.1
 platform: ruby
 authors:
 - Fabio Napoleoni
@@ -93,6 +93,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: concurrent-ruby
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.2'
 - !ruby/object:Gem::Dependency
   name: health-monitor-rails
   requirement: !ruby/object:Gem::Requirement
@@ -123,10 +137,12 @@ dependencies:
         version: '5.0'
 description: |
   Packages the yabeda ecosystem and health-monitor-rails into a single,
-  opinionated kit for running Rails applications on Kubernetes: Puma and
-  Sidekiq runtime metrics in Prometheus text format on an in-process /metrics
-  endpoint (default port 9394), plus a /healthz endpoint wired for
-  liveness/readiness/startup probes. No sidecar, no separate collector.
+  opinionated kit for running Rails applications on Kubernetes: Puma, Sidekiq
+  and SolidQueue runtime metrics in Prometheus text format on an in-process
+  /metrics endpoint (default port 9394), plus a /healthz endpoint wired for
+  liveness/readiness/startup probes. No sidecar, no separate collector. For
+  SolidQueue it also ships the supervised scheduler thread that lets the job
+  executor scale to zero.
 email:
 - f.napoleoni@gmail.com
 executables: []
@@ -138,11 +154,16 @@ files:
 - VERSION
 - lib/rails_pod_kit.rb
 - lib/rails_pod_kit/config.rb
+- lib/rails_pod_kit/error_reporter.rb
+- lib/rails_pod_kit/exporter.rb
 - lib/rails_pod_kit/global_exporter.rb
 - lib/rails_pod_kit/health.rb
 - lib/rails_pod_kit/puma.rb
 - lib/rails_pod_kit/railtie.rb
 - lib/rails_pod_kit/sidekiq.rb
+- lib/rails_pod_kit/solid_queue.rb
+- lib/rails_pod_kit/solid_queue/metrics.rb
+- lib/rails_pod_kit/solid_queue/scheduler_runner.rb
 - lib/rails_pod_kit/version.rb
 homepage: https://github.com/fabn/rails_pod_kit
 licenses:
@@ -167,6 +188,6 @@ required_rubygems_version: !ruby/object:Gem::Requirement
 requirements: []
 rubygems_version: 3.6.9
 specification_version: 4
-summary: 'Operational endpoints for Rails pods: Prometheus metrics (Puma + Sidekiq)
-  and health checks'
+summary: 'Operational endpoints for Rails pods: Prometheus metrics (Puma, Sidekiq,
+  SolidQueue) and health checks'
 test_files: []