rails_pod_kit 0.0.3 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 24c0e428faf1a3c46646a1d7a0e693b5cde8373f6aea61f9dc005d6b6dcba192
4
- data.tar.gz: 968fc08545c75f5b22f91c2c882068b20069d7bb2bb6f48d1abf2aa09f4bc71b
3
+ metadata.gz: 85fea66a44c5b2198b5c3c7bfd05f5f1f4c3655559ed701748661d0e148650a5
4
+ data.tar.gz: 9c921528f839ea1d5b962035600316ea37affe17cf8fdc910453c30108a248dd
5
5
  SHA512:
6
- metadata.gz: 266b27abb0911f031e54522dfcea870d603e1ec143465b4441a206590654087c1604a9aeb4438745bc10600bf3696b7d4938c0c936e0145322b76cb0279a5513
7
- data.tar.gz: dedc4665fb51d38b9e713caead650ee875108aed4a24e9f27bb3053f2ec0152bdcc0f3840cc0ac76056858f330b723c502bc5f66d726261db47a7033e6ae2142
6
+ metadata.gz: d7fbfee31ce338a6dd7c11d65214dc442cd2cb261848ae2403a415dc4a6e1333fa55b39e2d06441b4e05fcacff2ec3971ee009613a532f384fcc19fbe84f070d
7
+ data.tar.gz: 894f99f9fba4596aa7d69e143f5fa23d2ebc64ef85f6b7cac310096c902005dd003103cb12c3933e49dfe26263c68f9656cfc3e68c0d9551a8f118e79496fd48
data/README.md CHANGED
@@ -3,14 +3,17 @@
3
3
  The operational endpoints a Rails pod needs to be a good Kubernetes citizen,
4
4
  packaged behind a single, opinionated entry point:
5
5
 
6
- - **Prometheus metrics** for Puma and Sidekiq, served **in-process** on a
7
- single `/metrics` endpoint (default port **9394**) in both the web (Puma) and
8
- worker (Sidekiq) processes — no sidecar, no separate collector process. A
9
- metrics agent (e.g. the Datadog Agent via OpenMetrics autodiscovery) scrapes
10
- the pod directly.
6
+ - **Prometheus metrics** for Puma, Sidekiq and SolidQueue, served **in-process**
7
+ on a single `/metrics` endpoint (default port **9394**) in the web (Puma) and
8
+ worker processes — no sidecar, no separate collector process. A metrics agent
9
+ (e.g. the Datadog Agent via OpenMetrics autodiscovery) scrapes the pod
10
+ directly.
11
11
  - **Health checks** on `/healthz` (database, cache, optionally Redis and Sidekiq),
12
12
  wired for Kubernetes startup/liveness/readiness probes — a thin, opinionated
13
13
  wrapper around [health-monitor-rails](https://github.com/lbeder/health-monitor-rails).
14
+ - **Scheduler hosting for scale-to-zero**, so a job executor can be autoscaled to
15
+ zero without stranding its recurring and scheduled jobs: a supervised
16
+ SolidQueue scheduler thread, and a supervised sidekiq-cron poller for Sidekiq.
14
17
 
15
18
  The metrics side is a thin wrapper around the
16
19
  [yabeda](https://github.com/yabeda-rb) ecosystem:
@@ -21,11 +24,14 @@ The metrics side is a thin wrapper around the
21
24
  | `yabeda-puma-plugin` | Puma thread-pool / worker stats + the `:yabeda` / `:yabeda_prometheus` Puma plugins |
22
25
  | `yabeda-sidekiq` | Sidekiq per-process job metrics + global/Redis-wide queue metrics |
23
26
  | `yabeda-prometheus-mmap` | Prometheus text exporter, multiprocess-safe via `prometheus-client-mmap` |
24
- | `webrick` | HTTP server for the Sidekiq exporter |
27
+ | `webrick` | HTTP server for the non-Puma exporters |
25
28
 
26
- > Scope: **runtime/worker metrics only** (Puma backlog/threads, Sidekiq queues
27
- > and jobs). HTTP request-level metrics are intentionally out of scope — that is
28
- > covered by APM.
29
+ The SolidQueue queue gauges are the gem's own: SolidQueue ships no metrics
30
+ endpoint and there is no `yabeda-solid_queue` plugin to wrap.
31
+
32
+ > Scope: **runtime/worker metrics only** (Puma backlog/threads, Sidekiq and
33
+ > SolidQueue queues and jobs). HTTP request-level metrics are intentionally out
34
+ > of scope — that is covered by APM.
29
35
 
30
36
  The gem is deliberately **connection-agnostic**: it never reads `REDIS_URL` and
31
37
  makes no TLS decisions. Wherever a Redis connection is needed (health checks,
@@ -67,19 +73,24 @@ RailsPodKit::Puma.activate(self)
67
73
  RailsPodKit::Sidekiq.install!(config)
68
74
  ```
69
75
 
76
+ On a SolidQueue stack there is no step 3 — see
77
+ [SolidQueue](#solidqueue-scale-to-zero) instead.
78
+
70
79
  ## Configuration
71
80
 
72
81
  `RailsPodKit::Config` is an [anyway_config](https://github.com/palkan/anyway_config)
73
82
  config: besides the `RailsPodKit.configure` block (which runs last and wins),
74
83
  every setting can come from a `RAILS_POD_KIT_*` env var (e.g.
75
84
  `RAILS_POD_KIT_ENABLED=false`, `RAILS_POD_KIT_PORT=9500`,
76
- `RAILS_POD_KIT_SIDEKIQ_GLOBAL_METRICS=off`) or an optional
85
+ `RAILS_POD_KIT_SIDEKIQ_GLOBAL_METRICS=off`,
86
+ `RAILS_POD_KIT_SCHEDULER_ENABLED=false`) or an optional
77
87
  `config/rails_pod_kit.yml` — handy for the Rails-free exporter pod, which runs
78
88
  no initializers.
79
89
 
80
90
  | setting | default | meaning |
81
91
  |---------|---------|---------|
82
- | `enabled` | on, except in `test` | master switch; `false` ⇒ no exporter, no port bound |
92
+ | `enabled` | on, except in `test` | master switch for the **exporter**; `false` ⇒ no exporter, no port bound |
93
+ | `scheduler_enabled` | on, everywhere | kill switch for the **hosted schedulers** (sidekiq-cron poller, SolidQueue scheduler thread). Separate from `enabled` — an app may want one without the other, and this is the one you may need to flip in a hurry, since the scheduler is a single point of failure for the whole schedule. On even in `test`: nothing starts a scheduler implicitly, so there is no port to protect, and a switch that failed closed on a typo would silently stop a schedule. Turning it off logs a warning naming the scheduler that did not start. |
83
94
  | `port` | `9394` (env `PROMETHEUS_EXPORTER_PORT`) | exporter bind port for Puma **and** Sidekiq |
84
95
  | `sidekiq_global_metrics` | `:web` | who exports the Redis-wide queue metrics: `:web` = only the always-on web process (no per-worker duplication); `:all` = every worker; `:off` = nobody |
85
96
  | `puma_control_url` | `tcp://127.0.0.1:9293` (env `PUMA_CONTROL_URL`) | localhost-only Puma control app the stats reader queries |
@@ -148,6 +159,8 @@ endpoint every few seconds); pass `silence_controller_log: false` to keep it.
148
159
  `sidekiq_queue_latency`, `sidekiq_active_processes`,
149
160
  `sidekiq_active_workers_count`, `sidekiq_jobs_retry_count`,
150
161
  `sidekiq_jobs_dead_count`, `sidekiq_jobs_scheduled_count`.
162
+ - **SolidQueue (DB-wide):** `solid_queue_backlog`,
163
+ `solid_queue_latency_seconds`.
151
164
 
152
165
  Series are intentionally **untagged**: a scraping agent adds
153
166
  `service`/`env`/`version` and `kube_*` tags at scrape time, so the gem doesn't
@@ -182,35 +195,53 @@ instances:
182
195
  raw_metric_prefix: "sidekiq_" # sidekiq_queue_latency -> sidekiq.queue_latency
183
196
  metrics: [".*"]
184
197
  tag_by_endpoint: false
198
+
199
+ # SolidQueue (the pod publishing the queue gauges) — :9394/metrics
200
+ instances:
201
+ - openmetrics_endpoint: "http://%%host%%:9394/metrics"
202
+ namespace: "solid_queue"
203
+ raw_metric_prefix: "solid_queue_" # solid_queue_backlog -> solid_queue.backlog
204
+ metrics: [".*"]
205
+ tag_by_endpoint: false
185
206
  ```
186
207
 
208
+ **One prefix per endpoint.** `metrics: [".*"]` ingests *everything* served
209
+ there, so each block above assumes its endpoint carries a single group — which
210
+ is why the SolidQueue gauges are best published from their own pod (below).
211
+ Where one endpoint really must carry two groups (e.g. a web pod exposing both
212
+ `puma_*` and `solid_queue_*`), give **every** instance on it an explicit filter
213
+ — `metrics: ["puma_.*"]` and `metrics: ["solid_queue_.*"]` — or each namespace
214
+ will swallow the other's series un-stripped.
215
+
187
216
  On Kubernetes this is typically wired as pod-annotation autodiscovery. The
188
217
  `openmetrics_endpoint` above uses the Datadog Agent's `%%host%%` autodiscovery
189
218
  template (resolves to the pod IP) — a **non-k8s adopter** (a plain `conf.yaml`
190
219
  check) swaps `%%host%%:9394` for the real `host:port`; everything else
191
220
  (`namespace`, `raw_metric_prefix`, `metrics`) is identical.
192
221
 
193
- ### Invariant — the endpoint must stay prefix-pure
222
+ ### Invariant — every series must carry a group prefix
194
223
 
195
224
  The Datadog check uses `metrics: [".*"]`, which ingests **every** series on the
196
225
  endpoint. `raw_metric_prefix` only *strips* the prefix when present — **it does
197
226
  not filter**. So the naming scheme above holds only because the `/metrics`
198
- endpoint exposes **solely** yabeda-registered series, all sharing the
199
- `puma_` / `sidekiq_` group prefix:
227
+ endpoint exposes **solely** yabeda-registered series, each carrying its
228
+ `puma_` / `sidekiq_` / `solid_queue_` group prefix:
200
229
 
201
- - the gem registers only the `:puma` and `:sidekiq` yabeda groups (no process,
202
- GC, or Ruby-runtime collectors);
230
+ - the gem registers only the `:puma`, `:sidekiq` and `:solid_queue` yabeda
231
+ groups (no process, GC, or Ruby-runtime collectors);
203
232
  - the exposition serves Yabeda's registry only — the Prometheus client's HTTP
204
233
  request collector (`http_*`) is **not** mounted on the exporter.
205
234
 
206
- If a series without the `puma_` / `sidekiq_` prefix ever appeared on the
207
- endpoint, `metrics: [".*"]` would ingest it **un-stripped** under the namespace
208
- (e.g. `puma.http_requests_total`). **Do not** add cross-cutting metrics (process,
209
- runtime, HTTP request) to this exporter, and do not mount the Prometheus Rack
210
- collector on it. If you ever need such metrics, expose them on a separate
211
- endpoint with its own check rather than polluting this one. Conversely, any new
212
- metric you *do* add to the `:puma` / `:sidekiq` groups is picked up automatically
213
- by `[".*"]` — no check change needed.
235
+ If an unprefixed series ever appeared on the endpoint, `metrics: [".*"]` would
236
+ ingest it **un-stripped** under the namespace (e.g. `puma.http_requests_total`).
237
+ **Do not** add cross-cutting metrics (process, runtime, HTTP request) to this
238
+ exporter, and do not mount the Prometheus Rack collector on it. If you ever need
239
+ such metrics, expose them on a separate endpoint with its own check rather than
240
+ polluting this one. Conversely, any new metric you *do* add to an existing group
241
+ is picked up automatically by `[".*"]` no check change needed.
242
+
243
+ `spec/rails_pod_kit/metrics_invariant_spec.rb` guards this at both the registry
244
+ and the exposition level.
214
245
 
215
246
  > ⚠️ Renaming the namespace prefix is a **breaking metric rename** — existing
216
247
  > dashboards/monitors built on the old `puma.puma_*` / `sidekiq.sidekiq_*` series
@@ -258,6 +289,13 @@ Sidekiq job class.
258
289
  | `sidekiq.job_runtime` | histogram | `queue`, `worker` |
259
290
  | `sidekiq.job_latency` | histogram | `queue`, `worker` |
260
291
 
292
+ **SolidQueue — DB-wide queue state** (`namespace: solid_queue`):
293
+
294
+ | canonical Datadog metric | type | functional tags |
295
+ |---|---|---|
296
+ | `solid_queue.backlog` | gauge | `queue` |
297
+ | `solid_queue.latency_seconds` | gauge | `queue` |
298
+
261
299
  ### Where Sidekiq global metrics come from
262
300
 
263
301
  The global (Redis-wide) queue gauges — `sidekiq_jobs_waiting_count`,
@@ -313,10 +351,227 @@ declares the cluster gauges, starts the exporter and blocks until SIGTERM.
313
351
  Booting the full host app just to read a handful of Redis counters would cost
314
352
  ~300Mi RSS for nothing — this process sits at ~60Mi.
315
353
 
354
+ ## Sidekiq: scale-to-zero
355
+
356
+ Being an always-on singleton makes that same pod the right home for the
357
+ **sidekiq-cron poller**, which is what lets the worker fleet scale to zero.
358
+
359
+ sidekiq-cron installs its poller from inside `Sidekiq.configure_server`, so on
360
+ its own the schedule exists only while a Sidekiq server is alive. At zero
361
+ replicas nothing polls, nothing is enqueued, and nothing ever raises the queue
362
+ depth that would wake a worker back up — a closed loop that forces a permanent
363
+ floor of one replica just to keep a poller alive. Missed runs are not caught up
364
+ afterwards either: `reschedule_grace_period` (60s by default) discards any run
365
+ older than itself.
366
+
367
+ The poller has no such requirement of its own — `Sidekiq::Cron::Poller` is a
368
+ Redis-polling thread that runs in any process holding a Sidekiq config — so
369
+ `scheduler: true` hosts it here:
370
+
371
+ ```ruby
372
+ RailsPodKit::GlobalExporter.run!(
373
+ redis: { url: ENV['REDIS_URL'] },
374
+ scheduler: true,
375
+ schedule_file: File.expand_path('../config/schedule.yml', __dir__)
376
+ )
377
+ ```
378
+
379
+ `schedule_file:`, `poll_interval:` and `reschedule_grace_period:` override
380
+ sidekiq-cron's defaults (`config/schedule.yml` resolved against the working
381
+ directory, polled every 30s, catching up runs at most 60s late);
382
+ `supervision_interval:` tunes the liveness check. `RailsPodKit::GlobalScheduler`
383
+ is usable on its own (`start!` / `stop!`) if the always-on process is something
384
+ other than the exporter.
385
+
386
+ > **Size `reschedule_grace_period` over your worst restart.** It is what makes
387
+ > restarting the *only* scheduling process free: below it a missed occurrence is
388
+ > caught up on the next poll, above it the run is skipped silently. sidekiq-cron
389
+ > defaults to 60s, which a node drain or an evicted pod can easily exceed —
390
+ > rolling updates are covered anyway, since a `maxSurge` overlap means there is
391
+ > no gap at all. Catching up is bounded, not repeated: `last_enqueue_time` in
392
+ > Redis still gates each occurrence to exactly one enqueue.
393
+ >
394
+ > This is the reason a singleton scheduler does **not** need to become an HA
395
+ > pair. A second replica is safe for the poller (same `zadd` lock) but doubles
396
+ > every metric series the pod publishes, and a `PodDisruptionBudget` on a
397
+ > single-replica Deployment stalls node drains rather than protecting anything.
398
+
399
+ The poller runs under `RailsPodKit::Supervisor` — the same supervising timer
400
+ that keeps the SolidQueue scheduler thread alive, since both share the failure
401
+ mode: the thread dies, the host process notices nothing, and the schedule stops
402
+ silently. The cron poller's own loop swallows StandardError, so a Redis blip
403
+ costs one skipped tick; the supervisor makes anything it does *not* catch a
404
+ skipped tick too.
405
+
406
+ > **Every schedule entry must declare `active_job: true`.** This process has no
407
+ > Rails, so it cannot resolve the job classes; sidekiq-cron then falls back to
408
+ > pushing a raw message, and only that flag makes the message an ActiveJob
409
+ > wrapper (naming the class as a *string*, which the worker resolves). Without it
410
+ > the job is pushed as a bare Sidekiq job and runs outside ActiveJob entirely.
411
+ > `start!` logs a warning naming any entry in that state. For the same reason the
412
+ > schedule file's ERB must not reach for Rails.
413
+
414
+ Leaving the workers' own poller in place is fine and costs nothing: enqueueing is
415
+ gated on a Redis `zadd` that exactly one caller wins — the same lock that already
416
+ lets multiple worker replicas coexist without double-firing. Both processes must
417
+ then read the *same* schedule file, though: `load_from_hash!` removes the
418
+ schedule-sourced jobs that are absent from the file it is given, so two processes
419
+ loading different files will delete each other's entries.
420
+
421
+ ## SolidQueue: scale-to-zero
422
+
423
+ SolidQueue's executor has nothing to do while the queue is empty, so it is the
424
+ natural candidate for **scale-to-zero** autoscaling (KEDA, or an HPA). Two things
425
+ stand in the way, and the gem covers both. Everything here is opt-in; requiring
426
+ the gem alone changes nothing.
427
+
428
+ ### 1. The scheduler has to move off the executor
429
+
430
+ With the executor at zero there is no scheduler, so nothing enqueues the
431
+ recurring and scheduled jobs that would wake one — the queue stays empty because
432
+ it is empty. A k8s CronJob can't take over either: it can't own **dynamic**
433
+ recurring tasks, the ones created and updated at runtime through
434
+ `SolidQueue.schedule_recurring_task`.
435
+
436
+ The fix is to run the *scheduler alone* on a process that is always on, and let
437
+ the executor be nothing but dispatcher + workers (`bin/jobs` with
438
+ `SOLID_QUEUE_SKIP_RECURRING=true`):
439
+
440
+ ```ruby
441
+ # config/puma.rb — after_booted only runs in the real Puma process, never in a
442
+ # console, a rake task or the test suite.
443
+ after_booted { RailsPodKit::SolidQueue.start_scheduler! }
444
+ at_exit { RailsPodKit::SolidQueue.stop_scheduler! }
445
+ ```
446
+
447
+ This is deliberately **not** `plugin :solid_queue`. That one runs the full
448
+ supervisor, which forks and whose watchdog takes Puma down when the supervisor
449
+ exits — and a transient Postgres disconnect is enough to cause that
450
+ ([rails/solid_queue#512](https://github.com/rails/solid_queue/issues/512)). Here
451
+ a DB blip at worst kills the scheduler thread; `RailsPodKit::Supervisor` — a
452
+ `Concurrent::TimerTask`, the same primitive SolidQueue supervises its own
453
+ processes with, and the same one that keeps the sidekiq-cron poller alive —
454
+ notices on the next tick and starts a fresh one, the process itself never
455
+ notices, and the scheduler re-registers on recovery.
456
+
457
+ Running it on every replica is safe: enqueues stay exactly-once via the unique
458
+ index on `solid_queue_recurring_executions (task_key, run_at)`. Static tasks come
459
+ from `config/recurring.yml` (honouring `SOLID_QUEUE_RECURRING_SCHEDULE`), dynamic
460
+ ones from the DB.
461
+
462
+ | option | default | meaning |
463
+ |---|---|---|
464
+ | `polling_interval` | `5` | how often the scheduler re-reads the dynamic tasks |
465
+ | `supervision_interval` | `5` | how often we check the scheduler thread is alive |
466
+ | `recurring_schedule_file` | `config/recurring.yml` | static task definitions; skipped when absent |
467
+
468
+ `start_scheduler!` is **not** gated on `enabled` — that switch owns the metrics
469
+ exporter, and an app may well want the scheduler with metrics off.
470
+ `scheduler_enabled` is the switch that does own it, shared with the sidekiq-cron
471
+ poller. Beyond that, what keeps it out of consoles and specs is *where* you call
472
+ it from.
473
+
474
+ ### 2. Queue depth has to be visible
475
+
476
+ SolidQueue publishes no metrics, so the autoscaler and the dashboards have
477
+ nothing to read. `install_metrics!` adds two gauges, computed **at scrape time**
478
+ from the SolidQueue tables (a yabeda `collect` block — no background thread, no
479
+ cached snapshot):
480
+
481
+ ```ruby
482
+ # config/initializers/rails_pod_kit.rb
483
+ RailsPodKit::SolidQueue.install_metrics!
484
+ ```
485
+
486
+ | metric | meaning |
487
+ |---|---|
488
+ | `solid_queue_backlog` | how many jobs could be claimed right now, per `queue` |
489
+ | `solid_queue_latency_seconds` | how long the oldest of them has been waiting, per `queue` |
490
+
491
+ "Claimable right now" is ready executions **plus** scheduled ones whose time has
492
+ come — the dispatcher has only to move those across. A scheduled job's wait is
493
+ measured from its `scheduled_at`, not its `created_at`: enqueuing a week ahead of
494
+ the slot doesn't make it a week late.
495
+
496
+ Both matter, and neither alone is enough: backlog misses a small-but-stalled
497
+ queue, latency misses a large-but-moving one.
498
+
499
+ **An idle system reads 0, not no-data.** A queue that drains is explicitly zeroed
500
+ rather than left pinned at its last reading, and the zeroing starts from a
501
+ baseline of every queue the app is known to use — discovered once per process
502
+ from the jobs table, or pinned by the host:
503
+
504
+ ```ruby
505
+ RailsPodKit::SolidQueue.install_metrics!(queues: %w[default mailers])
506
+ ```
507
+
508
+ Without that baseline a process booting while the queue is empty — the steady
509
+ state of a scale-to-zero deployment — would publish no series at all, since a
510
+ gauge only exists once it has been set. Discovery is best-effort: the jobs table
511
+ is bounded by `clear_finished_jobs_after`, so a queue idle for longer than the
512
+ retention window leaves no trace in it. Pin `queues:` where the zero has to be
513
+ guaranteed.
514
+
515
+ **A collection error is always reported** (logged, plus handed to
516
+ `Rails.error`), then either swallowed or raised:
517
+
518
+ | | |
519
+ |---|---|
520
+ | `fail_scrape_on_error: false` (default) | serve the last reading. Right on an endpoint shared with the Puma or Sidekiq series, where failing the response would lose those too — at the cost of a stale gauge a consumer cannot tell apart from a live one. |
521
+ | `fail_scrape_on_error: true` (`run_exporter!`'s default) | fail the scrape. Right on the dedicated pod, where there is nothing else to protect: the gauges go to no-data and the scrape failure shows up in the scraper's own `up` series. |
522
+
523
+ ### The always-on pod
524
+
525
+ The cleanest home for both is a **1-replica Deployment** that hosts the scheduler
526
+ and publishes the gauges, decoupled from the web and the executor — so the
527
+ signals survive either scaling to zero, and each series has exactly one source.
528
+ `run_exporter!` is that process: it declares the gauges, starts the scheduler,
529
+ serves `/metrics` and blocks until SIGTERM (winding the scheduler down so it
530
+ deregisters rather than expiring).
531
+
532
+ Unlike the Sidekiq global exporter it is **not** Rails-free — SolidQueue is
533
+ ActiveRecord-backed and reads the app's own tables — so the host's entrypoint
534
+ boots the environment first. The gem ships no executable; e.g.
535
+ `bin/solid-queue-pod`:
536
+
537
+ ```ruby
538
+ #!/usr/bin/env ruby
539
+ require_relative '../config/environment'
540
+
541
+ RailsPodKit::SolidQueue.run_exporter!
542
+ ```
543
+
544
+ ```
545
+ command: ["bin/solid-queue-pod"]
546
+ ```
547
+
548
+ Pass `scheduler: false` to serve the gauges only, on an app whose web process
549
+ already hosts the scheduler, and `metrics:` to override the gauge options (this
550
+ endpoint is the collector's own, so `fail_scrape_on_error` defaults to `true`
551
+ here):
552
+
553
+ ```ruby
554
+ RailsPodKit::SolidQueue.run_exporter!(scheduler: false, metrics: { queues: %w[default mailers] })
555
+ ```
556
+
557
+ Keep the endpoint single-prefix (see
558
+ [One prefix per endpoint](#datadog-naming--the-canonical-metric-set)) — that pod
559
+ serves `solid_queue_*` and nothing else, so the check config needs no filters.
560
+
316
561
  ## Caveats
317
562
 
318
563
  - **Puma only.** The Puma plugins only activate under Puma; under any other
319
- app server the in-process `/metrics` endpoint is **not** exposed.
564
+ app server the in-process `/metrics` endpoint is **not** exposed. The non-Puma
565
+ entry points (Sidekiq, the global exporter, the SolidQueue pod) serve it from
566
+ a WEBrick thread instead, started at most once per process.
567
+ - **SolidQueue and Sidekiq are the host's.** The gem depends on neither; the
568
+ SolidQueue integration is inert until you call it, exactly like the Sidekiq one.
569
+ `sidekiq-cron` too: it is required only when `GlobalScheduler.start!` is called,
570
+ so an app that does not schedule anything need not carry it.
571
+ - **Queue-gauge query cost.** The gauges run four small grouped aggregates per
572
+ scrape. `MIN(created_at)` is not covered by SolidQueue's indexes, so on a
573
+ backlog of many thousands of rows it is a scan — cheap at a normal scrape
574
+ interval, worth knowing about if you scrape aggressively.
320
575
  - **Puma control app.** `yabeda-puma-plugin` reads Puma's thread-pool stats
321
576
  through Puma's control app, so `Puma.activate` activates one on a
322
577
  localhost-only socket (`no_token: true`, never network-exposed).
@@ -352,6 +607,10 @@ curl -s localhost:9394/metrics | grep '^puma_'
352
607
  bundle exec sidekiq
353
608
  curl -s localhost:9394/metrics | grep '^sidekiq_'
354
609
 
610
+ # SolidQueue: solid_queue_* series (enqueue a job first so the queue isn't empty)
611
+ bin/solid-queue-pod
612
+ curl -s localhost:9394/metrics | grep '^solid_queue_'
613
+
355
614
  # Health endpoint
356
615
  curl -s localhost:3000/healthz -H 'Accept: application/json'
357
616
  ```
data/VERSION CHANGED
@@ -1 +1 @@
1
- 0.0.3
1
+ 0.2.0
@@ -60,17 +60,29 @@ module RailsPodKit
60
60
  # transports — the Puma plugin's exporter (`prometheus_silence_logger`)
61
61
  # and the WEBrick exporter used by Sidekiq / the dedicated global exporter
62
62
  # (`Rack::CommonLogger`). Flip to false only to debug the exporter itself.
63
+ # scheduler_enabled: kill switch for the hosted schedulers (the sidekiq-cron
64
+ # poller and the SolidQueue scheduler thread). Separate from `enabled`,
65
+ # which owns the metrics exporter: an app may well want one without the
66
+ # other, and the scheduler is the piece you may need to turn off in a
67
+ # hurry — it is a single point of failure for the whole schedule, and
68
+ # `RAILS_POD_KIT_SCHEDULER_ENABLED=false` + a restart beats a deploy at
69
+ # 3am. Defaults to on *including* in the test env, unlike `enabled`:
70
+ # nothing starts a scheduler implicitly (it takes an explicit call from an
71
+ # entry point), so there is no port to protect, and a flag that fails
72
+ # closed on a typo would silently stop a schedule.
63
73
  attr_config :enabled,
64
74
  :port,
65
75
  :puma_control_url,
66
76
  sidekiq_global_metrics: :web,
67
77
  retries_segmented_by_queue: false,
68
- silence_exporter_access_log: true
78
+ silence_exporter_access_log: true,
79
+ scheduler_enabled: true
69
80
 
70
81
  coerce_types port: :integer,
71
82
  enabled: :boolean,
72
83
  retries_segmented_by_queue: :boolean,
73
- silence_exporter_access_log: :boolean
84
+ silence_exporter_access_log: :boolean,
85
+ scheduler_enabled: :boolean
74
86
 
75
87
  def initialize(overrides = nil)
76
88
  super
@@ -87,6 +99,10 @@ module RailsPodKit
87
99
  !!enabled
88
100
  end
89
101
 
102
+ def scheduler_enabled?
103
+ !!scheduler_enabled
104
+ end
105
+
90
106
  # True unless we're clearly in a test environment. Kept independent of
91
107
  # Rails so the gem's own specs (which don't load the host app) get a safe
92
108
  # default and never bind a socket.
@@ -114,5 +130,15 @@ module RailsPodKit
114
130
  def enabled?
115
131
  config.enabled?
116
132
  end
133
+
134
+ # Gate shared by both hosted schedulers. Says so out loud when it turns one
135
+ # off: a process that starts, stays up and quietly schedules nothing is the
136
+ # one failure this whole feature exists to avoid.
137
+ def scheduler_enabled?(what = 'scheduler')
138
+ return true if config.scheduler_enabled?
139
+
140
+ warn "[rails_pod_kit] scheduler_enabled=false — #{what} not started"
141
+ false
142
+ end
117
143
  end
118
144
  end
@@ -0,0 +1,34 @@
1
+ # frozen_string_literal: true
2
+
3
+ module RailsPodKit
4
+ # Where the gem's background work (the SolidQueue scheduler supervisor, the
5
+ # scrape-time metric collectors) sends the errors it swallows. Those errors
6
+ # must never propagate — one would kill the supervising timer or fail the whole
7
+ # /metrics response — but they must not vanish either.
8
+ #
9
+ # Always logs, and additionally hands the error to the Rails error reporter
10
+ # when there is one, so the host's Rollbar / Sentry / Datadog subscriber picks
11
+ # it up. The log line is not redundant: `Rails.error.report` only fans out to
12
+ # subscribers, so on an app with none — or with one that is not wired in a
13
+ # given environment — the failure would otherwise leave no trace at all, and
14
+ # the only symptom of a broken collector is a gauge quietly serving a stale
15
+ # value.
16
+ module ErrorReporter
17
+ module_function
18
+
19
+ def report(error, source:)
20
+ message = "[#{source}] #{error.class}: #{error.message}"
21
+ logger ? logger.error(message) : warn(message)
22
+
23
+ rails_reporter&.report(error, handled: true, source: source)
24
+ end
25
+
26
+ def logger
27
+ ::Rails.logger if defined?(::Rails) && ::Rails.respond_to?(:logger)
28
+ end
29
+
30
+ def rails_reporter
31
+ ::Rails.error if defined?(::Rails) && ::Rails.respond_to?(:error)
32
+ end
33
+ end
34
+ end
@@ -0,0 +1,42 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'rails_pod_kit/config'
4
+
5
+ module RailsPodKit
6
+ # The in-process WEBrick /metrics server, shared by every non-Puma entry
7
+ # point: the Sidekiq worker, the Rails-free global exporter and the SolidQueue
8
+ # exporter. Under Puma the exporter comes from the `:yabeda_prometheus` plugin
9
+ # instead (see RailsPodKit::Puma), so this is never used there.
10
+ module Exporter
11
+ module_function
12
+
13
+ # Starts the background exporter and returns whether it did. Idempotent: the
14
+ # latch keeps a re-entrant boot from double-binding the port.
15
+ def start!
16
+ return false unless RailsPodKit.enabled?
17
+ return false if @started
18
+
19
+ require 'yabeda/prometheus/mmap'
20
+
21
+ # `start_metrics_server!` reads the bind port from the env, so publish the
22
+ # configured one first.
23
+ ENV['PROMETHEUS_EXPORTER_PORT'] ||= RailsPodKit.config.port.to_s
24
+ # Drop the exporter's per-scrape access log (Rack::CommonLogger, which the
25
+ # mmap exporter mounts unless this is exactly 'false'). See
26
+ # Config#silence_exporter_access_log.
27
+ ENV['PROMETHEUS_EXPORTER_LOG_REQUESTS'] = 'false' if RailsPodKit.config.silence_exporter_access_log
28
+
29
+ Yabeda::Prometheus::Exporter.start_metrics_server!
30
+ @started = true
31
+ end
32
+
33
+ def started?
34
+ !!@started
35
+ end
36
+
37
+ # Test/reset hook — drops the "already started" latch.
38
+ def reset!
39
+ @started = false
40
+ end
41
+ end
42
+ end
@@ -1,6 +1,8 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  require 'rails_pod_kit/config'
4
+ require 'rails_pod_kit/global_scheduler'
5
+ require 'rails_pod_kit/shutdown'
4
6
 
5
7
  module RailsPodKit
6
8
  # Standalone, always-on exporter for the Sidekiq global (Redis-wide) queue
@@ -26,6 +28,10 @@ module RailsPodKit
26
28
  # require 'bundler/setup'
27
29
  # require 'rails_pod_kit/global_exporter'
28
30
  # RailsPodKit::GlobalExporter.run!(redis: { url: ENV['REDIS_URL'] })
31
+ #
32
+ # Being an always-on singleton also makes it the natural host for the
33
+ # sidekiq-cron poller (`scheduler: true`, see GlobalScheduler), which is what
34
+ # lets the workers scale to zero without losing their schedule.
29
35
  module GlobalExporter
30
36
  module_function
31
37
 
@@ -47,22 +53,34 @@ module RailsPodKit
47
53
  # `redis:` takes the same options hash the host passes to its own
48
54
  # `Sidekiq.configure_*` blocks (`url:`, `ssl_params:`, …), so the connection
49
55
  # config stays a host decision with a single source of truth.
50
- def run!(redis:)
51
- unless RailsPodKit.enabled?
52
- warn '[rails_pod_kit] disabled global exporter not started'
53
- return
54
- end
56
+ #
57
+ # `scheduler: true` additionally runs the sidekiq-cron poller here; any
58
+ # extra keywords are GlobalScheduler.start!'s (`schedule_file:`,
59
+ # `poll_interval:`, `supervision_interval:`). The scheduler is not gated on
60
+ # `RailsPodKit.enabled?` —
61
+ # that switch owns the metrics exporter, and an app may well want the
62
+ # singleton scheduler with metrics turned off.
63
+ def run!(redis:, scheduler: false, **scheduler_options)
64
+ serve_metrics = RailsPodKit.enabled?
65
+ warn '[rails_pod_kit] disabled — /metrics not served by the global exporter' unless serve_metrics
66
+ return unless serve_metrics || scheduler
55
67
 
56
68
  configure_redis!(redis)
69
+ start_metrics! if serve_metrics
70
+ GlobalScheduler.start!(**scheduler_options) if scheduler
71
+
72
+ # Both the exporter and the poller serve from background threads; block
73
+ # the main thread so the process stays up until the kubelet sends SIGTERM.
74
+ Shutdown.await
75
+ GlobalScheduler.stop! if scheduler
76
+ end
77
+
78
+ def start_metrics!
57
79
  install!
58
80
  RailsPodKit::Sidekiq.start_metrics_server!
59
81
 
60
82
  require 'yabeda'
61
83
  Yabeda.configure! unless Yabeda.already_configured?
62
-
63
- # The exporter serves from a background thread; block the main thread so
64
- # the process stays up until the kubelet sends SIGTERM.
65
- sleep
66
84
  end
67
85
 
68
86
  # Configure the Sidekiq client's Redis connection so this Rails-free process