rails_pod_kit 0.0.3 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +284 -25
- data/VERSION +1 -1
- data/lib/rails_pod_kit/config.rb +28 -2
- data/lib/rails_pod_kit/error_reporter.rb +34 -0
- data/lib/rails_pod_kit/exporter.rb +42 -0
- data/lib/rails_pod_kit/global_exporter.rb +27 -9
- data/lib/rails_pod_kit/global_scheduler.rb +156 -0
- data/lib/rails_pod_kit/shutdown.rb +18 -0
- data/lib/rails_pod_kit/sidekiq.rb +5 -12
- data/lib/rails_pod_kit/solid_queue/metrics.rb +186 -0
- data/lib/rails_pod_kit/solid_queue/scheduler_runner.rb +118 -0
- data/lib/rails_pod_kit/solid_queue.rb +108 -0
- data/lib/rails_pod_kit/supervisor.rb +90 -0
- data/lib/rails_pod_kit.rb +8 -5
- metadata +32 -7
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 85fea66a44c5b2198b5c3c7bfd05f5f1f4c3655559ed701748661d0e148650a5
|
|
4
|
+
data.tar.gz: 9c921528f839ea1d5b962035600316ea37affe17cf8fdc910453c30108a248dd
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: d7fbfee31ce338a6dd7c11d65214dc442cd2cb261848ae2403a415dc4a6e1333fa55b39e2d06441b4e05fcacff2ec3971ee009613a532f384fcc19fbe84f070d
|
|
7
|
+
data.tar.gz: 894f99f9fba4596aa7d69e143f5fa23d2ebc64ef85f6b7cac310096c902005dd003103cb12c3933e49dfe26263c68f9656cfc3e68c0d9551a8f118e79496fd48
|
data/README.md
CHANGED
|
@@ -3,14 +3,17 @@
|
|
|
3
3
|
The operational endpoints a Rails pod needs to be a good Kubernetes citizen,
|
|
4
4
|
packaged behind a single, opinionated entry point:
|
|
5
5
|
|
|
6
|
-
- **Prometheus metrics** for Puma and
|
|
7
|
-
single `/metrics` endpoint (default port **9394**) in
|
|
8
|
-
worker
|
|
9
|
-
|
|
10
|
-
|
|
6
|
+
- **Prometheus metrics** for Puma, Sidekiq and SolidQueue, served **in-process**
|
|
7
|
+
on a single `/metrics` endpoint (default port **9394**) in the web (Puma) and
|
|
8
|
+
worker processes — no sidecar, no separate collector process. A metrics agent
|
|
9
|
+
(e.g. the Datadog Agent via OpenMetrics autodiscovery) scrapes the pod
|
|
10
|
+
directly.
|
|
11
11
|
- **Health checks** on `/healthz` (database, cache, optionally Redis and Sidekiq),
|
|
12
12
|
wired for Kubernetes startup/liveness/readiness probes — a thin, opinionated
|
|
13
13
|
wrapper around [health-monitor-rails](https://github.com/lbeder/health-monitor-rails).
|
|
14
|
+
- **Scheduler hosting for scale-to-zero**, so a job executor can be autoscaled to
|
|
15
|
+
zero without stranding its recurring and scheduled jobs: a supervised
|
|
16
|
+
SolidQueue scheduler thread, and a supervised sidekiq-cron poller for Sidekiq.
|
|
14
17
|
|
|
15
18
|
The metrics side is a thin wrapper around the
|
|
16
19
|
[yabeda](https://github.com/yabeda-rb) ecosystem:
|
|
@@ -21,11 +24,14 @@ The metrics side is a thin wrapper around the
|
|
|
21
24
|
| `yabeda-puma-plugin` | Puma thread-pool / worker stats + the `:yabeda` / `:yabeda_prometheus` Puma plugins |
|
|
22
25
|
| `yabeda-sidekiq` | Sidekiq per-process job metrics + global/Redis-wide queue metrics |
|
|
23
26
|
| `yabeda-prometheus-mmap` | Prometheus text exporter, multiprocess-safe via `prometheus-client-mmap` |
|
|
24
|
-
| `webrick` | HTTP server for the
|
|
27
|
+
| `webrick` | HTTP server for the non-Puma exporters |
|
|
25
28
|
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
+
The SolidQueue queue gauges are the gem's own: SolidQueue ships no metrics
|
|
30
|
+
endpoint and there is no `yabeda-solid_queue` plugin to wrap.
|
|
31
|
+
|
|
32
|
+
> Scope: **runtime/worker metrics only** (Puma backlog/threads, Sidekiq and
|
|
33
|
+
> SolidQueue queues and jobs). HTTP request-level metrics are intentionally out
|
|
34
|
+
> of scope — that is covered by APM.
|
|
29
35
|
|
|
30
36
|
The gem is deliberately **connection-agnostic**: it never reads `REDIS_URL` and
|
|
31
37
|
makes no TLS decisions. Wherever a Redis connection is needed (health checks,
|
|
@@ -67,19 +73,24 @@ RailsPodKit::Puma.activate(self)
|
|
|
67
73
|
RailsPodKit::Sidekiq.install!(config)
|
|
68
74
|
```
|
|
69
75
|
|
|
76
|
+
On a SolidQueue stack there is no step 3 — see
|
|
77
|
+
[SolidQueue](#solidqueue-scale-to-zero) instead.
|
|
78
|
+
|
|
70
79
|
## Configuration
|
|
71
80
|
|
|
72
81
|
`RailsPodKit::Config` is an [anyway_config](https://github.com/palkan/anyway_config)
|
|
73
82
|
config: besides the `RailsPodKit.configure` block (which runs last and wins),
|
|
74
83
|
every setting can come from a `RAILS_POD_KIT_*` env var (e.g.
|
|
75
84
|
`RAILS_POD_KIT_ENABLED=false`, `RAILS_POD_KIT_PORT=9500`,
|
|
76
|
-
`RAILS_POD_KIT_SIDEKIQ_GLOBAL_METRICS=off
|
|
85
|
+
`RAILS_POD_KIT_SIDEKIQ_GLOBAL_METRICS=off`,
|
|
86
|
+
`RAILS_POD_KIT_SCHEDULER_ENABLED=false`) or an optional
|
|
77
87
|
`config/rails_pod_kit.yml` — handy for the Rails-free exporter pod, which runs
|
|
78
88
|
no initializers.
|
|
79
89
|
|
|
80
90
|
| setting | default | meaning |
|
|
81
91
|
|---------|---------|---------|
|
|
82
|
-
| `enabled` | on, except in `test` | master switch
|
|
92
|
+
| `enabled` | on, except in `test` | master switch for the **exporter**; `false` ⇒ no exporter, no port bound |
|
|
93
|
+
| `scheduler_enabled` | on, everywhere | kill switch for the **hosted schedulers** (sidekiq-cron poller, SolidQueue scheduler thread). Separate from `enabled` — an app may want one without the other, and this is the one you may need to flip in a hurry, since the scheduler is a single point of failure for the whole schedule. On even in `test`: nothing starts a scheduler implicitly, so there is no port to protect, and a switch that failed closed on a typo would silently stop a schedule. Turning it off logs a warning naming the scheduler that did not start. |
|
|
83
94
|
| `port` | `9394` (env `PROMETHEUS_EXPORTER_PORT`) | exporter bind port for Puma **and** Sidekiq |
|
|
84
95
|
| `sidekiq_global_metrics` | `:web` | who exports the Redis-wide queue metrics: `:web` = only the always-on web process (no per-worker duplication); `:all` = every worker; `:off` = nobody |
|
|
85
96
|
| `puma_control_url` | `tcp://127.0.0.1:9293` (env `PUMA_CONTROL_URL`) | localhost-only Puma control app the stats reader queries |
|
|
@@ -148,6 +159,8 @@ endpoint every few seconds); pass `silence_controller_log: false` to keep it.
|
|
|
148
159
|
`sidekiq_queue_latency`, `sidekiq_active_processes`,
|
|
149
160
|
`sidekiq_active_workers_count`, `sidekiq_jobs_retry_count`,
|
|
150
161
|
`sidekiq_jobs_dead_count`, `sidekiq_jobs_scheduled_count`.
|
|
162
|
+
- **SolidQueue (DB-wide):** `solid_queue_backlog`,
|
|
163
|
+
`solid_queue_latency_seconds`.
|
|
151
164
|
|
|
152
165
|
Series are intentionally **untagged**: a scraping agent adds
|
|
153
166
|
`service`/`env`/`version` and `kube_*` tags at scrape time, so the gem doesn't
|
|
@@ -182,35 +195,53 @@ instances:
|
|
|
182
195
|
raw_metric_prefix: "sidekiq_" # sidekiq_queue_latency -> sidekiq.queue_latency
|
|
183
196
|
metrics: [".*"]
|
|
184
197
|
tag_by_endpoint: false
|
|
198
|
+
|
|
199
|
+
# SolidQueue (the pod publishing the queue gauges) — :9394/metrics
|
|
200
|
+
instances:
|
|
201
|
+
- openmetrics_endpoint: "http://%%host%%:9394/metrics"
|
|
202
|
+
namespace: "solid_queue"
|
|
203
|
+
raw_metric_prefix: "solid_queue_" # solid_queue_backlog -> solid_queue.backlog
|
|
204
|
+
metrics: [".*"]
|
|
205
|
+
tag_by_endpoint: false
|
|
185
206
|
```
|
|
186
207
|
|
|
208
|
+
**One prefix per endpoint.** `metrics: [".*"]` ingests *everything* served
|
|
209
|
+
there, so each block above assumes its endpoint carries a single group — which
|
|
210
|
+
is why the SolidQueue gauges are best published from their own pod (below).
|
|
211
|
+
Where one endpoint really must carry two groups (e.g. a web pod exposing both
|
|
212
|
+
`puma_*` and `solid_queue_*`), give **every** instance on it an explicit filter
|
|
213
|
+
— `metrics: ["puma_.*"]` and `metrics: ["solid_queue_.*"]` — or each namespace
|
|
214
|
+
will swallow the other's series un-stripped.
|
|
215
|
+
|
|
187
216
|
On Kubernetes this is typically wired as pod-annotation autodiscovery. The
|
|
188
217
|
`openmetrics_endpoint` above uses the Datadog Agent's `%%host%%` autodiscovery
|
|
189
218
|
template (resolves to the pod IP) — a **non-k8s adopter** (a plain `conf.yaml`
|
|
190
219
|
check) swaps `%%host%%:9394` for the real `host:port`; everything else
|
|
191
220
|
(`namespace`, `raw_metric_prefix`, `metrics`) is identical.
|
|
192
221
|
|
|
193
|
-
### Invariant —
|
|
222
|
+
### Invariant — every series must carry a group prefix
|
|
194
223
|
|
|
195
224
|
The Datadog check uses `metrics: [".*"]`, which ingests **every** series on the
|
|
196
225
|
endpoint. `raw_metric_prefix` only *strips* the prefix when present — **it does
|
|
197
226
|
not filter**. So the naming scheme above holds only because the `/metrics`
|
|
198
|
-
endpoint exposes **solely** yabeda-registered series,
|
|
199
|
-
`puma_` / `sidekiq_` group prefix:
|
|
227
|
+
endpoint exposes **solely** yabeda-registered series, each carrying its
|
|
228
|
+
`puma_` / `sidekiq_` / `solid_queue_` group prefix:
|
|
200
229
|
|
|
201
|
-
- the gem registers only the `:puma` and `:
|
|
202
|
-
GC, or Ruby-runtime collectors);
|
|
230
|
+
- the gem registers only the `:puma`, `:sidekiq` and `:solid_queue` yabeda
|
|
231
|
+
groups (no process, GC, or Ruby-runtime collectors);
|
|
203
232
|
- the exposition serves Yabeda's registry only — the Prometheus client's HTTP
|
|
204
233
|
request collector (`http_*`) is **not** mounted on the exporter.
|
|
205
234
|
|
|
206
|
-
If
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
235
|
+
If an unprefixed series ever appeared on the endpoint, `metrics: [".*"]` would
|
|
236
|
+
ingest it **un-stripped** under the namespace (e.g. `puma.http_requests_total`).
|
|
237
|
+
**Do not** add cross-cutting metrics (process, runtime, HTTP request) to this
|
|
238
|
+
exporter, and do not mount the Prometheus Rack collector on it. If you ever need
|
|
239
|
+
such metrics, expose them on a separate endpoint with its own check rather than
|
|
240
|
+
polluting this one. Conversely, any new metric you *do* add to an existing group
|
|
241
|
+
is picked up automatically by `[".*"]` — no check change needed.
|
|
242
|
+
|
|
243
|
+
`spec/rails_pod_kit/metrics_invariant_spec.rb` guards this at both the registry
|
|
244
|
+
and the exposition level.
|
|
214
245
|
|
|
215
246
|
> ⚠️ Renaming the namespace prefix is a **breaking metric rename** — existing
|
|
216
247
|
> dashboards/monitors built on the old `puma.puma_*` / `sidekiq.sidekiq_*` series
|
|
@@ -258,6 +289,13 @@ Sidekiq job class.
|
|
|
258
289
|
| `sidekiq.job_runtime` | histogram | `queue`, `worker` |
|
|
259
290
|
| `sidekiq.job_latency` | histogram | `queue`, `worker` |
|
|
260
291
|
|
|
292
|
+
**SolidQueue — DB-wide queue state** (`namespace: solid_queue`):
|
|
293
|
+
|
|
294
|
+
| canonical Datadog metric | type | functional tags |
|
|
295
|
+
|---|---|---|
|
|
296
|
+
| `solid_queue.backlog` | gauge | `queue` |
|
|
297
|
+
| `solid_queue.latency_seconds` | gauge | `queue` |
|
|
298
|
+
|
|
261
299
|
### Where Sidekiq global metrics come from
|
|
262
300
|
|
|
263
301
|
The global (Redis-wide) queue gauges — `sidekiq_jobs_waiting_count`,
|
|
@@ -313,10 +351,227 @@ declares the cluster gauges, starts the exporter and blocks until SIGTERM.
|
|
|
313
351
|
Booting the full host app just to read a handful of Redis counters would cost
|
|
314
352
|
~300Mi RSS for nothing — this process sits at ~60Mi.
|
|
315
353
|
|
|
354
|
+
## Sidekiq: scale-to-zero
|
|
355
|
+
|
|
356
|
+
Being an always-on singleton makes that same pod the right home for the
|
|
357
|
+
**sidekiq-cron poller**, which is what lets the worker fleet scale to zero.
|
|
358
|
+
|
|
359
|
+
sidekiq-cron installs its poller from inside `Sidekiq.configure_server`, so on
|
|
360
|
+
its own the schedule exists only while a Sidekiq server is alive. At zero
|
|
361
|
+
replicas nothing polls, nothing is enqueued, and nothing ever raises the queue
|
|
362
|
+
depth that would wake a worker back up — a closed loop that forces a permanent
|
|
363
|
+
floor of one replica just to keep a poller alive. Missed runs are not caught up
|
|
364
|
+
afterwards either: `reschedule_grace_period` (60s by default) discards any run
|
|
365
|
+
older than itself.
|
|
366
|
+
|
|
367
|
+
The poller has no such requirement of its own — `Sidekiq::Cron::Poller` is a
|
|
368
|
+
Redis-polling thread that runs in any process holding a Sidekiq config — so
|
|
369
|
+
`scheduler: true` hosts it here:
|
|
370
|
+
|
|
371
|
+
```ruby
|
|
372
|
+
RailsPodKit::GlobalExporter.run!(
|
|
373
|
+
redis: { url: ENV['REDIS_URL'] },
|
|
374
|
+
scheduler: true,
|
|
375
|
+
schedule_file: File.expand_path('../config/schedule.yml', __dir__)
|
|
376
|
+
)
|
|
377
|
+
```
|
|
378
|
+
|
|
379
|
+
`schedule_file:`, `poll_interval:` and `reschedule_grace_period:` override
|
|
380
|
+
sidekiq-cron's defaults (`config/schedule.yml` resolved against the working
|
|
381
|
+
directory, polled every 30s, catching up runs at most 60s late);
|
|
382
|
+
`supervision_interval:` tunes the liveness check. `RailsPodKit::GlobalScheduler`
|
|
383
|
+
is usable on its own (`start!` / `stop!`) if the always-on process is something
|
|
384
|
+
other than the exporter.
|
|
385
|
+
|
|
386
|
+
> **Size `reschedule_grace_period` over your worst restart.** It is what makes
|
|
387
|
+
> restarting the *only* scheduling process free: below it a missed occurrence is
|
|
388
|
+
> caught up on the next poll, above it the run is skipped silently. sidekiq-cron
|
|
389
|
+
> defaults to 60s, which a node drain or an evicted pod can easily exceed —
|
|
390
|
+
> rolling updates are covered anyway, since a `maxSurge` overlap means there is
|
|
391
|
+
> no gap at all. Catching up is bounded, not repeated: `last_enqueue_time` in
|
|
392
|
+
> Redis still gates each occurrence to exactly one enqueue.
|
|
393
|
+
>
|
|
394
|
+
> This is the reason a singleton scheduler does **not** need to become an HA
|
|
395
|
+
> pair. A second replica is safe for the poller (same `zadd` lock) but doubles
|
|
396
|
+
> every metric series the pod publishes, and a `PodDisruptionBudget` on a
|
|
397
|
+
> single-replica Deployment stalls node drains rather than protecting anything.
|
|
398
|
+
|
|
399
|
+
The poller runs under `RailsPodKit::Supervisor` — the same supervising timer
|
|
400
|
+
that keeps the SolidQueue scheduler thread alive, since both share the failure
|
|
401
|
+
mode: the thread dies, the host process notices nothing, and the schedule stops
|
|
402
|
+
silently. The cron poller's own loop swallows StandardError, so a Redis blip
|
|
403
|
+
costs one skipped tick; the supervisor makes anything it does *not* catch a
|
|
404
|
+
skipped tick too.
|
|
405
|
+
|
|
406
|
+
> **Every schedule entry must declare `active_job: true`.** This process has no
|
|
407
|
+
> Rails, so it cannot resolve the job classes; sidekiq-cron then falls back to
|
|
408
|
+
> pushing a raw message, and only that flag makes the message an ActiveJob
|
|
409
|
+
> wrapper (naming the class as a *string*, which the worker resolves). Without it
|
|
410
|
+
> the job is pushed as a bare Sidekiq job and runs outside ActiveJob entirely.
|
|
411
|
+
> `start!` logs a warning naming any entry in that state. For the same reason the
|
|
412
|
+
> schedule file's ERB must not reach for Rails.
|
|
413
|
+
|
|
414
|
+
Leaving the workers' own poller in place is fine and costs nothing: enqueueing is
|
|
415
|
+
gated on a Redis `zadd` that exactly one caller wins — the same lock that already
|
|
416
|
+
lets multiple worker replicas coexist without double-firing. Both processes must
|
|
417
|
+
then read the *same* schedule file, though: `load_from_hash!` removes the
|
|
418
|
+
schedule-sourced jobs that are absent from the file it is given, so two processes
|
|
419
|
+
loading different files will delete each other's entries.
|
|
420
|
+
|
|
421
|
+
## SolidQueue: scale-to-zero
|
|
422
|
+
|
|
423
|
+
SolidQueue's executor has nothing to do while the queue is empty, so it is the
|
|
424
|
+
natural candidate for **scale-to-zero** autoscaling (KEDA, or an HPA). Two things
|
|
425
|
+
stand in the way, and the gem covers both. Everything here is opt-in; requiring
|
|
426
|
+
the gem alone changes nothing.
|
|
427
|
+
|
|
428
|
+
### 1. The scheduler has to move off the executor
|
|
429
|
+
|
|
430
|
+
With the executor at zero there is no scheduler, so nothing enqueues the
|
|
431
|
+
recurring and scheduled jobs that would wake one — the queue stays empty because
|
|
432
|
+
it is empty. A k8s CronJob can't take over either: it can't own **dynamic**
|
|
433
|
+
recurring tasks, the ones created and updated at runtime through
|
|
434
|
+
`SolidQueue.schedule_recurring_task`.
|
|
435
|
+
|
|
436
|
+
The fix is to run the *scheduler alone* on a process that is always on, and let
|
|
437
|
+
the executor be nothing but dispatcher + workers (`bin/jobs` with
|
|
438
|
+
`SOLID_QUEUE_SKIP_RECURRING=true`):
|
|
439
|
+
|
|
440
|
+
```ruby
|
|
441
|
+
# config/puma.rb — after_booted only runs in the real Puma process, never in a
|
|
442
|
+
# console, a rake task or the test suite.
|
|
443
|
+
after_booted { RailsPodKit::SolidQueue.start_scheduler! }
|
|
444
|
+
at_exit { RailsPodKit::SolidQueue.stop_scheduler! }
|
|
445
|
+
```
|
|
446
|
+
|
|
447
|
+
This is deliberately **not** `plugin :solid_queue`. That one runs the full
|
|
448
|
+
supervisor, which forks and whose watchdog takes Puma down when the supervisor
|
|
449
|
+
exits — and a transient Postgres disconnect is enough to cause that
|
|
450
|
+
([rails/solid_queue#512](https://github.com/rails/solid_queue/issues/512)). Here
|
|
451
|
+
a DB blip at worst kills the scheduler thread; `RailsPodKit::Supervisor` — a
|
|
452
|
+
`Concurrent::TimerTask`, the same primitive SolidQueue supervises its own
|
|
453
|
+
processes with, and the same one that keeps the sidekiq-cron poller alive —
|
|
454
|
+
notices on the next tick and starts a fresh one, the process itself never
|
|
455
|
+
notices, and the scheduler re-registers on recovery.
|
|
456
|
+
|
|
457
|
+
Running it on every replica is safe: enqueues stay exactly-once via the unique
|
|
458
|
+
index on `solid_queue_recurring_executions (task_key, run_at)`. Static tasks come
|
|
459
|
+
from `config/recurring.yml` (honouring `SOLID_QUEUE_RECURRING_SCHEDULE`), dynamic
|
|
460
|
+
ones from the DB.
|
|
461
|
+
|
|
462
|
+
| option | default | meaning |
|
|
463
|
+
|---|---|---|
|
|
464
|
+
| `polling_interval` | `5` | how often the scheduler re-reads the dynamic tasks |
|
|
465
|
+
| `supervision_interval` | `5` | how often we check the scheduler thread is alive |
|
|
466
|
+
| `recurring_schedule_file` | `config/recurring.yml` | static task definitions; skipped when absent |
|
|
467
|
+
|
|
468
|
+
`start_scheduler!` is **not** gated on `enabled` — that switch owns the metrics
|
|
469
|
+
exporter, and an app may well want the scheduler with metrics off.
|
|
470
|
+
`scheduler_enabled` is the switch that does own it, shared with the sidekiq-cron
|
|
471
|
+
poller. Beyond that, what keeps it out of consoles and specs is *where* you call
|
|
472
|
+
it from.
|
|
473
|
+
|
|
474
|
+
### 2. Queue depth has to be visible
|
|
475
|
+
|
|
476
|
+
SolidQueue publishes no metrics, so the autoscaler and the dashboards have
|
|
477
|
+
nothing to read. `install_metrics!` adds two gauges, computed **at scrape time**
|
|
478
|
+
from the SolidQueue tables (a yabeda `collect` block — no background thread, no
|
|
479
|
+
cached snapshot):
|
|
480
|
+
|
|
481
|
+
```ruby
|
|
482
|
+
# config/initializers/rails_pod_kit.rb
|
|
483
|
+
RailsPodKit::SolidQueue.install_metrics!
|
|
484
|
+
```
|
|
485
|
+
|
|
486
|
+
| metric | meaning |
|
|
487
|
+
|---|---|
|
|
488
|
+
| `solid_queue_backlog` | how many jobs could be claimed right now, per `queue` |
|
|
489
|
+
| `solid_queue_latency_seconds` | how long the oldest of them has been waiting, per `queue` |
|
|
490
|
+
|
|
491
|
+
"Claimable right now" is ready executions **plus** scheduled ones whose time has
|
|
492
|
+
come — the dispatcher has only to move those across. A scheduled job's wait is
|
|
493
|
+
measured from its `scheduled_at`, not its `created_at`: enqueuing a week ahead of
|
|
494
|
+
the slot doesn't make it a week late.
|
|
495
|
+
|
|
496
|
+
Both matter, and neither alone is enough: backlog misses a small-but-stalled
|
|
497
|
+
queue, latency misses a large-but-moving one.
|
|
498
|
+
|
|
499
|
+
**An idle system reads 0, not no-data.** A queue that drains is explicitly zeroed
|
|
500
|
+
rather than left pinned at its last reading, and the zeroing starts from a
|
|
501
|
+
baseline of every queue the app is known to use — discovered once per process
|
|
502
|
+
from the jobs table, or pinned by the host:
|
|
503
|
+
|
|
504
|
+
```ruby
|
|
505
|
+
RailsPodKit::SolidQueue.install_metrics!(queues: %w[default mailers])
|
|
506
|
+
```
|
|
507
|
+
|
|
508
|
+
Without that baseline a process booting while the queue is empty — the steady
|
|
509
|
+
state of a scale-to-zero deployment — would publish no series at all, since a
|
|
510
|
+
gauge only exists once it has been set. Discovery is best-effort: the jobs table
|
|
511
|
+
is bounded by `clear_finished_jobs_after`, so a queue idle for longer than the
|
|
512
|
+
retention window leaves no trace in it. Pin `queues:` where the zero has to be
|
|
513
|
+
guaranteed.
|
|
514
|
+
|
|
515
|
+
**A collection error is always reported** (logged, plus handed to
|
|
516
|
+
`Rails.error`), then either swallowed or raised:
|
|
517
|
+
|
|
518
|
+
| | |
|
|
519
|
+
|---|---|
|
|
520
|
+
| `fail_scrape_on_error: false` (default) | serve the last reading. Right on an endpoint shared with the Puma or Sidekiq series, where failing the response would lose those too — at the cost of a stale gauge a consumer cannot tell apart from a live one. |
|
|
521
|
+
| `fail_scrape_on_error: true` (`run_exporter!`'s default) | fail the scrape. Right on the dedicated pod, where there is nothing else to protect: the gauges go to no-data and the scrape failure shows up in the scraper's own `up` series. |
|
|
522
|
+
|
|
523
|
+
### The always-on pod
|
|
524
|
+
|
|
525
|
+
The cleanest home for both is a **1-replica Deployment** that hosts the scheduler
|
|
526
|
+
and publishes the gauges, decoupled from the web and the executor — so the
|
|
527
|
+
signals survive either scaling to zero, and each series has exactly one source.
|
|
528
|
+
`run_exporter!` is that process: it declares the gauges, starts the scheduler,
|
|
529
|
+
serves `/metrics` and blocks until SIGTERM (winding the scheduler down so it
|
|
530
|
+
deregisters rather than expiring).
|
|
531
|
+
|
|
532
|
+
Unlike the Sidekiq global exporter it is **not** Rails-free — SolidQueue is
|
|
533
|
+
ActiveRecord-backed and reads the app's own tables — so the host's entrypoint
|
|
534
|
+
boots the environment first. The gem ships no executable; e.g.
|
|
535
|
+
`bin/solid-queue-pod`:
|
|
536
|
+
|
|
537
|
+
```ruby
|
|
538
|
+
#!/usr/bin/env ruby
|
|
539
|
+
require_relative '../config/environment'
|
|
540
|
+
|
|
541
|
+
RailsPodKit::SolidQueue.run_exporter!
|
|
542
|
+
```
|
|
543
|
+
|
|
544
|
+
```
|
|
545
|
+
command: ["bin/solid-queue-pod"]
|
|
546
|
+
```
|
|
547
|
+
|
|
548
|
+
Pass `scheduler: false` to serve the gauges only, on an app whose web process
|
|
549
|
+
already hosts the scheduler, and `metrics:` to override the gauge options (this
|
|
550
|
+
endpoint is the collector's own, so `fail_scrape_on_error` defaults to `true`
|
|
551
|
+
here):
|
|
552
|
+
|
|
553
|
+
```ruby
|
|
554
|
+
RailsPodKit::SolidQueue.run_exporter!(scheduler: false, metrics: { queues: %w[default mailers] })
|
|
555
|
+
```
|
|
556
|
+
|
|
557
|
+
Keep the endpoint single-prefix (see
|
|
558
|
+
[One prefix per endpoint](#datadog-naming--the-canonical-metric-set)) — that pod
|
|
559
|
+
serves `solid_queue_*` and nothing else, so the check config needs no filters.
|
|
560
|
+
|
|
316
561
|
## Caveats
|
|
317
562
|
|
|
318
563
|
- **Puma only.** The Puma plugins only activate under Puma; under any other
|
|
319
|
-
app server the in-process `/metrics` endpoint is **not** exposed.
|
|
564
|
+
app server the in-process `/metrics` endpoint is **not** exposed. The non-Puma
|
|
565
|
+
entry points (Sidekiq, the global exporter, the SolidQueue pod) serve it from
|
|
566
|
+
a WEBrick thread instead, started at most once per process.
|
|
567
|
+
- **SolidQueue and Sidekiq are the host's.** The gem depends on neither; the
|
|
568
|
+
SolidQueue integration is inert until you call it, exactly like the Sidekiq one.
|
|
569
|
+
`sidekiq-cron` too: it is required only when `GlobalScheduler.start!` is called,
|
|
570
|
+
so an app that does not schedule anything need not carry it.
|
|
571
|
+
- **Queue-gauge query cost.** The gauges run four small grouped aggregates per
|
|
572
|
+
scrape. `MIN(created_at)` is not covered by SolidQueue's indexes, so on a
|
|
573
|
+
backlog of many thousands of rows it is a scan — cheap at a normal scrape
|
|
574
|
+
interval, worth knowing about if you scrape aggressively.
|
|
320
575
|
- **Puma control app.** `yabeda-puma-plugin` reads Puma's thread-pool stats
|
|
321
576
|
through Puma's control app, so `Puma.activate` activates one on a
|
|
322
577
|
localhost-only socket (`no_token: true`, never network-exposed).
|
|
@@ -352,6 +607,10 @@ curl -s localhost:9394/metrics | grep '^puma_'
|
|
|
352
607
|
bundle exec sidekiq
|
|
353
608
|
curl -s localhost:9394/metrics | grep '^sidekiq_'
|
|
354
609
|
|
|
610
|
+
# SolidQueue: solid_queue_* series (enqueue a job first so the queue isn't empty)
|
|
611
|
+
bin/solid-queue-pod
|
|
612
|
+
curl -s localhost:9394/metrics | grep '^solid_queue_'
|
|
613
|
+
|
|
355
614
|
# Health endpoint
|
|
356
615
|
curl -s localhost:3000/healthz -H 'Accept: application/json'
|
|
357
616
|
```
|
data/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
0.0
|
|
1
|
+
0.2.0
|
data/lib/rails_pod_kit/config.rb
CHANGED
|
@@ -60,17 +60,29 @@ module RailsPodKit
|
|
|
60
60
|
# transports — the Puma plugin's exporter (`prometheus_silence_logger`)
|
|
61
61
|
# and the WEBrick exporter used by Sidekiq / the dedicated global exporter
|
|
62
62
|
# (`Rack::CommonLogger`). Flip to false only to debug the exporter itself.
|
|
63
|
+
# scheduler_enabled: kill switch for the hosted schedulers (the sidekiq-cron
|
|
64
|
+
# poller and the SolidQueue scheduler thread). Separate from `enabled`,
|
|
65
|
+
# which owns the metrics exporter: an app may well want one without the
|
|
66
|
+
# other, and the scheduler is the piece you may need to turn off in a
|
|
67
|
+
# hurry — it is a single point of failure for the whole schedule, and
|
|
68
|
+
# `RAILS_POD_KIT_SCHEDULER_ENABLED=false` + a restart beats a deploy at
|
|
69
|
+
# 3am. Defaults to on *including* in the test env, unlike `enabled`:
|
|
70
|
+
# nothing starts a scheduler implicitly (it takes an explicit call from an
|
|
71
|
+
# entry point), so there is no port to protect, and a flag that fails
|
|
72
|
+
# closed on a typo would silently stop a schedule.
|
|
63
73
|
attr_config :enabled,
|
|
64
74
|
:port,
|
|
65
75
|
:puma_control_url,
|
|
66
76
|
sidekiq_global_metrics: :web,
|
|
67
77
|
retries_segmented_by_queue: false,
|
|
68
|
-
silence_exporter_access_log: true
|
|
78
|
+
silence_exporter_access_log: true,
|
|
79
|
+
scheduler_enabled: true
|
|
69
80
|
|
|
70
81
|
coerce_types port: :integer,
|
|
71
82
|
enabled: :boolean,
|
|
72
83
|
retries_segmented_by_queue: :boolean,
|
|
73
|
-
silence_exporter_access_log: :boolean
|
|
84
|
+
silence_exporter_access_log: :boolean,
|
|
85
|
+
scheduler_enabled: :boolean
|
|
74
86
|
|
|
75
87
|
def initialize(overrides = nil)
|
|
76
88
|
super
|
|
@@ -87,6 +99,10 @@ module RailsPodKit
|
|
|
87
99
|
!!enabled
|
|
88
100
|
end
|
|
89
101
|
|
|
102
|
+
def scheduler_enabled?
|
|
103
|
+
!!scheduler_enabled
|
|
104
|
+
end
|
|
105
|
+
|
|
90
106
|
# True unless we're clearly in a test environment. Kept independent of
|
|
91
107
|
# Rails so the gem's own specs (which don't load the host app) get a safe
|
|
92
108
|
# default and never bind a socket.
|
|
@@ -114,5 +130,15 @@ module RailsPodKit
|
|
|
114
130
|
def enabled?
|
|
115
131
|
config.enabled?
|
|
116
132
|
end
|
|
133
|
+
|
|
134
|
+
# Gate shared by both hosted schedulers. Says so out loud when it turns one
|
|
135
|
+
# off: a process that starts, stays up and quietly schedules nothing is the
|
|
136
|
+
# one failure this whole feature exists to avoid.
|
|
137
|
+
def scheduler_enabled?(what = 'scheduler')
|
|
138
|
+
return true if config.scheduler_enabled?
|
|
139
|
+
|
|
140
|
+
warn "[rails_pod_kit] scheduler_enabled=false — #{what} not started"
|
|
141
|
+
false
|
|
142
|
+
end
|
|
117
143
|
end
|
|
118
144
|
end
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module RailsPodKit
|
|
4
|
+
# Where the gem's background work (the SolidQueue scheduler supervisor, the
|
|
5
|
+
# scrape-time metric collectors) sends the errors it swallows. Those errors
|
|
6
|
+
# must never propagate — one would kill the supervising timer or fail the whole
|
|
7
|
+
# /metrics response — but they must not vanish either.
|
|
8
|
+
#
|
|
9
|
+
# Always logs, and additionally hands the error to the Rails error reporter
|
|
10
|
+
# when there is one, so the host's Rollbar / Sentry / Datadog subscriber picks
|
|
11
|
+
# it up. The log line is not redundant: `Rails.error.report` only fans out to
|
|
12
|
+
# subscribers, so on an app with none — or with one that is not wired in a
|
|
13
|
+
# given environment — the failure would otherwise leave no trace at all, and
|
|
14
|
+
# the only symptom of a broken collector is a gauge quietly serving a stale
|
|
15
|
+
# value.
|
|
16
|
+
module ErrorReporter
|
|
17
|
+
module_function
|
|
18
|
+
|
|
19
|
+
def report(error, source:)
|
|
20
|
+
message = "[#{source}] #{error.class}: #{error.message}"
|
|
21
|
+
logger ? logger.error(message) : warn(message)
|
|
22
|
+
|
|
23
|
+
rails_reporter&.report(error, handled: true, source: source)
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def logger
|
|
27
|
+
::Rails.logger if defined?(::Rails) && ::Rails.respond_to?(:logger)
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def rails_reporter
|
|
31
|
+
::Rails.error if defined?(::Rails) && ::Rails.respond_to?(:error)
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
end
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'rails_pod_kit/config'
|
|
4
|
+
|
|
5
|
+
module RailsPodKit
|
|
6
|
+
# The in-process WEBrick /metrics server, shared by every non-Puma entry
|
|
7
|
+
# point: the Sidekiq worker, the Rails-free global exporter and the SolidQueue
|
|
8
|
+
# exporter. Under Puma the exporter comes from the `:yabeda_prometheus` plugin
|
|
9
|
+
# instead (see RailsPodKit::Puma), so this is never used there.
|
|
10
|
+
module Exporter
|
|
11
|
+
module_function
|
|
12
|
+
|
|
13
|
+
# Starts the background exporter and returns whether it did. Idempotent: the
|
|
14
|
+
# latch keeps a re-entrant boot from double-binding the port.
|
|
15
|
+
def start!
|
|
16
|
+
return false unless RailsPodKit.enabled?
|
|
17
|
+
return false if @started
|
|
18
|
+
|
|
19
|
+
require 'yabeda/prometheus/mmap'
|
|
20
|
+
|
|
21
|
+
# `start_metrics_server!` reads the bind port from the env, so publish the
|
|
22
|
+
# configured one first.
|
|
23
|
+
ENV['PROMETHEUS_EXPORTER_PORT'] ||= RailsPodKit.config.port.to_s
|
|
24
|
+
# Drop the exporter's per-scrape access log (Rack::CommonLogger, which the
|
|
25
|
+
# mmap exporter mounts unless this is exactly 'false'). See
|
|
26
|
+
# Config#silence_exporter_access_log.
|
|
27
|
+
ENV['PROMETHEUS_EXPORTER_LOG_REQUESTS'] = 'false' if RailsPodKit.config.silence_exporter_access_log
|
|
28
|
+
|
|
29
|
+
Yabeda::Prometheus::Exporter.start_metrics_server!
|
|
30
|
+
@started = true
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
def started?
|
|
34
|
+
!!@started
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
# Test/reset hook — drops the "already started" latch.
|
|
38
|
+
def reset!
|
|
39
|
+
@started = false
|
|
40
|
+
end
|
|
41
|
+
end
|
|
42
|
+
end
|
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require 'rails_pod_kit/config'
|
|
4
|
+
require 'rails_pod_kit/global_scheduler'
|
|
5
|
+
require 'rails_pod_kit/shutdown'
|
|
4
6
|
|
|
5
7
|
module RailsPodKit
|
|
6
8
|
# Standalone, always-on exporter for the Sidekiq global (Redis-wide) queue
|
|
@@ -26,6 +28,10 @@ module RailsPodKit
|
|
|
26
28
|
# require 'bundler/setup'
|
|
27
29
|
# require 'rails_pod_kit/global_exporter'
|
|
28
30
|
# RailsPodKit::GlobalExporter.run!(redis: { url: ENV['REDIS_URL'] })
|
|
31
|
+
#
|
|
32
|
+
# Being an always-on singleton also makes it the natural host for the
|
|
33
|
+
# sidekiq-cron poller (`scheduler: true`, see GlobalScheduler), which is what
|
|
34
|
+
# lets the workers scale to zero without losing their schedule.
|
|
29
35
|
module GlobalExporter
|
|
30
36
|
module_function
|
|
31
37
|
|
|
@@ -47,22 +53,34 @@ module RailsPodKit
|
|
|
47
53
|
# `redis:` takes the same options hash the host passes to its own
|
|
48
54
|
# `Sidekiq.configure_*` blocks (`url:`, `ssl_params:`, …), so the connection
|
|
49
55
|
# config stays a host decision with a single source of truth.
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
56
|
+
#
|
|
57
|
+
# `scheduler: true` additionally runs the sidekiq-cron poller here; any
|
|
58
|
+
# extra keywords are GlobalScheduler.start!'s (`schedule_file:`,
|
|
59
|
+
# `poll_interval:`, `supervision_interval:`). The scheduler is not gated on
|
|
60
|
+
# `RailsPodKit.enabled?` —
|
|
61
|
+
# that switch owns the metrics exporter, and an app may well want the
|
|
62
|
+
# singleton scheduler with metrics turned off.
|
|
63
|
+
def run!(redis:, scheduler: false, **scheduler_options)
|
|
64
|
+
serve_metrics = RailsPodKit.enabled?
|
|
65
|
+
warn '[rails_pod_kit] disabled — /metrics not served by the global exporter' unless serve_metrics
|
|
66
|
+
return unless serve_metrics || scheduler
|
|
55
67
|
|
|
56
68
|
configure_redis!(redis)
|
|
69
|
+
start_metrics! if serve_metrics
|
|
70
|
+
GlobalScheduler.start!(**scheduler_options) if scheduler
|
|
71
|
+
|
|
72
|
+
# Both the exporter and the poller serve from background threads; block
|
|
73
|
+
# the main thread so the process stays up until the kubelet sends SIGTERM.
|
|
74
|
+
Shutdown.await
|
|
75
|
+
GlobalScheduler.stop! if scheduler
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
def start_metrics!
|
|
57
79
|
install!
|
|
58
80
|
RailsPodKit::Sidekiq.start_metrics_server!
|
|
59
81
|
|
|
60
82
|
require 'yabeda'
|
|
61
83
|
Yabeda.configure! unless Yabeda.already_configured?
|
|
62
|
-
|
|
63
|
-
# The exporter serves from a background thread; block the main thread so
|
|
64
|
-
# the process stays up until the kubelet sends SIGTERM.
|
|
65
|
-
sleep
|
|
66
84
|
end
|
|
67
85
|
|
|
68
86
|
# Configure the Sidekiq client's Redis connection so this Rails-free process
|