solid_queue_guard 1.1.3 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +27 -0
- data/CONTRIBUTING.md +57 -0
- data/Gemfile.lock +1 -1
- data/README.md +64 -392
- data/app/assets/images/solid_queue_guard/mission-control-guard-dashboard.png +0 -0
- data/bin/console +8 -0
- data/docs/mission-control-guard-dashboard.png +0 -0
- data/lib/generators/solid_queue_guard/install/templates/solid_queue_guard.rb +1 -0
- data/lib/solid_queue_guard/checks/registry.rb +7 -0
- data/lib/solid_queue_guard/configuration.rb +49 -1
- data/lib/solid_queue_guard/engine.rb +1 -0
- data/lib/solid_queue_guard/health/cache.rb +8 -2
- data/lib/solid_queue_guard/health/payload.rb +1 -0
- data/lib/solid_queue_guard/metrics/exporter.rb +3 -0
- data/lib/solid_queue_guard/metrics/open_telemetry.rb +13 -5
- data/lib/solid_queue_guard/metrics/prometheus.rb +18 -4
- data/lib/solid_queue_guard/metrics/statsd.rb +16 -4
- data/lib/solid_queue_guard/mission_control/integration.rb +4 -3
- data/lib/solid_queue_guard/mission_control/navigation_extension.rb +2 -1
- data/lib/solid_queue_guard/notifier.rb +1 -0
- data/lib/solid_queue_guard/observability.rb +35 -0
- data/lib/solid_queue_guard/optional_dependency.rb +25 -0
- data/lib/solid_queue_guard/report.rb +5 -0
- data/lib/solid_queue_guard/runner.rb +1 -1
- data/lib/solid_queue_guard/version.rb +1 -1
- data/solid_queue_guard.gemspec +2 -1
- metadata +7 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 4ab3926371f9714622f421164d1436dcc74b17eb0d311627fff319c74a9369be
|
|
4
|
+
data.tar.gz: 13bd4acab34e3a6757bd5a4c445e471ab3021fc88d0748ca95a89a92cc597804
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 02d75b6056ddd2f698441b892c16d29f4155a0d213d598c5b0f813d0094529b32a761461071ca302cf170c2d1acfbd0e39339e2818b260261c006ba11ad73581
|
|
7
|
+
data.tar.gz: 9e57de7cf88e4ab6ef5d5ded2b98cd15c09374f0a46ecc311fadbe02d2952f4097fef99025848454b0734af35ed734b61f7f54093df255ad27c9c8ff5cf15f1a
|
data/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,33 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [1.2.0] - 2026-07-10
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
|
|
12
|
+
- `config.validate!` at boot — rejects invalid HTTP status codes, thresholds, and `on_status_change` types
|
|
13
|
+
- `config.on_status_change` callback when `/solid_queue_guard/health` detects a status transition
|
|
14
|
+
- Structured logging per check result (debug for pass/skip, warn/error for warn/fail)
|
|
15
|
+
- `status_counts` in report and health JSON payload
|
|
16
|
+
- Per-check metrics for StatsD, Prometheus, and OpenTelemetry exporters
|
|
17
|
+
- `SolidQueueGuard::Checks::Registry.catalog` for listing check IDs
|
|
18
|
+
- `bin/console` for dummy app debugging
|
|
19
|
+
- `CONTRIBUTING.md`, `docs/configuration.md`, and `docs/checks.md`
|
|
20
|
+
- CI: Ruby 3.1 matrix, dedicated Appraisal job, and `gem build` verification
|
|
21
|
+
- Mission Control navigation integration test from recurring tasks page
|
|
22
|
+
|
|
23
|
+
### Changed
|
|
24
|
+
|
|
25
|
+
- README trimmed with links to dedicated docs
|
|
26
|
+
- Mission Control screenshot shipped in gem package for RubyGems README rendering
|
|
27
|
+
- OpenTelemetry exporter requires `opentelemetry-sdk` with clear log when missing
|
|
28
|
+
|
|
29
|
+
## [1.1.4] - 2026-07-10
|
|
30
|
+
|
|
31
|
+
### Fixed
|
|
32
|
+
|
|
33
|
+
- Mission Control Guard tab now uses application-scoped URLs (`/applications/:application_id/guard`) with the engine mount prefix and `server_id`, matching Queues, Workers, and Recurring tasks navigation
|
|
34
|
+
|
|
8
35
|
## [1.1.3] - 2026-07-09
|
|
9
36
|
|
|
10
37
|
### Fixed
|
data/CONTRIBUTING.md
ADDED
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
# Contributing
|
|
2
|
+
|
|
3
|
+
Issues and PRs welcome. Keep changes focused on **production safety**, not another job dashboard.
|
|
4
|
+
|
|
5
|
+
## Setup
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
git clone https://github.com/rafael-pissardo/solid_queue_guard.git
|
|
9
|
+
cd solid_queue_guard
|
|
10
|
+
bundle install
|
|
11
|
+
cd test/dummy && bin/setup
|
|
12
|
+
cd ../..
|
|
13
|
+
bundle exec rake test
|
|
14
|
+
bundle exec rubocop
|
|
15
|
+
bundle exec appraisal install
|
|
16
|
+
bundle exec appraisal rake test
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Interactive console against the dummy app:
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
bin/console
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
## Adding a check
|
|
26
|
+
|
|
27
|
+
1. Create `lib/solid_queue_guard/checks/config/my_check.rb` or `checks/runtime/my_check.rb`
|
|
28
|
+
2. Register it in `lib/solid_queue_guard/checks/registry.rb`
|
|
29
|
+
3. Add tests under `test/solid_queue_guard/checks/`
|
|
30
|
+
4. Document the check ID in [docs/checks.md](docs/checks.md)
|
|
31
|
+
|
|
32
|
+
Check IDs are derived from the class name (`MyFeatureCheck` → `my_feature`).
|
|
33
|
+
|
|
34
|
+
## Testing against Revelo-style apps
|
|
35
|
+
|
|
36
|
+
Use `script/validate_revelo.sh` to run the doctor and health tasks against local Revelo backends with a path gem:
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
./script/validate_revelo.sh
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Edit the script to point at your checkout paths. The script temporarily adds a `path:` gem line, runs `solid_queue_guard:doctor` and `solid_queue_guard:health`, then restores the Gemfile.
|
|
43
|
+
|
|
44
|
+
## Release checklist
|
|
45
|
+
|
|
46
|
+
- [ ] Tests pass locally and in CI (matrix + Appraisal + `gem build`)
|
|
47
|
+
- [ ] RuboCop clean
|
|
48
|
+
- [ ] CHANGELOG updated
|
|
49
|
+
- [ ] Version bumped in `lib/solid_queue_guard/version.rb` and lockfiles synced
|
|
50
|
+
- [ ] README / docs updated for public API changes
|
|
51
|
+
- [ ] Tag matches gem version: `git tag vX.Y.Z && git push origin vX.Y.Z`
|
|
52
|
+
|
|
53
|
+
## Code style
|
|
54
|
+
|
|
55
|
+
- Follow existing patterns: small checks, `Check::Result`, registry-driven runner
|
|
56
|
+
- Mark internal APIs with `# @api private`
|
|
57
|
+
- Public API changes require semver discipline (see README)
|
data/Gemfile.lock
CHANGED
data/README.md
CHANGED
|
@@ -1,44 +1,57 @@
|
|
|
1
|
-
|
|
1
|
+
<p align="center">
|
|
2
|
+
<img src="https://img.shields.io/gem/v/solid_queue_guard?style=for-the-badge" alt="Gem Version">
|
|
3
|
+
<img src="https://img.shields.io/badge/Ruby-3.1%2B-red?style=for-the-badge&logo=ruby" alt="Ruby 3.1+">
|
|
4
|
+
<img src="https://img.shields.io/badge/Rails-7.1–8.0-red?style=for-the-badge&logo=rubyonrails" alt="Rails 7.1–8.0">
|
|
5
|
+
<img src="https://img.shields.io/github/actions/workflow/status/rafael-pissardo/solid_queue_guard/ci.yml?branch=main&style=for-the-badge&label=CI" alt="CI">
|
|
6
|
+
<img src="https://img.shields.io/badge/License-MIT-blue?style=for-the-badge" alt="MIT License">
|
|
7
|
+
</p>
|
|
8
|
+
|
|
9
|
+
<h1 align="center">solid_queue_guard</h1>
|
|
2
10
|
|
|
3
|
-
|
|
11
|
+
<p align="center">
|
|
12
|
+
<strong>Your Solid Queue production doctor.</strong><br>
|
|
13
|
+
Catch dead workers, queue lag, and broken config <em>before</em> the 3am page.
|
|
14
|
+
</p>
|
|
4
15
|
|
|
5
|
-
|
|
6
|
-
|
|
16
|
+
<p align="center">
|
|
17
|
+
<a href="#-quick-start"><strong>Quick start</strong></a> ·
|
|
18
|
+
<a href="docs/checks.md"><strong>Checks</strong></a> ·
|
|
19
|
+
<a href="docs/configuration.md"><strong>Configuration</strong></a> ·
|
|
20
|
+
<a href="#-mission-control-dashboard"><strong>Mission Control</strong></a>
|
|
21
|
+
</p>
|
|
7
22
|
|
|
8
|
-
|
|
23
|
+
---
|
|
9
24
|
|
|
10
|
-
|
|
25
|
+
Rails 8 ships with [Solid Queue](https://github.com/rails/solid_queue). Redis is optional. **Production config is not.**
|
|
11
26
|
|
|
12
|
-
**solid_queue_guard** catches that *before* it becomes
|
|
27
|
+
Your web tier can be **green** while your jobs are **dead**. **solid_queue_guard** catches that *before* it becomes an incident.
|
|
13
28
|
|
|
14
29
|
> **Mission Control** shows what is happening.
|
|
15
30
|
> **solid_queue_guard** warns what is dangerous.
|
|
16
31
|
|
|
32
|
+
### Install in 30 seconds
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
bundle add solid_queue_guard
|
|
36
|
+
bin/rails solid_queue_guard:install
|
|
37
|
+
bin/rails solid_queue_guard:doctor
|
|
38
|
+
```
|
|
39
|
+
|
|
17
40
|
---
|
|
18
41
|
|
|
19
42
|
## Why this exists
|
|
20
43
|
|
|
21
|
-
Solid Queue is excellent. Operating it blindly is not.
|
|
22
|
-
|
|
23
44
|
| Symptom | What actually happened |
|
|
24
45
|
| ------- | ---------------------- |
|
|
25
46
|
| "Site is up, emails stopped" | Workers dead, heartbeats stale |
|
|
26
47
|
| "Only 3 jobs in the queue, why the panic?" | Oldest job waiting 40 minutes — **lag**, not depth |
|
|
27
|
-
| "Recurring billing just… stopped" | Scheduler not running
|
|
28
|
-
| "Jobs hang after deploy" | Thread count > DB pool
|
|
48
|
+
| "Recurring billing just… stopped" | Scheduler not running |
|
|
49
|
+
| "Jobs hang after deploy" | Thread count > DB pool |
|
|
29
50
|
| "Health check passes, jobs don't" | `/up` doesn't know Solid Queue exists |
|
|
30
51
|
|
|
31
|
-
You don't need another dashboard. You need a **doctor**.
|
|
32
|
-
|
|
33
52
|
---
|
|
34
53
|
|
|
35
|
-
## See it in action
|
|
36
|
-
|
|
37
|
-
```bash
|
|
38
|
-
bundle add solid_queue_guard
|
|
39
|
-
bin/rails solid_queue_guard:install
|
|
40
|
-
bin/rails solid_queue_guard:doctor
|
|
41
|
-
```
|
|
54
|
+
## 👨⚕️ See it in action
|
|
42
55
|
|
|
43
56
|
```text
|
|
44
57
|
SolidQueueGuard Report
|
|
@@ -47,109 +60,29 @@ Status: DEGRADED
|
|
|
47
60
|
|
|
48
61
|
Checks:
|
|
49
62
|
✅ Active Job adapter is :solid_queue
|
|
50
|
-
✅ Queue database configured in database.yml
|
|
51
|
-
✅ Solid Queue connects_to queue database (pool: 10)
|
|
52
|
-
✅ Solid Queue schema tables present
|
|
53
63
|
❌ Worker threads: 10, queue DB pool: 5
|
|
54
64
|
⚠️ No workers configured for "mailers" queue
|
|
55
|
-
⚠️ recurring.yml exists but scheduler may not run
|
|
56
|
-
✅ Solid Queue is not co-located with Puma
|
|
57
65
|
|
|
58
66
|
Suggested fixes:
|
|
59
67
|
- Increase queue DB pool to at least 12 or reduce worker threads
|
|
60
|
-
- Add a worker for the mailers queue
|
|
61
|
-
- Verify scheduler process is running in production
|
|
62
68
|
```
|
|
63
69
|
|
|
64
|
-
One command. Actionable output. No Datadog required to get started
|
|
70
|
+
**One command. Actionable output. No Datadog required to get started.**
|
|
65
71
|
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
## What it checks
|
|
69
|
-
|
|
70
|
-
### Configuration doctor
|
|
71
|
-
|
|
72
|
-
Runs locally, in CI, or pre-deploy. **No extra infrastructure.**
|
|
73
|
-
|
|
74
|
-
| Check | Catches |
|
|
75
|
-
| ----- | ------- |
|
|
76
|
-
| **Adapter** | Active Job not set to `:solid_queue` |
|
|
77
|
-
| **Queue database** | Missing `queue` entry in `database.yml` |
|
|
78
|
-
| **connects_to** | Solid Queue pointing at the wrong DB / pool |
|
|
79
|
-
| **Queue schema** | Missing Solid Queue tables in schema files or queue database |
|
|
80
|
-
| **Thread pool** | `threads` > available queue DB connections |
|
|
81
|
-
| **Worker coverage** | Queues with no worker assigned |
|
|
82
|
-
| **Scheduler** | `recurring.yml` tasks without a scheduler |
|
|
83
|
-
| **Env flags** | `SOLID_QUEUE_SKIP_RECURRING=true` in production |
|
|
84
|
-
| **Heartbeats** | Default thresholds that may not fit your deploy |
|
|
85
|
-
| **Puma plugin** | Solid Queue co-located with web in production |
|
|
86
|
-
| **Async supervisor** | `async` mode with thread/pool sizing risks |
|
|
87
|
-
| **Topology** | `queue.yml` worker and pool recommendations |
|
|
88
|
-
|
|
89
|
-
### Runtime guards
|
|
90
|
-
|
|
91
|
-
| Check | Catches |
|
|
92
|
-
| ----- | ------- |
|
|
93
|
-
| **Queue lag** | Oldest ready job waiting too long |
|
|
94
|
-
| **Stale processes** | Dead workers, dispatchers, schedulers |
|
|
95
|
-
| **Dispatcher health** | Scheduled jobs never becoming ready |
|
|
96
|
-
| **Blocked jobs** | Concurrency control silently holding jobs |
|
|
97
|
-
| **Recurring staleness** | Cron tasks that stopped firing |
|
|
98
|
-
| **Puma plugin runtime** | Plugin enabled but no active Solid Queue processes |
|
|
99
|
-
| **HTTP `/health`** | Kamal, ECS, K8s, UptimeRobot integration |
|
|
72
|
+
Full check list: [docs/checks.md](docs/checks.md)
|
|
100
73
|
|
|
101
74
|
---
|
|
102
75
|
|
|
103
|
-
## Quick start
|
|
104
|
-
|
|
105
|
-
### Install
|
|
76
|
+
## 🚀 Quick start
|
|
106
77
|
|
|
107
78
|
```bash
|
|
108
79
|
bundle add solid_queue_guard
|
|
109
80
|
bin/rails solid_queue_guard:install
|
|
110
|
-
bin/rails generate solid_queue_guard:install:ci # optional GitHub Actions workflow
|
|
111
|
-
```
|
|
112
|
-
|
|
113
|
-
### Run the doctor
|
|
114
|
-
|
|
115
|
-
```bash
|
|
116
|
-
# Human-readable report
|
|
117
81
|
bin/rails solid_queue_guard:doctor
|
|
118
|
-
|
|
119
|
-
#
|
|
120
|
-
bin/rails "solid_queue_guard:doctor[json]"
|
|
121
|
-
SOLID_QUEUE_GUARD_FORMAT=json bin/rails solid_queue_guard:doctor
|
|
122
|
-
|
|
123
|
-
# Fail CI on warnings too
|
|
124
|
-
SOLID_QUEUE_GUARD_STRICT=1 bin/rails solid_queue_guard:doctor
|
|
125
|
-
|
|
126
|
-
# Full report (config + runtime when available)
|
|
127
|
-
bin/rails solid_queue_guard:report
|
|
128
|
-
```
|
|
129
|
-
|
|
130
|
-
### Exit codes
|
|
131
|
-
|
|
132
|
-
| Code | Meaning |
|
|
133
|
-
| ---- | ------- |
|
|
134
|
-
| `0` | Healthy or degraded |
|
|
135
|
-
| `1` | Unhealthy (or degraded in `--strict` / `SOLID_QUEUE_GUARD_STRICT=1`) |
|
|
136
|
-
|
|
137
|
-
Perfect for deploy pipelines:
|
|
138
|
-
|
|
139
|
-
```yaml
|
|
140
|
-
# GitHub Actions
|
|
141
|
-
- name: Solid Queue production readiness
|
|
142
|
-
run: SOLID_QUEUE_GUARD_STRICT=1 bin/rails solid_queue_guard:doctor
|
|
143
|
-
```
|
|
144
|
-
|
|
145
|
-
Or generate a workflow:
|
|
146
|
-
|
|
147
|
-
```bash
|
|
148
|
-
bin/rails generate solid_queue_guard:install:ci
|
|
82
|
+
SOLID_QUEUE_GUARD_STRICT=1 bin/rails solid_queue_guard:doctor # CI gate
|
|
83
|
+
bin/rails solid_queue_guard:report # config + runtime
|
|
149
84
|
```
|
|
150
85
|
|
|
151
|
-
### HTTP health
|
|
152
|
-
|
|
153
86
|
```ruby
|
|
154
87
|
# config/routes.rb
|
|
155
88
|
mount SolidQueueGuard::Engine, at: "/solid_queue_guard"
|
|
@@ -157,298 +90,66 @@ mount SolidQueueGuard::Engine, at: "/solid_queue_guard"
|
|
|
157
90
|
|
|
158
91
|
```bash
|
|
159
92
|
curl localhost:3000/solid_queue_guard/health
|
|
160
|
-
# => { "status": "degraded", "queue_lag_seconds": 245, "warnings": [...] }
|
|
161
|
-
```
|
|
162
|
-
|
|
163
|
-
Optional token protection:
|
|
164
|
-
|
|
165
|
-
```ruby
|
|
166
|
-
config.health_token = ENV["SOLID_QUEUE_GUARD_TOKEN"]
|
|
167
|
-
# curl -H "X-Solid-Queue-Guard-Token: $TOKEN" ...
|
|
168
93
|
```
|
|
169
94
|
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
```ruby
|
|
173
|
-
config.degraded_http_status = 207 # or :ok (200), 503, etc.
|
|
174
|
-
config.unhealthy_http_status = 503 # default
|
|
175
|
-
```
|
|
176
|
-
|
|
177
|
-
Works with **Kamal**, **Heroku**, **Fly.io**, **ECS/Fargate**, **Kubernetes**, **Better Stack**, **UptimeRobot**.
|
|
178
|
-
|
|
179
|
-
### Mission Control dashboard (opt-in)
|
|
180
|
-
|
|
181
|
-
Add a **Guard** tab to [Mission Control — Jobs](https://github.com/rails/mission_control-jobs) — same Bulma UI, same auth, same navigation:
|
|
182
|
-
|
|
183
|
-
```ruby
|
|
184
|
-
# Gemfile
|
|
185
|
-
gem "mission_control-jobs"
|
|
186
|
-
gem "solid_queue_guard"
|
|
95
|
+
All configuration options: [docs/configuration.md](docs/configuration.md)
|
|
187
96
|
|
|
188
|
-
|
|
189
|
-
SolidQueueGuard.configure do |config|
|
|
190
|
-
config.integrate_mission_control = true
|
|
191
|
-
end
|
|
192
|
-
|
|
193
|
-
# config/routes.rb
|
|
194
|
-
mount MissionControl::Jobs::Engine, at: "/jobs"
|
|
195
|
-
mount SolidQueueGuard::Engine, at: "/solid_queue_guard" # keep /health for probes
|
|
196
|
-
```
|
|
197
|
-
|
|
198
|
-
Open `/jobs/guard` (or click the **Guard** tab inside Mission Control). The page shows:
|
|
199
|
-
|
|
200
|
-
| Section | Content |
|
|
201
|
-
| ------- | ------- |
|
|
202
|
-
| **Overall status** | `healthy` / `degraded` / `unhealthy` badge |
|
|
203
|
-
| **Metrics** | Queue lag, failed jobs (1h), dead processes, check count |
|
|
204
|
-
| **Warnings & suggestions** | Same messages as `doctor` / `/health` |
|
|
205
|
-
| **Checks table** | Every check with status tag, message, and suggestion |
|
|
206
|
-
|
|
207
|
-
**Authentication:** the Guard tab inherits Mission Control auth — HTTP basic (default) or your admin controller via `MissionControl::Jobs.base_controller_class`. No separate login.
|
|
208
|
-
|
|
209
|
-
**Load balancers:** keep using `GET /solid_queue_guard/health` with optional `health_token`. Probes do not go through Mission Control.
|
|
210
|
-
|
|
211
|
-
**Requirements:** an asset pipeline (Propshaft or Sprockets), same as Mission Control. `mission_control-jobs` is optional — only needed when `integrate_mission_control` is enabled.
|
|
212
|
-
|
|
213
|
-
---
|
|
214
|
-
|
|
215
|
-
## How to use it
|
|
216
|
-
|
|
217
|
-
Solid Queue does not show up in Rails `/up`. **solid_queue_guard** gives you four operational surfaces:
|
|
97
|
+
### Operational surfaces
|
|
218
98
|
|
|
219
99
|
| Surface | Command / URL | Best for |
|
|
220
100
|
| ------- | ------------- | -------- |
|
|
221
101
|
| **Doctor** | `bin/rails solid_queue_guard:doctor` | Local pre-deploy, config review |
|
|
222
|
-
| **CI gate** | `SOLID_QUEUE_GUARD_STRICT=1 bin/rails solid_queue_guard:doctor` | Block merges
|
|
223
|
-
| **HTTP health** | `GET /solid_queue_guard/health` |
|
|
224
|
-
| **Guard tab** | `GET /
|
|
225
|
-
|
|
226
|
-
**Mission Control** shows what is happening. **solid_queue_guard** warns what is dangerous. Use both.
|
|
102
|
+
| **CI gate** | `SOLID_QUEUE_GUARD_STRICT=1 bin/rails solid_queue_guard:doctor` | Block bad merges |
|
|
103
|
+
| **HTTP health** | `GET /solid_queue_guard/health` | Kamal, ECS, K8s, UptimeRobot |
|
|
104
|
+
| **Guard tab** | `GET …/applications/:application_id/guard` | Human checks in Mission Control |
|
|
227
105
|
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
Run before changing `queue.yml`, `database.yml`, or recurring tasks:
|
|
106
|
+
---
|
|
231
107
|
|
|
232
|
-
|
|
233
|
-
bin/rails solid_queue_guard:doctor # config checks (default scope)
|
|
234
|
-
bin/rails solid_queue_guard:report # config + runtime when the queue DB is available
|
|
235
|
-
```
|
|
108
|
+
## 🛡️ Mission Control dashboard
|
|
236
109
|
|
|
237
|
-
|
|
110
|
+
Opt-in **Guard** tab for [Mission Control — Jobs](https://github.com/rails/mission_control-jobs):
|
|
238
111
|
|
|
239
112
|
```ruby
|
|
240
|
-
|
|
241
|
-
```
|
|
242
|
-
|
|
243
|
-
Or run with production config locally when validating deploy readiness.
|
|
244
|
-
|
|
245
|
-
### CI pipelines
|
|
246
|
-
|
|
247
|
-
Validate **configuration** before deploy. Process/runtime checks usually **skip** in CI because there is no `bin/jobs` supervisor on the runner.
|
|
248
|
-
|
|
249
|
-
```yaml
|
|
250
|
-
- name: Solid Queue production readiness
|
|
251
|
-
run: SOLID_QUEUE_GUARD_STRICT=1 bin/rails solid_queue_guard:doctor
|
|
252
|
-
```
|
|
253
|
-
|
|
254
|
-
`STRICT=1` turns warnings into exit code `1`, so the pipeline fails on misconfigured pools, missing worker coverage, or missing schema tables — not only hard failures.
|
|
255
|
-
|
|
256
|
-
Generate a starter workflow:
|
|
257
|
-
|
|
258
|
-
```bash
|
|
259
|
-
bin/rails generate solid_queue_guard:install:ci
|
|
260
|
-
```
|
|
261
|
-
|
|
262
|
-
### Production monitoring
|
|
113
|
+
gem "mission_control-jobs"
|
|
263
114
|
|
|
264
|
-
|
|
115
|
+
SolidQueueGuard.configure { |config| config.integrate_mission_control = true }
|
|
265
116
|
|
|
266
|
-
|
|
267
|
-
# config/routes.rb
|
|
117
|
+
mount MissionControl::Jobs::Engine, at: "/jobs"
|
|
268
118
|
mount SolidQueueGuard::Engine, at: "/solid_queue_guard"
|
|
269
119
|
```
|
|
270
120
|
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
Optional hardening:
|
|
121
|
+

|
|
274
122
|
|
|
275
|
-
|
|
276
|
-
config.health_token = ENV["SOLID_QUEUE_GUARD_TOKEN"]
|
|
277
|
-
config.health_cache_ttl = 15.seconds
|
|
278
|
-
config.degraded_http_status = 207 # or :ok (200), 503, etc.
|
|
279
|
-
config.notify_with = [:rails_logger, :slack]
|
|
280
|
-
```
|
|
281
|
-
|
|
282
|
-
### Typical flow
|
|
283
|
-
|
|
284
|
-
```text
|
|
285
|
-
Developer CI Production
|
|
286
|
-
│ │ │
|
|
287
|
-
│ doctor │ doctor --strict │ GET /health (every 30–60s)
|
|
288
|
-
▼ ▼ ▼
|
|
289
|
-
"pool wrong?" block bad deploy alert: worker dead
|
|
290
|
-
"schema ok?" before merge alert: queue lag
|
|
291
|
-
```
|
|
292
|
-
|
|
293
|
-
---
|
|
294
|
-
|
|
295
|
-
## Runtime process checks
|
|
296
|
-
|
|
297
|
-
Runtime checks query the **queue database** (via `SolidQueue::Record`). If `connects_to` is missing or the DB is unreachable, they **skip** — they do not pass silently as healthy.
|
|
298
|
-
|
|
299
|
-
### `process_topology` — are the expected roles present?
|
|
300
|
-
|
|
301
|
-
Reads distinct `kind` values from `solid_queue_processes`:
|
|
302
|
-
|
|
303
|
-
| Kind | Role |
|
|
304
|
-
| ---- | ---- |
|
|
305
|
-
| `Supervisor` | Parent process (`bin/jobs`) |
|
|
306
|
-
| `Worker` | Consumes ready jobs |
|
|
307
|
-
| `Dispatcher` | Moves scheduled jobs to ready |
|
|
308
|
-
| `Scheduler` | Runs recurring tasks |
|
|
309
|
-
|
|
310
|
-
| Result | Meaning |
|
|
311
|
-
| ------ | ------- |
|
|
312
|
-
| ⚠️ No processes | Nothing registered — supervisor not running |
|
|
313
|
-
| ⚠️ No Worker | Jobs will not be processed |
|
|
314
|
-
| ⚠️ No Dispatcher | Scheduled work may stall (when recurring/scheduled jobs exist) |
|
|
315
|
-
| ✅ Pass | Expected kinds are present |
|
|
316
|
-
|
|
317
|
-
This check looks at **presence of records**, not heartbeat freshness.
|
|
318
|
-
|
|
319
|
-
### `stale_process` — are heartbeats fresh?
|
|
320
|
-
|
|
321
|
-
Finds processes where `last_heartbeat_at` is older than `stale_process_threshold` (default **5 minutes**):
|
|
322
|
-
|
|
323
|
-
| Result | Meaning |
|
|
324
|
-
| ------ | ------- |
|
|
325
|
-
| ✅ Pass | All processes reported recently |
|
|
326
|
-
| ❌ Fail | One or more workers/dispatchers look dead or stuck |
|
|
327
|
-
|
|
328
|
-
Use this in production health to catch workers that died after deploy.
|
|
329
|
-
|
|
330
|
-
### `pidfile` — optional supervisor liveness
|
|
331
|
-
|
|
332
|
-
When `tmp/pids/solid_queue.pid` (or `SOLID_QUEUE_PIDFILE`) exists, verifies the PID is alive. Often **warns in development** where `bin/jobs` is not running. Disable if you do not use pidfiles:
|
|
333
|
-
|
|
334
|
-
```ruby
|
|
335
|
-
config.disabled_checks = [:pidfile]
|
|
336
|
-
```
|
|
337
|
-
|
|
338
|
-
### `puma_plugin_runtime` — Solid Queue inside Puma
|
|
339
|
-
|
|
340
|
-
When `plugin :solid_queue` is in `config/puma.rb`, verifies active processes with recent heartbeats exist. **Skips** when the Puma plugin is not enabled.
|
|
341
|
-
|
|
342
|
-
### Queue schema detection
|
|
343
|
-
|
|
344
|
-
`QueueSchemaCheck` does **not** require `db/queue_schema.rb`. It validates that all tables for your installed **solid_queue** version exist in any of:
|
|
345
|
-
|
|
346
|
-
- `db/queue_schema.rb`
|
|
347
|
-
- `db/schema.rb`
|
|
348
|
-
- `db/structure.sql`
|
|
349
|
-
- the connected queue database
|
|
350
|
-
|
|
351
|
-
Apps that keep Solid Queue tables only in `structure.sql` (common in Revelo-style repos) pass correctly.
|
|
352
|
-
|
|
353
|
-
### Where each check type applies
|
|
354
|
-
|
|
355
|
-
| Context | Config checks | Process / runtime checks |
|
|
356
|
-
| ------- | ------------- | ------------------------ |
|
|
357
|
-
| Local `doctor` | ✅ Primary value | ⚠️ Partial without `bin/jobs` |
|
|
358
|
-
| CI `--strict` | ✅ Primary value | ⏭️ Usually skip |
|
|
359
|
-
| Production `/health` | ✅ When DB up | ✅ Primary value |
|
|
360
|
-
|
|
361
|
-
---
|
|
362
|
-
|
|
363
|
-
## Configuration
|
|
364
|
-
|
|
365
|
-
```ruby
|
|
366
|
-
# config/initializers/solid_queue_guard.rb
|
|
367
|
-
SolidQueueGuard.configure do |config|
|
|
368
|
-
config.enabled = %w[production staging].include?(Rails.env)
|
|
369
|
-
|
|
370
|
-
config.queue_lag_thresholds = {
|
|
371
|
-
critical: 30.seconds,
|
|
372
|
-
default: 5.minutes,
|
|
373
|
-
mailers: 15.minutes
|
|
374
|
-
}
|
|
375
|
-
|
|
376
|
-
config.failed_jobs_threshold = 20
|
|
377
|
-
config.stale_process_threshold = 5.minutes
|
|
378
|
-
|
|
379
|
-
# Per-check overrides (v0.6+)
|
|
380
|
-
config.disabled_checks = [:pidfile]
|
|
381
|
-
config.checks.queue_lag = { threshold: 10.minutes }
|
|
382
|
-
config.checks.failed_jobs = { threshold: 5, enabled: true }
|
|
383
|
-
|
|
384
|
-
# HTTP status policy (v0.8+)
|
|
385
|
-
# config.degraded_http_status = 207
|
|
386
|
-
# config.unhealthy_http_status = 503
|
|
387
|
-
|
|
388
|
-
# config.health_token = ENV["SOLID_QUEUE_GUARD_TOKEN"]
|
|
389
|
-
# config.integrate_rails_health = true
|
|
390
|
-
# config.integrate_mission_control = true # Guard tab in Mission Control (requires mission_control-jobs)
|
|
391
|
-
# config.notify_with = [:rails_logger, :slack, :datadog, :webhook]
|
|
392
|
-
# config.metrics_backends = [:statsd, :prometheus, :opentelemetry]
|
|
393
|
-
end
|
|
394
|
-
```
|
|
123
|
+
Requires `mission_control-jobs` and an asset pipeline (Propshaft or Sprockets). Load balancers should keep using `/solid_queue_guard/health`.
|
|
395
124
|
|
|
396
125
|
---
|
|
397
126
|
|
|
398
127
|
## Public API (v1.0+)
|
|
399
128
|
|
|
400
|
-
|
|
129
|
+
Stable until `2.0` — [semantic versioning](https://semver.org/):
|
|
401
130
|
|
|
402
131
|
| API | Description |
|
|
403
132
|
| --- | ----------- |
|
|
404
133
|
| `SolidQueueGuard.configure` | Block-style configuration |
|
|
405
|
-
| `SolidQueueGuard.config` | Current configuration object |
|
|
406
|
-
| `SolidQueueGuard.enabled?` | Whether checks run |
|
|
407
134
|
| `solid_queue_guard:doctor` | Config readiness report |
|
|
408
|
-
| `solid_queue_guard:health` | Runtime health report |
|
|
409
135
|
| `solid_queue_guard:report` | Full diagnostic report |
|
|
410
|
-
| `solid_queue_guard:install` | Initializer generator |
|
|
411
|
-
| `solid_queue_guard:install:ci` | GitHub Actions workflow generator |
|
|
412
136
|
| `mount SolidQueueGuard::Engine` | HTTP health endpoint |
|
|
413
|
-
| `config.integrate_mission_control` | Guard tab
|
|
414
|
-
| `GET /jobs/guard` | Human-readable health dashboard (when integration enabled) |
|
|
137
|
+
| `config.integrate_mission_control` | Guard tab (requires `mission_control-jobs`) |
|
|
415
138
|
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
Breaking changes ship only in major versions (`2.0+`). Deprecations warn one minor version ahead.
|
|
139
|
+
Internal check classes and registry are `@api private`.
|
|
419
140
|
|
|
420
141
|
---
|
|
421
142
|
|
|
422
|
-
##
|
|
143
|
+
## ⚔️ vs Mission Control
|
|
423
144
|
|
|
424
|
-
| |
|
|
145
|
+
| | Mission Control | solid_queue_guard |
|
|
425
146
|
| --- | --- | --- |
|
|
426
147
|
| **Purpose** | Inspect & manage jobs | Detect production risk |
|
|
427
|
-
| **UI** | Dashboard | CLI + JSON + health endpoint + optional Guard tab in Mission Control |
|
|
428
|
-
| **Retry / discard** | Yes | No (use Mission Control) |
|
|
429
148
|
| **Config doctor** | No | Yes |
|
|
430
|
-
| **Queue lag alerts** | No | Yes |
|
|
431
149
|
| **Pre-deploy checks** | No | Yes |
|
|
432
|
-
| **
|
|
433
|
-
|
|
434
|
-
**Use both.** They solve different problems.
|
|
435
|
-
|
|
436
|
-
---
|
|
150
|
+
| **Queue lag alerts** | No | Yes |
|
|
437
151
|
|
|
438
|
-
|
|
439
|
-
|
|
440
|
-
| Version | Ships | Status |
|
|
441
|
-
| ------- | ----- | ------ |
|
|
442
|
-
| **v0.1** | `doctor` — config checks, CI integration, install generator | ✅ Released |
|
|
443
|
-
| **v0.2** | Runtime health, queue lag, dispatcher/blocked jobs, HTTP endpoint | ✅ Released |
|
|
444
|
-
| **v0.3** | Slack, Datadog, webhook notifications | ✅ Released |
|
|
445
|
-
| **v0.4** | StatsD, Prometheus, OpenTelemetry metrics | ✅ Released |
|
|
446
|
-
| **v0.5** | Auto-recommendations for `queue.yml` topology | ✅ Released |
|
|
447
|
-
| **v0.6** | Per-check configuration (`disabled_checks`, `config.checks`) | ✅ Released |
|
|
448
|
-
| **v0.7** | Puma plugin runtime check, async supervisor awareness | ✅ Released |
|
|
449
|
-
| **v0.8** | HTTP status policy, `install:ci` generator | ✅ Released |
|
|
450
|
-
| **v1.0** | Stable public API, strict semver | ✅ Released |
|
|
451
|
-
| **v1.1** | Mission Control Guard tab (opt-in) | ✅ Released |
|
|
152
|
+
**Use both.**
|
|
452
153
|
|
|
453
154
|
---
|
|
454
155
|
|
|
@@ -456,55 +157,26 @@ Breaking changes ship only in major versions (`2.0+`). Deprecations warn one min
|
|
|
456
157
|
|
|
457
158
|
| Gem version | Ruby | Rails |
|
|
458
159
|
| ----------- | ---- | ----- |
|
|
160
|
+
| 1.2.x | 3.1+ | 7.1, 7.2, 8.0 |
|
|
459
161
|
| 1.1.x | 3.1+ | 7.1, 7.2, 8.0 |
|
|
460
162
|
| 1.0.x | 3.1+ | 7.1, 7.2, 8.0 |
|
|
461
|
-
| 0.5.x | 3.1+ | 7.1, 7.2, 8.0 |
|
|
462
|
-
| 0.1.x | 3.1+ | 7.1, 7.2, 8.0 |
|
|
463
|
-
|
|
464
|
-
## Requirements
|
|
465
163
|
|
|
466
|
-
- Ruby >= 3.1
|
|
467
|
-
- Rails >= 7.1, < 9.0
|
|
468
164
|
- [solid_queue](https://github.com/rails/solid_queue) >= 1.0, < 2.0
|
|
469
|
-
- [mission_control-jobs](https://github.com/rails/mission_control-jobs)
|
|
165
|
+
- [mission_control-jobs](https://github.com/rails/mission_control-jobs) — optional
|
|
470
166
|
|
|
471
167
|
---
|
|
472
168
|
|
|
473
169
|
## Development
|
|
474
170
|
|
|
171
|
+
See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, `bin/console`, Appraisal matrix, and `script/validate_revelo.sh`.
|
|
172
|
+
|
|
475
173
|
```bash
|
|
476
|
-
|
|
477
|
-
cd solid_queue_guard
|
|
478
|
-
bundle install
|
|
479
|
-
cd test/dummy && bin/setup
|
|
480
|
-
cd ../.. && bundle exec rake test
|
|
174
|
+
bundle exec rake test
|
|
481
175
|
bundle exec rubocop
|
|
482
|
-
bundle exec appraisal install
|
|
483
176
|
bundle exec appraisal rake test
|
|
177
|
+
gem build solid_queue_guard.gemspec
|
|
484
178
|
```
|
|
485
179
|
|
|
486
|
-
Release a new version after CI passes:
|
|
487
|
-
|
|
488
|
-
```bash
|
|
489
|
-
# Tag must match lib/solid_queue_guard/version.rb (currently 1.1.3)
|
|
490
|
-
git tag v1.1.3
|
|
491
|
-
git push origin v1.1.3
|
|
492
|
-
```
|
|
493
|
-
|
|
494
|
-
Trusted Publishing on RubyGems publishes automatically when the tag is pushed.
|
|
495
|
-
|
|
496
|
-
---
|
|
497
|
-
|
|
498
|
-
## Contributing
|
|
499
|
-
|
|
500
|
-
Issues and PRs welcome. Keep it focused: **production safety**, not another job dashboard.
|
|
501
|
-
|
|
502
|
-
1. Fork it
|
|
503
|
-
2. Create your branch (`git checkout -b my-new-check`)
|
|
504
|
-
3. Write a test first
|
|
505
|
-
4. Make the check pass
|
|
506
|
-
5. Open a PR
|
|
507
|
-
|
|
508
180
|
---
|
|
509
181
|
|
|
510
182
|
## License
|
data/bin/console
ADDED
|
Binary file
|
|
@@ -23,6 +23,7 @@ SolidQueueGuard.configure do |config|
|
|
|
23
23
|
# config.health_token = ENV["SOLID_QUEUE_GUARD_TOKEN"]
|
|
24
24
|
# config.integrate_rails_health = true
|
|
25
25
|
# config.integrate_mission_control = true
|
|
26
|
+
# config.on_status_change = ->(previous, current, report) { ... }
|
|
26
27
|
# config.notify_with = [:rails_logger, :slack, :datadog, :webhook]
|
|
27
28
|
# config.metrics_backends = [:statsd, :prometheus, :opentelemetry]
|
|
28
29
|
end
|
|
@@ -40,6 +40,13 @@ module SolidQueueGuard
|
|
|
40
40
|
check_class.check_id
|
|
41
41
|
end
|
|
42
42
|
|
|
43
|
+
def catalog
|
|
44
|
+
{
|
|
45
|
+
config: CONFIG_CHECKS.map(&:check_id),
|
|
46
|
+
runtime: RUNTIME_CHECKS.map(&:check_id)
|
|
47
|
+
}
|
|
48
|
+
end
|
|
49
|
+
|
|
43
50
|
def for(scope)
|
|
44
51
|
case scope.to_sym
|
|
45
52
|
when :config then CONFIG_CHECKS
|
|
@@ -2,6 +2,10 @@
|
|
|
2
2
|
|
|
3
3
|
module SolidQueueGuard
|
|
4
4
|
class Configuration
|
|
5
|
+
class ValidationError < StandardError; end
|
|
6
|
+
|
|
7
|
+
HTTP_STATUS_OPTIONS = HttpStatusPolicy::STATUS_MAP.keys.freeze
|
|
8
|
+
|
|
5
9
|
attr_accessor :enabled,
|
|
6
10
|
:queue_lag_thresholds,
|
|
7
11
|
:failed_jobs_threshold,
|
|
@@ -17,7 +21,8 @@ module SolidQueueGuard
|
|
|
17
21
|
:disabled_checks,
|
|
18
22
|
:checks,
|
|
19
23
|
:degraded_http_status,
|
|
20
|
-
:unhealthy_http_status
|
|
24
|
+
:unhealthy_http_status,
|
|
25
|
+
:on_status_change
|
|
21
26
|
|
|
22
27
|
def initialize
|
|
23
28
|
@enabled = true
|
|
@@ -36,6 +41,18 @@ module SolidQueueGuard
|
|
|
36
41
|
@checks = ActiveSupport::OrderedOptions.new
|
|
37
42
|
@degraded_http_status = :ok
|
|
38
43
|
@unhealthy_http_status = :service_unavailable
|
|
44
|
+
@on_status_change = nil
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
def validate!
|
|
48
|
+
validate_http_status!(:degraded_http_status, degraded_http_status)
|
|
49
|
+
validate_http_status!(:unhealthy_http_status, unhealthy_http_status)
|
|
50
|
+
validate_duration!(:stale_process_threshold, stale_process_threshold)
|
|
51
|
+
validate_duration!(:health_cache_ttl, health_cache_ttl)
|
|
52
|
+
validate_positive_integer!(:failed_jobs_threshold, failed_jobs_threshold)
|
|
53
|
+
validate_positive_integer!(:scheduled_backlog_threshold, scheduled_backlog_threshold)
|
|
54
|
+
validate_callback!(:on_status_change, on_status_change)
|
|
55
|
+
self
|
|
39
56
|
end
|
|
40
57
|
|
|
41
58
|
def strict?
|
|
@@ -66,5 +83,36 @@ module SolidQueueGuard
|
|
|
66
83
|
value = settings[key.to_sym] if value.nil?
|
|
67
84
|
value.nil? ? default : value
|
|
68
85
|
end
|
|
86
|
+
|
|
87
|
+
private
|
|
88
|
+
|
|
89
|
+
def validate_http_status!(attribute, value)
|
|
90
|
+
return if HTTP_STATUS_OPTIONS.include?(normalize_http_status_key(value))
|
|
91
|
+
|
|
92
|
+
raise ValidationError,
|
|
93
|
+
"#{attribute} must be one of #{HTTP_STATUS_OPTIONS.inspect}, got #{value.inspect}"
|
|
94
|
+
end
|
|
95
|
+
|
|
96
|
+
def normalize_http_status_key(value)
|
|
97
|
+
value.is_a?(Integer) ? value : value.to_sym
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
def validate_duration!(attribute, value)
|
|
101
|
+
return if value.is_a?(ActiveSupport::Duration) && value.positive?
|
|
102
|
+
|
|
103
|
+
raise ValidationError, "#{attribute} must be a positive ActiveSupport::Duration"
|
|
104
|
+
end
|
|
105
|
+
|
|
106
|
+
def validate_positive_integer!(attribute, value)
|
|
107
|
+
return if value.is_a?(Integer) && value.positive?
|
|
108
|
+
|
|
109
|
+
raise ValidationError, "#{attribute} must be a positive Integer"
|
|
110
|
+
end
|
|
111
|
+
|
|
112
|
+
def validate_callback!(attribute, value)
|
|
113
|
+
return if value.nil? || value.respond_to?(:call)
|
|
114
|
+
|
|
115
|
+
raise ValidationError, "#{attribute} must respond to #call"
|
|
116
|
+
end
|
|
69
117
|
end
|
|
70
118
|
end
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
module SolidQueueGuard
|
|
4
4
|
module Health
|
|
5
|
+
# @api private
|
|
5
6
|
class Cache
|
|
6
7
|
def self.fetch
|
|
7
8
|
new.fetch
|
|
@@ -9,7 +10,7 @@ module SolidQueueGuard
|
|
|
9
10
|
|
|
10
11
|
def fetch
|
|
11
12
|
Rails.cache.fetch(cache_key, expires_in: SolidQueueGuard.config.health_cache_ttl) do
|
|
12
|
-
build_payload
|
|
13
|
+
build_payload(previous_status: read_cached_status)
|
|
13
14
|
end
|
|
14
15
|
end
|
|
15
16
|
|
|
@@ -19,8 +20,13 @@ module SolidQueueGuard
|
|
|
19
20
|
'solid_queue_guard/health'
|
|
20
21
|
end
|
|
21
22
|
|
|
22
|
-
def
|
|
23
|
+
def read_cached_status
|
|
24
|
+
Rails.cache.read(cache_key)&.dig(:status)
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
def build_payload(previous_status: nil)
|
|
23
28
|
report = Runner.new(scope: :all).run
|
|
29
|
+
Observability.notify_status_change(report, previous_status: previous_status&.to_sym)
|
|
24
30
|
Payload.new(report).to_h
|
|
25
31
|
end
|
|
26
32
|
end
|
|
@@ -4,6 +4,7 @@ module SolidQueueGuard
|
|
|
4
4
|
module Metrics
|
|
5
5
|
class Exporter
|
|
6
6
|
STATUS_VALUES = { healthy: 0, degraded: 1, unhealthy: 2 }.freeze
|
|
7
|
+
CHECK_STATUS_VALUES = { pass: 0, warn: 1, fail: 2, skip: 3 }.freeze
|
|
7
8
|
|
|
8
9
|
def self.export(report, backends: SolidQueueGuard.config.metrics_backends)
|
|
9
10
|
new(report, backends: backends).export
|
|
@@ -27,6 +28,8 @@ module SolidQueueGuard
|
|
|
27
28
|
when :statsd then Statsd.export(report)
|
|
28
29
|
when :prometheus then Prometheus.export(report)
|
|
29
30
|
when :opentelemetry then OpenTelemetry.export(report)
|
|
31
|
+
else
|
|
32
|
+
OptionalDependency.log_missing("metrics backend #{backend}")
|
|
30
33
|
end
|
|
31
34
|
end
|
|
32
35
|
end
|
|
@@ -6,14 +6,22 @@ module SolidQueueGuard
|
|
|
6
6
|
module_function
|
|
7
7
|
|
|
8
8
|
def export(report)
|
|
9
|
-
return unless
|
|
9
|
+
return unless OptionalDependency.require!('opentelemetry-sdk', 'opentelemetry-sdk')
|
|
10
10
|
|
|
11
11
|
meter = ::OpenTelemetry.meter_provider.meter('solid_queue_guard')
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
12
|
+
overall_gauge = meter.create_gauge('solid_queue.guard.overall_status', unit: 'status')
|
|
13
|
+
overall_gauge.record(STATUS_VALUES.fetch(report.status))
|
|
14
|
+
|
|
15
|
+
check_gauge = meter.create_gauge('solid_queue.guard.check_status', unit: 'status')
|
|
16
|
+
report.results.each do |result|
|
|
17
|
+
check_gauge.record(CHECK_STATUS_VALUES.fetch(result.status), attributes: { 'check.id' => result.id })
|
|
18
|
+
end
|
|
19
|
+
rescue StandardError => e
|
|
20
|
+
Rails.logger.warn("[solid_queue_guard] OpenTelemetry export failed: #{e.class}: #{e.message}")
|
|
16
21
|
end
|
|
22
|
+
|
|
23
|
+
STATUS_VALUES = Exporter::STATUS_VALUES
|
|
24
|
+
CHECK_STATUS_VALUES = Exporter::CHECK_STATUS_VALUES
|
|
17
25
|
end
|
|
18
26
|
end
|
|
19
27
|
end
|
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require 'fileutils'
|
|
4
|
+
|
|
3
5
|
module SolidQueueGuard
|
|
4
6
|
module Metrics
|
|
5
7
|
module Prometheus
|
|
@@ -7,15 +9,27 @@ module SolidQueueGuard
|
|
|
7
9
|
|
|
8
10
|
def export(report)
|
|
9
11
|
path = ENV.fetch('SOLID_QUEUE_GUARD_PROMETHEUS_FILE', Rails.root.join('tmp/solid_queue_guard.prom'))
|
|
12
|
+
FileUtils.mkdir_p(File.dirname(path))
|
|
10
13
|
File.write(path, render(report))
|
|
11
14
|
end
|
|
12
15
|
|
|
13
16
|
def render(report)
|
|
14
|
-
|
|
15
|
-
# TYPE solid_queue_guard_overall_status gauge
|
|
16
|
-
solid_queue_guard_overall_status #{
|
|
17
|
-
|
|
17
|
+
lines = [
|
|
18
|
+
'# TYPE solid_queue_guard_overall_status gauge',
|
|
19
|
+
"solid_queue_guard_overall_status #{STATUS_VALUES.fetch(report.status)}",
|
|
20
|
+
'# TYPE solid_queue_guard_check_status gauge'
|
|
21
|
+
]
|
|
22
|
+
|
|
23
|
+
report.results.each do |result|
|
|
24
|
+
value = CHECK_STATUS_VALUES.fetch(result.status)
|
|
25
|
+
lines << "solid_queue_guard_check_status{check=\"#{result.id}\"} #{value}"
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
"#{lines.join("\n")}\n"
|
|
18
29
|
end
|
|
30
|
+
|
|
31
|
+
STATUS_VALUES = Exporter::STATUS_VALUES
|
|
32
|
+
CHECK_STATUS_VALUES = Exporter::CHECK_STATUS_VALUES
|
|
19
33
|
end
|
|
20
34
|
end
|
|
21
35
|
end
|
|
@@ -6,20 +6,32 @@ module SolidQueueGuard
|
|
|
6
6
|
module_function
|
|
7
7
|
|
|
8
8
|
def export(report)
|
|
9
|
-
require 'socket'
|
|
10
9
|
host = ENV.fetch('SOLID_QUEUE_GUARD_STATSD_HOST', '127.0.0.1')
|
|
11
10
|
port = ENV.fetch('SOLID_QUEUE_GUARD_STATSD_PORT', 8125).to_i
|
|
12
11
|
socket = UDPSocket.new
|
|
13
|
-
|
|
12
|
+
metric_lines(report).each do |line|
|
|
13
|
+
socket.send(line, 0, host, port)
|
|
14
|
+
end
|
|
14
15
|
ensure
|
|
15
16
|
socket&.close
|
|
16
17
|
end
|
|
17
18
|
|
|
18
|
-
def
|
|
19
|
-
|
|
19
|
+
def metric_lines(report)
|
|
20
|
+
[
|
|
21
|
+
"solid_queue.guard.overall_status:#{STATUS_VALUES.fetch(report.status)}|g",
|
|
22
|
+
*check_status_lines(report)
|
|
23
|
+
]
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def check_status_lines(report)
|
|
27
|
+
report.results.map do |result|
|
|
28
|
+
value = CHECK_STATUS_VALUES.fetch(result.status)
|
|
29
|
+
"solid_queue.guard.check_status:#{value}|g|##{result.id}"
|
|
30
|
+
end
|
|
20
31
|
end
|
|
21
32
|
|
|
22
33
|
STATUS_VALUES = Exporter::STATUS_VALUES
|
|
34
|
+
CHECK_STATUS_VALUES = Exporter::CHECK_STATUS_VALUES
|
|
23
35
|
end
|
|
24
36
|
end
|
|
25
37
|
end
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
module SolidQueueGuard
|
|
4
4
|
module MissionControl
|
|
5
|
+
# @api private
|
|
5
6
|
class Integration
|
|
6
7
|
class << self
|
|
7
8
|
def install!
|
|
@@ -13,10 +14,10 @@ module SolidQueueGuard
|
|
|
13
14
|
return if routes_installed?
|
|
14
15
|
|
|
15
16
|
::MissionControl::Jobs::Engine.routes.append do
|
|
16
|
-
get 'guard',
|
|
17
|
+
get 'applications/:application_id/guard',
|
|
17
18
|
controller: '/solid_queue_guard/mission_control/dashboard',
|
|
18
19
|
action: 'show',
|
|
19
|
-
as: :
|
|
20
|
+
as: :application_solid_queue_guard_dashboard
|
|
20
21
|
end
|
|
21
22
|
end
|
|
22
23
|
|
|
@@ -31,7 +32,7 @@ module SolidQueueGuard
|
|
|
31
32
|
|
|
32
33
|
def routes_installed?
|
|
33
34
|
::MissionControl::Jobs::Engine.routes.routes.any? do |route|
|
|
34
|
-
route.name == '
|
|
35
|
+
route.name == 'application_solid_queue_guard_dashboard'
|
|
35
36
|
end
|
|
36
37
|
end
|
|
37
38
|
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
module SolidQueueGuard
|
|
4
4
|
module MissionControl
|
|
5
|
+
# @api private
|
|
5
6
|
module NavigationExtension
|
|
6
7
|
def navigation_sections
|
|
7
8
|
super.tap do |sections|
|
|
@@ -12,7 +13,7 @@ module SolidQueueGuard
|
|
|
12
13
|
private
|
|
13
14
|
|
|
14
15
|
def guard_dashboard_path
|
|
15
|
-
|
|
16
|
+
application_solid_queue_guard_dashboard_path(@application)
|
|
16
17
|
end
|
|
17
18
|
end
|
|
18
19
|
end
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SolidQueueGuard
|
|
4
|
+
# @api private
|
|
5
|
+
module Observability
|
|
6
|
+
module_function
|
|
7
|
+
|
|
8
|
+
def log_results(report)
|
|
9
|
+
return unless logger
|
|
10
|
+
|
|
11
|
+
report.results.each do |result|
|
|
12
|
+
message = "[SolidQueueGuard] #{result.status} check #{result.id}: #{result.message}"
|
|
13
|
+
case result.status
|
|
14
|
+
when :skip, :pass then logger.debug(message)
|
|
15
|
+
when :warn then logger.warn(message)
|
|
16
|
+
when :fail then logger.error(message)
|
|
17
|
+
end
|
|
18
|
+
end
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
def notify_status_change(report, previous_status: nil)
|
|
22
|
+
callback = SolidQueueGuard.config.on_status_change
|
|
23
|
+
return unless callback.respond_to?(:call)
|
|
24
|
+
|
|
25
|
+
current_status = report.status
|
|
26
|
+
return if previous_status == current_status
|
|
27
|
+
|
|
28
|
+
callback.call(previous_status, current_status, report)
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
def logger
|
|
32
|
+
defined?(Rails) && Rails.logger
|
|
33
|
+
end
|
|
34
|
+
end
|
|
35
|
+
end
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SolidQueueGuard
|
|
4
|
+
# @api private
|
|
5
|
+
module OptionalDependency
|
|
6
|
+
module_function
|
|
7
|
+
|
|
8
|
+
def require!(feature, gem_name = feature)
|
|
9
|
+
require feature
|
|
10
|
+
true
|
|
11
|
+
rescue LoadError
|
|
12
|
+
log_missing(gem_name)
|
|
13
|
+
false
|
|
14
|
+
end
|
|
15
|
+
|
|
16
|
+
def log_missing(gem_name)
|
|
17
|
+
return unless defined?(Rails) && Rails.logger
|
|
18
|
+
|
|
19
|
+
Rails.logger.warn(
|
|
20
|
+
"[solid_queue_guard] Optional dependency #{gem_name} is not installed. " \
|
|
21
|
+
'Add it to your Gemfile to enable this integration.'
|
|
22
|
+
)
|
|
23
|
+
end
|
|
24
|
+
end
|
|
25
|
+
end
|
|
@@ -37,8 +37,13 @@ module SolidQueueGuard
|
|
|
37
37
|
status: status.to_s,
|
|
38
38
|
warnings: warnings,
|
|
39
39
|
suggestions: suggestions,
|
|
40
|
+
status_counts: status_counts,
|
|
40
41
|
checks: results.map(&:to_h)
|
|
41
42
|
}
|
|
42
43
|
end
|
|
44
|
+
|
|
45
|
+
def status_counts
|
|
46
|
+
results.group_by(&:status).transform_values(&:size)
|
|
47
|
+
end
|
|
43
48
|
end
|
|
44
49
|
end
|
data/solid_queue_guard.gemspec
CHANGED
|
@@ -26,7 +26,8 @@ Gem::Specification.new do |spec|
|
|
|
26
26
|
|
|
27
27
|
spec.files = Dir.chdir(File.expand_path(__dir__)) do
|
|
28
28
|
`git ls-files -z`.split("\x0").reject do |file|
|
|
29
|
-
file.start_with?('test/', '.github/', '
|
|
29
|
+
file.start_with?('test/', '.github/', 'script/', 'gemfiles/', 'Appraisals') ||
|
|
30
|
+
(file.start_with?('docs/') && file != 'docs/mission-control-guard-dashboard.png')
|
|
30
31
|
end
|
|
31
32
|
end
|
|
32
33
|
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: solid_queue_guard
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Rafael Pissardo
|
|
@@ -301,11 +301,13 @@ files:
|
|
|
301
301
|
- ".rubocop.yml"
|
|
302
302
|
- ".ruby-version"
|
|
303
303
|
- CHANGELOG.md
|
|
304
|
+
- CONTRIBUTING.md
|
|
304
305
|
- Gemfile
|
|
305
306
|
- Gemfile.lock
|
|
306
307
|
- MIT-LICENSE
|
|
307
308
|
- README.md
|
|
308
309
|
- Rakefile
|
|
310
|
+
- app/assets/images/solid_queue_guard/mission-control-guard-dashboard.png
|
|
309
311
|
- app/controllers/solid_queue_guard/application_controller.rb
|
|
310
312
|
- app/controllers/solid_queue_guard/health_controller.rb
|
|
311
313
|
- app/controllers/solid_queue_guard/mission_control/base_controller.rb
|
|
@@ -314,7 +316,9 @@ files:
|
|
|
314
316
|
- app/views/solid_queue_guard/mission_control/dashboard/_check.html.erb
|
|
315
317
|
- app/views/solid_queue_guard/mission_control/dashboard/_summary_metrics.html.erb
|
|
316
318
|
- app/views/solid_queue_guard/mission_control/dashboard/show.html.erb
|
|
319
|
+
- bin/console
|
|
317
320
|
- config/routes.rb
|
|
321
|
+
- docs/mission-control-guard-dashboard.png
|
|
318
322
|
- lib/generators/solid_queue_guard/install/USAGE
|
|
319
323
|
- lib/generators/solid_queue_guard/install/USAGE.ci
|
|
320
324
|
- lib/generators/solid_queue_guard/install/ci_generator.rb
|
|
@@ -368,6 +372,8 @@ files:
|
|
|
368
372
|
- lib/solid_queue_guard/mission_control/navigation_extension.rb
|
|
369
373
|
- lib/solid_queue_guard/notifier.rb
|
|
370
374
|
- lib/solid_queue_guard/notifiers/base.rb
|
|
375
|
+
- lib/solid_queue_guard/observability.rb
|
|
376
|
+
- lib/solid_queue_guard/optional_dependency.rb
|
|
371
377
|
- lib/solid_queue_guard/puma_plugin_support.rb
|
|
372
378
|
- lib/solid_queue_guard/queue_coverage.rb
|
|
373
379
|
- lib/solid_queue_guard/recommendations/topology.rb
|