solid_queue_guard 1.1.4 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +21 -0
- data/CONTRIBUTING.md +57 -0
- data/Gemfile.lock +1 -1
- data/README.md +35 -411
- data/app/assets/images/solid_queue_guard/mission-control-guard-dashboard.png +0 -0
- data/bin/console +8 -0
- data/docs/mission-control-guard-dashboard.png +0 -0
- data/lib/generators/solid_queue_guard/install/templates/solid_queue_guard.rb +1 -0
- data/lib/solid_queue_guard/checks/registry.rb +7 -0
- data/lib/solid_queue_guard/configuration.rb +49 -1
- data/lib/solid_queue_guard/engine.rb +1 -0
- data/lib/solid_queue_guard/health/cache.rb +8 -2
- data/lib/solid_queue_guard/health/payload.rb +1 -0
- data/lib/solid_queue_guard/metrics/exporter.rb +3 -0
- data/lib/solid_queue_guard/metrics/open_telemetry.rb +13 -5
- data/lib/solid_queue_guard/metrics/prometheus.rb +18 -4
- data/lib/solid_queue_guard/metrics/statsd.rb +16 -4
- data/lib/solid_queue_guard/mission_control/integration.rb +1 -0
- data/lib/solid_queue_guard/mission_control/navigation_extension.rb +1 -0
- data/lib/solid_queue_guard/notifier.rb +1 -0
- data/lib/solid_queue_guard/observability.rb +35 -0
- data/lib/solid_queue_guard/optional_dependency.rb +25 -0
- data/lib/solid_queue_guard/report.rb +5 -0
- data/lib/solid_queue_guard/runner.rb +1 -1
- data/lib/solid_queue_guard/version.rb +1 -1
- data/solid_queue_guard.gemspec +2 -1
- metadata +7 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 4ab3926371f9714622f421164d1436dcc74b17eb0d311627fff319c74a9369be
|
|
4
|
+
data.tar.gz: 13bd4acab34e3a6757bd5a4c445e471ab3021fc88d0748ca95a89a92cc597804
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 02d75b6056ddd2f698441b892c16d29f4155a0d213d598c5b0f813d0094529b32a761461071ca302cf170c2d1acfbd0e39339e2818b260261c006ba11ad73581
|
|
7
|
+
data.tar.gz: 9e57de7cf88e4ab6ef5d5ded2b98cd15c09374f0a46ecc311fadbe02d2952f4097fef99025848454b0734af35ed734b61f7f54093df255ad27c9c8ff5cf15f1a
|
data/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,27 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [1.2.0] - 2026-07-10
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
|
|
12
|
+
- `config.validate!` at boot — rejects invalid HTTP status codes, thresholds, and `on_status_change` types
|
|
13
|
+
- `config.on_status_change` callback when `/solid_queue_guard/health` detects a status transition
|
|
14
|
+
- Structured logging per check result (debug for pass/skip, warn/error for warn/fail)
|
|
15
|
+
- `status_counts` in report and health JSON payload
|
|
16
|
+
- Per-check metrics for StatsD, Prometheus, and OpenTelemetry exporters
|
|
17
|
+
- `SolidQueueGuard::Checks::Registry.catalog` for listing check IDs
|
|
18
|
+
- `bin/console` for dummy app debugging
|
|
19
|
+
- `CONTRIBUTING.md`, `docs/configuration.md`, and `docs/checks.md`
|
|
20
|
+
- CI: Ruby 3.1 matrix, dedicated Appraisal job, and `gem build` verification
|
|
21
|
+
- Mission Control navigation integration test from recurring tasks page
|
|
22
|
+
|
|
23
|
+
### Changed
|
|
24
|
+
|
|
25
|
+
- README trimmed with links to dedicated docs
|
|
26
|
+
- Mission Control screenshot shipped in gem package for RubyGems README rendering
|
|
27
|
+
- OpenTelemetry exporter requires `opentelemetry-sdk` with clear log when missing
|
|
28
|
+
|
|
8
29
|
## [1.1.4] - 2026-07-10
|
|
9
30
|
|
|
10
31
|
### Fixed
|
data/CONTRIBUTING.md
ADDED
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
# Contributing
|
|
2
|
+
|
|
3
|
+
Issues and PRs welcome. Keep changes focused on **production safety**, not another job dashboard.
|
|
4
|
+
|
|
5
|
+
## Setup
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
git clone https://github.com/rafael-pissardo/solid_queue_guard.git
|
|
9
|
+
cd solid_queue_guard
|
|
10
|
+
bundle install
|
|
11
|
+
cd test/dummy && bin/setup
|
|
12
|
+
cd ../..
|
|
13
|
+
bundle exec rake test
|
|
14
|
+
bundle exec rubocop
|
|
15
|
+
bundle exec appraisal install
|
|
16
|
+
bundle exec appraisal rake test
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Interactive console against the dummy app:
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
bin/console
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
## Adding a check
|
|
26
|
+
|
|
27
|
+
1. Create `lib/solid_queue_guard/checks/config/my_check.rb` or `checks/runtime/my_check.rb`
|
|
28
|
+
2. Register it in `lib/solid_queue_guard/checks/registry.rb`
|
|
29
|
+
3. Add tests under `test/solid_queue_guard/checks/`
|
|
30
|
+
4. Document the check ID in [docs/checks.md](docs/checks.md)
|
|
31
|
+
|
|
32
|
+
Check IDs are derived from the class name (`MyFeatureCheck` → `my_feature`).
|
|
33
|
+
|
|
34
|
+
## Testing against Revelo-style apps
|
|
35
|
+
|
|
36
|
+
Use `script/validate_revelo.sh` to run the doctor and health tasks against local Revelo backends with a path gem:
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
./script/validate_revelo.sh
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Edit the script to point at your checkout paths. The script temporarily adds a `path:` gem line, runs `solid_queue_guard:doctor` and `solid_queue_guard:health`, then restores the Gemfile.
|
|
43
|
+
|
|
44
|
+
## Release checklist
|
|
45
|
+
|
|
46
|
+
- [ ] Tests pass locally and in CI (matrix + Appraisal + `gem build`)
|
|
47
|
+
- [ ] RuboCop clean
|
|
48
|
+
- [ ] CHANGELOG updated
|
|
49
|
+
- [ ] Version bumped in `lib/solid_queue_guard/version.rb` and lockfiles synced
|
|
50
|
+
- [ ] README / docs updated for public API changes
|
|
51
|
+
- [ ] Tag matches gem version: `git tag vX.Y.Z && git push origin vX.Y.Z`
|
|
52
|
+
|
|
53
|
+
## Code style
|
|
54
|
+
|
|
55
|
+
- Follow existing patterns: small checks, `Check::Result`, registry-driven runner
|
|
56
|
+
- Mark internal APIs with `# @api private`
|
|
57
|
+
- Public API changes require semver discipline (see README)
|
data/Gemfile.lock
CHANGED
data/README.md
CHANGED
|
@@ -15,18 +15,16 @@
|
|
|
15
15
|
|
|
16
16
|
<p align="center">
|
|
17
17
|
<a href="#-quick-start"><strong>Quick start</strong></a> ·
|
|
18
|
-
<a href="
|
|
19
|
-
<a href="
|
|
20
|
-
<a href="#-
|
|
18
|
+
<a href="docs/checks.md"><strong>Checks</strong></a> ·
|
|
19
|
+
<a href="docs/configuration.md"><strong>Configuration</strong></a> ·
|
|
20
|
+
<a href="#-mission-control-dashboard"><strong>Mission Control</strong></a>
|
|
21
21
|
</p>
|
|
22
22
|
|
|
23
23
|
---
|
|
24
24
|
|
|
25
25
|
Rails 8 ships with [Solid Queue](https://github.com/rails/solid_queue). Redis is optional. **Production config is not.**
|
|
26
26
|
|
|
27
|
-
Your web tier can be **green** while your jobs are **dead**.
|
|
28
|
-
|
|
29
|
-
**solid_queue_guard** catches that *before* it becomes an incident.
|
|
27
|
+
Your web tier can be **green** while your jobs are **dead**. **solid_queue_guard** catches that *before* it becomes an incident.
|
|
30
28
|
|
|
31
29
|
> **Mission Control** shows what is happening.
|
|
32
30
|
> **solid_queue_guard** warns what is dangerous.
|
|
@@ -41,49 +39,20 @@ bin/rails solid_queue_guard:doctor
|
|
|
41
39
|
|
|
42
40
|
---
|
|
43
41
|
|
|
44
|
-
## Table of contents
|
|
45
|
-
|
|
46
|
-
- [Why this exists](#why-this-exists)
|
|
47
|
-
- [See it in action](#-see-it-in-action)
|
|
48
|
-
- [What it checks](#what-it-checks)
|
|
49
|
-
- [Quick start](#-quick-start)
|
|
50
|
-
- [Mission Control dashboard](#-mission-control-dashboard-opt-in)
|
|
51
|
-
- [How to use it](#how-to-use-it)
|
|
52
|
-
- [Runtime process checks](#runtime-process-checks)
|
|
53
|
-
- [Configuration](#configuration)
|
|
54
|
-
- [Public API (v1.0+)](#public-api-v10)
|
|
55
|
-
- [solid_queue_guard vs Mission Control](#-solid_queue_guard-vs-mission-control)
|
|
56
|
-
- [Roadmap](#roadmap)
|
|
57
|
-
- [Compatibility](#compatibility)
|
|
58
|
-
- [Development](#development)
|
|
59
|
-
- [Contributing](#contributing)
|
|
60
|
-
|
|
61
|
-
---
|
|
62
|
-
|
|
63
42
|
## Why this exists
|
|
64
43
|
|
|
65
|
-
Solid Queue is excellent. Operating it blindly is not.
|
|
66
|
-
|
|
67
44
|
| Symptom | What actually happened |
|
|
68
45
|
| ------- | ---------------------- |
|
|
69
46
|
| "Site is up, emails stopped" | Workers dead, heartbeats stale |
|
|
70
47
|
| "Only 3 jobs in the queue, why the panic?" | Oldest job waiting 40 minutes — **lag**, not depth |
|
|
71
|
-
| "Recurring billing just… stopped" | Scheduler not running
|
|
72
|
-
| "Jobs hang after deploy" | Thread count > DB pool
|
|
48
|
+
| "Recurring billing just… stopped" | Scheduler not running |
|
|
49
|
+
| "Jobs hang after deploy" | Thread count > DB pool |
|
|
73
50
|
| "Health check passes, jobs don't" | `/up` doesn't know Solid Queue exists |
|
|
74
51
|
|
|
75
|
-
You don't need another dashboard. You need a **doctor**.
|
|
76
|
-
|
|
77
52
|
---
|
|
78
53
|
|
|
79
54
|
## 👨⚕️ See it in action
|
|
80
55
|
|
|
81
|
-
```bash
|
|
82
|
-
bundle add solid_queue_guard
|
|
83
|
-
bin/rails solid_queue_guard:install
|
|
84
|
-
bin/rails solid_queue_guard:doctor
|
|
85
|
-
```
|
|
86
|
-
|
|
87
56
|
```text
|
|
88
57
|
SolidQueueGuard Report
|
|
89
58
|
|
|
@@ -91,109 +60,29 @@ Status: DEGRADED
|
|
|
91
60
|
|
|
92
61
|
Checks:
|
|
93
62
|
✅ Active Job adapter is :solid_queue
|
|
94
|
-
✅ Queue database configured in database.yml
|
|
95
|
-
✅ Solid Queue connects_to queue database (pool: 10)
|
|
96
|
-
✅ Solid Queue schema tables present
|
|
97
63
|
❌ Worker threads: 10, queue DB pool: 5
|
|
98
64
|
⚠️ No workers configured for "mailers" queue
|
|
99
|
-
⚠️ recurring.yml exists but scheduler may not run
|
|
100
|
-
✅ Solid Queue is not co-located with Puma
|
|
101
65
|
|
|
102
66
|
Suggested fixes:
|
|
103
67
|
- Increase queue DB pool to at least 12 or reduce worker threads
|
|
104
|
-
- Add a worker for the mailers queue
|
|
105
|
-
- Verify scheduler process is running in production
|
|
106
68
|
```
|
|
107
69
|
|
|
108
70
|
**One command. Actionable output. No Datadog required to get started.**
|
|
109
71
|
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
## What it checks
|
|
113
|
-
|
|
114
|
-
### Configuration doctor
|
|
115
|
-
|
|
116
|
-
Runs locally, in CI, or pre-deploy. **No extra infrastructure.**
|
|
117
|
-
|
|
118
|
-
| Check | Catches |
|
|
119
|
-
| ----- | ------- |
|
|
120
|
-
| **Adapter** | Active Job not set to `:solid_queue` |
|
|
121
|
-
| **Queue database** | Missing `queue` entry in `database.yml` |
|
|
122
|
-
| **connects_to** | Solid Queue pointing at the wrong DB / pool |
|
|
123
|
-
| **Queue schema** | Missing Solid Queue tables in schema files or queue database |
|
|
124
|
-
| **Thread pool** | `threads` > available queue DB connections |
|
|
125
|
-
| **Worker coverage** | Queues with no worker assigned |
|
|
126
|
-
| **Scheduler** | `recurring.yml` tasks without a scheduler |
|
|
127
|
-
| **Env flags** | `SOLID_QUEUE_SKIP_RECURRING=true` in production |
|
|
128
|
-
| **Heartbeats** | Default thresholds that may not fit your deploy |
|
|
129
|
-
| **Puma plugin** | Solid Queue co-located with web in production |
|
|
130
|
-
| **Async supervisor** | `async` mode with thread/pool sizing risks |
|
|
131
|
-
| **Topology** | `queue.yml` worker and pool recommendations |
|
|
132
|
-
|
|
133
|
-
### Runtime guards
|
|
134
|
-
|
|
135
|
-
| Check | Catches |
|
|
136
|
-
| ----- | ------- |
|
|
137
|
-
| **Queue lag** | Oldest ready job waiting too long |
|
|
138
|
-
| **Stale processes** | Dead workers, dispatchers, schedulers |
|
|
139
|
-
| **Dispatcher health** | Scheduled jobs never becoming ready |
|
|
140
|
-
| **Blocked jobs** | Concurrency control silently holding jobs |
|
|
141
|
-
| **Recurring staleness** | Cron tasks that stopped firing |
|
|
142
|
-
| **Puma plugin runtime** | Plugin enabled but no active Solid Queue processes |
|
|
143
|
-
| **HTTP `/health`** | Kamal, ECS, K8s, UptimeRobot integration |
|
|
72
|
+
Full check list: [docs/checks.md](docs/checks.md)
|
|
144
73
|
|
|
145
74
|
---
|
|
146
75
|
|
|
147
76
|
## 🚀 Quick start
|
|
148
77
|
|
|
149
|
-
### Install
|
|
150
|
-
|
|
151
78
|
```bash
|
|
152
79
|
bundle add solid_queue_guard
|
|
153
80
|
bin/rails solid_queue_guard:install
|
|
154
|
-
bin/rails generate solid_queue_guard:install:ci # optional GitHub Actions workflow
|
|
155
|
-
```
|
|
156
|
-
|
|
157
|
-
### Run the doctor
|
|
158
|
-
|
|
159
|
-
```bash
|
|
160
|
-
# Human-readable report
|
|
161
81
|
bin/rails solid_queue_guard:doctor
|
|
162
|
-
|
|
163
|
-
#
|
|
164
|
-
bin/rails "solid_queue_guard:doctor[json]"
|
|
165
|
-
SOLID_QUEUE_GUARD_FORMAT=json bin/rails solid_queue_guard:doctor
|
|
166
|
-
|
|
167
|
-
# Fail CI on warnings too
|
|
168
|
-
SOLID_QUEUE_GUARD_STRICT=1 bin/rails solid_queue_guard:doctor
|
|
169
|
-
|
|
170
|
-
# Full report (config + runtime when available)
|
|
171
|
-
bin/rails solid_queue_guard:report
|
|
172
|
-
```
|
|
173
|
-
|
|
174
|
-
### Exit codes
|
|
175
|
-
|
|
176
|
-
| Code | Meaning |
|
|
177
|
-
| ---- | ------- |
|
|
178
|
-
| `0` | Healthy or degraded |
|
|
179
|
-
| `1` | Unhealthy (or degraded in `--strict` / `SOLID_QUEUE_GUARD_STRICT=1`) |
|
|
180
|
-
|
|
181
|
-
Perfect for deploy pipelines:
|
|
182
|
-
|
|
183
|
-
```yaml
|
|
184
|
-
# GitHub Actions
|
|
185
|
-
- name: Solid Queue production readiness
|
|
186
|
-
run: SOLID_QUEUE_GUARD_STRICT=1 bin/rails solid_queue_guard:doctor
|
|
82
|
+
SOLID_QUEUE_GUARD_STRICT=1 bin/rails solid_queue_guard:doctor # CI gate
|
|
83
|
+
bin/rails solid_queue_guard:report # config + runtime
|
|
187
84
|
```
|
|
188
85
|
|
|
189
|
-
Or generate a workflow:
|
|
190
|
-
|
|
191
|
-
```bash
|
|
192
|
-
bin/rails generate solid_queue_guard:install:ci
|
|
193
|
-
```
|
|
194
|
-
|
|
195
|
-
### HTTP health
|
|
196
|
-
|
|
197
86
|
```ruby
|
|
198
87
|
# config/routes.rb
|
|
199
88
|
mount SolidQueueGuard::Engine, at: "/solid_queue_guard"
|
|
@@ -201,302 +90,66 @@ mount SolidQueueGuard::Engine, at: "/solid_queue_guard"
|
|
|
201
90
|
|
|
202
91
|
```bash
|
|
203
92
|
curl localhost:3000/solid_queue_guard/health
|
|
204
|
-
# => { "status": "degraded", "queue_lag_seconds": 245, "warnings": [...] }
|
|
205
|
-
```
|
|
206
|
-
|
|
207
|
-
Optional token protection:
|
|
208
|
-
|
|
209
|
-
```ruby
|
|
210
|
-
config.health_token = ENV["SOLID_QUEUE_GUARD_TOKEN"]
|
|
211
|
-
# curl -H "X-Solid-Queue-Guard-Token: $TOKEN" ...
|
|
212
|
-
```
|
|
213
|
-
|
|
214
|
-
HTTP status policy (for Kamal, ECS, load balancers):
|
|
215
|
-
|
|
216
|
-
```ruby
|
|
217
|
-
config.degraded_http_status = 207 # or :ok (200), 503, etc.
|
|
218
|
-
config.unhealthy_http_status = 503 # default
|
|
219
|
-
```
|
|
220
|
-
|
|
221
|
-
Works with **Kamal**, **Heroku**, **Fly.io**, **ECS/Fargate**, **Kubernetes**, **Better Stack**, **UptimeRobot**.
|
|
222
|
-
|
|
223
|
-
---
|
|
224
|
-
|
|
225
|
-
## 🛡️ Mission Control dashboard (opt-in)
|
|
226
|
-
|
|
227
|
-
Add a **Guard** tab to [Mission Control — Jobs](https://github.com/rails/mission_control-jobs) — same Bulma UI, same auth, same navigation:
|
|
228
|
-
|
|
229
|
-
```ruby
|
|
230
|
-
# Gemfile
|
|
231
|
-
gem "mission_control-jobs"
|
|
232
|
-
gem "solid_queue_guard"
|
|
233
|
-
|
|
234
|
-
# config/initializers/solid_queue_guard.rb
|
|
235
|
-
SolidQueueGuard.configure do |config|
|
|
236
|
-
config.integrate_mission_control = true
|
|
237
|
-
end
|
|
238
|
-
|
|
239
|
-
# config/routes.rb
|
|
240
|
-
mount MissionControl::Jobs::Engine, at: "/jobs"
|
|
241
|
-
mount SolidQueueGuard::Engine, at: "/solid_queue_guard" # keep /health for probes
|
|
242
93
|
```
|
|
243
94
|
|
|
244
|
-
|
|
95
|
+
All configuration options: [docs/configuration.md](docs/configuration.md)
|
|
245
96
|
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
| Section | Content |
|
|
249
|
-
| ------- | ------- |
|
|
250
|
-
| **Overall status** | `healthy` / `degraded` / `unhealthy` badge |
|
|
251
|
-
| **Metrics** | Queue lag, failed jobs (1h), dead processes, check count |
|
|
252
|
-
| **Warnings & suggestions** | Same messages as `doctor` / `/health` |
|
|
253
|
-
| **Checks table** | Every check with status tag, message, and suggestion |
|
|
254
|
-
|
|
255
|
-
**Authentication:** the Guard tab inherits Mission Control auth — HTTP basic (default) or your admin controller via `MissionControl::Jobs.base_controller_class`. No separate login.
|
|
256
|
-
|
|
257
|
-
**Load balancers:** keep using `GET /solid_queue_guard/health` with optional `health_token`. Probes do not go through Mission Control.
|
|
258
|
-
|
|
259
|
-
**Requirements:** an asset pipeline (Propshaft or Sprockets), same as Mission Control. `mission_control-jobs` is optional — only needed when `integrate_mission_control` is enabled.
|
|
260
|
-
|
|
261
|
-
---
|
|
262
|
-
|
|
263
|
-
## How to use it
|
|
264
|
-
|
|
265
|
-
Solid Queue does not show up in Rails `/up`. **solid_queue_guard** gives you four operational surfaces:
|
|
97
|
+
### Operational surfaces
|
|
266
98
|
|
|
267
99
|
| Surface | Command / URL | Best for |
|
|
268
100
|
| ------- | ------------- | -------- |
|
|
269
101
|
| **Doctor** | `bin/rails solid_queue_guard:doctor` | Local pre-deploy, config review |
|
|
270
|
-
| **CI gate** | `SOLID_QUEUE_GUARD_STRICT=1 bin/rails solid_queue_guard:doctor` | Block merges
|
|
271
|
-
| **HTTP health** | `GET /solid_queue_guard/health` |
|
|
272
|
-
| **Guard tab** | `GET
|
|
273
|
-
|
|
274
|
-
**Mission Control** shows what is happening. **solid_queue_guard** warns what is dangerous. Use both.
|
|
102
|
+
| **CI gate** | `SOLID_QUEUE_GUARD_STRICT=1 bin/rails solid_queue_guard:doctor` | Block bad merges |
|
|
103
|
+
| **HTTP health** | `GET /solid_queue_guard/health` | Kamal, ECS, K8s, UptimeRobot |
|
|
104
|
+
| **Guard tab** | `GET …/applications/:application_id/guard` | Human checks in Mission Control |
|
|
275
105
|
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
Run before changing `queue.yml`, `database.yml`, or recurring tasks:
|
|
106
|
+
---
|
|
279
107
|
|
|
280
|
-
|
|
281
|
-
bin/rails solid_queue_guard:doctor # config checks (default scope)
|
|
282
|
-
bin/rails solid_queue_guard:report # config + runtime when the queue DB is available
|
|
283
|
-
```
|
|
108
|
+
## 🛡️ Mission Control dashboard
|
|
284
109
|
|
|
285
|
-
|
|
110
|
+
Opt-in **Guard** tab for [Mission Control — Jobs](https://github.com/rails/mission_control-jobs):
|
|
286
111
|
|
|
287
112
|
```ruby
|
|
288
|
-
|
|
289
|
-
```
|
|
290
|
-
|
|
291
|
-
Or run with production config locally when validating deploy readiness.
|
|
292
|
-
|
|
293
|
-
### CI pipelines
|
|
294
|
-
|
|
295
|
-
Validate **configuration** before deploy. Process/runtime checks usually **skip** in CI because there is no `bin/jobs` supervisor on the runner.
|
|
296
|
-
|
|
297
|
-
```yaml
|
|
298
|
-
- name: Solid Queue production readiness
|
|
299
|
-
run: SOLID_QUEUE_GUARD_STRICT=1 bin/rails solid_queue_guard:doctor
|
|
300
|
-
```
|
|
301
|
-
|
|
302
|
-
`STRICT=1` turns warnings into exit code `1`, so the pipeline fails on misconfigured pools, missing worker coverage, or missing schema tables — not only hard failures.
|
|
303
|
-
|
|
304
|
-
Generate a starter workflow:
|
|
305
|
-
|
|
306
|
-
```bash
|
|
307
|
-
bin/rails generate solid_queue_guard:install:ci
|
|
308
|
-
```
|
|
309
|
-
|
|
310
|
-
### Production monitoring
|
|
113
|
+
gem "mission_control-jobs"
|
|
311
114
|
|
|
312
|
-
|
|
115
|
+
SolidQueueGuard.configure { |config| config.integrate_mission_control = true }
|
|
313
116
|
|
|
314
|
-
|
|
315
|
-
# config/routes.rb
|
|
117
|
+
mount MissionControl::Jobs::Engine, at: "/jobs"
|
|
316
118
|
mount SolidQueueGuard::Engine, at: "/solid_queue_guard"
|
|
317
119
|
```
|
|
318
120
|
|
|
319
|
-
|
|
121
|
+

|
|
320
122
|
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
```ruby
|
|
324
|
-
config.health_token = ENV["SOLID_QUEUE_GUARD_TOKEN"]
|
|
325
|
-
config.health_cache_ttl = 15.seconds
|
|
326
|
-
config.degraded_http_status = 207 # or :ok (200), 503, etc.
|
|
327
|
-
config.notify_with = [:rails_logger, :slack]
|
|
328
|
-
```
|
|
329
|
-
|
|
330
|
-
### Typical flow
|
|
331
|
-
|
|
332
|
-
```text
|
|
333
|
-
Developer CI Production
|
|
334
|
-
│ │ │
|
|
335
|
-
│ doctor │ doctor --strict │ GET /health (every 30–60s)
|
|
336
|
-
▼ ▼ ▼
|
|
337
|
-
"pool wrong?" block bad deploy alert: worker dead
|
|
338
|
-
"schema ok?" before merge alert: queue lag
|
|
339
|
-
```
|
|
340
|
-
|
|
341
|
-
---
|
|
342
|
-
|
|
343
|
-
## Runtime process checks
|
|
344
|
-
|
|
345
|
-
Runtime checks query the **queue database** (via `SolidQueue::Record`). If `connects_to` is missing or the DB is unreachable, they **skip** — they do not pass silently as healthy.
|
|
346
|
-
|
|
347
|
-
### `process_topology` — are the expected roles present?
|
|
348
|
-
|
|
349
|
-
Reads distinct `kind` values from `solid_queue_processes`:
|
|
350
|
-
|
|
351
|
-
| Kind | Role |
|
|
352
|
-
| ---- | ---- |
|
|
353
|
-
| `Supervisor` | Parent process (`bin/jobs`) |
|
|
354
|
-
| `Worker` | Consumes ready jobs |
|
|
355
|
-
| `Dispatcher` | Moves scheduled jobs to ready |
|
|
356
|
-
| `Scheduler` | Runs recurring tasks |
|
|
357
|
-
|
|
358
|
-
| Result | Meaning |
|
|
359
|
-
| ------ | ------- |
|
|
360
|
-
| ⚠️ No processes | Nothing registered — supervisor not running |
|
|
361
|
-
| ⚠️ No Worker | Jobs will not be processed |
|
|
362
|
-
| ⚠️ No Dispatcher | Scheduled work may stall (when recurring/scheduled jobs exist) |
|
|
363
|
-
| ✅ Pass | Expected kinds are present |
|
|
364
|
-
|
|
365
|
-
This check looks at **presence of records**, not heartbeat freshness.
|
|
366
|
-
|
|
367
|
-
### `stale_process` — are heartbeats fresh?
|
|
368
|
-
|
|
369
|
-
Finds processes where `last_heartbeat_at` is older than `stale_process_threshold` (default **5 minutes**):
|
|
370
|
-
|
|
371
|
-
| Result | Meaning |
|
|
372
|
-
| ------ | ------- |
|
|
373
|
-
| ✅ Pass | All processes reported recently |
|
|
374
|
-
| ❌ Fail | One or more workers/dispatchers look dead or stuck |
|
|
375
|
-
|
|
376
|
-
Use this in production health to catch workers that died after deploy.
|
|
377
|
-
|
|
378
|
-
### `pidfile` — optional supervisor liveness
|
|
379
|
-
|
|
380
|
-
When `tmp/pids/solid_queue.pid` (or `SOLID_QUEUE_PIDFILE`) exists, verifies the PID is alive. Often **warns in development** where `bin/jobs` is not running. Disable if you do not use pidfiles:
|
|
381
|
-
|
|
382
|
-
```ruby
|
|
383
|
-
config.disabled_checks = [:pidfile]
|
|
384
|
-
```
|
|
385
|
-
|
|
386
|
-
### `puma_plugin_runtime` — Solid Queue inside Puma
|
|
387
|
-
|
|
388
|
-
When `plugin :solid_queue` is in `config/puma.rb`, verifies active processes with recent heartbeats exist. **Skips** when the Puma plugin is not enabled.
|
|
389
|
-
|
|
390
|
-
### Queue schema detection
|
|
391
|
-
|
|
392
|
-
`QueueSchemaCheck` does **not** require `db/queue_schema.rb`. It validates that all tables for your installed **solid_queue** version exist in any of:
|
|
393
|
-
|
|
394
|
-
- `db/queue_schema.rb`
|
|
395
|
-
- `db/schema.rb`
|
|
396
|
-
- `db/structure.sql`
|
|
397
|
-
- the connected queue database
|
|
398
|
-
|
|
399
|
-
Apps that keep Solid Queue tables only in `structure.sql` (common in Revelo-style repos) pass correctly.
|
|
400
|
-
|
|
401
|
-
### Where each check type applies
|
|
402
|
-
|
|
403
|
-
| Context | Config checks | Process / runtime checks |
|
|
404
|
-
| ------- | ------------- | ------------------------ |
|
|
405
|
-
| Local `doctor` | ✅ Primary value | ⚠️ Partial without `bin/jobs` |
|
|
406
|
-
| CI `--strict` | ✅ Primary value | ⏭️ Usually skip |
|
|
407
|
-
| Production `/health` | ✅ When DB up | ✅ Primary value |
|
|
408
|
-
|
|
409
|
-
---
|
|
410
|
-
|
|
411
|
-
## Configuration
|
|
412
|
-
|
|
413
|
-
```ruby
|
|
414
|
-
# config/initializers/solid_queue_guard.rb
|
|
415
|
-
SolidQueueGuard.configure do |config|
|
|
416
|
-
config.enabled = %w[production staging].include?(Rails.env)
|
|
417
|
-
|
|
418
|
-
config.queue_lag_thresholds = {
|
|
419
|
-
critical: 30.seconds,
|
|
420
|
-
default: 5.minutes,
|
|
421
|
-
mailers: 15.minutes
|
|
422
|
-
}
|
|
423
|
-
|
|
424
|
-
config.failed_jobs_threshold = 20
|
|
425
|
-
config.stale_process_threshold = 5.minutes
|
|
426
|
-
|
|
427
|
-
# Per-check overrides (v0.6+)
|
|
428
|
-
config.disabled_checks = [:pidfile]
|
|
429
|
-
config.checks.queue_lag = { threshold: 10.minutes }
|
|
430
|
-
config.checks.failed_jobs = { threshold: 5, enabled: true }
|
|
431
|
-
|
|
432
|
-
# HTTP status policy (v0.8+)
|
|
433
|
-
# config.degraded_http_status = 207
|
|
434
|
-
# config.unhealthy_http_status = 503
|
|
435
|
-
|
|
436
|
-
# config.health_token = ENV["SOLID_QUEUE_GUARD_TOKEN"]
|
|
437
|
-
# config.integrate_rails_health = true
|
|
438
|
-
# config.integrate_mission_control = true # Guard tab in Mission Control (requires mission_control-jobs)
|
|
439
|
-
# config.notify_with = [:rails_logger, :slack, :datadog, :webhook]
|
|
440
|
-
# config.metrics_backends = [:statsd, :prometheus, :opentelemetry]
|
|
441
|
-
end
|
|
442
|
-
```
|
|
123
|
+
Requires `mission_control-jobs` and an asset pipeline (Propshaft or Sprockets). Load balancers should keep using `/solid_queue_guard/health`.
|
|
443
124
|
|
|
444
125
|
---
|
|
445
126
|
|
|
446
127
|
## Public API (v1.0+)
|
|
447
128
|
|
|
448
|
-
|
|
129
|
+
Stable until `2.0` — [semantic versioning](https://semver.org/):
|
|
449
130
|
|
|
450
131
|
| API | Description |
|
|
451
132
|
| --- | ----------- |
|
|
452
133
|
| `SolidQueueGuard.configure` | Block-style configuration |
|
|
453
|
-
| `SolidQueueGuard.config` | Current configuration object |
|
|
454
|
-
| `SolidQueueGuard.enabled?` | Whether checks run |
|
|
455
134
|
| `solid_queue_guard:doctor` | Config readiness report |
|
|
456
|
-
| `solid_queue_guard:health` | Runtime health report |
|
|
457
135
|
| `solid_queue_guard:report` | Full diagnostic report |
|
|
458
|
-
| `solid_queue_guard:install` | Initializer generator |
|
|
459
|
-
| `solid_queue_guard:install:ci` | GitHub Actions workflow generator |
|
|
460
136
|
| `mount SolidQueueGuard::Engine` | HTTP health endpoint |
|
|
461
|
-
| `config.integrate_mission_control` | Guard tab
|
|
462
|
-
| `GET /jobs/applications/:application_id/guard` | Human-readable health dashboard (when integration enabled) |
|
|
463
|
-
|
|
464
|
-
Configuration attributes, rake tasks, health JSON shape, and Mission Control integration are public. Internal check classes and registry are `@api private`.
|
|
137
|
+
| `config.integrate_mission_control` | Guard tab (requires `mission_control-jobs`) |
|
|
465
138
|
|
|
466
|
-
|
|
139
|
+
Internal check classes and registry are `@api private`.
|
|
467
140
|
|
|
468
141
|
---
|
|
469
142
|
|
|
470
|
-
## ⚔️
|
|
143
|
+
## ⚔️ vs Mission Control
|
|
471
144
|
|
|
472
|
-
| |
|
|
145
|
+
| | Mission Control | solid_queue_guard |
|
|
473
146
|
| --- | --- | --- |
|
|
474
147
|
| **Purpose** | Inspect & manage jobs | Detect production risk |
|
|
475
|
-
| **UI** | Dashboard | CLI + JSON + health endpoint + optional Guard tab in Mission Control |
|
|
476
|
-
| **Retry / discard** | Yes | No (use Mission Control) |
|
|
477
148
|
| **Config doctor** | No | Yes |
|
|
478
|
-
| **Queue lag alerts** | No | Yes |
|
|
479
149
|
| **Pre-deploy checks** | No | Yes |
|
|
480
|
-
| **
|
|
481
|
-
|
|
482
|
-
**Use both.** They solve different problems.
|
|
483
|
-
|
|
484
|
-
---
|
|
150
|
+
| **Queue lag alerts** | No | Yes |
|
|
485
151
|
|
|
486
|
-
|
|
487
|
-
|
|
488
|
-
| Version | Ships | Status |
|
|
489
|
-
| ------- | ----- | ------ |
|
|
490
|
-
| **v0.1** | `doctor` — config checks, CI integration, install generator | ✅ Released |
|
|
491
|
-
| **v0.2** | Runtime health, queue lag, dispatcher/blocked jobs, HTTP endpoint | ✅ Released |
|
|
492
|
-
| **v0.3** | Slack, Datadog, webhook notifications | ✅ Released |
|
|
493
|
-
| **v0.4** | StatsD, Prometheus, OpenTelemetry metrics | ✅ Released |
|
|
494
|
-
| **v0.5** | Auto-recommendations for `queue.yml` topology | ✅ Released |
|
|
495
|
-
| **v0.6** | Per-check configuration (`disabled_checks`, `config.checks`) | ✅ Released |
|
|
496
|
-
| **v0.7** | Puma plugin runtime check, async supervisor awareness | ✅ Released |
|
|
497
|
-
| **v0.8** | HTTP status policy, `install:ci` generator | ✅ Released |
|
|
498
|
-
| **v1.0** | Stable public API, strict semver | ✅ Released |
|
|
499
|
-
| **v1.1** | Mission Control Guard tab (opt-in) | ✅ Released |
|
|
152
|
+
**Use both.**
|
|
500
153
|
|
|
501
154
|
---
|
|
502
155
|
|
|
@@ -504,55 +157,26 @@ Breaking changes ship only in major versions (`2.0+`). Deprecations warn one min
|
|
|
504
157
|
|
|
505
158
|
| Gem version | Ruby | Rails |
|
|
506
159
|
| ----------- | ---- | ----- |
|
|
160
|
+
| 1.2.x | 3.1+ | 7.1, 7.2, 8.0 |
|
|
507
161
|
| 1.1.x | 3.1+ | 7.1, 7.2, 8.0 |
|
|
508
162
|
| 1.0.x | 3.1+ | 7.1, 7.2, 8.0 |
|
|
509
|
-
| 0.5.x | 3.1+ | 7.1, 7.2, 8.0 |
|
|
510
|
-
| 0.1.x | 3.1+ | 7.1, 7.2, 8.0 |
|
|
511
163
|
|
|
512
|
-
## Requirements
|
|
513
|
-
|
|
514
|
-
- Ruby >= 3.1
|
|
515
|
-
- Rails >= 7.1, < 9.0
|
|
516
164
|
- [solid_queue](https://github.com/rails/solid_queue) >= 1.0, < 2.0
|
|
517
|
-
- [mission_control-jobs](https://github.com/rails/mission_control-jobs)
|
|
165
|
+
- [mission_control-jobs](https://github.com/rails/mission_control-jobs) — optional
|
|
518
166
|
|
|
519
167
|
---
|
|
520
168
|
|
|
521
169
|
## Development
|
|
522
170
|
|
|
171
|
+
See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, `bin/console`, Appraisal matrix, and `script/validate_revelo.sh`.
|
|
172
|
+
|
|
523
173
|
```bash
|
|
524
|
-
|
|
525
|
-
cd solid_queue_guard
|
|
526
|
-
bundle install
|
|
527
|
-
cd test/dummy && bin/setup
|
|
528
|
-
cd ../.. && bundle exec rake test
|
|
174
|
+
bundle exec rake test
|
|
529
175
|
bundle exec rubocop
|
|
530
|
-
bundle exec appraisal install
|
|
531
176
|
bundle exec appraisal rake test
|
|
177
|
+
gem build solid_queue_guard.gemspec
|
|
532
178
|
```
|
|
533
179
|
|
|
534
|
-
Release a new version after CI passes:
|
|
535
|
-
|
|
536
|
-
```bash
|
|
537
|
-
# Tag must match lib/solid_queue_guard/version.rb (currently 1.1.4)
|
|
538
|
-
git tag v1.1.4
|
|
539
|
-
git push origin v1.1.4
|
|
540
|
-
```
|
|
541
|
-
|
|
542
|
-
Trusted Publishing on RubyGems publishes automatically when the tag is pushed.
|
|
543
|
-
|
|
544
|
-
---
|
|
545
|
-
|
|
546
|
-
## Contributing
|
|
547
|
-
|
|
548
|
-
Issues and PRs welcome. Keep it focused: **production safety**, not another job dashboard.
|
|
549
|
-
|
|
550
|
-
1. Fork it
|
|
551
|
-
2. Create your branch (`git checkout -b my-new-check`)
|
|
552
|
-
3. Write a test first
|
|
553
|
-
4. Make the check pass
|
|
554
|
-
5. Open a PR
|
|
555
|
-
|
|
556
180
|
---
|
|
557
181
|
|
|
558
182
|
## License
|
data/bin/console
ADDED
|
Binary file
|
|
@@ -23,6 +23,7 @@ SolidQueueGuard.configure do |config|
|
|
|
23
23
|
# config.health_token = ENV["SOLID_QUEUE_GUARD_TOKEN"]
|
|
24
24
|
# config.integrate_rails_health = true
|
|
25
25
|
# config.integrate_mission_control = true
|
|
26
|
+
# config.on_status_change = ->(previous, current, report) { ... }
|
|
26
27
|
# config.notify_with = [:rails_logger, :slack, :datadog, :webhook]
|
|
27
28
|
# config.metrics_backends = [:statsd, :prometheus, :opentelemetry]
|
|
28
29
|
end
|
|
@@ -40,6 +40,13 @@ module SolidQueueGuard
|
|
|
40
40
|
check_class.check_id
|
|
41
41
|
end
|
|
42
42
|
|
|
43
|
+
def catalog
|
|
44
|
+
{
|
|
45
|
+
config: CONFIG_CHECKS.map(&:check_id),
|
|
46
|
+
runtime: RUNTIME_CHECKS.map(&:check_id)
|
|
47
|
+
}
|
|
48
|
+
end
|
|
49
|
+
|
|
43
50
|
def for(scope)
|
|
44
51
|
case scope.to_sym
|
|
45
52
|
when :config then CONFIG_CHECKS
|
|
@@ -2,6 +2,10 @@
|
|
|
2
2
|
|
|
3
3
|
module SolidQueueGuard
|
|
4
4
|
class Configuration
|
|
5
|
+
class ValidationError < StandardError; end
|
|
6
|
+
|
|
7
|
+
HTTP_STATUS_OPTIONS = HttpStatusPolicy::STATUS_MAP.keys.freeze
|
|
8
|
+
|
|
5
9
|
attr_accessor :enabled,
|
|
6
10
|
:queue_lag_thresholds,
|
|
7
11
|
:failed_jobs_threshold,
|
|
@@ -17,7 +21,8 @@ module SolidQueueGuard
|
|
|
17
21
|
:disabled_checks,
|
|
18
22
|
:checks,
|
|
19
23
|
:degraded_http_status,
|
|
20
|
-
:unhealthy_http_status
|
|
24
|
+
:unhealthy_http_status,
|
|
25
|
+
:on_status_change
|
|
21
26
|
|
|
22
27
|
def initialize
|
|
23
28
|
@enabled = true
|
|
@@ -36,6 +41,18 @@ module SolidQueueGuard
|
|
|
36
41
|
@checks = ActiveSupport::OrderedOptions.new
|
|
37
42
|
@degraded_http_status = :ok
|
|
38
43
|
@unhealthy_http_status = :service_unavailable
|
|
44
|
+
@on_status_change = nil
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
def validate!
|
|
48
|
+
validate_http_status!(:degraded_http_status, degraded_http_status)
|
|
49
|
+
validate_http_status!(:unhealthy_http_status, unhealthy_http_status)
|
|
50
|
+
validate_duration!(:stale_process_threshold, stale_process_threshold)
|
|
51
|
+
validate_duration!(:health_cache_ttl, health_cache_ttl)
|
|
52
|
+
validate_positive_integer!(:failed_jobs_threshold, failed_jobs_threshold)
|
|
53
|
+
validate_positive_integer!(:scheduled_backlog_threshold, scheduled_backlog_threshold)
|
|
54
|
+
validate_callback!(:on_status_change, on_status_change)
|
|
55
|
+
self
|
|
39
56
|
end
|
|
40
57
|
|
|
41
58
|
def strict?
|
|
@@ -66,5 +83,36 @@ module SolidQueueGuard
|
|
|
66
83
|
value = settings[key.to_sym] if value.nil?
|
|
67
84
|
value.nil? ? default : value
|
|
68
85
|
end
|
|
86
|
+
|
|
87
|
+
private
|
|
88
|
+
|
|
89
|
+
def validate_http_status!(attribute, value)
|
|
90
|
+
return if HTTP_STATUS_OPTIONS.include?(normalize_http_status_key(value))
|
|
91
|
+
|
|
92
|
+
raise ValidationError,
|
|
93
|
+
"#{attribute} must be one of #{HTTP_STATUS_OPTIONS.inspect}, got #{value.inspect}"
|
|
94
|
+
end
|
|
95
|
+
|
|
96
|
+
def normalize_http_status_key(value)
|
|
97
|
+
value.is_a?(Integer) ? value : value.to_sym
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
def validate_duration!(attribute, value)
|
|
101
|
+
return if value.is_a?(ActiveSupport::Duration) && value.positive?
|
|
102
|
+
|
|
103
|
+
raise ValidationError, "#{attribute} must be a positive ActiveSupport::Duration"
|
|
104
|
+
end
|
|
105
|
+
|
|
106
|
+
def validate_positive_integer!(attribute, value)
|
|
107
|
+
return if value.is_a?(Integer) && value.positive?
|
|
108
|
+
|
|
109
|
+
raise ValidationError, "#{attribute} must be a positive Integer"
|
|
110
|
+
end
|
|
111
|
+
|
|
112
|
+
def validate_callback!(attribute, value)
|
|
113
|
+
return if value.nil? || value.respond_to?(:call)
|
|
114
|
+
|
|
115
|
+
raise ValidationError, "#{attribute} must respond to #call"
|
|
116
|
+
end
|
|
69
117
|
end
|
|
70
118
|
end
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
module SolidQueueGuard
|
|
4
4
|
module Health
|
|
5
|
+
# @api private
|
|
5
6
|
class Cache
|
|
6
7
|
def self.fetch
|
|
7
8
|
new.fetch
|
|
@@ -9,7 +10,7 @@ module SolidQueueGuard
|
|
|
9
10
|
|
|
10
11
|
def fetch
|
|
11
12
|
Rails.cache.fetch(cache_key, expires_in: SolidQueueGuard.config.health_cache_ttl) do
|
|
12
|
-
build_payload
|
|
13
|
+
build_payload(previous_status: read_cached_status)
|
|
13
14
|
end
|
|
14
15
|
end
|
|
15
16
|
|
|
@@ -19,8 +20,13 @@ module SolidQueueGuard
|
|
|
19
20
|
'solid_queue_guard/health'
|
|
20
21
|
end
|
|
21
22
|
|
|
22
|
-
def
|
|
23
|
+
def read_cached_status
|
|
24
|
+
Rails.cache.read(cache_key)&.dig(:status)
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
def build_payload(previous_status: nil)
|
|
23
28
|
report = Runner.new(scope: :all).run
|
|
29
|
+
Observability.notify_status_change(report, previous_status: previous_status&.to_sym)
|
|
24
30
|
Payload.new(report).to_h
|
|
25
31
|
end
|
|
26
32
|
end
|
|
@@ -4,6 +4,7 @@ module SolidQueueGuard
|
|
|
4
4
|
module Metrics
|
|
5
5
|
class Exporter
|
|
6
6
|
STATUS_VALUES = { healthy: 0, degraded: 1, unhealthy: 2 }.freeze
|
|
7
|
+
CHECK_STATUS_VALUES = { pass: 0, warn: 1, fail: 2, skip: 3 }.freeze
|
|
7
8
|
|
|
8
9
|
def self.export(report, backends: SolidQueueGuard.config.metrics_backends)
|
|
9
10
|
new(report, backends: backends).export
|
|
@@ -27,6 +28,8 @@ module SolidQueueGuard
|
|
|
27
28
|
when :statsd then Statsd.export(report)
|
|
28
29
|
when :prometheus then Prometheus.export(report)
|
|
29
30
|
when :opentelemetry then OpenTelemetry.export(report)
|
|
31
|
+
else
|
|
32
|
+
OptionalDependency.log_missing("metrics backend #{backend}")
|
|
30
33
|
end
|
|
31
34
|
end
|
|
32
35
|
end
|
|
@@ -6,14 +6,22 @@ module SolidQueueGuard
|
|
|
6
6
|
module_function
|
|
7
7
|
|
|
8
8
|
def export(report)
|
|
9
|
-
return unless
|
|
9
|
+
return unless OptionalDependency.require!('opentelemetry-sdk', 'opentelemetry-sdk')
|
|
10
10
|
|
|
11
11
|
meter = ::OpenTelemetry.meter_provider.meter('solid_queue_guard')
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
12
|
+
overall_gauge = meter.create_gauge('solid_queue.guard.overall_status', unit: 'status')
|
|
13
|
+
overall_gauge.record(STATUS_VALUES.fetch(report.status))
|
|
14
|
+
|
|
15
|
+
check_gauge = meter.create_gauge('solid_queue.guard.check_status', unit: 'status')
|
|
16
|
+
report.results.each do |result|
|
|
17
|
+
check_gauge.record(CHECK_STATUS_VALUES.fetch(result.status), attributes: { 'check.id' => result.id })
|
|
18
|
+
end
|
|
19
|
+
rescue StandardError => e
|
|
20
|
+
Rails.logger.warn("[solid_queue_guard] OpenTelemetry export failed: #{e.class}: #{e.message}")
|
|
16
21
|
end
|
|
22
|
+
|
|
23
|
+
STATUS_VALUES = Exporter::STATUS_VALUES
|
|
24
|
+
CHECK_STATUS_VALUES = Exporter::CHECK_STATUS_VALUES
|
|
17
25
|
end
|
|
18
26
|
end
|
|
19
27
|
end
|
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require 'fileutils'
|
|
4
|
+
|
|
3
5
|
module SolidQueueGuard
|
|
4
6
|
module Metrics
|
|
5
7
|
module Prometheus
|
|
@@ -7,15 +9,27 @@ module SolidQueueGuard
|
|
|
7
9
|
|
|
8
10
|
def export(report)
|
|
9
11
|
path = ENV.fetch('SOLID_QUEUE_GUARD_PROMETHEUS_FILE', Rails.root.join('tmp/solid_queue_guard.prom'))
|
|
12
|
+
FileUtils.mkdir_p(File.dirname(path))
|
|
10
13
|
File.write(path, render(report))
|
|
11
14
|
end
|
|
12
15
|
|
|
13
16
|
def render(report)
|
|
14
|
-
|
|
15
|
-
# TYPE solid_queue_guard_overall_status gauge
|
|
16
|
-
solid_queue_guard_overall_status #{
|
|
17
|
-
|
|
17
|
+
lines = [
|
|
18
|
+
'# TYPE solid_queue_guard_overall_status gauge',
|
|
19
|
+
"solid_queue_guard_overall_status #{STATUS_VALUES.fetch(report.status)}",
|
|
20
|
+
'# TYPE solid_queue_guard_check_status gauge'
|
|
21
|
+
]
|
|
22
|
+
|
|
23
|
+
report.results.each do |result|
|
|
24
|
+
value = CHECK_STATUS_VALUES.fetch(result.status)
|
|
25
|
+
lines << "solid_queue_guard_check_status{check=\"#{result.id}\"} #{value}"
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
"#{lines.join("\n")}\n"
|
|
18
29
|
end
|
|
30
|
+
|
|
31
|
+
STATUS_VALUES = Exporter::STATUS_VALUES
|
|
32
|
+
CHECK_STATUS_VALUES = Exporter::CHECK_STATUS_VALUES
|
|
19
33
|
end
|
|
20
34
|
end
|
|
21
35
|
end
|
|
@@ -6,20 +6,32 @@ module SolidQueueGuard
|
|
|
6
6
|
module_function
|
|
7
7
|
|
|
8
8
|
def export(report)
|
|
9
|
-
require 'socket'
|
|
10
9
|
host = ENV.fetch('SOLID_QUEUE_GUARD_STATSD_HOST', '127.0.0.1')
|
|
11
10
|
port = ENV.fetch('SOLID_QUEUE_GUARD_STATSD_PORT', 8125).to_i
|
|
12
11
|
socket = UDPSocket.new
|
|
13
|
-
|
|
12
|
+
metric_lines(report).each do |line|
|
|
13
|
+
socket.send(line, 0, host, port)
|
|
14
|
+
end
|
|
14
15
|
ensure
|
|
15
16
|
socket&.close
|
|
16
17
|
end
|
|
17
18
|
|
|
18
|
-
def
|
|
19
|
-
|
|
19
|
+
def metric_lines(report)
|
|
20
|
+
[
|
|
21
|
+
"solid_queue.guard.overall_status:#{STATUS_VALUES.fetch(report.status)}|g",
|
|
22
|
+
*check_status_lines(report)
|
|
23
|
+
]
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def check_status_lines(report)
|
|
27
|
+
report.results.map do |result|
|
|
28
|
+
value = CHECK_STATUS_VALUES.fetch(result.status)
|
|
29
|
+
"solid_queue.guard.check_status:#{value}|g|##{result.id}"
|
|
30
|
+
end
|
|
20
31
|
end
|
|
21
32
|
|
|
22
33
|
STATUS_VALUES = Exporter::STATUS_VALUES
|
|
34
|
+
CHECK_STATUS_VALUES = Exporter::CHECK_STATUS_VALUES
|
|
23
35
|
end
|
|
24
36
|
end
|
|
25
37
|
end
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SolidQueueGuard
|
|
4
|
+
# @api private
|
|
5
|
+
module Observability
|
|
6
|
+
module_function
|
|
7
|
+
|
|
8
|
+
def log_results(report)
|
|
9
|
+
return unless logger
|
|
10
|
+
|
|
11
|
+
report.results.each do |result|
|
|
12
|
+
message = "[SolidQueueGuard] #{result.status} check #{result.id}: #{result.message}"
|
|
13
|
+
case result.status
|
|
14
|
+
when :skip, :pass then logger.debug(message)
|
|
15
|
+
when :warn then logger.warn(message)
|
|
16
|
+
when :fail then logger.error(message)
|
|
17
|
+
end
|
|
18
|
+
end
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
def notify_status_change(report, previous_status: nil)
|
|
22
|
+
callback = SolidQueueGuard.config.on_status_change
|
|
23
|
+
return unless callback.respond_to?(:call)
|
|
24
|
+
|
|
25
|
+
current_status = report.status
|
|
26
|
+
return if previous_status == current_status
|
|
27
|
+
|
|
28
|
+
callback.call(previous_status, current_status, report)
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
def logger
|
|
32
|
+
defined?(Rails) && Rails.logger
|
|
33
|
+
end
|
|
34
|
+
end
|
|
35
|
+
end
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SolidQueueGuard
|
|
4
|
+
# @api private
|
|
5
|
+
module OptionalDependency
|
|
6
|
+
module_function
|
|
7
|
+
|
|
8
|
+
def require!(feature, gem_name = feature)
|
|
9
|
+
require feature
|
|
10
|
+
true
|
|
11
|
+
rescue LoadError
|
|
12
|
+
log_missing(gem_name)
|
|
13
|
+
false
|
|
14
|
+
end
|
|
15
|
+
|
|
16
|
+
def log_missing(gem_name)
|
|
17
|
+
return unless defined?(Rails) && Rails.logger
|
|
18
|
+
|
|
19
|
+
Rails.logger.warn(
|
|
20
|
+
"[solid_queue_guard] Optional dependency #{gem_name} is not installed. " \
|
|
21
|
+
'Add it to your Gemfile to enable this integration.'
|
|
22
|
+
)
|
|
23
|
+
end
|
|
24
|
+
end
|
|
25
|
+
end
|
|
@@ -37,8 +37,13 @@ module SolidQueueGuard
|
|
|
37
37
|
status: status.to_s,
|
|
38
38
|
warnings: warnings,
|
|
39
39
|
suggestions: suggestions,
|
|
40
|
+
status_counts: status_counts,
|
|
40
41
|
checks: results.map(&:to_h)
|
|
41
42
|
}
|
|
42
43
|
end
|
|
44
|
+
|
|
45
|
+
def status_counts
|
|
46
|
+
results.group_by(&:status).transform_values(&:size)
|
|
47
|
+
end
|
|
43
48
|
end
|
|
44
49
|
end
|
data/solid_queue_guard.gemspec
CHANGED
|
@@ -26,7 +26,8 @@ Gem::Specification.new do |spec|
|
|
|
26
26
|
|
|
27
27
|
spec.files = Dir.chdir(File.expand_path(__dir__)) do
|
|
28
28
|
`git ls-files -z`.split("\x0").reject do |file|
|
|
29
|
-
file.start_with?('test/', '.github/', '
|
|
29
|
+
file.start_with?('test/', '.github/', 'script/', 'gemfiles/', 'Appraisals') ||
|
|
30
|
+
(file.start_with?('docs/') && file != 'docs/mission-control-guard-dashboard.png')
|
|
30
31
|
end
|
|
31
32
|
end
|
|
32
33
|
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: solid_queue_guard
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Rafael Pissardo
|
|
@@ -301,11 +301,13 @@ files:
|
|
|
301
301
|
- ".rubocop.yml"
|
|
302
302
|
- ".ruby-version"
|
|
303
303
|
- CHANGELOG.md
|
|
304
|
+
- CONTRIBUTING.md
|
|
304
305
|
- Gemfile
|
|
305
306
|
- Gemfile.lock
|
|
306
307
|
- MIT-LICENSE
|
|
307
308
|
- README.md
|
|
308
309
|
- Rakefile
|
|
310
|
+
- app/assets/images/solid_queue_guard/mission-control-guard-dashboard.png
|
|
309
311
|
- app/controllers/solid_queue_guard/application_controller.rb
|
|
310
312
|
- app/controllers/solid_queue_guard/health_controller.rb
|
|
311
313
|
- app/controllers/solid_queue_guard/mission_control/base_controller.rb
|
|
@@ -314,7 +316,9 @@ files:
|
|
|
314
316
|
- app/views/solid_queue_guard/mission_control/dashboard/_check.html.erb
|
|
315
317
|
- app/views/solid_queue_guard/mission_control/dashboard/_summary_metrics.html.erb
|
|
316
318
|
- app/views/solid_queue_guard/mission_control/dashboard/show.html.erb
|
|
319
|
+
- bin/console
|
|
317
320
|
- config/routes.rb
|
|
321
|
+
- docs/mission-control-guard-dashboard.png
|
|
318
322
|
- lib/generators/solid_queue_guard/install/USAGE
|
|
319
323
|
- lib/generators/solid_queue_guard/install/USAGE.ci
|
|
320
324
|
- lib/generators/solid_queue_guard/install/ci_generator.rb
|
|
@@ -368,6 +372,8 @@ files:
|
|
|
368
372
|
- lib/solid_queue_guard/mission_control/navigation_extension.rb
|
|
369
373
|
- lib/solid_queue_guard/notifier.rb
|
|
370
374
|
- lib/solid_queue_guard/notifiers/base.rb
|
|
375
|
+
- lib/solid_queue_guard/observability.rb
|
|
376
|
+
- lib/solid_queue_guard/optional_dependency.rb
|
|
371
377
|
- lib/solid_queue_guard/puma_plugin_support.rb
|
|
372
378
|
- lib/solid_queue_guard/queue_coverage.rb
|
|
373
379
|
- lib/solid_queue_guard/recommendations/topology.rb
|