standard_health 0.4.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +180 -0
- data/{MIT-LICENSE → LICENSE} +8 -6
- data/README.md +348 -5
- data/app/controllers/standard_health/diagnostics_controller.rb +27 -0
- data/app/controllers/standard_health/health_controller.rb +29 -1
- data/lib/generators/standard_health/install/install_generator.rb +150 -0
- data/lib/generators/standard_health/install/templates/initializer.rb.erb +99 -0
- data/lib/standard_health/aggregator.rb +150 -11
- data/lib/standard_health/check.rb +6 -1
- data/lib/standard_health/checks/env_spec_audit.rb +105 -0
- data/lib/standard_health/checks/solid_cable.rb +54 -0
- data/lib/standard_health/configuration.rb +114 -3
- data/lib/standard_health/engine.rb +13 -0
- data/lib/standard_health/env_spec.rb +108 -6
- data/lib/standard_health/event_emitter.rb +41 -0
- data/lib/standard_health/notifiers/logger.rb +88 -0
- data/lib/standard_health/notifiers/metrics.rb +88 -0
- data/lib/standard_health/notifiers/sentry.rb +153 -0
- data/lib/standard_health/redactor.rb +74 -0
- data/lib/standard_health/subscribers.rb +103 -0
- data/lib/standard_health/version.rb +1 -1
- data/lib/standard_health.rb +17 -0
- metadata +14 -4
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 3d6a6ea3266864aaa15232529241e2040a836d810c4898eb512746e78a582901
|
|
4
|
+
data.tar.gz: fc9575f2c3e425a6065e3e5285094d03ba5cd694848c386937251b0fea5d75af
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 83a04f1025e9e8b6e62344ff53828a716c262d7779d30f64afeb69b704fb34289e9138f751a93fb46215ba24c71612e17e9cfd7a37c859ac67feca64bbb2e25a
|
|
7
|
+
data.tar.gz: 6cf0b8d3a4100e8d3737eb1bbaddfb7c1a40d6dd77ad4e9c097ad547308ff196fb012b46ebb7c1d267b30d784abc863f734df4bd8520971d99af9d705e2af51b
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,186 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.5.0] - 2026-07-30
|
|
11
|
+
|
|
12
|
+
Closes the gap that had four host apps writing their own checks. The env-spec
|
|
13
|
+
DSL could only audit **presence**, so "this toggle must not be set on
|
|
14
|
+
production" and "this value must be exactly `false`" were inexpressible — and
|
|
15
|
+
the workaround was a hand-written check carrying a duplicate list of variable
|
|
16
|
+
names, kept in sync with the spec by comment.
|
|
17
|
+
|
|
18
|
+
Everything here is **additive and opt-in**. No check is auto-registered and no
|
|
19
|
+
existing status changes meaning, so this is a pure `bundle update` for
|
|
20
|
+
consumers on `~> 0.4`.
|
|
21
|
+
|
|
22
|
+
### Added
|
|
23
|
+
|
|
24
|
+
- **`forbidden` EnvSpec level.** Inverts the presence rule: absent is `:ok`,
|
|
25
|
+
present reports the new `:forbidden` status. For dangerous ops toggles —
|
|
26
|
+
demo modes, auth bypasses, bootstrap flags — that are legitimate on staging
|
|
27
|
+
and must never survive promotion. Composes with `in:`, `mode_alias`, groups
|
|
28
|
+
and the `if:`/`unless:` predicates like any other level.
|
|
29
|
+
- **`expected_value:` assertion** on `required` / `recommended`. Asserts the
|
|
30
|
+
value, not just presence; a present-but-wrong value reports the new
|
|
31
|
+
`:mismatch` status. Accepts a String, an Array (any-of), a Regexp (matched,
|
|
32
|
+
not compared), or any object comparable by its string form. The **actual
|
|
33
|
+
value is never surfaced** — env values are routinely secrets — only the
|
|
34
|
+
declared expectation.
|
|
35
|
+
- **`Checks::EnvSpecAudit`** (opt-in). Runs the configured `env_spec` and
|
|
36
|
+
fails on `:missing` / `:forbidden` / `:mismatch`, reporting offending
|
|
37
|
+
variable *names* grouped by status. Reads the spec directly, so there is no
|
|
38
|
+
second list to drift. Non-critical by default: config drift is visibility,
|
|
39
|
+
not a rotation signal. Skips `consumed_by` resolution — that is file IO per
|
|
40
|
+
entry, fine for an on-demand doctor endpoint and not on a polled tier.
|
|
41
|
+
- **`Checks::SolidCable`** (opt-in). Bounded read against
|
|
42
|
+
`solid_cable_messages`. Promoted from sidekick-web. Falls back to the
|
|
43
|
+
primary connection when SolidCable isn't pointed at its own database.
|
|
44
|
+
- **`standard_health:install` generator.** Writes the initializer and mounts
|
|
45
|
+
the engine, with the aggregate-route ordering requirement noted inline.
|
|
46
|
+
Idempotent; `--skip-initializer`, `--skip-routes`, `--force`.
|
|
47
|
+
|
|
48
|
+
### Changed
|
|
49
|
+
|
|
50
|
+
- **`/diagnostics/env` top-level `status`** now reports `incomplete` for
|
|
51
|
+
`:forbidden` and `:mismatch` rows as well as `:missing`. Callers already
|
|
52
|
+
gating on that one field pick up the new assertions for free, which is why
|
|
53
|
+
they joined the existing roll-up instead of getting a verdict of their own.
|
|
54
|
+
The endpoint still returns 200 either way. `:should_set` remains advisory
|
|
55
|
+
and still never affects the roll-up.
|
|
56
|
+
- **Rails dependency relaxed** from `~> 8.0` to `>= 8.0`, matching the rest of
|
|
57
|
+
the `standard_*` family. `~> 8.0` was the only floor in the family that
|
|
58
|
+
would have blocked a Rails 9 host.
|
|
59
|
+
|
|
60
|
+
### Not in this release
|
|
61
|
+
|
|
62
|
+
- **Timeout defaults remain unset.** 0.4.1 said sensible `default_check_timeout`
|
|
63
|
+
/ budget values would be chosen "in 0.5.0". They are not: choosing them still
|
|
64
|
+
needs enough real p99 latency data from the instrumentation 0.4.1 added, and
|
|
65
|
+
guessing a timeout on a health check is how you cause the outage you were
|
|
66
|
+
trying to prevent. The machinery stays opt-in per check. The stale promise in
|
|
67
|
+
`Configuration` has been repointed (#48).
|
|
68
|
+
- **`/diagnostics/env` still returns 200** when `status` is `incomplete`.
|
|
69
|
+
Turning that into a 503 is a breaking contract change for any caller gating on
|
|
70
|
+
the response code, so it wants its own decision rather than riding along with
|
|
71
|
+
an otherwise additive release.
|
|
72
|
+
|
|
73
|
+
### Documentation
|
|
74
|
+
|
|
75
|
+
- **The aggregate `GET /health` tier and its ordering requirement are now
|
|
76
|
+
documented.** The engine draws sub-paths only; the aggregate tier is the
|
|
77
|
+
host's job and must be drawn **before** `mount`. An app that mounts first
|
|
78
|
+
and relies on the engine to serve it silently has no aggregate tier — no
|
|
79
|
+
boot error, no failing route spec. Also documents what each of the four
|
|
80
|
+
tiers is for, and that readiness gates only on hard infra the app owns.
|
|
81
|
+
- New README section on opt-in checks, including why nothing here
|
|
82
|
+
auto-registers.
|
|
83
|
+
|
|
84
|
+
## [0.4.1] - 2026-07-29
|
|
85
|
+
|
|
86
|
+
Observability release. Until now the gem emitted **nothing** — no events, no
|
|
87
|
+
logs, no metrics. It computed `latency_ms` for every check and discarded it
|
|
88
|
+
into the response body. A failing check was invisible unless somebody happened
|
|
89
|
+
to call `/ready` at the right moment, so there was no history, no alerting
|
|
90
|
+
hook, and no way to chart check latency.
|
|
91
|
+
|
|
92
|
+
Everything here is **additive**: no status code, roll-up rule, or response
|
|
93
|
+
field changes meaning, so this is a pure `bundle update` for consumers on
|
|
94
|
+
`~> 0.4`.
|
|
95
|
+
|
|
96
|
+
### Added
|
|
97
|
+
|
|
98
|
+
- **Instrumentation.** Three events over whichever bus is live (`Rails.event`
|
|
99
|
+
on Rails 8.1+, `ActiveSupport::Notifications` otherwise), matching
|
|
100
|
+
`standard_circuit`'s `EventEmitter` idiom:
|
|
101
|
+
- `standard_health.check.completed` — `name`, `critical`, `status`,
|
|
102
|
+
`latency_ms`, `error_class`, `error_message`
|
|
103
|
+
- `standard_health.check.timed_out` — `name`, `critical`, `timeout_s`
|
|
104
|
+
- `standard_health.ready.evaluated` — `status`, `duration_ms`, `failed[]`
|
|
105
|
+
- **Three built-in subscribers**, registered by a new engine initializer
|
|
106
|
+
(`after: :load_config_initializers`, so host config is final):
|
|
107
|
+
- `Notifiers::Logger` — **silent while healthy.** `warn` on degraded,
|
|
108
|
+
`error` on unavailable. Health events are polls (~6/min/instance), not
|
|
109
|
+
transitions; a line per evaluation is ~8,640/day/instance of "everything
|
|
110
|
+
is fine".
|
|
111
|
+
- `Notifiers::Sentry` — **transition-only**, with a 60s repeat floor and
|
|
112
|
+
one `info` on recovery. Capturing every non-ok poll would turn a
|
|
113
|
+
five-minute outage into ~30 duplicate issues. Mutex-guarded (Puma is
|
|
114
|
+
threaded). Sentry stays a soft dependency.
|
|
115
|
+
- `Notifiers::Metrics` — per-poll counters and latency distributions. This
|
|
116
|
+
fires on every evaluation on purpose: it is what makes `latency_ms`
|
|
117
|
+
chartable.
|
|
118
|
+
- `config.add_notifier` for host-supplied `call(event_name, payload)`
|
|
119
|
+
subscribers, validated at add time.
|
|
120
|
+
- Config: `instrumentation_enabled`, `logger`, `sentry_enabled`,
|
|
121
|
+
`metric_prefix`.
|
|
122
|
+
- **Per-check timeout machinery** — `default_check_timeout`,
|
|
123
|
+
`total_check_budget`, and a `timeout:` option on `register_check`. Uses a
|
|
124
|
+
dedicated `StandardHealth::CheckTimeout` rather than bare `Timeout::Error`,
|
|
125
|
+
so a timeout the host raised for its own reasons is never mislabelled as
|
|
126
|
+
ours. **All default to `nil` (OFF)** — see below.
|
|
127
|
+
- `status` on `/diagnostics/env`: `:ok` or `:incomplete` (a `required` var is
|
|
128
|
+
missing). **Additive — the endpoint still returns 200 either way.**
|
|
129
|
+
|
|
130
|
+
### Fixed
|
|
131
|
+
|
|
132
|
+
- `Check#with_timing` now records `error_class` alongside the message. All
|
|
133
|
+
three built-in checks route through it, so without this every real driver
|
|
134
|
+
failure reached the aggregator with no class and the redacted body fell back
|
|
135
|
+
to a generic `StandardError` — exactly the useless label redaction exists to
|
|
136
|
+
replace.
|
|
137
|
+
- Failure detail (`error_class` + `error_message`) now rides on
|
|
138
|
+
`ready.evaluated`, not only on `check.completed`. Both built-in subscribers
|
|
139
|
+
are driven by `ready.evaluated` (deliberately — it is the transition-gated
|
|
140
|
+
event), so the message would otherwise have been deleted from the response
|
|
141
|
+
without ever reaching logs or Sentry.
|
|
142
|
+
|
|
143
|
+
### Security
|
|
144
|
+
|
|
145
|
+
- **`/ready` no longer returns raw exception messages.** The endpoint is
|
|
146
|
+
unauthenticated by design, and a driver error will happily tell an anonymous
|
|
147
|
+
caller the database host, port and username — during exactly the incident
|
|
148
|
+
you least want to be leaking. Failing rows now carry `error_class` and a
|
|
149
|
+
stable `error_code` instead. The full message is not lost: it goes to logs
|
|
150
|
+
and Sentry via the instrumentation above, which is why redaction ships in
|
|
151
|
+
the same release rather than separately.
|
|
152
|
+
- `config.expose_check_errors = true` restores the previous bodies.
|
|
153
|
+
- `config.detail_token` + an `X-Health-Token` header is a break-glass path
|
|
154
|
+
for on-call, compared in constant time.
|
|
155
|
+
- `status`, `name`, `critical`, `latency_ms` and `generated_at` are
|
|
156
|
+
unchanged, so dashboards and existing specs keep working.
|
|
157
|
+
|
|
158
|
+
### Notes on what is deliberately NOT on
|
|
159
|
+
|
|
160
|
+
Timeouts and the total budget default to **off**, giving byte-identical
|
|
161
|
+
behaviour to 0.4.0. Turning them on is a semantic change: a check that has
|
|
162
|
+
always been slow-but-fine starts reporting `:fail`, and for a critical check
|
|
163
|
+
that pulls the instance out of rotation. Shipping that in a patch, to five
|
|
164
|
+
apps, on a `bundle update`, is how you cause the outage you were preventing.
|
|
165
|
+
|
|
166
|
+
The machinery ships now so apps can opt in per check, and so the events above
|
|
167
|
+
can reveal the real p99 latencies. Defaults get chosen from that data in
|
|
168
|
+
0.5.0, which requires an explicit Gemfile edit.
|
|
169
|
+
|
|
170
|
+
One semantic guard is already in place for when they are enabled: a check
|
|
171
|
+
skipped because the budget ran out reports `:skipped` and floors the roll-up
|
|
172
|
+
at `:degraded` — **never `:unavailable`**, even when the skipped check is
|
|
173
|
+
critical. Otherwise a slow *non-critical* check could exhaust the budget,
|
|
174
|
+
leave the database check unrun, and pull a healthy instance out of rotation.
|
|
175
|
+
|
|
176
|
+
`total_check_budget` bounds **how many checks run**, not how long the probe
|
|
177
|
+
takes — it is evaluated before each check starts, not during one. Clamping
|
|
178
|
+
each check to the remaining budget would close that gap but would apply
|
|
179
|
+
`Timeout.timeout` to checks whose author never asked for one, so setting a
|
|
180
|
+
budget deliberately does not opt you into that. Asserted by spec, not just
|
|
181
|
+
documented.
|
|
182
|
+
|
|
183
|
+
A second reason to leave them off: `Timeout.timeout` raises into the running
|
|
184
|
+
thread at an arbitrary point, so firing one mid-connection-checkout can return
|
|
185
|
+
a broken connection to the pool. The dedicated `CheckTimeout` fixes
|
|
186
|
+
mislabelling, not this. The README now carries the full caveat — the right
|
|
187
|
+
answer for datastore checks is usually a driver-level timeout
|
|
188
|
+
(`connect_timeout` / `statement_timeout`), not this setting.
|
|
189
|
+
|
|
10
190
|
## [0.4.0] - 2026-05-05
|
|
11
191
|
|
|
12
192
|
### Added
|
data/{MIT-LICENSE → LICENSE}
RENAMED
|
@@ -1,4 +1,6 @@
|
|
|
1
|
-
|
|
1
|
+
The MIT License (MIT)
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Rarebit One
|
|
2
4
|
|
|
3
5
|
Permission is hereby granted, free of charge, to any person obtaining
|
|
4
6
|
a copy of this software and associated documentation files (the
|
|
@@ -13,8 +15,8 @@ included in all copies or substantial portions of the Software.
|
|
|
13
15
|
|
|
14
16
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
|
15
17
|
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
|
16
|
-
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
|
17
|
-
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
|
18
|
-
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
|
19
|
-
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
|
20
|
-
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
|
18
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
|
19
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
|
20
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
|
21
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
|
22
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
CHANGED
|
@@ -18,7 +18,16 @@ Add to your Gemfile:
|
|
|
18
18
|
gem "standard_health"
|
|
19
19
|
```
|
|
20
20
|
|
|
21
|
-
Then `bundle install
|
|
21
|
+
Then `bundle install`, and run the install generator:
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
bin/rails generate standard_health:install
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
It writes `config/initializers/standard_health.rb` and mounts the engine in
|
|
28
|
+
`config/routes.rb` with the ordering requirement below noted inline. It is
|
|
29
|
+
idempotent — re-running skips what is already installed. Flags:
|
|
30
|
+
`--skip-initializer`, `--skip-routes`, `--force` (overwrite the initializer).
|
|
22
31
|
|
|
23
32
|
## Mounting
|
|
24
33
|
|
|
@@ -34,9 +43,45 @@ This wires up:
|
|
|
34
43
|
- `GET /health/ready`
|
|
35
44
|
- `GET /health/diagnostics/env`
|
|
36
45
|
|
|
46
|
+
### The aggregate `GET /health` is yours to draw — before the mount
|
|
47
|
+
|
|
48
|
+
**The engine draws sub-paths only.** There is no `GET /health` in
|
|
49
|
+
`config/routes.rb` here; the aggregate tier is the host's responsibility, and
|
|
50
|
+
the ordering is not optional:
|
|
51
|
+
|
|
52
|
+
```ruby
|
|
53
|
+
get "/health", to: "health_aggregate#show" # aggregate — FIRST
|
|
54
|
+
mount StandardHealth::Engine => "/health", as: :standard_health
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
An app that mounts the engine **first** and relies on it to serve the aggregate
|
|
58
|
+
tier **silently has no aggregate tier at all** — no boot error, no 404 at boot,
|
|
59
|
+
and no failing route spec unless one exists. The failure surfaces as a
|
|
60
|
+
dashboard that has been reading nothing for months. Add a route spec asserting
|
|
61
|
+
`GET /health` resolves.
|
|
62
|
+
|
|
63
|
+
Why this way round: `mount` claims the `/health` prefix, so a bare `GET /health`
|
|
64
|
+
drawn *after* it still resolves (the engine has no route for the bare path) —
|
|
65
|
+
but nothing about that is obvious from reading the file, and getting it
|
|
66
|
+
backwards fails silently either way. Draw it first and the ordering question
|
|
67
|
+
never arises.
|
|
68
|
+
|
|
69
|
+
The four tiers, and what each is for:
|
|
70
|
+
|
|
71
|
+
| Route | Tier | Consumer |
|
|
72
|
+
|---|---|---|
|
|
73
|
+
| `GET /health/alive` | liveness — process is up | restart probes, CI reachability gates |
|
|
74
|
+
| `GET /health/ready` | readiness — **rotation gate** | load balancer / platform health check |
|
|
75
|
+
| `GET /health` | aggregate — full picture | dashboards, humans |
|
|
76
|
+
| `GET /health/diagnostics/env` | doctor (authed) | on-call, config audits |
|
|
77
|
+
|
|
78
|
+
Readiness gates **only** on hard infra the app owns. A soft upstream that
|
|
79
|
+
degrades must not pull an instance out of rotation — put those on the aggregate
|
|
80
|
+
tier instead.
|
|
81
|
+
|
|
37
82
|
## Configuration
|
|
38
83
|
|
|
39
|
-
Create `config/initializers/standard_health.rb
|
|
84
|
+
Create `config/initializers/standard_health.rb` (or let the generator write it):
|
|
40
85
|
|
|
41
86
|
```ruby
|
|
42
87
|
StandardHealth.configure do |c|
|
|
@@ -62,18 +107,21 @@ end
|
|
|
62
107
|
|
|
63
108
|
## EnvSpec
|
|
64
109
|
|
|
65
|
-
The DSL has
|
|
110
|
+
The DSL has three declarations:
|
|
66
111
|
|
|
67
112
|
- `required :NAME` — missing value reports `status: :missing`
|
|
68
113
|
- `recommended :NAME` — missing value reports `status: :should_set`
|
|
114
|
+
- `forbidden :NAME` — **present** value reports `status: :forbidden`; absent is `:ok`
|
|
69
115
|
|
|
70
|
-
|
|
116
|
+
All three accept:
|
|
71
117
|
|
|
72
118
|
- `in: %w[staging production]` — restricts the entry to those `APP_ENVIRONMENT` values; ignored otherwise. May also be a Symbol resolved via `mode_alias` (see below).
|
|
73
119
|
- `description: "..."` — surfaced verbatim in the audit JSON
|
|
74
120
|
- `consumed_by: "config/initializers/sentry.rb"` — pointer (or `Array<String>`) to where the value is read; surfaced verbatim
|
|
75
121
|
- `if: -> { ... }` / `unless: -> { ... }` — Proc predicates evaluated at audit time. When `unless:` returns truthy or `if:` returns falsy, the entry is reported with `status: :not_applicable`
|
|
76
122
|
|
|
123
|
+
`required` and `recommended` additionally accept `expected_value:` (see below). `forbidden` does not — its assertion *is* "absent" — and combining them raises an `ArgumentError` at `define` time rather than silently ignoring the option.
|
|
124
|
+
|
|
77
125
|
Audit output (one row per applicable entry):
|
|
78
126
|
|
|
79
127
|
```json
|
|
@@ -85,7 +133,68 @@ Audit output (one row per applicable entry):
|
|
|
85
133
|
}
|
|
86
134
|
```
|
|
87
135
|
|
|
88
|
-
Possible `status` values
|
|
136
|
+
Possible `status` values:
|
|
137
|
+
|
|
138
|
+
| status | meaning |
|
|
139
|
+
|---|---|
|
|
140
|
+
| `ok` | nothing to report |
|
|
141
|
+
| `missing` | `required` + absent |
|
|
142
|
+
| `should_set` | `recommended` + absent (advisory) |
|
|
143
|
+
| `forbidden` | `forbidden` + **present** |
|
|
144
|
+
| `mismatch` | present, but `expected_value:` says otherwise |
|
|
145
|
+
| `not_applicable` | suppressed by an `if:`/`unless:` predicate |
|
|
146
|
+
|
|
147
|
+
`missing`, `forbidden` and `mismatch` are the **violation** statuses — they drive the top-level `status: "incomplete"` and the optional `EnvSpecAudit` check. `should_set` is advisory and never counts as a violation.
|
|
148
|
+
|
|
149
|
+
### `forbidden`: vars that must NOT be set
|
|
150
|
+
|
|
151
|
+
For dangerous ops toggles — demo modes, auth bypasses, bootstrap flags — that are legitimate on staging and must never survive promotion to production:
|
|
152
|
+
|
|
153
|
+
```ruby
|
|
154
|
+
mode_alias :live, %w[production]
|
|
155
|
+
|
|
156
|
+
group "Production-forbidden toggles" do
|
|
157
|
+
forbidden :DEMO_MODE_ENABLED, in: :live,
|
|
158
|
+
description: "Demo/ops dashboard surfaces; unset before promoting"
|
|
159
|
+
forbidden :STANDARD_ID_BYPASS_CODE, in: :live,
|
|
160
|
+
description: "Fixed E2E OTP bypass code; staging only"
|
|
161
|
+
end
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
```json
|
|
165
|
+
{ "name": "DEMO_MODE_ENABLED", "level": "forbidden", "status": "forbidden", "mode": "production" }
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
Before this level existed, hosts expressed "forbidden" by declaring the var `recommended` with an `if: -> { ENV[...].present? }` predicate so that a set toggle at least produced a row — but the row was a green `:ok`, so the signal had to be carried by a hand-written custom check with its own duplicate list of toggle names. `forbidden` replaces both halves.
|
|
169
|
+
|
|
170
|
+
### `expected_value:`: assert the value, not just presence
|
|
171
|
+
|
|
172
|
+
Presence auditing misses the case where a var is set to the *wrong* thing — which for a security toggle is the failure mode that matters:
|
|
173
|
+
|
|
174
|
+
```ruby
|
|
175
|
+
# CSP is only enforced when this is exactly the string "false";
|
|
176
|
+
# "true" passes a presence audit while leaving the CSP report-only.
|
|
177
|
+
required :CONTENT_SECURITY_POLICY_REPORT_ONLY, in: :live,
|
|
178
|
+
expected_value: "false",
|
|
179
|
+
description: "Any other value leaves the CSP report-only"
|
|
180
|
+
|
|
181
|
+
required :LOG_LEVEL, expected_value: %w[info warn] # any of
|
|
182
|
+
required :DATABASE_URL, expected_value: /\Apostgres:/ # matched, not compared
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
A present-but-wrong value reports `status: :mismatch`. An absent value still reports `:missing` / `:should_set` — there is nothing to compare. Comparison is on the string form, so `expected_value: 3000` matches `"3000"`.
|
|
186
|
+
|
|
187
|
+
**The actual value is never surfaced.** Env values are routinely secrets, and the endpoint exists to report *that* something is wrong, not to echo it back. The row carries the declared `expected_value` (host config, safe) and nothing else:
|
|
188
|
+
|
|
189
|
+
```json
|
|
190
|
+
{
|
|
191
|
+
"name": "CONTENT_SECURITY_POLICY_REPORT_ONLY",
|
|
192
|
+
"level": "required",
|
|
193
|
+
"status": "mismatch",
|
|
194
|
+
"expected_value": "false",
|
|
195
|
+
"mode": "production"
|
|
196
|
+
}
|
|
197
|
+
```
|
|
89
198
|
|
|
90
199
|
### Predicates: `if:` and `unless:`
|
|
91
200
|
|
|
@@ -184,6 +293,41 @@ Audit rows for entries declared inside a `group` block carry a `group` key:
|
|
|
184
293
|
|
|
185
294
|
Entries declared outside any `group` block omit the `group` key entirely.
|
|
186
295
|
|
|
296
|
+
### Top-level status
|
|
297
|
+
|
|
298
|
+
Since 0.4.1 the response carries a `status` alongside the audit, so a caller
|
|
299
|
+
can gate on one field instead of re-implementing the roll-up:
|
|
300
|
+
|
|
301
|
+
```json
|
|
302
|
+
{ "mode": "production", "status": "incomplete", "audit": [ ... ] }
|
|
303
|
+
```
|
|
304
|
+
|
|
305
|
+
`incomplete` means at least one row is a **violation** — `missing`,
|
|
306
|
+
`forbidden`, or `mismatch`. `should_set` is advisory and never affects it.
|
|
307
|
+
|
|
308
|
+
The level is deliberately not consulted for `mismatch`: a `recommended` var
|
|
309
|
+
declared with an `expected_value:` that does not hold is a failed assertion,
|
|
310
|
+
not advice.
|
|
311
|
+
|
|
312
|
+
**The endpoint still returns 200 either way.** Callers that already gate on
|
|
313
|
+
`status == "incomplete"` pick up the `forbidden` and `mismatch` assertions for
|
|
314
|
+
free — which is why they joined this roll-up rather than getting a verdict of
|
|
315
|
+
their own.
|
|
316
|
+
|
|
317
|
+
### Surfacing the audit on a health tier
|
|
318
|
+
|
|
319
|
+
`/diagnostics/env` is authed and polled by nobody, so config drift declared in
|
|
320
|
+
the spec is invisible until someone looks. Register the opt-in `EnvSpecAudit`
|
|
321
|
+
check to put the same verdict on a health tier:
|
|
322
|
+
|
|
323
|
+
```ruby
|
|
324
|
+
c.register_check :env_spec, StandardHealth::Checks::EnvSpecAudit
|
|
325
|
+
```
|
|
326
|
+
|
|
327
|
+
It fails on `:missing` / `:forbidden` / `:mismatch` rows and reports the
|
|
328
|
+
offending **names** (never values). Non-critical by default — see
|
|
329
|
+
[Opt-in checks](#opt-in-checks).
|
|
330
|
+
|
|
187
331
|
### Backward compatibility
|
|
188
332
|
|
|
189
333
|
All v0.2.0 specs continue to produce identical audit output in v0.3.0. The new fields (`description`, `consumed_by`, `group`, `reason`) appear only when the corresponding feature is used; the new `:not_applicable` status only appears when a predicate suppresses an entry.
|
|
@@ -206,6 +350,81 @@ end
|
|
|
206
350
|
|
|
207
351
|
`with_timing` captures `latency_ms` on success and converts any `StandardError` into `{ status: :fail, error: <message> }`.
|
|
208
352
|
|
|
353
|
+
**A check must never raise.** `Aggregator` rescues `StandardError` per check, so a buggy check degrades to `:fail` rather than 500ing the endpoint — but don't rely on that as the only line of defence. Route fallible work through `with_timing`.
|
|
354
|
+
|
|
355
|
+
## Opt-in checks
|
|
356
|
+
|
|
357
|
+
Every check in this gem must be registered explicitly; **none are registered
|
|
358
|
+
automatically**, including the ones below. That is a deliberate constraint on
|
|
359
|
+
the gem: auto-registering a check turns it on for every host on a `bundle
|
|
360
|
+
update`, and a host that has never had SolidCable installed — or whose env spec
|
|
361
|
+
has a pre-existing violation — would go from green to yellow across its estate
|
|
362
|
+
for a check nobody asked for. New checks are additive only when they are opt-in.
|
|
363
|
+
|
|
364
|
+
| Check | Probes | `critical:` default |
|
|
365
|
+
|---|---|---|
|
|
366
|
+
| `Checks::ActiveRecord` | `SELECT 1` on the primary connection | `true` |
|
|
367
|
+
| `Checks::SolidQueue` | SolidQueue's tables | `true` |
|
|
368
|
+
| `Checks::SolidCache` | read-only `Rails.cache` probe | `false` |
|
|
369
|
+
| `Checks::SolidCable` | `solid_cable_messages` store | `false` |
|
|
370
|
+
| `Checks::EnvSpecAudit` | the configured `env_spec` | `false` |
|
|
371
|
+
|
|
372
|
+
The `critical:` default is only a default — `register_check` overrides it, and
|
|
373
|
+
what belongs on the rotation gate is a per-app decision.
|
|
374
|
+
|
|
375
|
+
```ruby
|
|
376
|
+
StandardHealth.configure do |c|
|
|
377
|
+
c.register_check :solid_cable, StandardHealth::Checks::SolidCable
|
|
378
|
+
c.register_check :env_spec, StandardHealth::Checks::EnvSpecAudit
|
|
379
|
+
end
|
|
380
|
+
```
|
|
381
|
+
|
|
382
|
+
### `Checks::SolidCable`
|
|
383
|
+
|
|
384
|
+
Bounded read against `solid_cable_messages`, confirming both that the cable
|
|
385
|
+
schema is migrated and that its connection is up. Uses
|
|
386
|
+
`SolidCable::Record.connection` when SolidCable is pointed at a separate
|
|
387
|
+
database, falling back to the primary connection when it isn't.
|
|
388
|
+
|
|
389
|
+
Cable is a **degradable feature dependency** — prefer this on the aggregate
|
|
390
|
+
tier. A broken cable store should mark the app `degraded`, never de-rotate it.
|
|
391
|
+
|
|
392
|
+
### `Checks::EnvSpecAudit`
|
|
393
|
+
|
|
394
|
+
Runs the configured `env_spec` and fails on the violation statuses
|
|
395
|
+
(`:missing`, `:forbidden`, `:mismatch`), reporting the offending variable
|
|
396
|
+
**names** grouped by status. Values never appear.
|
|
397
|
+
|
|
398
|
+
It reads the spec directly, so there is one declaration and no second list to
|
|
399
|
+
keep in sync — which is the whole reason it exists. It skips `consumed_by`
|
|
400
|
+
resolution (that does file IO per entry, fine for an on-demand doctor endpoint
|
|
401
|
+
and not fine on a tier polled every few seconds), and it is `:ok` when no
|
|
402
|
+
`env_spec` is configured.
|
|
403
|
+
|
|
404
|
+
Non-critical by default, and that default is load-bearing: config drift is
|
|
405
|
+
*visibility*, not a rotation signal. An instance with a stale toggle set is
|
|
406
|
+
still serving traffic correctly, and de-rotating it converts a warning into an
|
|
407
|
+
outage. Registering it `critical: true` asserts "this app must not serve at all
|
|
408
|
+
with a bad env" — a real but rare posture. Know which you want.
|
|
409
|
+
|
|
410
|
+
The aggregator instantiates checks with `name:`/`critical:` only, so to narrow
|
|
411
|
+
what counts as a failure, subclass:
|
|
412
|
+
|
|
413
|
+
```ruby
|
|
414
|
+
class ForbiddenTogglesOnly < StandardHealth::Checks::EnvSpecAudit
|
|
415
|
+
def initialize(name: :forbidden_toggles, critical: false)
|
|
416
|
+
super(name: name, critical: critical, fail_on: %i[forbidden])
|
|
417
|
+
end
|
|
418
|
+
end
|
|
419
|
+
```
|
|
420
|
+
|
|
421
|
+
Note the failure message is subject to [redaction](#failure-detail-is-redacted)
|
|
422
|
+
on `/ready` like any other check; the detail reaches you through logs and
|
|
423
|
+
Sentry. The row carries a stable `error_class` of
|
|
424
|
+
`StandardHealth::EnvSpecViolation`, so the redacted body still groups on
|
|
425
|
+
`error_code: "standard_health_env_spec_violation"` rather than a useless
|
|
426
|
+
`standard_error`.
|
|
427
|
+
|
|
209
428
|
## Auth
|
|
210
429
|
|
|
211
430
|
`/alive` and `/ready` are typically left open for orchestrator probes. `/diagnostics/env` enumerates which env vars are missing — that's potentially sensitive, so the host app is responsible for protecting it.
|
|
@@ -283,6 +502,130 @@ When `diagnostics_parent_controller` is unset, `DiagnosticsController` falls bac
|
|
|
283
502
|
|
|
284
503
|
The orchestrator should pull the instance out of rotation only on 503; degraded means "still serving, page someone."
|
|
285
504
|
|
|
505
|
+
`skipped` also exists, for a check the total budget never reached (see
|
|
506
|
+
Timeouts). A skip floors the roll-up at `degraded` and **never** produces
|
|
507
|
+
`unavailable`, even for a critical check.
|
|
508
|
+
|
|
509
|
+
## Failure detail is redacted
|
|
510
|
+
|
|
511
|
+
Since 0.4.1, a failing check reports the exception **class** rather than its
|
|
512
|
+
message:
|
|
513
|
+
|
|
514
|
+
```json
|
|
515
|
+
{ "name": "database", "critical": true, "status": "fail",
|
|
516
|
+
"error_class": "PG::ConnectionBad", "error_code": "pg_connection_bad" }
|
|
517
|
+
```
|
|
518
|
+
|
|
519
|
+
`/ready` is unauthenticated by design — probes carry no credentials — and a
|
|
520
|
+
raw driver message will happily tell an anonymous caller the database host,
|
|
521
|
+
port and username. The full message still reaches your logs and Sentry through
|
|
522
|
+
the instrumentation below; it just isn't in the public body.
|
|
523
|
+
|
|
524
|
+
```ruby
|
|
525
|
+
c.expose_check_errors = true # restore pre-0.4.1 verbose bodies
|
|
526
|
+
c.detail_token = ENV["HEALTH_DETAIL_TOKEN"] # X-Health-Token unlocks detail
|
|
527
|
+
```
|
|
528
|
+
|
|
529
|
+
`status`, `name`, `critical`, `latency_ms` and `generated_at` are unchanged.
|
|
530
|
+
|
|
531
|
+
## Instrumentation
|
|
532
|
+
|
|
533
|
+
The gem emits events on whichever bus is live — `Rails.event` on Rails 8.1+,
|
|
534
|
+
`ActiveSupport::Notifications` otherwise:
|
|
535
|
+
|
|
536
|
+
| Event | Payload |
|
|
537
|
+
|---|---|
|
|
538
|
+
| `standard_health.check.completed` | `name`, `critical`, `status`, `latency_ms`, `error_class`, `error_message` |
|
|
539
|
+
| `standard_health.check.timed_out` | `name`, `critical`, `timeout_s` |
|
|
540
|
+
| `standard_health.ready.evaluated` | `status`, `duration_ms`, `failed[]` |
|
|
541
|
+
|
|
542
|
+
Three subscribers are registered automatically. Their noise profiles differ on
|
|
543
|
+
purpose, because health events are **polls**, not state transitions — a 10s
|
|
544
|
+
probe period means ~6 evaluations/minute/instance:
|
|
545
|
+
|
|
546
|
+
- **Logger** — silent while healthy. `warn` on degraded, `error` on
|
|
547
|
+
unavailable. A line per evaluation would be ~8,640/day/instance of
|
|
548
|
+
"everything is fine".
|
|
549
|
+
- **Sentry** — only on status **change**, plus a 60s repeat floor for a
|
|
550
|
+
sustained bad state, plus one `info` on recovery. Capturing every non-ok
|
|
551
|
+
poll would turn a five-minute outage into ~30 duplicate issues. Sentry is a
|
|
552
|
+
soft dependency; no gemspec entry.
|
|
553
|
+
- **Metrics** — every poll, deliberately. Counters plus latency
|
|
554
|
+
distributions are what make `latency_ms` chartable.
|
|
555
|
+
|
|
556
|
+
```ruby
|
|
557
|
+
StandardHealth.configure do |c|
|
|
558
|
+
c.instrumentation_enabled = true # default
|
|
559
|
+
c.logger = Rails.logger # default: Rails.logger
|
|
560
|
+
c.sentry_enabled = true # default
|
|
561
|
+
c.metric_prefix = "health" # default
|
|
562
|
+
|
|
563
|
+
c.add_notifier(MyNotifier.new) # must respond to call(event_name, payload)
|
|
564
|
+
end
|
|
565
|
+
```
|
|
566
|
+
|
|
567
|
+
## Timeouts
|
|
568
|
+
|
|
569
|
+
**Off by default.** Both settings are `nil`, which is byte-identical to 0.4.0.
|
|
570
|
+
|
|
571
|
+
```ruby
|
|
572
|
+
c.default_check_timeout = 2.0 # seconds, per check
|
|
573
|
+
c.total_check_budget = 5.0 # checked BEFORE each check starts
|
|
574
|
+
c.register_check :db, MyCheck, critical: true, timeout: 1.0 # per-check override
|
|
575
|
+
```
|
|
576
|
+
|
|
577
|
+
**`total_check_budget` bounds how many checks RUN, not how long the probe
|
|
578
|
+
takes.** It is evaluated before each check starts, not during one. With a 1s
|
|
579
|
+
budget and a first check that blocks for 30s, `/ready` still takes 30s — only
|
|
580
|
+
the checks *after* it are skipped.
|
|
581
|
+
|
|
582
|
+
Clamping each check to the remaining budget would close that gap, and is
|
|
583
|
+
deliberately not done: it would apply `Timeout.timeout` to checks whose author
|
|
584
|
+
never asked for one, and that mechanism raises into the thread at an arbitrary
|
|
585
|
+
point (see below). Setting a budget must not silently opt you into that. To
|
|
586
|
+
bound wall-clock, put a `timeout:` on the checks that can safely take one.
|
|
587
|
+
|
|
588
|
+
Enabling them is a semantic change — a check that has always been
|
|
589
|
+
slow-but-fine starts reporting `:fail`, and for a critical check that pulls
|
|
590
|
+
the instance out of rotation. Pick values from observed `latency_ms` rather
|
|
591
|
+
than intuition; that is what the `check.completed` events are for.
|
|
592
|
+
|
|
593
|
+
Checks the total budget never reaches report `:skipped`, never silently `:ok`.
|
|
594
|
+
A skip alone floors the roll-up at `degraded` — otherwise a slow *non-critical*
|
|
595
|
+
check could exhaust the budget, leave the database check unrun, and pull a
|
|
596
|
+
healthy instance out of rotation.
|
|
597
|
+
|
|
598
|
+
Checks run **sequentially**. They are not parallelised on purpose: running
|
|
599
|
+
them in threads would mean connection checkouts from the health path, and a
|
|
600
|
+
health endpoint that can exhaust the connection pool is a worse problem than
|
|
601
|
+
the one it was added to solve.
|
|
602
|
+
|
|
603
|
+
### ⚠ Know what `Timeout.timeout` actually does before enabling this
|
|
604
|
+
|
|
605
|
+
Ruby's `Timeout.timeout` interrupts by raising **into the running thread at an
|
|
606
|
+
arbitrary point**. It cannot wait for a safe boundary. If a timeout fires
|
|
607
|
+
while a check is mid-way through a non-atomic operation — say a connection
|
|
608
|
+
checkout in the `ActiveRecord` check — the connection can be returned to the
|
|
609
|
+
pool in a broken state. You would have traded a slow check for a corrupted
|
|
610
|
+
pool, on the health path, during an incident.
|
|
611
|
+
|
|
612
|
+
The dedicated `CheckTimeout` class solves the *mislabelling* problem (a host's
|
|
613
|
+
own `Timeout::Error` is never mistaken for ours). **It does not solve this
|
|
614
|
+
one.** Nothing can, at this layer.
|
|
615
|
+
|
|
616
|
+
So, before turning timeouts on:
|
|
617
|
+
|
|
618
|
+
- Prefer a driver-level timeout where one exists — `connect_timeout` /
|
|
619
|
+
`statement_timeout` on the database, a client timeout on an HTTP dependency.
|
|
620
|
+
Those abort at a safe point because the driver owns the operation.
|
|
621
|
+
- Reserve `timeout:` here for checks whose `run` is genuinely interruptible —
|
|
622
|
+
typically your own custom checks — rather than the built-in datastore ones.
|
|
623
|
+
- Set the value comfortably above observed p99 (see the `latency_ms` in
|
|
624
|
+
`check.completed`), so it fires on a hang rather than on a slow day.
|
|
625
|
+
|
|
626
|
+
This is why the defaults ship as `nil` and why choosing them is deferred: the
|
|
627
|
+
right answer is usually a driver timeout, not this.
|
|
628
|
+
|
|
286
629
|
## License
|
|
287
630
|
|
|
288
631
|
MIT.
|
|
@@ -25,9 +25,36 @@ module StandardHealth
|
|
|
25
25
|
|
|
26
26
|
render json: {
|
|
27
27
|
mode: mode,
|
|
28
|
+
status: audit_status(audit),
|
|
28
29
|
audit: audit,
|
|
29
30
|
generated_at: Time.now.utc.iso8601
|
|
30
31
|
}
|
|
31
32
|
end
|
|
33
|
+
|
|
34
|
+
private
|
|
35
|
+
|
|
36
|
+
# Top-level verdict so a caller can gate on ONE field instead of
|
|
37
|
+
# re-implementing the roll-up over `audit`.
|
|
38
|
+
#
|
|
39
|
+
# :ok — no violations
|
|
40
|
+
# :incomplete — at least one row is `:missing`, `:forbidden`, or
|
|
41
|
+
# `:mismatch` (`EnvSpec::VIOLATION_STATUSES`)
|
|
42
|
+
#
|
|
43
|
+
# The `forbidden`/`mismatch` statuses join the roll-up rather than
|
|
44
|
+
# getting a verdict of their own, so callers that already gate on
|
|
45
|
+
# `status == "incomplete"` pick up the new assertions for free. Note the
|
|
46
|
+
# level is deliberately NOT consulted: a `recommended` var declared with
|
|
47
|
+
# an `expected_value:` that does not hold is a failed assertion, not
|
|
48
|
+
# advice. The advisory status stays `:should_set`, which is not a
|
|
49
|
+
# violation.
|
|
50
|
+
#
|
|
51
|
+
# The endpoint still returns 200 either way, so nothing that asserts on
|
|
52
|
+
# the status code breaks.
|
|
53
|
+
def audit_status(audit)
|
|
54
|
+
violated = Array(audit).any? do |row|
|
|
55
|
+
EnvSpec::VIOLATION_STATUSES.include?(row[:status])
|
|
56
|
+
end
|
|
57
|
+
violated ? :incomplete : :ok
|
|
58
|
+
end
|
|
32
59
|
end
|
|
33
60
|
end
|