standard_health 0.3.0 → 0.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +118 -0
- data/{MIT-LICENSE → LICENSE} +8 -6
- data/README.md +141 -0
- data/app/controllers/standard_health/diagnostics_controller.rb +23 -1
- data/app/controllers/standard_health/health_controller.rb +29 -1
- data/lib/standard_health/aggregator.rb +150 -11
- data/lib/standard_health/check.rb +6 -1
- data/lib/standard_health/configuration.rb +113 -3
- data/lib/standard_health/engine.rb +13 -0
- data/lib/standard_health/env_spec.rb +70 -5
- data/lib/standard_health/event_emitter.rb +41 -0
- data/lib/standard_health/notifiers/logger.rb +88 -0
- data/lib/standard_health/notifiers/metrics.rb +88 -0
- data/lib/standard_health/notifiers/sentry.rb +153 -0
- data/lib/standard_health/redactor.rb +74 -0
- data/lib/standard_health/subscribers.rb +103 -0
- data/lib/standard_health/version.rb +1 -1
- data/lib/standard_health.rb +13 -0
- metadata +8 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 68d0908117b9ae70545445465e84f94b4c6ccc5ebecb2aa6ab05b3aafa5ba9f7
|
|
4
|
+
data.tar.gz: 52c3ffe85837e265283011db180de726957df2eecd09b971efc369e879ae4ec3
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 379b80dbdf211c4c1a3e7639ec6cc926219949f9c352fbadddfada54bdea53f20df6ab52d5672baa0974bb6b2952b5f5e67b9069191d980baff1de0d3c77f972
|
|
7
|
+
data.tar.gz: feaba7a044e8e63e4dc60ad4d010edfbe197691596ea2520632c4d590f6e52907d9275a5825dea8deff42c716517cb569b54f2824460820327f5003e9e5b5157
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,124 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.4.1] - 2026-07-29
|
|
11
|
+
|
|
12
|
+
Observability release. Until now the gem emitted **nothing** — no events, no
|
|
13
|
+
logs, no metrics. It computed `latency_ms` for every check and discarded it
|
|
14
|
+
into the response body. A failing check was invisible unless somebody happened
|
|
15
|
+
to call `/ready` at the right moment, so there was no history, no alerting
|
|
16
|
+
hook, and no way to chart check latency.
|
|
17
|
+
|
|
18
|
+
Everything here is **additive**: no status code, roll-up rule, or response
|
|
19
|
+
field changes meaning, so this is a pure `bundle update` for consumers on
|
|
20
|
+
`~> 0.4`.
|
|
21
|
+
|
|
22
|
+
### Added
|
|
23
|
+
|
|
24
|
+
- **Instrumentation.** Three events over whichever bus is live (`Rails.event`
|
|
25
|
+
on Rails 8.1+, `ActiveSupport::Notifications` otherwise), matching
|
|
26
|
+
`standard_circuit`'s `EventEmitter` idiom:
|
|
27
|
+
- `standard_health.check.completed` — `name`, `critical`, `status`,
|
|
28
|
+
`latency_ms`, `error_class`, `error_message`
|
|
29
|
+
- `standard_health.check.timed_out` — `name`, `critical`, `timeout_s`
|
|
30
|
+
- `standard_health.ready.evaluated` — `status`, `duration_ms`, `failed[]`
|
|
31
|
+
- **Three built-in subscribers**, registered by a new engine initializer
|
|
32
|
+
(`after: :load_config_initializers`, so host config is final):
|
|
33
|
+
- `Notifiers::Logger` — **silent while healthy.** `warn` on degraded,
|
|
34
|
+
`error` on unavailable. Health events are polls (~6/min/instance), not
|
|
35
|
+
transitions; a line per evaluation is ~8,640/day/instance of "everything
|
|
36
|
+
is fine".
|
|
37
|
+
- `Notifiers::Sentry` — **transition-only**, with a 60s repeat floor and
|
|
38
|
+
one `info` on recovery. Capturing every non-ok poll would turn a
|
|
39
|
+
five-minute outage into ~30 duplicate issues. Mutex-guarded (Puma is
|
|
40
|
+
threaded). Sentry stays a soft dependency.
|
|
41
|
+
- `Notifiers::Metrics` — per-poll counters and latency distributions. This
|
|
42
|
+
fires on every evaluation on purpose: it is what makes `latency_ms`
|
|
43
|
+
chartable.
|
|
44
|
+
- `config.add_notifier` for host-supplied `call(event_name, payload)`
|
|
45
|
+
subscribers, validated at add time.
|
|
46
|
+
- Config: `instrumentation_enabled`, `logger`, `sentry_enabled`,
|
|
47
|
+
`metric_prefix`.
|
|
48
|
+
- **Per-check timeout machinery** — `default_check_timeout`,
|
|
49
|
+
`total_check_budget`, and a `timeout:` option on `register_check`. Uses a
|
|
50
|
+
dedicated `StandardHealth::CheckTimeout` rather than bare `Timeout::Error`,
|
|
51
|
+
so a timeout the host raised for its own reasons is never mislabelled as
|
|
52
|
+
ours. **All default to `nil` (OFF)** — see below.
|
|
53
|
+
- `status` on `/diagnostics/env`: `:ok` or `:incomplete` (a `required` var is
|
|
54
|
+
missing). **Additive — the endpoint still returns 200 either way.**
|
|
55
|
+
|
|
56
|
+
### Fixed
|
|
57
|
+
|
|
58
|
+
- `Check#with_timing` now records `error_class` alongside the message. All
|
|
59
|
+
three built-in checks route through it, so without this every real driver
|
|
60
|
+
failure reached the aggregator with no class and the redacted body fell back
|
|
61
|
+
to a generic `StandardError` — exactly the useless label redaction exists to
|
|
62
|
+
replace.
|
|
63
|
+
- Failure detail (`error_class` + `error_message`) now rides on
|
|
64
|
+
`ready.evaluated`, not only on `check.completed`. Both built-in subscribers
|
|
65
|
+
are driven by `ready.evaluated` (deliberately — it is the transition-gated
|
|
66
|
+
event), so the message would otherwise have been deleted from the response
|
|
67
|
+
without ever reaching logs or Sentry.
|
|
68
|
+
|
|
69
|
+
### Security
|
|
70
|
+
|
|
71
|
+
- **`/ready` no longer returns raw exception messages.** The endpoint is
|
|
72
|
+
unauthenticated by design, and a driver error will happily tell an anonymous
|
|
73
|
+
caller the database host, port and username — during exactly the incident
|
|
74
|
+
you least want to be leaking. Failing rows now carry `error_class` and a
|
|
75
|
+
stable `error_code` instead. The full message is not lost: it goes to logs
|
|
76
|
+
and Sentry via the instrumentation above, which is why redaction ships in
|
|
77
|
+
the same release rather than separately.
|
|
78
|
+
- `config.expose_check_errors = true` restores the previous bodies.
|
|
79
|
+
- `config.detail_token` + an `X-Health-Token` header is a break-glass path
|
|
80
|
+
for on-call, compared in constant time.
|
|
81
|
+
- `status`, `name`, `critical`, `latency_ms` and `generated_at` are
|
|
82
|
+
unchanged, so dashboards and existing specs keep working.
|
|
83
|
+
|
|
84
|
+
### Notes on what is deliberately NOT on
|
|
85
|
+
|
|
86
|
+
Timeouts and the total budget default to **off**, giving byte-identical
|
|
87
|
+
behaviour to 0.4.0. Turning them on is a semantic change: a check that has
|
|
88
|
+
always been slow-but-fine starts reporting `:fail`, and for a critical check
|
|
89
|
+
that pulls the instance out of rotation. Shipping that in a patch, to five
|
|
90
|
+
apps, on a `bundle update`, is how you cause the outage you were preventing.
|
|
91
|
+
|
|
92
|
+
The machinery ships now so apps can opt in per check, and so the events above
|
|
93
|
+
can reveal the real p99 latencies. Defaults get chosen from that data in
|
|
94
|
+
0.5.0, which requires an explicit Gemfile edit.
|
|
95
|
+
|
|
96
|
+
One semantic guard is already in place for when they are enabled: a check
|
|
97
|
+
skipped because the budget ran out reports `:skipped` and floors the roll-up
|
|
98
|
+
at `:degraded` — **never `:unavailable`**, even when the skipped check is
|
|
99
|
+
critical. Otherwise a slow *non-critical* check could exhaust the budget,
|
|
100
|
+
leave the database check unrun, and pull a healthy instance out of rotation.
|
|
101
|
+
|
|
102
|
+
`total_check_budget` bounds **how many checks run**, not how long the probe
|
|
103
|
+
takes — it is evaluated before each check starts, not during one. Clamping
|
|
104
|
+
each check to the remaining budget would close that gap but would apply
|
|
105
|
+
`Timeout.timeout` to checks whose author never asked for one, so setting a
|
|
106
|
+
budget deliberately does not opt you into that. Asserted by spec, not just
|
|
107
|
+
documented.
|
|
108
|
+
|
|
109
|
+
A second reason to leave them off: `Timeout.timeout` raises into the running
|
|
110
|
+
thread at an arbitrary point, so firing one mid-connection-checkout can return
|
|
111
|
+
a broken connection to the pool. The dedicated `CheckTimeout` fixes
|
|
112
|
+
mislabelling, not this. The README now carries the full caveat — the right
|
|
113
|
+
answer for datastore checks is usually a driver-level timeout
|
|
114
|
+
(`connect_timeout` / `statement_timeout`), not this setting.
|
|
115
|
+
|
|
116
|
+
## [0.4.0] - 2026-05-05
|
|
117
|
+
|
|
118
|
+
### Added
|
|
119
|
+
|
|
120
|
+
- Consumer-presence detection for `consumed_by:` paths. `audit()` accepts an optional `root:` keyword (host-app root). When given, each entry whose `consumed_by:` is set is checked against the host app's tree, producing a new `consumer:` field on the audit row: `:present` (file exists and references the var via `ENV[...]` or `ENV.fetch(...)`), `:file_missing` (path missing on disk), or `:not_referenced` (file exists but no `ENV` reference). Catches renamed/deleted consumer files, `consumed_by:` typos, and vars declared in env-spec but never actually `ENV.fetch`'d.
|
|
121
|
+
- `DiagnosticsController#env` now passes `Rails.root` automatically, so host apps get the new `consumer:` field with no host-side change.
|
|
122
|
+
- Deprecation metadata on `required` / `recommended`: `deprecated: true`, `sunset_on:` (target removal date), `replacement:` (what to use instead). Surfaced verbatim in audit rows. Lets vars be staged for removal with audit trail.
|
|
123
|
+
|
|
124
|
+
### Changed
|
|
125
|
+
|
|
126
|
+
- `Entry` struct extended with `deprecated`, `sunset_on`, `replacement`. Backward-compatible — every 0.3.0 spec produces identical audit output when the new opts aren't used and `root:` isn't passed.
|
|
127
|
+
|
|
10
128
|
## [0.3.0] - 2026-04-29
|
|
11
129
|
|
|
12
130
|
### Added
|
data/{MIT-LICENSE → LICENSE}
RENAMED
|
@@ -1,4 +1,6 @@
|
|
|
1
|
-
|
|
1
|
+
The MIT License (MIT)
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Rarebit One
|
|
2
4
|
|
|
3
5
|
Permission is hereby granted, free of charge, to any person obtaining
|
|
4
6
|
a copy of this software and associated documentation files (the
|
|
@@ -13,8 +15,8 @@ included in all copies or substantial portions of the Software.
|
|
|
13
15
|
|
|
14
16
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
|
15
17
|
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
|
16
|
-
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
|
17
|
-
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
|
18
|
-
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
|
19
|
-
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
|
20
|
-
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
|
18
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
|
19
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
|
20
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
|
21
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
|
22
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
CHANGED
|
@@ -184,6 +184,23 @@ Audit rows for entries declared inside a `group` block carry a `group` key:
|
|
|
184
184
|
|
|
185
185
|
Entries declared outside any `group` block omit the `group` key entirely.
|
|
186
186
|
|
|
187
|
+
### Top-level status
|
|
188
|
+
|
|
189
|
+
Since 0.4.1 the response carries a `status` alongside the audit, so a caller
|
|
190
|
+
can gate on one field instead of re-implementing the roll-up:
|
|
191
|
+
|
|
192
|
+
```json
|
|
193
|
+
{ "mode": "production", "status": "incomplete", "audit": [ ... ] }
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
`incomplete` means at least one `required` var is missing; `recommended` is
|
|
197
|
+
advisory and never affects it.
|
|
198
|
+
|
|
199
|
+
**The endpoint still returns 200 either way in 0.4.1.** That is deliberate: it
|
|
200
|
+
gives monitors a release to migrate onto the field before 0.5.0 makes
|
|
201
|
+
`incomplete` a 503. Flipping the status code without that window would
|
|
202
|
+
silently redden every existing caller.
|
|
203
|
+
|
|
187
204
|
### Backward compatibility
|
|
188
205
|
|
|
189
206
|
All v0.2.0 specs continue to produce identical audit output in v0.3.0. The new fields (`description`, `consumed_by`, `group`, `reason`) appear only when the corresponding feature is used; the new `:not_applicable` status only appears when a predicate suppresses an entry.
|
|
@@ -283,6 +300,130 @@ When `diagnostics_parent_controller` is unset, `DiagnosticsController` falls bac
|
|
|
283
300
|
|
|
284
301
|
The orchestrator should pull the instance out of rotation only on 503; degraded means "still serving, page someone."
|
|
285
302
|
|
|
303
|
+
`skipped` also exists, for a check the total budget never reached (see
|
|
304
|
+
Timeouts). A skip floors the roll-up at `degraded` and **never** produces
|
|
305
|
+
`unavailable`, even for a critical check.
|
|
306
|
+
|
|
307
|
+
## Failure detail is redacted
|
|
308
|
+
|
|
309
|
+
Since 0.4.1, a failing check reports the exception **class** rather than its
|
|
310
|
+
message:
|
|
311
|
+
|
|
312
|
+
```json
|
|
313
|
+
{ "name": "database", "critical": true, "status": "fail",
|
|
314
|
+
"error_class": "PG::ConnectionBad", "error_code": "pg_connection_bad" }
|
|
315
|
+
```
|
|
316
|
+
|
|
317
|
+
`/ready` is unauthenticated by design — probes carry no credentials — and a
|
|
318
|
+
raw driver message will happily tell an anonymous caller the database host,
|
|
319
|
+
port and username. The full message still reaches your logs and Sentry through
|
|
320
|
+
the instrumentation below; it just isn't in the public body.
|
|
321
|
+
|
|
322
|
+
```ruby
|
|
323
|
+
c.expose_check_errors = true # restore pre-0.4.1 verbose bodies
|
|
324
|
+
c.detail_token = ENV["HEALTH_DETAIL_TOKEN"] # X-Health-Token unlocks detail
|
|
325
|
+
```
|
|
326
|
+
|
|
327
|
+
`status`, `name`, `critical`, `latency_ms` and `generated_at` are unchanged.
|
|
328
|
+
|
|
329
|
+
## Instrumentation
|
|
330
|
+
|
|
331
|
+
The gem emits events on whichever bus is live — `Rails.event` on Rails 8.1+,
|
|
332
|
+
`ActiveSupport::Notifications` otherwise:
|
|
333
|
+
|
|
334
|
+
| Event | Payload |
|
|
335
|
+
|---|---|
|
|
336
|
+
| `standard_health.check.completed` | `name`, `critical`, `status`, `latency_ms`, `error_class`, `error_message` |
|
|
337
|
+
| `standard_health.check.timed_out` | `name`, `critical`, `timeout_s` |
|
|
338
|
+
| `standard_health.ready.evaluated` | `status`, `duration_ms`, `failed[]` |
|
|
339
|
+
|
|
340
|
+
Three subscribers are registered automatically. Their noise profiles differ on
|
|
341
|
+
purpose, because health events are **polls**, not state transitions — a 10s
|
|
342
|
+
probe period means ~6 evaluations/minute/instance:
|
|
343
|
+
|
|
344
|
+
- **Logger** — silent while healthy. `warn` on degraded, `error` on
|
|
345
|
+
unavailable. A line per evaluation would be ~8,640/day/instance of
|
|
346
|
+
"everything is fine".
|
|
347
|
+
- **Sentry** — only on status **change**, plus a 60s repeat floor for a
|
|
348
|
+
sustained bad state, plus one `info` on recovery. Capturing every non-ok
|
|
349
|
+
poll would turn a five-minute outage into ~30 duplicate issues. Sentry is a
|
|
350
|
+
soft dependency; no gemspec entry.
|
|
351
|
+
- **Metrics** — every poll, deliberately. Counters plus latency
|
|
352
|
+
distributions are what make `latency_ms` chartable.
|
|
353
|
+
|
|
354
|
+
```ruby
|
|
355
|
+
StandardHealth.configure do |c|
|
|
356
|
+
c.instrumentation_enabled = true # default
|
|
357
|
+
c.logger = Rails.logger # default: Rails.logger
|
|
358
|
+
c.sentry_enabled = true # default
|
|
359
|
+
c.metric_prefix = "health" # default
|
|
360
|
+
|
|
361
|
+
c.add_notifier(MyNotifier.new) # must respond to call(event_name, payload)
|
|
362
|
+
end
|
|
363
|
+
```
|
|
364
|
+
|
|
365
|
+
## Timeouts
|
|
366
|
+
|
|
367
|
+
**Off by default.** Both settings are `nil`, which is byte-identical to 0.4.0.
|
|
368
|
+
|
|
369
|
+
```ruby
|
|
370
|
+
c.default_check_timeout = 2.0 # seconds, per check
|
|
371
|
+
c.total_check_budget = 5.0 # checked BEFORE each check starts
|
|
372
|
+
c.register_check :db, MyCheck, critical: true, timeout: 1.0 # per-check override
|
|
373
|
+
```
|
|
374
|
+
|
|
375
|
+
**`total_check_budget` bounds how many checks RUN, not how long the probe
|
|
376
|
+
takes.** It is evaluated before each check starts, not during one. With a 1s
|
|
377
|
+
budget and a first check that blocks for 30s, `/ready` still takes 30s — only
|
|
378
|
+
the checks *after* it are skipped.
|
|
379
|
+
|
|
380
|
+
Clamping each check to the remaining budget would close that gap, and is
|
|
381
|
+
deliberately not done: it would apply `Timeout.timeout` to checks whose author
|
|
382
|
+
never asked for one, and that mechanism raises into the thread at an arbitrary
|
|
383
|
+
point (see below). Setting a budget must not silently opt you into that. To
|
|
384
|
+
bound wall-clock, put a `timeout:` on the checks that can safely take one.
|
|
385
|
+
|
|
386
|
+
Enabling them is a semantic change — a check that has always been
|
|
387
|
+
slow-but-fine starts reporting `:fail`, and for a critical check that pulls
|
|
388
|
+
the instance out of rotation. Pick values from observed `latency_ms` rather
|
|
389
|
+
than intuition; that is what the `check.completed` events are for.
|
|
390
|
+
|
|
391
|
+
Checks the total budget never reaches report `:skipped`, never silently `:ok`.
|
|
392
|
+
A skip alone floors the roll-up at `degraded` — otherwise a slow *non-critical*
|
|
393
|
+
check could exhaust the budget, leave the database check unrun, and pull a
|
|
394
|
+
healthy instance out of rotation.
|
|
395
|
+
|
|
396
|
+
Checks run **sequentially**. They are not parallelised on purpose: running
|
|
397
|
+
them in threads would mean connection checkouts from the health path, and a
|
|
398
|
+
health endpoint that can exhaust the connection pool is a worse problem than
|
|
399
|
+
the one it was added to solve.
|
|
400
|
+
|
|
401
|
+
### ⚠ Know what `Timeout.timeout` actually does before enabling this
|
|
402
|
+
|
|
403
|
+
Ruby's `Timeout.timeout` interrupts by raising **into the running thread at an
|
|
404
|
+
arbitrary point**. It cannot wait for a safe boundary. If a timeout fires
|
|
405
|
+
while a check is mid-way through a non-atomic operation — say a connection
|
|
406
|
+
checkout in the `ActiveRecord` check — the connection can be returned to the
|
|
407
|
+
pool in a broken state. You would have traded a slow check for a corrupted
|
|
408
|
+
pool, on the health path, during an incident.
|
|
409
|
+
|
|
410
|
+
The dedicated `CheckTimeout` class solves the *mislabelling* problem (a host's
|
|
411
|
+
own `Timeout::Error` is never mistaken for ours). **It does not solve this
|
|
412
|
+
one.** Nothing can, at this layer.
|
|
413
|
+
|
|
414
|
+
So, before turning timeouts on:
|
|
415
|
+
|
|
416
|
+
- Prefer a driver-level timeout where one exists — `connect_timeout` /
|
|
417
|
+
`statement_timeout` on the database, a client timeout on an HTTP dependency.
|
|
418
|
+
Those abort at a safe point because the driver owns the operation.
|
|
419
|
+
- Reserve `timeout:` here for checks whose `run` is genuinely interruptible —
|
|
420
|
+
typically your own custom checks — rather than the built-in datastore ones.
|
|
421
|
+
- Set the value comfortably above observed p99 (see the `latency_ms` in
|
|
422
|
+
`check.completed`), so it fires on a hang rather than on a slow day.
|
|
423
|
+
|
|
424
|
+
This is why the defaults ship as `nil` and why choosing them is deferred: the
|
|
425
|
+
right answer is usually a driver timeout, not this.
|
|
426
|
+
|
|
286
427
|
## License
|
|
287
428
|
|
|
288
429
|
MIT.
|
|
@@ -19,14 +19,36 @@ module StandardHealth
|
|
|
19
19
|
def env
|
|
20
20
|
spec = StandardHealth.config.env_spec
|
|
21
21
|
mode = ENV["APP_ENVIRONMENT"].to_s
|
|
22
|
+
root = defined?(Rails) ? Rails.root : nil
|
|
22
23
|
|
|
23
|
-
audit = spec ? spec.audit(ENV.to_h, mode: mode) : []
|
|
24
|
+
audit = spec ? spec.audit(ENV.to_h, mode: mode, root: root) : []
|
|
24
25
|
|
|
25
26
|
render json: {
|
|
26
27
|
mode: mode,
|
|
28
|
+
status: audit_status(audit),
|
|
27
29
|
audit: audit,
|
|
28
30
|
generated_at: Time.now.utc.iso8601
|
|
29
31
|
}
|
|
30
32
|
end
|
|
33
|
+
|
|
34
|
+
private
|
|
35
|
+
|
|
36
|
+
# Top-level verdict so a caller can gate on ONE field instead of
|
|
37
|
+
# re-implementing the roll-up over `audit`.
|
|
38
|
+
#
|
|
39
|
+
# :ok — nothing required is missing
|
|
40
|
+
# :incomplete — at least one `required` var is missing
|
|
41
|
+
#
|
|
42
|
+
# ADDITIVE ONLY in 0.4.1: the endpoint still returns 200 either way, so
|
|
43
|
+
# nothing that asserts on the status code breaks. That is the point —
|
|
44
|
+
# it gives monitors a release in which to migrate onto this field before
|
|
45
|
+
# 0.5.0 makes `:incomplete` a 503. Without the migration window, turning
|
|
46
|
+
# the code non-200 would silently redden every existing caller.
|
|
47
|
+
def audit_status(audit)
|
|
48
|
+
incomplete = Array(audit).any? do |row|
|
|
49
|
+
row[:level].to_s == "required" && row[:status].to_s == "missing"
|
|
50
|
+
end
|
|
51
|
+
incomplete ? :incomplete : :ok
|
|
52
|
+
end
|
|
31
53
|
end
|
|
32
54
|
end
|
|
@@ -12,10 +12,38 @@ module StandardHealth
|
|
|
12
12
|
# Readiness probe. Runs every registered check and returns:
|
|
13
13
|
# 200 if the rolled-up status is :ok or :degraded
|
|
14
14
|
# 503 if any critical check failed (:unavailable)
|
|
15
|
+
#
|
|
16
|
+
# Failure detail is REDACTED by default — see StandardHealth::Redactor.
|
|
17
|
+
# This endpoint is unauthenticated so probes need no credentials, and a
|
|
18
|
+
# raw driver error will happily tell an anonymous caller the database
|
|
19
|
+
# host, port and username. The full message goes to logs and Sentry
|
|
20
|
+
# instead. Status codes and every other field are unchanged.
|
|
15
21
|
def ready
|
|
16
22
|
result = StandardHealth::Aggregator.call
|
|
17
23
|
http_status = result[:status] == :unavailable ? :service_unavailable : :ok
|
|
18
|
-
render json: result,
|
|
24
|
+
render json: StandardHealth::Redactor.call(result, expose: expose_errors?),
|
|
25
|
+
status: http_status
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
private
|
|
29
|
+
|
|
30
|
+
# Detail is exposed when the host opted in globally, or when the caller
|
|
31
|
+
# presents the break-glass token. Compared in constant time: the token is
|
|
32
|
+
# a secret, and a naive == leaks its length and prefix to a patient
|
|
33
|
+
# attacker on an endpoint that is public by design.
|
|
34
|
+
def expose_errors?
|
|
35
|
+
config = StandardHealth.config
|
|
36
|
+
return true if config.expose_check_errors
|
|
37
|
+
|
|
38
|
+
token = config.detail_token
|
|
39
|
+
return false if token.nil? || token.empty?
|
|
40
|
+
|
|
41
|
+
presented = request.headers["X-Health-Token"].to_s
|
|
42
|
+
return false if presented.empty?
|
|
43
|
+
|
|
44
|
+
ActiveSupport::SecurityUtils.secure_compare(presented, token)
|
|
45
|
+
rescue StandardError
|
|
46
|
+
false
|
|
19
47
|
end
|
|
20
48
|
end
|
|
21
49
|
end
|
|
@@ -1,48 +1,187 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require "timeout"
|
|
4
|
+
|
|
3
5
|
module StandardHealth
|
|
6
|
+
# Raised internally when a check exceeds its timeout.
|
|
7
|
+
#
|
|
8
|
+
# A DEDICATED subclass, never bare `Timeout::Error`: `Timeout.timeout` with
|
|
9
|
+
# the default class would also swallow a timeout the host app raised for its
|
|
10
|
+
# own reasons and mislabel it as ours. Passing our own class means we only
|
|
11
|
+
# ever catch the one we threw.
|
|
12
|
+
class CheckTimeout < ::Timeout::Error; end
|
|
13
|
+
|
|
4
14
|
# Runs all registered checks and rolls them up into a single status.
|
|
5
15
|
#
|
|
6
16
|
# Status semantics:
|
|
7
17
|
# :ok — every check returned :ok
|
|
8
|
-
# :degraded — at least one non-critical check failed
|
|
18
|
+
# :degraded — at least one non-critical check failed, OR a check was
|
|
19
|
+
# skipped because the total budget ran out
|
|
9
20
|
# :unavailable — at least one critical check failed
|
|
10
21
|
#
|
|
11
|
-
# The aggregator never raises. Each check is invoked through
|
|
12
|
-
#
|
|
13
|
-
#
|
|
22
|
+
# The aggregator never raises. Each check is invoked through `safe_run`
|
|
23
|
+
# which catches `StandardError` so a buggy custom check cannot take down
|
|
24
|
+
# /ready. Instrumentation is held to the same bar — every emit is wrapped
|
|
25
|
+
# so a broken subscriber cannot become a new way for /ready to 500.
|
|
14
26
|
class Aggregator
|
|
15
27
|
def self.call(checks: StandardHealth.config.checks, now: Time.now.utc)
|
|
16
|
-
|
|
28
|
+
started = monotonic
|
|
29
|
+
budget = StandardHealth.config.total_check_budget
|
|
30
|
+
|
|
31
|
+
check_rows = checks.map do |reg|
|
|
32
|
+
if budget_exhausted?(budget, started)
|
|
33
|
+
row = skipped_row(reg, budget)
|
|
34
|
+
# Skips emit too. Without this a skip is invisible to the Metrics
|
|
35
|
+
# notifier — the only trace would be a name inside ready.evaluated's
|
|
36
|
+
# `failed[]`, with no per-check counter — so once a budget is
|
|
37
|
+
# enabled you could not answer "how often is check X getting
|
|
38
|
+
# skipped", which is exactly the question a budget creates.
|
|
39
|
+
emit_check(row)
|
|
40
|
+
row
|
|
41
|
+
else
|
|
42
|
+
safe_run(reg)
|
|
43
|
+
end
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
duration_ms = ((monotonic - started) * 1000).round
|
|
47
|
+
status = overall_status(check_rows)
|
|
48
|
+
|
|
49
|
+
failing = check_rows.reject { |r| r[:status] == :ok }
|
|
50
|
+
|
|
51
|
+
emit(
|
|
52
|
+
"standard_health.ready.evaluated",
|
|
53
|
+
status: status,
|
|
54
|
+
duration_ms: duration_ms,
|
|
55
|
+
failed: failing.map { |r| r[:name] },
|
|
56
|
+
# The full failure detail rides on THIS event, not only on the
|
|
57
|
+
# per-check one. Redaction removes the message from the HTTP body on
|
|
58
|
+
# the promise that it still reaches logs and Sentry — but both of
|
|
59
|
+
# those subscribers are driven by ready.evaluated (deliberately: it
|
|
60
|
+
# is the transition-gated event, so they don't fire per poll). If the
|
|
61
|
+
# message only existed on check.completed, redaction would delete the
|
|
62
|
+
# last copy instead of relocating it.
|
|
63
|
+
failures: failing.map do |r|
|
|
64
|
+
{
|
|
65
|
+
name: r[:name],
|
|
66
|
+
critical: r[:critical],
|
|
67
|
+
status: r[:status],
|
|
68
|
+
error_class: r[:error_class],
|
|
69
|
+
error_message: r[:error]
|
|
70
|
+
}.compact
|
|
71
|
+
end
|
|
72
|
+
)
|
|
73
|
+
|
|
17
74
|
{
|
|
18
|
-
status:
|
|
75
|
+
status: status,
|
|
19
76
|
checks: check_rows,
|
|
20
77
|
generated_at: now.iso8601
|
|
21
78
|
}
|
|
22
79
|
end
|
|
23
80
|
|
|
24
81
|
def self.safe_run(reg)
|
|
25
|
-
|
|
26
|
-
|
|
82
|
+
timeout = reg.timeout || StandardHealth.config.default_check_timeout
|
|
83
|
+
instance = reg.klass.new(name: reg.name, critical: reg.critical)
|
|
84
|
+
|
|
85
|
+
result =
|
|
86
|
+
if timeout
|
|
87
|
+
Timeout.timeout(timeout, CheckTimeout) { instance.run }
|
|
88
|
+
else
|
|
89
|
+
instance.run
|
|
90
|
+
end
|
|
91
|
+
|
|
92
|
+
row = result.merge(name: reg.name, critical: reg.critical)
|
|
93
|
+
emit_check(row)
|
|
94
|
+
row
|
|
95
|
+
rescue CheckTimeout
|
|
96
|
+
emit("standard_health.check.timed_out",
|
|
97
|
+
name: reg.name, critical: reg.critical, timeout_s: timeout)
|
|
98
|
+
row = {
|
|
99
|
+
name: reg.name,
|
|
100
|
+
critical: reg.critical,
|
|
101
|
+
status: :fail,
|
|
102
|
+
error: "timed out after #{timeout}s",
|
|
103
|
+
error_class: "StandardHealth::CheckTimeout"
|
|
104
|
+
}
|
|
105
|
+
emit_check(row)
|
|
106
|
+
row
|
|
27
107
|
rescue StandardError => e
|
|
28
|
-
{
|
|
108
|
+
row = {
|
|
29
109
|
name: reg.name,
|
|
30
110
|
critical: reg.critical,
|
|
31
111
|
status: :fail,
|
|
32
|
-
error: e.message
|
|
112
|
+
error: e.message,
|
|
113
|
+
error_class: e.class.name
|
|
33
114
|
}
|
|
115
|
+
emit_check(row)
|
|
116
|
+
row
|
|
34
117
|
end
|
|
35
118
|
private_class_method :safe_run
|
|
36
119
|
|
|
120
|
+
def self.budget_exhausted?(budget, started)
|
|
121
|
+
return false unless budget
|
|
122
|
+
|
|
123
|
+
(monotonic - started) >= budget
|
|
124
|
+
end
|
|
125
|
+
private_class_method :budget_exhausted?
|
|
126
|
+
|
|
127
|
+
# A check the total budget never reached. `:skipped`, never silently
|
|
128
|
+
# `:ok` — an unperformed check is not a healthy one.
|
|
129
|
+
def self.skipped_row(reg, budget)
|
|
130
|
+
{
|
|
131
|
+
name: reg.name,
|
|
132
|
+
critical: reg.critical,
|
|
133
|
+
status: :skipped,
|
|
134
|
+
error: "skipped — total check budget of #{budget}s exhausted"
|
|
135
|
+
}
|
|
136
|
+
end
|
|
137
|
+
private_class_method :skipped_row
|
|
138
|
+
|
|
37
139
|
def self.overall_status(rows)
|
|
38
140
|
return :ok if rows.empty?
|
|
39
141
|
|
|
40
142
|
failures = rows.reject { |r| r[:status] == :ok }
|
|
41
143
|
return :ok if failures.empty?
|
|
42
|
-
|
|
144
|
+
|
|
145
|
+
# A SKIP MUST NEVER PRODUCE :unavailable, even for a critical check.
|
|
146
|
+
# Otherwise a slow *non-critical* check could exhaust the budget, leave
|
|
147
|
+
# the database check unrun, and pull a perfectly healthy instance out of
|
|
148
|
+
# rotation — a self-inflicted outage caused by the safety mechanism.
|
|
149
|
+
# Skips floor the result at :degraded; only a real critical FAILURE
|
|
150
|
+
# reaches :unavailable.
|
|
151
|
+
return :unavailable if failures.any? { |r| r[:critical] && r[:status] != :skipped }
|
|
43
152
|
|
|
44
153
|
:degraded
|
|
45
154
|
end
|
|
46
155
|
private_class_method :overall_status
|
|
156
|
+
|
|
157
|
+
def self.emit_check(row)
|
|
158
|
+
emit(
|
|
159
|
+
"standard_health.check.completed",
|
|
160
|
+
name: row[:name],
|
|
161
|
+
critical: row[:critical],
|
|
162
|
+
status: row[:status],
|
|
163
|
+
latency_ms: row[:latency_ms],
|
|
164
|
+
error_class: row[:error_class],
|
|
165
|
+
error_message: row[:error]
|
|
166
|
+
)
|
|
167
|
+
end
|
|
168
|
+
private_class_method :emit_check
|
|
169
|
+
|
|
170
|
+
# Belt and braces on top of EventEmitter's own rescue. The never-raise
|
|
171
|
+
# rule names this file, and instrumentation is the newest way to violate
|
|
172
|
+
# it — so the call site guards too.
|
|
173
|
+
def self.emit(event_name, **payload)
|
|
174
|
+
return unless StandardHealth.config.instrumentation_enabled
|
|
175
|
+
|
|
176
|
+
EventEmitter.emit(event_name, payload)
|
|
177
|
+
rescue StandardError
|
|
178
|
+
nil
|
|
179
|
+
end
|
|
180
|
+
private_class_method :emit
|
|
181
|
+
|
|
182
|
+
def self.monotonic
|
|
183
|
+
Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
184
|
+
end
|
|
185
|
+
private_class_method :monotonic
|
|
47
186
|
end
|
|
48
187
|
end
|
|
@@ -43,7 +43,12 @@ module StandardHealth
|
|
|
43
43
|
latency_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - start) * 1000).round
|
|
44
44
|
{ status: :ok, latency_ms: latency_ms }
|
|
45
45
|
rescue StandardError => e
|
|
46
|
-
|
|
46
|
+
# `error_class` matters as much as the message. All three built-in
|
|
47
|
+
# checks route through here, so without it every real driver failure
|
|
48
|
+
# reaches the aggregator with no class — and the redacted response then
|
|
49
|
+
# falls back to a generic "StandardError", which is precisely the
|
|
50
|
+
# useless label redaction exists to replace with something groupable.
|
|
51
|
+
{ status: :fail, error: e.message, error_class: e.class.name }
|
|
47
52
|
end
|
|
48
53
|
end
|
|
49
54
|
end
|