standard_health 0.6.1 → 0.7.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +101 -0
- data/README.md +87 -14
- data/app/controllers/concerns/standard_health/diagnostics_authentication.rb +9 -0
- data/app/controllers/standard_health/diagnostics_controller.rb +8 -3
- data/lib/standard_health/aggregate_report.rb +5 -3
- data/lib/standard_health/aggregator.rb +47 -5
- data/lib/standard_health/check.rb +10 -0
- data/lib/standard_health/configuration.rb +49 -1
- data/lib/standard_health/diagnostics_assertions.rb +62 -0
- data/lib/standard_health/version.rb +1 -1
- data/lib/standard_health.rb +1 -0
- metadata +5 -4
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 162374f071e6634d43876ef75f8fa6f50b1ec7bfd6a8d01bd94a91c854151d09
|
|
4
|
+
data.tar.gz: 58c40844359768a3d905c75575599dc1bf61dab15ebabc372c3e5e56cd4e169d
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: a93890d618342991d02699f9a68d0cf432596f4fba5eff7659d4d4c8675d22e5e0bac3d996abed41e02ddd7f53b84c771f21e093a6c6e22b39404ec9ccc7498c
|
|
7
|
+
data.tar.gz: 13b347704251b47c6a877ce5fb559686d9fee639610109300da1fd989627010a3823f678ab420330a26ac851548647da3ab12405081cd13a2d9024c90d4bb5dc
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,107 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.7.1] - 2026-09-25
|
|
11
|
+
|
|
12
|
+
### Fixed
|
|
13
|
+
|
|
14
|
+
- **Skipped checks no longer degrade the aggregate.** A check that returns
|
|
15
|
+
`status: :skipped` itself means "not applicable here" (the feature it
|
|
16
|
+
covers is not configured or not enforced). It is now **neutral** in the
|
|
17
|
+
roll-up on every tier (`/ready` and the aggregate `/health`): it neither
|
|
18
|
+
degrades nor fails the status. Before, it counted like a failing
|
|
19
|
+
non-critical check, which held sidekick-web's `/health` at `degraded`
|
|
20
|
+
permanently while attestation was unconfigured. The row still renders in
|
|
21
|
+
`checks[]` with `"status": "skipped"`, and `check.completed` still fires,
|
|
22
|
+
so it stays visible. It is no longer listed in the `failed[]` / `failures`
|
|
23
|
+
of `ready.evaluated` / `aggregate.evaluated`, so the Logger and Sentry
|
|
24
|
+
notifiers don't name it as failing when something else degrades.
|
|
25
|
+
- **A critical check that reports `:skipped` is neutral too**, on purpose.
|
|
26
|
+
"Not applicable" says nothing about whether the instance can serve, so it
|
|
27
|
+
must not pull it out of rotation. Return `:fail` for a state that should
|
|
28
|
+
page.
|
|
29
|
+
- Real failures roll up exactly as before: a failing critical check is
|
|
30
|
+
`unavailable` (503), a failing non-critical one `degraded`, including
|
|
31
|
+
alongside a skipped check.
|
|
32
|
+
- **Budget skips are unchanged.** A check the `total_check_budget` never
|
|
33
|
+
reached was not performed, which is not the same as healthy. It still
|
|
34
|
+
floors the roll-up at `degraded` (never `unavailable`) and is still listed
|
|
35
|
+
in `failed[]`. Its row now carries `budget_exhausted: true`, which is how
|
|
36
|
+
the gem tells the two kinds of skip apart.
|
|
37
|
+
- `/diagnostics/env` is unaffected. Its roll-up is over env-audit rows and
|
|
38
|
+
assertions (`:ok` / `:warn` / `:error`), which have no skipped state.
|
|
39
|
+
|
|
40
|
+
### Changed
|
|
41
|
+
|
|
42
|
+
- **Requires Rails 8.1** (`rails >= 8.1`, was `>= 8.0`). Every consumer app
|
|
43
|
+
runs 8.1; 8.0 was never exercised in CI. Ruby 3.4 remains supported.
|
|
44
|
+
|
|
45
|
+
### Upgrade notes (0.7.0 → 0.7.1)
|
|
46
|
+
|
|
47
|
+
- **No required host change.** An app whose aggregate was `degraded` only
|
|
48
|
+
because of a self-reported skip now reports `ok`. Monitors or smoke tests
|
|
49
|
+
that expect `degraded` in that state need updating (sidekick-web).
|
|
50
|
+
|
|
51
|
+
## [0.7.0] - 2026-09-24
|
|
52
|
+
|
|
53
|
+
The Phase 4 release. 0.6 deprecated nothing, so nothing is removed. It adds
|
|
54
|
+
the one extension point the app adoptions still needed their own controller
|
|
55
|
+
for. The response of `/diagnostics/env` gains a key, and a failing assertion
|
|
56
|
+
changes its `status`, so this ships as a minor.
|
|
57
|
+
|
|
58
|
+
### Added
|
|
59
|
+
|
|
60
|
+
- **`Configuration#register_diagnostics_assertion(name, callable = nil, &block)`**
|
|
61
|
+
adds a runtime assertion to the doctor tier. `/diagnostics/env` now renders
|
|
62
|
+
`assertions: [...]` next to `audit`. The key is always present (`[]` when
|
|
63
|
+
none are registered). Each callable returns a Hash with `status:` (`:ok` /
|
|
64
|
+
`:warn` / `:error`) plus detail keys, and the gem sets `name:`. A raise
|
|
65
|
+
becomes an `:error` row with `error_class` / `error` and is reported to
|
|
66
|
+
`Rails.error` as handled (`context: { diagnostics_assertion: name }`,
|
|
67
|
+
`source: "standard_health"`). A non-Hash result or an unknown status also
|
|
68
|
+
becomes an `:error` row. Re-registering a name replaces it (reload-safe).
|
|
69
|
+
`diagnostics_assertions` / `reset_diagnostics_assertions!` are provided.
|
|
70
|
+
Assertions never run on `/alive`, `/ready` or the aggregate tier, and stay
|
|
71
|
+
behind `diagnostics_basic_auth`.
|
|
72
|
+
- **`StandardHealth::DiagnosticsAssertions.run` / `.failing?`**, public for
|
|
73
|
+
hosts that render their own diagnostics endpoint.
|
|
74
|
+
|
|
75
|
+
### Changed
|
|
76
|
+
|
|
77
|
+
- **`/diagnostics/env` `status` also rolls up assertions.** An `:error`
|
|
78
|
+
assertion makes it `incomplete` (`:warn` does not). The HTTP status is still
|
|
79
|
+
200. Nothing changes unless you register an assertion.
|
|
80
|
+
- **`StandardHealth::DiagnosticsAuthentication` is public, semver-stable
|
|
81
|
+
API.** Including it into a host controller (as sidekick-web does) is now a
|
|
82
|
+
supported extension point: the include, its single `before_action`, and the
|
|
83
|
+
401 challenge / 403 fail-closed refusal. Its private method names are not
|
|
84
|
+
part of the contract.
|
|
85
|
+
|
|
86
|
+
### Upgrade notes (0.6.x → 0.7.0)
|
|
87
|
+
|
|
88
|
+
Grepped `origin/main` of sidekick-web, jumpdrive-web, fundbright-web,
|
|
89
|
+
luminality-web and nutripod-web on 2026-09-24.
|
|
90
|
+
|
|
91
|
+
- **No required host change.** Nothing was removed.
|
|
92
|
+
- **Monitors or scripts that parse `/diagnostics/env`:** a new `assertions`
|
|
93
|
+
key appears, and `status` can become `incomplete` from an `:error`
|
|
94
|
+
assertion once you register one.
|
|
95
|
+
- **sidekick-web (optional, recommended):** move the three assertions in
|
|
96
|
+
`app/controllers/health_diagnostics_controller.rb` (`statement_timeout`,
|
|
97
|
+
`rate_limit_store`, `attestation_roots`) into
|
|
98
|
+
`register_diagnostics_assertion` calls in
|
|
99
|
+
`config/initializers/standard_health.rb`. Then delete the controller and its
|
|
100
|
+
`config/routes.rb` route that shadows the engine's `/health/diagnostics/env`.
|
|
101
|
+
Keep each assertion's Hash shape. Drop the per-assertion
|
|
102
|
+
`rescue => e; { status: :error, error: e.message }`, which the gem now
|
|
103
|
+
provides (it also adds `error_class`). The response gains `status`, which
|
|
104
|
+
the host controller never rendered.
|
|
105
|
+
- Other apps have no diagnostics controller of their own. Nothing to do.
|
|
106
|
+
|
|
107
|
+
### Documentation
|
|
108
|
+
|
|
109
|
+
- Aggregate-tier check **timeouts** are deliberately not sent to `Rails.error` (true since 0.6.1). A timeout is the gem's own budget firing on a slow dependency, already emitted as `standard_health.check.timed_out`, and the pre-0.6 host controllers had no per-check timeouts. A code comment and a spec now pin this.
|
|
110
|
+
|
|
10
111
|
## [0.6.1] - 2026-09-24
|
|
11
112
|
|
|
12
113
|
### Fixed
|
data/README.md
CHANGED
|
@@ -106,7 +106,7 @@ end
|
|
|
106
106
|
| Status | HTTP | When |
|
|
107
107
|
|---|---|---|
|
|
108
108
|
| `unavailable` | 503 | a critical check failed, or a `:critical` circuit is red |
|
|
109
|
-
| `degraded` | 200 | a non-critical check failed
|
|
109
|
+
| `degraded` | 200 | a non-critical check failed, a check was skipped by the total budget, or the circuit roll-up is `:degraded` |
|
|
110
110
|
| `ok` | 200 | otherwise |
|
|
111
111
|
|
|
112
112
|
**The aggregate tier re-runs your readiness checks by default.** With
|
|
@@ -130,7 +130,8 @@ A check that **raises** on this tier, rather than returning a `:fail` row, is
|
|
|
130
130
|
also reported to `Rails.error` as handled (since 0.6.1), with context
|
|
131
131
|
`{ health_check:, tier: "aggregate" }` and severity `:error` for a critical
|
|
132
132
|
check or `:warning` otherwise. That matches what the pre-0.6 host controllers
|
|
133
|
-
did.
|
|
133
|
+
did. Per-check timeouts are not reported; they surface as
|
|
134
|
+
`standard_health.check.timed_out`. `/ready` does not do this: its failures reach Sentry through the
|
|
134
135
|
transition-gated `ready.evaluated` notifier.
|
|
135
136
|
|
|
136
137
|
With `aggregate_endpoint` off (the default) the engine's root route carries a
|
|
@@ -418,11 +419,13 @@ Since 0.4.1 the response carries a `status` alongside the audit, so a caller
|
|
|
418
419
|
can gate on one field instead of re-implementing the roll-up:
|
|
419
420
|
|
|
420
421
|
```json
|
|
421
|
-
{ "mode": "production", "status": "incomplete", "audit": [ ... ] }
|
|
422
|
+
{ "mode": "production", "status": "incomplete", "audit": [ ... ], "assertions": [ ... ] }
|
|
422
423
|
```
|
|
423
424
|
|
|
424
425
|
`incomplete` means at least one row is a **violation** — `missing`,
|
|
425
|
-
`forbidden`, or `mismatch
|
|
426
|
+
`forbidden`, or `mismatch` — or (since 0.7.0) a registered diagnostics
|
|
427
|
+
assertion reported `error`. `should_set` and an assertion's `warn` are
|
|
428
|
+
advisory and never affect it.
|
|
426
429
|
|
|
427
430
|
The level is deliberately not consulted for `mismatch`: a `recommended` var
|
|
428
431
|
declared with an `expected_value:` that does not hold is a failed assertion,
|
|
@@ -433,6 +436,56 @@ not advice.
|
|
|
433
436
|
free — which is why they joined this roll-up rather than getting a verdict of
|
|
434
437
|
their own.
|
|
435
438
|
|
|
439
|
+
### Runtime assertions: `register_diagnostics_assertion` (0.7.0)
|
|
440
|
+
|
|
441
|
+
Env presence can't prove that a setting took effect. For example, the live
|
|
442
|
+
`statement_timeout` may still be `0`, a rate limiter may have fallen back to
|
|
443
|
+
Solid Cache, or a pinned certificate may be about to expire. Register those
|
|
444
|
+
checks as assertions and the engine renders them on `/diagnostics/env` under
|
|
445
|
+
`assertions:`, behind the same `diagnostics_basic_auth` gate:
|
|
446
|
+
|
|
447
|
+
```ruby
|
|
448
|
+
StandardHealth.configure do |c|
|
|
449
|
+
c.register_diagnostics_assertion(:statement_timeout) do
|
|
450
|
+
value = ActiveRecord::Base.connection.select_value("SHOW statement_timeout").to_s
|
|
451
|
+
{ status: value.strip == "0" ? :warn : :ok, value: value, expected: "non-zero" }
|
|
452
|
+
end
|
|
453
|
+
|
|
454
|
+
# Any callable works; reference app constants inside it, not at boot.
|
|
455
|
+
c.register_diagnostics_assertion(:rate_limit_store, -> { RateLimitAssertion.call })
|
|
456
|
+
end
|
|
457
|
+
```
|
|
458
|
+
|
|
459
|
+
```json
|
|
460
|
+
{ "mode": "production", "status": "ok", "audit": [ ... ],
|
|
461
|
+
"assertions": [ { "name": "statement_timeout", "status": "ok", "value": "15s", "expected": "non-zero" } ] }
|
|
462
|
+
```
|
|
463
|
+
|
|
464
|
+
- The callable takes no arguments and returns a Hash with `status:` set to
|
|
465
|
+
`:ok`, `:warn` or `:error`. Other keys are rendered as-is. The gem sets
|
|
466
|
+
`name:`, and the callable can't override it.
|
|
467
|
+
- An assertion that raises becomes `{ status: "error", error_class:, error: }`
|
|
468
|
+
and is reported to `Rails.error` as handled. The tier is authenticated, so
|
|
469
|
+
the message is shown. A non-Hash result or an unknown status also becomes an
|
|
470
|
+
`:error` row. One broken assertion never takes the endpoint down.
|
|
471
|
+
- An `:error` row makes the top-level `status` `incomplete`. `:warn` does not.
|
|
472
|
+
The endpoint still returns 200.
|
|
473
|
+
- Assertions run per request, and only on this tier, never on `/alive`,
|
|
474
|
+
`/ready` or the aggregate. They have no timeout, so keep them cheap.
|
|
475
|
+
- Re-registering a name replaces it, so it is safe inside `to_prepare`.
|
|
476
|
+
`reset_diagnostics_assertions!` clears them in specs.
|
|
477
|
+
- A host that keeps its own diagnostics controller can render
|
|
478
|
+
`StandardHealth::DiagnosticsAssertions.run` itself.
|
|
479
|
+
|
|
480
|
+
**Replace your host code with it.** sidekick-web's
|
|
481
|
+
`app/controllers/health_diagnostics_controller.rb` exists to add three
|
|
482
|
+
assertions (`statement_timeout`, `rate_limit_store`, `attestation_roots`) to
|
|
483
|
+
the env audit. Move each into a `register_diagnostics_assertion`, delete the
|
|
484
|
+
controller, and delete its route
|
|
485
|
+
(`get "/health/diagnostics/env", to: "health_diagnostics#env"` ahead of the
|
|
486
|
+
engine mount). Its `rescue => e; { status: :error, error: e.message }`
|
|
487
|
+
wrappers go too, because the gem does that now.
|
|
488
|
+
|
|
436
489
|
### Surfacing the audit on a health tier
|
|
437
490
|
|
|
438
491
|
`/diagnostics/env` is authed and polled by nobody, so config drift declared in
|
|
@@ -608,11 +661,15 @@ c.diagnostics_basic_auth = {
|
|
|
608
661
|
both, the parent's callbacks run first).
|
|
609
662
|
- The request-time gate is the `StandardHealth::DiagnosticsAuthentication`
|
|
610
663
|
concern, which the engine includes into its own `DiagnosticsController`.
|
|
611
|
-
|
|
612
|
-
|
|
613
|
-
|
|
614
|
-
|
|
615
|
-
|
|
664
|
+
**Since 0.7.0 it is public, semver-stable API:** `include
|
|
665
|
+
StandardHealth::DiagnosticsAuthentication` into any `ActionController::API`
|
|
666
|
+
or `::Base` controller to put a host endpoint behind the same fail-closed
|
|
667
|
+
gate. The contract is the include, the one `before_action` it installs, and
|
|
668
|
+
the 401 challenge / 403 refusal behaviour. Its private method names are not
|
|
669
|
+
part of it. It is a no-op until `diagnostics_basic_auth` is set. If your
|
|
670
|
+
controller only exists to add runtime assertions, use
|
|
671
|
+
[`register_diagnostics_assertion`](#runtime-assertions-register_diagnostics_assertion-070)
|
|
672
|
+
instead and delete the controller.
|
|
616
673
|
|
|
617
674
|
**Replace your host code with it.** Delete
|
|
618
675
|
`app/controllers/standard_health_host_controller.rb` and the
|
|
@@ -749,9 +806,25 @@ go in `extra:`.
|
|
|
749
806
|
|
|
750
807
|
The orchestrator should pull the instance out of rotation only on 503; degraded means "still serving, page someone."
|
|
751
808
|
|
|
752
|
-
`skipped` also exists,
|
|
753
|
-
|
|
754
|
-
`
|
|
809
|
+
`skipped` also exists, in two flavours that roll up differently:
|
|
810
|
+
|
|
811
|
+
- **A check that returns `status: :skipped` itself** — "not applicable here"
|
|
812
|
+
(the feature it covers isn't configured or enforced). This is **neutral**
|
|
813
|
+
(since 0.7.1): it neither degrades nor fails the roll-up, whether the check
|
|
814
|
+
is critical or not, and it is left out of the evaluation events' `failed[]`.
|
|
815
|
+
It still renders in `checks[]` with `"status": "skipped"` and still emits
|
|
816
|
+
`check.completed`, so it stays visible. A critical check is neutral too on
|
|
817
|
+
purpose: "not applicable" says nothing about whether the instance can serve,
|
|
818
|
+
so it must not pull it out of rotation — and it must not mark it healthy
|
|
819
|
+
*because of* that check either; it simply doesn't count. If the state should
|
|
820
|
+
page, return `:fail`.
|
|
821
|
+
- **A check the total budget never reached** (see Timeouts) — rendered with
|
|
822
|
+
`"budget_exhausted": true`. That check was not performed, which is not the
|
|
823
|
+
same as healthy, so it floors the roll-up at `degraded` and **never**
|
|
824
|
+
produces `unavailable`, even for a critical check.
|
|
825
|
+
|
|
826
|
+
A real failure still rolls up exactly as before alongside any skip: a failing
|
|
827
|
+
critical check is `unavailable`, a failing non-critical one `degraded`.
|
|
755
828
|
|
|
756
829
|
## Failure detail is redacted
|
|
757
830
|
|
|
@@ -877,8 +950,8 @@ slow-but-fine starts reporting `:fail`, and for a critical check that pulls
|
|
|
877
950
|
the instance out of rotation. Pick values from observed `latency_ms` rather
|
|
878
951
|
than intuition; that is what the `check.completed` events are for.
|
|
879
952
|
|
|
880
|
-
Checks the total budget never reaches report `:skipped
|
|
881
|
-
A skip alone floors the roll-up at `degraded` — otherwise a slow *non-critical*
|
|
953
|
+
Checks the total budget never reaches report `:skipped` (with
|
|
954
|
+
`budget_exhausted: true`), never silently `:ok`. A budget skip alone floors the roll-up at `degraded` — otherwise a slow *non-critical*
|
|
882
955
|
check could exhaust the budget, leave the database check unrun, and pull a
|
|
883
956
|
healthy instance out of rotation.
|
|
884
957
|
|
|
@@ -7,6 +7,15 @@ module StandardHealth
|
|
|
7
7
|
# `DiagnosticsController` only — never `HealthController` — so /alive and
|
|
8
8
|
# /ready stay anonymous for orchestrator probes.
|
|
9
9
|
#
|
|
10
|
+
# PUBLIC API (since 0.7.0, semver-stable): a host that serves its own
|
|
11
|
+
# diagnostics endpoint can `include StandardHealth::DiagnosticsAuthentication`
|
|
12
|
+
# into any ActionController::API or ::Base controller to put it behind the
|
|
13
|
+
# same fail-closed gate. The contract is the include, the `before_action`
|
|
14
|
+
# it installs, and the 401 challenge / 403 refusal behaviour below; the
|
|
15
|
+
# private method names are not part of it. Prefer
|
|
16
|
+
# `register_diagnostics_assertion` when all the host controller adds is
|
|
17
|
+
# assertions.
|
|
18
|
+
#
|
|
10
19
|
# A NO-OP when `diagnostics_basic_auth` is unset, so hosts that gate
|
|
11
20
|
# diagnostics through `diagnostics_parent_controller` see no change. Both
|
|
12
21
|
# can be used together; the parent's callbacks run first.
|
|
@@ -27,11 +27,13 @@ module StandardHealth
|
|
|
27
27
|
root = defined?(Rails) ? Rails.root : nil
|
|
28
28
|
|
|
29
29
|
audit = spec ? spec.audit(ENV.to_h, mode: mode, root: root) : []
|
|
30
|
+
assertions = DiagnosticsAssertions.run
|
|
30
31
|
|
|
31
32
|
render json: {
|
|
32
33
|
mode: mode,
|
|
33
|
-
status: audit_status(audit),
|
|
34
|
+
status: audit_status(audit, assertions),
|
|
34
35
|
audit: audit,
|
|
36
|
+
assertions: assertions,
|
|
35
37
|
generated_at: Time.now.utc.iso8601
|
|
36
38
|
}
|
|
37
39
|
end
|
|
@@ -53,13 +55,16 @@ module StandardHealth
|
|
|
53
55
|
# advice. The advisory status stays `:should_set`, which is not a
|
|
54
56
|
# violation.
|
|
55
57
|
#
|
|
58
|
+
# A registered diagnostics assertion (`register_diagnostics_assertion`)
|
|
59
|
+
# that reports `:error` also makes it `:incomplete`; `:warn` does not.
|
|
60
|
+
#
|
|
56
61
|
# The endpoint still returns 200 either way, so nothing that asserts on
|
|
57
62
|
# the status code breaks.
|
|
58
|
-
def audit_status(audit)
|
|
63
|
+
def audit_status(audit, assertions = [])
|
|
59
64
|
violated = Array(audit).any? do |row|
|
|
60
65
|
EnvSpec::VIOLATION_STATUSES.include?(row[:status])
|
|
61
66
|
end
|
|
62
|
-
violated ? :incomplete : :ok
|
|
67
|
+
(violated || DiagnosticsAssertions.failing?(assertions)) ? :incomplete : :ok
|
|
63
68
|
end
|
|
64
69
|
end
|
|
65
70
|
end
|
|
@@ -18,8 +18,10 @@ module StandardHealth
|
|
|
18
18
|
#
|
|
19
19
|
# Status roll-up, in the estate's readiness vocabulary:
|
|
20
20
|
# :unavailable (503) — a critical check failed, or a :critical circuit is RED
|
|
21
|
-
# :degraded (200) — a non-critical check failed
|
|
22
|
-
# circuit roll-up is
|
|
21
|
+
# :degraded (200) — a non-critical check failed, a check was skipped
|
|
22
|
+
# by the total budget, or the circuit roll-up is
|
|
23
|
+
# :degraded. A check that reports :skipped itself
|
|
24
|
+
# (not applicable) is neutral — see Aggregator.
|
|
23
25
|
# :ok (200) — otherwise
|
|
24
26
|
#
|
|
25
27
|
# StandardCircuit's own word for its worst state is `:critical`; it is
|
|
@@ -92,7 +94,7 @@ module StandardHealth
|
|
|
92
94
|
# notifier logs this one (so redacted messages still reach logs); Sentry
|
|
93
95
|
# and Metrics ignore it.
|
|
94
96
|
def emit_evaluation(report, rows, circuits, started)
|
|
95
|
-
failing = rows
|
|
97
|
+
failing = Aggregator.failing_rows(rows)
|
|
96
98
|
payload = {
|
|
97
99
|
status: report[:status],
|
|
98
100
|
duration_ms: ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started) * 1000).round,
|
|
@@ -14,11 +14,23 @@ module StandardHealth
|
|
|
14
14
|
# Runs all registered checks and rolls them up into a single status.
|
|
15
15
|
#
|
|
16
16
|
# Status semantics:
|
|
17
|
-
# :ok — every check returned :ok
|
|
17
|
+
# :ok — every check returned :ok, or reported :skipped itself
|
|
18
18
|
# :degraded — at least one non-critical check failed, OR a check was
|
|
19
19
|
# skipped because the total budget ran out
|
|
20
20
|
# :unavailable — at least one critical check failed
|
|
21
21
|
#
|
|
22
|
+
# Two kinds of :skipped row exist, and they roll up differently (0.7.1):
|
|
23
|
+
#
|
|
24
|
+
# * A check that RETURNS `status: :skipped` is saying "not applicable
|
|
25
|
+
# here" (the feature it covers is not configured or not enforced). That
|
|
26
|
+
# is NEUTRAL — it neither degrades nor fails the tier, whatever its
|
|
27
|
+
# `critical:` flag, and it is not listed in an evaluation event's
|
|
28
|
+
# `failed[]`. The row still renders with status "skipped", and
|
|
29
|
+
# `check.completed` still fires, so it stays visible.
|
|
30
|
+
# * A check the total budget never REACHED (`budget_exhausted: true` on
|
|
31
|
+
# the row) was not performed at all, and an unperformed check is not a
|
|
32
|
+
# healthy one. It floors the tier at :degraded — never :unavailable.
|
|
33
|
+
#
|
|
22
34
|
# The aggregator never raises. Each check is invoked through `safe_run`
|
|
23
35
|
# which catches `StandardError` so a buggy custom check cannot take down
|
|
24
36
|
# /ready. Instrumentation is held to the same bar — every emit is wrapped
|
|
@@ -55,7 +67,7 @@ module StandardHealth
|
|
|
55
67
|
duration_ms = ((monotonic - started) * 1000).round
|
|
56
68
|
status = overall_status(check_rows)
|
|
57
69
|
|
|
58
|
-
failing = check_rows
|
|
70
|
+
failing = failing_rows(check_rows)
|
|
59
71
|
|
|
60
72
|
return { status: status, checks: check_rows, generated_at: now.iso8601 } unless tier == :ready
|
|
61
73
|
|
|
@@ -81,6 +93,21 @@ module StandardHealth
|
|
|
81
93
|
}
|
|
82
94
|
end
|
|
83
95
|
|
|
96
|
+
# The rows an evaluation event reports as failing: everything that is not
|
|
97
|
+
# :ok, except a check that reported :skipped itself (not applicable, so
|
|
98
|
+
# not a failure). A budget skip IS listed — it is the only place a skip
|
|
99
|
+
# shows up in the transition-gated event. @api private (also used by
|
|
100
|
+
# AggregateReport).
|
|
101
|
+
def self.failing_rows(rows)
|
|
102
|
+
rows.reject { |r| r[:status] == :ok || neutral_skip?(r) }
|
|
103
|
+
end
|
|
104
|
+
|
|
105
|
+
# A row the check itself reported as :skipped — "not applicable here".
|
|
106
|
+
# Neutral in the roll-up. A budget skip is not neutral. @api private
|
|
107
|
+
def self.neutral_skip?(row)
|
|
108
|
+
row[:status] == :skipped && !row[:budget_exhausted]
|
|
109
|
+
end
|
|
110
|
+
|
|
84
111
|
# Per-failure detail for an evaluation event. Public (but @api private) so
|
|
85
112
|
# the aggregate tier's evaluation event carries the same shape.
|
|
86
113
|
# @api private
|
|
@@ -111,6 +138,12 @@ module StandardHealth
|
|
|
111
138
|
emit_check(row, tier)
|
|
112
139
|
row
|
|
113
140
|
rescue CheckTimeout
|
|
141
|
+
# Deliberately NOT sent to Rails.error (unlike the StandardError branch
|
|
142
|
+
# below). A timeout is our own budget firing on a slow dependency, not a
|
|
143
|
+
# bug in the check: it is already visible as `check.timed_out` and as a
|
|
144
|
+
# failing row, and reporting it per poll would page on latency. The
|
|
145
|
+
# pre-0.6 host aggregate controllers had no per-check timeouts, so
|
|
146
|
+
# nothing reported there is lost.
|
|
114
147
|
emit("standard_health.check.timed_out",
|
|
115
148
|
name: reg.name, critical: reg.critical, timeout_s: timeout, **tier_attrs(tier))
|
|
116
149
|
row = {
|
|
@@ -171,12 +204,15 @@ module StandardHealth
|
|
|
171
204
|
private_class_method :budget_exhausted?
|
|
172
205
|
|
|
173
206
|
# A check the total budget never reached. `:skipped`, never silently
|
|
174
|
-
# `:ok` — an unperformed check is not a healthy one.
|
|
207
|
+
# `:ok` — an unperformed check is not a healthy one. `budget_exhausted`
|
|
208
|
+
# is what separates it from a check that reported :skipped itself (which
|
|
209
|
+
# is neutral); see overall_status.
|
|
175
210
|
def self.skipped_row(reg, budget)
|
|
176
211
|
{
|
|
177
212
|
name: reg.name,
|
|
178
213
|
critical: reg.critical,
|
|
179
214
|
status: :skipped,
|
|
215
|
+
budget_exhausted: true,
|
|
180
216
|
error: "skipped — total check budget of #{budget}s exhausted"
|
|
181
217
|
}
|
|
182
218
|
end
|
|
@@ -185,10 +221,16 @@ module StandardHealth
|
|
|
185
221
|
def self.overall_status(rows)
|
|
186
222
|
return :ok if rows.empty?
|
|
187
223
|
|
|
188
|
-
|
|
224
|
+
# A check that reported :skipped itself is not applicable here, so it is
|
|
225
|
+
# NEUTRAL: it must not degrade the tier (sidekick-web's non-critical
|
|
226
|
+
# attestation check held /health at "degraded" forever while attestation
|
|
227
|
+
# was simply not configured), and a critical one must not fail it
|
|
228
|
+
# either — "not applicable" says nothing about whether the app can
|
|
229
|
+
# serve. It stays in checks[] as "skipped" for visibility.
|
|
230
|
+
failures = failing_rows(rows)
|
|
189
231
|
return :ok if failures.empty?
|
|
190
232
|
|
|
191
|
-
# A SKIP MUST NEVER PRODUCE :unavailable, even for a critical check.
|
|
233
|
+
# A BUDGET SKIP MUST NEVER PRODUCE :unavailable, even for a critical check.
|
|
192
234
|
# Otherwise a slow *non-critical* check could exhaust the budget, leave
|
|
193
235
|
# the database check unrun, and pull a perfectly healthy instance out of
|
|
194
236
|
# rotation — a self-inflicted outage caused by the safety mechanism.
|
|
@@ -11,6 +11,16 @@ module StandardHealth
|
|
|
11
11
|
#
|
|
12
12
|
# { status: :fail, error: "connection refused" }
|
|
13
13
|
#
|
|
14
|
+
# or, when the check does not apply to this deployment (the feature it
|
|
15
|
+
# covers is not configured or not enforced):
|
|
16
|
+
#
|
|
17
|
+
# { status: :skipped }
|
|
18
|
+
#
|
|
19
|
+
# A self-reported :skipped is NEUTRAL in the roll-up (0.7.1): it never
|
|
20
|
+
# degrades or fails the tier, even for a critical check, but it still
|
|
21
|
+
# renders in `checks[]` as "skipped". Return :fail, not :skipped, for
|
|
22
|
+
# anything that should page.
|
|
23
|
+
#
|
|
14
24
|
# The `with_timing` helper wraps a block, captures latency, and converts
|
|
15
25
|
# any unhandled `StandardError` into a `:fail` row so subclasses don't
|
|
16
26
|
# have to repeat the pattern.
|
|
@@ -192,7 +192,9 @@ module StandardHealth
|
|
|
192
192
|
attr_accessor :default_check_timeout
|
|
193
193
|
|
|
194
194
|
# Budget across all checks, evaluated BEFORE each check starts. Checks not
|
|
195
|
-
# reached are reported :skipped
|
|
195
|
+
# reached are reported :skipped with `budget_exhausted: true` (unlike a
|
|
196
|
+
# check that reports :skipped itself, which is neutral), and such a skip
|
|
197
|
+
# alone floors the roll-up at
|
|
196
198
|
# :degraded — never :unavailable. Otherwise a slow NON-critical check could
|
|
197
199
|
# exhaust the budget, leave a critical check unrun, and pull a healthy app
|
|
198
200
|
# out of rotation. nil = no budget.
|
|
@@ -220,6 +222,7 @@ module StandardHealth
|
|
|
220
222
|
@env_spec = nil
|
|
221
223
|
@checks = []
|
|
222
224
|
@aggregate_checks = []
|
|
225
|
+
@diagnostics_assertions = []
|
|
223
226
|
@aggregate_endpoint = false
|
|
224
227
|
@aggregate_readiness_checks = true
|
|
225
228
|
@aggregate_circuits = true
|
|
@@ -368,6 +371,51 @@ module StandardHealth
|
|
|
368
371
|
registration
|
|
369
372
|
end
|
|
370
373
|
|
|
374
|
+
DiagnosticsAssertion = Struct.new(:name, :callable)
|
|
375
|
+
|
|
376
|
+
# Adds a runtime assertion to the doctor tier (`GET /diagnostics/env`),
|
|
377
|
+
# rendered under `assertions:` next to the env audit. For facts env
|
|
378
|
+
# presence can't prove: the live `statement_timeout`, the cache store a
|
|
379
|
+
# rate limiter actually resolved to, a pinned root's expiry. Replaces a
|
|
380
|
+
# host diagnostics controller that exists only to add these.
|
|
381
|
+
#
|
|
382
|
+
# c.register_diagnostics_assertion(:statement_timeout) do
|
|
383
|
+
# value = ActiveRecord::Base.connection.select_value("SHOW statement_timeout").to_s
|
|
384
|
+
# { status: value == "0" ? :warn : :ok, value: value, expected: "non-zero" }
|
|
385
|
+
# end
|
|
386
|
+
#
|
|
387
|
+
# The callable takes no arguments and returns a Hash with `status:`
|
|
388
|
+
# (`:ok`, `:warn` or `:error`) plus any detail keys. `name:` is set by
|
|
389
|
+
# the gem. It runs per request, on the authenticated tier only, never on
|
|
390
|
+
# /alive, /ready or the aggregate. A raise, a non-Hash, or an unknown
|
|
391
|
+
# status becomes an `:error` row (a raise is also reported to
|
|
392
|
+
# `Rails.error` as handled). An `:error` row makes the endpoint's
|
|
393
|
+
# `status` `incomplete`; `:warn` does not.
|
|
394
|
+
#
|
|
395
|
+
# Re-registering a name replaces it, so `configure` can safely re-run
|
|
396
|
+
# from `to_prepare` on reload.
|
|
397
|
+
def register_diagnostics_assertion(name, callable = nil, &block)
|
|
398
|
+
callable ||= block
|
|
399
|
+
unless callable.respond_to?(:call)
|
|
400
|
+
raise ArgumentError, "register_diagnostics_assertion(#{name.inspect}) needs a callable or a block"
|
|
401
|
+
end
|
|
402
|
+
|
|
403
|
+
assertion = DiagnosticsAssertion.new(name.to_sym, callable)
|
|
404
|
+
@diagnostics_assertions.reject! { |existing| existing.name == assertion.name }
|
|
405
|
+
@diagnostics_assertions << assertion
|
|
406
|
+
assertion
|
|
407
|
+
end
|
|
408
|
+
|
|
409
|
+
# @return [Array<DiagnosticsAssertion>] in registration order
|
|
410
|
+
def diagnostics_assertions
|
|
411
|
+
@diagnostics_assertions.dup
|
|
412
|
+
end
|
|
413
|
+
|
|
414
|
+
# Drop registered diagnostics assertions. Test hygiene.
|
|
415
|
+
def reset_diagnostics_assertions!
|
|
416
|
+
@diagnostics_assertions = []
|
|
417
|
+
end
|
|
418
|
+
|
|
371
419
|
# @return [Array<Registration>] aggregate-only checks
|
|
372
420
|
def aggregate_checks
|
|
373
421
|
@aggregate_checks.dup
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module StandardHealth
|
|
4
|
+
# Runs the host's `register_diagnostics_assertion` callables for the doctor
|
|
5
|
+
# tier (`/diagnostics/env`, behind `diagnostics_basic_auth`).
|
|
6
|
+
#
|
|
7
|
+
# An assertion is a runtime fact that env presence cannot express: the live
|
|
8
|
+
# `statement_timeout`, which cache store a limiter actually resolved to, how
|
|
9
|
+
# close a pinned certificate is to expiry. Each callable returns a Hash with
|
|
10
|
+
# at least `status:` (`:ok`, `:warn` or `:error`); any other keys (`value:`,
|
|
11
|
+
# `expected:`, ...) are rendered as-is. The gem sets `name:`.
|
|
12
|
+
#
|
|
13
|
+
# Public: a host with its own diagnostics controller can render
|
|
14
|
+
# `StandardHealth::DiagnosticsAssertions.run` itself.
|
|
15
|
+
module DiagnosticsAssertions
|
|
16
|
+
STATUSES = %i[ok warn error].freeze
|
|
17
|
+
|
|
18
|
+
module_function
|
|
19
|
+
|
|
20
|
+
# @return [Array<Hash>] one row per registered assertion, in registration
|
|
21
|
+
# order. Never raises: an assertion that raises, returns a non-Hash, or
|
|
22
|
+
# returns an unknown status becomes an `:error` row.
|
|
23
|
+
def run(config = StandardHealth.config)
|
|
24
|
+
config.diagnostics_assertions.map { |assertion| evaluate(assertion) }
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
# True when any row is `:error`. `:warn` is advisory.
|
|
28
|
+
def failing?(rows)
|
|
29
|
+
Array(rows).any? { |row| row[:status] == :error }
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def evaluate(assertion)
|
|
33
|
+
result = assertion.callable.call
|
|
34
|
+
return invalid(assertion.name, "returned #{result.class}, expected a Hash") unless result.is_a?(Hash)
|
|
35
|
+
|
|
36
|
+
row = result.to_h.transform_keys(&:to_sym)
|
|
37
|
+
status = row[:status].respond_to?(:to_sym) ? row[:status].to_sym : nil
|
|
38
|
+
return invalid(assertion.name, "returned status #{row[:status].inspect}, expected one of #{STATUSES.inspect}") unless STATUSES.include?(status)
|
|
39
|
+
|
|
40
|
+
{ name: assertion.name }.merge(row.except(:name)).merge(status: status)
|
|
41
|
+
rescue StandardError => e
|
|
42
|
+
report(e, assertion.name)
|
|
43
|
+
# The doctor tier is authenticated, so the message is shown (the /ready
|
|
44
|
+
# redaction rationale does not apply); the class is kept separately
|
|
45
|
+
# for grouping.
|
|
46
|
+
{ name: assertion.name, status: :error, error_class: e.class.name, error: e.message }
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
def invalid(name, detail)
|
|
50
|
+
{ name: name, status: :error, error: "diagnostics assertion #{name} #{detail}" }
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
def report(error, name)
|
|
54
|
+
return unless defined?(::Rails) && ::Rails.respond_to?(:error) && ::Rails.error
|
|
55
|
+
|
|
56
|
+
::Rails.error.report(error, handled: true, severity: :warning,
|
|
57
|
+
context: { diagnostics_assertion: name }, source: "standard_health")
|
|
58
|
+
rescue StandardError
|
|
59
|
+
nil
|
|
60
|
+
end
|
|
61
|
+
end
|
|
62
|
+
end
|
data/lib/standard_health.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: standard_health
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.7.1
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Jaryl Sim
|
|
@@ -15,14 +15,14 @@ dependencies:
|
|
|
15
15
|
requirements:
|
|
16
16
|
- - ">="
|
|
17
17
|
- !ruby/object:Gem::Version
|
|
18
|
-
version: '8.
|
|
18
|
+
version: '8.1'
|
|
19
19
|
type: :runtime
|
|
20
20
|
prerelease: false
|
|
21
21
|
version_requirements: !ruby/object:Gem::Requirement
|
|
22
22
|
requirements:
|
|
23
23
|
- - ">="
|
|
24
24
|
- !ruby/object:Gem::Version
|
|
25
|
-
version: '8.
|
|
25
|
+
version: '8.1'
|
|
26
26
|
description: StandardHealth is a mountable Rails engine providing /alive, /ready,
|
|
27
27
|
and /diagnostics/env endpoints, with a configuration block for registering custom
|
|
28
28
|
checks and a DSL for declaring required and recommended environment variables.
|
|
@@ -54,6 +54,7 @@ files:
|
|
|
54
54
|
- lib/standard_health/checks/solid_cache.rb
|
|
55
55
|
- lib/standard_health/checks/solid_queue.rb
|
|
56
56
|
- lib/standard_health/configuration.rb
|
|
57
|
+
- lib/standard_health/diagnostics_assertions.rb
|
|
57
58
|
- lib/standard_health/diagnostics_basic_auth.rb
|
|
58
59
|
- lib/standard_health/engine.rb
|
|
59
60
|
- lib/standard_health/env_spec.rb
|
|
@@ -87,7 +88,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
87
88
|
- !ruby/object:Gem::Version
|
|
88
89
|
version: '0'
|
|
89
90
|
requirements: []
|
|
90
|
-
rubygems_version: 4.0.
|
|
91
|
+
rubygems_version: 4.0.10
|
|
91
92
|
specification_version: 4
|
|
92
93
|
summary: A drop-in health check and environment-spec engine for Rails 8 host apps.
|
|
93
94
|
test_files: []
|