standard_circuit 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +42 -0
- data/README.md +174 -10
- data/app/controllers/standard_circuit/health_controller.rb +28 -0
- data/lib/generators/standard_circuit/install/install_generator.rb +11 -22
- data/lib/generators/standard_circuit/install/templates/initializer.rb.tt +23 -1
- data/lib/standard_circuit/active_storage/s3_service.rb +9 -1
- data/lib/standard_circuit/adapter_errors/faraday.rb +8 -0
- data/lib/standard_circuit/adapter_errors/postmark.rb +31 -0
- data/lib/standard_circuit/config.rb +95 -5
- data/lib/standard_circuit/controller_support.rb +3 -11
- data/lib/standard_circuit/deprecator.rb +12 -0
- data/lib/standard_circuit/engine.rb +7 -0
- data/lib/standard_circuit/error_taxonomies.rb +36 -0
- data/lib/standard_circuit/health.rb +4 -4
- data/lib/standard_circuit/health_controller.rb +20 -25
- data/lib/standard_circuit/mailer/retry.rb +121 -0
- data/lib/standard_circuit/presets.rb +90 -0
- data/lib/standard_circuit/version.rb +1 -1
- data/lib/standard_circuit.rb +15 -0
- metadata +34 -2
- data/lib/generators/standard_circuit/install/templates/health_initializer.rb.tt +0 -25
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 60bd3d9dbd17cad5bdf8062be096c8f7b1d8baa00f167e25f1ea96581a078cca
|
|
4
|
+
data.tar.gz: 5abdcf7eb5bf00b65bc0b724d3ced093f71f6ee1cb59a8d594c6a8d3944998e1
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: f88dc067ff12e4651a0f6f5317d55f22f5a9b78507d0bc31bf7fc737fa2dc3a7efa09853b64bb89f3fb608660eb4da79b126921d90a87e7ad9396bcb81cb317b
|
|
7
|
+
data.tar.gz: 1ed24fba0bdd0e373dfbe4cc4a18cc9e22675db527c109b151395ba2681d403ea64dbf0a12672b53250edfd8886aa66be1cedd42a36099833c23562a828b89c0
|
data/CHANGELOG.md
CHANGED
|
@@ -2,8 +2,50 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
|
4
4
|
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
5
8
|
## [Unreleased]
|
|
6
9
|
|
|
10
|
+
## [0.4.0] - 2026-09-24
|
|
11
|
+
|
|
12
|
+
A developer-experience release that moves code copy-pasted across the five consumer apps into the gem. Everything is additive: behaviour is unchanged unless you opt in, and deprecated APIs keep working until 0.5.
|
|
13
|
+
|
|
14
|
+
### Added
|
|
15
|
+
- **`ErrorTaxonomies::Postmark` / `AdapterErrors::Postmark`.** `server_errors` is `[Postmark::HttpServerError, Postmark::TimeoutError]` and `caller_errors` is `[Postmark::ApiInputError, Postmark::InvalidApiKeyError]`. Both return `[]` when the postmark gem isn't loaded. `ErrorTaxonomies.default_skipped_for` now also covers Postmark, because `ApiInputError` and `InvalidApiKeyError` subclass the tracked `HttpServerError`. A circuit that tracks `HttpServerError` and has no `skipped_errors:` now skips them by default, the same way AWS caller errors are skipped since 0.3.1. None of the consumers is affected: all three Postmark circuits pass `skipped_errors:` explicitly.
|
|
16
|
+
- **`Config#register_preset(:postmark | :s3, name:, **overrides)`.** It replaces the identical `:postmark` registrations in jumpdrive-web, fundbright-web and luminality-web, and the identical `register_prefix(:s3, ...)` in luminality-web, sidekick-web and nutripod-web. Each preset requires its SDK first (`postmark`, `aws-sdk-s3`) and raises `ArgumentError` if the SDK is missing. This matters when `aws-sdk-s3` is declared `require: false`: in that case `ErrorTaxonomies::Aws.tracked` evaluated at configure time contains only network errors. There is deliberately no `:stripe` preset, because the apps' Stripe registrations differ.
|
|
17
|
+
- **Opt-in `config.mailer_retry`** (`true`, or `{ wait:, attempts:, jitter: }` over the defaults `90 / 5 / 0.15`; off by default). It installs `retry_on StandardCircuit::Mailer::CircuitOpenError` on `ActionMailer::MailDeliveryJob`. The install is idempotent and reload-safe, and it stands down if a `CircuitOpenError` handler is already present. On exhaustion it writes an error log line, sends a Sentry `:error` event and emits a new `standard_circuit.mailer.retries_exhausted` event, all carrying recipient domains only. It replaces `config/initializers/mail_delivery_retry.rb` in jumpdrive-web, sidekick-web and nutripod-web (their provider-specific retries and discards stay in the apps), and the `CircuitOpenError` `retry_on` in fundbright-web's `ApplicationMailDeliveryJob`. The Sentry fingerprint is new, so the first exhaustion after switching opens a new issue.
|
|
18
|
+
- `StandardCircuit.deprecator`, registered as `Rails.application.deprecators[:standard_circuit]`.
|
|
19
|
+
- `Config#add_notifier(notifier, key:)`: an optional `key:` for registering several instances of one notifier class (see Changed).
|
|
20
|
+
- README sections: presets, mail delivery and `mailer_retry`, calling `configure` more than once, supported test API (`force_open`, `force_closed`, `reset_force!`, `reset!`, `standard_circuit/rspec`), and deprecations. Each feature has a "replace your host code with" snippet.
|
|
21
|
+
|
|
22
|
+
### Changed
|
|
23
|
+
- **`StandardCircuit::HealthController` is autoloaded by the engine.** It moved to `app/controllers/standard_circuit/health_controller.rb`, so drawing `get "/health", to: "standard_circuit/health#show"` is all a host needs. The boot probe spec now covers autoload, eager load and both legacy `require` placements. The install generator's `--with-health-endpoint` no longer writes `config/initializers/standard_circuit_health.rb`, which only ever contained the require; it prints the route hint.
|
|
24
|
+
- **Repeated `configure` calls no longer stack notifiers.** Every consumer calls `configure` from `to_prepare`, which re-runs on every code reload. `add_notifier` now replaces an existing entry with the same identity in place: the same explicit `key:`, otherwise the same class name, or for procs and methods the same source location. The newest instance wins. `extra_notifiers` is deliberately not reset on each `configure`, because apps split configuration across several calls and a reset would silently drop notifiers added earlier.
|
|
25
|
+
- **gemspec declares `actionmailer` and `actionpack` (`>= 8.0`).** `require "standard_circuit"` has always loaded both, but only `railties` was declared, and railties doesn't pull in actionmailer. `activestorage` stays undeclared because the S3 service is only ever loaded by ActiveStorage's own configurator. The SDKs stay optional. The gem package now includes `app/**`.
|
|
26
|
+
|
|
27
|
+
### Deprecated
|
|
28
|
+
Each item below still works and warns through `StandardCircuit.deprecator`. All are removed in 0.5.
|
|
29
|
+
- `require "standard_circuit/health_controller"`: remove it, the controller is autoloaded. Carried in `config/routes.rb` by jumpdrive-web, fundbright-web and nutripod-web.
|
|
30
|
+
- `StandardCircuit.health_snapshot` and `StandardCircuit.health_overall`: use `StandardCircuit.health_report[:circuits]` / `[:status]`. No consumer calls them.
|
|
31
|
+
- `StandardCircuit::AdapterErrors::Faraday.caller_errors`: `Faraday::ClientError` is never tracked, so skipping it is a no-op. No consumer calls it. (`Aws.caller_errors` is kept because `default_skipped_for` uses it.)
|
|
32
|
+
- `StandardCircuit::ActiveStorage::S3Service`: use `ActiveStorage::Service::StandardCircuitS3Service`, or `service: StandardCircuitS3` in storage.yml. No consumer references it.
|
|
33
|
+
|
|
34
|
+
## [0.3.1] - 2026-09-24
|
|
35
|
+
|
|
36
|
+
Two behaviour changes — both are bug fixes, but both change numbers or breaker behaviour you may be alerting on.
|
|
37
|
+
|
|
38
|
+
### Fixed
|
|
39
|
+
- **Circuit-open requests were counted twice in `<metric_prefix>.request`.** `Runner` already emits `<metric_prefix>.request{status: circuit_open}` when a call is rejected by a tripped or forced-open circuit; `ControllerSupport#handle_circuit_open` then emitted the same metric again when the `Stoplight::Error::RedLight` reached a controller. The controller-side emission (and the private `emit_circuit_open_metric` helper) is removed; fallback dispatch is unchanged. **Behaviour change:** `circuit_open` request counts for rejections that reach a controller drop by half, back to the true value — dashboards and alert thresholds tuned against the doubled number should be revisited.
|
|
40
|
+
- **S3 caller errors tripped the S3 breaker.** `ErrorTaxonomies::Aws.tracked` includes `Aws::Errors::ServiceError`, which is the superclass of `Aws::S3::Errors::AccessDenied` and `NoSuchKey` as well as of the dynamically generated 5xx errors, so a burst of missing-key lookups or permission errors opened the circuit. New `ErrorTaxonomies.default_skipped_for(tracked)` returns the AWS caller errors whenever the tracked list covers them, and `register` / `register_prefix` now use it as the default `skipped_errors`. **Behaviour change:** AWS circuits registered without `skipped_errors:` no longer count `AccessDenied` / `NoSuchKey` toward the threshold; 5xx (`ServiceUnavailable` etc.) and `Seahorse::Client::NetworkingError` still trip. An explicit `skipped_errors:` — including `[]` — always wins, and circuits for other adapters keep defaulting to `[]`.
|
|
41
|
+
|
|
42
|
+
### Added
|
|
43
|
+
- Specs for `NetworkErrors` and the Stripe / AWS / Faraday / SMTP `AdapterErrors` modules.
|
|
44
|
+
- README section on error taxonomies and the AWS `skipped_errors` default; the initializer template notes it too.
|
|
45
|
+
|
|
46
|
+
### Documentation
|
|
47
|
+
- **Consumer list corrected in `CLAUDE.md`: this gem has five consumers, not four.** `sidekick-web` was missing. The same entry also claimed the `workspace-os` → `jumpdrive-web` directory rename "was deferred" and pointed at `~/Workspace/rarebit-one/workspace-os`; that rename completed 2026-07-14 and the old husk is gone, so an agent following the note was looking in a directory that no longer exists. Verified against the canonical matrix in the workspace's `rollout-gem/SKILL.md`, which the new advisory `check-gem-family-drift.sh` now diffs this list against on every sweep.
|
|
48
|
+
|
|
7
49
|
## [0.3.0] - 2026-07-30
|
|
8
50
|
|
|
9
51
|
### Added
|
data/README.md
CHANGED
|
@@ -5,10 +5,10 @@ Circuit breaker primitives for Rails apps, built on [stoplight](https://github.c
|
|
|
5
5
|
Wraps the upstream `stoplight` gem with:
|
|
6
6
|
|
|
7
7
|
- Opinionated default error taxonomy (network errors track; caller/config errors do not)
|
|
8
|
-
- SDK-specific adapter error bundles (Stripe, AWS, Faraday, SMTP)
|
|
8
|
+
- SDK-specific adapter error bundles (Stripe, AWS, Faraday, SMTP, Postmark) and circuit presets (`:postmark`, `:s3`)
|
|
9
9
|
- Rails event emission (`standard_circuit.circuit.{opened,closed,degraded,fallback_invoked,registered}`) with built-in Logger, Sentry, and Sentry::Metrics subscribers
|
|
10
10
|
- ActiveStorage S3 adapter with per-bucket circuit keying
|
|
11
|
-
- Generic ActionMailer delivery-method wrapper (supports both instance and symbol `underlying:` forms)
|
|
11
|
+
- Generic ActionMailer delivery-method wrapper (supports both instance and symbol `underlying:` forms) with opt-in `mailer_retry` for `deliver_later` jobs
|
|
12
12
|
- Controller concern for standardized 503 responses on `Stoplight::Error::RedLight`
|
|
13
13
|
- Test helpers (`force_open`, `force_closed`, `reset_force!`) with RSpec auto-cleanup
|
|
14
14
|
|
|
@@ -27,10 +27,8 @@ bundle add standard_circuit
|
|
|
27
27
|
rails g standard_circuit:install
|
|
28
28
|
```
|
|
29
29
|
|
|
30
|
-
Pass `--with-health-endpoint` to also
|
|
31
|
-
`config/
|
|
32
|
-
health controller); the generator prints the matching route line for you to
|
|
33
|
-
add to `config/routes.rb`.
|
|
30
|
+
Pass `--with-health-endpoint` to also print the route line for the health
|
|
31
|
+
endpoint, for you to add to `config/routes.rb`.
|
|
34
32
|
|
|
35
33
|
The generator is idempotent — re-running skips an existing initializer
|
|
36
34
|
unless you pass `--force`.
|
|
@@ -58,6 +56,58 @@ StandardCircuit.run(:stripe) do
|
|
|
58
56
|
end
|
|
59
57
|
```
|
|
60
58
|
|
|
59
|
+
## Error taxonomies
|
|
60
|
+
|
|
61
|
+
`tracked_errors` decide what counts toward tripping a circuit; `skipped_errors` are re-raised without counting (and win over `tracked_errors` when a class matches both). `StandardCircuit::ErrorTaxonomies::<Adapter>.tracked` combines `NetworkErrors.defaults` with the adapter's server-side errors for `Stripe`, `Smtp`, `Aws`, `Faraday`, and `Postmark`; `StandardCircuit::AdapterErrors::<Adapter>.caller_errors` lists the adapter's caller-side (4xx-style) errors.
|
|
62
|
+
|
|
63
|
+
When `skipped_errors:` is omitted it defaults to `[]` — except when the tracked list covers the AWS or Postmark caller errors. (Postmark's `ApiInputError` and `InvalidApiKeyError` subclass `Postmark::HttpServerError`, so a circuit tracking `ErrorTaxonomies::Postmark.tracked` without `skipped_errors:` defaults to skipping them.) AWS 5xx responses are dynamically generated `Aws::Errors::ServiceError` subclasses, so `ErrorTaxonomies::Aws.tracked` has to track `ServiceError` itself, which is also the superclass of `Aws::S3::Errors::AccessDenied` and `NoSuchKey`. So for such circuits `skipped_errors` defaults to `AdapterErrors::Aws.caller_errors` (via `ErrorTaxonomies.default_skipped_for`), and a burst of missing-key lookups or permission errors no longer trips the S3 breaker:
|
|
64
|
+
|
|
65
|
+
```ruby
|
|
66
|
+
StandardCircuit.configure do |c|
|
|
67
|
+
# skipped_errors defaults to [Aws::S3::Errors::NoSuchKey, Aws::S3::Errors::AccessDenied]
|
|
68
|
+
c.register_prefix(:s3, tracked_errors: StandardCircuit::ErrorTaxonomies::Aws.tracked)
|
|
69
|
+
|
|
70
|
+
# An explicit skipped_errors — even [] — always wins over the default.
|
|
71
|
+
# c.register(:s3_strict, tracked_errors: StandardCircuit::ErrorTaxonomies::Aws.tracked, skipped_errors: [])
|
|
72
|
+
end
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
## Presets
|
|
76
|
+
|
|
77
|
+
Some integrations were registered identically, line for line, in several apps. `c.register_preset` registers them with the shared settings; any `register` option you pass alongside wins, and `name:` renames the circuit.
|
|
78
|
+
|
|
79
|
+
| Preset | Registers | Settings |
|
|
80
|
+
|--------|-----------|----------|
|
|
81
|
+
| `:postmark` | `register(:postmark, ...)` | threshold 3, cool-off 60s, `:standard`, `ErrorTaxonomies::Postmark.tracked`, skips `AdapterErrors::Postmark.caller_errors` |
|
|
82
|
+
| `:s3` | `register_prefix(:s3, ...)` — matches the `s3_<bucket>` circuits `StandardCircuitS3` opens | threshold 3, cool-off 30s, `:standard`, `ErrorTaxonomies::Aws.tracked`, skips `NoSuchKey` / `AccessDenied` |
|
|
83
|
+
|
|
84
|
+
Each preset requires its SDK (`postmark`, `aws-sdk-s3`) before building the error lists and raises `ArgumentError` if the gem is missing. That matters for `gem "aws-sdk-s3", require: false`: `ErrorTaxonomies::Aws.tracked` returns only network errors when the SDK hasn't been loaded yet at configure time, which quietly leaves S3 5xx responses untracked.
|
|
85
|
+
|
|
86
|
+
Replace your host code with:
|
|
87
|
+
|
|
88
|
+
```ruby
|
|
89
|
+
# Before — config/initializers/standard_circuit.rb
|
|
90
|
+
c.register(:postmark,
|
|
91
|
+
threshold: 3,
|
|
92
|
+
cool_off_time: 60,
|
|
93
|
+
criticality: :standard,
|
|
94
|
+
tracked_errors: StandardCircuit::NetworkErrors.defaults +
|
|
95
|
+
[ Postmark::HttpServerError, Postmark::TimeoutError ],
|
|
96
|
+
skipped_errors: [ Postmark::ApiInputError, Postmark::InvalidApiKeyError ])
|
|
97
|
+
|
|
98
|
+
c.register_prefix(:s3,
|
|
99
|
+
threshold: 3,
|
|
100
|
+
cool_off_time: 30,
|
|
101
|
+
criticality: :standard,
|
|
102
|
+
tracked_errors: StandardCircuit::ErrorTaxonomies::Aws.tracked)
|
|
103
|
+
|
|
104
|
+
# After
|
|
105
|
+
c.register_preset(:postmark)
|
|
106
|
+
c.register_preset(:s3)
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
There is deliberately no `:stripe` preset — the apps that wrap Stripe disagree on threshold, criticality, and skips, so `ErrorTaxonomies::Stripe.tracked` stays the shared piece.
|
|
110
|
+
|
|
61
111
|
## Circuit state storage (`data_store`)
|
|
62
112
|
|
|
63
113
|
Circuit state (failure counts, colors, locks) lives in a Stoplight data store. StandardCircuit defaults to `Stoplight::DataStore::Memory.new`, which is **per-process**: each Puma worker, Sidekiq/SolidQueue worker, and console gets its own independent view of every circuit.
|
|
@@ -140,6 +190,85 @@ StandardCircuit.configure do |c|
|
|
|
140
190
|
end
|
|
141
191
|
```
|
|
142
192
|
|
|
193
|
+
## Mail delivery
|
|
194
|
+
|
|
195
|
+
The `:standard_circuit` delivery method wraps any registered ActionMailer delivery method in a circuit and raises `StandardCircuit::Mailer::CircuitOpenError` (instead of attempting the send) while that circuit is open:
|
|
196
|
+
|
|
197
|
+
```ruby
|
|
198
|
+
# config/environments/production.rb
|
|
199
|
+
config.action_mailer.delivery_method = :standard_circuit
|
|
200
|
+
config.action_mailer.standard_circuit_settings = {
|
|
201
|
+
underlying: :postmark,
|
|
202
|
+
underlying_settings: { api_token: ENV.fetch("POSTMARK_API_TOKEN") },
|
|
203
|
+
circuit: :postmark
|
|
204
|
+
}
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
### Retrying `deliver_later` while the circuit is open (`mailer_retry`)
|
|
208
|
+
|
|
209
|
+
Without a retry, a `deliver_later` job that runs during an outage fails once and the email is lost. Opt in and the gem installs `retry_on CircuitOpenError` on `ActionMailer::MailDeliveryJob`:
|
|
210
|
+
|
|
211
|
+
```ruby
|
|
212
|
+
StandardCircuit.configure do |c|
|
|
213
|
+
c.mailer_retry = true # wait: 90, attempts: 5, jitter: 0.15
|
|
214
|
+
# c.mailer_retry = { wait: 120, attempts: 8 } # partial overrides; wait: takes anything retry_on does
|
|
215
|
+
end
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
- **Off by default.** Nothing is installed unless you set it.
|
|
219
|
+
- **Keep `wait` longer than the mail circuit's `cool_off_time`** (the `:postmark` preset uses 60s), or retries land on a circuit that is still open.
|
|
220
|
+
- **Reload-safe and idempotent.** Calling `configure` from `to_prepare` on every reload installs the handler once. If a `retry_on CircuitOpenError` is already on the job (for example your old initializer, during migration), the gem leaves it alone. The first install wins, so changing the options later needs a restart.
|
|
221
|
+
- **Subclasses inherit it.** A custom `self.delivery_job = MyJob < ActionMailer::MailDeliveryJob` gets the handler, and a `retry_on CircuitOpenError` declared on the subclass takes precedence.
|
|
222
|
+
- **On exhaustion** it writes one `error` log line, sends a Sentry `:error` event (when `sentry_enabled` and Sentry is initialized, fingerprinted by mailer and action), and emits `standard_circuit.mailer.retries_exhausted`. Every payload holds only `mailer_class`, `mail_action`, `recipient_domains`, `job_id` and `executions`: recipient **domains**, never addresses or subjects.
|
|
223
|
+
- Only `CircuitOpenError` is retried. Errors from the provider while the circuit is closed, such as a SendGrid 429 or a Postmark 422, keep their existing behaviour. Add your own `retry_on` / `discard_on` for those.
|
|
224
|
+
|
|
225
|
+
Replace your host code with:
|
|
226
|
+
|
|
227
|
+
```ruby
|
|
228
|
+
# Before — config/initializers/mail_delivery_retry.rb (~40–60 lines)
|
|
229
|
+
Rails.application.config.to_prepare do
|
|
230
|
+
next if ActionMailer::MailDeliveryJob.rescue_handlers.any? { |h| h.first == StandardCircuit::Mailer::CircuitOpenError.name }
|
|
231
|
+
|
|
232
|
+
ActionMailer::MailDeliveryJob.retry_on(StandardCircuit::Mailer::CircuitOpenError,
|
|
233
|
+
wait: 90.seconds, attempts: 5, jitter: 0.15) do |job, error|
|
|
234
|
+
# ...recipient-domain extraction, Rails.logger.error, Sentry.capture_message...
|
|
235
|
+
end
|
|
236
|
+
end
|
|
237
|
+
|
|
238
|
+
# After — inside your existing StandardCircuit.configure block
|
|
239
|
+
c.mailer_retry = true
|
|
240
|
+
# or, keeping ENV tunables:
|
|
241
|
+
c.mailer_retry = {
|
|
242
|
+
wait: ENV.fetch("SENDGRID_CIRCUIT_RETRY_WAIT", 90).to_i,
|
|
243
|
+
attempts: ENV.fetch("SENDGRID_CIRCUIT_RETRY_ATTEMPTS", 5).to_i
|
|
244
|
+
}
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
The Sentry fingerprint changes to `["standard_circuit-mailer-retries-exhausted", mailer, action]`, so the first exhaustion after switching opens a new Sentry issue rather than regrouping into your old one.
|
|
248
|
+
|
|
249
|
+
### Calling `configure` more than once
|
|
250
|
+
|
|
251
|
+
`configure` is safe to call repeatedly. Apps register circuits from `to_prepare` because it's the first point where autoloaded error classes are available, and `to_prepare` re-runs on every code reload. Each call re-applies the config and rebuilds the subscribers. `add_notifier` is idempotent: adding "the same" notifier again replaces the earlier entry in place and does not stack a duplicate. Two notifiers are the same when they share an explicit `key:`, or else the same class name (a reloaded class is a new object with the same name), or for lambdas and methods the same source location. The newest instance wins. To register two instances of one class, give them distinct keys:
|
|
252
|
+
|
|
253
|
+
```ruby
|
|
254
|
+
c.add_notifier(WebhookNotifier.new(ops_url), key: :ops)
|
|
255
|
+
c.add_notifier(WebhookNotifier.new(audit_url), key: :audit)
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
Replace your host code with:
|
|
259
|
+
|
|
260
|
+
```ruby
|
|
261
|
+
# Before — hand-rolled guard against stacking a notifier on every reload
|
|
262
|
+
unless c.extra_notifiers.any? { |n| n.class.name == "CircuitAuditNotifier" }
|
|
263
|
+
c.add_notifier(CircuitAuditNotifier.new)
|
|
264
|
+
end
|
|
265
|
+
|
|
266
|
+
# After
|
|
267
|
+
c.add_notifier(CircuitAuditNotifier.new)
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
Circuit registrations (`register` / `register_prefix` / `register_preset`) were already idempotent: the same name overwrites.
|
|
271
|
+
|
|
143
272
|
## Streaming and non-controller contexts
|
|
144
273
|
|
|
145
274
|
`ControllerSupport.circuit_open_fallback` only works for non-streaming responses — once a `Live` controller has flushed any output, Rails can't render an error template over the wire. For a streaming controller, catch `Stoplight::Error::RedLight` *inside* the streaming proc and write a degraded payload before the stream closes:
|
|
@@ -174,19 +303,28 @@ Same pattern applies in background jobs (where `circuit_open_fallback` doesn't h
|
|
|
174
303
|
|
|
175
304
|
## Health endpoint
|
|
176
305
|
|
|
177
|
-
StandardCircuit ships
|
|
306
|
+
StandardCircuit ships a controller that renders `StandardCircuit.health_report` as JSON. It returns 503 when the rolled-up status is `:critical` (so orchestrators pull the instance out of rotation) and 200 otherwise.
|
|
178
307
|
|
|
179
|
-
|
|
308
|
+
The controller lives in the engine's `app/controllers`, so it is autoloaded. The route is the only opt-in, and apps that don't draw it never load the controller.
|
|
180
309
|
|
|
181
310
|
```ruby
|
|
182
311
|
# config/routes.rb
|
|
183
|
-
require "standard_circuit/health_controller"
|
|
184
|
-
|
|
185
312
|
Rails.application.routes.draw do
|
|
186
313
|
get "/health", to: "standard_circuit/health#show"
|
|
187
314
|
end
|
|
188
315
|
```
|
|
189
316
|
|
|
317
|
+
Replace your host code with:
|
|
318
|
+
|
|
319
|
+
```ruby
|
|
320
|
+
# Before — top of config/routes.rb (or config/initializers/standard_circuit_health.rb)
|
|
321
|
+
require "standard_circuit/health_controller"
|
|
322
|
+
|
|
323
|
+
# After — delete the line (and the initializer, if that's all it contained).
|
|
324
|
+
```
|
|
325
|
+
|
|
326
|
+
The old `require` still works in 0.4 but emits a deprecation through `Rails.application.deprecators[:standard_circuit]`. It will be removed in 0.5.
|
|
327
|
+
|
|
190
328
|
The controller inherits from `ActionController::API` to sidestep app-level filters (authentication, bootstrap redirects, etc.) so probes can call it anonymously.
|
|
191
329
|
|
|
192
330
|
**If your app also mounts `StandardHealth::Engine` at `/health`, draw the aggregate route first:**
|
|
@@ -198,6 +336,32 @@ mount StandardHealth::Engine => "/health", as: :standard_health
|
|
|
198
336
|
|
|
199
337
|
`StandardHealth::Engine` registers sub-paths only (`/alive`, `/ready`, `/diagnostics/env`) — it never serves the aggregate tier itself. An app that mounts the engine and assumes `/health` is covered silently has no aggregate tier at all, with no boot error and no failing route spec to reveal it. The ordering is load-bearing; draw the aggregate route explicitly, first.
|
|
200
338
|
|
|
339
|
+
## Test API
|
|
340
|
+
|
|
341
|
+
These are supported public API for host test suites, not internals:
|
|
342
|
+
|
|
343
|
+
| Method | What it does |
|
|
344
|
+
|--------|--------------|
|
|
345
|
+
| `StandardCircuit.force_open(name) { ... }` | Treat `name` as open for the block (or until `reset_force!` without a block): `run` raises `Stoplight::Error::RedLight`, or returns the fallback. Emits the same `run.completed` / `fallback_invoked` events as a real open circuit. |
|
|
346
|
+
| `StandardCircuit.force_closed(name) { ... }` | Bypass the circuit for the block — `run` just yields. No events. |
|
|
347
|
+
| `StandardCircuit.reset_force!` | Clear every forced state. |
|
|
348
|
+
| `StandardCircuit.reset!` | Clear the light cache and forced states, and swap in a fresh `Memory` data store (left alone when the store is Redis). |
|
|
349
|
+
| `require "standard_circuit/rspec"` | Adds a `before(:each)` that runs `reset!` and tears down subscribers. Circuit registrations are kept. |
|
|
350
|
+
|
|
351
|
+
All of these are **process-local**. They are test and console tools, not an operational kill switch (see `data_store` above).
|
|
352
|
+
|
|
353
|
+
## Deprecations
|
|
354
|
+
|
|
355
|
+
0.4 deprecates the following. Each still works and warns through `Rails.application.deprecators[:standard_circuit]`, so your app's `config.active_support.deprecation` setting applies. They will be removed in 0.5.
|
|
356
|
+
|
|
357
|
+
| Deprecated | Use instead |
|
|
358
|
+
|------------|-------------|
|
|
359
|
+
| `require "standard_circuit/health_controller"` | nothing — the controller is autoloaded |
|
|
360
|
+
| `StandardCircuit.health_snapshot` | `StandardCircuit.health_report[:circuits]` |
|
|
361
|
+
| `StandardCircuit.health_overall` | `StandardCircuit.health_report[:status]` |
|
|
362
|
+
| `StandardCircuit::AdapterErrors::Faraday.caller_errors` | drop it — `Faraday::ClientError` is never tracked, so skipping it is a no-op |
|
|
363
|
+
| `StandardCircuit::ActiveStorage::S3Service` | `ActiveStorage::Service::StandardCircuitS3Service` / `service: StandardCircuitS3` |
|
|
364
|
+
|
|
201
365
|
## License
|
|
202
366
|
|
|
203
367
|
MIT
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
module StandardCircuit
|
|
2
|
+
# Health-check controller. Renders +StandardCircuit.health_report+ as JSON
|
|
3
|
+
# and returns 503 when the rolled-up status is +:critical+ so upstream
|
|
4
|
+
# orchestrators pull the instance out of rotation. :degraded and :ok both
|
|
5
|
+
# return 200 — the app can still serve most traffic.
|
|
6
|
+
#
|
|
7
|
+
# Autoloaded by the engine (it lives under app/controllers), so a host only
|
|
8
|
+
# draws the route — no `require` needed:
|
|
9
|
+
#
|
|
10
|
+
# # config/routes.rb
|
|
11
|
+
# Rails.application.routes.draw do
|
|
12
|
+
# get "/health", to: "standard_circuit/health#show"
|
|
13
|
+
# end
|
|
14
|
+
#
|
|
15
|
+
# The pre-0.4 `require "standard_circuit/health_controller"` still works but
|
|
16
|
+
# is deprecated (see lib/standard_circuit/health_controller.rb).
|
|
17
|
+
#
|
|
18
|
+
# Inherits from +ActionController::API+ to sidestep any ApplicationController
|
|
19
|
+
# filters (authentication, bootstrap redirects, etc.) — health probes must be
|
|
20
|
+
# callable anonymously from load balancers and uptime monitors.
|
|
21
|
+
class HealthController < ::ActionController::API
|
|
22
|
+
def show
|
|
23
|
+
report = StandardCircuit.health_report
|
|
24
|
+
http_status = report[:status] == :critical ? :service_unavailable : :ok
|
|
25
|
+
render json: report, status: http_status
|
|
26
|
+
end
|
|
27
|
+
end
|
|
28
|
+
end
|
|
@@ -7,10 +7,10 @@ module StandardCircuit
|
|
|
7
7
|
# By default, writes config/initializers/standard_circuit.rb with
|
|
8
8
|
# commented-out examples covering the public Config DSL.
|
|
9
9
|
#
|
|
10
|
-
# When +--with-health-endpoint+ is passed, also
|
|
11
|
-
# config/
|
|
12
|
-
#
|
|
13
|
-
#
|
|
10
|
+
# When +--with-health-endpoint+ is passed, also prints the route line the
|
|
11
|
+
# host should add to config/routes.rb to expose the endpoint. (Before 0.4
|
|
12
|
+
# it also wrote config/initializers/standard_circuit_health.rb to require
|
|
13
|
+
# the controller; the engine now autoloads it, so no file is needed.)
|
|
14
14
|
#
|
|
15
15
|
# Idempotent: re-running on an existing initializer logs and skips. Pass
|
|
16
16
|
# +--force+ to overwrite.
|
|
@@ -23,17 +23,16 @@ module StandardCircuit
|
|
|
23
23
|
covering circuit registration, prefix registration, and notifier
|
|
24
24
|
wiring.
|
|
25
25
|
|
|
26
|
-
Pass --with-health-endpoint to also
|
|
27
|
-
config/
|
|
28
|
-
|
|
29
|
-
config/routes.rb.
|
|
26
|
+
Pass --with-health-endpoint to also print the route line to add to
|
|
27
|
+
config/routes.rb. The health controller is autoloaded by the engine,
|
|
28
|
+
so the route is all you need.
|
|
30
29
|
|
|
31
30
|
The generator is idempotent — already-installed initializers are
|
|
32
31
|
skipped with a clear message. Pass --force to overwrite.
|
|
33
32
|
DESC
|
|
34
33
|
|
|
35
34
|
class_option :with_health_endpoint, type: :boolean, default: false,
|
|
36
|
-
desc: "Also
|
|
35
|
+
desc: "Also print the config/routes.rb line for the health endpoint"
|
|
37
36
|
|
|
38
37
|
def create_initializer_file
|
|
39
38
|
path = "config/initializers/standard_circuit.rb"
|
|
@@ -45,25 +44,15 @@ module StandardCircuit
|
|
|
45
44
|
template "initializer.rb.tt", path
|
|
46
45
|
end
|
|
47
46
|
|
|
48
|
-
def create_health_initializer_file
|
|
49
|
-
return unless options[:with_health_endpoint]
|
|
50
|
-
|
|
51
|
-
path = "config/initializers/standard_circuit_health.rb"
|
|
52
|
-
if File.exist?(File.join(destination_root, path)) && !options[:force]
|
|
53
|
-
say_status("skip", "#{path} already present, skipping (use --force to overwrite)", :yellow)
|
|
54
|
-
else
|
|
55
|
-
template "health_initializer.rb.tt", path
|
|
56
|
-
end
|
|
57
|
-
end
|
|
58
|
-
|
|
59
47
|
def print_health_route_hint
|
|
60
48
|
return unless options[:with_health_endpoint]
|
|
61
49
|
|
|
62
50
|
say ""
|
|
63
51
|
say "=" * 79
|
|
64
|
-
say "StandardCircuit health endpoint
|
|
52
|
+
say "StandardCircuit health endpoint"
|
|
65
53
|
say ""
|
|
66
|
-
say "Add the following to config/routes.rb to expose the endpoint
|
|
54
|
+
say "Add the following to config/routes.rb to expose the endpoint (the"
|
|
55
|
+
say "controller is autoloaded by the engine — no require needed):"
|
|
67
56
|
say ""
|
|
68
57
|
say ' get "/health", to: "standard_circuit/health#show"'
|
|
69
58
|
say ""
|
|
@@ -37,6 +37,14 @@ StandardCircuit.configure do |config|
|
|
|
37
37
|
# Default: nil
|
|
38
38
|
# config.logger = Rails.logger
|
|
39
39
|
|
|
40
|
+
# Retry ActionMailer::MailDeliveryJob on
|
|
41
|
+
# StandardCircuit::Mailer::CircuitOpenError while the mail circuit is open.
|
|
42
|
+
# Keep `wait` longer than the mail circuit's cool_off_time. On exhaustion:
|
|
43
|
+
# error log + Sentry :error (recipient domains only).
|
|
44
|
+
# true -> { wait: 90, attempts: 5, jitter: 0.15 }
|
|
45
|
+
# Default: nil (off)
|
|
46
|
+
# config.mailer_retry = true
|
|
47
|
+
|
|
40
48
|
# Stoplight data store. Default: in-memory (per-process). Use a shared
|
|
41
49
|
# store (e.g. Stoplight::DataStore::Redis) when you want circuit state
|
|
42
50
|
# shared across processes/hosts.
|
|
@@ -54,7 +62,10 @@ StandardCircuit.configure do |config|
|
|
|
54
62
|
# tracked_errors — error classes that count toward the threshold
|
|
55
63
|
# (default: StandardCircuit::NetworkErrors.defaults)
|
|
56
64
|
# skipped_errors — error classes that bypass the circuit (re-raised
|
|
57
|
-
# without counting — typically caller/validation errors
|
|
65
|
+
# without counting — typically caller/validation errors;
|
|
66
|
+
# default: [], or the AWS / Postmark caller errors
|
|
67
|
+
# when tracked_errors covers them — see
|
|
68
|
+
# StandardCircuit::ErrorTaxonomies.default_skipped_for)
|
|
58
69
|
# criticality — :critical | :standard | :optional
|
|
59
70
|
# (affects health rollup; default: :standard)
|
|
60
71
|
|
|
@@ -69,12 +80,20 @@ StandardCircuit.configure do |config|
|
|
|
69
80
|
# tracked_errors: StandardCircuit::ErrorTaxonomies::Smtp.tracked,
|
|
70
81
|
# criticality: :standard)
|
|
71
82
|
|
|
83
|
+
# Presets for integrations with shared settings (any option passed wins):
|
|
84
|
+
# :postmark — register(:postmark, ...) with the Postmark taxonomy + skips
|
|
85
|
+
# :s3 — register_prefix(:s3, ...) for `service: StandardCircuitS3`
|
|
86
|
+
# config.register_preset(:postmark)
|
|
87
|
+
# config.register_preset(:s3)
|
|
88
|
+
|
|
72
89
|
# ---------------------------------------------------------------------------
|
|
73
90
|
# Prefix registration
|
|
74
91
|
# ---------------------------------------------------------------------------
|
|
75
92
|
# Use `register_prefix` to dynamically name circuits matching a pattern.
|
|
76
93
|
# The first matching prefix wins; the circuit name is "<prefix>_<suffix>".
|
|
77
94
|
|
|
95
|
+
# AWS caller errors (NoSuchKey, AccessDenied) are skipped by default here,
|
|
96
|
+
# so missing keys and permission errors don't trip the S3 breaker.
|
|
78
97
|
# config.register_prefix(:s3,
|
|
79
98
|
# threshold: 10,
|
|
80
99
|
# tracked_errors: StandardCircuit::ErrorTaxonomies::Aws.tracked,
|
|
@@ -91,11 +110,14 @@ StandardCircuit.configure do |config|
|
|
|
91
110
|
# contract as the built-in subscribers.
|
|
92
111
|
|
|
93
112
|
# config.add_notifier(->(name, payload) { MyTracer.record(name, payload) })
|
|
113
|
+
# Idempotent across repeated configure calls (e.g. from to_prepare):
|
|
114
|
+
# re-adding the same notifier replaces it rather than stacking a duplicate.
|
|
94
115
|
|
|
95
116
|
# ---------------------------------------------------------------------------
|
|
96
117
|
# Health endpoint (see --with-health-endpoint)
|
|
97
118
|
# ---------------------------------------------------------------------------
|
|
98
119
|
# The aggregate route is `get "/health", to: "standard_circuit/health#show"`.
|
|
120
|
+
# The controller is autoloaded by the engine — no require needed.
|
|
99
121
|
# If your app also mounts StandardHealth::Engine at "/health", draw the
|
|
100
122
|
# aggregate route BEFORE the mount. That engine registers sub-paths only
|
|
101
123
|
# ("/alive", "/ready", "/diagnostics/env") and never serves the aggregate tier
|
|
@@ -35,7 +35,15 @@ module ActiveStorage
|
|
|
35
35
|
end
|
|
36
36
|
|
|
37
37
|
module StandardCircuit
|
|
38
|
+
# @deprecated The StandardCircuit::ActiveStorage::S3Service alias is unused —
|
|
39
|
+
# storage.yml resolves `service: StandardCircuitS3` to
|
|
40
|
+
# ActiveStorage::Service::StandardCircuitS3Service — and is removed in 0.5.
|
|
38
41
|
module ActiveStorage
|
|
39
|
-
|
|
42
|
+
include ::ActiveSupport::Deprecation::DeprecatedConstantAccessor
|
|
43
|
+
|
|
44
|
+
deprecate_constant "S3Service", "::ActiveStorage::Service::StandardCircuitS3Service",
|
|
45
|
+
deprecator: StandardCircuit.deprecator,
|
|
46
|
+
message: "StandardCircuit::ActiveStorage::S3Service is deprecated and will be removed in 0.5; " \
|
|
47
|
+
"use ActiveStorage::Service::StandardCircuitS3Service (or `service: StandardCircuitS3` in storage.yml)."
|
|
40
48
|
end
|
|
41
49
|
end
|
|
@@ -10,7 +10,15 @@ module StandardCircuit
|
|
|
10
10
|
errors.select { |klass| klass.is_a?(Class) }
|
|
11
11
|
end
|
|
12
12
|
|
|
13
|
+
# @deprecated Removed in 0.5. Faraday::ClientError (4xx) is not in
|
|
14
|
+
# +server_errors+, so it never counts toward a Faraday circuit —
|
|
15
|
+
# listing it in +skipped_errors+ is a no-op. Unused by every consumer.
|
|
13
16
|
def caller_errors
|
|
17
|
+
StandardCircuit.deprecator.warn(
|
|
18
|
+
"StandardCircuit::AdapterErrors::Faraday.caller_errors is deprecated and will be removed in 0.5: " \
|
|
19
|
+
"Faraday::ClientError is never tracked by ErrorTaxonomies::Faraday.tracked, so skipping it is a no-op. " \
|
|
20
|
+
"Drop it from skipped_errors."
|
|
21
|
+
)
|
|
14
22
|
return [] unless defined?(::Faraday::ClientError)
|
|
15
23
|
|
|
16
24
|
[ ::Faraday::ClientError ]
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
module StandardCircuit
|
|
2
|
+
module AdapterErrors
|
|
3
|
+
# Postmark (the `postmark` gem, usually via `postmark-rails`).
|
|
4
|
+
#
|
|
5
|
+
# Guarded like the other adapters: every method returns `[]` when the
|
|
6
|
+
# postmark gem isn't loaded, so this file is safe to require in apps that
|
|
7
|
+
# don't send mail through Postmark.
|
|
8
|
+
#
|
|
9
|
+
# Postmark's error hierarchy is the reason `caller_errors` matters here:
|
|
10
|
+
# `ApiInputError` (422 — invalid payload, inactive recipient, 429-style
|
|
11
|
+
# throttling) and `InvalidApiKeyError` (401 — rotated / revoked token) both
|
|
12
|
+
# subclass `HttpServerError`, and Stoplight matches tracked errors with
|
|
13
|
+
# `is_a?`. Tracking `HttpServerError` without skipping them would let a
|
|
14
|
+
# burst of bad payloads or a config bug trip the breaker.
|
|
15
|
+
module Postmark
|
|
16
|
+
class << self
|
|
17
|
+
def server_errors
|
|
18
|
+
return [] unless defined?(::Postmark::HttpServerError)
|
|
19
|
+
|
|
20
|
+
[ ::Postmark::HttpServerError, ::Postmark::TimeoutError ]
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
def caller_errors
|
|
24
|
+
return [] unless defined?(::Postmark::HttpServerError)
|
|
25
|
+
|
|
26
|
+
[ ::Postmark::ApiInputError, ::Postmark::InvalidApiKeyError ]
|
|
27
|
+
end
|
|
28
|
+
end
|
|
29
|
+
end
|
|
30
|
+
end
|
|
31
|
+
end
|
|
@@ -5,6 +5,10 @@ module StandardCircuit
|
|
|
5
5
|
DEFAULT_WINDOW = 60
|
|
6
6
|
DEFAULT_CRITICALITY = :standard
|
|
7
7
|
CRITICALITIES = [ :critical, :standard, :optional ].freeze
|
|
8
|
+
# 90s > the :postmark preset's 60s cool-off, so a retry lands after the
|
|
9
|
+
# breaker has had a chance to half-open. 5 attempts ≈ 7.5 minutes; 15%
|
|
10
|
+
# jitter so a backlog queued during an outage doesn't retry in lockstep.
|
|
11
|
+
MAILER_RETRY_DEFAULTS = { wait: 90, attempts: 5, jitter: 0.15 }.freeze
|
|
8
12
|
|
|
9
13
|
CircuitSpec = Struct.new(
|
|
10
14
|
:threshold,
|
|
@@ -22,19 +26,24 @@ module StandardCircuit
|
|
|
22
26
|
"invalid criticality #{criticality.inspect}; must be one of #{CRITICALITIES.inspect}"
|
|
23
27
|
end
|
|
24
28
|
|
|
29
|
+
tracked_errors = opts.fetch(:tracked_errors, NetworkErrors.defaults)
|
|
30
|
+
|
|
25
31
|
new(
|
|
26
32
|
threshold: opts.fetch(:threshold, DEFAULT_THRESHOLD),
|
|
27
33
|
cool_off_time: opts.fetch(:cool_off_time, DEFAULT_COOL_OFF),
|
|
28
34
|
window_size: opts.fetch(:window_size, DEFAULT_WINDOW),
|
|
29
|
-
tracked_errors:
|
|
30
|
-
skipped_errors
|
|
35
|
+
tracked_errors: tracked_errors,
|
|
36
|
+
# An explicit `skipped_errors:` (including `[]`) always wins; the
|
|
37
|
+
# default only kicks in when the key is absent. See
|
|
38
|
+
# ErrorTaxonomies.default_skipped_for for why AWS needs one.
|
|
39
|
+
skipped_errors: opts.fetch(:skipped_errors) { ErrorTaxonomies.default_skipped_for(tracked_errors) },
|
|
31
40
|
criticality: criticality
|
|
32
41
|
)
|
|
33
42
|
end
|
|
34
43
|
end
|
|
35
44
|
|
|
36
45
|
attr_accessor :sentry_enabled, :metric_prefix, :data_store, :logger
|
|
37
|
-
attr_reader :circuits, :prefixes, :extra_notifiers, :sentry_criticality_levels
|
|
46
|
+
attr_reader :circuits, :prefixes, :extra_notifiers, :sentry_criticality_levels, :mailer_retry
|
|
38
47
|
|
|
39
48
|
def initialize
|
|
40
49
|
@sentry_enabled = true
|
|
@@ -45,6 +54,22 @@ module StandardCircuit
|
|
|
45
54
|
@circuits = {}
|
|
46
55
|
@prefixes = {}
|
|
47
56
|
@extra_notifiers = []
|
|
57
|
+
@extra_notifier_keys = []
|
|
58
|
+
@mailer_retry = nil
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
# Opt in to retrying ActionMailer::MailDeliveryJob on
|
|
62
|
+
# StandardCircuit::Mailer::CircuitOpenError (see Mailer::Retry). Accepts:
|
|
63
|
+
#
|
|
64
|
+
# nil / false — (default) off; the gem installs nothing.
|
|
65
|
+
# true — MAILER_RETRY_DEFAULTS (wait: 90, attempts: 5, jitter: 0.15)
|
|
66
|
+
# Hash — MAILER_RETRY_DEFAULTS merged with :wait / :attempts /
|
|
67
|
+
# :jitter. `wait:` takes anything `retry_on` does (seconds,
|
|
68
|
+
# a Duration, :polynomially_longer, a Proc).
|
|
69
|
+
#
|
|
70
|
+
# The reader returns nil or a frozen, complete Hash.
|
|
71
|
+
def mailer_retry=(value)
|
|
72
|
+
@mailer_retry = normalize_mailer_retry(value)
|
|
48
73
|
end
|
|
49
74
|
|
|
50
75
|
# Opt in to criticality-aware Sentry reporting for the built-in Sentry
|
|
@@ -67,6 +92,7 @@ module StandardCircuit
|
|
|
67
92
|
@circuits.clear
|
|
68
93
|
@prefixes.clear
|
|
69
94
|
@extra_notifiers.clear
|
|
95
|
+
@extra_notifier_keys.clear
|
|
70
96
|
end
|
|
71
97
|
|
|
72
98
|
def register(name, **opts)
|
|
@@ -89,16 +115,53 @@ module StandardCircuit
|
|
|
89
115
|
spec
|
|
90
116
|
end
|
|
91
117
|
|
|
118
|
+
# Register a circuit from a named preset (see StandardCircuit::Presets):
|
|
119
|
+
#
|
|
120
|
+
# c.register_preset(:postmark) # c.register(:postmark, ...)
|
|
121
|
+
# c.register_preset(:s3) # c.register_prefix(:s3, ...)
|
|
122
|
+
# c.register_preset(:postmark, name: :mail, threshold: 5)
|
|
123
|
+
#
|
|
124
|
+
# +name:+ overrides the circuit name (or prefix); any other keyword is a
|
|
125
|
+
# `register` option that wins over the preset's value.
|
|
126
|
+
def register_preset(preset, name: preset, **overrides)
|
|
127
|
+
scope, opts = Presets.resolve(preset, **overrides)
|
|
128
|
+
scope == :prefix ? register_prefix(name, **opts) : register(name, **opts)
|
|
129
|
+
end
|
|
130
|
+
|
|
92
131
|
# Register a host-supplied subscriber. Subscribers must respond to
|
|
93
132
|
# `call(event_name, payload)` — Stoplight-shaped 4-arg notifiers from the
|
|
94
133
|
# 0.1.x API are no longer accepted as extras (Logger / Sentry / Metrics
|
|
95
134
|
# demonstrate the new shape).
|
|
96
|
-
|
|
135
|
+
#
|
|
136
|
+
# Idempotent: hosts call `configure` from `to_prepare`, which re-runs on
|
|
137
|
+
# every code reload, so re-adding "the same" notifier replaces the earlier
|
|
138
|
+
# entry in place instead of stacking a duplicate. "The same" means the same
|
|
139
|
+
# +key:+ when one is given, otherwise:
|
|
140
|
+
#
|
|
141
|
+
# * a Proc / Method — same source location (a lambda re-created on reload)
|
|
142
|
+
# * a Class / Module used directly — same name
|
|
143
|
+
# * any other object — same class name (a reloaded class is a new class
|
|
144
|
+
# object with the same name, so compare names, not classes)
|
|
145
|
+
# * an instance of an anonymous class — never deduped
|
|
146
|
+
#
|
|
147
|
+
# The newest instance wins, so a reloaded notifier class takes effect.
|
|
148
|
+
# Pass distinct +key:+ values to register two instances of one class
|
|
149
|
+
# (e.g. two webhook notifiers pointed at different URLs).
|
|
150
|
+
def add_notifier(notifier, key: nil)
|
|
97
151
|
unless notifier.respond_to?(:call)
|
|
98
152
|
raise ArgumentError,
|
|
99
153
|
"extra notifiers must respond to `call(event_name, payload)`; got #{notifier.class}"
|
|
100
154
|
end
|
|
101
|
-
|
|
155
|
+
|
|
156
|
+
identity = key.nil? ? notifier_identity(notifier) : [ :key, key ]
|
|
157
|
+
index = identity && @extra_notifier_keys.index(identity)
|
|
158
|
+
if index
|
|
159
|
+
@extra_notifiers[index] = notifier
|
|
160
|
+
else
|
|
161
|
+
@extra_notifiers << notifier
|
|
162
|
+
@extra_notifier_keys << identity
|
|
163
|
+
end
|
|
164
|
+
notifier
|
|
102
165
|
end
|
|
103
166
|
|
|
104
167
|
def spec_for(name)
|
|
@@ -119,6 +182,33 @@ module StandardCircuit
|
|
|
119
182
|
end
|
|
120
183
|
end
|
|
121
184
|
|
|
185
|
+
def notifier_identity(notifier)
|
|
186
|
+
if notifier.respond_to?(:source_location) && notifier.source_location
|
|
187
|
+
[ :source, notifier.source_location ]
|
|
188
|
+
elsif notifier.is_a?(Module)
|
|
189
|
+
notifier.name && [ :module, notifier.name ]
|
|
190
|
+
else
|
|
191
|
+
notifier.class.name && [ :class, notifier.class.name ]
|
|
192
|
+
end
|
|
193
|
+
end
|
|
194
|
+
|
|
195
|
+
def normalize_mailer_retry(value)
|
|
196
|
+
case value
|
|
197
|
+
when nil, false then nil
|
|
198
|
+
when true then MAILER_RETRY_DEFAULTS
|
|
199
|
+
when Hash
|
|
200
|
+
options = value.transform_keys(&:to_sym)
|
|
201
|
+
unknown = options.keys - MAILER_RETRY_DEFAULTS.keys
|
|
202
|
+
unless unknown.empty?
|
|
203
|
+
raise ArgumentError,
|
|
204
|
+
"unknown mailer_retry option(s) #{unknown.inspect}; allowed: #{MAILER_RETRY_DEFAULTS.keys.inspect}"
|
|
205
|
+
end
|
|
206
|
+
MAILER_RETRY_DEFAULTS.merge(options).freeze
|
|
207
|
+
else
|
|
208
|
+
raise ArgumentError, "mailer_retry must be nil, true, false, or a Hash; got #{value.class}"
|
|
209
|
+
end
|
|
210
|
+
end
|
|
211
|
+
|
|
122
212
|
def merge_sentry_criticality_levels(map)
|
|
123
213
|
normalized = map.to_h do |criticality, level|
|
|
124
214
|
# Check symbolizability before coercing: a key like 42 would otherwise
|
|
@@ -47,9 +47,10 @@ module StandardCircuit
|
|
|
47
47
|
|
|
48
48
|
private
|
|
49
49
|
|
|
50
|
-
def handle_circuit_open(
|
|
50
|
+
def handle_circuit_open(_error)
|
|
51
|
+
# No metric here: Runner already counted this rejection as
|
|
52
|
+
# `<metric_prefix>.request{status: circuit_open}` before re-raising.
|
|
51
53
|
fallback = self.class._circuit_open_fallback || {}
|
|
52
|
-
emit_circuit_open_metric(error)
|
|
53
54
|
|
|
54
55
|
return instance_exec(&fallback[:json]) if request.format.json? && fallback[:json]
|
|
55
56
|
return instance_exec(response, &fallback[:stream]) if streaming_controller? && fallback[:stream]
|
|
@@ -61,14 +62,5 @@ module StandardCircuit
|
|
|
61
62
|
def streaming_controller?
|
|
62
63
|
defined?(::ActionController::Live) && self.class.include?(::ActionController::Live)
|
|
63
64
|
end
|
|
64
|
-
|
|
65
|
-
def emit_circuit_open_metric(error)
|
|
66
|
-
prefix = StandardCircuit.config.metric_prefix
|
|
67
|
-
::Sentry::Metrics.count(
|
|
68
|
-
"#{prefix}.request",
|
|
69
|
-
value: 1,
|
|
70
|
-
attributes: { service: error.light_name, status: "circuit_open" }
|
|
71
|
-
)
|
|
72
|
-
end
|
|
73
65
|
end
|
|
74
66
|
end
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
require "active_support"
|
|
2
|
+
require "active_support/deprecation"
|
|
3
|
+
|
|
4
|
+
module StandardCircuit
|
|
5
|
+
# The gem's ActiveSupport::Deprecation instance. Registered with the host as
|
|
6
|
+
# `Rails.application.deprecators[:standard_circuit]` by the engine, so the
|
|
7
|
+
# host's `config.active_support.deprecation` behaviour (log / raise /
|
|
8
|
+
# silence / report) applies to StandardCircuit deprecations too.
|
|
9
|
+
def self.deprecator
|
|
10
|
+
@deprecator ||= ActiveSupport::Deprecation.new("0.5", "StandardCircuit")
|
|
11
|
+
end
|
|
12
|
+
end
|
|
@@ -23,12 +23,19 @@ module StandardCircuit
|
|
|
23
23
|
# spec/integration/health_route_boot_spec.rb, which fails if this ever
|
|
24
24
|
# regresses).
|
|
25
25
|
#
|
|
26
|
+
# `StandardCircuit::HealthController` lives in the engine's app/controllers
|
|
27
|
+
# and is autoloaded (and eager-loaded in production) by the host.
|
|
28
|
+
#
|
|
26
29
|
# One real consequence: `StandardCircuit::HealthController` now picks up the
|
|
27
30
|
# engine's (empty) url_helpers instead of the application's, so app path
|
|
28
31
|
# helpers inside it — or inside a host subclass of it — need a `main_app.`
|
|
29
32
|
# prefix. The controller only renders JSON, so nothing in-gem is affected.
|
|
30
33
|
isolate_namespace StandardCircuit
|
|
31
34
|
|
|
35
|
+
initializer "standard_circuit.deprecator" do |app|
|
|
36
|
+
app.deprecators[:standard_circuit] = StandardCircuit.deprecator if app.respond_to?(:deprecators)
|
|
37
|
+
end
|
|
38
|
+
|
|
32
39
|
initializer "standard_circuit.subscribers", after: :load_config_initializers do
|
|
33
40
|
StandardCircuit.subscribers.setup!
|
|
34
41
|
end
|
|
@@ -16,6 +16,34 @@ module StandardCircuit
|
|
|
16
16
|
# `AdapterErrors::*` because the right `skipped_errors` set is usually
|
|
17
17
|
# app-specific and a shared taxonomy would over-skip.
|
|
18
18
|
module ErrorTaxonomies
|
|
19
|
+
# Default `skipped_errors` for a circuit registered without an explicit
|
|
20
|
+
# `skipped_errors:`. Most adapters default to `[]`, but two can't, because
|
|
21
|
+
# their server-side error class is also the superclass of caller errors:
|
|
22
|
+
#
|
|
23
|
+
# * AWS — 5xx errors are dynamically generated `Aws::Errors::ServiceError`
|
|
24
|
+
# subclasses (e.g. `Aws::S3::Errors::ServiceUnavailable`), so `Aws.tracked`
|
|
25
|
+
# has to track `ServiceError` itself — which is also the superclass of
|
|
26
|
+
# `AccessDenied` and `NoSuchKey`. Without a skip list, a burst of
|
|
27
|
+
# missing-key lookups or permission errors would trip the S3 breaker.
|
|
28
|
+
# * Postmark — `ApiInputError` and `InvalidApiKeyError` subclass
|
|
29
|
+
# `Postmark::HttpServerError`, which `Postmark.tracked` tracks.
|
|
30
|
+
#
|
|
31
|
+
# Returns the caller errors that some entry of +tracked+ covers (is the
|
|
32
|
+
# same class or an ancestor of), or `[]` when none are covered. Returns a
|
|
33
|
+
# fresh array each call.
|
|
34
|
+
def self.default_skipped_for(tracked)
|
|
35
|
+
trackable = Array(tracked).grep(Module)
|
|
36
|
+
covered_caller_errors.select do |caller_error|
|
|
37
|
+
trackable.any? { |klass| caller_error <= klass }
|
|
38
|
+
end
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
# @api private
|
|
42
|
+
def self.covered_caller_errors
|
|
43
|
+
AdapterErrors::Aws.caller_errors + AdapterErrors::Postmark.caller_errors
|
|
44
|
+
end
|
|
45
|
+
private_class_method :covered_caller_errors
|
|
46
|
+
|
|
19
47
|
module Stripe
|
|
20
48
|
def self.tracked
|
|
21
49
|
NetworkErrors.defaults + AdapterErrors::Stripe.server_errors
|
|
@@ -39,5 +67,13 @@ module StandardCircuit
|
|
|
39
67
|
NetworkErrors.defaults + AdapterErrors::Faraday.server_errors
|
|
40
68
|
end
|
|
41
69
|
end
|
|
70
|
+
|
|
71
|
+
# Pair with `skipped_errors: AdapterErrors::Postmark.caller_errors` — or
|
|
72
|
+
# omit `skipped_errors:` and let `default_skipped_for` supply it.
|
|
73
|
+
module Postmark
|
|
74
|
+
def self.tracked
|
|
75
|
+
NetworkErrors.defaults + AdapterErrors::Postmark.server_errors
|
|
76
|
+
end
|
|
77
|
+
end
|
|
42
78
|
end
|
|
43
79
|
end
|
|
@@ -3,10 +3,10 @@ module StandardCircuit
|
|
|
3
3
|
# and the Config's registered circuits / prefixes and return structured
|
|
4
4
|
# snapshots plus an overall health status.
|
|
5
5
|
#
|
|
6
|
-
# Intended for mounting in a Rails HealthController.
|
|
7
|
-
#
|
|
8
|
-
#
|
|
9
|
-
#
|
|
6
|
+
# Intended for mounting in a Rails HealthController. Use +health_report+ —
|
|
7
|
+
# it takes a single atomic snapshot, so the rendered status and circuits
|
|
8
|
+
# always describe the same moment (the separate module-level
|
|
9
|
+
# +health_snapshot+ / +health_overall+ readers are deprecated):
|
|
10
10
|
#
|
|
11
11
|
# report = StandardCircuit.health_report
|
|
12
12
|
# render json: report, status: (report[:status] == :critical ? 503 : 200)
|
|
@@ -1,28 +1,23 @@
|
|
|
1
|
+
# Deprecated load path, kept so `require "standard_circuit/health_controller"`
|
|
2
|
+
# at the top of a host's config/routes.rb (the pre-0.4 opt-in) keeps working.
|
|
3
|
+
#
|
|
4
|
+
# The controller now lives at app/controllers/standard_circuit/health_controller.rb
|
|
5
|
+
# and the engine autoloads it — delete the require and keep the route.
|
|
1
6
|
require "action_controller"
|
|
7
|
+
require "standard_circuit"
|
|
2
8
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
#
|
|
18
|
-
# Inherits from +ActionController::API+ to sidestep any ApplicationController
|
|
19
|
-
# filters (authentication, bootstrap redirects, etc.) — health probes must be
|
|
20
|
-
# callable anonymously from load balancers and uptime monitors.
|
|
21
|
-
class HealthController < ::ActionController::API
|
|
22
|
-
def show
|
|
23
|
-
report = StandardCircuit.health_report
|
|
24
|
-
http_status = report[:status] == :critical ? :service_unavailable : :ok
|
|
25
|
-
render json: report, status: http_status
|
|
26
|
-
end
|
|
27
|
-
end
|
|
9
|
+
StandardCircuit.deprecator.warn(
|
|
10
|
+
'require "standard_circuit/health_controller" is no longer needed: the engine ' \
|
|
11
|
+
"autoloads StandardCircuit::HealthController. Remove the require and keep the " \
|
|
12
|
+
'`get "/health", to: "standard_circuit/health#show"` route.'
|
|
13
|
+
)
|
|
14
|
+
|
|
15
|
+
# Booted Rails app (e.g. required from config/routes.rb): the engine's
|
|
16
|
+
# autoloader already has an autoload set for the constant, so there is nothing
|
|
17
|
+
# to do — loading the file here would race Zeitwerk for the same path.
|
|
18
|
+
# Otherwise (required from an initializer before autoloaders are set up, or
|
|
19
|
+
# outside Rails entirely) load it directly; Zeitwerk later sees the constant
|
|
20
|
+
# is defined and leaves it alone.
|
|
21
|
+
unless StandardCircuit.autoload?(:HealthController) || StandardCircuit.const_defined?(:HealthController, false)
|
|
22
|
+
require_relative "../../app/controllers/standard_circuit/health_controller"
|
|
28
23
|
end
|
|
@@ -0,0 +1,121 @@
|
|
|
1
|
+
require_relative "circuit_open_error"
|
|
2
|
+
|
|
3
|
+
module StandardCircuit
|
|
4
|
+
module Mailer
|
|
5
|
+
# Opt-in `retry_on CircuitOpenError` for ActionMailer's delivery job, so a
|
|
6
|
+
# `deliver_later` that runs while the mail circuit is open is retried after
|
|
7
|
+
# the breaker has had a chance to half-open, instead of failing once and
|
|
8
|
+
# losing the email. Enabled with:
|
|
9
|
+
#
|
|
10
|
+
# StandardCircuit.configure do |c|
|
|
11
|
+
# c.mailer_retry = true # 90s wait, 5 attempts, 15% jitter
|
|
12
|
+
# c.mailer_retry = { wait: 120, attempts: 8 } # partial overrides
|
|
13
|
+
# end
|
|
14
|
+
#
|
|
15
|
+
# Keep `wait` longer than the mail circuit's `cool_off_time`, or each retry
|
|
16
|
+
# lands on a still-open circuit.
|
|
17
|
+
#
|
|
18
|
+
# `ActionMailer::MailDeliveryJob` inherits from `ActiveJob::Base`, not the
|
|
19
|
+
# host's ApplicationJob, which is why every app was patching it from an
|
|
20
|
+
# initializer. Installation is idempotent — `configure` runs on every
|
|
21
|
+
# `to_prepare` (so on every code reload in development), and `retry_on`
|
|
22
|
+
# appends a rescue handler each time it's called — and it also stands down
|
|
23
|
+
# when a handler for CircuitOpenError is already present, so a host that
|
|
24
|
+
# still carries its own initializer doesn't end up with two.
|
|
25
|
+
#
|
|
26
|
+
# Subclasses of MailDeliveryJob (`self.delivery_job = MyJob`) inherit the
|
|
27
|
+
# handler; a `retry_on CircuitOpenError` declared on the subclass wins.
|
|
28
|
+
#
|
|
29
|
+
# On exhaustion: one error log line, a Sentry `:error` event (when
|
|
30
|
+
# `sentry_enabled` and Sentry is initialized), and a
|
|
31
|
+
# `standard_circuit.mailer.retries_exhausted` event. PII: recipient
|
|
32
|
+
# DOMAINS only — never addresses or subjects.
|
|
33
|
+
module Retry
|
|
34
|
+
EXHAUSTED_EVENT = "standard_circuit.mailer.retries_exhausted".freeze
|
|
35
|
+
EXHAUSTED_MESSAGE = "Email delivery failed: mail circuit breaker retries exhausted".freeze
|
|
36
|
+
FINGERPRINT = "standard_circuit-mailer-retries-exhausted".freeze
|
|
37
|
+
|
|
38
|
+
class << self
|
|
39
|
+
# Defers installation until ActiveJob::Base has loaded (immediately if
|
|
40
|
+
# it already has), so enabling this from an early initializer doesn't
|
|
41
|
+
# force ActiveJob to load before its own configuration is applied. The
|
|
42
|
+
# hook reads the live config when it runs; it is registered once.
|
|
43
|
+
def install_on_load
|
|
44
|
+
return if @on_load_registered
|
|
45
|
+
|
|
46
|
+
@on_load_registered = true
|
|
47
|
+
::ActiveSupport.on_load(:active_job) do
|
|
48
|
+
options = StandardCircuit.config.mailer_retry
|
|
49
|
+
StandardCircuit::Mailer::Retry.install(::ActionMailer::MailDeliveryJob, options) if options
|
|
50
|
+
end
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
# Adds the retry_on to +job_class+ unless a CircuitOpenError handler is
|
|
54
|
+
# already there. Returns true when it installed one.
|
|
55
|
+
def install(job_class, options)
|
|
56
|
+
return false if installed?(job_class)
|
|
57
|
+
|
|
58
|
+
job_class.retry_on(
|
|
59
|
+
CircuitOpenError,
|
|
60
|
+
wait: options.fetch(:wait),
|
|
61
|
+
attempts: options.fetch(:attempts),
|
|
62
|
+
jitter: options.fetch(:jitter)
|
|
63
|
+
) do |job, error|
|
|
64
|
+
StandardCircuit::Mailer::Retry.report_exhausted(job, error)
|
|
65
|
+
end
|
|
66
|
+
true
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
def installed?(job_class)
|
|
70
|
+
job_class.rescue_handlers.any? { |(error_name, _handler)| error_name == CircuitOpenError.name }
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
def report_exhausted(job, error)
|
|
74
|
+
mailer_class, mail_action = job.arguments
|
|
75
|
+
details = {
|
|
76
|
+
mailer_class: mailer_class,
|
|
77
|
+
mail_action: mail_action,
|
|
78
|
+
recipient_domains: recipient_domains(error.recipients),
|
|
79
|
+
job_id: job.job_id,
|
|
80
|
+
executions: job.executions
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
logger&.error("[standard_circuit] #{EXHAUSTED_MESSAGE} #{details.to_json}")
|
|
84
|
+
capture_sentry(details)
|
|
85
|
+
EventEmitter.emit(EXHAUSTED_EVENT, details)
|
|
86
|
+
details
|
|
87
|
+
end
|
|
88
|
+
|
|
89
|
+
def recipient_domains(recipients)
|
|
90
|
+
Array(recipients)
|
|
91
|
+
.filter_map { |address| address.to_s.split("@", 2)[1]&.strip&.downcase }
|
|
92
|
+
.reject(&:empty?)
|
|
93
|
+
.uniq
|
|
94
|
+
end
|
|
95
|
+
|
|
96
|
+
# @api private — test isolation.
|
|
97
|
+
def reset_on_load_registration!
|
|
98
|
+
@on_load_registered = false
|
|
99
|
+
end
|
|
100
|
+
|
|
101
|
+
private
|
|
102
|
+
|
|
103
|
+
def logger
|
|
104
|
+
StandardCircuit.config.logger || (::Rails.logger if defined?(::Rails) && ::Rails.respond_to?(:logger))
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
def capture_sentry(details)
|
|
108
|
+
return unless StandardCircuit.config.sentry_enabled
|
|
109
|
+
return unless defined?(::Sentry) && ::Sentry.respond_to?(:initialized?) && ::Sentry.initialized?
|
|
110
|
+
|
|
111
|
+
::Sentry.capture_message(
|
|
112
|
+
EXHAUSTED_MESSAGE,
|
|
113
|
+
level: :error,
|
|
114
|
+
fingerprint: [ FINGERPRINT, details[:mailer_class], details[:mail_action] ].map(&:to_s),
|
|
115
|
+
extra: details
|
|
116
|
+
)
|
|
117
|
+
end
|
|
118
|
+
end
|
|
119
|
+
end
|
|
120
|
+
end
|
|
121
|
+
end
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
module StandardCircuit
|
|
2
|
+
# Named circuit registrations for integrations whose settings were being
|
|
3
|
+
# copy-pasted, line for line, across host apps. Use via
|
|
4
|
+
# `Config#register_preset`:
|
|
5
|
+
#
|
|
6
|
+
# StandardCircuit.configure do |c|
|
|
7
|
+
# c.register_preset(:postmark) # named circuit :postmark
|
|
8
|
+
# c.register_preset(:s3) # prefix "s3" (s3_<bucket>)
|
|
9
|
+
# c.register_preset(:postmark, criticality: :critical) # override any option
|
|
10
|
+
# end
|
|
11
|
+
#
|
|
12
|
+
# A preset only fills in defaults — every `register` option passed alongside
|
|
13
|
+
# it wins. Each preset also requires its SDK before computing the error
|
|
14
|
+
# lists, so the taxonomy can't silently degrade to network errors only when
|
|
15
|
+
# the SDK is declared `require: false` and hasn't been loaded yet at
|
|
16
|
+
# configure time.
|
|
17
|
+
module Presets
|
|
18
|
+
Preset = Struct.new(:scope, :sdk_require, :sdk_constant, :options, keyword_init: true)
|
|
19
|
+
|
|
20
|
+
REGISTRY = {
|
|
21
|
+
# Postmark API delivery behind the `:standard_circuit` mailer delivery
|
|
22
|
+
# method. 60s cool-off pairs with `mailer_retry`'s 90s default wait, so a
|
|
23
|
+
# retried delivery lands after the breaker has had a chance to half-open.
|
|
24
|
+
# :standard — an email outage is user-visible but non-blocking.
|
|
25
|
+
postmark: Preset.new(
|
|
26
|
+
scope: :name,
|
|
27
|
+
sdk_require: "postmark",
|
|
28
|
+
sdk_constant: "Postmark::HttpServerError",
|
|
29
|
+
options: -> {
|
|
30
|
+
{
|
|
31
|
+
threshold: 3,
|
|
32
|
+
cool_off_time: 60,
|
|
33
|
+
criticality: :standard,
|
|
34
|
+
tracked_errors: ErrorTaxonomies::Postmark.tracked,
|
|
35
|
+
skipped_errors: AdapterErrors::Postmark.caller_errors
|
|
36
|
+
}
|
|
37
|
+
}
|
|
38
|
+
),
|
|
39
|
+
# Per-bucket S3 circuits for `service: StandardCircuitS3` in storage.yml,
|
|
40
|
+
# which keys its circuit as `s3_<bucket>` — hence a prefix registration.
|
|
41
|
+
# `skipped_errors` is left to `ErrorTaxonomies.default_skipped_for`, which
|
|
42
|
+
# skips AccessDenied / NoSuchKey.
|
|
43
|
+
s3: Preset.new(
|
|
44
|
+
scope: :prefix,
|
|
45
|
+
sdk_require: "aws-sdk-s3",
|
|
46
|
+
sdk_constant: "Aws::S3::Errors::NoSuchKey",
|
|
47
|
+
options: -> {
|
|
48
|
+
{
|
|
49
|
+
threshold: 3,
|
|
50
|
+
cool_off_time: 30,
|
|
51
|
+
criticality: :standard,
|
|
52
|
+
tracked_errors: ErrorTaxonomies::Aws.tracked
|
|
53
|
+
}
|
|
54
|
+
}
|
|
55
|
+
)
|
|
56
|
+
}.freeze
|
|
57
|
+
|
|
58
|
+
class << self
|
|
59
|
+
def names
|
|
60
|
+
REGISTRY.keys
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
# Returns `[scope, options]` for +preset+ with +overrides+ merged on top.
|
|
64
|
+
# Raises ArgumentError for an unknown preset or a missing SDK.
|
|
65
|
+
def resolve(preset, **overrides)
|
|
66
|
+
entry = REGISTRY.fetch(preset.to_sym) do
|
|
67
|
+
raise ArgumentError, "unknown preset #{preset.inspect}; available: #{names.inspect}"
|
|
68
|
+
end
|
|
69
|
+
load_sdk!(preset, entry)
|
|
70
|
+
[ entry.scope, entry.options.call.merge(overrides) ]
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
private
|
|
74
|
+
|
|
75
|
+
def load_sdk!(preset, entry)
|
|
76
|
+
return if Object.const_defined?(entry.sdk_constant)
|
|
77
|
+
|
|
78
|
+
begin
|
|
79
|
+
require entry.sdk_require
|
|
80
|
+
rescue LoadError
|
|
81
|
+
nil
|
|
82
|
+
end
|
|
83
|
+
return if Object.const_defined?(entry.sdk_constant)
|
|
84
|
+
|
|
85
|
+
raise ArgumentError,
|
|
86
|
+
"preset #{preset.inspect} needs the #{entry.sdk_require} gem (#{entry.sdk_constant} is not defined)"
|
|
87
|
+
end
|
|
88
|
+
end
|
|
89
|
+
end
|
|
90
|
+
end
|
data/lib/standard_circuit.rb
CHANGED
|
@@ -3,12 +3,15 @@ require "concurrent"
|
|
|
3
3
|
require "sentry-ruby"
|
|
4
4
|
|
|
5
5
|
require "standard_circuit/version"
|
|
6
|
+
require "standard_circuit/deprecator"
|
|
6
7
|
require "standard_circuit/network_errors"
|
|
7
8
|
require "standard_circuit/adapter_errors/stripe"
|
|
8
9
|
require "standard_circuit/adapter_errors/aws"
|
|
9
10
|
require "standard_circuit/adapter_errors/faraday"
|
|
10
11
|
require "standard_circuit/adapter_errors/smtp"
|
|
12
|
+
require "standard_circuit/adapter_errors/postmark"
|
|
11
13
|
require "standard_circuit/error_taxonomies"
|
|
14
|
+
require "standard_circuit/presets"
|
|
12
15
|
require "standard_circuit/event_emitter"
|
|
13
16
|
require "standard_circuit/notifier_bridge"
|
|
14
17
|
require "standard_circuit/notifiers/logger"
|
|
@@ -20,6 +23,7 @@ require "standard_circuit/health"
|
|
|
20
23
|
require "standard_circuit/runner"
|
|
21
24
|
require "standard_circuit/mailer/circuit_open_error"
|
|
22
25
|
require "standard_circuit/mailer/delivery_method"
|
|
26
|
+
require "standard_circuit/mailer/retry"
|
|
23
27
|
require "standard_circuit/controller_support"
|
|
24
28
|
require "standard_circuit/engine" if defined?(::Rails::Engine)
|
|
25
29
|
|
|
@@ -32,6 +36,7 @@ module StandardCircuit
|
|
|
32
36
|
yield config
|
|
33
37
|
runner.apply_config!(config)
|
|
34
38
|
subscribers.setup!
|
|
39
|
+
Mailer::Retry.install_on_load if config.mailer_retry
|
|
35
40
|
config
|
|
36
41
|
end
|
|
37
42
|
|
|
@@ -67,11 +72,21 @@ module StandardCircuit
|
|
|
67
72
|
runner.reset!
|
|
68
73
|
end
|
|
69
74
|
|
|
75
|
+
# @deprecated Use +health_report[:circuits]+. Removed in 0.5.
|
|
70
76
|
def health_snapshot
|
|
77
|
+
deprecator.warn(
|
|
78
|
+
"StandardCircuit.health_snapshot is deprecated and will be removed in 0.5; " \
|
|
79
|
+
"use StandardCircuit.health_report[:circuits] (one atomic read of status + circuits)."
|
|
80
|
+
)
|
|
71
81
|
runner.health_snapshot
|
|
72
82
|
end
|
|
73
83
|
|
|
84
|
+
# @deprecated Use +health_report[:status]+. Removed in 0.5.
|
|
74
85
|
def health_overall(snapshot = nil)
|
|
86
|
+
deprecator.warn(
|
|
87
|
+
"StandardCircuit.health_overall is deprecated and will be removed in 0.5; " \
|
|
88
|
+
"use StandardCircuit.health_report[:status] (one atomic read of status + circuits)."
|
|
89
|
+
)
|
|
75
90
|
runner.health_overall(snapshot)
|
|
76
91
|
end
|
|
77
92
|
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: standard_circuit
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.4.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Jaryl Sim
|
|
@@ -65,6 +65,34 @@ dependencies:
|
|
|
65
65
|
- - ">="
|
|
66
66
|
- !ruby/object:Gem::Version
|
|
67
67
|
version: '8.0'
|
|
68
|
+
- !ruby/object:Gem::Dependency
|
|
69
|
+
name: actionmailer
|
|
70
|
+
requirement: !ruby/object:Gem::Requirement
|
|
71
|
+
requirements:
|
|
72
|
+
- - ">="
|
|
73
|
+
- !ruby/object:Gem::Version
|
|
74
|
+
version: '8.0'
|
|
75
|
+
type: :runtime
|
|
76
|
+
prerelease: false
|
|
77
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
78
|
+
requirements:
|
|
79
|
+
- - ">="
|
|
80
|
+
- !ruby/object:Gem::Version
|
|
81
|
+
version: '8.0'
|
|
82
|
+
- !ruby/object:Gem::Dependency
|
|
83
|
+
name: actionpack
|
|
84
|
+
requirement: !ruby/object:Gem::Requirement
|
|
85
|
+
requirements:
|
|
86
|
+
- - ">="
|
|
87
|
+
- !ruby/object:Gem::Version
|
|
88
|
+
version: '8.0'
|
|
89
|
+
type: :runtime
|
|
90
|
+
prerelease: false
|
|
91
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
92
|
+
requirements:
|
|
93
|
+
- - ">="
|
|
94
|
+
- !ruby/object:Gem::Version
|
|
95
|
+
version: '8.0'
|
|
68
96
|
- !ruby/object:Gem::Dependency
|
|
69
97
|
name: brakeman
|
|
70
98
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -120,18 +148,20 @@ files:
|
|
|
120
148
|
- LICENSE
|
|
121
149
|
- README.md
|
|
122
150
|
- Rakefile
|
|
151
|
+
- app/controllers/standard_circuit/health_controller.rb
|
|
123
152
|
- lib/active_storage/service/standard_circuit_s3_service.rb
|
|
124
153
|
- lib/generators/standard_circuit/install/install_generator.rb
|
|
125
|
-
- lib/generators/standard_circuit/install/templates/health_initializer.rb.tt
|
|
126
154
|
- lib/generators/standard_circuit/install/templates/initializer.rb.tt
|
|
127
155
|
- lib/standard_circuit.rb
|
|
128
156
|
- lib/standard_circuit/active_storage/s3_service.rb
|
|
129
157
|
- lib/standard_circuit/adapter_errors/aws.rb
|
|
130
158
|
- lib/standard_circuit/adapter_errors/faraday.rb
|
|
159
|
+
- lib/standard_circuit/adapter_errors/postmark.rb
|
|
131
160
|
- lib/standard_circuit/adapter_errors/smtp.rb
|
|
132
161
|
- lib/standard_circuit/adapter_errors/stripe.rb
|
|
133
162
|
- lib/standard_circuit/config.rb
|
|
134
163
|
- lib/standard_circuit/controller_support.rb
|
|
164
|
+
- lib/standard_circuit/deprecator.rb
|
|
135
165
|
- lib/standard_circuit/engine.rb
|
|
136
166
|
- lib/standard_circuit/error_taxonomies.rb
|
|
137
167
|
- lib/standard_circuit/event_emitter.rb
|
|
@@ -139,11 +169,13 @@ files:
|
|
|
139
169
|
- lib/standard_circuit/health_controller.rb
|
|
140
170
|
- lib/standard_circuit/mailer/circuit_open_error.rb
|
|
141
171
|
- lib/standard_circuit/mailer/delivery_method.rb
|
|
172
|
+
- lib/standard_circuit/mailer/retry.rb
|
|
142
173
|
- lib/standard_circuit/network_errors.rb
|
|
143
174
|
- lib/standard_circuit/notifier_bridge.rb
|
|
144
175
|
- lib/standard_circuit/notifiers/logger.rb
|
|
145
176
|
- lib/standard_circuit/notifiers/metrics.rb
|
|
146
177
|
- lib/standard_circuit/notifiers/sentry.rb
|
|
178
|
+
- lib/standard_circuit/presets.rb
|
|
147
179
|
- lib/standard_circuit/rspec.rb
|
|
148
180
|
- lib/standard_circuit/runner.rb
|
|
149
181
|
- lib/standard_circuit/subscribers.rb
|
|
@@ -1,25 +0,0 @@
|
|
|
1
|
-
# StandardCircuit health-check controller
|
|
2
|
-
# Generated by: rails g standard_circuit:install --with-health-endpoint
|
|
3
|
-
#
|
|
4
|
-
# The controller is opt-in: this require pulls it onto the autoload path so
|
|
5
|
-
# the route below resolves. Add the matching route to config/routes.rb:
|
|
6
|
-
#
|
|
7
|
-
# get "/health", to: "standard_circuit/health#show"
|
|
8
|
-
#
|
|
9
|
-
# ORDERING IS LOAD-BEARING: if your app also mounts StandardHealth::Engine at
|
|
10
|
-
# "/health", draw this aggregate route BEFORE the mount —
|
|
11
|
-
#
|
|
12
|
-
# get "/health", to: "standard_circuit/health#show" # aggregate — FIRST
|
|
13
|
-
# mount StandardHealth::Engine => "/health", as: :standard_health
|
|
14
|
-
#
|
|
15
|
-
# StandardHealth::Engine registers sub-paths only ("/alive", "/ready",
|
|
16
|
-
# "/diagnostics/env") — it never serves the aggregate tier itself. So an app
|
|
17
|
-
# that mounts the engine and assumes "/health" is covered has no aggregate tier
|
|
18
|
-
# at all, and nothing says so: no boot error, and no failing route spec unless
|
|
19
|
-
# one already exists. Draw this route explicitly, first.
|
|
20
|
-
#
|
|
21
|
-
# The endpoint renders StandardCircuit.health_report as JSON and returns
|
|
22
|
-
# 503 when the rolled-up status is :critical (so orchestrators pull the
|
|
23
|
-
# instance out of rotation) and 200 otherwise.
|
|
24
|
-
|
|
25
|
-
require "standard_circuit/health_controller"
|