standard_circuit 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +29 -0
- data/MIT-LICENSE +21 -0
- data/README.md +66 -0
- data/Rakefile +6 -0
- data/lib/active_storage/service/standard_circuit_s3_service.rb +13 -0
- data/lib/standard_circuit/active_storage/s3_service.rb +41 -0
- data/lib/standard_circuit/adapter_errors/aws.rb +23 -0
- data/lib/standard_circuit/adapter_errors/faraday.rb +21 -0
- data/lib/standard_circuit/adapter_errors/smtp.rb +27 -0
- data/lib/standard_circuit/adapter_errors/stripe.rb +27 -0
- data/lib/standard_circuit/config.rb +86 -0
- data/lib/standard_circuit/controller_support.rb +53 -0
- data/lib/standard_circuit/health.rb +102 -0
- data/lib/standard_circuit/health_controller.rb +28 -0
- data/lib/standard_circuit/mailer/circuit_open_error.rb +23 -0
- data/lib/standard_circuit/mailer/delivery_method.rb +77 -0
- data/lib/standard_circuit/network_errors.rb +22 -0
- data/lib/standard_circuit/notifiers/logger.rb +26 -0
- data/lib/standard_circuit/notifiers/metrics.rb +24 -0
- data/lib/standard_circuit/notifiers/sentry.rb +24 -0
- data/lib/standard_circuit/rspec.rb +18 -0
- data/lib/standard_circuit/runner.rb +164 -0
- data/lib/standard_circuit/version.rb +3 -0
- data/lib/standard_circuit.rb +72 -0
- metadata +126 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 2faee952d1a2a7a131f721af4ac58ce46cbbe94dadc50cd50d4e6ac24af08600
|
|
4
|
+
data.tar.gz: e20f8adb374398ad0fab643b6333e329b6c7f5e187fc927092b1d167b4e290f8
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: '09dd4b8be7afd25bb06044c3b80d41fa39c302ccbb9357cd770127050af30d517cef73d8f6ad03a6112f837ae5d6a278068bc7edc147824403081afe63670f13'
|
|
7
|
+
data.tar.gz: ef93377225b985b979ee54f35d86f9e52354030d0c5cad2131653828ee3a767239206268eb0b18b776b3ec3fa3cfc69fc9d9ed8118a9cd53ca0e9eb74cc6c834
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project will be documented in this file.
|
|
4
|
+
|
|
5
|
+
## [Unreleased]
|
|
6
|
+
|
|
7
|
+
## [0.1.0] - 2026-04-27
|
|
8
|
+
|
|
9
|
+
### Fixed
|
|
10
|
+
- `Mailer::Railtie` is now idempotent: skips `add_delivery_method` when `:standard_circuit` is already in `delivery_methods`. Previously, a host app that pre-registered the delivery method from an env-file `on_load(:action_mailer)` block (a common workaround for the `NoMethodError` that `config.action_mailer.standard_circuit_settings=` triggers during eager_load) would have its settings hash wiped when the gem Railtie's `on_load` fired afterwards, causing `KeyError: key not found: :circuit` at delivery time. Reproduced in production at nutripod-web (Sentry NUTRIPOD-WEB-EE / Linear LMT-454).
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
- `StandardCircuit::Mailer::CircuitOpenError` — now the default `retry_error_class` for the mailer delivery method; consumers no longer need to define their own.
|
|
14
|
+
- Opt-in `StandardCircuit::HealthController` — `require "standard_circuit/health_controller"` then route `get "/health", to: "standard_circuit/health#show"`. Renders `StandardCircuit.health_report` as JSON and returns 503 on `:critical`.
|
|
15
|
+
- Initial extraction from `sidekick-web/app/services/circuit_breaker.rb` (see design doc §4 for the five flaws addressed).
|
|
16
|
+
- `StandardCircuit.run(:name, fallback:, &block)` module-method API.
|
|
17
|
+
- `StandardCircuit::Config` with `register`, `register_prefix`.
|
|
18
|
+
- `StandardCircuit::NetworkErrors` narrow default tracked-errors list.
|
|
19
|
+
- `StandardCircuit::AdapterErrors::{Stripe,Aws,Faraday,Smtp}` modules exposing `server_errors` and `caller_errors`.
|
|
20
|
+
- `StandardCircuit::Notifiers::{Logger,Sentry,Metrics}`.
|
|
21
|
+
- `StandardCircuit::ActiveStorage::S3Service` — per-bucket circuit keying (`:s3_<bucket>`), wraps `upload`/`download`/`download_chunk`/`delete`/`delete_prefixed`/`exist?`/`compose`/`update_metadata`.
|
|
22
|
+
- `StandardCircuit::Mailer::DeliveryMethod` — accepts `underlying:` as instance or symbol.
|
|
23
|
+
- `StandardCircuit::ControllerSupport` — `circuit_open_fallback` DSL for production 503 handling.
|
|
24
|
+
- `StandardCircuit.force_open`, `force_closed`, `reset_force!` + `require "standard_circuit/rspec"` for auto-cleanup.
|
|
25
|
+
|
|
26
|
+
### Changed
|
|
27
|
+
- Removed redundant `defined?(::Sentry::Metrics)` guards in `Runner`, `ControllerSupport`, and `Notifiers::Metrics`. `sentry-ruby` is a hard runtime dependency; the guards were dead code.
|
|
28
|
+
- Tightened `sentry-ruby` lower bound from `>= 5.0` to `>= 5.17`. `Sentry::Metrics` was introduced in 5.17; the previous floor let Bundler resolve a version where the metrics API does not exist.
|
|
29
|
+
|
data/MIT-LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Jaryl Sim
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
|
13
|
+
all copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
|
21
|
+
THE SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# StandardCircuit
|
|
2
|
+
|
|
3
|
+
Circuit breaker primitives for Rails apps, built on [stoplight](https://github.com/bolshakov/stoplight).
|
|
4
|
+
|
|
5
|
+
Wraps the upstream `stoplight` gem with:
|
|
6
|
+
|
|
7
|
+
- Opinionated default error taxonomy (network errors track; caller/config errors do not)
|
|
8
|
+
- SDK-specific adapter error bundles (Stripe, AWS, Faraday, SMTP)
|
|
9
|
+
- Sentry notifier (warning on GREEN→RED) and metrics notifier (`external.circuit_breaker`, `external.request`)
|
|
10
|
+
- ActiveStorage S3 adapter with per-bucket circuit keying
|
|
11
|
+
- Generic ActionMailer delivery-method wrapper (supports both instance and symbol `underlying:` forms)
|
|
12
|
+
- Controller concern for standardized 503 responses on `Stoplight::Error::RedLight`
|
|
13
|
+
- Test helpers (`force_open`, `force_closed`, `reset_force!`) with RSpec auto-cleanup
|
|
14
|
+
|
|
15
|
+
## Installation
|
|
16
|
+
|
|
17
|
+
```ruby
|
|
18
|
+
# Gemfile
|
|
19
|
+
gem "standard_circuit", git: "https://github.com/rarebit-one/standard_circuit", ref: "<sha>"
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
## Quick start
|
|
23
|
+
|
|
24
|
+
```ruby
|
|
25
|
+
# config/initializers/standard_circuit.rb
|
|
26
|
+
StandardCircuit.configure do |c|
|
|
27
|
+
c.sentry_enabled = true
|
|
28
|
+
c.metric_prefix = "external"
|
|
29
|
+
|
|
30
|
+
c.register(:stripe,
|
|
31
|
+
threshold: 5,
|
|
32
|
+
cool_off_time: 30,
|
|
33
|
+
tracked_errors: StandardCircuit::NetworkErrors.defaults + StandardCircuit::AdapterErrors::Stripe.server_errors,
|
|
34
|
+
skipped_errors: StandardCircuit::AdapterErrors::Stripe.caller_errors)
|
|
35
|
+
end
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
```ruby
|
|
39
|
+
# anywhere in app code
|
|
40
|
+
StandardCircuit.run(:stripe) do
|
|
41
|
+
Stripe::PaymentIntent.create(amount:, currency:)
|
|
42
|
+
end
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
## Health endpoint
|
|
46
|
+
|
|
47
|
+
StandardCircuit ships an opt-in controller that renders `StandardCircuit.health_report` as JSON. It returns 503 when the rolled-up status is `:critical` (so orchestrators pull the instance out of rotation) and 200 otherwise.
|
|
48
|
+
|
|
49
|
+
It's opt-in — not auto-required — so apps that don't want a health route don't pay for it.
|
|
50
|
+
|
|
51
|
+
```ruby
|
|
52
|
+
# config/routes.rb
|
|
53
|
+
require "standard_circuit/health_controller"
|
|
54
|
+
|
|
55
|
+
Rails.application.routes.draw do
|
|
56
|
+
get "/health", to: "standard_circuit/health#show"
|
|
57
|
+
end
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
The controller inherits from `ActionController::API` to sidestep app-level filters (authentication, bootstrap redirects, etc.) so probes can call it anonymously.
|
|
61
|
+
|
|
62
|
+
See [`standard_circuit-design.md`](../standard_circuit-design.md) for the full design.
|
|
63
|
+
|
|
64
|
+
## License
|
|
65
|
+
|
|
66
|
+
MIT
|
data/Rakefile
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
# Shim for ActiveStorage::Service::Configurator
|
|
2
|
+
#
|
|
3
|
+
# When storage.yml says `service: StandardCircuitS3`, ActiveStorage's
|
|
4
|
+
# Configurator calls:
|
|
5
|
+
# require "active_storage/service/standard_circuit_s3_service"
|
|
6
|
+
# const_get("StandardCircuitS3Service")
|
|
7
|
+
#
|
|
8
|
+
# This file's only purpose is to exist at that load path so the require
|
|
9
|
+
# succeeds. The actual implementation lives at
|
|
10
|
+
# lib/standard_circuit/active_storage/s3_service.rb so the gem's layout
|
|
11
|
+
# follows the StandardCircuit:: namespace convention.
|
|
12
|
+
|
|
13
|
+
require "standard_circuit/active_storage/s3_service"
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
require "active_storage"
|
|
2
|
+
require "active_storage/service"
|
|
3
|
+
require "active_storage/service/s3_service"
|
|
4
|
+
|
|
5
|
+
# NOTE: ActiveStorage's Configurator does
|
|
6
|
+
# require "active_storage/service/standard_circuit_s3_service"
|
|
7
|
+
# when storage.yml has `service: StandardCircuitS3`. We ship a shim at
|
|
8
|
+
# lib/active_storage/service/standard_circuit_s3_service.rb that requires this
|
|
9
|
+
# file, so the conventional require path works.
|
|
10
|
+
module ActiveStorage
|
|
11
|
+
class Service::StandardCircuitS3Service < Service::S3Service
|
|
12
|
+
WRAPPED_METHODS = %i[
|
|
13
|
+
upload
|
|
14
|
+
download
|
|
15
|
+
download_chunk
|
|
16
|
+
delete
|
|
17
|
+
delete_prefixed
|
|
18
|
+
exist?
|
|
19
|
+
compose
|
|
20
|
+
update_metadata
|
|
21
|
+
].freeze
|
|
22
|
+
|
|
23
|
+
WRAPPED_METHODS.each do |method_name|
|
|
24
|
+
define_method(method_name) do |*args, **kwargs, &block|
|
|
25
|
+
StandardCircuit.run(circuit_name) { super(*args, **kwargs, &block) }
|
|
26
|
+
end
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
private
|
|
30
|
+
|
|
31
|
+
def circuit_name
|
|
32
|
+
:"s3_#{bucket.name}"
|
|
33
|
+
end
|
|
34
|
+
end
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
module StandardCircuit
|
|
38
|
+
module ActiveStorage
|
|
39
|
+
S3Service = ::ActiveStorage::Service::StandardCircuitS3Service
|
|
40
|
+
end
|
|
41
|
+
end
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
module StandardCircuit
|
|
2
|
+
module AdapterErrors
|
|
3
|
+
module Aws
|
|
4
|
+
class << self
|
|
5
|
+
def server_errors
|
|
6
|
+
errors = []
|
|
7
|
+
errors << ::Seahorse::Client::NetworkingError if defined?(::Seahorse::Client::NetworkingError)
|
|
8
|
+
errors << ::Aws::Errors::ServiceError if defined?(::Aws::Errors::ServiceError)
|
|
9
|
+
errors
|
|
10
|
+
end
|
|
11
|
+
|
|
12
|
+
def caller_errors
|
|
13
|
+
return [] unless defined?(::Aws::S3::Errors::NoSuchKey)
|
|
14
|
+
|
|
15
|
+
[
|
|
16
|
+
::Aws::S3::Errors::NoSuchKey,
|
|
17
|
+
::Aws::S3::Errors::AccessDenied
|
|
18
|
+
].select { |klass| klass.is_a?(Class) }
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
end
|
|
22
|
+
end
|
|
23
|
+
end
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
module StandardCircuit
|
|
2
|
+
module AdapterErrors
|
|
3
|
+
module Faraday
|
|
4
|
+
class << self
|
|
5
|
+
def server_errors
|
|
6
|
+
return [] unless defined?(::Faraday::Error)
|
|
7
|
+
|
|
8
|
+
errors = [ ::Faraday::TimeoutError, ::Faraday::ConnectionFailed ]
|
|
9
|
+
errors << ::Faraday::ServerError if defined?(::Faraday::ServerError)
|
|
10
|
+
errors.select { |klass| klass.is_a?(Class) }
|
|
11
|
+
end
|
|
12
|
+
|
|
13
|
+
def caller_errors
|
|
14
|
+
return [] unless defined?(::Faraday::ClientError)
|
|
15
|
+
|
|
16
|
+
[ ::Faraday::ClientError ]
|
|
17
|
+
end
|
|
18
|
+
end
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
end
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
require "net/smtp"
|
|
2
|
+
|
|
3
|
+
module StandardCircuit
|
|
4
|
+
module AdapterErrors
|
|
5
|
+
module Smtp
|
|
6
|
+
SERVER_ERRORS = [
|
|
7
|
+
Net::SMTPServerBusy,
|
|
8
|
+
Net::SMTPFatalError,
|
|
9
|
+
Net::SMTPUnknownError,
|
|
10
|
+
EOFError
|
|
11
|
+
].freeze
|
|
12
|
+
|
|
13
|
+
CALLER_ERRORS = [
|
|
14
|
+
Net::SMTPSyntaxError,
|
|
15
|
+
Net::SMTPAuthenticationError
|
|
16
|
+
].freeze
|
|
17
|
+
|
|
18
|
+
def self.server_errors
|
|
19
|
+
SERVER_ERRORS.dup
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def self.caller_errors
|
|
23
|
+
CALLER_ERRORS.dup
|
|
24
|
+
end
|
|
25
|
+
end
|
|
26
|
+
end
|
|
27
|
+
end
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
module StandardCircuit
|
|
2
|
+
module AdapterErrors
|
|
3
|
+
module Stripe
|
|
4
|
+
class << self
|
|
5
|
+
def server_errors
|
|
6
|
+
return [] unless defined?(::Stripe::StripeError)
|
|
7
|
+
|
|
8
|
+
[
|
|
9
|
+
::Stripe::APIConnectionError,
|
|
10
|
+
::Stripe::RateLimitError,
|
|
11
|
+
::Stripe::APIError
|
|
12
|
+
]
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
def caller_errors
|
|
16
|
+
return [] unless defined?(::Stripe::StripeError)
|
|
17
|
+
|
|
18
|
+
[
|
|
19
|
+
::Stripe::InvalidRequestError,
|
|
20
|
+
::Stripe::CardError,
|
|
21
|
+
::Stripe::AuthenticationError
|
|
22
|
+
]
|
|
23
|
+
end
|
|
24
|
+
end
|
|
25
|
+
end
|
|
26
|
+
end
|
|
27
|
+
end
|
|
@@ -0,0 +1,86 @@
|
|
|
1
|
+
module StandardCircuit
|
|
2
|
+
class Config
|
|
3
|
+
DEFAULT_THRESHOLD = 3
|
|
4
|
+
DEFAULT_COOL_OFF = 30
|
|
5
|
+
DEFAULT_WINDOW = 60
|
|
6
|
+
DEFAULT_CRITICALITY = :standard
|
|
7
|
+
CRITICALITIES = [ :critical, :standard, :optional ].freeze
|
|
8
|
+
|
|
9
|
+
CircuitSpec = Struct.new(
|
|
10
|
+
:threshold,
|
|
11
|
+
:cool_off_time,
|
|
12
|
+
:window_size,
|
|
13
|
+
:tracked_errors,
|
|
14
|
+
:skipped_errors,
|
|
15
|
+
:criticality,
|
|
16
|
+
keyword_init: true
|
|
17
|
+
) do
|
|
18
|
+
def self.build(**opts)
|
|
19
|
+
criticality = opts.fetch(:criticality, DEFAULT_CRITICALITY)
|
|
20
|
+
unless CRITICALITIES.include?(criticality)
|
|
21
|
+
raise ArgumentError,
|
|
22
|
+
"invalid criticality #{criticality.inspect}; must be one of #{CRITICALITIES.inspect}"
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
new(
|
|
26
|
+
threshold: opts.fetch(:threshold, DEFAULT_THRESHOLD),
|
|
27
|
+
cool_off_time: opts.fetch(:cool_off_time, DEFAULT_COOL_OFF),
|
|
28
|
+
window_size: opts.fetch(:window_size, DEFAULT_WINDOW),
|
|
29
|
+
tracked_errors: opts.fetch(:tracked_errors, NetworkErrors.defaults),
|
|
30
|
+
skipped_errors: opts.fetch(:skipped_errors, []),
|
|
31
|
+
criticality: criticality
|
|
32
|
+
)
|
|
33
|
+
end
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
attr_accessor :sentry_enabled, :metric_prefix, :data_store, :logger
|
|
37
|
+
attr_reader :circuits, :prefixes, :extra_notifiers
|
|
38
|
+
|
|
39
|
+
def initialize
|
|
40
|
+
@sentry_enabled = true
|
|
41
|
+
@metric_prefix = "external"
|
|
42
|
+
@data_store = Stoplight::DataStore::Memory.new
|
|
43
|
+
@logger = nil
|
|
44
|
+
@circuits = {}
|
|
45
|
+
@prefixes = {}
|
|
46
|
+
@extra_notifiers = []
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
def notifiers
|
|
50
|
+
built = [ Notifiers::Logger.new(@logger) ]
|
|
51
|
+
built << Notifiers::Sentry.new if @sentry_enabled
|
|
52
|
+
built << Notifiers::Metrics.new(metric_prefix: @metric_prefix)
|
|
53
|
+
built + @extra_notifiers
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
def reset_registry!
|
|
57
|
+
@circuits.clear
|
|
58
|
+
@prefixes.clear
|
|
59
|
+
@extra_notifiers.clear
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
def register(name, **opts)
|
|
63
|
+
@circuits[name.to_sym] = CircuitSpec.build(**opts)
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
def register_prefix(prefix, **opts)
|
|
67
|
+
@prefixes[prefix.to_s] = CircuitSpec.build(**opts)
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
def add_notifier(notifier)
|
|
71
|
+
@extra_notifiers << notifier
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
def spec_for(name)
|
|
75
|
+
@circuits[name.to_sym] || spec_for_prefix(name)
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
private
|
|
79
|
+
|
|
80
|
+
def spec_for_prefix(name)
|
|
81
|
+
key = name.to_s
|
|
82
|
+
_matched_prefix, spec = @prefixes.find { |prefix, _| key.start_with?("#{prefix}_") }
|
|
83
|
+
spec
|
|
84
|
+
end
|
|
85
|
+
end
|
|
86
|
+
end
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
require "active_support/concern"
|
|
2
|
+
require "action_controller"
|
|
3
|
+
|
|
4
|
+
module StandardCircuit
|
|
5
|
+
module ControllerSupport
|
|
6
|
+
extend ActiveSupport::Concern
|
|
7
|
+
|
|
8
|
+
included do
|
|
9
|
+
rescue_from Stoplight::Error::RedLight do |error|
|
|
10
|
+
handle_circuit_open(error)
|
|
11
|
+
end
|
|
12
|
+
|
|
13
|
+
class_attribute :_circuit_open_fallback, instance_accessor: false, default: nil
|
|
14
|
+
end
|
|
15
|
+
|
|
16
|
+
class_methods do
|
|
17
|
+
def circuit_open_fallback(status: :service_unavailable, html: nil, json: nil, stream: nil)
|
|
18
|
+
self._circuit_open_fallback = {
|
|
19
|
+
status: status,
|
|
20
|
+
html: html,
|
|
21
|
+
json: json,
|
|
22
|
+
stream: stream
|
|
23
|
+
}
|
|
24
|
+
end
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
private
|
|
28
|
+
|
|
29
|
+
def handle_circuit_open(error)
|
|
30
|
+
fallback = self.class._circuit_open_fallback || {}
|
|
31
|
+
emit_circuit_open_metric(error)
|
|
32
|
+
|
|
33
|
+
return instance_exec(&fallback[:json]) if request.format.json? && fallback[:json]
|
|
34
|
+
return instance_exec(response, &fallback[:stream]) if streaming_controller? && fallback[:stream]
|
|
35
|
+
return instance_exec(&fallback[:html]) if fallback[:html]
|
|
36
|
+
|
|
37
|
+
head(fallback[:status] || :service_unavailable)
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
def streaming_controller?
|
|
41
|
+
defined?(::ActionController::Live) && self.class.include?(::ActionController::Live)
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
def emit_circuit_open_metric(error)
|
|
45
|
+
prefix = StandardCircuit.config.metric_prefix
|
|
46
|
+
::Sentry::Metrics.count(
|
|
47
|
+
"#{prefix}.request",
|
|
48
|
+
value: 1,
|
|
49
|
+
attributes: { service: error.light_name, status: "circuit_open" }
|
|
50
|
+
)
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
end
|
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
module StandardCircuit
|
|
2
|
+
# Health-reporting helpers that inspect the Runner's Stoplight Light cache
|
|
3
|
+
# and the Config's registered circuits / prefixes and return structured
|
|
4
|
+
# snapshots plus an overall health status.
|
|
5
|
+
#
|
|
6
|
+
# Intended for mounting in a Rails HealthController. Prefer +health_report+
|
|
7
|
+
# over calling +health_snapshot+ and +health_overall+ separately — the
|
|
8
|
+
# combined call takes a single atomic snapshot, so the rendered status and
|
|
9
|
+
# circuits always describe the same moment:
|
|
10
|
+
#
|
|
11
|
+
# report = StandardCircuit.health_report
|
|
12
|
+
# render json: report, status: (report[:status] == :critical ? 503 : 200)
|
|
13
|
+
#
|
|
14
|
+
module Health
|
|
15
|
+
module_function
|
|
16
|
+
|
|
17
|
+
# Build a snapshot of every relevant circuit.
|
|
18
|
+
#
|
|
19
|
+
# Includes: every named circuit registered via +Config#register+ (lights
|
|
20
|
+
# are eagerly built if not yet cached so the snapshot reflects real state
|
|
21
|
+
# instead of "never exercised"); plus every circuit already present in the
|
|
22
|
+
# runner's light cache that was matched via a prefix registration.
|
|
23
|
+
#
|
|
24
|
+
# Prefix-registered circuits that have never been exercised are not
|
|
25
|
+
# enumerable and are therefore omitted.
|
|
26
|
+
def snapshot(runner, config)
|
|
27
|
+
entries = named_entries(runner, config) + prefix_entries(runner, config)
|
|
28
|
+
# Dedupe by name — a named circuit might also match a prefix; the named
|
|
29
|
+
# registration wins and we keep its entry.
|
|
30
|
+
entries.uniq { |entry| entry[:name] }
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
# Roll the snapshot up to :ok | :degraded | :critical.
|
|
34
|
+
#
|
|
35
|
+
# - any :critical circuit RED -> :critical
|
|
36
|
+
# - any :critical circuit YELLOW -> :degraded
|
|
37
|
+
# - any :standard circuit RED -> :degraded
|
|
38
|
+
# - otherwise -> :ok
|
|
39
|
+
#
|
|
40
|
+
# :optional circuits never elevate the overall state.
|
|
41
|
+
def overall(snapshot)
|
|
42
|
+
return :critical if snapshot.any? { |e| e[:criticality] == :critical && e[:color] == "red" }
|
|
43
|
+
|
|
44
|
+
degraded = snapshot.any? do |e|
|
|
45
|
+
(e[:criticality] == :critical && e[:color] == "yellow") ||
|
|
46
|
+
(e[:criticality] == :standard && e[:color] == "red")
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
degraded ? :degraded : :ok
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
# --- internals -------------------------------------------------------
|
|
53
|
+
|
|
54
|
+
def named_entries(runner, config)
|
|
55
|
+
config.circuits.map do |name, spec|
|
|
56
|
+
build_entry(runner.light_for(name), spec, config)
|
|
57
|
+
end
|
|
58
|
+
end
|
|
59
|
+
private_class_method :named_entries
|
|
60
|
+
|
|
61
|
+
def prefix_entries(runner, config)
|
|
62
|
+
runner.cached_lights.filter_map do |name, light|
|
|
63
|
+
next if config.circuits.key?(name.to_sym)
|
|
64
|
+
|
|
65
|
+
spec = config.spec_for(name)
|
|
66
|
+
next unless spec # defensive: light exists but no matching registration
|
|
67
|
+
|
|
68
|
+
build_entry(light, spec, config)
|
|
69
|
+
end
|
|
70
|
+
end
|
|
71
|
+
private_class_method :prefix_entries
|
|
72
|
+
|
|
73
|
+
def build_entry(light, spec, config)
|
|
74
|
+
{
|
|
75
|
+
name: light.name.to_sym,
|
|
76
|
+
color: light.color,
|
|
77
|
+
locked: locked?(light, config),
|
|
78
|
+
criticality: spec.criticality
|
|
79
|
+
}
|
|
80
|
+
end
|
|
81
|
+
private_class_method :build_entry
|
|
82
|
+
|
|
83
|
+
def locked?(light, config)
|
|
84
|
+
state = safe_state(light, config)
|
|
85
|
+
state == Stoplight::State::LOCKED_RED || state == Stoplight::State::LOCKED_GREEN
|
|
86
|
+
end
|
|
87
|
+
private_class_method :locked?
|
|
88
|
+
|
|
89
|
+
# Reading +light.state+ can raise when the data store is unreachable
|
|
90
|
+
# (e.g. Redis connection dropped). Swallow the failure so the health
|
|
91
|
+
# endpoint still returns the rest of the snapshot, but log via the
|
|
92
|
+
# passed-in config's logger so operators see the underlying fault and
|
|
93
|
+
# tests that inject a throwaway Config don't leak to the global logger.
|
|
94
|
+
def safe_state(light, config)
|
|
95
|
+
light.state
|
|
96
|
+
rescue StandardError => e
|
|
97
|
+
config.logger&.warn("StandardCircuit::Health.safe_state failed for #{light.name}: #{e.class}: #{e.message}")
|
|
98
|
+
nil
|
|
99
|
+
end
|
|
100
|
+
private_class_method :safe_state
|
|
101
|
+
end
|
|
102
|
+
end
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
require "action_controller"
|
|
2
|
+
|
|
3
|
+
module StandardCircuit
|
|
4
|
+
# Opt-in health-check controller. Renders +StandardCircuit.health_report+ as
|
|
5
|
+
# JSON and returns 503 when the rolled-up status is +:critical+ so upstream
|
|
6
|
+
# orchestrators pull the instance out of rotation. :degraded and :ok both
|
|
7
|
+
# return 200 — the app can still serve most traffic.
|
|
8
|
+
#
|
|
9
|
+
# This controller is not auto-loaded. Consumers opt in:
|
|
10
|
+
#
|
|
11
|
+
# # config/routes.rb
|
|
12
|
+
# require "standard_circuit/health_controller"
|
|
13
|
+
#
|
|
14
|
+
# Rails.application.routes.draw do
|
|
15
|
+
# get "/health", to: "standard_circuit/health#show"
|
|
16
|
+
# end
|
|
17
|
+
#
|
|
18
|
+
# Inherits from +ActionController::API+ to sidestep any ApplicationController
|
|
19
|
+
# filters (authentication, bootstrap redirects, etc.) — health probes must be
|
|
20
|
+
# callable anonymously from load balancers and uptime monitors.
|
|
21
|
+
class HealthController < ::ActionController::API
|
|
22
|
+
def show
|
|
23
|
+
report = StandardCircuit.health_report
|
|
24
|
+
http_status = report[:status] == :critical ? :service_unavailable : :ok
|
|
25
|
+
render json: report, status: http_status
|
|
26
|
+
end
|
|
27
|
+
end
|
|
28
|
+
end
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
module StandardCircuit
|
|
2
|
+
module Mailer
|
|
3
|
+
# Default error class raised by +StandardCircuit::Mailer::DeliveryMethod+
|
|
4
|
+
# when the wrapped mailer circuit is open. Mailer jobs can rescue/retry on
|
|
5
|
+
# this class to defer delivery until the upstream recovers.
|
|
6
|
+
#
|
|
7
|
+
# Constructor contract (matches DeliveryMethod#deliver!):
|
|
8
|
+
# new(recipients:, subject:)
|
|
9
|
+
#
|
|
10
|
+
# Consumers that want their own error class can still pass
|
|
11
|
+
# +retry_error_class:+ to the delivery method settings; this class is the
|
|
12
|
+
# default when none is provided.
|
|
13
|
+
class CircuitOpenError < StandardError
|
|
14
|
+
attr_reader :recipients, :subject
|
|
15
|
+
|
|
16
|
+
def initialize(recipients:, subject:)
|
|
17
|
+
@recipients = recipients
|
|
18
|
+
@subject = subject
|
|
19
|
+
super("Circuit breaker is open: to=#{recipients.inspect} subject=#{subject.inspect}")
|
|
20
|
+
end
|
|
21
|
+
end
|
|
22
|
+
end
|
|
23
|
+
end
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
require "action_mailer"
|
|
2
|
+
require_relative "circuit_open_error"
|
|
3
|
+
|
|
4
|
+
module StandardCircuit
|
|
5
|
+
module Mailer
|
|
6
|
+
class DeliveryMethod
|
|
7
|
+
attr_accessor :settings
|
|
8
|
+
|
|
9
|
+
def initialize(settings)
|
|
10
|
+
@settings = settings
|
|
11
|
+
@underlying = nil
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
def deliver!(mail)
|
|
15
|
+
StandardCircuit.run(circuit_name) do
|
|
16
|
+
underlying_instance.deliver!(mail)
|
|
17
|
+
end
|
|
18
|
+
rescue Stoplight::Error::RedLight
|
|
19
|
+
raise retry_error_class.new(
|
|
20
|
+
recipients: Array(mail.to),
|
|
21
|
+
subject: mail.subject
|
|
22
|
+
)
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
private
|
|
26
|
+
|
|
27
|
+
def circuit_name
|
|
28
|
+
settings.fetch(:circuit)
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
def retry_error_class
|
|
32
|
+
settings.fetch(:retry_error_class, StandardCircuit::Mailer::CircuitOpenError)
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
def underlying_instance
|
|
36
|
+
@underlying ||= resolve_underlying
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
def resolve_underlying
|
|
40
|
+
target = settings.fetch(:underlying)
|
|
41
|
+
return target unless target.is_a?(Symbol)
|
|
42
|
+
|
|
43
|
+
registry = ::ActionMailer::Base.delivery_methods
|
|
44
|
+
klass = registry.fetch(target) do
|
|
45
|
+
raise ArgumentError, "unknown delivery method #{target.inspect}. Registered: #{registry.keys.inspect}"
|
|
46
|
+
end
|
|
47
|
+
klass.new(settings.fetch(:underlying_settings, {}))
|
|
48
|
+
end
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
if defined?(::Rails::Railtie)
|
|
52
|
+
class Railtie < ::Rails::Railtie
|
|
53
|
+
# ActionMailer's `add_delivery_method` unconditionally resets
|
|
54
|
+
# `<symbol>_settings` to its `default_options` arg. If the host app
|
|
55
|
+
# has already registered :standard_circuit (e.g. from its own
|
|
56
|
+
# `on_load(:action_mailer)` block in an env file, queued earlier),
|
|
57
|
+
# calling `add_delivery_method` again would wipe the settings the
|
|
58
|
+
# app just wrote — causing `KeyError: key not found: :circuit` at
|
|
59
|
+
# delivery time. Skip if already registered. (This guard assumes the
|
|
60
|
+
# host app registers first; if you call `add_delivery_method` from
|
|
61
|
+
# `after_initialize` or otherwise after the gem's Railtie hook fires,
|
|
62
|
+
# use `standard_circuit_settings=` directly to avoid the reset.)
|
|
63
|
+
def self.install(mailer_class)
|
|
64
|
+
return if mailer_class.delivery_methods.key?(:standard_circuit)
|
|
65
|
+
|
|
66
|
+
mailer_class.add_delivery_method :standard_circuit, StandardCircuit::Mailer::DeliveryMethod
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
initializer "standard_circuit.action_mailer" do
|
|
70
|
+
ActiveSupport.on_load(:action_mailer) do
|
|
71
|
+
StandardCircuit::Mailer::Railtie.install(self)
|
|
72
|
+
end
|
|
73
|
+
end
|
|
74
|
+
end
|
|
75
|
+
end
|
|
76
|
+
end
|
|
77
|
+
end
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
require "net/http"
|
|
2
|
+
require "openssl"
|
|
3
|
+
require "socket"
|
|
4
|
+
|
|
5
|
+
module StandardCircuit
|
|
6
|
+
module NetworkErrors
|
|
7
|
+
DEFAULTS = [
|
|
8
|
+
Net::OpenTimeout,
|
|
9
|
+
Net::ReadTimeout,
|
|
10
|
+
Errno::ECONNREFUSED,
|
|
11
|
+
Errno::ECONNRESET,
|
|
12
|
+
Errno::EHOSTUNREACH,
|
|
13
|
+
Errno::ETIMEDOUT,
|
|
14
|
+
SocketError,
|
|
15
|
+
OpenSSL::SSL::SSLError
|
|
16
|
+
].freeze
|
|
17
|
+
|
|
18
|
+
def self.defaults
|
|
19
|
+
DEFAULTS.dup
|
|
20
|
+
end
|
|
21
|
+
end
|
|
22
|
+
end
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
require "logger"
|
|
2
|
+
|
|
3
|
+
module StandardCircuit
|
|
4
|
+
module Notifiers
|
|
5
|
+
class Logger
|
|
6
|
+
def initialize(logger = nil)
|
|
7
|
+
@logger = logger || ::Logger.new($stdout)
|
|
8
|
+
end
|
|
9
|
+
|
|
10
|
+
def notify(light, from_color, to_color, error)
|
|
11
|
+
message = build_message(light, from_color, to_color, error)
|
|
12
|
+
level = to_color == Stoplight::Color::RED ? :warn : :info
|
|
13
|
+
@logger.public_send(level, message)
|
|
14
|
+
message
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
private
|
|
18
|
+
|
|
19
|
+
def build_message(light, from_color, to_color, error)
|
|
20
|
+
words = [ "Stoplight", light.name, "switched from", from_color, "to", to_color ]
|
|
21
|
+
words += [ "because", error.class.name, error.message ] if error
|
|
22
|
+
words.join(" ")
|
|
23
|
+
end
|
|
24
|
+
end
|
|
25
|
+
end
|
|
26
|
+
end
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
module StandardCircuit
|
|
2
|
+
module Notifiers
|
|
3
|
+
class Metrics
|
|
4
|
+
STATE_FOR_COLOR = {
|
|
5
|
+
Stoplight::Color::RED => "opened",
|
|
6
|
+
Stoplight::Color::GREEN => "closed",
|
|
7
|
+
Stoplight::Color::YELLOW => "half_open"
|
|
8
|
+
}.freeze
|
|
9
|
+
|
|
10
|
+
def initialize(metric_prefix: "external")
|
|
11
|
+
@metric_prefix = metric_prefix
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
def notify(light, _from_color, to_color, _error)
|
|
15
|
+
state = STATE_FOR_COLOR.fetch(to_color, to_color)
|
|
16
|
+
::Sentry::Metrics.count(
|
|
17
|
+
"#{@metric_prefix}.circuit_breaker",
|
|
18
|
+
value: 1,
|
|
19
|
+
attributes: { service: light.name, state: state }
|
|
20
|
+
)
|
|
21
|
+
end
|
|
22
|
+
end
|
|
23
|
+
end
|
|
24
|
+
end
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
module StandardCircuit
|
|
2
|
+
module Notifiers
|
|
3
|
+
class Sentry
|
|
4
|
+
def notify(light, from_color, to_color, error)
|
|
5
|
+
return unless to_color == Stoplight::Color::RED
|
|
6
|
+
return unless defined?(::Sentry) && ::Sentry.respond_to?(:capture_message)
|
|
7
|
+
|
|
8
|
+
message = "Circuit breaker opened: #{light.name}"
|
|
9
|
+
::Sentry.capture_message(
|
|
10
|
+
message,
|
|
11
|
+
level: :warning,
|
|
12
|
+
extra: {
|
|
13
|
+
circuit: light.name,
|
|
14
|
+
from_color: from_color,
|
|
15
|
+
to_color: to_color,
|
|
16
|
+
error_class: error&.class&.name,
|
|
17
|
+
error_message: error&.message
|
|
18
|
+
}.compact
|
|
19
|
+
)
|
|
20
|
+
message
|
|
21
|
+
end
|
|
22
|
+
end
|
|
23
|
+
end
|
|
24
|
+
end
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
require "standard_circuit"
|
|
2
|
+
|
|
3
|
+
# Full StandardCircuit state reset between examples.
|
|
4
|
+
#
|
|
5
|
+
# - Clears the light cache so rebuilt lights pick up fresh config if a spec
|
|
6
|
+
# mutates it.
|
|
7
|
+
# - Clears forced states (force_open / force_closed).
|
|
8
|
+
# - Swaps a fresh Stoplight::DataStore::Memory into the Config when the
|
|
9
|
+
# current store is already Memory, so failure counters from one spec don't
|
|
10
|
+
# leak into the next. Redis stores are left alone.
|
|
11
|
+
#
|
|
12
|
+
# This is intentionally `before(:each)` rather than `after(:each)` so the
|
|
13
|
+
# setup happens even when a previous example aborted in an after hook.
|
|
14
|
+
RSpec.configure do |config|
|
|
15
|
+
config.before(:each) do
|
|
16
|
+
StandardCircuit.reset!
|
|
17
|
+
end
|
|
18
|
+
end
|
|
@@ -0,0 +1,164 @@
|
|
|
1
|
+
module StandardCircuit
|
|
2
|
+
class Runner
|
|
3
|
+
def initialize
|
|
4
|
+
@lights = Concurrent::Map.new
|
|
5
|
+
@forced_states = Concurrent::Map.new
|
|
6
|
+
@config = Config.new
|
|
7
|
+
end
|
|
8
|
+
|
|
9
|
+
def apply_config!(config)
|
|
10
|
+
@config = config
|
|
11
|
+
@lights.clear
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
def run(name, fallback: nil, &block)
|
|
15
|
+
forced = @forced_states[name.to_sym]
|
|
16
|
+
return run_forced_open(name, fallback) if forced == :open
|
|
17
|
+
return yield if forced == :closed
|
|
18
|
+
|
|
19
|
+
execute(name, fallback, &block)
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def force_open(name, &block)
|
|
23
|
+
apply_force(name, :open, &block)
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def force_closed(name, &block)
|
|
27
|
+
apply_force(name, :closed, &block)
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def reset_force!
|
|
31
|
+
@forced_states.clear
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
# Full state reset for tests. Clears the light cache, forced states, AND
|
|
35
|
+
# swaps the Stoplight data store to a fresh Memory instance. Clearing the
|
|
36
|
+
# cache alone is not sufficient: failure counters live in the data store,
|
|
37
|
+
# so a spec that deliberately trips a circuit would otherwise leak RedLight
|
|
38
|
+
# into later specs. We only swap the store when the existing one is already
|
|
39
|
+
# a Memory store, because replacing a Redis store (production config) would
|
|
40
|
+
# silently defeat shared-process coordination.
|
|
41
|
+
def reset!
|
|
42
|
+
@lights.clear
|
|
43
|
+
@forced_states.clear
|
|
44
|
+
return unless @config.data_store.is_a?(Stoplight::DataStore::Memory)
|
|
45
|
+
|
|
46
|
+
@config.data_store = Stoplight::DataStore::Memory.new
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
def light_for(name)
|
|
50
|
+
@lights.compute_if_absent(name.to_sym) { build_light(name.to_sym) }
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
# Snapshot Hash of the current cached lights. Used by Health to discover
|
|
54
|
+
# prefix-matched circuits that have been exercised at least once — we can't
|
|
55
|
+
# enumerate prefix-matched dynamic names any other way.
|
|
56
|
+
def cached_lights
|
|
57
|
+
hash = {}
|
|
58
|
+
@lights.each_pair { |name, light| hash[name] = light }
|
|
59
|
+
hash
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
def health_snapshot
|
|
63
|
+
Health.snapshot(self, @config)
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
# Accepts an optional pre-computed snapshot so callers that also need the
|
|
67
|
+
# raw circuits list don't read the data store twice (which could yield
|
|
68
|
+
# an inconsistent status/circuits pair). Prefer +health_report+ when you
|
|
69
|
+
# need both.
|
|
70
|
+
def health_overall(snapshot = nil)
|
|
71
|
+
Health.overall(snapshot || health_snapshot)
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
# Atomic health report — takes a single snapshot and returns both the
|
|
75
|
+
# rolled-up status and the per-circuit list. Preferred for rendering a
|
|
76
|
+
# health-check endpoint so status and circuits describe the same moment.
|
|
77
|
+
def health_report
|
|
78
|
+
snapshot = health_snapshot
|
|
79
|
+
{ status: Health.overall(snapshot), circuits: snapshot }
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
private
|
|
83
|
+
|
|
84
|
+
def execute(name, fallback, &block)
|
|
85
|
+
started_at = monotonic_now
|
|
86
|
+
light = light_for(name)
|
|
87
|
+
|
|
88
|
+
result = fallback ? light.run(fallback, &block) : light.run(&block)
|
|
89
|
+
emit_request_metric(name, :success, duration_ms(started_at))
|
|
90
|
+
result
|
|
91
|
+
rescue Stoplight::Error::RedLight => e
|
|
92
|
+
emit_request_metric(name, :circuit_open, duration_ms(started_at))
|
|
93
|
+
raise e unless fallback
|
|
94
|
+
|
|
95
|
+
fallback.call(nil)
|
|
96
|
+
rescue StandardError => e
|
|
97
|
+
emit_request_metric(name, :failure, duration_ms(started_at))
|
|
98
|
+
raise e
|
|
99
|
+
end
|
|
100
|
+
|
|
101
|
+
def run_forced_open(name, fallback)
|
|
102
|
+
emit_request_metric(name, :circuit_open, 0)
|
|
103
|
+
return fallback.call(nil) if fallback
|
|
104
|
+
|
|
105
|
+
spec = @config.spec_for(name)
|
|
106
|
+
raise Stoplight::Error::RedLight.new(
|
|
107
|
+
name.to_s,
|
|
108
|
+
cool_off_time: spec&.cool_off_time || Config::DEFAULT_COOL_OFF,
|
|
109
|
+
retry_after: nil
|
|
110
|
+
)
|
|
111
|
+
end
|
|
112
|
+
|
|
113
|
+
def apply_force(name, state)
|
|
114
|
+
key = name.to_sym
|
|
115
|
+
return set_force(key, state) unless block_given?
|
|
116
|
+
|
|
117
|
+
prior = @forced_states[key]
|
|
118
|
+
set_force(key, state)
|
|
119
|
+
begin
|
|
120
|
+
yield
|
|
121
|
+
ensure
|
|
122
|
+
prior.nil? ? @forced_states.delete(key) : set_force(key, prior)
|
|
123
|
+
end
|
|
124
|
+
end
|
|
125
|
+
|
|
126
|
+
def set_force(key, state)
|
|
127
|
+
@forced_states[key] = state
|
|
128
|
+
end
|
|
129
|
+
|
|
130
|
+
def build_light(name)
|
|
131
|
+
spec = @config.spec_for(name) or raise UnknownCircuit, "no circuit registered for #{name.inspect}"
|
|
132
|
+
|
|
133
|
+
Stoplight.light(
|
|
134
|
+
name.to_s,
|
|
135
|
+
threshold: spec.threshold,
|
|
136
|
+
cool_off_time: spec.cool_off_time,
|
|
137
|
+
window_size: spec.window_size,
|
|
138
|
+
tracked_errors: spec.tracked_errors,
|
|
139
|
+
skipped_errors: spec.skipped_errors,
|
|
140
|
+
data_store: @config.data_store,
|
|
141
|
+
notifiers: notifiers
|
|
142
|
+
)
|
|
143
|
+
end
|
|
144
|
+
|
|
145
|
+
def notifiers
|
|
146
|
+
@config.notifiers
|
|
147
|
+
end
|
|
148
|
+
|
|
149
|
+
def emit_request_metric(name, status, duration)
|
|
150
|
+
prefix = @config.metric_prefix
|
|
151
|
+
attrs = { service: name.to_s, status: status.to_s }
|
|
152
|
+
::Sentry::Metrics.count("#{prefix}.request", value: 1, attributes: attrs)
|
|
153
|
+
::Sentry::Metrics.distribution("#{prefix}.request.duration", duration, unit: "millisecond", attributes: attrs)
|
|
154
|
+
end
|
|
155
|
+
|
|
156
|
+
def monotonic_now
|
|
157
|
+
Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
158
|
+
end
|
|
159
|
+
|
|
160
|
+
def duration_ms(started_at)
|
|
161
|
+
((monotonic_now - started_at) * 1000.0).round(2)
|
|
162
|
+
end
|
|
163
|
+
end
|
|
164
|
+
end
|
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
require "stoplight"
|
|
2
|
+
require "concurrent"
|
|
3
|
+
require "sentry-ruby"
|
|
4
|
+
|
|
5
|
+
require "standard_circuit/version"
|
|
6
|
+
require "standard_circuit/network_errors"
|
|
7
|
+
require "standard_circuit/adapter_errors/stripe"
|
|
8
|
+
require "standard_circuit/adapter_errors/aws"
|
|
9
|
+
require "standard_circuit/adapter_errors/faraday"
|
|
10
|
+
require "standard_circuit/adapter_errors/smtp"
|
|
11
|
+
require "standard_circuit/notifiers/logger"
|
|
12
|
+
require "standard_circuit/notifiers/sentry"
|
|
13
|
+
require "standard_circuit/notifiers/metrics"
|
|
14
|
+
require "standard_circuit/config"
|
|
15
|
+
require "standard_circuit/health"
|
|
16
|
+
require "standard_circuit/runner"
|
|
17
|
+
require "standard_circuit/mailer/circuit_open_error"
|
|
18
|
+
require "standard_circuit/mailer/delivery_method"
|
|
19
|
+
require "standard_circuit/controller_support"
|
|
20
|
+
|
|
21
|
+
module StandardCircuit
|
|
22
|
+
class Error < StandardError; end
|
|
23
|
+
class UnknownCircuit < Error; end
|
|
24
|
+
|
|
25
|
+
class << self
|
|
26
|
+
def configure
|
|
27
|
+
yield config
|
|
28
|
+
runner.apply_config!(config)
|
|
29
|
+
config
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def config
|
|
33
|
+
@config ||= Config.new
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
def runner
|
|
37
|
+
@runner ||= Runner.new
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
def run(name, fallback: nil, &block)
|
|
41
|
+
runner.run(name, fallback: fallback, &block)
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
def force_open(name, &block)
|
|
45
|
+
runner.force_open(name, &block)
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
def force_closed(name, &block)
|
|
49
|
+
runner.force_closed(name, &block)
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
def reset_force!
|
|
53
|
+
runner.reset_force!
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
def reset!
|
|
57
|
+
runner.reset!
|
|
58
|
+
end
|
|
59
|
+
|
|
60
|
+
def health_snapshot
|
|
61
|
+
runner.health_snapshot
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
def health_overall(snapshot = nil)
|
|
65
|
+
runner.health_overall(snapshot)
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
def health_report
|
|
69
|
+
runner.health_report
|
|
70
|
+
end
|
|
71
|
+
end
|
|
72
|
+
end
|
metadata
ADDED
|
@@ -0,0 +1,126 @@
|
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
|
2
|
+
name: standard_circuit
|
|
3
|
+
version: !ruby/object:Gem::Version
|
|
4
|
+
version: 0.1.0
|
|
5
|
+
platform: ruby
|
|
6
|
+
authors:
|
|
7
|
+
- Jaryl Sim
|
|
8
|
+
bindir: bin
|
|
9
|
+
cert_chain: []
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
11
|
+
dependencies:
|
|
12
|
+
- !ruby/object:Gem::Dependency
|
|
13
|
+
name: stoplight
|
|
14
|
+
requirement: !ruby/object:Gem::Requirement
|
|
15
|
+
requirements:
|
|
16
|
+
- - "~>"
|
|
17
|
+
- !ruby/object:Gem::Version
|
|
18
|
+
version: '5.8'
|
|
19
|
+
type: :runtime
|
|
20
|
+
prerelease: false
|
|
21
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
22
|
+
requirements:
|
|
23
|
+
- - "~>"
|
|
24
|
+
- !ruby/object:Gem::Version
|
|
25
|
+
version: '5.8'
|
|
26
|
+
- !ruby/object:Gem::Dependency
|
|
27
|
+
name: concurrent-ruby
|
|
28
|
+
requirement: !ruby/object:Gem::Requirement
|
|
29
|
+
requirements:
|
|
30
|
+
- - "~>"
|
|
31
|
+
- !ruby/object:Gem::Version
|
|
32
|
+
version: '1.3'
|
|
33
|
+
type: :runtime
|
|
34
|
+
prerelease: false
|
|
35
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
36
|
+
requirements:
|
|
37
|
+
- - "~>"
|
|
38
|
+
- !ruby/object:Gem::Version
|
|
39
|
+
version: '1.3'
|
|
40
|
+
- !ruby/object:Gem::Dependency
|
|
41
|
+
name: sentry-ruby
|
|
42
|
+
requirement: !ruby/object:Gem::Requirement
|
|
43
|
+
requirements:
|
|
44
|
+
- - ">="
|
|
45
|
+
- !ruby/object:Gem::Version
|
|
46
|
+
version: '5.17'
|
|
47
|
+
type: :runtime
|
|
48
|
+
prerelease: false
|
|
49
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
50
|
+
requirements:
|
|
51
|
+
- - ">="
|
|
52
|
+
- !ruby/object:Gem::Version
|
|
53
|
+
version: '5.17'
|
|
54
|
+
- !ruby/object:Gem::Dependency
|
|
55
|
+
name: railties
|
|
56
|
+
requirement: !ruby/object:Gem::Requirement
|
|
57
|
+
requirements:
|
|
58
|
+
- - ">="
|
|
59
|
+
- !ruby/object:Gem::Version
|
|
60
|
+
version: '8.0'
|
|
61
|
+
type: :runtime
|
|
62
|
+
prerelease: false
|
|
63
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
64
|
+
requirements:
|
|
65
|
+
- - ">="
|
|
66
|
+
- !ruby/object:Gem::Version
|
|
67
|
+
version: '8.0'
|
|
68
|
+
description: StandardCircuit wraps the stoplight gem with opinionated error taxonomy,
|
|
69
|
+
Sentry notifiers, ActiveStorage S3 and ActionMailer adapters, and test helpers shared
|
|
70
|
+
across Rails apps.
|
|
71
|
+
email:
|
|
72
|
+
- code@jaryl.dev
|
|
73
|
+
executables: []
|
|
74
|
+
extensions: []
|
|
75
|
+
extra_rdoc_files: []
|
|
76
|
+
files:
|
|
77
|
+
- CHANGELOG.md
|
|
78
|
+
- MIT-LICENSE
|
|
79
|
+
- README.md
|
|
80
|
+
- Rakefile
|
|
81
|
+
- lib/active_storage/service/standard_circuit_s3_service.rb
|
|
82
|
+
- lib/standard_circuit.rb
|
|
83
|
+
- lib/standard_circuit/active_storage/s3_service.rb
|
|
84
|
+
- lib/standard_circuit/adapter_errors/aws.rb
|
|
85
|
+
- lib/standard_circuit/adapter_errors/faraday.rb
|
|
86
|
+
- lib/standard_circuit/adapter_errors/smtp.rb
|
|
87
|
+
- lib/standard_circuit/adapter_errors/stripe.rb
|
|
88
|
+
- lib/standard_circuit/config.rb
|
|
89
|
+
- lib/standard_circuit/controller_support.rb
|
|
90
|
+
- lib/standard_circuit/health.rb
|
|
91
|
+
- lib/standard_circuit/health_controller.rb
|
|
92
|
+
- lib/standard_circuit/mailer/circuit_open_error.rb
|
|
93
|
+
- lib/standard_circuit/mailer/delivery_method.rb
|
|
94
|
+
- lib/standard_circuit/network_errors.rb
|
|
95
|
+
- lib/standard_circuit/notifiers/logger.rb
|
|
96
|
+
- lib/standard_circuit/notifiers/metrics.rb
|
|
97
|
+
- lib/standard_circuit/notifiers/sentry.rb
|
|
98
|
+
- lib/standard_circuit/rspec.rb
|
|
99
|
+
- lib/standard_circuit/runner.rb
|
|
100
|
+
- lib/standard_circuit/version.rb
|
|
101
|
+
homepage: https://github.com/rarebit-one/standard_circuit
|
|
102
|
+
licenses:
|
|
103
|
+
- MIT
|
|
104
|
+
metadata:
|
|
105
|
+
homepage_uri: https://github.com/rarebit-one/standard_circuit
|
|
106
|
+
source_code_uri: https://github.com/rarebit-one/standard_circuit
|
|
107
|
+
changelog_uri: https://github.com/rarebit-one/standard_circuit/blob/main/CHANGELOG.md
|
|
108
|
+
bug_tracker_uri: https://github.com/rarebit-one/standard_circuit/issues
|
|
109
|
+
rdoc_options: []
|
|
110
|
+
require_paths:
|
|
111
|
+
- lib
|
|
112
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
|
113
|
+
requirements:
|
|
114
|
+
- - ">="
|
|
115
|
+
- !ruby/object:Gem::Version
|
|
116
|
+
version: '3.4'
|
|
117
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
118
|
+
requirements:
|
|
119
|
+
- - ">="
|
|
120
|
+
- !ruby/object:Gem::Version
|
|
121
|
+
version: '0'
|
|
122
|
+
requirements: []
|
|
123
|
+
rubygems_version: 4.0.3
|
|
124
|
+
specification_version: 4
|
|
125
|
+
summary: Circuit breaker primitives for Rails apps, built on stoplight.
|
|
126
|
+
test_files: []
|