faulty 0.1.1 → 0.3.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.rubocop.yml +9 -0
- data/.travis.yml +4 -2
- data/CHANGELOG.md +37 -1
- data/Gemfile +17 -0
- data/README.md +333 -55
- data/bin/check-version +5 -1
- data/bin/console +1 -1
- data/faulty.gemspec +3 -10
- data/lib/faulty.rb +149 -43
- data/lib/faulty/cache.rb +3 -1
- data/lib/faulty/cache/auto_wire.rb +65 -0
- data/lib/faulty/cache/circuit_proxy.rb +61 -0
- data/lib/faulty/cache/default.rb +10 -21
- data/lib/faulty/cache/fault_tolerant_proxy.rb +15 -4
- data/lib/faulty/cache/interface.rb +1 -1
- data/lib/faulty/cache/mock.rb +1 -1
- data/lib/faulty/cache/null.rb +1 -1
- data/lib/faulty/cache/rails.rb +9 -10
- data/lib/faulty/circuit.rb +10 -5
- data/lib/faulty/error.rb +18 -4
- data/lib/faulty/events.rb +3 -2
- data/lib/faulty/events/callback_listener.rb +1 -1
- data/lib/faulty/events/honeybadger_listener.rb +53 -0
- data/lib/faulty/events/listener_interface.rb +1 -1
- data/lib/faulty/events/log_listener.rb +1 -1
- data/lib/faulty/events/notifier.rb +11 -2
- data/lib/faulty/immutable_options.rb +1 -1
- data/lib/faulty/result.rb +2 -2
- data/lib/faulty/status.rb +1 -1
- data/lib/faulty/storage.rb +4 -1
- data/lib/faulty/storage/auto_wire.rb +122 -0
- data/lib/faulty/storage/circuit_proxy.rb +64 -0
- data/lib/faulty/storage/fallback_chain.rb +207 -0
- data/lib/faulty/storage/fault_tolerant_proxy.rb +55 -60
- data/lib/faulty/storage/interface.rb +1 -1
- data/lib/faulty/storage/memory.rb +8 -4
- data/lib/faulty/storage/redis.rb +75 -13
- data/lib/faulty/version.rb +2 -2
- metadata +13 -118
- data/lib/faulty/scope.rb +0 -117
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: c0712dd797dfe2922e4e52a792f48c5ac72c4b9766d218dfa597e7ad630f9fd0
|
4
|
+
data.tar.gz: fb4a787507006131f34a301fc164466fd8befb72051550ca0a63fc2d69ec949c
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 5381dd13f058045906330c67ff80aeefac55ae81077eb4552d5461d743344fa097e205b8ac3a39b385c52976e64315aec4e903c1a0c298a05452fab7788d1df2
|
7
|
+
data.tar.gz: af31cbf86cac54226189f8f63b1d8b0bcf82f9023085e98d07e4e9db65e402883c2219dbf255e21427c40b0c0a66ec71bd4dc9dfdd94fb9bc7b574ddf8426674
|
data/.rubocop.yml
CHANGED
@@ -29,6 +29,9 @@ Layout/LineLength:
|
|
29
29
|
Layout/MultilineMethodCallIndentation:
|
30
30
|
EnforcedStyle: indented
|
31
31
|
|
32
|
+
Layout/RescueEnsureAlignment:
|
33
|
+
Enabled: false
|
34
|
+
|
32
35
|
Lint/RaiseException:
|
33
36
|
Enabled: true
|
34
37
|
|
@@ -44,6 +47,9 @@ RSpec/FilePath:
|
|
44
47
|
RSpec/NamedSubject:
|
45
48
|
Enabled: false
|
46
49
|
|
50
|
+
RSpec/MessageSpies:
|
51
|
+
Enabled: false
|
52
|
+
|
47
53
|
RSpec/MultipleExpectations:
|
48
54
|
Enabled: false
|
49
55
|
|
@@ -59,6 +65,9 @@ Metrics/BlockLength:
|
|
59
65
|
Metrics/MethodLength:
|
60
66
|
Max: 30
|
61
67
|
|
68
|
+
Naming/MethodParameterName:
|
69
|
+
MinNameLength: 1
|
70
|
+
|
62
71
|
Style/Documentation:
|
63
72
|
Enabled: false
|
64
73
|
|
data/.travis.yml
CHANGED
@@ -6,6 +6,8 @@ rvm:
|
|
6
6
|
- 2.5
|
7
7
|
- 2.6
|
8
8
|
- 2.7
|
9
|
+
- jruby-9.2.10
|
10
|
+
- truffleruby-20.2.0
|
9
11
|
|
10
12
|
env:
|
11
13
|
global:
|
@@ -22,8 +24,8 @@ before_script:
|
|
22
24
|
- ./cc-test-reporter before-build
|
23
25
|
|
24
26
|
script:
|
25
|
-
-
|
26
|
-
-
|
27
|
+
- bundle exec rubocop
|
28
|
+
- bundle exec rspec --format doc
|
27
29
|
- bin/check-version
|
28
30
|
|
29
31
|
after_script:
|
data/CHANGELOG.md
CHANGED
@@ -1,6 +1,42 @@
|
|
1
|
+
## Release v0.3.0
|
2
|
+
|
3
|
+
* Add tools for backend fault-tolerance #10
|
4
|
+
* CircuitProxy for wrapping storage in an internal circuit
|
5
|
+
* FallbackChain storage backend for falling back to stable storage
|
6
|
+
* Timeout warnings for Redis backend
|
7
|
+
* AutoWire wrappers for automatically configuring storage and cache
|
8
|
+
* Better documentation for fault-tolerance
|
9
|
+
|
10
|
+
## Release v0.2.0
|
11
|
+
|
12
|
+
* Remove Scopes and replace them with Faulty instances #9
|
13
|
+
|
14
|
+
### Breaking Changes
|
15
|
+
|
16
|
+
* `Faulty::Scope` has been removed. Use `Faulty.new` instead.
|
17
|
+
* `Faulty` is now a class, not a module
|
18
|
+
|
19
|
+
## Release v0.1.5
|
20
|
+
|
21
|
+
* Fix redis storage to expire state key when using CAS #8
|
22
|
+
|
23
|
+
## Release v0.1.4
|
24
|
+
|
25
|
+
* Improve spec coverage for supporting classes #6
|
26
|
+
* Fix redis bug where concurrent CAS requests could crash #7
|
27
|
+
|
28
|
+
## Release v0.1.3
|
29
|
+
|
30
|
+
* Fix bug where memory storage would delete the newest entries #5
|
31
|
+
* Add HoneybadgerListener for error reporting #4
|
32
|
+
|
33
|
+
## Release v0.1.2
|
34
|
+
|
35
|
+
* Fix Storage::FaultTolerantProxy open and reopen methods #2
|
36
|
+
|
1
37
|
## Release v0.1.1
|
2
38
|
|
3
|
-
* Fix a crash when Storage::FaultTolerantProxy created a status stub
|
39
|
+
* Fix a crash when Storage::FaultTolerantProxy created a status stub #1
|
4
40
|
|
5
41
|
## Release v0.1.0
|
6
42
|
|
data/Gemfile
CHANGED
@@ -3,3 +3,20 @@
|
|
3
3
|
source 'https://rubygems.org'
|
4
4
|
|
5
5
|
gemspec
|
6
|
+
|
7
|
+
# We add non-essential gems like debugging tools and CI dependencies
|
8
|
+
# here. This also allows us to use conditional dependencies that depend on the
|
9
|
+
# platform
|
10
|
+
|
11
|
+
not_jruby = %i[ruby mingw x64_mingw].freeze
|
12
|
+
|
13
|
+
gem 'activesupport', '>= 4.2'
|
14
|
+
gem 'bundler', '>= 1.17', '< 3'
|
15
|
+
gem 'byebug', platforms: not_jruby
|
16
|
+
gem 'irb', '~> 1.0'
|
17
|
+
gem 'redcarpet', '~> 3.5', platforms: not_jruby
|
18
|
+
gem 'rspec_junit_formatter', '~> 0.4'
|
19
|
+
# For now, code climate doesn't support simplecov 0.18
|
20
|
+
# https://github.com/codeclimate/test-reporter/issues/413
|
21
|
+
gem 'simplecov', '>= 0.17.1', '< 0.18'
|
22
|
+
gem 'yard', '~> 0.9.25', platforms: not_jruby
|
data/README.md
CHANGED
@@ -67,6 +67,51 @@ end
|
|
67
67
|
For a full list of configuration options, see the
|
68
68
|
[Global Configuration](#global-configuration) section.
|
69
69
|
|
70
|
+
## What is this for?
|
71
|
+
|
72
|
+
Circuit breakers are a fault-tolerance tool for creating separation between your
|
73
|
+
application and external dependencies. For example, your application may call an
|
74
|
+
external API to send a text message:
|
75
|
+
|
76
|
+
```ruby
|
77
|
+
TextApi.send(message)
|
78
|
+
```
|
79
|
+
|
80
|
+
In normal operation, this API call is very fast. However what if the texting
|
81
|
+
service started hanging? Your application would quickly use up a lot of
|
82
|
+
resources waiting for requests to return from the service. You could consider
|
83
|
+
adding a timeout to your request:
|
84
|
+
|
85
|
+
```ruby
|
86
|
+
TextApi.send(message, timeout: 5)
|
87
|
+
```
|
88
|
+
|
89
|
+
Now your application will terminate requests after 5 seconds, but that could
|
90
|
+
still add up to a lot of resources if you call this thousands of times. Circuit
|
91
|
+
breakers solve this problem.
|
92
|
+
|
93
|
+
```ruby
|
94
|
+
Faulty.circuit(:text_api).run do
|
95
|
+
TextApi.send(message, timeout: 5)
|
96
|
+
end
|
97
|
+
```
|
98
|
+
|
99
|
+
Now, when the text API hangs, the first few will run and start timing out. This
|
100
|
+
will trip the circuit. After the circuit trips
|
101
|
+
(see [How it Works](#how-it-works)), calls to the text API will be paused for
|
102
|
+
the configured cool down period. Your application resources are not overwhelmed.
|
103
|
+
|
104
|
+
You are free to implement a fallback or error handling however you wish, for
|
105
|
+
example, in this case, you might add the text message to a failure queue:
|
106
|
+
|
107
|
+
```ruby
|
108
|
+
Faulty.circuit(:text_api).run do
|
109
|
+
TextApi.send(message, timeout: 5)
|
110
|
+
rescue Faulty::CircuitError => e
|
111
|
+
FailureQueue.enqueue(message)
|
112
|
+
end
|
113
|
+
```
|
114
|
+
|
70
115
|
## Basic Usage
|
71
116
|
|
72
117
|
To create a circuit, call `Faulty.circuit`. This can be done as you use the
|
@@ -127,7 +172,8 @@ end.or_default([])
|
|
127
172
|
|
128
173
|
## How it Works
|
129
174
|
|
130
|
-
Faulty implements a version of circuit breakers inspired by
|
175
|
+
Faulty implements a version of circuit breakers inspired by "Release It!: Design
|
176
|
+
and Deploy Production-Ready Software" by [Michael T. Nygard][michael nygard] and
|
131
177
|
[Martin Fowler's post][martin fowler] on the subject. A few notable features of
|
132
178
|
Faulty's implementation are:
|
133
179
|
|
@@ -135,11 +181,12 @@ Faulty's implementation are:
|
|
135
181
|
- Integrated caching inspired by Netflix's [Hystrix][hystrix] with automatic
|
136
182
|
cache jitter and error fallback.
|
137
183
|
- Event-based monitoring
|
184
|
+
- Flexible fault-tolerant storage with optional fallbacks
|
138
185
|
|
139
186
|
Following the principals of the circuit-breaker pattern, the block given to
|
140
|
-
`run` or `try_run` will always be executed as long as
|
141
|
-
|
142
|
-
|
187
|
+
`run` or `try_run` will always be executed as long as it never raises an error.
|
188
|
+
If the block _does_ raise an error, then the circuit keeps track of the number
|
189
|
+
of runs and the failure rate.
|
143
190
|
|
144
191
|
Once both thresholds are breached, the circuit is opened. Once open, the
|
145
192
|
circuit starts the cool-down period. Any executions within that cool-down are
|
@@ -158,22 +205,30 @@ In addition to the classic circuit breaker design, Faulty implements caching
|
|
158
205
|
that is integrated with the circuit state. See [Caching](#caching) for more
|
159
206
|
detail.
|
160
207
|
|
161
|
-
##
|
208
|
+
## Configuration
|
162
209
|
|
163
|
-
|
164
|
-
illustrates the default values.
|
165
|
-
|
210
|
+
Faulty can be configured with the following configuration options. This example
|
211
|
+
illustrates the default values. In the first example, we configure Faulty
|
212
|
+
globally. The second example shows the same configuration using an instance of
|
213
|
+
Faulty instead of global configuration.
|
166
214
|
|
167
215
|
```ruby
|
168
216
|
Faulty.init do |config|
|
169
217
|
# The cache backend to use. By default, Faulty looks for a Rails cache. If
|
170
218
|
# that's not available, it uses an ActiveSupport::Cache::Memory instance.
|
171
219
|
# Otherwise, it uses a Faulty::Cache::Null and caching is disabled.
|
220
|
+
# Whatever backend is given here is automatically wrapped in
|
221
|
+
# Faulty::Cache::AutoWire. This adds fault-tolerance features, see the
|
222
|
+
# AutoWire API docs for more details.
|
172
223
|
config.cache = Faulty::Cache::Default.new
|
173
224
|
|
174
225
|
# The storage backend. By default, Faulty uses an in-memory store. For most
|
175
226
|
# production applications, you'll want a more robust backend. Faulty also
|
176
227
|
# provides Faulty::Storage::Redis for this.
|
228
|
+
# Whatever backend is given here is automatically wrapped in
|
229
|
+
# Faulty::Storage::AutoWire. This adds fault-tolerance features, see the
|
230
|
+
# AutoWire APi docs for more details. If an array of storage backends is
|
231
|
+
# given, each one will be tried in order until one succeeds.
|
177
232
|
config.storage = Faulty::Storage::Memory.new
|
178
233
|
|
179
234
|
# An array of event listeners. Each object in the array should implement
|
@@ -188,6 +243,25 @@ Faulty.init do |config|
|
|
188
243
|
end
|
189
244
|
```
|
190
245
|
|
246
|
+
Here is the same configuration using an instance of `Faulty`. This is a more
|
247
|
+
object-oriented approach.
|
248
|
+
|
249
|
+
```ruby
|
250
|
+
faulty = Faulty.new do |config|
|
251
|
+
config.cache = Faulty::Cache::Default.new
|
252
|
+
config.storage = Faulty::Storage::Memory.new
|
253
|
+
config.listeners = [Faulty::Events::LogListener.new]
|
254
|
+
config.notifier = Faulty::Events::Notifier.new(config.listeners)
|
255
|
+
end
|
256
|
+
```
|
257
|
+
|
258
|
+
Most of the examples in this README use the global Faulty class methods, but
|
259
|
+
they work the same way when using an instance. Just substitute your instance
|
260
|
+
instead of `Faulty`. There is no preferred way to use Faulty. Choose whichever
|
261
|
+
configuration mechanism works best for your application. Also see
|
262
|
+
[Multiple Configurations](#multiple-configurations) if your application needs
|
263
|
+
to set different options in different scenarios.
|
264
|
+
|
191
265
|
For all Faulty APIs that have configuration, you can also pass in an options
|
192
266
|
hash. For example, `Faulty.init` could be called like this:
|
193
267
|
|
@@ -199,9 +273,9 @@ Faulty.init(cache: Faulty::Cache::Null.new)
|
|
199
273
|
|
200
274
|
A circuit can be created with the following configuration options. Those options
|
201
275
|
are only set once, synchronized across threads, and will persist in-memory until
|
202
|
-
the process exits. If you're using [
|
203
|
-
within the context of each
|
204
|
-
`Faulty.circuit` (or `
|
276
|
+
the process exits. If you're using [multiple configurations](#multiple-configurations),
|
277
|
+
the options are retained within the context of each instance. All options given
|
278
|
+
after the first call to `Faulty.circuit` (or `Faulty#circuit`) are ignored.
|
205
279
|
|
206
280
|
This is because the circuit objects themselves are internally memoized, and are
|
207
281
|
read-only once created.
|
@@ -262,8 +336,8 @@ Faulty.circuit(:api, cache_expires_in: 1800)
|
|
262
336
|
|
263
337
|
Faulty integrates caching into it's circuits in a way that is particularly
|
264
338
|
suited to fault-tolerance. To make use of caching, you must specify the `cache`
|
265
|
-
configuration option when initializing Faulty or creating a
|
266
|
-
using Rails, this is automatically set to the Rails cache.
|
339
|
+
configuration option when initializing Faulty or creating a new Faulty instance.
|
340
|
+
If you're using Rails, this is automatically set to the Rails cache.
|
267
341
|
|
268
342
|
Once your cache is configured, you can use the `cache` parameter when running
|
269
343
|
a circuit to specify a cache key:
|
@@ -315,7 +389,12 @@ Faulty backends are fault-tolerant by default. Any `StandardError`s raised by
|
|
315
389
|
the storage or cache backends are captured and suppressed. Failure events for
|
316
390
|
these errors are sent to the notifier.
|
317
391
|
|
318
|
-
|
392
|
+
In case of a flaky storage or cache backend, Faulty also uses independent
|
393
|
+
in-memory circuits to track failures so that we don't keep calling a backend
|
394
|
+
that is failing. See the API docs for `Cache::AutoWire`, and `Storage::AutoWire`
|
395
|
+
for more details.
|
396
|
+
|
397
|
+
If the storage backend fails, circuits will default to closed. If the cache
|
319
398
|
backend fails, all cache queries will miss.
|
320
399
|
|
321
400
|
## Event Handling
|
@@ -338,12 +417,17 @@ events. The full list of events is available from `Faulty::Events::EVENTS`.
|
|
338
417
|
closed. Payload: `circuit`
|
339
418
|
- `circuit_success` - A circuit execution was successful. Payload: `circuit`,
|
340
419
|
`status`
|
341
|
-
- `storage_failure` - A storage backend raised an error. Payload `circuit
|
342
|
-
`action`, `error`
|
420
|
+
- `storage_failure` - A storage backend raised an error. Payload `circuit` (can
|
421
|
+
be nil), `action`, `error`
|
343
422
|
|
344
423
|
By default events are logged using `Faulty::Events::LogListener`, but that can
|
345
424
|
be replaced, or additional listeners can be added.
|
346
425
|
|
426
|
+
### CallbackListener
|
427
|
+
|
428
|
+
The callback listener is useful for ad-hoc handling of events. You can specify
|
429
|
+
an event handler by calling a method on the callback handler by the same name.
|
430
|
+
|
347
431
|
```ruby
|
348
432
|
Faulty.init do |config|
|
349
433
|
# Replace the default listener with a custom callback listener
|
@@ -356,6 +440,17 @@ Faulty.init do |config|
|
|
356
440
|
end
|
357
441
|
```
|
358
442
|
|
443
|
+
### Other Built-in Listeners
|
444
|
+
|
445
|
+
In addition to the log and callback listeners, Faulty intends to implement
|
446
|
+
built-in service-specific handlers to make it easy to integrate with monitoring
|
447
|
+
and reporting software.
|
448
|
+
|
449
|
+
- `Faulty::Events::HoneybadgerListener`: Reports circuit and backend errors to
|
450
|
+
the Honeybadger error reporting service.
|
451
|
+
|
452
|
+
### Custom Listeners
|
453
|
+
|
359
454
|
You can implement your own listener by following the documentation in
|
360
455
|
`Faulty::Events::ListenerInterface`. For example:
|
361
456
|
|
@@ -375,10 +470,19 @@ end
|
|
375
470
|
|
376
471
|
## Configuring the Storage Backend
|
377
472
|
|
473
|
+
A storage backend is required to use Faulty. By default, it uses in-memory
|
474
|
+
storage, but Redis is also available, along with a number of wrappers used to
|
475
|
+
improve resiliency and fault-tolerance.
|
476
|
+
|
378
477
|
### Memory
|
379
478
|
|
380
|
-
The `Faulty::
|
381
|
-
|
479
|
+
The `Faulty::Storage::Memory` backend is the default storage backend. You may
|
480
|
+
prefer this implementation if you want to avoid the complexity and potential
|
481
|
+
failure-mode of cross-network circuit storage. The trade-off is that circuit
|
482
|
+
state is only contained within a single process and will not be saved across
|
483
|
+
application restarts. Locks will also be cleared on restart.
|
484
|
+
|
485
|
+
The default configuration:
|
382
486
|
|
383
487
|
```ruby
|
384
488
|
Faulty.init do |config|
|
@@ -391,36 +495,159 @@ end
|
|
391
495
|
|
392
496
|
### Redis
|
393
497
|
|
394
|
-
The `Faulty::
|
395
|
-
Redis.
|
498
|
+
The `Faulty::Storage::Redis` backend provides distributed circuit storage using
|
499
|
+
Redis. Although Faulty takes steps to reduce risk
|
500
|
+
(See [Fault Tolerance](#fault-tolerance)), using cross-network storage does
|
501
|
+
introduce some additional failure modes. To reduce this risk, be sure to set
|
502
|
+
conservative timeouts for your Redis connection. Setting high timeouts will
|
503
|
+
print warnings to stderr.
|
504
|
+
|
505
|
+
The default configuration:
|
396
506
|
|
397
507
|
```ruby
|
398
508
|
Faulty.init do |config|
|
399
509
|
config.storage = Faulty::Storage::Redis.new do |storage|
|
400
510
|
# The Redis client. Accepts either a Redis instance, or a ConnectionPool
|
401
|
-
# of Redis instances.
|
402
|
-
|
511
|
+
# of Redis instances. A low timeout is highly recommended to prevent
|
512
|
+
# cascading failures when evaluating circuits.
|
513
|
+
storage.client = ::Redis.new(timeout: 1)
|
403
514
|
|
404
515
|
# The prefix to prepend to all redis keys used by Faulty circuits
|
405
516
|
storage.key_prefix = 'faulty'
|
406
517
|
|
407
518
|
# A string to separate the parts of the redis key
|
408
|
-
storage.key_separator
|
519
|
+
storage.key_separator = ':'
|
409
520
|
|
410
521
|
# The maximum number of circuit runs that will be stored
|
411
522
|
storage.max_sample_size = 100
|
412
523
|
|
413
524
|
# The maximum number of seconds that a circuit run will be stored
|
414
525
|
storage.sample_ttl = 1800
|
526
|
+
|
527
|
+
# The maximum number of seconds to store a circuit. Does not apply to
|
528
|
+
# locks, which are indefinite.
|
529
|
+
storage.circuit_ttl = 604_800 # 1 Week
|
530
|
+
|
531
|
+
# The number of seconds between circuit expirations. Changing this setting
|
532
|
+
# is not recommended. See API docs for more implementation details.
|
533
|
+
storage.list_granularity = 3600
|
534
|
+
|
535
|
+
# If true, disables warnings about recommended client settings like timeouts
|
536
|
+
storage.disable_warnings = false
|
415
537
|
end
|
416
538
|
end
|
417
539
|
```
|
418
540
|
|
419
|
-
###
|
541
|
+
### FallbackChain
|
542
|
+
|
543
|
+
The `Faulty::Storage::FallbackChain` backend is a wrapper for multiple
|
544
|
+
prioritized storage backends. If the first backend in the chain fails,
|
545
|
+
consecutive backends are tried until one succeeds. The recommended use-case for
|
546
|
+
this is to fall back on reliable storage if a networked storage backend fails.
|
547
|
+
|
548
|
+
For example, you may configure Redis as your primary storage backend, with an
|
549
|
+
in-memory storage backend as a fallback:
|
550
|
+
|
551
|
+
```ruby
|
552
|
+
Faulty.init do |config|
|
553
|
+
config.storage = Faulty::Storage::FallbackChain.new([
|
554
|
+
Faulty::Storage::Redis.new,
|
555
|
+
Faulty::Storage::Memory.new
|
556
|
+
])
|
557
|
+
end
|
558
|
+
```
|
559
|
+
|
560
|
+
Faulty instances will automatically use a fallback chain if an array is given to
|
561
|
+
the `storage` option, so this example is equivalent to the above:
|
562
|
+
|
563
|
+
```ruby
|
564
|
+
Faulty.init do |config|
|
565
|
+
config.storage = [
|
566
|
+
Faulty::Storage::Redis.new,
|
567
|
+
Faulty::Storage::Memory.new
|
568
|
+
]
|
569
|
+
end
|
570
|
+
```
|
571
|
+
|
572
|
+
If the fallback chain fails-over to backup storage, circuit states will not
|
573
|
+
carry over, so failover could be temporarily disruptive to your application.
|
574
|
+
However, any calls to `#lock` or `#unlock` will always be persisted to all
|
575
|
+
backends so that locks are maintained during failover.
|
576
|
+
|
577
|
+
### Storage::FaultTolerantProxy
|
578
|
+
|
579
|
+
This wrapper is applied to all non-fault-tolerant storage backends by default
|
580
|
+
(see the [API docs for `Faulty::Storage::AutoWire`](https://www.rubydoc.info/gems/faulty/Faulty/Storage/AutoWire)).
|
581
|
+
|
582
|
+
`Faulty::Storage::FaultTolerantProxy` is a wrapper that suppresses storage
|
583
|
+
errors and returns sensible defaults during failures. If a storage backend is
|
584
|
+
failing, all circuits will be treated as closed regardless of locks or previous
|
585
|
+
history.
|
586
|
+
|
587
|
+
If you wish your application to use a secondary storage backend instead of
|
588
|
+
failing closed, use `FallbackChain`.
|
589
|
+
|
590
|
+
### Storage::CircuitProxy
|
591
|
+
|
592
|
+
This wrapper is applied to all non-fault-tolerant storage backends by default
|
593
|
+
(see the [API docs for `Faulty::Storage::AutoWire`](https://www.rubydoc.info/gems/faulty/Faulty/Cache/AutoWire)).
|
594
|
+
|
595
|
+
`Faulty::Storage::CircuitProxy` is a wrapper that uses an independent in-memory
|
596
|
+
circuit to track failures to storage backends. If a storage backend fails
|
597
|
+
continuously, it will be temporarily disabled and raise `Faulty::CircuitError`s.
|
598
|
+
|
599
|
+
Typically this is used inside a `FaultTolerantProxy` or `FallbackChain` so that
|
600
|
+
these storage failures are handled gracefully.
|
601
|
+
|
602
|
+
## Configuring the Cache Backend
|
603
|
+
|
604
|
+
### Null
|
605
|
+
|
606
|
+
The `Faulty::Cache::Null` cache disables caching. It is the default if Rails and
|
607
|
+
ActiveSupport are not present.
|
608
|
+
|
609
|
+
### Rails
|
610
|
+
|
611
|
+
`Faulty::Cache::Rails` is the default cache if Rails or ActiveSupport are
|
612
|
+
present. If Rails is present, it uses `Rails.cache` as the backend. If
|
613
|
+
ActiveSupport is present, but Rails is not, it creates a new
|
614
|
+
`ActiveSupport::Cache::MemoryStore` by default. This backend can be used with
|
615
|
+
any `ActiveSupport::Cache`.
|
616
|
+
|
617
|
+
```ruby
|
618
|
+
Faulty.init do |config|
|
619
|
+
config.cache = Faulty::Cache::Rails.new(
|
620
|
+
ActiveSupport::Cache::RedisCacheStore.new
|
621
|
+
)
|
622
|
+
end
|
623
|
+
```
|
624
|
+
|
625
|
+
### Cache::FaultTolerantProxy
|
626
|
+
|
627
|
+
This wrapper is applied to all non-fault-tolerant cache backends by default
|
628
|
+
(see the API docs for `Faulty::Cache::AutoWire`).
|
629
|
+
|
630
|
+
`Faulty::Cache::FaultTolerantProxy` is a wrapper that suppresses cache errors
|
631
|
+
and acts like a null cache during failures. Reads always return `nil`, and
|
632
|
+
writes are no-ops.
|
633
|
+
|
634
|
+
### Cache::CircuitProxy
|
635
|
+
|
636
|
+
This wrapper is applied to all non-fault-tolerant circuit backends by default
|
637
|
+
(see the API docs for `Faulty::Circuit::AutoWire`).
|
638
|
+
|
639
|
+
`Faulty::Circuit::CircuitProxy` is a wrapper that uses an independent in-memory
|
640
|
+
circuit to track failures to circuit backends. If a circuit backend fails
|
641
|
+
continuously, it will be temporarily disabled and raise `Faulty::CircuitError`s.
|
642
|
+
|
643
|
+
Typically this is used inside a `FaultTolerantProxy` so that these cache
|
644
|
+
failures are handled gracefully.
|
645
|
+
|
646
|
+
## Listing Circuits
|
420
647
|
|
421
648
|
For monitoring or debugging, you may need to retrieve a list of all circuit
|
422
|
-
names. This is possible with `Faulty.list_circuits` (or
|
423
|
-
|
649
|
+
names. This is possible with `Faulty.list_circuits` (or `Faulty#list_circuits`
|
650
|
+
if you're using an instance).
|
424
651
|
|
425
652
|
You can get a list of all circuit statuses by mapping those names to their
|
426
653
|
status objects. Be careful though, since this could cause performance issues for
|
@@ -432,73 +659,103 @@ statuses = Faulty.list_circuits.map do |name|
|
|
432
659
|
end
|
433
660
|
```
|
434
661
|
|
435
|
-
##
|
662
|
+
## Locking Circuits
|
663
|
+
|
664
|
+
It is possible to lock a circuit open or closed. A circuit that is locked open
|
665
|
+
will never execute its block, and always raise an `Faulty::OpenCircuitError`.
|
666
|
+
This is useful in cases where you need to manually disable a dependency
|
667
|
+
entirely. If a cached value is available, that will be returned from the circuit
|
668
|
+
until it expires, even outside its refresh period.
|
669
|
+
|
670
|
+
```ruby
|
671
|
+
Faulty.circuit(:broken_api).lock_open!
|
672
|
+
```
|
673
|
+
|
674
|
+
A circuit that is locked closed will never trip. This is useful in cases where a
|
675
|
+
circuit is continuously tripping incorrectly. If a cached value is available, it
|
676
|
+
will have the same behavior as an unlocked circuit.
|
677
|
+
|
678
|
+
```ruby
|
679
|
+
Faulty.circuit(:false_positive).lock_closed!
|
680
|
+
```
|
681
|
+
|
682
|
+
To remove a lock of either type:
|
683
|
+
|
684
|
+
```ruby
|
685
|
+
Faulty.circuit(:fixed).unlock!
|
686
|
+
```
|
687
|
+
|
688
|
+
Locking or unlocking a circuit has no concurrency guarantees, so it's not
|
689
|
+
recommended to lock or unlock circuits from production code. Instead, locks are
|
690
|
+
intended as an emergency tool for troubleshooting and debugging.
|
691
|
+
|
692
|
+
## Multiple Configurations
|
436
693
|
|
437
694
|
It is possible to have multiple configurations of Faulty running within the same
|
438
|
-
process. The most common
|
695
|
+
process. The most common setup is to simply use `Faulty.init` to
|
439
696
|
configure Faulty globally, however it is possible to have additional
|
440
|
-
configurations
|
697
|
+
configurations.
|
441
698
|
|
442
|
-
### The default
|
699
|
+
### The default instance
|
443
700
|
|
444
|
-
When you call `Faulty.init`, you are actually creating the default
|
445
|
-
can access this
|
701
|
+
When you call `Faulty.init`, you are actually creating the default instance of
|
702
|
+
`Faulty`. You can access this instance directly by calling `Faulty.default`.
|
446
703
|
|
447
704
|
```ruby
|
448
|
-
# We create the default
|
705
|
+
# We create the default instance
|
449
706
|
Faulty.init
|
450
707
|
|
451
|
-
# Access the default
|
452
|
-
|
708
|
+
# Access the default instance
|
709
|
+
faulty = Faulty.default
|
453
710
|
|
454
|
-
# Alternatively, access the
|
455
|
-
|
711
|
+
# Alternatively, access the instance by name
|
712
|
+
faulty = Faulty[:default]
|
456
713
|
```
|
457
714
|
|
458
|
-
You can rename the default
|
715
|
+
You can rename the default instance if desired:
|
459
716
|
|
460
717
|
```ruby
|
461
718
|
Faulty.init(:custom_default)
|
462
719
|
|
463
|
-
|
464
|
-
|
720
|
+
instance = Faulty.default
|
721
|
+
instance = Faulty[:custom_default]
|
465
722
|
```
|
466
723
|
|
467
|
-
### Multiple
|
724
|
+
### Multiple Instances
|
468
725
|
|
469
|
-
If you want multiple
|
726
|
+
If you want multiple instance, but want global, thread-safe access to
|
470
727
|
them, you can use `Faulty.register`:
|
471
728
|
|
472
729
|
```ruby
|
473
|
-
|
730
|
+
api_faulty = Faulty.new do |config|
|
474
731
|
# This accepts the same options as Faulty.init
|
475
732
|
end
|
476
733
|
|
477
|
-
Faulty.register(:api,
|
734
|
+
Faulty.register(:api, api_faulty)
|
478
735
|
|
479
|
-
# Now access the
|
736
|
+
# Now access the instance globally
|
480
737
|
Faulty[:api]
|
481
738
|
```
|
482
739
|
|
483
740
|
When you call `Faulty.circuit`, that's the same as calling
|
484
|
-
`Faulty.default.circuit`, so you can apply the same
|
485
|
-
|
741
|
+
`Faulty.default.circuit`, so you can apply the same principal to any other
|
742
|
+
registered Faulty instance:
|
486
743
|
|
487
744
|
```ruby
|
488
745
|
Faulty[:api].circuit(:api_circuit).run { 'ok' }
|
489
746
|
```
|
490
747
|
|
491
|
-
### Standalone
|
748
|
+
### Standalone Instances
|
492
749
|
|
493
|
-
If you choose, you can use Faulty
|
494
|
-
|
750
|
+
If you choose, you can use Faulty instances without registering them globally.
|
751
|
+
This is more object-oriented and is necessary if you use dependency injection.
|
495
752
|
|
496
753
|
```ruby
|
497
|
-
faulty = Faulty
|
754
|
+
faulty = Faulty.new
|
498
755
|
faulty.circuit(:standalone_circuit)
|
499
756
|
```
|
500
757
|
|
501
|
-
Calling
|
758
|
+
Calling `#circuit` on the instance still has the same memoization behavior that
|
502
759
|
`Faulty.circuit` has, so subsequent calls to the same circuit will return a
|
503
760
|
memoized circuit object.
|
504
761
|
|
@@ -545,15 +802,36 @@ would be useful for other users.
|
|
545
802
|
Faulty has its own opinions about how to implement a circuit breaker in Ruby,
|
546
803
|
but there are and have been many other options:
|
547
804
|
|
548
|
-
|
549
|
-
|
550
|
-
- [
|
805
|
+
### Currently Active
|
806
|
+
|
807
|
+
- [semian](https://github.com/Shopify/semian): A resiliency toolkit that
|
808
|
+
includes circuit breakers. It uses adapters to auto-wire circuits, and it has
|
809
|
+
only host-local storage by design.
|
810
|
+
- [circuitbox](https://github.com/yammer/circuitbox): Similar in design to
|
811
|
+
Faulty, but with a different API. It uses Moneta to abstract circuit storage
|
812
|
+
to allow any key-value store.
|
813
|
+
|
814
|
+
### Previous Work
|
815
|
+
|
816
|
+
- [circuit_breaker-ruby](https://github.com/scripbox/circuit_breaker-ruby) (no
|
817
|
+
recent activity)
|
818
|
+
- [stoplight](https://github.com/orgsync/stoplight) (unmaintained)
|
551
819
|
- [circuit_breaker](https://github.com/wooga/circuit_breaker) (archived)
|
552
820
|
- [simple_circuit_breaker](https://github.com/soundcloud/simple_circuit_breaker)
|
553
821
|
(unmaintained)
|
554
822
|
- [breaker](https://github.com/ahawkins/breaker) (unmaintained)
|
555
823
|
- [circuit_b](https://github.com/alg/circuit_b) (unmaintained)
|
556
824
|
|
825
|
+
### Faulty's Unique Features
|
826
|
+
|
827
|
+
- Simple API but configurable for advanced users
|
828
|
+
- Pluggable storage backends (circuitbox also has this)
|
829
|
+
- Protected storage access with fallback to safe storage
|
830
|
+
- Global, or object-oriented configuration with multiple instances
|
831
|
+
- Integrated caching support tailored for fault-tolerance
|
832
|
+
- Manually lock circuits open or closed
|
833
|
+
|
557
834
|
[api docs]: https://www.rubydoc.info/github/ParentSquare/faulty/master
|
835
|
+
[michael nygard]: https://www.michaelnygard.com/
|
558
836
|
[martin fowler]: https://www.martinfowler.com/bliki/CircuitBreaker.html
|
559
837
|
[hystrix]: https://github.com/Netflix/Hystrix/wiki/How-it-Works
|