faulty 0.7.1 → 0.8.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +35 -1
- data/README.md +27 -9
- data/bin/benchmark +30 -7
- data/faulty.gemspec +1 -0
- data/lib/faulty/cache/auto_wire.rb +0 -2
- data/lib/faulty/cache/circuit_proxy.rb +1 -3
- data/lib/faulty/cache/fault_tolerant_proxy.rb +0 -2
- data/lib/faulty/circuit.rb +104 -8
- data/lib/faulty/circuit_registry.rb +49 -0
- data/lib/faulty/error.rb +4 -3
- data/lib/faulty/events/filter_notifier.rb +31 -0
- data/lib/faulty/events.rb +1 -0
- data/lib/faulty/immutable_options.rb +21 -7
- data/lib/faulty/status.rb +0 -2
- data/lib/faulty/storage/auto_wire.rb +0 -2
- data/lib/faulty/storage/circuit_proxy.rb +15 -4
- data/lib/faulty/storage/fallback_chain.rb +18 -2
- data/lib/faulty/storage/fault_tolerant_proxy.rb +24 -2
- data/lib/faulty/storage/interface.rb +24 -1
- data/lib/faulty/storage/memory.rb +19 -3
- data/lib/faulty/storage/null.rb +94 -0
- data/lib/faulty/storage/redis.rb +35 -3
- data/lib/faulty/storage.rb +1 -0
- data/lib/faulty/version.rb +1 -1
- data/lib/faulty.rb +31 -6
- metadata +19 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: ff7d5c217cd03c4b1a752cef13a7547d0623f9810cb8c7602e91e8aa6a941358
|
4
|
+
data.tar.gz: c6df02b5ac50a0078fb43cc03ee7fad9275f5fe41c471a4a649eda05e6fa1782
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 0bbd345bbfe4d1acd1ba941ae73ba51fdf7a954fe1ca7a87e214dcbb78f8b44de643cb8cc05d7eb7dbd95de9b767039dbe24535717712dcb4c22d641333a9e33
|
7
|
+
data.tar.gz: afa9ccb28dec10f811c3076def8722323fb491b8e8e6ebf6f6917023a6c2a92eec515f0aaf86b74f9fd99f7947ce7e0e859358d6a16412675066a90e547d62ec
|
data/CHANGELOG.md
CHANGED
@@ -1,6 +1,40 @@
|
|
1
|
+
## Releae v0.8.2
|
2
|
+
|
3
|
+
* Fix crash for older versions of concurrent-ruby #42 justinhoward
|
4
|
+
|
5
|
+
## Releae v0.8.1
|
6
|
+
|
7
|
+
* Add cause message to CircuitTrippedError #40 justinhoward
|
8
|
+
* Record failures for cache hits #41 justinhoward
|
9
|
+
|
10
|
+
|
11
|
+
## Release v0.8.0
|
12
|
+
|
13
|
+
* Store circuit options in the backend when run #34 justinhoward
|
14
|
+
|
15
|
+
### Breaking Changes
|
16
|
+
|
17
|
+
* Added #get_options and #set_options to Faulty::Storage::Interface.
|
18
|
+
These will need to be added to any custom backends
|
19
|
+
* Faulty::Storage::Interface#reset now requires removing options in
|
20
|
+
addition to other stored values
|
21
|
+
* Circuit options will now be supplemented by stored options until they
|
22
|
+
are run. This is technically a breaking change in behavior, although
|
23
|
+
in most cases this should cause the expected result.
|
24
|
+
* Circuits are not memoized until they are run. Subsequent calls
|
25
|
+
to Faulty#circuit can return different instances if the circuit is
|
26
|
+
not run. However, once run, options are synchronized between
|
27
|
+
instances, so likely this will not be a breaking change for most
|
28
|
+
cases.
|
29
|
+
|
30
|
+
## Release v0.7.2
|
31
|
+
|
32
|
+
* Add Faulty.disable! for disabling globally #38 justinhoward
|
33
|
+
* Suppress circuit_success for proxy circuits #39 justinhoward
|
34
|
+
|
1
35
|
## Release v0.7.1
|
2
36
|
|
3
|
-
|
37
|
+
* Fix success event crash in log listener #37 justinhoward
|
4
38
|
|
5
39
|
## Release v0.7.0
|
6
40
|
|
data/README.md
CHANGED
@@ -88,6 +88,7 @@ Also see "Release It!: Design and Deploy Production-Ready Software" by
|
|
88
88
|
+ [CallbackListener](#callbacklistener)
|
89
89
|
+ [Other Built-in Listeners](#other-built-in-listeners)
|
90
90
|
+ [Custom Listeners](#custom-listeners)
|
91
|
+
* [Disabling Faulty Globally](#disabling-faulty-globally)
|
91
92
|
* [How it Works](#how-it-works)
|
92
93
|
+ [Caching](#caching)
|
93
94
|
+ [Fault Tolerance](#fault-tolerance)
|
@@ -607,7 +608,7 @@ faulty.circuit('standalone_circuit')
|
|
607
608
|
```
|
608
609
|
|
609
610
|
Calling `#circuit` on the instance still has the same memoization behavior that
|
610
|
-
`Faulty.circuit` has, so subsequent
|
611
|
+
`Faulty.circuit` has, so subsequent runs for the same circuit will use a
|
611
612
|
memoized circuit object.
|
612
613
|
|
613
614
|
|
@@ -732,7 +733,7 @@ Both options can even be specified together.
|
|
732
733
|
```ruby
|
733
734
|
Faulty.circuit(
|
734
735
|
'api',
|
735
|
-
errors: [ActiveRecord::ActiveRecordError]
|
736
|
+
errors: [ActiveRecord::ActiveRecordError],
|
736
737
|
exclude: [ActiveRecord::RecordNotFound, ActiveRecord::RecordNotUnique]
|
737
738
|
).run do
|
738
739
|
# This only captures ActiveRecord::ActiveRecordError errors, but not
|
@@ -812,7 +813,7 @@ the options are retained within the context of each instance. All options given
|
|
812
813
|
after the first call to `Faulty.circuit` (or `Faulty#circuit`) are ignored.
|
813
814
|
|
814
815
|
```ruby
|
815
|
-
Faulty.circuit('api', rate_threshold: 0.7)
|
816
|
+
Faulty.circuit('api', rate_threshold: 0.7).run { api.call }
|
816
817
|
|
817
818
|
# These options are ignored since with already initialized the circuit
|
818
819
|
circuit = Faulty.circuit('api', rate_threshold: 0.3)
|
@@ -820,7 +821,7 @@ circuit.options.rate_threshold # => 0.7
|
|
820
821
|
```
|
821
822
|
|
822
823
|
This is because the circuit objects themselves are internally memoized, and are
|
823
|
-
read-only once
|
824
|
+
read-only once they are run.
|
824
825
|
|
825
826
|
The following example represents the defaults for a new circuit:
|
826
827
|
|
@@ -1124,6 +1125,21 @@ Faulty.init do |config|
|
|
1124
1125
|
end
|
1125
1126
|
```
|
1126
1127
|
|
1128
|
+
## Disabling Faulty Globally
|
1129
|
+
|
1130
|
+
For testing or for some environments, you may wish to disable Faulty circuits
|
1131
|
+
at a global level.
|
1132
|
+
|
1133
|
+
```ruby
|
1134
|
+
Faulty.disable!
|
1135
|
+
```
|
1136
|
+
|
1137
|
+
This only affects the process where you run the `#disable!` method and it does
|
1138
|
+
not affect the stored state of circuits.
|
1139
|
+
|
1140
|
+
Faulty will **still use the cache** even when disabled. If you also want to
|
1141
|
+
disable the cache, configure Faulty to use a `Faulty::Cache::Null` cache.
|
1142
|
+
|
1127
1143
|
## How it Works
|
1128
1144
|
|
1129
1145
|
Faulty implements a version of circuit breakers inspired by "Release It!: Design
|
@@ -1272,11 +1288,12 @@ but there are and have been many other options:
|
|
1272
1288
|
### Currently Active
|
1273
1289
|
|
1274
1290
|
- [semian](https://github.com/Shopify/semian): A resiliency toolkit that
|
1275
|
-
includes circuit breakers. It
|
1276
|
-
only in-memory storage by design.
|
1277
|
-
|
1278
|
-
|
1279
|
-
to
|
1291
|
+
includes circuit breakers. It auto-wires circuits for MySQL, Net::HTTP, and
|
1292
|
+
Redis. It has only in-memory storage by design. Its core components are
|
1293
|
+
written in C, which allows it to be faster than pure ruby.
|
1294
|
+
- [circuitbox](https://github.com/yammer/circuitbox): Also uses a block syntax
|
1295
|
+
to manually define circuits. It uses Moneta to abstract circuit storage to
|
1296
|
+
allow any key-value store.
|
1280
1297
|
|
1281
1298
|
### Previous Work
|
1282
1299
|
|
@@ -1293,6 +1310,7 @@ but there are and have been many other options:
|
|
1293
1310
|
|
1294
1311
|
- Simple API but configurable for advanced users
|
1295
1312
|
- Pluggable storage backends (circuitbox also has this)
|
1313
|
+
- Patches for common core dependencies (semian also has this)
|
1296
1314
|
- Protected storage access with fallback to safe storage
|
1297
1315
|
- Global, or object-oriented configuration with multiple instances
|
1298
1316
|
- Integrated caching support tailored for fault-tolerance
|
data/bin/benchmark
CHANGED
@@ -4,12 +4,13 @@
|
|
4
4
|
require 'bundler/setup'
|
5
5
|
require 'benchmark'
|
6
6
|
require 'faulty'
|
7
|
+
require 'redis'
|
8
|
+
require 'json'
|
7
9
|
|
8
10
|
n = 100_000
|
9
|
-
|
10
|
-
puts "
|
11
|
-
|
12
|
-
Benchmark.bm(25) do |b|
|
11
|
+
width = 25
|
12
|
+
puts "In memory circuits x#{n}"
|
13
|
+
Benchmark.bm(width) do |b|
|
13
14
|
in_memory = Faulty.new(listeners: [])
|
14
15
|
b.report('memory storage') do
|
15
16
|
n.times { in_memory.circuit(:memory).run { true } }
|
@@ -31,11 +32,33 @@ Benchmark.bm(25) do |b|
|
|
31
32
|
end
|
32
33
|
end
|
33
34
|
|
34
|
-
n =
|
35
|
+
n = 1000
|
36
|
+
puts "\n\Redis circuits x#{n}"
|
37
|
+
Benchmark.bm(width) do |b|
|
38
|
+
redis = Faulty.new(listeners: [], storage: Faulty::Storage::Redis.new)
|
39
|
+
b.report('redis storage') do
|
40
|
+
n.times { redis.circuit(:memory).run { true } }
|
41
|
+
end
|
35
42
|
|
36
|
-
|
43
|
+
b.report('redis storage failures') do
|
44
|
+
n.times do
|
45
|
+
begin
|
46
|
+
redis.circuit(:memory_fail, sample_threshold: n + 1).run { raise 'fail' }
|
47
|
+
rescue StandardError
|
48
|
+
# Expected to raise here
|
49
|
+
end
|
50
|
+
end
|
51
|
+
end
|
37
52
|
|
38
|
-
|
53
|
+
redis_large = Faulty.new(listeners: [], storage: Faulty::Storage::Redis.new(max_sample_size: 1000))
|
54
|
+
b.report('large redis storage') do
|
55
|
+
n.times { redis_large.circuit(:memory).run { true } }
|
56
|
+
end
|
57
|
+
end
|
58
|
+
|
59
|
+
n = 1_000_000
|
60
|
+
puts "\n\nExtra x#{n}"
|
61
|
+
Benchmark.bm(width) do |b|
|
39
62
|
in_memory = Faulty.new(listeners: [])
|
40
63
|
|
41
64
|
log_listener = Faulty::Events::LogListener.new(Logger.new(File::NULL))
|
data/faulty.gemspec
CHANGED
@@ -26,6 +26,7 @@ Gem::Specification.new do |spec|
|
|
26
26
|
# Only essential development tools and dependencies go here.
|
27
27
|
# Other non-essential development dependencies go in the Gemfile.
|
28
28
|
spec.add_development_dependency 'connection_pool', '~> 2.0'
|
29
|
+
spec.add_development_dependency 'json'
|
29
30
|
spec.add_development_dependency 'redis', '>= 3.0'
|
30
31
|
spec.add_development_dependency 'rspec', '~> 3.8'
|
31
32
|
# 0.81 is the last rubocop version with Ruby 2.3 support
|
@@ -26,14 +26,12 @@ class Faulty
|
|
26
26
|
) do
|
27
27
|
include ImmutableOptions
|
28
28
|
|
29
|
-
private
|
30
|
-
|
31
29
|
def finalize
|
32
30
|
raise ArgumentError, 'The circuit or notifier option must be given' unless notifier || circuit
|
33
31
|
|
34
32
|
self.circuit ||= Circuit.new(
|
35
33
|
Faulty::Storage::CircuitProxy.name,
|
36
|
-
notifier: notifier,
|
34
|
+
notifier: Events::FilterNotifier.new(notifier, exclude: %i[circuit_success]),
|
37
35
|
cache: Cache::Null.new
|
38
36
|
)
|
39
37
|
end
|
data/lib/faulty/circuit.rb
CHANGED
@@ -27,7 +27,6 @@ class Faulty
|
|
27
27
|
CACHE_REFRESH_SUFFIX = '.faulty_refresh'
|
28
28
|
|
29
29
|
attr_reader :name
|
30
|
-
attr_reader :options
|
31
30
|
|
32
31
|
# Options for {Circuit}
|
33
32
|
#
|
@@ -82,6 +81,9 @@ class Faulty
|
|
82
81
|
# @return [Storage::Interface] The storage backend. Default
|
83
82
|
# `Storage::Memory.new`. Unlike {Faulty#initialize}, this is not wrapped
|
84
83
|
# in {Storage::AutoWire} by default.
|
84
|
+
# @!attribute [r] registry
|
85
|
+
# @return [CircuitRegistry] For use by {Faulty} instances to facilitate
|
86
|
+
# memoization of circuits.
|
85
87
|
Options = Struct.new(
|
86
88
|
:cache_expires_in,
|
87
89
|
:cache_refreshes_after,
|
@@ -95,11 +97,22 @@ class Faulty
|
|
95
97
|
:exclude,
|
96
98
|
:cache,
|
97
99
|
:notifier,
|
98
|
-
:storage
|
100
|
+
:storage,
|
101
|
+
:registry
|
99
102
|
) do
|
100
103
|
include ImmutableOptions
|
101
104
|
|
102
|
-
|
105
|
+
# Get the options stored in the storage backend
|
106
|
+
#
|
107
|
+
# @return [Hash] A hash of stored options
|
108
|
+
def for_storage
|
109
|
+
{
|
110
|
+
cool_down: cool_down,
|
111
|
+
evaluation_window: evaluation_window,
|
112
|
+
rate_threshold: rate_threshold,
|
113
|
+
sample_threshold: sample_threshold
|
114
|
+
}
|
115
|
+
end
|
103
116
|
|
104
117
|
def defaults
|
105
118
|
{
|
@@ -150,7 +163,59 @@ class Faulty
|
|
150
163
|
raise ArgumentError, 'name must be a String' unless name.is_a?(String)
|
151
164
|
|
152
165
|
@name = name
|
153
|
-
@
|
166
|
+
@given_options = Options.new(options, &block)
|
167
|
+
@pulled_options = nil
|
168
|
+
@options_pushed = false
|
169
|
+
end
|
170
|
+
|
171
|
+
# Get the options for this circuit
|
172
|
+
#
|
173
|
+
# If this circuit has been run, these will the options exactly as given
|
174
|
+
# to {.new}. However, if this circuit has not yet been run, these options
|
175
|
+
# will be supplemented by the last-known options from the circuit storage.
|
176
|
+
#
|
177
|
+
# Once a circuit is run, the given options are pushed to circuit storage to
|
178
|
+
# be persisted.
|
179
|
+
#
|
180
|
+
# This is to allow circuit objects to behave as expected in contexts where
|
181
|
+
# the exact options for a circuit are not known such as an admin dashboard
|
182
|
+
# or in a debug console.
|
183
|
+
#
|
184
|
+
# Note that this distinction isn't usually important unless using
|
185
|
+
# distributed circuit storage like the Redis storage backend.
|
186
|
+
#
|
187
|
+
# @example
|
188
|
+
# Faulty.circuit('api', cool_down: 5).run { api.users }
|
189
|
+
# # This status will be calculated using the cool_down of 5 because
|
190
|
+
# # the circuit was already run
|
191
|
+
# Faulty.circuit('api').status
|
192
|
+
#
|
193
|
+
# @example
|
194
|
+
# # This status will be calculated using the cool_down in circuit storage
|
195
|
+
# # if it is available instead of using the default value.
|
196
|
+
# Faulty.circuit('api').status
|
197
|
+
#
|
198
|
+
# @example
|
199
|
+
# # For typical usage, this behaves as expected, but note that it's
|
200
|
+
# # possible to run into some unexpected behavior when creating circuits
|
201
|
+
# # in unusual ways.
|
202
|
+
#
|
203
|
+
# # For example, this status will be calculated using the cool_down in
|
204
|
+
# # circuit storage if it is available despite the given value of 5.
|
205
|
+
# Faulty.circuit('api', cool_down: 5).status
|
206
|
+
# Faulty.circuit('api').run { api.users }
|
207
|
+
# # However now, after the circuit is run, status will be calculated
|
208
|
+
# # using the given cool_down of 5 and the value of 5 will be pushed
|
209
|
+
# # permanently to circuit storage
|
210
|
+
# Faulty.circuit('api').status
|
211
|
+
#
|
212
|
+
# @return [Options] The resolved options
|
213
|
+
def options
|
214
|
+
return @given_options if @options_pushed
|
215
|
+
return @pulled_options if @pulled_options
|
216
|
+
|
217
|
+
stored = @given_options.storage.get_options(self)
|
218
|
+
@pulled_options = stored ? @given_options.dup_with(stored) : @given_options
|
154
219
|
end
|
155
220
|
|
156
221
|
# Run the circuit as with {#run}, but return a {Result}
|
@@ -204,6 +269,8 @@ class Faulty
|
|
204
269
|
# a second cool down period. However, if the circuit completes successfully,
|
205
270
|
# the circuit will be closed and reset to its initial state.
|
206
271
|
#
|
272
|
+
# When this is run, the given options are persisted to the storage backend.
|
273
|
+
#
|
207
274
|
# @param cache [String, nil] A cache key, or nil if caching is not desired
|
208
275
|
# @yield The block to protect with this circuit
|
209
276
|
# @raise If the block raises an error not in the error list, or if the error
|
@@ -216,6 +283,7 @@ class Faulty
|
|
216
283
|
# circuit to trip
|
217
284
|
# @return The return value of the block
|
218
285
|
def run(cache: nil, &block)
|
286
|
+
push_options
|
219
287
|
cached_value = cache_read(cache)
|
220
288
|
# return cached unless cached.nil?
|
221
289
|
return cached_value if !cached_value.nil? && !cache_should_refresh?(cache)
|
@@ -256,6 +324,8 @@ class Faulty
|
|
256
324
|
#
|
257
325
|
# @return [self]
|
258
326
|
def reset!
|
327
|
+
@options_pushed = false
|
328
|
+
@pulled_options = nil
|
259
329
|
storage.reset(self)
|
260
330
|
self
|
261
331
|
end
|
@@ -284,6 +354,25 @@ class Faulty
|
|
284
354
|
|
285
355
|
private
|
286
356
|
|
357
|
+
# Push the given options to circuit storage and set those as the current
|
358
|
+
# options
|
359
|
+
#
|
360
|
+
# @return [void]
|
361
|
+
def push_options
|
362
|
+
return if @options_pushed
|
363
|
+
|
364
|
+
@pulled_options = nil
|
365
|
+
@options_pushed = true
|
366
|
+
resolved = options.registry&.resolve(self)
|
367
|
+
if resolved
|
368
|
+
# If another circuit instance was resolved, don't store these options
|
369
|
+
# Instead, copy the options from that circuit as if we were given those
|
370
|
+
@given_options = resolved.options
|
371
|
+
else
|
372
|
+
storage.set_options(self, @given_options.for_storage)
|
373
|
+
end
|
374
|
+
end
|
375
|
+
|
287
376
|
# Process a skipped run
|
288
377
|
#
|
289
378
|
# @param cached_value The cached value if one is available
|
@@ -308,10 +397,13 @@ class Faulty
|
|
308
397
|
rescue *options.errors => e
|
309
398
|
raise if options.exclude.any? { |ex| e.is_a?(ex) }
|
310
399
|
|
400
|
+
opened = failure!(status, e)
|
311
401
|
if cached_value.nil?
|
312
|
-
|
313
|
-
|
314
|
-
|
402
|
+
if opened
|
403
|
+
raise options.error_module::CircuitTrippedError.new(e.message, self)
|
404
|
+
else
|
405
|
+
raise options.error_module::CircuitFailureError.new(e.message, self)
|
406
|
+
end
|
315
407
|
else
|
316
408
|
cached_value
|
317
409
|
end
|
@@ -431,9 +523,13 @@ class Faulty
|
|
431
523
|
|
432
524
|
# Alias to the storage engine from options
|
433
525
|
#
|
526
|
+
# Always returns the value from the given options
|
527
|
+
#
|
434
528
|
# @return [Storage::Interface]
|
435
529
|
def storage
|
436
|
-
|
530
|
+
return Faulty::Storage::Null.new if Faulty.disabled?
|
531
|
+
|
532
|
+
@given_options.storage
|
437
533
|
end
|
438
534
|
end
|
439
535
|
end
|
@@ -0,0 +1,49 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
class Faulty
|
4
|
+
# Used by Faulty instances to track and memoize Circuits
|
5
|
+
#
|
6
|
+
# Whenever a circuit is requested by `Faulty#circuit`, it calls
|
7
|
+
# `#retrieve`. That will return a resolved circuit if there is one, or
|
8
|
+
# otherwise, it will create a new circuit instance.
|
9
|
+
#
|
10
|
+
# Once any circuit is run, the circuit calls `#resolve`. That saves
|
11
|
+
# the instance into the registry. Any calls to `#retrieve` after
|
12
|
+
# the circuit is resolved will result in the same instance being returned.
|
13
|
+
#
|
14
|
+
# However, before a circuit is resolved, calling `Faulty#circuit` will result
|
15
|
+
# in a new Circuit instance being created for every call. If multiples of
|
16
|
+
# these call `resolve`, only the first one will "win" and be memoized.
|
17
|
+
class CircuitRegistry
|
18
|
+
def initialize(circuit_options)
|
19
|
+
@circuit_options = circuit_options
|
20
|
+
@circuit_options[:registry] = self
|
21
|
+
@circuits = Concurrent::Map.new
|
22
|
+
end
|
23
|
+
|
24
|
+
# Retrieve a memoized circuit with the same name, or if none is yet
|
25
|
+
# resolved, create a new one.
|
26
|
+
#
|
27
|
+
# @param name [String] The name of the circuit
|
28
|
+
# @param options [Hash] Options for {Circuit::Options}
|
29
|
+
# @yield [Circuit::Options] For setting options in a block
|
30
|
+
# @return [Circuit] The new or memoized circuit
|
31
|
+
def retrieve(name, options, &block)
|
32
|
+
@circuits.fetch(name) do
|
33
|
+
options = @circuit_options.merge(options)
|
34
|
+
Circuit.new(name, **options, &block)
|
35
|
+
end
|
36
|
+
end
|
37
|
+
|
38
|
+
# Save and memoize the given circuit as the "canonical" instance for
|
39
|
+
# the circuit name
|
40
|
+
#
|
41
|
+
# If the name is already resolved, this will be ignored
|
42
|
+
#
|
43
|
+
# @return [Circuit, nil] If this circuit name is already resolved, the
|
44
|
+
# already-resolved circuit
|
45
|
+
def resolve(circuit)
|
46
|
+
@circuits.put_if_absent(circuit.name, circuit)
|
47
|
+
end
|
48
|
+
end
|
49
|
+
end
|
data/lib/faulty/error.rb
CHANGED
@@ -36,10 +36,11 @@ class Faulty
|
|
36
36
|
# @param message [String]
|
37
37
|
# @param circuit [Circuit] The circuit that raised the error
|
38
38
|
def initialize(message, circuit)
|
39
|
-
|
40
|
-
|
39
|
+
full_message = %(circuit error for "#{circuit.name}")
|
40
|
+
full_message = %(#{full_message}: #{message}) if message
|
41
41
|
|
42
|
-
|
42
|
+
@circuit = circuit
|
43
|
+
super(full_message)
|
43
44
|
end
|
44
45
|
end
|
45
46
|
|
@@ -0,0 +1,31 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
class Faulty
|
4
|
+
module Events
|
5
|
+
# Wraps a Notifier and filters events by name
|
6
|
+
class FilterNotifier
|
7
|
+
# @param notifier [Notifier] The internal notifier to filter events for
|
8
|
+
# @param events [Array, nil] An array of events to allow. If nil, all
|
9
|
+
# {EVENTS} will be used
|
10
|
+
# @param exclude [Array, nil] An array of events to disallow. If nil,
|
11
|
+
# no events will be disallowed. Takes priority over `events`.
|
12
|
+
def initialize(notifier, events: nil, exclude: nil)
|
13
|
+
@notifier = notifier
|
14
|
+
@events = Set.new(events || EVENTS)
|
15
|
+
exclude&.each { |e| @events.delete(e) }
|
16
|
+
end
|
17
|
+
|
18
|
+
# Notify all listeners of an event
|
19
|
+
#
|
20
|
+
# If a listener raises an error while handling an event, that error will
|
21
|
+
# be captured and written to STDERR.
|
22
|
+
#
|
23
|
+
# @param (see Notifier)
|
24
|
+
def notify(event, payload)
|
25
|
+
return unless @events.include?(event)
|
26
|
+
|
27
|
+
@notifier.notify(event, payload)
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
31
|
+
end
|
data/lib/faulty/events.rb
CHANGED
@@ -5,18 +5,23 @@ class Faulty
|
|
5
5
|
module ImmutableOptions
|
6
6
|
# @param hash [Hash] A hash of attributes to initialize with
|
7
7
|
# @yield [self] Yields itself to the block to set options before freezing
|
8
|
-
def initialize(hash)
|
9
|
-
defaults.merge(hash)
|
8
|
+
def initialize(hash, &block)
|
9
|
+
setup(defaults.merge(hash), &block)
|
10
|
+
end
|
11
|
+
|
12
|
+
def dup_with(hash, &block)
|
13
|
+
dup.setup(hash, &block)
|
14
|
+
end
|
15
|
+
|
16
|
+
def setup(hash)
|
17
|
+
hash&.each { |key, value| self[key] = value }
|
10
18
|
yield self if block_given?
|
11
19
|
finalize
|
12
|
-
|
13
|
-
raise ArgumentError, "Missing required attribute #{key}" if self[key].nil?
|
14
|
-
end
|
20
|
+
guard_required!
|
15
21
|
freeze
|
22
|
+
self
|
16
23
|
end
|
17
24
|
|
18
|
-
private
|
19
|
-
|
20
25
|
# A hash of default values to set before yielding to the block
|
21
26
|
#
|
22
27
|
# @return [Hash<Symbol, Object>]
|
@@ -36,5 +41,14 @@ class Faulty
|
|
36
41
|
# @return [void]
|
37
42
|
def finalize
|
38
43
|
end
|
44
|
+
|
45
|
+
private
|
46
|
+
|
47
|
+
# Raise an error if required options are missing
|
48
|
+
def guard_required!
|
49
|
+
required.each do |key|
|
50
|
+
raise ArgumentError, "Missing required attribute #{key}" if self[key].nil?
|
51
|
+
end
|
52
|
+
end
|
39
53
|
end
|
40
54
|
end
|
data/lib/faulty/status.rb
CHANGED
@@ -26,14 +26,12 @@ class Faulty
|
|
26
26
|
) do
|
27
27
|
include ImmutableOptions
|
28
28
|
|
29
|
-
private
|
30
|
-
|
31
29
|
def finalize
|
32
30
|
raise ArgumentError, 'The circuit or notifier option must be given' unless notifier || circuit
|
33
31
|
|
34
32
|
self.circuit ||= Circuit.new(
|
35
33
|
Faulty::Storage::CircuitProxy.name,
|
36
|
-
notifier: notifier,
|
34
|
+
notifier: Events::FilterNotifier.new(notifier, exclude: %i[circuit_success]),
|
37
35
|
cache: Cache::Null.new
|
38
36
|
)
|
39
37
|
end
|
@@ -47,7 +45,20 @@ class Faulty
|
|
47
45
|
@options = Options.new(options, &block)
|
48
46
|
end
|
49
47
|
|
50
|
-
%i[
|
48
|
+
%i[
|
49
|
+
get_options
|
50
|
+
set_options
|
51
|
+
entry
|
52
|
+
open
|
53
|
+
reopen
|
54
|
+
close
|
55
|
+
lock
|
56
|
+
unlock
|
57
|
+
reset
|
58
|
+
status
|
59
|
+
history
|
60
|
+
list
|
61
|
+
].each do |method|
|
51
62
|
define_method(method) do |*args|
|
52
63
|
options.circuit.run { @storage.public_send(method, *args) }
|
53
64
|
end
|
@@ -30,8 +30,6 @@ class Faulty
|
|
30
30
|
) do
|
31
31
|
include ImmutableOptions
|
32
32
|
|
33
|
-
private
|
34
|
-
|
35
33
|
def required
|
36
34
|
%i[notifier]
|
37
35
|
end
|
@@ -49,6 +47,24 @@ class Faulty
|
|
49
47
|
@options = Options.new(options, &block)
|
50
48
|
end
|
51
49
|
|
50
|
+
# Get options from the first available storage backend
|
51
|
+
#
|
52
|
+
# @param (see Interface#get_options)
|
53
|
+
# @return (see Interface#get_options)
|
54
|
+
def get_options(circuit)
|
55
|
+
send_chain(:get_options, circuit) do |e|
|
56
|
+
options.notifier.notify(:storage_failure, circuit: circuit, action: :get_options, error: e)
|
57
|
+
end
|
58
|
+
end
|
59
|
+
|
60
|
+
# Try to set circuit options on all backends
|
61
|
+
#
|
62
|
+
# @param (see Interface#set_options)
|
63
|
+
# @return (see Interface#set_options)
|
64
|
+
def set_options(circuit, stored_options)
|
65
|
+
send_all(:set_options, circuit, stored_options)
|
66
|
+
end
|
67
|
+
|
52
68
|
# Create a circuit entry in the first available storage backend
|
53
69
|
#
|
54
70
|
# @param (see Interface#entry)
|
@@ -23,8 +23,6 @@ class Faulty
|
|
23
23
|
) do
|
24
24
|
include ImmutableOptions
|
25
25
|
|
26
|
-
private
|
27
|
-
|
28
26
|
def required
|
29
27
|
%i[notifier]
|
30
28
|
end
|
@@ -85,6 +83,30 @@ class Faulty
|
|
85
83
|
# @return (see Interface#list)
|
86
84
|
def_delegators :@storage, :lock, :unlock, :reset, :history, :list
|
87
85
|
|
86
|
+
# Get circuit options safely
|
87
|
+
#
|
88
|
+
# @see Interface#get_options
|
89
|
+
# @param (see Interface#get_options)
|
90
|
+
# @return (see Interface#get_options)
|
91
|
+
def get_options(circuit)
|
92
|
+
@storage.get_options(circuit)
|
93
|
+
rescue StandardError => e
|
94
|
+
options.notifier.notify(:storage_failure, circuit: circuit, action: :get_options, error: e)
|
95
|
+
nil
|
96
|
+
end
|
97
|
+
|
98
|
+
# Set circuit options safely
|
99
|
+
#
|
100
|
+
# @see Interface#get_options
|
101
|
+
# @param (see Interface#set_options)
|
102
|
+
# @return (see Interface#set_options)
|
103
|
+
def set_options(circuit, stored_options)
|
104
|
+
@storage.set_options(circuit, stored_options)
|
105
|
+
rescue StandardError => e
|
106
|
+
options.notifier.notify(:storage_failure, circuit: circuit, action: :set_options, error: e)
|
107
|
+
nil
|
108
|
+
end
|
109
|
+
|
88
110
|
# Add a history entry safely
|
89
111
|
#
|
90
112
|
# @see Interface#entry
|
@@ -6,6 +6,29 @@ class Faulty
|
|
6
6
|
#
|
7
7
|
# This is for documentation only and is not loaded
|
8
8
|
class Interface
|
9
|
+
# Get the options stored for circuit
|
10
|
+
#
|
11
|
+
# They should be returned exactly as given by {#set_options}
|
12
|
+
#
|
13
|
+
# @return [Hash] A hash of the options stored by {#set_options}. The keys
|
14
|
+
# must be symbols.
|
15
|
+
def get_options(circuit)
|
16
|
+
raise NotImplementedError
|
17
|
+
end
|
18
|
+
|
19
|
+
# Store the options for a circuit
|
20
|
+
#
|
21
|
+
# They should be returned exactly as given by {#set_options}
|
22
|
+
#
|
23
|
+
# @param circuit [Circuit] The circuit to set options for
|
24
|
+
# @param options [Hash<Symbol, Object>] A hash of symbol option names to
|
25
|
+
# circuit options. These option values are guranteed to be primive
|
26
|
+
# values.
|
27
|
+
# @return [void]
|
28
|
+
def set_options(circuit, stored_options)
|
29
|
+
raise NotImplementedError
|
30
|
+
end
|
31
|
+
|
9
32
|
# Add a circuit run entry to storage
|
10
33
|
#
|
11
34
|
# The backend may choose to store this in whatever manner it chooses as
|
@@ -99,7 +122,7 @@ class Faulty
|
|
99
122
|
# Reset the circuit to a fresh state
|
100
123
|
#
|
101
124
|
# Clears all circuit status including entries, state, locks,
|
102
|
-
# opened_at, and any other values that would affect Status.
|
125
|
+
# opened_at, options, and any other values that would affect Status.
|
103
126
|
#
|
104
127
|
# No concurrency gurantees are provided for resetting
|
105
128
|
#
|
@@ -33,8 +33,6 @@ class Faulty
|
|
33
33
|
Options = Struct.new(:max_sample_size) do
|
34
34
|
include ImmutableOptions
|
35
35
|
|
36
|
-
private
|
37
|
-
|
38
36
|
def defaults
|
39
37
|
{ max_sample_size: 100 }
|
40
38
|
end
|
@@ -43,7 +41,7 @@ class Faulty
|
|
43
41
|
# The internal object for storing a circuit
|
44
42
|
#
|
45
43
|
# @private
|
46
|
-
MemoryCircuit = Struct.new(:state, :runs, :opened_at, :lock) do
|
44
|
+
MemoryCircuit = Struct.new(:state, :runs, :opened_at, :lock, :options) do
|
47
45
|
def initialize
|
48
46
|
self.state = Concurrent::Atom.new(:closed)
|
49
47
|
self.runs = Concurrent::MVar.new([], dup_on_deref: true)
|
@@ -78,6 +76,24 @@ class Faulty
|
|
78
76
|
@options = Options.new(options, &block)
|
79
77
|
end
|
80
78
|
|
79
|
+
# Get the options stored for circuit
|
80
|
+
#
|
81
|
+
# @see Interface#get_options
|
82
|
+
# @param (see Interface#get_options)
|
83
|
+
# @return (see Interface#get_options)
|
84
|
+
def get_options(circuit)
|
85
|
+
fetch(circuit).options
|
86
|
+
end
|
87
|
+
|
88
|
+
# Store the options for a circuit
|
89
|
+
#
|
90
|
+
# @see Interface#set_options
|
91
|
+
# @param (see Interface#set_options)
|
92
|
+
# @return (see Interface#set_options)
|
93
|
+
def set_options(circuit, stored_options)
|
94
|
+
fetch(circuit).options = stored_options
|
95
|
+
end
|
96
|
+
|
81
97
|
# Add an entry to storage
|
82
98
|
#
|
83
99
|
# @see Interface#entry
|
@@ -0,0 +1,94 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
class Faulty
|
4
|
+
module Storage
|
5
|
+
# A no-op backend for disabling circuits
|
6
|
+
class Null
|
7
|
+
# Define a single global instance
|
8
|
+
@instance = new
|
9
|
+
|
10
|
+
def self.new
|
11
|
+
@instance
|
12
|
+
end
|
13
|
+
|
14
|
+
# @param (see Interface#get_options)
|
15
|
+
# @return (see Interface#get_options)
|
16
|
+
def get_options(_circuit)
|
17
|
+
{}
|
18
|
+
end
|
19
|
+
|
20
|
+
# @param (see Interface#set_options)
|
21
|
+
# @return (see Interface#set_options)
|
22
|
+
def set_options(_circuit, _stored_options)
|
23
|
+
end
|
24
|
+
|
25
|
+
# @param (see Interface#entry)
|
26
|
+
# @return (see Interface#entry)
|
27
|
+
def entry(_circuit, _time, _success)
|
28
|
+
[]
|
29
|
+
end
|
30
|
+
|
31
|
+
# @param (see Interface#open)
|
32
|
+
# @return (see Interface#open)
|
33
|
+
def open(_circuit, _opened_at)
|
34
|
+
true
|
35
|
+
end
|
36
|
+
|
37
|
+
# @param (see Interface#reopen)
|
38
|
+
# @return (see Interface#reopen)
|
39
|
+
def reopen(_circuit, _opened_at, _previous_opened_at)
|
40
|
+
true
|
41
|
+
end
|
42
|
+
|
43
|
+
# @param (see Interface#close)
|
44
|
+
# @return (see Interface#close)
|
45
|
+
def close(_circuit)
|
46
|
+
true
|
47
|
+
end
|
48
|
+
|
49
|
+
# @param (see Interface#lock)
|
50
|
+
# @return (see Interface#lock)
|
51
|
+
def lock(_circuit, _state)
|
52
|
+
end
|
53
|
+
|
54
|
+
# @param (see Interface#unlock)
|
55
|
+
# @return (see Interface#unlock)
|
56
|
+
def unlock(_circuit)
|
57
|
+
end
|
58
|
+
|
59
|
+
# @param (see Interface#reset)
|
60
|
+
# @return (see Interface#reset)
|
61
|
+
def reset(_circuit)
|
62
|
+
end
|
63
|
+
|
64
|
+
# @param (see Interface#status)
|
65
|
+
# @return (see Interface#status)
|
66
|
+
def status(circuit)
|
67
|
+
Faulty::Status.new(
|
68
|
+
options: circuit.options,
|
69
|
+
stub: true
|
70
|
+
)
|
71
|
+
end
|
72
|
+
|
73
|
+
# @param (see Interface#history)
|
74
|
+
# @return (see Interface#history)
|
75
|
+
def history(_circuit)
|
76
|
+
[]
|
77
|
+
end
|
78
|
+
|
79
|
+
# @param (see Interface#list)
|
80
|
+
# @return (see Interface#list)
|
81
|
+
def list
|
82
|
+
[]
|
83
|
+
end
|
84
|
+
|
85
|
+
# This backend is fault tolerant
|
86
|
+
#
|
87
|
+
# @param (see Interface#fault_tolerant?)
|
88
|
+
# @return (see Interface#fault_tolerant?)
|
89
|
+
def fault_tolerant?
|
90
|
+
true
|
91
|
+
end
|
92
|
+
end
|
93
|
+
end
|
94
|
+
end
|
data/lib/faulty/storage/redis.rb
CHANGED
@@ -57,8 +57,6 @@ class Faulty
|
|
57
57
|
) do
|
58
58
|
include ImmutableOptions
|
59
59
|
|
60
|
-
private
|
61
|
-
|
62
60
|
def defaults
|
63
61
|
{
|
64
62
|
key_prefix: 'faulty',
|
@@ -85,9 +83,37 @@ class Faulty
|
|
85
83
|
def initialize(**options, &block)
|
86
84
|
@options = Options.new(options, &block)
|
87
85
|
|
86
|
+
# Ensure JSON is available since we don't explicitly require it
|
87
|
+
JSON # rubocop:disable Lint/Void
|
88
|
+
|
88
89
|
check_client_options!
|
89
90
|
end
|
90
91
|
|
92
|
+
# Get the options stored for circuit
|
93
|
+
#
|
94
|
+
# @see Interface#get_options
|
95
|
+
# @param (see Interface#get_options)
|
96
|
+
# @return (see Interface#get_options)
|
97
|
+
def get_options(circuit)
|
98
|
+
json = redis { |r| r.get(options_key(circuit)) }
|
99
|
+
return if json.nil?
|
100
|
+
|
101
|
+
JSON.parse(json, symbolize_names: true)
|
102
|
+
end
|
103
|
+
|
104
|
+
# Store the options for a circuit
|
105
|
+
#
|
106
|
+
# These will be serialized as JSON
|
107
|
+
#
|
108
|
+
# @see Interface#set_options
|
109
|
+
# @param (see Interface#set_options)
|
110
|
+
# @return (see Interface#set_options)
|
111
|
+
def set_options(circuit, stored_options)
|
112
|
+
redis do |r|
|
113
|
+
r.set(options_key(circuit), JSON.dump(stored_options), ex: options.circuit_ttl)
|
114
|
+
end
|
115
|
+
end
|
116
|
+
|
91
117
|
# Add an entry to storage
|
92
118
|
#
|
93
119
|
# @see Interface#entry
|
@@ -173,7 +199,8 @@ class Faulty
|
|
173
199
|
r.del(
|
174
200
|
entries_key(circuit),
|
175
201
|
opened_at_key(circuit),
|
176
|
-
lock_key(circuit)
|
202
|
+
lock_key(circuit),
|
203
|
+
options_key(circuit)
|
177
204
|
)
|
178
205
|
r.set(state_key(circuit), 'closed', ex: options.circuit_ttl)
|
179
206
|
end
|
@@ -239,6 +266,11 @@ class Faulty
|
|
239
266
|
key('circuit', circuit.name, *parts)
|
240
267
|
end
|
241
268
|
|
269
|
+
# @return [String] The key for circuit options
|
270
|
+
def options_key(circuit)
|
271
|
+
ckey(circuit, 'options')
|
272
|
+
end
|
273
|
+
|
242
274
|
# @return [String] The key for circuit state
|
243
275
|
def state_key(circuit)
|
244
276
|
ckey(circuit, 'state')
|
data/lib/faulty/storage.rb
CHANGED
@@ -10,5 +10,6 @@ require 'faulty/storage/auto_wire'
|
|
10
10
|
require 'faulty/storage/circuit_proxy'
|
11
11
|
require 'faulty/storage/fallback_chain'
|
12
12
|
require 'faulty/storage/fault_tolerant_proxy'
|
13
|
+
require 'faulty/storage/null'
|
13
14
|
require 'faulty/storage/memory'
|
14
15
|
require 'faulty/storage/redis'
|
data/lib/faulty/version.rb
CHANGED
data/lib/faulty.rb
CHANGED
@@ -2,7 +2,7 @@
|
|
2
2
|
|
3
3
|
require 'securerandom'
|
4
4
|
require 'forwardable'
|
5
|
-
require 'concurrent
|
5
|
+
require 'concurrent'
|
6
6
|
|
7
7
|
require 'faulty/immutable_options'
|
8
8
|
require 'faulty/cache'
|
@@ -10,6 +10,7 @@ require 'faulty/circuit'
|
|
10
10
|
require 'faulty/error'
|
11
11
|
require 'faulty/events'
|
12
12
|
require 'faulty/patch'
|
13
|
+
require 'faulty/circuit_registry'
|
13
14
|
require 'faulty/result'
|
14
15
|
require 'faulty/status'
|
15
16
|
require 'faulty/storage'
|
@@ -128,6 +129,33 @@ class Faulty
|
|
128
129
|
def current_time
|
129
130
|
Time.now.to_i
|
130
131
|
end
|
132
|
+
|
133
|
+
# Disable Faulty circuits
|
134
|
+
#
|
135
|
+
# This allows circuits to run as if they were always closed. Does
|
136
|
+
# not disable caching.
|
137
|
+
#
|
138
|
+
# Intended for use in tests, or to disable Faulty entirely for an
|
139
|
+
# environment.
|
140
|
+
#
|
141
|
+
# @return [void]
|
142
|
+
def disable!
|
143
|
+
@disabled = true
|
144
|
+
end
|
145
|
+
|
146
|
+
# Re-enable Faulty if disabled with {#disable!}
|
147
|
+
#
|
148
|
+
# @return [void]
|
149
|
+
def enable!
|
150
|
+
@disabled = false
|
151
|
+
end
|
152
|
+
|
153
|
+
# Check whether Faulty was disabled with {#disable!}
|
154
|
+
#
|
155
|
+
# @return [Boolean] True if disabled
|
156
|
+
def disabled?
|
157
|
+
@disabled == true
|
158
|
+
end
|
131
159
|
end
|
132
160
|
|
133
161
|
attr_reader :options
|
@@ -199,8 +227,8 @@ class Faulty
|
|
199
227
|
# @param options [Hash] Attributes for {Options}
|
200
228
|
# @yield [Options] For setting options in a block
|
201
229
|
def initialize(**options, &block)
|
202
|
-
@circuits = Concurrent::Map.new
|
203
230
|
@options = Options.new(options, &block)
|
231
|
+
@registry = CircuitRegistry.new(circuit_options)
|
204
232
|
end
|
205
233
|
|
206
234
|
# Create or retrieve a circuit
|
@@ -216,10 +244,7 @@ class Faulty
|
|
216
244
|
# @return [Circuit] The new circuit or the existing circuit if it already exists
|
217
245
|
def circuit(name, **options, &block)
|
218
246
|
name = name.to_s
|
219
|
-
@
|
220
|
-
options = circuit_options.merge(options)
|
221
|
-
Circuit.new(name, **options, &block)
|
222
|
-
end
|
247
|
+
@registry.retrieve(name, options, &block)
|
223
248
|
end
|
224
249
|
|
225
250
|
# Get a list of all circuit names
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: faulty
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.8.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Justin Howard
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-10-18 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: concurrent-ruby
|
@@ -38,6 +38,20 @@ dependencies:
|
|
38
38
|
- - "~>"
|
39
39
|
- !ruby/object:Gem::Version
|
40
40
|
version: '2.0'
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: json
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - ">="
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '0'
|
48
|
+
type: :development
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - ">="
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: '0'
|
41
55
|
- !ruby/object:Gem::Dependency
|
42
56
|
name: redis
|
43
57
|
requirement: !ruby/object:Gem::Requirement
|
@@ -144,9 +158,11 @@ files:
|
|
144
158
|
- lib/faulty/cache/null.rb
|
145
159
|
- lib/faulty/cache/rails.rb
|
146
160
|
- lib/faulty/circuit.rb
|
161
|
+
- lib/faulty/circuit_registry.rb
|
147
162
|
- lib/faulty/error.rb
|
148
163
|
- lib/faulty/events.rb
|
149
164
|
- lib/faulty/events/callback_listener.rb
|
165
|
+
- lib/faulty/events/filter_notifier.rb
|
150
166
|
- lib/faulty/events/honeybadger_listener.rb
|
151
167
|
- lib/faulty/events/listener_interface.rb
|
152
168
|
- lib/faulty/events/log_listener.rb
|
@@ -165,6 +181,7 @@ files:
|
|
165
181
|
- lib/faulty/storage/fault_tolerant_proxy.rb
|
166
182
|
- lib/faulty/storage/interface.rb
|
167
183
|
- lib/faulty/storage/memory.rb
|
184
|
+
- lib/faulty/storage/null.rb
|
168
185
|
- lib/faulty/storage/redis.rb
|
169
186
|
- lib/faulty/version.rb
|
170
187
|
homepage: https://github.com/ParentSquare/faulty
|