faulty 0.1.1 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a2d947943e6b9d5bf8481bc3b37f417c14f6ee37f7ccc359b37eb83c5d1da63f
-  data.tar.gz: 2ba752577fbf1a9e9ddf4b9754a1382d603435efb463158b55f5ac1eabe1b75e
+  metadata.gz: c0712dd797dfe2922e4e52a792f48c5ac72c4b9766d218dfa597e7ad630f9fd0
+  data.tar.gz: fb4a787507006131f34a301fc164466fd8befb72051550ca0a63fc2d69ec949c
 SHA512:
-  metadata.gz: 3d23f2e4b3aee70fceca72c2cc73f712a01ee6b86fdcc8d6fb68433573e224724d9f181300695bba7780265d1a294be1b991c4bff5ee24f6eb152d6650f834f9
-  data.tar.gz: 589ac68d52e00f82b7f7b562ae64f140bb7accc86b34247324e95b8e4c67ba9b79f40d71b6929398ab75738d9ea27c1a61a1162cde8003cc54dd0ade2171c79d
+  metadata.gz: 5381dd13f058045906330c67ff80aeefac55ae81077eb4552d5461d743344fa097e205b8ac3a39b385c52976e64315aec4e903c1a0c298a05452fab7788d1df2
+  data.tar.gz: af31cbf86cac54226189f8f63b1d8b0bcf82f9023085e98d07e4e9db65e402883c2219dbf255e21427c40b0c0a66ec71bd4dc9dfdd94fb9bc7b574ddf8426674

data/.rubocop.yml

@@ -29,6 +29,9 @@ Layout/LineLength:
 Layout/MultilineMethodCallIndentation:
   EnforcedStyle: indented
 
+Layout/RescueEnsureAlignment:
+  Enabled: false
+
 Lint/RaiseException:
   Enabled: true
 
@@ -44,6 +47,9 @@ RSpec/FilePath:
 RSpec/NamedSubject:
   Enabled: false
 
+RSpec/MessageSpies:
+  Enabled: false
+
 RSpec/MultipleExpectations:
   Enabled: false
 
@@ -59,6 +65,9 @@ Metrics/BlockLength:
 Metrics/MethodLength:
   Max: 30
 
+Naming/MethodParameterName:
+  MinNameLength: 1
+
 Style/Documentation:
   Enabled: false
 

data/.travis.yml

@@ -6,6 +6,8 @@ rvm:
   - 2.5
   - 2.6
   - 2.7
+  - jruby-9.2.10
+  - truffleruby-20.2.0
 
 env:
   global:
@@ -22,8 +24,8 @@ before_script:
   - ./cc-test-reporter before-build
 
 script:
-  - bin/rubocop
-  - bin/rspec --format doc
+  - bundle exec rubocop
+  - bundle exec rspec --format doc
   - bin/check-version
 
 after_script:

data/CHANGELOG.md

@@ -1,6 +1,42 @@
+## Release v0.3.0
+
+* Add tools for backend fault-tolerance #10
+  * CircuitProxy for wrapping storage in an internal circuit
+  * FallbackChain storage backend for falling back to stable storage
+  * Timeout warnings for Redis backend
+  * AutoWire wrappers for automatically configuring storage and cache
+  * Better documentation for fault-tolerance
+
+## Release v0.2.0
+
+* Remove Scopes and replace them with Faulty instances #9
+
+### Breaking Changes
+
+* `Faulty::Scope` has been removed. Use `Faulty.new` instead.
+* `Faulty` is now a class, not a module
+
+## Release v0.1.5
+
+* Fix redis storage to expire state key when using CAS #8
+
+## Release v0.1.4
+
+* Improve spec coverage for supporting classes #6
+* Fix redis bug where concurrent CAS requests could crash #7
+
+## Release v0.1.3
+
+* Fix bug where memory storage would delete the newest entries #5
+* Add HoneybadgerListener for error reporting #4
+
+## Release v0.1.2
+
+* Fix Storage::FaultTolerantProxy open and reopen methods #2
+
 ## Release v0.1.1
 
-* Fix a crash when Storage::FaultTolerantProxy created a status stub
+* Fix a crash when Storage::FaultTolerantProxy created a status stub #1
 
 ## Release v0.1.0
 

data/Gemfile

@@ -3,3 +3,20 @@
 source 'https://rubygems.org'
 
 gemspec
+
+# We add non-essential gems like debugging tools and CI dependencies
+# here. This also allows us to use conditional dependencies that depend on the
+# platform
+
+not_jruby = %i[ruby mingw x64_mingw].freeze
+
+gem 'activesupport', '>= 4.2'
+gem 'bundler', '>= 1.17', '< 3'
+gem 'byebug', platforms: not_jruby
+gem 'irb', '~> 1.0'
+gem 'redcarpet', '~> 3.5', platforms: not_jruby
+gem 'rspec_junit_formatter', '~> 0.4'
+# For now, code climate doesn't support simplecov 0.18
+# https://github.com/codeclimate/test-reporter/issues/413
+gem 'simplecov', '>= 0.17.1', '< 0.18'
+gem 'yard', '~> 0.9.25', platforms: not_jruby

data/README.md

@@ -67,6 +67,51 @@ end
 For a full list of configuration options, see the
 [Global Configuration](#global-configuration) section.
 
+## What is this for?
+
+Circuit breakers are a fault-tolerance tool for creating separation between your
+application and external dependencies. For example, your application may call an
+external API to send a text message:
+
+```ruby
+TextApi.send(message)
+```
+
+In normal operation, this API call is very fast. However what if the texting
+service started hanging? Your application would quickly use up a lot of
+resources waiting for requests to return from the service. You could consider
+adding a timeout to your request:
+
+```ruby
+TextApi.send(message, timeout: 5)
+```
+
+Now your application will terminate requests after 5 seconds, but that could
+still add up to a lot of resources if you call this thousands of times. Circuit
+breakers solve this problem.
+
+```ruby
+Faulty.circuit(:text_api).run do
+  TextApi.send(message, timeout: 5)
+end
+```
+
+Now, when the text API hangs, the first few will run and start timing out. This
+will trip the circuit. After the circuit trips
+(see [How it Works](#how-it-works)), calls to the text API will be paused for
+the configured cool down period. Your application resources are not overwhelmed.
+
+You are free to implement a fallback or error handling however you wish, for
+example, in this case, you might add the text message to a failure queue:
+
+```ruby
+Faulty.circuit(:text_api).run do
+  TextApi.send(message, timeout: 5)
+rescue Faulty::CircuitError => e
+  FailureQueue.enqueue(message)
+end
+```
+
 ## Basic Usage
 
 To create a circuit, call `Faulty.circuit`. This can be done as you use the
@@ -127,7 +172,8 @@ end.or_default([])
 
 ## How it Works
 
-Faulty implements a version of circuit breakers inspired by
+Faulty implements a version of circuit breakers inspired by "Release It!: Design
+and Deploy Production-Ready Software" by [Michael T. Nygard][michael nygard] and
 [Martin Fowler's post][martin fowler] on the subject. A few notable features of
 Faulty's implementation are:
 
@@ -135,11 +181,12 @@ Faulty's implementation are:
 - Integrated caching inspired by Netflix's [Hystrix][hystrix] with automatic
   cache jitter and error fallback.
 - Event-based monitoring
+- Flexible fault-tolerant storage with optional fallbacks
 
 Following the principals of the circuit-breaker pattern, the block given to
-`run` or `try_run` will always be executed as long as long as it never raises an
-error. If the block _does_ raise an error, then the circuit keeps track of the
-number of runs and the failure rate.
+`run` or `try_run` will always be executed as long as it never raises an error.
+If the block _does_ raise an error, then the circuit keeps track of the number
+of runs and the failure rate.
 
 Once both thresholds are breached, the circuit is opened. Once open, the
 circuit starts the cool-down period. Any executions within that cool-down are
@@ -158,22 +205,30 @@ In addition to the classic circuit breaker design, Faulty implements caching
 that is integrated with the circuit state. See [Caching](#caching) for more
 detail.
 
-## Global Configuration
+## Configuration
 
-`Faulty.init` can set the following global configuration options. This example
-illustrates the default values. It is also possible to define multiple
-non-global configuration scopes (see [Scopes](#scopes)).
+Faulty can be configured with the following configuration options. This example
+illustrates the default values. In the first example, we configure Faulty
+globally. The second example shows the same configuration using an instance of
+Faulty instead of global configuration.
 
 ```ruby
 Faulty.init do |config|
   # The cache backend to use. By default, Faulty looks for a Rails cache. If
   # that's not available, it uses an ActiveSupport::Cache::Memory instance.
   # Otherwise, it uses a Faulty::Cache::Null and caching is disabled.
+  # Whatever backend is given here is automatically wrapped in
+  # Faulty::Cache::AutoWire. This adds fault-tolerance features, see the
+  # AutoWire API docs for more details.
   config.cache = Faulty::Cache::Default.new
 
   # The storage backend. By default, Faulty uses an in-memory store. For most
   # production applications, you'll want a more robust backend. Faulty also
   # provides Faulty::Storage::Redis for this.
+  # Whatever backend is given here is automatically wrapped in
+  # Faulty::Storage::AutoWire. This adds fault-tolerance features, see the
+  # AutoWire APi docs for more details. If an array of storage backends is
+  # given, each one will be tried in order until one succeeds.
   config.storage = Faulty::Storage::Memory.new
 
   # An array of event listeners. Each object in the array should implement
@@ -188,6 +243,25 @@ Faulty.init do |config|
 end
 ```
 
+Here is the same configuration using an instance of `Faulty`. This is a more
+object-oriented approach.
+
+```ruby
+faulty = Faulty.new do |config|
+  config.cache = Faulty::Cache::Default.new
+  config.storage = Faulty::Storage::Memory.new
+  config.listeners = [Faulty::Events::LogListener.new]
+  config.notifier = Faulty::Events::Notifier.new(config.listeners)
+end
+```
+
+Most of the examples in this README use the global Faulty class methods, but
+they work the same way when using an instance. Just substitute your instance
+instead of `Faulty`. There is no preferred way to use Faulty. Choose whichever
+configuration mechanism works best for your application. Also see
+[Multiple Configurations](#multiple-configurations) if your application needs
+to set different options in different scenarios.
+
 For all Faulty APIs that have configuration, you can also pass in an options
 hash. For example, `Faulty.init` could be called like this:
 
@@ -199,9 +273,9 @@ Faulty.init(cache: Faulty::Cache::Null.new)
 
 A circuit can be created with the following configuration options. Those options
 are only set once, synchronized across threads, and will persist in-memory until
-the process exits. If you're using [scopes](#scopes), the options are retained
-within the context of each scope. All options given after the first call to
-`Faulty.circuit` (or `Scope.circuit` are ignored.
+the process exits. If you're using [multiple configurations](#multiple-configurations),
+the options are retained within the context of each instance. All options given
+after the first call to `Faulty.circuit` (or `Faulty#circuit`) are ignored.
 
 This is because the circuit objects themselves are internally memoized, and are
 read-only once created.
@@ -262,8 +336,8 @@ Faulty.circuit(:api, cache_expires_in: 1800)
 
 Faulty integrates caching into it's circuits in a way that is particularly
 suited to fault-tolerance. To make use of caching, you must specify the `cache`
-configuration option when initializing Faulty or creating a scope. If you're
-using Rails, this is automatically set to the Rails cache.
+configuration option when initializing Faulty or creating a new Faulty instance.
+If you're using Rails, this is automatically set to the Rails cache.
 
 Once your cache is configured, you can use the `cache` parameter when running
 a circuit to specify a cache key:
@@ -315,7 +389,12 @@ Faulty backends are fault-tolerant by default. Any `StandardError`s raised by
 the storage or cache backends are captured and suppressed. Failure events for
 these errors are sent to the notifier.
 
-If the storage backend fails, all circuits will default to open. If the cache
+In case of a flaky storage or cache backend, Faulty also uses independent
+in-memory circuits to track failures so that we don't keep calling a backend
+that is failing. See the API docs for `Cache::AutoWire`, and `Storage::AutoWire`
+for more details.
+
+If the storage backend fails, circuits will default to closed. If the cache
 backend fails, all cache queries will miss.
 
 ## Event Handling
@@ -338,12 +417,17 @@ events. The full list of events is available from `Faulty::Events::EVENTS`.
   closed. Payload: `circuit`
 - `circuit_success` - A circuit execution was successful. Payload: `circuit`,
   `status`
-- `storage_failure` - A storage backend raised an error. Payload `circuit`,
-  `action`, `error`
+- `storage_failure` - A storage backend raised an error. Payload `circuit` (can
+  be nil), `action`, `error`
 
 By default events are logged using `Faulty::Events::LogListener`, but that can
 be replaced, or additional listeners can be added.
 
+### CallbackListener
+
+The callback listener is useful for ad-hoc handling of events. You can specify
+an event handler by calling a method on the callback handler by the same name.
+
 ```ruby
 Faulty.init do |config|
   # Replace the default listener with a custom callback listener
@@ -356,6 +440,17 @@ Faulty.init do |config|
 end
 ```
 
+### Other Built-in Listeners
+
+In addition to the log and callback listeners, Faulty intends to implement
+built-in service-specific handlers to make it easy to integrate with monitoring
+and reporting software.
+
+- `Faulty::Events::HoneybadgerListener`: Reports circuit and backend errors to
+  the Honeybadger error reporting service.
+
+### Custom Listeners
+
 You can implement your own listener by following the documentation in
 `Faulty::Events::ListenerInterface`. For example:
 
@@ -375,10 +470,19 @@ end
 
 ## Configuring the Storage Backend
 
+A storage backend is required to use Faulty. By default, it uses in-memory
+storage, but Redis is also available, along with a number of wrappers used to
+improve resiliency and fault-tolerance.
+
 ### Memory
 
-The `Faulty::Cache::Memory` backend is the default storage backend. The default
-configuration:
+The `Faulty::Storage::Memory` backend is the default storage backend. You may
+prefer this implementation if you want to avoid the complexity and potential
+failure-mode of cross-network circuit storage. The trade-off is that circuit
+state is only contained within a single process and will not be saved across
+application restarts. Locks will also be cleared on restart.
+
+The default configuration:
 
 ```ruby
 Faulty.init do |config|
@@ -391,36 +495,159 @@ end
 
 ### Redis
 
-The `Faulty::Cache::Redis` backend provides distributed circuit storage using
-Redis. The default configuration:
+The `Faulty::Storage::Redis` backend provides distributed circuit storage using
+Redis. Although Faulty takes steps to reduce risk
+(See [Fault Tolerance](#fault-tolerance)), using cross-network storage does
+introduce some additional failure modes. To reduce this risk, be sure to set
+conservative timeouts for your Redis connection. Setting high timeouts will
+print warnings to stderr.
+
+The default configuration:
 
 ```ruby
 Faulty.init do |config|
   config.storage = Faulty::Storage::Redis.new do |storage|
     # The Redis client. Accepts either a Redis instance, or a ConnectionPool
-    # of Redis instances.
-    storage.client = ::Redis.new
+    # of Redis instances. A low timeout is highly recommended to prevent
+    # cascading failures when evaluating circuits.
+    storage.client = ::Redis.new(timeout: 1)
 
     # The prefix to prepend to all redis keys used by Faulty circuits
     storage.key_prefix = 'faulty'
 
     # A string to separate the parts of the redis key
-    storage.key_separator: ':'
+    storage.key_separator = ':'
 
     # The maximum number of circuit runs that will be stored
     storage.max_sample_size = 100
 
     # The maximum number of seconds that a circuit run will be stored
     storage.sample_ttl = 1800
+
+    # The maximum number of seconds to store a circuit. Does not apply to
+    # locks, which are indefinite.
+    storage.circuit_ttl = 604_800 # 1 Week
+
+    # The number of seconds between circuit expirations. Changing this setting
+    # is not recommended. See API docs for more implementation details.
+    storage.list_granularity = 3600
+
+    # If true, disables warnings about recommended client settings like timeouts
+    storage.disable_warnings = false
   end
 end
 ```
 
-### Listing Circuits
+### FallbackChain
+
+The `Faulty::Storage::FallbackChain` backend is a wrapper for multiple
+prioritized storage backends. If the first backend in the chain fails,
+consecutive backends are tried until one succeeds. The recommended use-case for
+this is to fall back on reliable storage if a networked storage backend fails.
+
+For example, you may configure Redis as your primary storage backend, with an
+in-memory storage backend as a fallback:
+
+```ruby
+Faulty.init do |config|
+  config.storage = Faulty::Storage::FallbackChain.new([
+    Faulty::Storage::Redis.new,
+    Faulty::Storage::Memory.new
+  ])
+end
+```
+
+Faulty instances will automatically use a fallback chain if an array is given to
+the `storage` option, so this example is equivalent to the above:
+
+```ruby
+Faulty.init do |config|
+  config.storage = [
+    Faulty::Storage::Redis.new,
+    Faulty::Storage::Memory.new
+  ]
+end
+```
+
+If the fallback chain fails-over to backup storage, circuit states will not
+carry over, so failover could be temporarily disruptive to your application.
+However, any calls to `#lock` or `#unlock` will always be persisted to all
+backends so that locks are maintained during failover.
+
+### Storage::FaultTolerantProxy
+
+This wrapper is applied to all non-fault-tolerant storage backends by default
+(see the [API docs for `Faulty::Storage::AutoWire`](https://www.rubydoc.info/gems/faulty/Faulty/Storage/AutoWire)).
+
+`Faulty::Storage::FaultTolerantProxy` is a wrapper that suppresses storage
+errors and returns sensible defaults during failures. If a storage backend is
+failing, all circuits will be treated as closed regardless of locks or previous
+history.
+
+If you wish your application to use a secondary storage backend instead of
+failing closed, use `FallbackChain`.
+
+### Storage::CircuitProxy
+
+This wrapper is applied to all non-fault-tolerant storage backends by default
+(see the [API docs for `Faulty::Storage::AutoWire`](https://www.rubydoc.info/gems/faulty/Faulty/Cache/AutoWire)).
+
+`Faulty::Storage::CircuitProxy` is a wrapper that uses an independent in-memory
+circuit to track failures to storage backends. If a storage backend fails
+continuously, it will be temporarily disabled and raise `Faulty::CircuitError`s.
+
+Typically this is used inside a `FaultTolerantProxy` or `FallbackChain` so that
+these storage failures are handled gracefully.
+
+## Configuring the Cache Backend
+
+### Null
+
+The `Faulty::Cache::Null` cache disables caching. It is the default if Rails and
+ActiveSupport are not present.
+
+### Rails
+
+`Faulty::Cache::Rails` is the default cache if Rails or ActiveSupport are
+present. If Rails is present, it uses `Rails.cache` as the backend. If
+ActiveSupport is present, but Rails is not, it creates a new
+`ActiveSupport::Cache::MemoryStore` by default.  This backend can be used with
+any `ActiveSupport::Cache`.
+
+```ruby
+Faulty.init do |config|
+  config.cache = Faulty::Cache::Rails.new(
+    ActiveSupport::Cache::RedisCacheStore.new
+  )
+end
+```
+
+### Cache::FaultTolerantProxy
+
+This wrapper is applied to all non-fault-tolerant cache backends by default
+(see the API docs for `Faulty::Cache::AutoWire`).
+
+`Faulty::Cache::FaultTolerantProxy` is a wrapper that suppresses cache errors
+and acts like a null cache during failures. Reads always return `nil`, and
+writes are no-ops.
+
+### Cache::CircuitProxy
+
+This wrapper is applied to all non-fault-tolerant circuit backends by default
+(see the API docs for `Faulty::Circuit::AutoWire`).
+
+`Faulty::Circuit::CircuitProxy` is a wrapper that uses an independent in-memory
+circuit to track failures to circuit backends. If a circuit backend fails
+continuously, it will be temporarily disabled and raise `Faulty::CircuitError`s.
+
+Typically this is used inside a `FaultTolerantProxy` so that these cache
+failures are handled gracefully.
+
+## Listing Circuits
 
 For monitoring or debugging, you may need to retrieve a list of all circuit
-names. This is possible with `Faulty.list_circuits` (or the equivalent method on
-your [scope](#scopes)).
+names. This is possible with `Faulty.list_circuits` (or `Faulty#list_circuits`
+if you're using an instance).
 
 You can get a list of all circuit statuses by mapping those names to their
 status objects. Be careful though, since this could cause performance issues for
@@ -432,73 +659,103 @@ statuses = Faulty.list_circuits.map do |name|
 end
 ```
 
-## Scopes
+## Locking Circuits
+
+It is possible to lock a circuit open or closed. A circuit that is locked open
+will never execute its block, and always raise an `Faulty::OpenCircuitError`.
+This is useful in cases where you need to manually disable a dependency
+entirely. If a cached value is available, that will be returned from the circuit
+until it expires, even outside its refresh period.
+
+```ruby
+Faulty.circuit(:broken_api).lock_open!
+```
+
+A circuit that is locked closed will never trip. This is useful in cases where a
+circuit is continuously tripping incorrectly. If a cached value is available, it
+will have the same behavior as an unlocked circuit.
+
+```ruby
+Faulty.circuit(:false_positive).lock_closed!
+```
+
+To remove a lock of either type:
+
+```ruby
+Faulty.circuit(:fixed).unlock!
+```
+
+Locking or unlocking a circuit has no concurrency guarantees, so it's not
+recommended to lock or unlock circuits from production code. Instead, locks are
+intended as an emergency tool for troubleshooting and debugging.
+
+## Multiple Configurations
 
 It is possible to have multiple configurations of Faulty running within the same
-process. The most common configuration is to simply use `Faulty.init` to
+process. The most common setup is to simply use `Faulty.init` to
 configure Faulty globally, however it is possible to have additional
-configurations using scopes.
+configurations.
 
-### The default scope
+### The default instance
 
-When you call `Faulty.init`, you are actually creating the default scope. You
-can access this scope directly by calling `Faulty.default`.
+When you call `Faulty.init`, you are actually creating the default instance of
+`Faulty`. You can access this instance directly by calling `Faulty.default`.
 
 ```ruby
-# We create the default scope
+# We create the default instance
 Faulty.init
 
-# Access the default scope
-scope = Faulty.default
+# Access the default instance
+faulty = Faulty.default
 
-# Alternatively, access the scope by name
-scope = Faulty[:default]
+# Alternatively, access the instance by name
+faulty = Faulty[:default]
 ```
 
-You can rename the default scope if desired:
+You can rename the default instance if desired:
 
 ```ruby
 Faulty.init(:custom_default)
 
-scope = Faulty.default
-scope = Faulty[:custom_default]
+instance = Faulty.default
+instance = Faulty[:custom_default]
 ```
 
-### Multiple Scopes
+### Multiple Instances
 
-If you want multiple scopes, but want global, thread-safe access to
+If you want multiple instance, but want global, thread-safe access to
 them, you can use `Faulty.register`:
 
 ```ruby
-api_scope = Faulty::Scope.new do |config|
+api_faulty = Faulty.new do |config|
   # This accepts the same options as Faulty.init
 end
 
-Faulty.register(:api, api_scope)
+Faulty.register(:api, api_faulty)
 
-# Now access the scope globally
+# Now access the instance globally
 Faulty[:api]
 ```
 
 When you call `Faulty.circuit`, that's the same as calling
-`Faulty.default.circuit`, so you can apply the same API to any other Faulty
-scope:
+`Faulty.default.circuit`, so you can apply the same principal to any other
+registered Faulty instance:
 
 ```ruby
 Faulty[:api].circuit(:api_circuit).run { 'ok' }
 ```
 
-### Standalone Scopes
+### Standalone Instances
 
-If you choose, you can use Faulty scopes without registering them globally. This
-could be useful if you prefer dependency injection over global state.
+If you choose, you can use Faulty instances without registering them globally.
+This is more object-oriented and is necessary if you use dependency injection.
 
 ```ruby
-faulty = Faulty::Scope.new
+faulty = Faulty.new
 faulty.circuit(:standalone_circuit)
 ```
 
-Calling `circuit` on the scope still has the same memoization behavior that
+Calling `#circuit` on the instance still has the same memoization behavior that
 `Faulty.circuit` has, so subsequent calls to the same circuit will return a
 memoized circuit object.
 
@@ -545,15 +802,36 @@ would be useful for other users.
 Faulty has its own opinions about how to implement a circuit breaker in Ruby,
 but there are and have been many other options:
 
-- [circuitbox](https://github.com/yammer/circuitbox)
-- [circuit_breaker-ruby](https://github.com/scripbox/circuit_breaker-ruby)
-- [stoplight](https://github.com/orgsync/stoplight) (currently unmaintained)
+### Currently Active
+
+- [semian](https://github.com/Shopify/semian): A resiliency toolkit that
+  includes circuit breakers. It uses adapters to auto-wire circuits, and it has
+  only host-local storage by design.
+- [circuitbox](https://github.com/yammer/circuitbox): Similar in design to
+  Faulty, but with a different API. It uses Moneta to abstract circuit storage
+  to allow any key-value store.
+
+### Previous Work
+
+- [circuit_breaker-ruby](https://github.com/scripbox/circuit_breaker-ruby) (no
+  recent activity)
+- [stoplight](https://github.com/orgsync/stoplight) (unmaintained)
 - [circuit_breaker](https://github.com/wooga/circuit_breaker) (archived)
 - [simple_circuit_breaker](https://github.com/soundcloud/simple_circuit_breaker)
   (unmaintained)
 - [breaker](https://github.com/ahawkins/breaker) (unmaintained)
 - [circuit_b](https://github.com/alg/circuit_b) (unmaintained)
 
+### Faulty's Unique Features
+
+- Simple API but configurable for advanced users
+- Pluggable storage backends (circuitbox also has this)
+- Protected storage access with fallback to safe storage
+- Global, or object-oriented configuration with multiple instances
+- Integrated caching support tailored for fault-tolerance
+- Manually lock circuits open or closed
+
 [api docs]: https://www.rubydoc.info/github/ParentSquare/faulty/master
+[michael nygard]: https://www.michaelnygard.com/
 [martin fowler]: https://www.martinfowler.com/bliki/CircuitBreaker.html
 [hystrix]: https://github.com/Netflix/Hystrix/wiki/How-it-Works