faulty 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 55bf688c3615a59f51103633db0902abf3458324945ab855894e5975981868f5
-  data.tar.gz: dab412aad317a79364782a85c2a6a184e55fa2214cd966655d74eeab69e248c0
+  metadata.gz: c0712dd797dfe2922e4e52a792f48c5ac72c4b9766d218dfa597e7ad630f9fd0
+  data.tar.gz: fb4a787507006131f34a301fc164466fd8befb72051550ca0a63fc2d69ec949c
 SHA512:
-  metadata.gz: 7527a23df0c042ea317e99b8fafdf8c8f96ac5f9f68323ba4c4a2901f8e32dc7ca2b7c62b6b7197e446985eab6901f85dd41c6463407ae3632d86e022626d733
-  data.tar.gz: 7e3d48b82905b0ad2303449a532e85736a0681fa4dcb8a5f9ee64e6ffd10c76fdc760ac97aa9dd882f7a019d335440f9d91f0ba1efbb92f978c5e9e43c5b977a
+  metadata.gz: 5381dd13f058045906330c67ff80aeefac55ae81077eb4552d5461d743344fa097e205b8ac3a39b385c52976e64315aec4e903c1a0c298a05452fab7788d1df2
+  data.tar.gz: af31cbf86cac54226189f8f63b1d8b0bcf82f9023085e98d07e4e9db65e402883c2219dbf255e21427c40b0c0a66ec71bd4dc9dfdd94fb9bc7b574ddf8426674

data/.rubocop.yml

@@ -47,6 +47,9 @@ RSpec/FilePath:
 RSpec/NamedSubject:
   Enabled: false
 
+RSpec/MessageSpies:
+  Enabled: false
+
 RSpec/MultipleExpectations:
   Enabled: false
 

data/.travis.yml

@@ -6,8 +6,8 @@ rvm:
   - 2.5
   - 2.6
   - 2.7
-  - jruby
-  - truffleruby
+  - jruby-9.2.10
+  - truffleruby-20.2.0
 
 env:
   global:

data/CHANGELOG.md

@@ -1,3 +1,12 @@
+## Release v0.3.0
+
+* Add tools for backend fault-tolerance #10
+  * CircuitProxy for wrapping storage in an internal circuit
+  * FallbackChain storage backend for falling back to stable storage
+  * Timeout warnings for Redis backend
+  * AutoWire wrappers for automatically configuring storage and cache
+  * Better documentation for fault-tolerance
+
 ## Release v0.2.0
 
 * Remove Scopes and replace them with Faulty instances #9

data/README.md

@@ -172,7 +172,8 @@ end.or_default([])
 
 ## How it Works
 
-Faulty implements a version of circuit breakers inspired by
+Faulty implements a version of circuit breakers inspired by "Release It!: Design
+and Deploy Production-Ready Software" by [Michael T. Nygard][michael nygard] and
 [Martin Fowler's post][martin fowler] on the subject. A few notable features of
 Faulty's implementation are:
 
@@ -180,6 +181,7 @@ Faulty's implementation are:
 - Integrated caching inspired by Netflix's [Hystrix][hystrix] with automatic
   cache jitter and error fallback.
 - Event-based monitoring
+- Flexible fault-tolerant storage with optional fallbacks
 
 Following the principals of the circuit-breaker pattern, the block given to
 `run` or `try_run` will always be executed as long as it never raises an error.
@@ -215,11 +217,18 @@ Faulty.init do |config|
   # The cache backend to use. By default, Faulty looks for a Rails cache. If
   # that's not available, it uses an ActiveSupport::Cache::Memory instance.
   # Otherwise, it uses a Faulty::Cache::Null and caching is disabled.
+  # Whatever backend is given here is automatically wrapped in
+  # Faulty::Cache::AutoWire. This adds fault-tolerance features, see the
+  # AutoWire API docs for more details.
   config.cache = Faulty::Cache::Default.new
 
   # The storage backend. By default, Faulty uses an in-memory store. For most
   # production applications, you'll want a more robust backend. Faulty also
   # provides Faulty::Storage::Redis for this.
+  # Whatever backend is given here is automatically wrapped in
+  # Faulty::Storage::AutoWire. This adds fault-tolerance features, see the
+  # AutoWire APi docs for more details. If an array of storage backends is
+  # given, each one will be tried in order until one succeeds.
   config.storage = Faulty::Storage::Memory.new
 
   # An array of event listeners. Each object in the array should implement
@@ -380,7 +389,12 @@ Faulty backends are fault-tolerant by default. Any `StandardError`s raised by
 the storage or cache backends are captured and suppressed. Failure events for
 these errors are sent to the notifier.
 
-If the storage backend fails, all circuits will default to open. If the cache
+In case of a flaky storage or cache backend, Faulty also uses independent
+in-memory circuits to track failures so that we don't keep calling a backend
+that is failing. See the API docs for `Cache::AutoWire`, and `Storage::AutoWire`
+for more details.
+
+If the storage backend fails, circuits will default to closed. If the cache
 backend fails, all cache queries will miss.
 
 ## Event Handling
@@ -456,10 +470,19 @@ end
 
 ## Configuring the Storage Backend
 
+A storage backend is required to use Faulty. By default, it uses in-memory
+storage, but Redis is also available, along with a number of wrappers used to
+improve resiliency and fault-tolerance.
+
 ### Memory
 
-The `Faulty::Cache::Memory` backend is the default storage backend. The default
-configuration:
+The `Faulty::Storage::Memory` backend is the default storage backend. You may
+prefer this implementation if you want to avoid the complexity and potential
+failure-mode of cross-network circuit storage. The trade-off is that circuit
+state is only contained within a single process and will not be saved across
+application restarts. Locks will also be cleared on restart.
+
+The default configuration:
 
 ```ruby
 Faulty.init do |config|
@@ -472,15 +495,22 @@ end
 
 ### Redis
 
-The `Faulty::Cache::Redis` backend provides distributed circuit storage using
-Redis. The default configuration:
+The `Faulty::Storage::Redis` backend provides distributed circuit storage using
+Redis. Although Faulty takes steps to reduce risk
+(See [Fault Tolerance](#fault-tolerance)), using cross-network storage does
+introduce some additional failure modes. To reduce this risk, be sure to set
+conservative timeouts for your Redis connection. Setting high timeouts will
+print warnings to stderr.
+
+The default configuration:
 
 ```ruby
 Faulty.init do |config|
   config.storage = Faulty::Storage::Redis.new do |storage|
     # The Redis client. Accepts either a Redis instance, or a ConnectionPool
-    # of Redis instances.
-    storage.client = ::Redis.new
+    # of Redis instances. A low timeout is highly recommended to prevent
+    # cascading failures when evaluating circuits.
+    storage.client = ::Redis.new(timeout: 1)
 
     # The prefix to prepend to all redis keys used by Faulty circuits
     storage.key_prefix = 'faulty'
@@ -493,10 +523,126 @@ Faulty.init do |config|
 
     # The maximum number of seconds that a circuit run will be stored
     storage.sample_ttl = 1800
+
+    # The maximum number of seconds to store a circuit. Does not apply to
+    # locks, which are indefinite.
+    storage.circuit_ttl = 604_800 # 1 Week
+
+    # The number of seconds between circuit expirations. Changing this setting
+    # is not recommended. See API docs for more implementation details.
+    storage.list_granularity = 3600
+
+    # If true, disables warnings about recommended client settings like timeouts
+    storage.disable_warnings = false
   end
 end
 ```
 
+### FallbackChain
+
+The `Faulty::Storage::FallbackChain` backend is a wrapper for multiple
+prioritized storage backends. If the first backend in the chain fails,
+consecutive backends are tried until one succeeds. The recommended use-case for
+this is to fall back on reliable storage if a networked storage backend fails.
+
+For example, you may configure Redis as your primary storage backend, with an
+in-memory storage backend as a fallback:
+
+```ruby
+Faulty.init do |config|
+  config.storage = Faulty::Storage::FallbackChain.new([
+    Faulty::Storage::Redis.new,
+    Faulty::Storage::Memory.new
+  ])
+end
+```
+
+Faulty instances will automatically use a fallback chain if an array is given to
+the `storage` option, so this example is equivalent to the above:
+
+```ruby
+Faulty.init do |config|
+  config.storage = [
+    Faulty::Storage::Redis.new,
+    Faulty::Storage::Memory.new
+  ]
+end
+```
+
+If the fallback chain fails-over to backup storage, circuit states will not
+carry over, so failover could be temporarily disruptive to your application.
+However, any calls to `#lock` or `#unlock` will always be persisted to all
+backends so that locks are maintained during failover.
+
+### Storage::FaultTolerantProxy
+
+This wrapper is applied to all non-fault-tolerant storage backends by default
+(see the [API docs for `Faulty::Storage::AutoWire`](https://www.rubydoc.info/gems/faulty/Faulty/Storage/AutoWire)).
+
+`Faulty::Storage::FaultTolerantProxy` is a wrapper that suppresses storage
+errors and returns sensible defaults during failures. If a storage backend is
+failing, all circuits will be treated as closed regardless of locks or previous
+history.
+
+If you wish your application to use a secondary storage backend instead of
+failing closed, use `FallbackChain`.
+
+### Storage::CircuitProxy
+
+This wrapper is applied to all non-fault-tolerant storage backends by default
+(see the [API docs for `Faulty::Storage::AutoWire`](https://www.rubydoc.info/gems/faulty/Faulty/Cache/AutoWire)).
+
+`Faulty::Storage::CircuitProxy` is a wrapper that uses an independent in-memory
+circuit to track failures to storage backends. If a storage backend fails
+continuously, it will be temporarily disabled and raise `Faulty::CircuitError`s.
+
+Typically this is used inside a `FaultTolerantProxy` or `FallbackChain` so that
+these storage failures are handled gracefully.
+
+## Configuring the Cache Backend
+
+### Null
+
+The `Faulty::Cache::Null` cache disables caching. It is the default if Rails and
+ActiveSupport are not present.
+
+### Rails
+
+`Faulty::Cache::Rails` is the default cache if Rails or ActiveSupport are
+present. If Rails is present, it uses `Rails.cache` as the backend. If
+ActiveSupport is present, but Rails is not, it creates a new
+`ActiveSupport::Cache::MemoryStore` by default.  This backend can be used with
+any `ActiveSupport::Cache`.
+
+```ruby
+Faulty.init do |config|
+  config.cache = Faulty::Cache::Rails.new(
+    ActiveSupport::Cache::RedisCacheStore.new
+  )
+end
+```
+
+### Cache::FaultTolerantProxy
+
+This wrapper is applied to all non-fault-tolerant cache backends by default
+(see the API docs for `Faulty::Cache::AutoWire`).
+
+`Faulty::Cache::FaultTolerantProxy` is a wrapper that suppresses cache errors
+and acts like a null cache during failures. Reads always return `nil`, and
+writes are no-ops.
+
+### Cache::CircuitProxy
+
+This wrapper is applied to all non-fault-tolerant circuit backends by default
+(see the API docs for `Faulty::Circuit::AutoWire`).
+
+`Faulty::Circuit::CircuitProxy` is a wrapper that uses an independent in-memory
+circuit to track failures to circuit backends. If a circuit backend fails
+continuously, it will be temporarily disabled and raise `Faulty::CircuitError`s.
+
+Typically this is used inside a `FaultTolerantProxy` so that these cache
+failures are handled gracefully.
+
 ## Listing Circuits
 
 For monitoring or debugging, you may need to retrieve a list of all circuit
@@ -661,7 +807,7 @@ but there are and have been many other options:
 - [semian](https://github.com/Shopify/semian): A resiliency toolkit that
   includes circuit breakers. It uses adapters to auto-wire circuits, and it has
   only host-local storage by design.
-- [circuitbox](https://github.com/yammer/circuitbox): Similar in function to
+- [circuitbox](https://github.com/yammer/circuitbox): Similar in design to
   Faulty, but with a different API. It uses Moneta to abstract circuit storage
   to allow any key-value store.
 
@@ -680,10 +826,12 @@ but there are and have been many other options:
 
 - Simple API but configurable for advanced users
 - Pluggable storage backends (circuitbox also has this)
+- Protected storage access with fallback to safe storage
 - Global, or object-oriented configuration with multiple instances
 - Integrated caching support tailored for fault-tolerance
 - Manually lock circuits open or closed
 
 [api docs]: https://www.rubydoc.info/github/ParentSquare/faulty/master
+[michael nygard]: https://www.michaelnygard.com/
 [martin fowler]: https://www.martinfowler.com/bliki/CircuitBreaker.html
 [hystrix]: https://github.com/Netflix/Hystrix/wiki/How-it-Works

data/bin/check-version

@@ -3,8 +3,12 @@
 set -e
 
 tag="$(git describe --abbrev=0 2>/dev/null || echo)"
+echo "Tag: ${tag}"
 tag="${tag#v}"
+echo "Git Version: ${tag}"
 [ "$tag" = '' ] && exit 0
+gem_version="$(ruby -r ./lib/faulty/version -e "puts Faulty.version" | tail -n1)"
+echo "Gem Version: ${gem_version}"
 
-tag_gt_version=$(ruby -r ./lib/faulty/version -e "puts Faulty.version >= Gem::Version.new('${tag}')")
+tag_gt_version="$(ruby -r ./lib/faulty/version -e "puts Faulty.version >= Gem::Version.new('${tag}')" | tail -n1)"
 test "$tag_gt_version" = true

data/lib/faulty.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require 'securerandom'
+require 'forwardable'
 require 'concurrent-ruby'
 
 require 'faulty/immutable_options'
@@ -124,14 +125,17 @@ class Faulty
   # Options for {Faulty}
   #
   # @!attribute [r] cache
+  #   @see Cache::AutoWire
   #   @return [Cache::Interface] A cache backend if you want
   #     to use Faulty's cache support. Automatically wrapped in a
-  #     {Cache::FaultTolerantProxy}. Default `Cache::Default.new`.
+  #     {Cache::AutoWire}. Default `Cache::AutoWire.new`.
   # @!attribute [r] storage
-  #   @return [Storage::Interface] The storage backend.
-  #     Automatically wrapped in a {Storage::FaultTolerantProxy}.
-  #     Default `Storage::Memory.new`.
+  #   @see Storage::AutoWire
+  #   @return [Storage::Interface, Array<Storage::Interface>] The storage
+  #   backend. Automatically wrapped in a {Storage::AutoWire}, so this can also
+  #   be given an array of prioritized backends. Default `Storage::AutoWire.new`.
   # @!attribute [r] listeners
+  #   @see Events::ListenerInterface
   #   @return [Array] listeners Faulty event listeners
   # @!attribute [r] notifier
   #   @return [Events::Notifier] A Faulty notifier. If given, listeners are
@@ -148,16 +152,8 @@ class Faulty
 
     def finalize
       self.notifier ||= Events::Notifier.new(listeners || [])
-
-      self.storage ||= Storage::Memory.new
-      unless storage.fault_tolerant?
-        self.storage = Storage::FaultTolerantProxy.new(storage, notifier: notifier)
-      end
-
-      self.cache ||= Cache::Default.new
-      unless cache.fault_tolerant?
-        self.cache = Cache::FaultTolerantProxy.new(cache, notifier: notifier)
-      end
+      self.storage = Storage::AutoWire.new(storage, notifier: notifier)
+      self.cache = Cache::AutoWire.new(cache, notifier: notifier)
     end
 
     def required
@@ -223,8 +219,6 @@ class Faulty
   #
   # @return [Hash] The circuit options
   def circuit_options
-    options = @options.to_h
-    options.delete(:listeners)
-    options
+    @options.to_h.select { |k, _v| %i[cache storage notifier].include?(k) }
   end
 end

data/lib/faulty/cache.rb

@@ -6,7 +6,9 @@ class Faulty
   end
 end
 
+require 'faulty/cache/auto_wire'
 require 'faulty/cache/default'
+require 'faulty/cache/circuit_proxy'
 require 'faulty/cache/fault_tolerant_proxy'
 require 'faulty/cache/mock'
 require 'faulty/cache/null'

data/lib/faulty/cache/auto_wire.rb

@@ -0,0 +1,65 @@
+# frozen_string_literal: true
+
+class Faulty
+  module Cache
+    # Automatically configure a cache backend
+    #
+    # Used by {Faulty#initialize} to setup sensible cache defaults
+    class AutoWire
+      extend Forwardable
+
+      # Options for {AutoWire}
+      Options = Struct.new(
+        :notifier
+      ) do
+        include ImmutableOptions
+
+        private
+
+        def required
+          %i[notifier]
+        end
+      end
+
+      # Wrap a cache backend with sensible defaults
+      #
+      # If the cache is `nil`, create a new {Default}.
+      #
+      # If the backend is not fault tolerant, wrap it in {CircuitProxy} and
+      # {FaultTolerantProxy}.
+      #
+      # @param cache [Interface] A cache backend
+      # @param options [Hash] Attributes for {Options}
+      # @yield [Options] For setting options in a block
+      def initialize(cache, **options, &block)
+        @options = Options.new(options, &block)
+        @cache = if cache.nil?
+          Cache::Default.new
+        elsif cache.fault_tolerant?
+          cache
+        else
+          Cache::FaultTolerantProxy.new(
+            Cache::CircuitProxy.new(cache, notifier: @options.notifier),
+            notifier: @options.notifier
+          )
+        end
+
+        freeze
+      end
+
+      # @!method read(key)
+      #   (see Faulty::Cache::Interface#read)
+      #
+      # @!method write(key, value, expires_in: expires_in)
+      #   (see Faulty::Cache::Interface#write)
+      def_delegators :@cache, :read, :write
+
+      # Auto-wired caches are always fault tolerant
+      #
+      # @return [true]
+      def fault_tolerant?
+        true
+      end
+    end
+  end
+end

data/lib/faulty/cache/circuit_proxy.rb

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+class Faulty
+  module Cache
+    # A circuit wrapper for cache backends
+    #
+    # This class uses an internal {Circuit} to prevent the cache backend from
+    # causing application issues. If the backend fails continuously, this
+    # circuit will trip to prevent cascading failures. This internal circuit
+    # uses an independent in-memory backend by default.
+    class CircuitProxy
+      attr_reader :options
+
+      # Options for {CircuitProxy}
+      #
+      # @!attribute [r] circuit
+      #   @return [Circuit] A replacement for the internal circuit. When
+      #     modifying this, be careful to use only a reliable circuit storage
+      #     backend so that you don't introduce cascading failures.
+      # @!attribute [r] notifier
+      #   @return [Events::Notifier] A Faulty notifier to use for failure
+      #     notifications. If `circuit` is given, this is ignored.
+      Options = Struct.new(
+        :circuit,
+        :notifier
+      ) do
+        include ImmutableOptions
+
+        private
+
+        def finalize
+          raise ArgumentError, 'The circuit or notifier option must be given' unless notifier || circuit
+
+          self.circuit ||= Circuit.new(
+            Faulty::Storage::CircuitProxy.name,
+            notifier: notifier,
+            cache: Cache::Null.new
+          )
+        end
+      end
+
+      # @param cache [Cache::Interface] The cache backend to wrap
+      # @param options [Hash] Attributes for {Options}
+      # @yield [Options] For setting options in a block
+      def initialize(cache, **options, &block)
+        @cache = cache
+        @options = Options.new(options, &block)
+      end
+
+      %i[read write].each do |method|
+        define_method(method) do |*args|
+          options.circuit.run { @cache.public_send(method, *args) }
+        end
+      end
+
+      def fault_tolerant?
+        @cache.fault_tolerant?
+      end
+    end
+  end
+end