semian 0.27.0 → 0.28.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cc25bcc8e0291e3f53d1ecb6ac19e56b375ce0c16974d16c20b9a847b6d2104d
-  data.tar.gz: 8c4ab8f6259e8118fa94801b28392738a05403d346941738f68bea258bc41f37
+  metadata.gz: c6423964e3bf474c3f1c6c31ab4c52a14fc28a35d51d91480c537f46f0c1e5f2
+  data.tar.gz: a7ec4ad1154ceef88bdb3dd2bbd40d419d8dc3162a81a17804afdd9573d380be
 SHA512:
-  metadata.gz: 4f0936f08aa93d042241bece483742cc03d9e50ebaf27a363aec1973026d989ee3c6c6067de13f9016a46b5a227fb40ac2136746c8da25d33895f5a4312da510
-  data.tar.gz: 6cada0bbab474810b4bc12617e2d6e323034e6b088cebfa0d41352f0f1c8e522b4fc37d2e9dde77328c1ac5a7fc76e34e76b94f313313da9363049a72fb83859
+  metadata.gz: ccc4f1217740efde74e84acf26c2f73de7ac9c088575ac03d7be5408bc1c2711cd95ffd27a9b524071db95de3ce24ad3673b5a4f960a4d96b6b6c1c083c85183
+  data.tar.gz: 1f94bc2139a6397ba5a963a230496ab694f98cb1ae29e42b69017972d2268afc97c8fdde0d49ffd8d6514036199c15a5d62f67439a6790ed9d4696771ddf7b8e

data/README.md

@@ -588,6 +588,10 @@ There are four configuration parameters for circuit breakers in Semian:
   Defaults to `error_timeout` seconds if not set.
 - **error_timeout**. The amount of time in seconds until trying to query the resource
   again.
+- **exponential_backoff_error_timeout**. If set to `true`, we will progress towards error_timeout exponentially, instead of committing to it directly.
+This is useful to avoid rejecting too many requests if the dependency is not really degraded.
+- **exponential_backoff_initial_timeout**. Where to start the exponential backoff towards `error_timeout` from. Defaults to 1 second.
+- **exponential_backoff_multiplier**. The exponential multiplier to use during the exponential backoff towards the `error_timeout`. Defaults to 2.
 - **error_threshold_timeout_enabled**. If set to false it will disable
   the time window for evicting old exceptions. `error_timeout` is still used and
   will reset the circuit. Defaults to `true` if not set.
@@ -603,6 +607,77 @@ It is possible to disable Circuit Breaker with environment variable
 For more information about configuring these parameters, please read
 [this post](https://shopify.engineering/circuit-breaker-misconfigured).
 
+#### Adaptive Circuit Breaker (Experimental)
+
+Semian also includes an experimental adaptive circuit breaker that uses a [PID controller](https://en.wikipedia.org/wiki/Proportional%E2%80%93integral%E2%80%93derivative_controller)
+to dynamically adjust the rejection rate based on real-time error rates. Unlike the
+traditional circuit breaker with fixed thresholds, the adaptive circuit breaker continuously
+monitors error rates and adjusts its behavior accordingly.
+
+##### How It Works
+
+The adaptive circuit breaker has two components:
+
+1. An ideal error rate estimator that determines when the service is starting to become unhealthy
+2. A PID controller that opens the circuit fully or partially based on how bad the situation is.
+
+The ideal error rate estimator uses a "simple exponential smoother", which means it simply takes the average error rate
+that it observes as the ideal. With the following caveat:
+
+1. It ignores any data that is too high from its calculations. For example, we know that 20% error rate is an anamolous
+observation so we ignore it.
+1. It starts with an educated guess about the ideal error rate,
+and then converges down quickly if it observes a lower error rate, and slowly if it observes a higher error rate.
+1. After 30 minutes, it becomes more confident of its guess, and thus converges even slower in either directions.
+
+The PID controller uses the following equation to determine whether to open or close the circuit:
+
+```
+P = (error_rate - ideal_error_rate) - (1 - (error_rate - ideal_error_rate)) * rejection_rate
+```
+
+Or, more simply, if you define `delta_error = error_rate - ideal_error_rate` then:
+
+```
+P = delta_error - (1 - delta_error) * rejection_rate
+```
+
+In simple terms: This equation says: open more when the error rate is higher than the rejection rate,
+and less when the opposite. The multiplier of `(1 - delta_error)` is called the aggressiveness multiplier.
+It allows the circuit to open more aggressively depending on how bad the situation is.
+
+This P is fed into a typical PID equation, and is used to control the rejection rate of the circuit breaker.
+
+##### Adaptive Circuit Breaker Configuration
+
+To enable the adaptive circuit breaker, simply set **adaptive_circuit_breaker** to true.
+
+Example configuration:
+```ruby
+Semian.register(
+  :my_service,
+  adaptive_circuit_breaker: true,  # Use adaptive instead of traditional
+  bulkhead: false                   # Can be combined with bulkhead
+)
+```
+
+**Note**: When `adaptive_circuit_breaker: true` is set, traditional circuit breaker
+parameters (`error_threshold`, `error_timeout`, etc.) are ignored.
+
+
+We **_highly_** recommend just setting that configuration and not any other.
+One of the main goals of the adaptive circuit breaker is that it "just works".
+Configuring it might be difficult and not provide much value. That said, here are the configurations you can set:
+* **kp:** The contribution of P in the PID equation. Increasing it means you react more quickly to the latest data. Defaults to 1.0
+* **ki**: The contribution of the integral in the PID equation. Increasing it means adding more "memory", which is useful to ignoring noise. Defaults to 0.2
+* **kd**: The contribution of the derivative in the PID equation. Its behaviour can be complex because of our complex P equation. Defaults to 0.0
+* **integral_upper_cap**: Maximum value of the integral, prevents integral windup. Default to 10.0
+* **integral_lower_cap**: Minimum value of the integral, prevents integral windup. Default to -10.0
+* **window_size**: How many seconds of observations to take into account. Note that this window is a sliding window of 1 second sliding interval. To control the sliding interval you should set the environment variable SEMIAN_ADAPTIVE_CIRCUIT_BREAKER_SLIDING_INTERVAL (shared among all adaptive circuit breakers). window_size default to 10 seconds
+* **dead_zone_ratio**: An error percentage above the ideal_error_rate to ignore. This helps remove noise. Defaults to 0.25
+* **initial_error_rate**: The guess to start with for the ideal error rate. Defaults to 0.05 (5%)
+* **ideal_error_rate_estimator_cap_value**: The value above which we ignore observations for the ideal error rate. Defaults to 0.1 (10%)
+
 ### Bulkheading
 
 For some applications, circuit breakers are not enough. This is best illustrated

data/lib/semian/adapter.rb

@@ -45,7 +45,7 @@ module Semian
       end
     rescue ::Semian::OpenCircuitError => error
       last_error = semian_resource.circuit_breaker.last_error
-      message = "#{error.message} caused by #{last_error.message}"
+      message = "#{error.message} caused by #{last_error&.message}"
       last_error = nil unless last_error.is_a?(Exception) # Net::HTTPServerError is not an exception
       raise self.class::CircuitOpenError.new(semian_identifier, message), cause: last_error
     rescue ::Semian::BaseError => error

data/lib/semian/adaptive_circuit_breaker.rb

@@ -0,0 +1,136 @@
+# frozen_string_literal: true
+
+require_relative "circuit_breaker_behaviour"
+require_relative "pid_controller_thread"
+
+module Semian
+  # Adaptive Circuit Breaker that uses PID controller for dynamic rejection
+  class AdaptiveCircuitBreaker
+    include CircuitBreakerBehaviour
+
+    attr_reader :pid_controller, :update_thread, :sliding_interval, :pid_controller_thread, :stopped
+
+    @pid_controller_thread = nil
+
+    def initialize(name:, exceptions:, kp:, ki:, kd:, window_size:, initial_error_rate:, implementation:,
+      sliding_interval:, dead_zone_ratio:, ideal_error_rate_estimator_cap_value:, integral_upper_cap:,
+      integral_lower_cap:)
+      initialize_behaviour(name: name)
+
+      @exceptions = exceptions
+      @stopped = false
+
+      @pid_controller = implementation::PIDController.new(
+        kp: kp,
+        ki: ki,
+        kd: kd,
+        window_size: window_size,
+        implementation: implementation,
+        sliding_interval: sliding_interval,
+        initial_error_rate: initial_error_rate,
+        dead_zone_ratio: dead_zone_ratio,
+        ideal_error_rate_estimator_cap_value: ideal_error_rate_estimator_cap_value,
+        integral_upper_cap: integral_upper_cap,
+        integral_lower_cap: integral_lower_cap,
+      )
+
+      @pid_controller_thread = PIDControllerThread.instance.register_resource(self)
+    end
+
+    def acquire(resource = nil, scope: nil, adapter: nil, &block)
+      unless request_allowed?
+        mark_rejected(scope:, adapter:)
+        raise OpenCircuitError, "Rejected by adaptive circuit breaker"
+      end
+
+      result = nil
+      begin
+        result = block.call
+      rescue *@exceptions => error
+        if !error.respond_to?(:marks_semian_circuits?) || error.marks_semian_circuits?
+          mark_failed(error, scope:, adapter:)
+        end
+        raise error
+      else
+        mark_success(scope:, adapter:)
+      end
+      result
+    end
+
+    def reset(scope: nil, adapter: nil)
+      @last_error = nil
+      @pid_controller.reset
+    end
+
+    def stop
+      destroy
+    end
+
+    def destroy
+      @stopped = true
+      PIDControllerThread.instance.unregister_resource(self)
+      @pid_controller.reset
+    end
+
+    def metrics
+      @pid_controller.metrics
+    end
+
+    def open?
+      @pid_controller.rejection_rate == 1
+    end
+
+    def closed?
+      @pid_controller.rejection_rate == 0
+    end
+
+    # Compatibility with ProtectedResource - Adaptive circuit breaker does not have a half open state
+    def half_open?
+      !open? && !closed?
+    end
+
+    def mark_failed(error, scope: nil, adapter: nil)
+      @last_error = error
+      @pid_controller.record_request(:error)
+    end
+
+    def mark_success(scope: nil, adapter: nil)
+      @pid_controller.record_request(:success)
+    end
+
+    def mark_rejected(scope: nil, adapter: nil)
+      @pid_controller.record_request(:rejected)
+    end
+
+    def request_allowed?
+      !@pid_controller.should_reject?
+    end
+
+    def in_use?
+      true
+    end
+
+    def pid_controller_update
+      @pid_controller.update
+      notify_metrics_update(@pid_controller.metrics(full: false))
+    end
+
+    private
+
+    def notify_metrics_update(metrics)
+      Semian.notify(
+        :adaptive_update,
+        self,
+        nil,
+        nil,
+        rejection_rate: metrics[:rejection_rate],
+        error_rate: metrics[:error_rate],
+        ideal_error_rate: metrics[:ideal_error_rate],
+        p_value: metrics[:p_value],
+        integral: metrics[:integral],
+        derivative: metrics[:derivative],
+        previous_p_value: metrics[:previous_p_value],
+      )
+    end
+  end
+end

data/lib/semian/circuit_breaker.rb

@@ -1,33 +1,43 @@
 # frozen_string_literal: true
 
+require_relative "circuit_breaker_behaviour"
+
 module Semian
   class CircuitBreaker # :nodoc:
+    include CircuitBreakerBehaviour
     extend Forwardable
 
     def_delegators :@state, :closed?, :open?, :half_open?
 
     attr_reader(
-      :name,
       :half_open_resource_timeout,
       :error_timeout,
       :state,
-      :last_error,
       :error_threshold_timeout_enabled,
+      :exponential_backoff_error_timeout,
+      :exponential_backoff_initial_timeout,
+      :exponential_backoff_multiplier,
     )
 
     def initialize(name, exceptions:, success_threshold:, error_threshold:,
       error_timeout:, implementation:, half_open_resource_timeout: nil,
       error_threshold_timeout: nil, error_threshold_timeout_enabled: true,
-      lumping_interval: 0)
-      @name = name.to_sym
+      lumping_interval: 0, exponential_backoff_error_timeout: false,
+      exponential_backoff_initial_timeout: 1, exponential_backoff_multiplier: 2)
+      initialize_behaviour(name: name)
+
+      @exceptions = exceptions
       @success_count_threshold = success_threshold
       @error_count_threshold = error_threshold
       @error_threshold_timeout = error_threshold_timeout || error_timeout
       @error_threshold_timeout_enabled = error_threshold_timeout_enabled.nil? ? true : error_threshold_timeout_enabled
       @error_timeout = error_timeout
-      @exceptions = exceptions
       @half_open_resource_timeout = half_open_resource_timeout
       @lumping_interval = lumping_interval
+      @exponential_backoff_error_timeout = exponential_backoff_error_timeout
+      @exponential_backoff_initial_timeout = exponential_backoff_initial_timeout
+      @exponential_backoff_multiplier = exponential_backoff_multiplier
+      @current_error_timeout = exponential_backoff_error_timeout ? exponential_backoff_initial_timeout : error_timeout
 
       @errors = implementation::SlidingWindow.new(max_size: @error_count_threshold)
       @successes = implementation::Integer.new
@@ -36,8 +46,8 @@ module Semian
       reset
     end
 
-    def acquire(resource = nil, &block)
-      transition_to_half_open if transition_to_half_open?
+    def acquire(resource = nil, scope: nil, adapter: nil, &block)
+      transition_to_half_open(scope: scope, adapter: adapter) if transition_to_half_open?
 
       raise OpenCircuitError unless request_allowed?
 
@@ -46,11 +56,11 @@ module Semian
         result = maybe_with_half_open_resource_timeout(resource, &block)
       rescue *@exceptions => error
         if !error.respond_to?(:marks_semian_circuits?) || error.marks_semian_circuits?
-          mark_failed(error)
+          mark_failed(error, scope: scope, adapter: adapter)
         end
         raise error
       else
-        mark_success
+        mark_success(scope: scope, adapter: adapter)
       end
       result
     end
@@ -63,26 +73,26 @@ module Semian
       closed? || half_open? || transition_to_half_open?
     end
 
-    def mark_failed(error)
+    def mark_failed(error, scope: nil, adapter: nil)
       push_error(error)
       if closed?
-        transition_to_open if error_threshold_reached?
+        transition_to_open(scope: scope, adapter: adapter) if error_threshold_reached?
       elsif half_open?
-        transition_to_open
+        transition_to_open(scope: scope, adapter: adapter)
       end
     end
 
-    def mark_success
+    def mark_success(scope: nil, adapter: nil)
       return unless half_open?
 
       @successes.increment
-      transition_to_close if success_threshold_reached?
+      transition_to_close(scope: scope, adapter: adapter) if success_threshold_reached?
     end
 
-    def reset
+    def reset(scope: nil, adapter: nil)
       @errors.clear
       @successes.reset
-      transition_to_close
+      transition_to_close(scope: scope, adapter: adapter)
     end
 
     def destroy
@@ -97,24 +107,30 @@ module Semian
 
     private
 
-    def transition_to_close
-      notify_state_transition(:closed)
+    def transition_to_close(scope: nil, adapter: nil)
+      notify_state_transition(:closed, scope: scope, adapter: adapter)
       log_state_transition(:closed)
       @state.close!
       @errors.clear
+      # Reset exponential backoff when circuit closes
+      @current_error_timeout = @exponential_backoff_error_timeout ? @exponential_backoff_initial_timeout : @error_timeout
     end
 
-    def transition_to_open
-      notify_state_transition(:open)
+    def transition_to_open(scope: nil, adapter: nil)
+      notify_state_transition(:open, scope: scope, adapter: adapter)
       log_state_transition(:open)
       @state.open!
     end
 
-    def transition_to_half_open
-      notify_state_transition(:half_open)
+    def transition_to_half_open(scope: nil, adapter: nil)
+      notify_state_transition(:half_open, scope: scope, adapter: adapter)
       log_state_transition(:half_open)
       @state.half_open!
       @successes.reset
+      # Multiply the backoff timeout when circuit opens (up to the max error_timeout)
+      if @exponential_backoff_error_timeout && @current_error_timeout < @error_timeout
+        @current_error_timeout = [@current_error_timeout * @exponential_backoff_multiplier, @error_timeout].min
+      end
     end
 
     def success_threshold_reached?
@@ -129,7 +145,7 @@ module Semian
       last_error_time = @errors.last
       return false unless last_error_time
 
-      last_error_time + @error_timeout < Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      last_error_time + @current_error_timeout < Process.clock_gettime(Process::CLOCK_MONOTONIC)
     end
 
     def push_error(error)
@@ -153,6 +169,9 @@ module Semian
       str += " success_count_threshold=#{@success_count_threshold}"
       str += " error_count_threshold=#{@error_count_threshold}"
       str += " error_timeout=#{@error_timeout} error_last_at=\"#{@errors.last}\""
+      if @exponential_backoff_error_timeout
+        str += " current_error_timeout=#{@current_error_timeout}"
+      end
       str += " name=\"#{@name}\""
       if new_state == :open && @last_error
         str += " last_error_message=#{@last_error.message.inspect}"
@@ -161,8 +180,8 @@ module Semian
       Semian.logger.info(str)
     end
 
-    def notify_state_transition(new_state)
-      Semian.notify(:state_change, self, nil, nil, state: new_state)
+    def notify_state_transition(new_state, scope: nil, adapter: nil)
+      Semian.notify(:state_change, self, scope, adapter, state: new_state)
     end
 
     def maybe_with_half_open_resource_timeout(resource, &block)

data/lib/semian/circuit_breaker_behaviour.rb

@@ -0,0 +1,64 @@
+# frozen_string_literal: true
+
+module Semian
+  module CircuitBreakerBehaviour
+    attr_reader :name, :last_error
+    attr_accessor :exceptions
+
+    # Initialize common circuit breaker attributes
+    def initialize_behaviour(name:)
+      @name = name.to_sym
+      @last_error = nil
+    end
+
+    # Main method to execute a block with circuit breaker protection
+    def acquire(resource = nil, scope: nil, adapter: nil, &block)
+      raise NotImplementedError, "#{self.class} must implement #acquire"
+    end
+
+    # Reset the circuit breaker to its initial state
+    def reset(scope: nil, adapter: nil)
+      raise NotImplementedError, "#{self.class} must implement #reset"
+    end
+
+    # Clean up resources
+    def destroy
+      raise NotImplementedError, "#{self.class} must implement #destroy"
+    end
+
+    # Check if the circuit is open (rejecting requests)
+    def open?
+      raise NotImplementedError, "#{self.class} must implement #open?"
+    end
+
+    # Check if the circuit is closed (allowing requests)
+    def closed?
+      raise NotImplementedError, "#{self.class} must implement #closed?"
+    end
+
+    # Check if the circuit is half-open (testing if service recovered)
+    def half_open?
+      raise NotImplementedError, "#{self.class} must implement #half_open?"
+    end
+
+    # Check if requests are currently allowed
+    def request_allowed?
+      raise NotImplementedError, "#{self.class} must implement #request_allowed?"
+    end
+
+    # Mark a request as failed
+    def mark_failed(error, scope: nil, adapter: nil)
+      raise NotImplementedError, "#{self.class} must implement #mark_failed"
+    end
+
+    # Mark a request as successful
+    def mark_success(scope: nil, adapter: nil)
+      raise NotImplementedError, "#{self.class} must implement #mark_success"
+    end
+
+    # Check if the circuit breaker is actively tracking failures
+    def in_use?
+      raise NotImplementedError, "#{self.class} must implement #in_use?"
+    end
+  end
+end

data/lib/semian/configuration_validator.rb

@@ -66,6 +66,7 @@ module Semian
     def validate_circuit_breaker_configuration!
       return if ENV.key?("SEMIAN_CIRCUIT_BREAKER_DISABLED")
       return unless @configuration.fetch(:circuit_breaker, true)
+      return if @configuration[:adaptive_circuit_breaker] # Skip traditional validation if using adaptive
 
       require_keys!([:success_threshold, :error_threshold, :error_timeout], @configuration)
       validate_thresholds!
@@ -103,6 +104,7 @@ module Semian
       error_threshold = @configuration[:error_threshold]
       lumping_interval = @configuration[:lumping_interval]
       half_open_resource_timeout = @configuration[:half_open_resource_timeout]
+      exponential_backoff_error_timeout = @configuration[:exponential_backoff_error_timeout]
 
       unless error_timeout.is_a?(Numeric) && error_timeout > 0
         err = "error_timeout must be a positive number, got #{error_timeout}"
@@ -174,6 +176,56 @@ module Semian
 
         raise_or_log_validation_required!(err)
       end
+
+      unless exponential_backoff_error_timeout.nil? || [true, false].include?(exponential_backoff_error_timeout)
+        err = "exponential_backoff_error_timeout must be a boolean, got #{exponential_backoff_error_timeout}"
+        err += hint_format("Use true to enable exponential backoff for error timeout. Use false to disable.")
+
+        raise_or_log_validation_required!(err)
+      end
+
+      # Validate exponential backoff initial timeout
+      exponential_backoff_initial_timeout = @configuration[:exponential_backoff_initial_timeout]
+      unless exponential_backoff_initial_timeout.nil? || (exponential_backoff_initial_timeout.is_a?(Numeric) && exponential_backoff_initial_timeout > 0)
+        err = "exponential_backoff_initial_timeout must be a positive number, got #{exponential_backoff_initial_timeout}"
+        err += hint_format("This is the initial timeout when exponential backoff is enabled. Must be less than error_timeout.")
+
+        raise_or_log_validation_required!(err)
+      end
+
+      # Validate exponential backoff multiplier
+      exponential_backoff_multiplier = @configuration[:exponential_backoff_multiplier]
+      unless exponential_backoff_multiplier.nil? || (exponential_backoff_multiplier.is_a?(Numeric) && exponential_backoff_multiplier > 1)
+        err = "exponential_backoff_multiplier must be a number greater than 1, got #{exponential_backoff_multiplier}"
+        err += hint_format("This is the factor by which the timeout increases on each subsequent opening. Common values are 2 (double) or 1.5.")
+
+        raise_or_log_validation_required!(err)
+      end
+
+      # Ensure exponential backoff parameters are only provided when exponential_backoff_error_timeout is true
+      unless exponential_backoff_error_timeout
+        if exponential_backoff_initial_timeout
+          err = "exponential_backoff_initial_timeout can only be specified when exponential_backoff_error_timeout is true"
+          err += hint_format("Set exponential_backoff_error_timeout: true to use exponential backoff features.")
+
+          raise_or_log_validation_required!(err)
+        end
+
+        if exponential_backoff_multiplier
+          err = "exponential_backoff_multiplier can only be specified when exponential_backoff_error_timeout is true"
+          err += hint_format("Set exponential_backoff_error_timeout: true to use exponential backoff features.")
+
+          raise_or_log_validation_required!(err)
+        end
+      end
+
+      # Ensure initial timeout is less than error_timeout when using exponential backoff
+      if exponential_backoff_error_timeout && exponential_backoff_initial_timeout && exponential_backoff_initial_timeout >= error_timeout
+        err = "exponential_backoff_initial_timeout (#{exponential_backoff_initial_timeout}) must be less than error_timeout (#{error_timeout})"
+        err += hint_format("The initial timeout should be smaller than the maximum timeout for exponential backoff to be effective.")
+
+        raise_or_log_validation_required!(err)
+      end
     end
 
     def validate_quota!(quota)