semian 0.16.0 → 0.17.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 03aa7b6a715e0990279468f71a5bacbd03b94f983c16083570a09dc46c69d325
-  data.tar.gz: 1b53f85622bfa3868585ea8836df867a37da5b4142b769ad8b968d9865c001c2
+  metadata.gz: '0461852ea5a5ffbacaf9a20062213609066e3ef8ade9a93a980d2af73a0feb39'
+  data.tar.gz: 8e21023912cd7c71cc74c0eb6a384771018f5fa3f53c212a62c23446801573e2
 SHA512:
-  metadata.gz: f7f481a7a77c6d91574500e2a636eb6bd9721e61be5c8740265f58ac90895525ed9b34750b7dbb23d97360a3449a7a990994d870c8a9730e8d06350afda7c7de
-  data.tar.gz: 216189dea6814080db529843164dd02235541127ce9058ff803cc3843842220e87340b45a73109a2004d6205194b166b0cbcbf55e46601a5bf5f00c75cb5678c
+  metadata.gz: 0bd910ad2f7891d4dc2660135adea394db252afa7f53f8c707a55fa569cbf481514af485fe9056d5959e98faffd8bd20990ac9f7b6abe0b0f453ca29d3ad3e9f
+  data.tar.gz: a4149e3b4021fe47be636e28bbd5ece3eb4246d65bbc95b07cc50e1b45a328980e47c32d15e81a52e2516fe01b2622fedf31d5aac8bcedbee7e3bdb73a1a0c8c

data/README.md

@@ -73,6 +73,7 @@ version is the version of the public gem with the same name:
 * [`semian/mysql2`][mysql-semian-adapter] (~> 0.3.16)
 * [`semian/redis`][redis-semian-adapter] (~> 3.2.1)
 * [`semian/net_http`][nethttp-semian-adapter]
+* [`semian-postgres`][postgres-semian-adapter]
 
 ### Creating Adapters
 
@@ -110,7 +111,7 @@ There are some global configuration options that can be set for Semian:
 # Note: Setting this to 0 enables aggressive garbage collection.
 Semian.maximum_lru_size = 0
 
-# Minimum time a resource should be resident in the LRU cache (default: 300s)
+# Minimum time in seconds a resource should be resident in the LRU cache (default: 300s)
 Semian.minimum_lru_time = 60
 ```
 
@@ -418,9 +419,11 @@ response time. This is the problem Semian solves by failing fast.
 
 ## How does Semian work?
 
-Semian consists of two parts: **Circuit breaker** and **Bulkheading**. To understand
-Semian, and especially how to configure it, we must understand these patterns
-and their implementation.
+Semian consists of two parts: **Circuit Breaker** and **Bulkheading**.
+To understand Semian, and especially how to configure it,
+we must understand these patterns and their implementation.
+
+Disable Semian via environment variable `SEMIAN_DISABLED=1`.
 
 ### Circuit Breaker
 
@@ -451,18 +454,28 @@ all workers on a server.
 
 There are four configuration parameters for circuit breakers in Semian:
 
-* **error_threshold**. The amount of errors a worker encounters within error_threshold_timeout amount of time before
-  opening the circuit, that is to start rejecting requests instantly.
-* **error_threshold_timeout**. The amount of time in seconds that error_threshold errors must occur to open the circuit. Defaults to error_timeout seconds if not set.
+* **circuit_breaker**. Enable or Disable Circuit Breaker. Defaults to `true` if not set.
+* **error_threshold**. The amount of errors a worker encounters within `error_threshold_timeout`
+  amount of time before opening the circuit,
+  that is to start rejecting requests instantly.
+* **error_threshold_timeout**. The amount of time in seconds that `error_threshold`
+  errors must occur to open the circuit.
+  Defaults to `error_timeout` seconds if not set.
 * **error_timeout**. The amount of time in seconds until trying to query the resource
   again.
-* **error_threshold_timeout_enabled**. If set to false it will disable the time window for evicting old exceptions. `error_timeout` is still used and will reset
-  the circuit. Defaults to `true` if not set.
+* **error_threshold_timeout_enabled**. If set to false it will disable
+  the time window for evicting old exceptions. `error_timeout` is still used and
+  will reset the circuit. Defaults to `true` if not set.
 * **success_threshold**. The amount of successes on the circuit until closing it
   again, that is to start accepting all requests to the circuit.
-* **half_open_resource_timeout**. Timeout for the resource in seconds when the circuit is half-open (supported for MySQL, Net::HTTP and Redis).
+* **half_open_resource_timeout**. Timeout for the resource in seconds when
+  the circuit is half-open (supported for MySQL, Net::HTTP and Redis).
+
+It is possible to disable Circuit Breaker with environment variable
+`SEMIAN_CIRCUIT_BREAKER_DISABLED=1`.
 
-For more information about configuring these parameters, please read [this post](https://engineering.shopify.com/blogs/engineering/circuit-breaker-misconfigured).
+For more information about configuring these parameters, please read
+[this post](https://engineering.shopify.com/blogs/engineering/circuit-breaker-misconfigured).
 
 ### Bulkheading
 
@@ -515,11 +528,15 @@ still experimenting with ways to figure out optimal ticket numbers. Generally
 something below half the number of workers on the server for endpoints that are
 queried frequently has worked well for us.
 
+* **bulkhead**. Enable or Disable Bulkhead. Defaults to `true` if not set.
 * **tickets**. Number of workers that can concurrently access a resource.
 * **timeout**. Time to wait in seconds to acquire a ticket if there are no tickets left.
   We recommend this to be `0` unless you have very few workers running (i.e.
   less than ~5).
 
+It is possible to disable Bulkhead with environment variable
+`SEMIAN_BULKHEAD_DISABLED=1`.
+
 Note that there are system-wide limitations on how many tickets can be allocated
 on a system. `cat /proc/sys/kernel/sem` will tell you.
 
@@ -549,7 +566,6 @@ ipcs -si $(ipcs -s | grep 0x48af51ea | awk '{print $2}')
 Which should output something like:
 
 ```
-
 Semaphore Array semid=5570729
 uid=8192         gid=8192        cuid=8192       cgid=8192
 mode=0660, access_perms=0660
@@ -841,6 +857,7 @@ $ bundle install
 [release-it]: https://pragprog.com/titles/mnee2/release-it-second-edition/
 [shopify]: http://www.shopify.com/
 [mysql-semian-adapter]: lib/semian/mysql2.rb
+[postgres-semian-adapter]: https://github.com/mschoenlaub/semian-postgres
 [redis-semian-adapter]: lib/semian/redis.rb
 [semian-adapter]: lib/semian/adapter.rb
 [nethttp-semian-adapter]: lib/semian/net_http.rb

data/ext/semian/resource.c

@@ -359,13 +359,13 @@ ms_to_timespec(long ms, struct timespec *ts)
   ts->tv_nsec = (ms % 1000) * 1000000;
 }
 
-static inline void
+static void
 semian_resource_mark(void *ptr)
 {
   /* noop */
 }
 
-static inline void
+static void
 semian_resource_free(void *ptr)
 {
   semian_resource_t *res = (semian_resource_t *) ptr;
@@ -376,7 +376,7 @@ semian_resource_free(void *ptr)
   xfree(res);
 }
 
-static inline size_t
+static size_t
 semian_resource_memsize(const void *ptr)
 {
   return sizeof(semian_resource_t);
@@ -390,5 +390,5 @@ semian_resource_type = {
     semian_resource_free,
     semian_resource_memsize
   },
-  NULL, NULL, RUBY_TYPED_FREE_IMMEDIATELY
+  NULL, NULL, RUBY_TYPED_FREE_IMMEDIATELY | RUBY_TYPED_WB_PROTECTED
 };

data/ext/semian/sysv_semaphores.c

@@ -269,3 +269,19 @@ diff_timespec_ms(struct timespec *end, struct timespec *begin)
   long begin_ms = (begin->tv_sec * 1e3) + (begin->tv_nsec / 1e6);
   return end_ms - begin_ms;
 }
+
+#ifdef DEBUG
+VALUE
+print_sem_vals_without_rescue(VALUE v_sem_id)
+{
+  int sem_id = NUM2INT(v_sem_id);
+  printf("[pid=%d][semian] semaphore values lock: %d, tickets: %d configured: %d, registered workers: %d\n",
+   getpid(),
+   get_sem_val(sem_id, SI_SEM_LOCK),
+   get_sem_val(sem_id, SI_SEM_TICKETS),
+   get_sem_val(sem_id, SI_SEM_CONFIGURED_TICKETS),
+   get_sem_val(sem_id, SI_SEM_REGISTERED_WORKERS)
+  );
+  return (VALUE)0;
+}
+#endif

data/ext/semian/sysv_semaphores.h

@@ -110,16 +110,14 @@ void *
 acquire_semaphore_without_gvl(void *p);
 
 #ifdef DEBUG
+VALUE
+print_sem_vals_without_rescue(VALUE v_sem_id);
+
 static inline void
 print_sem_vals(int sem_id)
 {
-  printf("[pid=%d][semian] semaphore values lock: %d, tickets: %d configured: %d, registered workers: %d\n",
-   getpid(),
-   get_sem_val(sem_id, SI_SEM_LOCK),
-   get_sem_val(sem_id, SI_SEM_TICKETS),
-   get_sem_val(sem_id, SI_SEM_CONFIGURED_TICKETS),
-   get_sem_val(sem_id, SI_SEM_REGISTERED_WORKERS)
-  );
+  int state;
+  rb_protect(print_sem_vals_without_rescue, INT2NUM(sem_id), &state);
 }
 #endif
 

data/lib/semian/circuit_breaker.rb

@@ -36,8 +36,6 @@ module Semian
     end
 
     def acquire(resource = nil, &block)
-      return yield if disabled?
-
       transition_to_half_open if transition_to_half_open?
 
       raise OpenCircuitError unless request_allowed?
@@ -66,7 +64,7 @@ module Semian
 
     def mark_failed(error)
       push_error(error)
-      push_time(@errors)
+      push_time
       if closed?
         transition_to_open if error_threshold_reached?
       elsif half_open?
@@ -131,19 +129,20 @@ module Semian
       last_error_time = @errors.last
       return false unless last_error_time
 
-      Time.at(last_error_time) + @error_timeout < Time.now
+      last_error_time + @error_timeout < Process.clock_gettime(Process::CLOCK_MONOTONIC)
     end
 
     def push_error(error)
       @last_error = error
     end
 
-    def push_time(window, time: Time.now)
+    def push_time
+      time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
       if error_threshold_timeout_enabled
-        window.reject! { |err_time| err_time + @error_threshold_timeout < time.to_i }
+        @errors.reject! { |err_time| err_time + @error_threshold_timeout < time }
       end
 
-      window << time.to_i
+      @errors << time
     end
 
     def log_state_transition(new_state)
@@ -151,7 +150,8 @@ module Semian
 
       str = "[#{self.class.name}] State transition from #{@state.value} to #{new_state}."
       str += " success_count=#{@successes.value} error_count=#{@errors.size}"
-      str += " success_count_threshold=#{@success_count_threshold} error_count_threshold=#{@error_count_threshold}"
+      str += " success_count_threshold=#{@success_count_threshold}"
+      str += " error_count_threshold=#{@error_count_threshold}"
       str += " error_timeout=#{@error_timeout} error_last_at=\"#{@errors.last}\""
       str += " name=\"#{@name}\""
       if new_state == :open && @last_error
@@ -165,21 +165,14 @@ module Semian
       Semian.notify(:state_change, self, nil, nil, state: new_state)
     end
 
-    def disabled?
-      ENV["SEMIAN_CIRCUIT_BREAKER_DISABLED"] || ENV["SEMIAN_DISABLED"]
-    end
-
     def maybe_with_half_open_resource_timeout(resource, &block)
-      result =
-        if half_open? && @half_open_resource_timeout && resource.respond_to?(:with_resource_timeout)
-          resource.with_resource_timeout(@half_open_resource_timeout) do
-            block.call
-          end
-        else
+      if half_open? && @half_open_resource_timeout && resource.respond_to?(:with_resource_timeout)
+        resource.with_resource_timeout(@half_open_resource_timeout) do
           block.call
         end
-
-      result
+      else
+        block.call
+      end
     end
   end
 end

data/lib/semian/grpc.rb

@@ -117,7 +117,7 @@ module Semian
       execute = operation.singleton_method(:execute)
       operation.instance_variable_set(:@semian, self)
       operation.define_singleton_method(:execute) do
-        @semian.send(:acquire_semian_resource, **{ adapter: :grpc, scope: scope }) { execute.call }
+        @semian.send(:acquire_semian_resource, adapter: :grpc, scope: scope) { execute.call }
       end
     end
   end

data/lib/semian/lru_hash.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require "thread"
+
 class LRUHash
   # This LRU (Least Recently Used) hash will allow
   # the cleaning of resources as time goes on.
@@ -41,7 +43,7 @@ class LRUHash
   #
   # Arguments:
   #   +max_size+ The maximum size of the table
-  #   +min_time+ The minimum time a resource can live in the cache
+  #   +min_time+ The minimum time in seconds a resource can live in the cache
   #
   # Note:
   #   The +min_time+ is a stronger guarantee than +max_size+. That is, if there are
@@ -57,9 +59,9 @@ class LRUHash
     @table = {}
     @lock =
       if Semian.thread_safe?
-        Mutex.new
+        ::Thread::Mutex.new
       else
-        NoopMutex.new
+        ::LRUHash::NoopMutex.new
       end
   end
 
@@ -83,7 +85,7 @@ class LRUHash
     @lock.synchronize do
       @table.delete(key)
       @table[key] = resource
-      resource.updated_at = Time.now
+      resource.updated_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
     end
     clear_unused_resources if @table.length > @max_size
   end
@@ -98,7 +100,7 @@ class LRUHash
       found = @table.delete(key)
       if found
         @table[key] = found
-        found.updated_at = Time.now
+        found.updated_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
       end
       found
     end
@@ -130,9 +132,10 @@ class LRUHash
     timer_start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
 
     ran = try_synchronize do
-      # Clears resources that have not been used in the last 5 minutes.
+      # Clears resources that have not been used
+      # in the last 5 minutes (default value of Semian.minimum_lru_time).
 
-      stop_time = Time.now - @min_time # Don't process resources updated after this time
+      stop_time = Process.clock_gettime(Process::CLOCK_MONOTONIC) - @min_time
       @table.each do |_, resource|
         payload[:examined] += 1
 

data/lib/semian/platform.rb

@@ -13,6 +13,6 @@ module Semian
   end
 
   def disabled?
-    ENV["SEMIAN_SEMAPHORES_DISABLED"] || ENV["SEMIAN_DISABLED"]
+    ENV.key?("SEMIAN_SEMAPHORES_DISABLED") || ENV.key?("SEMIAN_DISABLED")
   end
 end

data/lib/semian/protected_resource.rb

@@ -5,8 +5,14 @@ module Semian
     extend Forwardable
 
     def_delegators :@bulkhead, :destroy, :count, :semid, :tickets, :registered_workers
-    def_delegators :@circuit_breaker, :reset, :mark_failed, :mark_success, :request_allowed?,
-      :open?, :closed?, :half_open?
+    def_delegators :@circuit_breaker,
+      :reset,
+      :mark_failed,
+      :mark_success,
+      :request_allowed?,
+      :open?,
+      :closed?,
+      :half_open?
 
     attr_reader :bulkhead, :circuit_breaker, :name
     attr_accessor :updated_at
@@ -15,7 +21,7 @@ module Semian
       @name = name
       @bulkhead = bulkhead
       @circuit_breaker = circuit_breaker
-      @updated_at = Time.now
+      @updated_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
     end
 
     def destroy

data/lib/semian/unprotected_resource.rb

@@ -9,7 +9,7 @@ module Semian
 
     def initialize(name)
       @name = name
-      @updated_at = Time.now
+      @updated_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
     end
 
     def registered_workers

data/lib/semian/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Semian
-  VERSION = "0.16.0"
+  VERSION = "0.17.0"
 end

data/lib/semian.rb

@@ -104,7 +104,7 @@ module Semian
   attr_accessor :maximum_lru_size, :minimum_lru_time, :default_permissions, :namespace
 
   self.maximum_lru_size = 500
-  self.minimum_lru_time = 300
+  self.minimum_lru_time = 300 # 300 seconds / 5 minutes
   self.default_permissions = 0660
 
   def issue_disabled_semaphores_warning
@@ -122,11 +122,16 @@ module Semian
     attr_accessor :semian_identifier
 
     def to_s
+      message = super
       if @semian_identifier
-        "[#{@semian_identifier}] #{super}"
-      else
-        super
+        prefix = "[#{@semian_identifier}] "
+        # When an error is created from another error's message it might
+        # already have a semian identifier in their message
+        unless message.start_with?(prefix)
+          message = "#{prefix}#{message}"
+        end
       end
+      message
     end
   end
 
@@ -156,11 +161,17 @@ module Semian
   #
   # +timeout+: Default timeout in seconds. Default 0. (bulkhead)
   #
+  # +error_timeout+: The duration in seconds since the last error after which the error count is reset to 0.
+  # (circuit breaker required)
+  #
   # +error_threshold+: The amount of errors that must happen within error_timeout amount of time to open
   # the circuit. (circuit breaker required)
   #
-  # +error_timeout+: The duration in seconds since the last error after which the error count is reset to 0.
-  # (circuit breaker required)
+  # +error_threshold_timeout+: The duration in seconds to examine number of errors to compare with error_threshold.
+  # Default same as error_timeout. (circuit breaker)
+  #
+  # +error_threshold_timeout_enabled+: flag to enable/disable filter time window based error eviction
+  # (error_threshold_timeout). Default true. (circuit breaker)
   #
   # +success_threshold+: The number of consecutive success after which an half-open circuit will be fully closed.
   # (circuit breaker required)
@@ -170,6 +181,8 @@ module Semian
   #
   # Returns the registered resource.
   def register(name, **options)
+    return UnprotectedResource.new(name) if ENV.key?("SEMIAN_DISABLED")
+
     circuit_breaker = create_circuit_breaker(name, **options)
     bulkhead = create_bulkhead(name, **options)
 
@@ -264,8 +277,8 @@ module Semian
   private
 
   def create_circuit_breaker(name, **options)
-    circuit_breaker = options.fetch(:circuit_breaker, true)
-    return unless circuit_breaker
+    return if ENV.key?("SEMIAN_CIRCUIT_BREAKER_DISABLED")
+    return unless options.fetch(:circuit_breaker, true)
 
     require_keys!([:success_threshold, :error_threshold, :error_timeout], options)
 
@@ -304,16 +317,18 @@ module Semian
   end
 
   def create_bulkhead(name, **options)
-    bulkhead = options.fetch(:bulkhead, true)
-    return unless bulkhead
+    return if ENV.key?("SEMIAN_BULKHEAD_DISABLED")
+    return unless options.fetch(:bulkhead, true)
 
     permissions = options[:permissions] || default_permissions
     timeout = options[:timeout] || 0
-    ::Semian::Resource.new(name,
+    ::Semian::Resource.new(
+      name,
       tickets: options[:tickets],
       quota: options[:quota],
       permissions: permissions,
-      timeout: timeout)
+      timeout: timeout,
+    )
   end
 
   def require_keys!(required, options)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: semian
 version: !ruby/object:Gem::Version
-  version: 0.16.0
+  version: 0.17.0
 platform: ruby
 authors:
 - Scott Francis
@@ -10,7 +10,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2022-10-05 00:00:00.000000000 Z
+date: 2023-02-08 00:00:00.000000000 Z
 dependencies: []
 description: |2
       A Ruby C extention that is used to control access to shared resources