eventq 4.1.0 → 4.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 830a90e777c5896d7d7d7f7d02bf400ec53b2e46d37235956f489719742724f1
-  data.tar.gz: 9d874da06b6190367a382ddac2cfc0dafe937bb622136474bf2430d32805af34
+  metadata.gz: c0bbacb646ab33aa08c10738ceeba304c17c0c5f80024b0b8c5e22cb4a11e41a
+  data.tar.gz: d05f206a2231a46750ab172426725dc7a7347acfd084c6371ea888955182042c
 SHA512:
-  metadata.gz: 5ef60aecf44e22fe3ed07efd0e7d86205e3191c75c020e9a722715334598d284445fee1548cab1873868128f28556bc30fe9f5810628cb388d17add6c20bc4d9
-  data.tar.gz: 2157ca99b534f9d34156f444da0343814bc7f415649b7aeda6ae9d6c4e97d3b2ce131549bff9ebf57973a6d859fa95d03b89c2dbbd22d18dc3c4c84c04c8a370
+  metadata.gz: 8daf3d93b434bae5ca09c3d2c51e9058df10f3355efbfcf116bd44eb52fd4769209f90a5638204de5e9eff93b418f07596bd746d33db5510a5c5a3471fea1f7f
+  data.tar.gz: 2cf23e913f766e2f2dff17cb1ea47dfbb5b6c1f476728113225cf50801ade03dfcacdfec810a44a8ac2c3c50d84b3af2fc91bbbda1dda7ea5ea2eb324040e7c2

data/README.md

@@ -69,17 +69,72 @@ A subscription queue should be defined to receive any events raised for the subs
 **Example**
 
 ```ruby
-# Create a queue that allows retries and accepts a maximum of 3 retries with a 20 second delay between retries.
+# Create a queue that allows retries and accepts a maximum of 5 retries with a 20 second delay between retries.
 class DataChangeAddressQueue < Queue
   def initialize
     @name = 'Data.Change.Address'
     @allow_retry = true
     @retry_delay = 20_000
-    @max_retry_attempts = 3
+    @max_retry_attempts = 5
   end
 end
 ```
 
+**Retry Strategies**
+
+In distributed systems, it is expected for some events to fail.
+Thankfully, those events can be put "on hold" and will be processed again after a given waiting time.
+The attributes affecting your retry strategy the most are:
+* `retry_delay` (base duration that events are waiting before being reprocessed)
+* `max_receive_count` and `max_retry_attempts` (limiting how often an event can be seen / processed)
+* `allow_retry`, `allow_retry_back_off` and `allow_exponential_back_off` (defining if retries are allowed and how duration between retries should be calculated)
+
+If only `retry_delay` is set to `true`, while `allow_retry_back_off` and `allow_exponential_back_off` remain `false`, the duration between retries will be `retry_delay` each time ("fixed back off").
+So there is a fixed duration between events, like in the example for `DataChangeAddressQueue` above.
+With the configuration of that class, the event will be retried 5 times, with at least 20 seconds between retries.
+Therefore we can calculate that the final retry will have happened after `retry_duration * max_retry_attempts`, which results in 100 seconds here.
+
+If also `allow_retry_back_off` is set to `true`, the duration between retries will scale with the number of retries ("incremental back off").
+So the first retry will happen after `retry_duration`, the second after `2 * retry_duration`, the third after `3 * retry_duration` and so on.
+So the retries will be spread out further apart each time.
+The last retry will be processed after `(max_retry_attempts * (max_retry_attempts + 1))/2 * retry_duration`.
+So in the example above, it would result in 300 seconds until the last retry.
+
+If also `allow_exponential_back_off` is set to `true`, the duration between retries will double each time ("exponential back off").
+So the first retry will happen after `retry_duration`, the second after `2 * retry_duration`, the third after `4 * retry_duration` and so on.
+The last retry will be processed after `(2^max_retry_attempts - 1) * retry_duration`.
+So in the example above, it would result in 620 seconds until the last retry.
+
+You can run experiments on your retry configuration using [plot_visibility_timeout.rb](https://github.com/Sage/eventq/blob/master/utilities/plot_visibility_timeout.rb), which will output the retry duration on each retry given your settings.
+
+
+![Graph comparing back off strategies](images/back-off-strategy.png)
+
+**Randomness**
+
+By default, there will be no randomness in your retry strategy.
+However, that means that with a fixed 20 second back off, many events overloading your service will all come back after exactly 20 seconds, overloading it again.
+Therefore it can be useful to introduce randomness to your retry duration, so the events that initially hit the queue at the same time, are spread out when scheduling them for retry.
+
+The attribute `retry_jitter_ratio` allows you to configure how much randomness ("jitter") is allowed for the retry duration.
+Let's assume we have a `retry_duration = 20_000` (20 seconds).
+Then the `retry_jitter_ratio` would have the following effect:
+* 0 means no randomness, so retry duration of 20 seconds is used every time
+* 20 means 20% randomness, so the duration will be randomly chosen between 80% to 100% of the value, i.e. between 16 to 20 seconds
+* 50 means 50% randomness, i.e. between 10 to 20 seconds
+* 80 means 80% randomness, i.e. between 4 to 20 seconds
+* 100 means 100% randomness, i.e. between 0 to 20 seconds
+
+In the graphs below you can see how adding 50% randomness can help avoid overloading the service.
+In the first graph ("Fixed Retry Duration"), all failures are hitting the queue again after exactly 20 seconds.
+This leads to only a couple of events to succeed, as the others fail due to too many concurrent requests running into locks etc.
+However, in the second graph ("Randomised Retry Duration"), the events are randomnly spread out over the next 10 to 20 seconds.
+This means less events hit the service concurrently, allowing it to succesfully process more events and processing all of the events in a shorter duration, reducing the overall load on the service.
+
+![Graph showing that events overload the service repeatedly with fixed retry duration](images/fixed-retry-duration.png)
+
+![Graph showing that events are spread out on retries when randomising retry duration](images/randomised-retry-duration.png)
+
 ### SubscriptionManager
 
 In order to receive events within a subscription queue it must subscribe to the type of the event it should receive.
@@ -323,10 +378,11 @@ This method is called to verify connection to an event_type (topic/exchange).
 
 ## Development
 
-After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
-
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in  the file, `EVENTQ_VERSION`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+### Setup
 
+After checking out the repo, run `bin/setup` to install dependencies.
+You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+To install this gem onto your local machine, run `bundle exec rake install`.
 
 ### Preparing the Docker images
 
@@ -352,6 +408,14 @@ You can run the specs that don't depend on an AWS account with:
 
     $ ./script/test.sh --tag ~integration
 
+### Release new version
+
+To release a new version, first update the version number in  the file [`EVENTQ_VERSION`](https://github.com/Sage/eventq/blob/master/EVENTQ_VERSION).
+With that change merged to `master`, just [draft a new release](https://github.com/Sage/eventq/releases/new) with the same version you specified in `EVENTQ_VERSION`.
+Use "Generate Release Notes" to generate details for this release.
+
+This will create a git tag for the version and triggers the GitHub [Workflow to publish the new gem](https://github.com/Sage/eventq/actions/workflows/publish.yml) (defined in [publish.yml](https://github.com/Sage/eventq/blob/master/.github/workflows/publish.yml)) to [rubygems.org](https://rubygems.org).
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/sage/eventq. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.

data/lib/eventq/eventq_base/nonce_manager.rb

@@ -1,10 +1,24 @@
 module EventQ
+  class NonceManagerNotConfiguredError < StandardError; end
+
   class NonceManager
 
-    def self.configure(server:,timeout:10000,lifespan:3600)
+    def self.configure(server:,timeout:10000,lifespan:3600, pool_size: 5, pool_timeout: 5)
       @server_url = server
       @timeout = timeout
       @lifespan = lifespan
+      @pool_size = pool_size
+      @pool_timeout = pool_timeout
+
+      @redis_pool = begin
+        require 'connection_pool'
+        require 'redis'
+
+        ConnectionPool.new(size: @pool_size, timeout: @pool_timeout) do
+          Redis.new(url: @server_url)
+        end
+      end
+      @configured = true
     end
 
     def self.server_url
@@ -19,39 +33,77 @@ module EventQ
       @lifespan
     end
 
-    def self.is_allowed?(nonce)
-      if @server_url == nil
-        return true
+    def self.pool_size
+      @pool_size
+    end
+
+    def self.pool_timeout
+      @pool_timeout
+    end
+
+    def self.lock(nonce)
+      # act as if successfully locked if not nonce manager configured - makes it a no-op
+      return true if !configured?
+
+      successfully_locked = false
+      with_redis_connection do |conn|
+        successfully_locked = conn.set(nonce, 1, ex: lifespan, nx: true)
       end
 
-      require 'redlock'
-      lock = Redlock::Client.new([ @server_url ]).lock(nonce, @timeout)
-      if lock == false
+      if !successfully_locked
         EventQ.log(:info, "[#{self.class}] - Message has already been processed: #{nonce}")
-        return false
       end
 
-      return true
+      successfully_locked
     end
 
+    # if the message was successfully procesed, lock for another lifespan length
+    # so it isn't reprocessed
     def self.complete(nonce)
-      if @server_url != nil
-        Redis.new(url: @server_url).expire(nonce, @lifespan)
+      return true if !configured?
+
+      with_redis_connection do |conn|
+        conn.expire(nonce, lifespan)
       end
-      return true
+
+      true
     end
 
+    # if it failed, unlock immediately so that retries can kick in
     def self.failed(nonce)
-      if @server_url != nil
-        Redis.new(url: @server_url).del(nonce)
+      return true if !configured?
+
+      with_redis_connection do |conn|
+        conn.del(nonce)
       end
-      return true
+
+      true
     end
 
     def self.reset
       @server_url = nil
       @timeout = nil
       @lifespan = nil
+      @pool_size = nil
+      @pool_timeout = nil
+      @configured = false
+      @redis_pool.reload(&:close)
+    end
+
+    def self.configured?
+      @configured == true
+    end
+
+    private
+
+    def self.with_redis_connection
+      if !configured?
+        raise NonceManagerNotConfiguredError, 'Unable to checkout redis connection from pool, nonce manager has not been configured. Call .configure on NonceManager.'
+      end
+
+      @redis_pool.with do |conn|
+        yield conn
+      end
     end
   end
 end

data/lib/eventq/queue_worker.rb

@@ -127,7 +127,7 @@ module EventQ
 
       EventQ.logger.debug("[#{self.class}] - Message received. Id: #{message.id}. Retry Attempts: #{retry_attempts}")
 
-      if (!EventQ::NonceManager.is_allowed?(message.id))
+      if (!EventQ::NonceManager.lock(message.id))
         EventQ.logger.warn("[#{self.class}] - Duplicate Message received. Id: #{message.id}. Ignoring message.")
         status = :duplicate
         return status, message_args

data/lib/eventq.rb

@@ -1,7 +1,6 @@
 # frozen_string_literal: true
 
 require 'securerandom'
-require 'redlock'
 require 'class_kit'
 require 'hash_kit'
 require 'oj'

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: eventq
 version: !ruby/object:Gem::Version
-  version: 4.1.0
+  version: 4.2.0
 platform: ruby
 authors:
 - SageOne
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-01-12 00:00:00.000000000 Z
+date: 2025-01-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -235,7 +235,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
-  name: redlock
+  name: connection_pool
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -309,7 +309,7 @@ homepage: https://github.com/sage/eventq
 licenses:
 - MIT
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -324,8 +324,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.3.5
-signing_key: 
+rubygems_version: 3.4.20
+signing_key:
 specification_version: 4
 summary: EventQ is a pub/sub system that uses async notifications and message queues
 test_files: []