cuniculus 0.1.0 → 0.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ca0cc8cbcc9cc9607488d5c7c88903548fe4106799e6517e82b2efb5824a9971
-  data.tar.gz: 6a0589c882ae8de39a0192e2e5b54c537a709fbe6ca496ce3b999d3a03ff6da1
+  metadata.gz: 836705af66bf5ceb94715ce41b70906928611e4b37eaee3e343e4ee8d6a52256
+  data.tar.gz: da7c6168ea3aeddd9896c956f9cb89b5beeda5ded1aa7d663a6e8662eae2100d
 SHA512:
-  metadata.gz: 73fc5ea97169bdbdf5a72b95dec76e2b529a13c6b308a3c3c9c947d891bc3845d48ad5b4344820d0f1fed8cbed8b0bb1ffac3a4c3ea043ca0cb3c1737eb8014d
-  data.tar.gz: 7dbdd0c30c1dc808457fc408506aade75b65de83f5318799f8b7a5aab4c0891ed71fbb927e58eaa70b099442790b1c1d20b06d34d12a53967def89052f068887
+  metadata.gz: 69ae000142f4d761c10a37e37df70bbc257f28b602df19d7aeb9508df844d1a0165e22502ec8c9a8b6f195d4db0220faf0c7be8900489c12a1a1460ec7220fa7
+  data.tar.gz: 516789a202f6000f4145dfc218f60645437ee9476f129670dd3d464c38084399e97b930b022efee40906519752243caccb526e6d4d5e3714bb268a776de6ab75

data/README.md

@@ -1,36 +1,7 @@
-# Cuniculus
+![cuniculus_logo](https://user-images.githubusercontent.com/4718658/115109242-de53f900-9f74-11eb-8a46-b93d6f484370.png)
 
-Ruby job queue backed by RabbitMQ. The word _cuniculus_ comes from the scientific name of the European rabbit (Oryctolagus cuniculus).
-
-## Benchmarks
-
-The following measurements were performed with the `bin/run_benchmarks` utility, with different command parameters. Run it with `-h` to see its usage.
-
-To simulate network latency, [Toxiproxy](https://github.com/Shopify/toxiproxy) was used. It needs to be started with `toxiproxy-server` before running the benchmarks.
 
-Network latency (_ms_) | Prefetch count       | Throughput (_jobs/s_) | Average latency (_ms_)
-----------------------:|---------------------:|----------------------:|----------------------:
-1                      | 65535 (max. allowed) | 10225                 | 2
-10                     | 65535 (max. allowed) | 9990                  | 13
-1                      |                   50 | 8051                  | 2
-10                     |                   50 | 2500                  | 13
-100                    |                   50 | 481                   | 103
-1                      |                   25 | 7824                  | 2
-10                     |                   25 | 1824                  | 13
-50                     |                   25 | 469                   | 53
-1                      |         10 (default) | 5266                  | 2
-10                     |         10 (default) | 807                   | 13
-1                      |                    1 | 481                   | 2
-10                     |                    1 | 81                    | 13
-
-Additional benchmark parameters:
-- throughput was measured by consuming 100k jobs;
-- job latency was averaged over 200 samples;
-- Ruby 2.7.2 was used.
-
-Several remarks can be made:
-- Higher prefetch counts lead to higher throughput, but there are downsides of having it too high; see [this reference](https://www.cloudamqp.com/blog/2017-12-29-part1-rabbitmq-best-practice.html#prefetch) on how to properly tune it.
-- Network latency has a severe impact on the throughput, and the effect is larger the smaller the prefetch count is.
+Ruby job queue backed by RabbitMQ. The word _cuniculus_ comes from the scientific name of the European rabbit (Oryctolagus cuniculus).
 
 ## Getting started
 
@@ -46,7 +17,7 @@ Create a worker class:
 require 'cuniculus/worker'
 
 class MyWorker
-  include Cuniculus::Worker
+  extend Cuniculus::Worker
 
   # the queue name is not explicitly given, so "cun_default" is used.
 
@@ -68,23 +39,33 @@ Start the job consumer:
 cuniculus -r my_worker.rb
 ```
 
-### Example
+## Benchmarks
 
-There is also a more complete example in the Cuniculus repository itself. To run it, clone the repository, then
-- start the Ruby and RabbitMQ containers using [Docker Compose](https://docs.docker.com/compose/):
-  ```
-  docker-compose up -d
-  ```
-- from within the _cuniculus_ container, produce a job:
-  ```
-  ruby -Ilib examples/produce.rb
-  ```
-- also from within the container, start the consumer:
-  ```
-  bin/cuniculus -I examples/ -r example/init_cuniculus.rb
-  ```
+The following measurements were performed with the `bin/run_benchmarks` utility, with different command parameters. Run it with `-h` to see its usage.
+
+To simulate network latency, [Toxiproxy](https://github.com/Shopify/toxiproxy) was used. It needs to be started with `toxiproxy-server` before running the benchmarks.
+
+Network latency (_ms_) | Prefetch count       | Throughput (_jobs/s_) | Average latency (_ms_)
+----------------------:|---------------------:|----------------------:|----------------------:
+1                      | 65535 (max. allowed) | 10225                 | 2
+10                     | 65535 (max. allowed) | 9990                  | 13
+1                      |                   50 | 8051                  | 2
+10                     |                   50 | 2500                  | 13
+100                    |                   50 | 481                   | 103
+1                      |         10 (default) | 5266                  | 2
+10                     |         10 (default) | 807                   | 13
+1                      |                    1 | 481                   | 2
+10                     |                    1 | 81                    | 13
+
+Additional benchmark parameters:
+- throughput was measured by consuming 100k jobs;
+- job latency was averaged over 200 samples;
+- Ruby 2.7.2 was used.
+
+Several remarks can be made:
+- Higher prefetch counts lead to higher throughput, but there are downsides of having it too high; see [this reference](https://www.cloudamqp.com/blog/2017-12-29-part1-rabbitmq-best-practice.html#prefetch) on how to properly tune it.
+- Network latency has a severe impact on the throughput, and the effect is larger the smaller the prefetch count is.
 
-The `-I examples` option adds the `examples/` directory into the load path, and `-r example/init_cuniculus.rb` requires `init_cuniculus.rb` prior to starting the consumer. The latter is where configurations such as that described in the next section should be.
 
 ## Configuration
 
@@ -107,13 +88,13 @@ rabbitmq_conn = {
 
 Cuniculus.configure do |cfg|
   cfg.rabbitmq_opts = rabbitmq_conn
-  cfg.pub_thr_pool_size = 5         # Only affects job producers
+  cfg.pub_pool_size = 5         # Only affects job producers
   cfg.dead_queue_ttl = 1000 * 60 * 60 * 24 * 30 # keep failed jobs for 30 days
   cfg.add_queue({ name: "critical", durable: true, max_retry: 10, prefetch_count: 1})
 end
 ```
 
-To configure the queue used by a worker, used `cuniculus_options`:
+To configure the queue used by a worker, use `cuniculus_options`:
 
 ```ruby
 class MyWorker
@@ -126,6 +107,24 @@ class MyWorker
 end
 ```
 
+## More examples
+
+There is also a more complete example in the Cuniculus repository itself. To run it, clone the repository, then
+- start the Ruby and RabbitMQ containers using [Docker Compose](https://docs.docker.com/compose/):
+  ```
+  docker-compose up -d
+  ```
+- from within the _cuniculus_ container, produce a job:
+  ```
+  ruby -Ilib examples/produce.rb
+  ```
+- also from within the container, start the consumer:
+  ```
+  bin/cuniculus -I examples/ -r example/init_cuniculus.rb
+  ```
+
+The `-I examples` option adds the `examples/` directory into the load path, and `-r example/init_cuniculus.rb` requires `init_cuniculus.rb` prior to starting the consumer. The latter is where configurations such as that described in the [configuration section](#configuration) section should be.
+
 ## Error handling
 
 By default, exceptions raised when consuming a job are logged to STDOUT. This can be overriden with the `Cuniculus.error_handler` method:
@@ -133,22 +132,36 @@ By default, exceptions raised when consuming a job are logged to STDOUT. This ca
 ```ruby
 Cuniculus.error_handler do |e|
   puts "Oh nein! #{e}"
+  LoggingService.send(e)
 end
 ```
 
 The method expects a block that will receive an exception, and run in the scope of the Worker instance.
 
+## Publisher proper shutdown
+
+When `perform_async` is called, the job is first put into a local (in-memory) queue that is published to RabbitMQ by a worker in a worker pool (the size of which is configured with `config.pub_pool_size`).
+
+To ensure Cuniculus tries to finish publishing jobs on shutdown, it's important that `Cuniculus.shutdown` is called. Once this method is called, workers have a grace period to publish enqueued jobs, after which the shutdown is forced. The period is set in seconds in `config.pub_shutdown_grace_period` (defaults to 50).
+
+Example code for the [Puma web server](https://github.com/puma/puma):
+```ruby
+on_worker_shutdown do
+  Cuniculus.shutdown
+end
+```
+
 ## Retry mechanism
 
 Retries are enabled by default (with 8 retries) with an exponential backoff, meaning the time between retries increases the more failures happen. The formula for calculating the times between retries can be found in {Cuniculus::QueueConfig}, namely in the `x-message-ttl` line. As an example, the time between the 7th and 8th retries is roughly 29 days.
 
-Given a declared queue in the configuration, Cuniculus starts on RabbitMQ the corresponding base queue, in addition to its retry queues. As an example, let's consider the default queue `cun_default`: Cuniculus declares a `cun_default` queue, together with some `cun_default_{n}` queues used for job retries.
+Given a queue in the configuration, Cuniculus declares on RabbitMQ the corresponding base queue, in addition to its retry queues. As an example, let's consider the default queue `cun_default`: Cuniculus declares a `cun_default` queue, together with some `cun_default_{n}` queues used for job retries.
 
 When a job raises an exception, it is placed into the `cun_default_1` queue for the first retry. It stays there for some pre-defined time, and then gets moved back into the `cun_default` queue for execution.
 
 If it fails again, it gets moved to `cun_default_2`, where it stays for a longer period until it's moved back directly into the `cun_default` queue again.
 
-This goes on until there are no more retry attempts, in which case the job gets moved into the `cun_dead` queue. It can be then only be moved back into the `cun_default` queue manually; otherwise it is discarded after some time, defined as the {Cuniculus::Config.dead_queue_ttl}, in milliseconds (by default, 180 days).
+This goes on until there are no more retry attempts, in which case the job gets moved into the `cun_dead` queue. It can be then only be moved back into the `cun_default` queue manually (from RabbitMQ itself, not with Cuniculus); otherwise it is discarded after some time, defined as the {Cuniculus::Config.dead_queue_ttl}, in milliseconds (by default, 180 days).
 
 Note that if a job cannot even be parsed, it is moved straight to the dead queue, as there's no point in retrying.
 

data/lib/cuniculus.rb

@@ -7,7 +7,7 @@ raise "Cuniculus #{Cuniculus.version} does not support Ruby versions below 2.6."
 require "cuniculus/logger"
 require "cuniculus/config"
 require "cuniculus/plugins"
-require "cuniculus/rmq_pool"
+require "cuniculus/dispatcher"
 require "cuniculus/supervisor"
 
 # Base definition of the Cuniculus Module
@@ -28,7 +28,20 @@ module Cuniculus
     yield cfg
     cfg.declare!
     @config = cfg
-    Cuniculus::RMQPool.configure(cfg)
+    @dispatcher = Cuniculus::Dispatcher.new(cfg)
+  end
+
+  def self.enqueue(job)
+    dispatcher.job_queue << job
+    dispatcher.start!
+  end
+
+  def self.shutdown
+    dispatcher.shutdown
+  end
+
+  def self.dispatcher
+    @dispatcher ||= Cuniculus::Dispatcher.new(config)
   end
 
   # Current config of Cuniculus
@@ -64,6 +77,9 @@ module Cuniculus
   def self.error_handler(&block)
     Cuniculus::Consumer.define_method(:handle_error, &block)
     Cuniculus::Consumer.instance_eval { private :handle_error }
+
+    Cuniculus::Dispatcher.define_method(:handle_error, &block)
+    Cuniculus::Dispatcher.instance_eval { private :handle_error }
   end
 
   # Load a plugin. If plugin is a Module, it is loaded directly.

data/lib/cuniculus/config.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require "logger"
 require "cuniculus/core"
 require "cuniculus/exceptions"
 require "cuniculus/queue_config"
@@ -7,14 +8,29 @@ require "cuniculus/queue_config"
 module Cuniculus
   class Config
     ENFORCED_CONN_OPTS = {
-      threaded: false # No need for a reader thread, since this connection is only used for publishing
+      threaded: false, # No need for a reader thread, since this connection is only used for declaring exchanges and queues.
+      automatically_recover: false,
+      log_level: ::Logger::ERROR
     }.freeze
 
-    attr_accessor :dead_queue_ttl, :exchange_name, :pub_thr_pool_size, :rabbitmq_opts
-    attr_reader :queues, :opts
+    attr_reader(
+      :dead_queue_ttl,
+      :exchange_name,
+      :opts,
+      :pub_pool_size,
+      :pub_reconnect_attempts,
+      :pub_reconnect_delay,
+      :pub_reconnect_delay_max,
+      :pub_shutdown_grace_period,
+      :queues
+    )
+
+    attr_accessor :rabbitmq_opts
 
     def initialize
       @opts = {}
+
+      # ---- Default values
       @queues = { "cun_default" => QueueConfig.new({ "name" => "cun_default" }) }
       @rabbitmq_opts = {
         host: "127.0.0.1",
@@ -23,11 +39,51 @@ module Cuniculus
         pass: "guest",
         vhost: "/"
       }
-      @exchange_name = "cuniculus"
-      @pub_thr_pool_size = 5
+      @exchange_name = Cuniculus::CUNICULUS_EXCHANGE
       @dead_queue_ttl = 1000 * 60 * 60 * 24 * 180 # 180 days
+      @pub_reconnect_attempts = :infinite
+      @pub_reconnect_delay = 1.5
+      @pub_reconnect_delay_max = 10
+      @pub_shutdown_grace_period = 50
+      @pub_pool_size = 5
+      ## ---- End of default values
     end
 
+      def dead_queue_ttl=(ttl)
+        raise Cuniculus::ConfigError, "dead_queue_ttl should be a positive integer, given #{ttl.inspect}" if ttl.to_i <= 0
+        @dead_queue_ttl = ttl
+      end
+
+      def exchange_name=(xname)
+        raise Cuniculus::ConfigError, "exchange_name should not be blank" if xname.to_s.empty?
+        @exchange_name = xname
+      end
+
+      def pub_pool_size=(pool_size)
+        raise Cuniculus::ConfigError, "pub_pool_size should be a positive integer, given #{pool_size.inspect}" if pool_size.to_i <= 0
+        @pub_pool_size = pool_size
+      end
+
+      def pub_reconnect_attempts=(attempts)
+        raise Cuniculus::ConfigError, "pub_reconnect_attempts should be either :infinite or a non-negative integer, was given #{attempts.inspect}" if attempts != :infinite && attempts.to_i < 0
+        @pub_reconnect_attempts = attempts
+      end
+
+      def pub_reconnect_delay=(delay)
+        raise Cuniculus::ConfigError, "pub_reconnect_delay should be a non-negative integer, was given #{delay.inspect}" if delay.to_i < 0
+        @pub_reconnect_delay = delay
+      end
+
+      def pub_reconnect_delay_max=(delay)
+        raise Cuniculus::ConfigError, "pub_reconnect_delay_max should be a non-negative integer, was given #{delay.inspect}" if delay.to_i < 0
+        @pub_reconnect_delay_max = delay
+      end
+
+      def pub_shutdown_grace_period=(period)
+        raise Cuniculus::ConfigError, "pub_shutdown_grace_period should be a non-negative integer, was given #{period.inspect}" if period.to_i < 0
+        @pub_shutdown_grace_period = period
+      end
+
     # Configure an additional queue
     #
     # Note that a single call to `add_queue` might lead to the creation of multiple queues on RabbitMQ: one base queue, and an additional queue for every retry attempt.
@@ -62,6 +118,9 @@ module Cuniculus
       declare_exchanges!(ch)
       declare_dead_queue!(ch)
       @queues.each_value { |q| q.declare!(ch) }
+      conn.close unless conn.closed?
+    rescue Bunny::TCPConnectionFailed => ex
+      raise Cuniculus.convert_exception_class(ex, Cuniculus::RMQConnectionError)
     end
 
     # Specify if the default queue `cun_default` should be created.
@@ -87,8 +146,7 @@ module Cuniculus
         arguments: {
           "x-message-ttl" => dead_queue_ttl
         }
-      ).
-        bind(Cuniculus::CUNICULUS_DLX_EXCHANGE)
+      ).bind(Cuniculus::CUNICULUS_DLX_EXCHANGE)
     end
   end
 end

data/lib/cuniculus/core.rb

@@ -34,6 +34,10 @@ module Cuniculus
       e.set_backtrace(exception.backtrace)
       e
     end
+
+    def mark_time
+      Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    end
   end
 
   extend CuniculusMethods

data/lib/cuniculus/dispatcher.rb

@@ -0,0 +1,156 @@
+# frozen_string_literal: true
+
+require "bunny"
+require "cuniculus/pub_worker"
+
+module Cuniculus
+  # The dispatcher forwards jobs to a worker pool to be published to RabbitMQ.
+  # It holds a RabbitMQ session and, when it receives information from one of its workers
+  # that a network exception occurred, tries to reestablish the connection and restarts
+  # the pool.
+  #
+  # The dispatcher background thread, which monitors for connection errors, is started
+  # whenever the first job is enqueued by a {Cuniculus::Worker}.
+  class Dispatcher
+    ENFORCED_CONN_OPTS = {
+      threaded: false, # No need for a reader thread, since this connection is only used for publishing
+      automatically_recover: false,
+      logger: ::Logger.new(IO::NULL)
+    }.freeze
+    RECOVERABLE_ERRORS = [AMQ::Protocol::Error, ::Bunny::Exception, Errno::ECONNRESET].freeze
+
+    attr_reader :dispatcher_chan, :job_queue, :reconnect_attempts, :reconnect_delay, :reconnect_delay_max, :shutdown_grace_period
+
+    # Instantiates a dispatcher using the passed {Cuniculus::Config}.
+    #
+    # @param config [Cuniculus::Config]
+    def initialize(config)
+      @config = config
+      @conn = nil
+      @job_queue = Queue.new
+      @dispatcher_chan = Queue.new
+      @shutdown = false
+      @workers = config.pub_pool_size.times.map do |i|
+        Cuniculus::PubWorker.new(config, @job_queue, @dispatcher_chan)
+      end
+      @reconnect_attempts = config.pub_reconnect_attempts
+      @reconnect_delay = config.pub_reconnect_delay
+      @reconnect_delay_max = config.pub_reconnect_delay_max
+      @shutdown_grace_period = config.pub_shutdown_grace_period
+      @thread = nil
+      @shutdown = false
+    end
+
+    def describe(log_level = Logger::DEBUG)
+      Cuniculus.logger.info @thread&.backtrace
+      @workers.each do |w|
+        Cuniculus.logger.log(log_level, w.instance_variable_get(:@thread)&.backtrace)
+      end
+    end
+
+
+    # Starts a thread responsible for reestablishing lost RabbitMQ connections and
+    # restarting {Cuniculus::PubWorker}s.
+    #
+    # It keeps track of the last time it had to reconnect, in case it receives outdated
+    # messages of failed connections from workers.
+    #
+    # PubWorkers communicate to it through its `dispatcher_chan` queue.
+    # Depending on the content fetched from the dispatcher channel, it takes different actions:
+    # - when a :shutdown message is received, it waits until current jobs are finished (up to the configured `shutdown_grace_period`) and stops its background thread.
+    # - when a timestamp is received that is smaller than the last reconnect timestamp, the message is ignored
+    # - when the timestamp is larger than the last reconnect timestamp, it tries to reestablish the connection to RabbitMQ and restarts its workers.
+    #
+    # Note that the first time the dispatcher is started, it sends a message to its own background thread with a timestamp to trigger the first connection.
+    def start!
+      return if @shutdown || @thread&.alive?
+      @thread = Thread.new do
+        last_connect_time = 0
+        loop do
+          disconnect_time = @dispatcher_chan.pop
+          break if disconnect_time == :shutdown
+          if disconnect_time > last_connect_time
+            recover_from_net_error
+            last_connect_time = Cuniculus.mark_time
+          end
+        end
+      end
+      @conn = ::Bunny.new(@config.rabbitmq_opts.merge(ENFORCED_CONN_OPTS).merge(session_error_handler: @thread))
+      @dispatcher_chan << Cuniculus.mark_time
+    end
+
+    # Whether its background thread is running.
+    #
+    # @return [Boolean]
+    def alive?
+      @thread&.alive? || false
+    end
+
+    # Starts connection to RabbitMQ followed by starting the workers background threads.
+    #
+    # if it fails to connect, it keeps retrying for a certain number of attempts, defined by
+    # {Config.pub_reconnect_attempts}. For unlimited retries, this value should be set to `:infinite`.
+    #
+    # The time between reconnect attempts follows an exponential backoff formula:
+    #
+    # ```
+    # t = delay * 2^(n-1)
+    # ```
+    #
+    # where n is the attempt number, and delay is defined by {Config.pub_reconnect_delay}.
+    #
+    # If {Config.pub_reconnect_delay_max} is defined, it works as a cap for the above time.
+    # @return [void]
+    def recover_from_net_error
+      attempt = 0
+      begin
+        @conn.start
+        Cuniculus.logger.info("Connection established")
+
+        @workers.each { |w| w.start!(@conn) }
+      rescue *RECOVERABLE_ERRORS => ex
+        handle_error(Cuniculus.convert_exception_class(ex, Cuniculus::RMQConnectionError))
+        sleep_time = @shutdown ? 1 : [(reconnect_delay * 2**(attempt-1)), reconnect_delay_max].min
+        sleep sleep_time
+        attempt += 1
+
+        retry if @shutdown && attempt <= reconnect_delay_max
+        retry if reconnect_attempts == :infinite || attempt <= reconnect_attempts
+      end
+    end
+
+    # Shutdown workers, giving them time to conclude outstanding tasks.
+    #
+    # Shutdown is forced after {Config.pub_shutdown_grace_period} seconds.
+    #
+    # @return [void]
+    def shutdown
+      Cuniculus.logger.info("Cuniculus: Shutting down dispatcher")
+      @shutdown = true
+      alive_size = @workers.size
+      shutdown_t0 = Cuniculus.mark_time
+
+      sleep 1 until Cuniculus.mark_time - shutdown_t0 > shutdown_grace_period || @job_queue.empty?
+
+      until Cuniculus.mark_time - shutdown_t0 > shutdown_grace_period || (alive_size = @workers.select(&:alive?).size) == 0
+        sleep 1
+        alive_size.times { @job_queue << :shutdown }
+      end
+
+      @dispatcher_chan << :shutdown
+      alive_size = @workers.select(&:alive?).size
+      return unless alive_size > 0
+
+      Cuniculus.logger.warn("Cuniculus: Forcing shutdown with #{alive_size} workers remaining")
+      describe
+    end
+
+    private
+
+    def handle_error(e)
+      Cuniculus.logger.error("#{e.class.name}: #{e.message}")
+      Cuniculus.logger.error(e.backtrace.join("\n")) unless e.backtrace.nil?
+    end
+  end
+end
+

data/lib/cuniculus/exceptions.rb

@@ -25,6 +25,10 @@ module Cuniculus
     def cause
       wrapped_exception || super
     end
+
+    def message
+      wrapped_exception&.message || to_s
+    end
   end
 
   # Dev note:

data/lib/cuniculus/pub_worker.rb

@@ -0,0 +1,83 @@
+# frozen_string_literal: true
+
+require "bunny"
+
+module Cuniculus
+  # Each PubWorker maintains a background thread in a loop, fetching jobs reaching
+  # its job queue and publishing the payloads to RabbitMQ. They are not instantiated
+  # directly, but are rather created and managed by a {Cuniculus::Dispatcher}.
+  class PubWorker
+    def initialize(config, job_queue, dispatcher_chan)
+      @config = config
+      @job_queue = job_queue
+      @dispatcher_chan = dispatcher_chan
+      @mutex = Mutex.new
+      @thread = nil
+    end
+
+    # Declares exchanges, and starts a background thread that consumes and publishes messages.
+    #
+    # If the connection to RabbitMQ it receives is not established, or if it fails to declare
+    # the exchanges, the background thread is not started and a message is sent to the
+    # dispatcher channel with the current timestamp. The dispatcher is then responsible for
+    # trying to set the connection up again and starting each of its workers.
+    #
+    # @param conn [::Bunny::Session] Connection to RabbitMQ. Expected to be open at this stage.
+    def start!(conn)
+      return @dispatcher_chan << Cuniculus.mark_time unless conn.open?
+
+      @channel = sync { conn.create_channel }
+      @x = sync { @channel.direct(Cuniculus::CUNICULUS_EXCHANGE, { durable: true }) }
+      @dlx = sync { @channel.fanout(Cuniculus::CUNICULUS_DLX_EXCHANGE, { durable: true }) }
+      @thread = Thread.new { run }
+    rescue Bunny::Exception
+      @dispatcher_chan << Cuniculus.mark_time
+    end
+
+    # Whether the background thread is running.
+    #
+    # @return [Boolean]
+    def alive?
+      @thread&.alive? || false
+    end
+
+    private
+
+    # Starts the job consuming loop. This is used internally by `start!` and runs in
+    # a background thread. Messages are published to RabbitMQ.
+    #
+    # The loop is finished if the message `:shutdown` is retrieved from the job queue or
+    # if an exception happens while trying to publish a message to RabbitMQ. In the
+    # latter case, the job is reinserted into the job queue, and a message with the timestamp
+    # is sent into the dispatcher channel, so that it can try restart the connection
+    # and the workers again.
+    def run
+      loop do
+        case msg = @job_queue.pop
+        when :shutdown
+          break
+        else
+          xname, payload, routing_key = msg
+          exchange = if xname == CUNICULUS_DLX_EXCHANGE
+                       sync { @dlx }
+                     else
+                       sync { @x }
+                     end
+          begin
+            publish_time = Cuniculus.mark_time
+            exchange.publish(payload, { routing_key: routing_key, persistent: true })
+          rescue *::Cuniculus::Dispatcher::RECOVERABLE_ERRORS
+            @job_queue << [xname, payload, routing_key]
+            @dispatcher_chan << publish_time
+            break
+          end
+        end
+      end
+      sync { @channel.close unless @channel.closed? }
+    end
+
+    def sync(&block)
+      @mutex.synchronize(&block)
+    end
+  end
+end

data/lib/cuniculus/version.rb

@@ -6,11 +6,11 @@ module Cuniculus
 
   # The minor version of Cuniculus. Bumped for every non-patch level
   # release.
-  MINOR = 1
+  MINOR = 2
 
   # The tiny version of Cuniculus.  Usually 0, only bumped for bugfix
   # releases that fix regressions from previous versions.
-  TINY  = 0
+  TINY  = 1
 
   # The version of Cuniculus you are using, as a string (e.g. "2.11.0")
   VERSION = [MAJOR, MINOR, TINY].join(".").freeze

data/lib/cuniculus/worker.rb

@@ -1,82 +1,66 @@
 # frozen_string_literal: true
 
 require "cuniculus/core"
-require "cuniculus/rmq_pool"
+require "cuniculus/exceptions"
 
 module Cuniculus
   module Worker
-    def self.included(base)
-      base.extend(ClassMethods)
+    DEFAULT_OPTS = { queue: "cun_default" }.freeze
+    VALID_OPT_KEYS = %i[queue].freeze
 
-      # Dev note:
-      # The point here is to allow options set via cuniculus_options to be
-      # inherited by subclasses.
-      # When reading the options, a subclass will call the singleton method cun_opts.
-      # If the subclass doesn't redefine this method via a call to cuniculus_options,
-      # it will still use the definition from its parent class.
-      base.define_singleton_method("cun_opts=") do |opts|
-        singleton_class.class_eval do
-          define_method("cun_opts") { opts }
-        end
-      end
+    def self.extended(base)
+      base.instance_variable_set(:@cun_opts, DEFAULT_OPTS)
+      super
     end
 
-    module ClassMethods
-      DEFAULT_OPTS = { "queue" => "cun_default" }.freeze
-      VALID_OPT_KEYS = %w[queue].freeze
+    def inherited(mod)
+      mod.instance_variable_set(:@cun_opts, @cun_opts)
+      super
+    end
 
-      # Read-only cuniculus option values
-      #
-      # @return opts [Hash] hash with current values
-      def cun_opts
-        DEFAULT_OPTS
-      end
+    attr_reader :cun_opts
 
-      # Worker-specific options for running cuniculus.
-      #
-      # Note that options set on a worker class are inherited by its subclasses.
-      #
-      # @param opts [Hash]
-      # @option opts [String] "queue" ("cun_default") Name of the underlying RabbitMQ queue.
-      #
-      # @example Change the queue name of a worker
-      #   class MyWorker
-      #     include Cuniculus::Worker
-      #
-      #     cuniculus_options queue: "critical"
-      #
-      #     def perform
-      #       # run the task
-      #     end
-      #   end
-      def cuniculus_options(opts)
-        opts = validate_opts!(opts)
-        self.cun_opts = opts
-      end
+    # Worker-specific options for running cuniculus.
+    #
+    # Note that options set on a worker class are inherited by its subclasses.
+    #
+    # @param opts [Hash]
+    # @option opts [String] "queue" ("cun_default") Name of the underlying RabbitMQ queue.
+    #
+    # @example Change the queue name of a worker
+    #   class MyWorker
+    #     include Cuniculus::Worker
+    #
+    #     cuniculus_options queue: "critical"
+    #
+    #     def perform
+    #       # run the task
+    #     end
+    #   end
+    def cuniculus_options(opts)
+      opts = validate_opts!(opts)
+      @cun_opts = opts
+    end
 
-      def validate_opts!(opts)
-        raise WorkerOptionsError, "Argument passed to 'cuniculus_options' should be a Hash" unless opts.is_a?(Hash)
-        opts = opts.transform_keys(&:to_s)
-        invalid_keys = opts.keys - VALID_OPT_KEYS
-        raise WorkerOptionsError, "Invalid keys passed to 'cuniculus_options': #{invalid_keys.inspect}" unless invalid_keys.empty?
-        opts
-      end
+    def validate_opts!(opts)
+      raise Cuniculus::WorkerOptionsError, "Argument passed to 'cuniculus_options' should be a Hash" unless opts.is_a?(Hash)
+      invalid_keys = opts.keys - VALID_OPT_KEYS
+      raise Cuniculus::WorkerOptionsError, "Invalid keys passed to 'cuniculus_options': #{invalid_keys.inspect}" unless invalid_keys.empty?
+      opts
+    end
 
-      def perform_async(*args)
-        publish({ "class" => self, "args" => args })
-      end
+    def perform_async(*args)
+      publish({ "class" => self, "args" => args })
+    end
 
-      def publish(item)
-        routing_key = cun_opts["queue"]
-        payload = normalize_item(item)
-        Cuniculus::RMQPool.with_exchange do |x|
-          x.publish(payload, { routing_key: routing_key, persistent: true })
-        end
-      end
+    def publish(item)
+      routing_key = cun_opts[:queue]
+      payload = normalize_item(item)
+      Cuniculus.enqueue [Cuniculus::CUNICULUS_EXCHANGE, payload, routing_key]
+    end
 
-      def normalize_item(item)
-        Cuniculus.dump_job(item)
-      end
+    def normalize_item(item)
+      Cuniculus.dump_job(item)
     end
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: cuniculus
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.1
 platform: ruby
 authors:
 - Marcelo Pereira
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2021-03-07 00:00:00.000000000 Z
+date: 2021-07-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bunny
@@ -24,20 +24,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: 2.15.0
-- !ruby/object:Gem::Dependency
-  name: connection_pool
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: 2.2.2
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: 2.2.2
 - !ruby/object:Gem::Dependency
   name: pry
   requirement: !ruby/object:Gem::Requirement
@@ -182,13 +168,14 @@ files:
 - lib/cuniculus/config.rb
 - lib/cuniculus/consumer.rb
 - lib/cuniculus/core.rb
+- lib/cuniculus/dispatcher.rb
 - lib/cuniculus/exceptions.rb
 - lib/cuniculus/job_queue.rb
 - lib/cuniculus/logger.rb
 - lib/cuniculus/plugins.rb
 - lib/cuniculus/plugins/health_check.rb
+- lib/cuniculus/pub_worker.rb
 - lib/cuniculus/queue_config.rb
-- lib/cuniculus/rmq_pool.rb
 - lib/cuniculus/supervisor.rb
 - lib/cuniculus/version.rb
 - lib/cuniculus/worker.rb
@@ -207,7 +194,7 @@ rdoc_options:
 - "--title"
 - 'Cuniculus: Background job processing with RabbitMQ'
 - "--main"
-- README.rdoc
+- README.md
 require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement

data/lib/cuniculus/rmq_pool.rb

@@ -1,44 +0,0 @@
-# frozen_string_literal: true
-
-require "bunny"
-require "connection_pool"
-require "cuniculus/core"
-require "cuniculus/config"
-
-module Cuniculus
-  class RMQPool
-    ENFORCED_CONN_OPTS = {
-      threaded: false # No need for a reader thread, since this connection is only used for publishing
-    }.freeze
-
-    class << self
-      def configure(config)
-        @config = config
-      end
-
-      def config
-        @config ||= Cuniculus::Config.new
-      end
-
-      def init!
-        @conn = ::Bunny.new(@config.rabbitmq_opts.merge(ENFORCED_CONN_OPTS))
-        @conn.start
-        @channel_pool = ConnectionPool.new(timeout: 1, size: @config.pub_thr_pool_size) do
-          ch = @conn.create_channel
-          ch.direct(Cuniculus::CUNICULUS_EXCHANGE, { durable: true })
-          ch.fanout(Cuniculus::CUNICULUS_DLX_EXCHANGE, { durable: true })
-          ch
-        end
-      end
-
-      def with_exchange(&block)
-        init! unless @channel_pool
-        @channel_pool.with do |ch|
-          block.call(ch.exchanges["cuniculus"])
-        ensure
-          ch.open if ch.closed?
-        end
-      end
-    end
-  end
-end