cuniculus 0.1.0 → 0.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: ca0cc8cbcc9cc9607488d5c7c88903548fe4106799e6517e82b2efb5824a9971
4
- data.tar.gz: 6a0589c882ae8de39a0192e2e5b54c537a709fbe6ca496ce3b999d3a03ff6da1
3
+ metadata.gz: 836705af66bf5ceb94715ce41b70906928611e4b37eaee3e343e4ee8d6a52256
4
+ data.tar.gz: da7c6168ea3aeddd9896c956f9cb89b5beeda5ded1aa7d663a6e8662eae2100d
5
5
  SHA512:
6
- metadata.gz: 73fc5ea97169bdbdf5a72b95dec76e2b529a13c6b308a3c3c9c947d891bc3845d48ad5b4344820d0f1fed8cbed8b0bb1ffac3a4c3ea043ca0cb3c1737eb8014d
7
- data.tar.gz: 7dbdd0c30c1dc808457fc408506aade75b65de83f5318799f8b7a5aab4c0891ed71fbb927e58eaa70b099442790b1c1d20b06d34d12a53967def89052f068887
6
+ metadata.gz: 69ae000142f4d761c10a37e37df70bbc257f28b602df19d7aeb9508df844d1a0165e22502ec8c9a8b6f195d4db0220faf0c7be8900489c12a1a1460ec7220fa7
7
+ data.tar.gz: 516789a202f6000f4145dfc218f60645437ee9476f129670dd3d464c38084399e97b930b022efee40906519752243caccb526e6d4d5e3714bb268a776de6ab75
data/README.md CHANGED
@@ -1,36 +1,7 @@
1
- # Cuniculus
1
+ ![cuniculus_logo](https://user-images.githubusercontent.com/4718658/115109242-de53f900-9f74-11eb-8a46-b93d6f484370.png)
2
2
 
3
- Ruby job queue backed by RabbitMQ. The word _cuniculus_ comes from the scientific name of the European rabbit (Oryctolagus cuniculus).
4
-
5
- ## Benchmarks
6
-
7
- The following measurements were performed with the `bin/run_benchmarks` utility, with different command parameters. Run it with `-h` to see its usage.
8
-
9
- To simulate network latency, [Toxiproxy](https://github.com/Shopify/toxiproxy) was used. It needs to be started with `toxiproxy-server` before running the benchmarks.
10
3
 
11
- Network latency (_ms_) | Prefetch count | Throughput (_jobs/s_) | Average latency (_ms_)
12
- ----------------------:|---------------------:|----------------------:|----------------------:
13
- 1 | 65535 (max. allowed) | 10225 | 2
14
- 10 | 65535 (max. allowed) | 9990 | 13
15
- 1 | 50 | 8051 | 2
16
- 10 | 50 | 2500 | 13
17
- 100 | 50 | 481 | 103
18
- 1 | 25 | 7824 | 2
19
- 10 | 25 | 1824 | 13
20
- 50 | 25 | 469 | 53
21
- 1 | 10 (default) | 5266 | 2
22
- 10 | 10 (default) | 807 | 13
23
- 1 | 1 | 481 | 2
24
- 10 | 1 | 81 | 13
25
-
26
- Additional benchmark parameters:
27
- - throughput was measured by consuming 100k jobs;
28
- - job latency was averaged over 200 samples;
29
- - Ruby 2.7.2 was used.
30
-
31
- Several remarks can be made:
32
- - Higher prefetch counts lead to higher throughput, but there are downsides of having it too high; see [this reference](https://www.cloudamqp.com/blog/2017-12-29-part1-rabbitmq-best-practice.html#prefetch) on how to properly tune it.
33
- - Network latency has a severe impact on the throughput, and the effect is larger the smaller the prefetch count is.
4
+ Ruby job queue backed by RabbitMQ. The word _cuniculus_ comes from the scientific name of the European rabbit (Oryctolagus cuniculus).
34
5
 
35
6
  ## Getting started
36
7
 
@@ -46,7 +17,7 @@ Create a worker class:
46
17
  require 'cuniculus/worker'
47
18
 
48
19
  class MyWorker
49
- include Cuniculus::Worker
20
+ extend Cuniculus::Worker
50
21
 
51
22
  # the queue name is not explicitly given, so "cun_default" is used.
52
23
 
@@ -68,23 +39,33 @@ Start the job consumer:
68
39
  cuniculus -r my_worker.rb
69
40
  ```
70
41
 
71
- ### Example
42
+ ## Benchmarks
72
43
 
73
- There is also a more complete example in the Cuniculus repository itself. To run it, clone the repository, then
74
- - start the Ruby and RabbitMQ containers using [Docker Compose](https://docs.docker.com/compose/):
75
- ```
76
- docker-compose up -d
77
- ```
78
- - from within the _cuniculus_ container, produce a job:
79
- ```
80
- ruby -Ilib examples/produce.rb
81
- ```
82
- - also from within the container, start the consumer:
83
- ```
84
- bin/cuniculus -I examples/ -r example/init_cuniculus.rb
85
- ```
44
+ The following measurements were performed with the `bin/run_benchmarks` utility, with different command parameters. Run it with `-h` to see its usage.
45
+
46
+ To simulate network latency, [Toxiproxy](https://github.com/Shopify/toxiproxy) was used. It needs to be started with `toxiproxy-server` before running the benchmarks.
47
+
48
+ Network latency (_ms_) | Prefetch count | Throughput (_jobs/s_) | Average latency (_ms_)
49
+ ----------------------:|---------------------:|----------------------:|----------------------:
50
+ 1 | 65535 (max. allowed) | 10225 | 2
51
+ 10 | 65535 (max. allowed) | 9990 | 13
52
+ 1 | 50 | 8051 | 2
53
+ 10 | 50 | 2500 | 13
54
+ 100 | 50 | 481 | 103
55
+ 1 | 10 (default) | 5266 | 2
56
+ 10 | 10 (default) | 807 | 13
57
+ 1 | 1 | 481 | 2
58
+ 10 | 1 | 81 | 13
59
+
60
+ Additional benchmark parameters:
61
+ - throughput was measured by consuming 100k jobs;
62
+ - job latency was averaged over 200 samples;
63
+ - Ruby 2.7.2 was used.
64
+
65
+ Several remarks can be made:
66
+ - Higher prefetch counts lead to higher throughput, but there are downsides of having it too high; see [this reference](https://www.cloudamqp.com/blog/2017-12-29-part1-rabbitmq-best-practice.html#prefetch) on how to properly tune it.
67
+ - Network latency has a severe impact on the throughput, and the effect is larger the smaller the prefetch count is.
86
68
 
87
- The `-I examples` option adds the `examples/` directory into the load path, and `-r example/init_cuniculus.rb` requires `init_cuniculus.rb` prior to starting the consumer. The latter is where configurations such as that described in the next section should be.
88
69
 
89
70
  ## Configuration
90
71
 
@@ -107,13 +88,13 @@ rabbitmq_conn = {
107
88
 
108
89
  Cuniculus.configure do |cfg|
109
90
  cfg.rabbitmq_opts = rabbitmq_conn
110
- cfg.pub_thr_pool_size = 5 # Only affects job producers
91
+ cfg.pub_pool_size = 5 # Only affects job producers
111
92
  cfg.dead_queue_ttl = 1000 * 60 * 60 * 24 * 30 # keep failed jobs for 30 days
112
93
  cfg.add_queue({ name: "critical", durable: true, max_retry: 10, prefetch_count: 1})
113
94
  end
114
95
  ```
115
96
 
116
- To configure the queue used by a worker, used `cuniculus_options`:
97
+ To configure the queue used by a worker, use `cuniculus_options`:
117
98
 
118
99
  ```ruby
119
100
  class MyWorker
@@ -126,6 +107,24 @@ class MyWorker
126
107
  end
127
108
  ```
128
109
 
110
+ ## More examples
111
+
112
+ There is also a more complete example in the Cuniculus repository itself. To run it, clone the repository, then
113
+ - start the Ruby and RabbitMQ containers using [Docker Compose](https://docs.docker.com/compose/):
114
+ ```
115
+ docker-compose up -d
116
+ ```
117
+ - from within the _cuniculus_ container, produce a job:
118
+ ```
119
+ ruby -Ilib examples/produce.rb
120
+ ```
121
+ - also from within the container, start the consumer:
122
+ ```
123
+ bin/cuniculus -I examples/ -r example/init_cuniculus.rb
124
+ ```
125
+
126
+ The `-I examples` option adds the `examples/` directory into the load path, and `-r example/init_cuniculus.rb` requires `init_cuniculus.rb` prior to starting the consumer. The latter is where configurations such as that described in the [configuration section](#configuration) section should be.
127
+
129
128
  ## Error handling
130
129
 
131
130
  By default, exceptions raised when consuming a job are logged to STDOUT. This can be overriden with the `Cuniculus.error_handler` method:
@@ -133,22 +132,36 @@ By default, exceptions raised when consuming a job are logged to STDOUT. This ca
133
132
  ```ruby
134
133
  Cuniculus.error_handler do |e|
135
134
  puts "Oh nein! #{e}"
135
+ LoggingService.send(e)
136
136
  end
137
137
  ```
138
138
 
139
139
  The method expects a block that will receive an exception, and run in the scope of the Worker instance.
140
140
 
141
+ ## Publisher proper shutdown
142
+
143
+ When `perform_async` is called, the job is first put into a local (in-memory) queue that is published to RabbitMQ by a worker in a worker pool (the size of which is configured with `config.pub_pool_size`).
144
+
145
+ To ensure Cuniculus tries to finish publishing jobs on shutdown, it's important that `Cuniculus.shutdown` is called. Once this method is called, workers have a grace period to publish enqueued jobs, after which the shutdown is forced. The period is set in seconds in `config.pub_shutdown_grace_period` (defaults to 50).
146
+
147
+ Example code for the [Puma web server](https://github.com/puma/puma):
148
+ ```ruby
149
+ on_worker_shutdown do
150
+ Cuniculus.shutdown
151
+ end
152
+ ```
153
+
141
154
  ## Retry mechanism
142
155
 
143
156
  Retries are enabled by default (with 8 retries) with an exponential backoff, meaning the time between retries increases the more failures happen. The formula for calculating the times between retries can be found in {Cuniculus::QueueConfig}, namely in the `x-message-ttl` line. As an example, the time between the 7th and 8th retries is roughly 29 days.
144
157
 
145
- Given a declared queue in the configuration, Cuniculus starts on RabbitMQ the corresponding base queue, in addition to its retry queues. As an example, let's consider the default queue `cun_default`: Cuniculus declares a `cun_default` queue, together with some `cun_default_{n}` queues used for job retries.
158
+ Given a queue in the configuration, Cuniculus declares on RabbitMQ the corresponding base queue, in addition to its retry queues. As an example, let's consider the default queue `cun_default`: Cuniculus declares a `cun_default` queue, together with some `cun_default_{n}` queues used for job retries.
146
159
 
147
160
  When a job raises an exception, it is placed into the `cun_default_1` queue for the first retry. It stays there for some pre-defined time, and then gets moved back into the `cun_default` queue for execution.
148
161
 
149
162
  If it fails again, it gets moved to `cun_default_2`, where it stays for a longer period until it's moved back directly into the `cun_default` queue again.
150
163
 
151
- This goes on until there are no more retry attempts, in which case the job gets moved into the `cun_dead` queue. It can be then only be moved back into the `cun_default` queue manually; otherwise it is discarded after some time, defined as the {Cuniculus::Config.dead_queue_ttl}, in milliseconds (by default, 180 days).
164
+ This goes on until there are no more retry attempts, in which case the job gets moved into the `cun_dead` queue. It can be then only be moved back into the `cun_default` queue manually (from RabbitMQ itself, not with Cuniculus); otherwise it is discarded after some time, defined as the {Cuniculus::Config.dead_queue_ttl}, in milliseconds (by default, 180 days).
152
165
 
153
166
  Note that if a job cannot even be parsed, it is moved straight to the dead queue, as there's no point in retrying.
154
167
 
data/lib/cuniculus.rb CHANGED
@@ -7,7 +7,7 @@ raise "Cuniculus #{Cuniculus.version} does not support Ruby versions below 2.6."
7
7
  require "cuniculus/logger"
8
8
  require "cuniculus/config"
9
9
  require "cuniculus/plugins"
10
- require "cuniculus/rmq_pool"
10
+ require "cuniculus/dispatcher"
11
11
  require "cuniculus/supervisor"
12
12
 
13
13
  # Base definition of the Cuniculus Module
@@ -28,7 +28,20 @@ module Cuniculus
28
28
  yield cfg
29
29
  cfg.declare!
30
30
  @config = cfg
31
- Cuniculus::RMQPool.configure(cfg)
31
+ @dispatcher = Cuniculus::Dispatcher.new(cfg)
32
+ end
33
+
34
+ def self.enqueue(job)
35
+ dispatcher.job_queue << job
36
+ dispatcher.start!
37
+ end
38
+
39
+ def self.shutdown
40
+ dispatcher.shutdown
41
+ end
42
+
43
+ def self.dispatcher
44
+ @dispatcher ||= Cuniculus::Dispatcher.new(config)
32
45
  end
33
46
 
34
47
  # Current config of Cuniculus
@@ -64,6 +77,9 @@ module Cuniculus
64
77
  def self.error_handler(&block)
65
78
  Cuniculus::Consumer.define_method(:handle_error, &block)
66
79
  Cuniculus::Consumer.instance_eval { private :handle_error }
80
+
81
+ Cuniculus::Dispatcher.define_method(:handle_error, &block)
82
+ Cuniculus::Dispatcher.instance_eval { private :handle_error }
67
83
  end
68
84
 
69
85
  # Load a plugin. If plugin is a Module, it is loaded directly.
@@ -1,5 +1,6 @@
1
1
  # frozen_string_literal: true
2
2
 
3
+ require "logger"
3
4
  require "cuniculus/core"
4
5
  require "cuniculus/exceptions"
5
6
  require "cuniculus/queue_config"
@@ -7,14 +8,29 @@ require "cuniculus/queue_config"
7
8
  module Cuniculus
8
9
  class Config
9
10
  ENFORCED_CONN_OPTS = {
10
- threaded: false # No need for a reader thread, since this connection is only used for publishing
11
+ threaded: false, # No need for a reader thread, since this connection is only used for declaring exchanges and queues.
12
+ automatically_recover: false,
13
+ log_level: ::Logger::ERROR
11
14
  }.freeze
12
15
 
13
- attr_accessor :dead_queue_ttl, :exchange_name, :pub_thr_pool_size, :rabbitmq_opts
14
- attr_reader :queues, :opts
16
+ attr_reader(
17
+ :dead_queue_ttl,
18
+ :exchange_name,
19
+ :opts,
20
+ :pub_pool_size,
21
+ :pub_reconnect_attempts,
22
+ :pub_reconnect_delay,
23
+ :pub_reconnect_delay_max,
24
+ :pub_shutdown_grace_period,
25
+ :queues
26
+ )
27
+
28
+ attr_accessor :rabbitmq_opts
15
29
 
16
30
  def initialize
17
31
  @opts = {}
32
+
33
+ # ---- Default values
18
34
  @queues = { "cun_default" => QueueConfig.new({ "name" => "cun_default" }) }
19
35
  @rabbitmq_opts = {
20
36
  host: "127.0.0.1",
@@ -23,11 +39,51 @@ module Cuniculus
23
39
  pass: "guest",
24
40
  vhost: "/"
25
41
  }
26
- @exchange_name = "cuniculus"
27
- @pub_thr_pool_size = 5
42
+ @exchange_name = Cuniculus::CUNICULUS_EXCHANGE
28
43
  @dead_queue_ttl = 1000 * 60 * 60 * 24 * 180 # 180 days
44
+ @pub_reconnect_attempts = :infinite
45
+ @pub_reconnect_delay = 1.5
46
+ @pub_reconnect_delay_max = 10
47
+ @pub_shutdown_grace_period = 50
48
+ @pub_pool_size = 5
49
+ ## ---- End of default values
29
50
  end
30
51
 
52
+ def dead_queue_ttl=(ttl)
53
+ raise Cuniculus::ConfigError, "dead_queue_ttl should be a positive integer, given #{ttl.inspect}" if ttl.to_i <= 0
54
+ @dead_queue_ttl = ttl
55
+ end
56
+
57
+ def exchange_name=(xname)
58
+ raise Cuniculus::ConfigError, "exchange_name should not be blank" if xname.to_s.empty?
59
+ @exchange_name = xname
60
+ end
61
+
62
+ def pub_pool_size=(pool_size)
63
+ raise Cuniculus::ConfigError, "pub_pool_size should be a positive integer, given #{pool_size.inspect}" if pool_size.to_i <= 0
64
+ @pub_pool_size = pool_size
65
+ end
66
+
67
+ def pub_reconnect_attempts=(attempts)
68
+ raise Cuniculus::ConfigError, "pub_reconnect_attempts should be either :infinite or a non-negative integer, was given #{attempts.inspect}" if attempts != :infinite && attempts.to_i < 0
69
+ @pub_reconnect_attempts = attempts
70
+ end
71
+
72
+ def pub_reconnect_delay=(delay)
73
+ raise Cuniculus::ConfigError, "pub_reconnect_delay should be a non-negative integer, was given #{delay.inspect}" if delay.to_i < 0
74
+ @pub_reconnect_delay = delay
75
+ end
76
+
77
+ def pub_reconnect_delay_max=(delay)
78
+ raise Cuniculus::ConfigError, "pub_reconnect_delay_max should be a non-negative integer, was given #{delay.inspect}" if delay.to_i < 0
79
+ @pub_reconnect_delay_max = delay
80
+ end
81
+
82
+ def pub_shutdown_grace_period=(period)
83
+ raise Cuniculus::ConfigError, "pub_shutdown_grace_period should be a non-negative integer, was given #{period.inspect}" if period.to_i < 0
84
+ @pub_shutdown_grace_period = period
85
+ end
86
+
31
87
  # Configure an additional queue
32
88
  #
33
89
  # Note that a single call to `add_queue` might lead to the creation of multiple queues on RabbitMQ: one base queue, and an additional queue for every retry attempt.
@@ -62,6 +118,9 @@ module Cuniculus
62
118
  declare_exchanges!(ch)
63
119
  declare_dead_queue!(ch)
64
120
  @queues.each_value { |q| q.declare!(ch) }
121
+ conn.close unless conn.closed?
122
+ rescue Bunny::TCPConnectionFailed => ex
123
+ raise Cuniculus.convert_exception_class(ex, Cuniculus::RMQConnectionError)
65
124
  end
66
125
 
67
126
  # Specify if the default queue `cun_default` should be created.
@@ -87,8 +146,7 @@ module Cuniculus
87
146
  arguments: {
88
147
  "x-message-ttl" => dead_queue_ttl
89
148
  }
90
- ).
91
- bind(Cuniculus::CUNICULUS_DLX_EXCHANGE)
149
+ ).bind(Cuniculus::CUNICULUS_DLX_EXCHANGE)
92
150
  end
93
151
  end
94
152
  end
@@ -34,6 +34,10 @@ module Cuniculus
34
34
  e.set_backtrace(exception.backtrace)
35
35
  e
36
36
  end
37
+
38
+ def mark_time
39
+ Process.clock_gettime(Process::CLOCK_MONOTONIC)
40
+ end
37
41
  end
38
42
 
39
43
  extend CuniculusMethods
@@ -0,0 +1,156 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "bunny"
4
+ require "cuniculus/pub_worker"
5
+
6
+ module Cuniculus
7
+ # The dispatcher forwards jobs to a worker pool to be published to RabbitMQ.
8
+ # It holds a RabbitMQ session and, when it receives information from one of its workers
9
+ # that a network exception occurred, tries to reestablish the connection and restarts
10
+ # the pool.
11
+ #
12
+ # The dispatcher background thread, which monitors for connection errors, is started
13
+ # whenever the first job is enqueued by a {Cuniculus::Worker}.
14
+ class Dispatcher
15
+ ENFORCED_CONN_OPTS = {
16
+ threaded: false, # No need for a reader thread, since this connection is only used for publishing
17
+ automatically_recover: false,
18
+ logger: ::Logger.new(IO::NULL)
19
+ }.freeze
20
+ RECOVERABLE_ERRORS = [AMQ::Protocol::Error, ::Bunny::Exception, Errno::ECONNRESET].freeze
21
+
22
+ attr_reader :dispatcher_chan, :job_queue, :reconnect_attempts, :reconnect_delay, :reconnect_delay_max, :shutdown_grace_period
23
+
24
+ # Instantiates a dispatcher using the passed {Cuniculus::Config}.
25
+ #
26
+ # @param config [Cuniculus::Config]
27
+ def initialize(config)
28
+ @config = config
29
+ @conn = nil
30
+ @job_queue = Queue.new
31
+ @dispatcher_chan = Queue.new
32
+ @shutdown = false
33
+ @workers = config.pub_pool_size.times.map do |i|
34
+ Cuniculus::PubWorker.new(config, @job_queue, @dispatcher_chan)
35
+ end
36
+ @reconnect_attempts = config.pub_reconnect_attempts
37
+ @reconnect_delay = config.pub_reconnect_delay
38
+ @reconnect_delay_max = config.pub_reconnect_delay_max
39
+ @shutdown_grace_period = config.pub_shutdown_grace_period
40
+ @thread = nil
41
+ @shutdown = false
42
+ end
43
+
44
+ def describe(log_level = Logger::DEBUG)
45
+ Cuniculus.logger.info @thread&.backtrace
46
+ @workers.each do |w|
47
+ Cuniculus.logger.log(log_level, w.instance_variable_get(:@thread)&.backtrace)
48
+ end
49
+ end
50
+
51
+
52
+ # Starts a thread responsible for reestablishing lost RabbitMQ connections and
53
+ # restarting {Cuniculus::PubWorker}s.
54
+ #
55
+ # It keeps track of the last time it had to reconnect, in case it receives outdated
56
+ # messages of failed connections from workers.
57
+ #
58
+ # PubWorkers communicate to it through its `dispatcher_chan` queue.
59
+ # Depending on the content fetched from the dispatcher channel, it takes different actions:
60
+ # - when a :shutdown message is received, it waits until current jobs are finished (up to the configured `shutdown_grace_period`) and stops its background thread.
61
+ # - when a timestamp is received that is smaller than the last reconnect timestamp, the message is ignored
62
+ # - when the timestamp is larger than the last reconnect timestamp, it tries to reestablish the connection to RabbitMQ and restarts its workers.
63
+ #
64
+ # Note that the first time the dispatcher is started, it sends a message to its own background thread with a timestamp to trigger the first connection.
65
+ def start!
66
+ return if @shutdown || @thread&.alive?
67
+ @thread = Thread.new do
68
+ last_connect_time = 0
69
+ loop do
70
+ disconnect_time = @dispatcher_chan.pop
71
+ break if disconnect_time == :shutdown
72
+ if disconnect_time > last_connect_time
73
+ recover_from_net_error
74
+ last_connect_time = Cuniculus.mark_time
75
+ end
76
+ end
77
+ end
78
+ @conn = ::Bunny.new(@config.rabbitmq_opts.merge(ENFORCED_CONN_OPTS).merge(session_error_handler: @thread))
79
+ @dispatcher_chan << Cuniculus.mark_time
80
+ end
81
+
82
+ # Whether its background thread is running.
83
+ #
84
+ # @return [Boolean]
85
+ def alive?
86
+ @thread&.alive? || false
87
+ end
88
+
89
+ # Starts connection to RabbitMQ followed by starting the workers background threads.
90
+ #
91
+ # if it fails to connect, it keeps retrying for a certain number of attempts, defined by
92
+ # {Config.pub_reconnect_attempts}. For unlimited retries, this value should be set to `:infinite`.
93
+ #
94
+ # The time between reconnect attempts follows an exponential backoff formula:
95
+ #
96
+ # ```
97
+ # t = delay * 2^(n-1)
98
+ # ```
99
+ #
100
+ # where n is the attempt number, and delay is defined by {Config.pub_reconnect_delay}.
101
+ #
102
+ # If {Config.pub_reconnect_delay_max} is defined, it works as a cap for the above time.
103
+ # @return [void]
104
+ def recover_from_net_error
105
+ attempt = 0
106
+ begin
107
+ @conn.start
108
+ Cuniculus.logger.info("Connection established")
109
+
110
+ @workers.each { |w| w.start!(@conn) }
111
+ rescue *RECOVERABLE_ERRORS => ex
112
+ handle_error(Cuniculus.convert_exception_class(ex, Cuniculus::RMQConnectionError))
113
+ sleep_time = @shutdown ? 1 : [(reconnect_delay * 2**(attempt-1)), reconnect_delay_max].min
114
+ sleep sleep_time
115
+ attempt += 1
116
+
117
+ retry if @shutdown && attempt <= reconnect_delay_max
118
+ retry if reconnect_attempts == :infinite || attempt <= reconnect_attempts
119
+ end
120
+ end
121
+
122
+ # Shutdown workers, giving them time to conclude outstanding tasks.
123
+ #
124
+ # Shutdown is forced after {Config.pub_shutdown_grace_period} seconds.
125
+ #
126
+ # @return [void]
127
+ def shutdown
128
+ Cuniculus.logger.info("Cuniculus: Shutting down dispatcher")
129
+ @shutdown = true
130
+ alive_size = @workers.size
131
+ shutdown_t0 = Cuniculus.mark_time
132
+
133
+ sleep 1 until Cuniculus.mark_time - shutdown_t0 > shutdown_grace_period || @job_queue.empty?
134
+
135
+ until Cuniculus.mark_time - shutdown_t0 > shutdown_grace_period || (alive_size = @workers.select(&:alive?).size) == 0
136
+ sleep 1
137
+ alive_size.times { @job_queue << :shutdown }
138
+ end
139
+
140
+ @dispatcher_chan << :shutdown
141
+ alive_size = @workers.select(&:alive?).size
142
+ return unless alive_size > 0
143
+
144
+ Cuniculus.logger.warn("Cuniculus: Forcing shutdown with #{alive_size} workers remaining")
145
+ describe
146
+ end
147
+
148
+ private
149
+
150
+ def handle_error(e)
151
+ Cuniculus.logger.error("#{e.class.name}: #{e.message}")
152
+ Cuniculus.logger.error(e.backtrace.join("\n")) unless e.backtrace.nil?
153
+ end
154
+ end
155
+ end
156
+
@@ -25,6 +25,10 @@ module Cuniculus
25
25
  def cause
26
26
  wrapped_exception || super
27
27
  end
28
+
29
+ def message
30
+ wrapped_exception&.message || to_s
31
+ end
28
32
  end
29
33
 
30
34
  # Dev note:
@@ -0,0 +1,83 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "bunny"
4
+
5
+ module Cuniculus
6
+ # Each PubWorker maintains a background thread in a loop, fetching jobs reaching
7
+ # its job queue and publishing the payloads to RabbitMQ. They are not instantiated
8
+ # directly, but are rather created and managed by a {Cuniculus::Dispatcher}.
9
+ class PubWorker
10
+ def initialize(config, job_queue, dispatcher_chan)
11
+ @config = config
12
+ @job_queue = job_queue
13
+ @dispatcher_chan = dispatcher_chan
14
+ @mutex = Mutex.new
15
+ @thread = nil
16
+ end
17
+
18
+ # Declares exchanges, and starts a background thread that consumes and publishes messages.
19
+ #
20
+ # If the connection to RabbitMQ it receives is not established, or if it fails to declare
21
+ # the exchanges, the background thread is not started and a message is sent to the
22
+ # dispatcher channel with the current timestamp. The dispatcher is then responsible for
23
+ # trying to set the connection up again and starting each of its workers.
24
+ #
25
+ # @param conn [::Bunny::Session] Connection to RabbitMQ. Expected to be open at this stage.
26
+ def start!(conn)
27
+ return @dispatcher_chan << Cuniculus.mark_time unless conn.open?
28
+
29
+ @channel = sync { conn.create_channel }
30
+ @x = sync { @channel.direct(Cuniculus::CUNICULUS_EXCHANGE, { durable: true }) }
31
+ @dlx = sync { @channel.fanout(Cuniculus::CUNICULUS_DLX_EXCHANGE, { durable: true }) }
32
+ @thread = Thread.new { run }
33
+ rescue Bunny::Exception
34
+ @dispatcher_chan << Cuniculus.mark_time
35
+ end
36
+
37
+ # Whether the background thread is running.
38
+ #
39
+ # @return [Boolean]
40
+ def alive?
41
+ @thread&.alive? || false
42
+ end
43
+
44
+ private
45
+
46
+ # Starts the job consuming loop. This is used internally by `start!` and runs in
47
+ # a background thread. Messages are published to RabbitMQ.
48
+ #
49
+ # The loop is finished if the message `:shutdown` is retrieved from the job queue or
50
+ # if an exception happens while trying to publish a message to RabbitMQ. In the
51
+ # latter case, the job is reinserted into the job queue, and a message with the timestamp
52
+ # is sent into the dispatcher channel, so that it can try restart the connection
53
+ # and the workers again.
54
+ def run
55
+ loop do
56
+ case msg = @job_queue.pop
57
+ when :shutdown
58
+ break
59
+ else
60
+ xname, payload, routing_key = msg
61
+ exchange = if xname == CUNICULUS_DLX_EXCHANGE
62
+ sync { @dlx }
63
+ else
64
+ sync { @x }
65
+ end
66
+ begin
67
+ publish_time = Cuniculus.mark_time
68
+ exchange.publish(payload, { routing_key: routing_key, persistent: true })
69
+ rescue *::Cuniculus::Dispatcher::RECOVERABLE_ERRORS
70
+ @job_queue << [xname, payload, routing_key]
71
+ @dispatcher_chan << publish_time
72
+ break
73
+ end
74
+ end
75
+ end
76
+ sync { @channel.close unless @channel.closed? }
77
+ end
78
+
79
+ def sync(&block)
80
+ @mutex.synchronize(&block)
81
+ end
82
+ end
83
+ end
@@ -6,11 +6,11 @@ module Cuniculus
6
6
 
7
7
  # The minor version of Cuniculus. Bumped for every non-patch level
8
8
  # release.
9
- MINOR = 1
9
+ MINOR = 2
10
10
 
11
11
  # The tiny version of Cuniculus. Usually 0, only bumped for bugfix
12
12
  # releases that fix regressions from previous versions.
13
- TINY = 0
13
+ TINY = 1
14
14
 
15
15
  # The version of Cuniculus you are using, as a string (e.g. "2.11.0")
16
16
  VERSION = [MAJOR, MINOR, TINY].join(".").freeze
@@ -1,82 +1,66 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  require "cuniculus/core"
4
- require "cuniculus/rmq_pool"
4
+ require "cuniculus/exceptions"
5
5
 
6
6
  module Cuniculus
7
7
  module Worker
8
- def self.included(base)
9
- base.extend(ClassMethods)
8
+ DEFAULT_OPTS = { queue: "cun_default" }.freeze
9
+ VALID_OPT_KEYS = %i[queue].freeze
10
10
 
11
- # Dev note:
12
- # The point here is to allow options set via cuniculus_options to be
13
- # inherited by subclasses.
14
- # When reading the options, a subclass will call the singleton method cun_opts.
15
- # If the subclass doesn't redefine this method via a call to cuniculus_options,
16
- # it will still use the definition from its parent class.
17
- base.define_singleton_method("cun_opts=") do |opts|
18
- singleton_class.class_eval do
19
- define_method("cun_opts") { opts }
20
- end
21
- end
11
+ def self.extended(base)
12
+ base.instance_variable_set(:@cun_opts, DEFAULT_OPTS)
13
+ super
22
14
  end
23
15
 
24
- module ClassMethods
25
- DEFAULT_OPTS = { "queue" => "cun_default" }.freeze
26
- VALID_OPT_KEYS = %w[queue].freeze
16
+ def inherited(mod)
17
+ mod.instance_variable_set(:@cun_opts, @cun_opts)
18
+ super
19
+ end
27
20
 
28
- # Read-only cuniculus option values
29
- #
30
- # @return opts [Hash] hash with current values
31
- def cun_opts
32
- DEFAULT_OPTS
33
- end
21
+ attr_reader :cun_opts
34
22
 
35
- # Worker-specific options for running cuniculus.
36
- #
37
- # Note that options set on a worker class are inherited by its subclasses.
38
- #
39
- # @param opts [Hash]
40
- # @option opts [String] "queue" ("cun_default") Name of the underlying RabbitMQ queue.
41
- #
42
- # @example Change the queue name of a worker
43
- # class MyWorker
44
- # include Cuniculus::Worker
45
- #
46
- # cuniculus_options queue: "critical"
47
- #
48
- # def perform
49
- # # run the task
50
- # end
51
- # end
52
- def cuniculus_options(opts)
53
- opts = validate_opts!(opts)
54
- self.cun_opts = opts
55
- end
23
+ # Worker-specific options for running cuniculus.
24
+ #
25
+ # Note that options set on a worker class are inherited by its subclasses.
26
+ #
27
+ # @param opts [Hash]
28
+ # @option opts [String] "queue" ("cun_default") Name of the underlying RabbitMQ queue.
29
+ #
30
+ # @example Change the queue name of a worker
31
+ # class MyWorker
32
+ # include Cuniculus::Worker
33
+ #
34
+ # cuniculus_options queue: "critical"
35
+ #
36
+ # def perform
37
+ # # run the task
38
+ # end
39
+ # end
40
+ def cuniculus_options(opts)
41
+ opts = validate_opts!(opts)
42
+ @cun_opts = opts
43
+ end
56
44
 
57
- def validate_opts!(opts)
58
- raise WorkerOptionsError, "Argument passed to 'cuniculus_options' should be a Hash" unless opts.is_a?(Hash)
59
- opts = opts.transform_keys(&:to_s)
60
- invalid_keys = opts.keys - VALID_OPT_KEYS
61
- raise WorkerOptionsError, "Invalid keys passed to 'cuniculus_options': #{invalid_keys.inspect}" unless invalid_keys.empty?
62
- opts
63
- end
45
+ def validate_opts!(opts)
46
+ raise Cuniculus::WorkerOptionsError, "Argument passed to 'cuniculus_options' should be a Hash" unless opts.is_a?(Hash)
47
+ invalid_keys = opts.keys - VALID_OPT_KEYS
48
+ raise Cuniculus::WorkerOptionsError, "Invalid keys passed to 'cuniculus_options': #{invalid_keys.inspect}" unless invalid_keys.empty?
49
+ opts
50
+ end
64
51
 
65
- def perform_async(*args)
66
- publish({ "class" => self, "args" => args })
67
- end
52
+ def perform_async(*args)
53
+ publish({ "class" => self, "args" => args })
54
+ end
68
55
 
69
- def publish(item)
70
- routing_key = cun_opts["queue"]
71
- payload = normalize_item(item)
72
- Cuniculus::RMQPool.with_exchange do |x|
73
- x.publish(payload, { routing_key: routing_key, persistent: true })
74
- end
75
- end
56
+ def publish(item)
57
+ routing_key = cun_opts[:queue]
58
+ payload = normalize_item(item)
59
+ Cuniculus.enqueue [Cuniculus::CUNICULUS_EXCHANGE, payload, routing_key]
60
+ end
76
61
 
77
- def normalize_item(item)
78
- Cuniculus.dump_job(item)
79
- end
62
+ def normalize_item(item)
63
+ Cuniculus.dump_job(item)
80
64
  end
81
65
  end
82
66
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: cuniculus
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 0.2.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Marcelo Pereira
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2021-03-07 00:00:00.000000000 Z
11
+ date: 2021-07-26 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bunny
@@ -24,20 +24,6 @@ dependencies:
24
24
  - - ">="
25
25
  - !ruby/object:Gem::Version
26
26
  version: 2.15.0
27
- - !ruby/object:Gem::Dependency
28
- name: connection_pool
29
- requirement: !ruby/object:Gem::Requirement
30
- requirements:
31
- - - ">="
32
- - !ruby/object:Gem::Version
33
- version: 2.2.2
34
- type: :runtime
35
- prerelease: false
36
- version_requirements: !ruby/object:Gem::Requirement
37
- requirements:
38
- - - ">="
39
- - !ruby/object:Gem::Version
40
- version: 2.2.2
41
27
  - !ruby/object:Gem::Dependency
42
28
  name: pry
43
29
  requirement: !ruby/object:Gem::Requirement
@@ -182,13 +168,14 @@ files:
182
168
  - lib/cuniculus/config.rb
183
169
  - lib/cuniculus/consumer.rb
184
170
  - lib/cuniculus/core.rb
171
+ - lib/cuniculus/dispatcher.rb
185
172
  - lib/cuniculus/exceptions.rb
186
173
  - lib/cuniculus/job_queue.rb
187
174
  - lib/cuniculus/logger.rb
188
175
  - lib/cuniculus/plugins.rb
189
176
  - lib/cuniculus/plugins/health_check.rb
177
+ - lib/cuniculus/pub_worker.rb
190
178
  - lib/cuniculus/queue_config.rb
191
- - lib/cuniculus/rmq_pool.rb
192
179
  - lib/cuniculus/supervisor.rb
193
180
  - lib/cuniculus/version.rb
194
181
  - lib/cuniculus/worker.rb
@@ -207,7 +194,7 @@ rdoc_options:
207
194
  - "--title"
208
195
  - 'Cuniculus: Background job processing with RabbitMQ'
209
196
  - "--main"
210
- - README.rdoc
197
+ - README.md
211
198
  require_paths:
212
199
  - lib
213
200
  required_ruby_version: !ruby/object:Gem::Requirement
@@ -1,44 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- require "bunny"
4
- require "connection_pool"
5
- require "cuniculus/core"
6
- require "cuniculus/config"
7
-
8
- module Cuniculus
9
- class RMQPool
10
- ENFORCED_CONN_OPTS = {
11
- threaded: false # No need for a reader thread, since this connection is only used for publishing
12
- }.freeze
13
-
14
- class << self
15
- def configure(config)
16
- @config = config
17
- end
18
-
19
- def config
20
- @config ||= Cuniculus::Config.new
21
- end
22
-
23
- def init!
24
- @conn = ::Bunny.new(@config.rabbitmq_opts.merge(ENFORCED_CONN_OPTS))
25
- @conn.start
26
- @channel_pool = ConnectionPool.new(timeout: 1, size: @config.pub_thr_pool_size) do
27
- ch = @conn.create_channel
28
- ch.direct(Cuniculus::CUNICULUS_EXCHANGE, { durable: true })
29
- ch.fanout(Cuniculus::CUNICULUS_DLX_EXCHANGE, { durable: true })
30
- ch
31
- end
32
- end
33
-
34
- def with_exchange(&block)
35
- init! unless @channel_pool
36
- @channel_pool.with do |ch|
37
- block.call(ch.exchanges["cuniculus"])
38
- ensure
39
- ch.open if ch.closed?
40
- end
41
- end
42
- end
43
- end
44
- end