cuniculus 0.1.0 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +64 -51
- data/lib/cuniculus.rb +18 -2
- data/lib/cuniculus/config.rb +65 -7
- data/lib/cuniculus/core.rb +4 -0
- data/lib/cuniculus/dispatcher.rb +156 -0
- data/lib/cuniculus/exceptions.rb +4 -0
- data/lib/cuniculus/pub_worker.rb +83 -0
- data/lib/cuniculus/version.rb +2 -2
- data/lib/cuniculus/worker.rb +48 -64
- metadata +5 -18
- data/lib/cuniculus/rmq_pool.rb +0 -44
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 836705af66bf5ceb94715ce41b70906928611e4b37eaee3e343e4ee8d6a52256
|
4
|
+
data.tar.gz: da7c6168ea3aeddd9896c956f9cb89b5beeda5ded1aa7d663a6e8662eae2100d
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 69ae000142f4d761c10a37e37df70bbc257f28b602df19d7aeb9508df844d1a0165e22502ec8c9a8b6f195d4db0220faf0c7be8900489c12a1a1460ec7220fa7
|
7
|
+
data.tar.gz: 516789a202f6000f4145dfc218f60645437ee9476f129670dd3d464c38084399e97b930b022efee40906519752243caccb526e6d4d5e3714bb268a776de6ab75
|
data/README.md
CHANGED
@@ -1,36 +1,7 @@
|
|
1
|
-
|
1
|
+

|
2
2
|
|
3
|
-
Ruby job queue backed by RabbitMQ. The word _cuniculus_ comes from the scientific name of the European rabbit (Oryctolagus cuniculus).
|
4
|
-
|
5
|
-
## Benchmarks
|
6
|
-
|
7
|
-
The following measurements were performed with the `bin/run_benchmarks` utility, with different command parameters. Run it with `-h` to see its usage.
|
8
|
-
|
9
|
-
To simulate network latency, [Toxiproxy](https://github.com/Shopify/toxiproxy) was used. It needs to be started with `toxiproxy-server` before running the benchmarks.
|
10
3
|
|
11
|
-
|
12
|
-
----------------------:|---------------------:|----------------------:|----------------------:
|
13
|
-
1 | 65535 (max. allowed) | 10225 | 2
|
14
|
-
10 | 65535 (max. allowed) | 9990 | 13
|
15
|
-
1 | 50 | 8051 | 2
|
16
|
-
10 | 50 | 2500 | 13
|
17
|
-
100 | 50 | 481 | 103
|
18
|
-
1 | 25 | 7824 | 2
|
19
|
-
10 | 25 | 1824 | 13
|
20
|
-
50 | 25 | 469 | 53
|
21
|
-
1 | 10 (default) | 5266 | 2
|
22
|
-
10 | 10 (default) | 807 | 13
|
23
|
-
1 | 1 | 481 | 2
|
24
|
-
10 | 1 | 81 | 13
|
25
|
-
|
26
|
-
Additional benchmark parameters:
|
27
|
-
- throughput was measured by consuming 100k jobs;
|
28
|
-
- job latency was averaged over 200 samples;
|
29
|
-
- Ruby 2.7.2 was used.
|
30
|
-
|
31
|
-
Several remarks can be made:
|
32
|
-
- Higher prefetch counts lead to higher throughput, but there are downsides of having it too high; see [this reference](https://www.cloudamqp.com/blog/2017-12-29-part1-rabbitmq-best-practice.html#prefetch) on how to properly tune it.
|
33
|
-
- Network latency has a severe impact on the throughput, and the effect is larger the smaller the prefetch count is.
|
4
|
+
Ruby job queue backed by RabbitMQ. The word _cuniculus_ comes from the scientific name of the European rabbit (Oryctolagus cuniculus).
|
34
5
|
|
35
6
|
## Getting started
|
36
7
|
|
@@ -46,7 +17,7 @@ Create a worker class:
|
|
46
17
|
require 'cuniculus/worker'
|
47
18
|
|
48
19
|
class MyWorker
|
49
|
-
|
20
|
+
extend Cuniculus::Worker
|
50
21
|
|
51
22
|
# the queue name is not explicitly given, so "cun_default" is used.
|
52
23
|
|
@@ -68,23 +39,33 @@ Start the job consumer:
|
|
68
39
|
cuniculus -r my_worker.rb
|
69
40
|
```
|
70
41
|
|
71
|
-
|
42
|
+
## Benchmarks
|
72
43
|
|
73
|
-
|
74
|
-
|
75
|
-
|
76
|
-
|
77
|
-
|
78
|
-
|
79
|
-
|
80
|
-
|
81
|
-
|
82
|
-
|
83
|
-
|
84
|
-
|
85
|
-
|
44
|
+
The following measurements were performed with the `bin/run_benchmarks` utility, with different command parameters. Run it with `-h` to see its usage.
|
45
|
+
|
46
|
+
To simulate network latency, [Toxiproxy](https://github.com/Shopify/toxiproxy) was used. It needs to be started with `toxiproxy-server` before running the benchmarks.
|
47
|
+
|
48
|
+
Network latency (_ms_) | Prefetch count | Throughput (_jobs/s_) | Average latency (_ms_)
|
49
|
+
----------------------:|---------------------:|----------------------:|----------------------:
|
50
|
+
1 | 65535 (max. allowed) | 10225 | 2
|
51
|
+
10 | 65535 (max. allowed) | 9990 | 13
|
52
|
+
1 | 50 | 8051 | 2
|
53
|
+
10 | 50 | 2500 | 13
|
54
|
+
100 | 50 | 481 | 103
|
55
|
+
1 | 10 (default) | 5266 | 2
|
56
|
+
10 | 10 (default) | 807 | 13
|
57
|
+
1 | 1 | 481 | 2
|
58
|
+
10 | 1 | 81 | 13
|
59
|
+
|
60
|
+
Additional benchmark parameters:
|
61
|
+
- throughput was measured by consuming 100k jobs;
|
62
|
+
- job latency was averaged over 200 samples;
|
63
|
+
- Ruby 2.7.2 was used.
|
64
|
+
|
65
|
+
Several remarks can be made:
|
66
|
+
- Higher prefetch counts lead to higher throughput, but there are downsides of having it too high; see [this reference](https://www.cloudamqp.com/blog/2017-12-29-part1-rabbitmq-best-practice.html#prefetch) on how to properly tune it.
|
67
|
+
- Network latency has a severe impact on the throughput, and the effect is larger the smaller the prefetch count is.
|
86
68
|
|
87
|
-
The `-I examples` option adds the `examples/` directory into the load path, and `-r example/init_cuniculus.rb` requires `init_cuniculus.rb` prior to starting the consumer. The latter is where configurations such as that described in the next section should be.
|
88
69
|
|
89
70
|
## Configuration
|
90
71
|
|
@@ -107,13 +88,13 @@ rabbitmq_conn = {
|
|
107
88
|
|
108
89
|
Cuniculus.configure do |cfg|
|
109
90
|
cfg.rabbitmq_opts = rabbitmq_conn
|
110
|
-
cfg.
|
91
|
+
cfg.pub_pool_size = 5 # Only affects job producers
|
111
92
|
cfg.dead_queue_ttl = 1000 * 60 * 60 * 24 * 30 # keep failed jobs for 30 days
|
112
93
|
cfg.add_queue({ name: "critical", durable: true, max_retry: 10, prefetch_count: 1})
|
113
94
|
end
|
114
95
|
```
|
115
96
|
|
116
|
-
To configure the queue used by a worker,
|
97
|
+
To configure the queue used by a worker, use `cuniculus_options`:
|
117
98
|
|
118
99
|
```ruby
|
119
100
|
class MyWorker
|
@@ -126,6 +107,24 @@ class MyWorker
|
|
126
107
|
end
|
127
108
|
```
|
128
109
|
|
110
|
+
## More examples
|
111
|
+
|
112
|
+
There is also a more complete example in the Cuniculus repository itself. To run it, clone the repository, then
|
113
|
+
- start the Ruby and RabbitMQ containers using [Docker Compose](https://docs.docker.com/compose/):
|
114
|
+
```
|
115
|
+
docker-compose up -d
|
116
|
+
```
|
117
|
+
- from within the _cuniculus_ container, produce a job:
|
118
|
+
```
|
119
|
+
ruby -Ilib examples/produce.rb
|
120
|
+
```
|
121
|
+
- also from within the container, start the consumer:
|
122
|
+
```
|
123
|
+
bin/cuniculus -I examples/ -r example/init_cuniculus.rb
|
124
|
+
```
|
125
|
+
|
126
|
+
The `-I examples` option adds the `examples/` directory into the load path, and `-r example/init_cuniculus.rb` requires `init_cuniculus.rb` prior to starting the consumer. The latter is where configurations such as that described in the [configuration section](#configuration) section should be.
|
127
|
+
|
129
128
|
## Error handling
|
130
129
|
|
131
130
|
By default, exceptions raised when consuming a job are logged to STDOUT. This can be overriden with the `Cuniculus.error_handler` method:
|
@@ -133,22 +132,36 @@ By default, exceptions raised when consuming a job are logged to STDOUT. This ca
|
|
133
132
|
```ruby
|
134
133
|
Cuniculus.error_handler do |e|
|
135
134
|
puts "Oh nein! #{e}"
|
135
|
+
LoggingService.send(e)
|
136
136
|
end
|
137
137
|
```
|
138
138
|
|
139
139
|
The method expects a block that will receive an exception, and run in the scope of the Worker instance.
|
140
140
|
|
141
|
+
## Publisher proper shutdown
|
142
|
+
|
143
|
+
When `perform_async` is called, the job is first put into a local (in-memory) queue that is published to RabbitMQ by a worker in a worker pool (the size of which is configured with `config.pub_pool_size`).
|
144
|
+
|
145
|
+
To ensure Cuniculus tries to finish publishing jobs on shutdown, it's important that `Cuniculus.shutdown` is called. Once this method is called, workers have a grace period to publish enqueued jobs, after which the shutdown is forced. The period is set in seconds in `config.pub_shutdown_grace_period` (defaults to 50).
|
146
|
+
|
147
|
+
Example code for the [Puma web server](https://github.com/puma/puma):
|
148
|
+
```ruby
|
149
|
+
on_worker_shutdown do
|
150
|
+
Cuniculus.shutdown
|
151
|
+
end
|
152
|
+
```
|
153
|
+
|
141
154
|
## Retry mechanism
|
142
155
|
|
143
156
|
Retries are enabled by default (with 8 retries) with an exponential backoff, meaning the time between retries increases the more failures happen. The formula for calculating the times between retries can be found in {Cuniculus::QueueConfig}, namely in the `x-message-ttl` line. As an example, the time between the 7th and 8th retries is roughly 29 days.
|
144
157
|
|
145
|
-
Given a
|
158
|
+
Given a queue in the configuration, Cuniculus declares on RabbitMQ the corresponding base queue, in addition to its retry queues. As an example, let's consider the default queue `cun_default`: Cuniculus declares a `cun_default` queue, together with some `cun_default_{n}` queues used for job retries.
|
146
159
|
|
147
160
|
When a job raises an exception, it is placed into the `cun_default_1` queue for the first retry. It stays there for some pre-defined time, and then gets moved back into the `cun_default` queue for execution.
|
148
161
|
|
149
162
|
If it fails again, it gets moved to `cun_default_2`, where it stays for a longer period until it's moved back directly into the `cun_default` queue again.
|
150
163
|
|
151
|
-
This goes on until there are no more retry attempts, in which case the job gets moved into the `cun_dead` queue. It can be then only be moved back into the `cun_default` queue manually; otherwise it is discarded after some time, defined as the {Cuniculus::Config.dead_queue_ttl}, in milliseconds (by default, 180 days).
|
164
|
+
This goes on until there are no more retry attempts, in which case the job gets moved into the `cun_dead` queue. It can be then only be moved back into the `cun_default` queue manually (from RabbitMQ itself, not with Cuniculus); otherwise it is discarded after some time, defined as the {Cuniculus::Config.dead_queue_ttl}, in milliseconds (by default, 180 days).
|
152
165
|
|
153
166
|
Note that if a job cannot even be parsed, it is moved straight to the dead queue, as there's no point in retrying.
|
154
167
|
|
data/lib/cuniculus.rb
CHANGED
@@ -7,7 +7,7 @@ raise "Cuniculus #{Cuniculus.version} does not support Ruby versions below 2.6."
|
|
7
7
|
require "cuniculus/logger"
|
8
8
|
require "cuniculus/config"
|
9
9
|
require "cuniculus/plugins"
|
10
|
-
require "cuniculus/
|
10
|
+
require "cuniculus/dispatcher"
|
11
11
|
require "cuniculus/supervisor"
|
12
12
|
|
13
13
|
# Base definition of the Cuniculus Module
|
@@ -28,7 +28,20 @@ module Cuniculus
|
|
28
28
|
yield cfg
|
29
29
|
cfg.declare!
|
30
30
|
@config = cfg
|
31
|
-
Cuniculus::
|
31
|
+
@dispatcher = Cuniculus::Dispatcher.new(cfg)
|
32
|
+
end
|
33
|
+
|
34
|
+
def self.enqueue(job)
|
35
|
+
dispatcher.job_queue << job
|
36
|
+
dispatcher.start!
|
37
|
+
end
|
38
|
+
|
39
|
+
def self.shutdown
|
40
|
+
dispatcher.shutdown
|
41
|
+
end
|
42
|
+
|
43
|
+
def self.dispatcher
|
44
|
+
@dispatcher ||= Cuniculus::Dispatcher.new(config)
|
32
45
|
end
|
33
46
|
|
34
47
|
# Current config of Cuniculus
|
@@ -64,6 +77,9 @@ module Cuniculus
|
|
64
77
|
def self.error_handler(&block)
|
65
78
|
Cuniculus::Consumer.define_method(:handle_error, &block)
|
66
79
|
Cuniculus::Consumer.instance_eval { private :handle_error }
|
80
|
+
|
81
|
+
Cuniculus::Dispatcher.define_method(:handle_error, &block)
|
82
|
+
Cuniculus::Dispatcher.instance_eval { private :handle_error }
|
67
83
|
end
|
68
84
|
|
69
85
|
# Load a plugin. If plugin is a Module, it is loaded directly.
|
data/lib/cuniculus/config.rb
CHANGED
@@ -1,5 +1,6 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
+
require "logger"
|
3
4
|
require "cuniculus/core"
|
4
5
|
require "cuniculus/exceptions"
|
5
6
|
require "cuniculus/queue_config"
|
@@ -7,14 +8,29 @@ require "cuniculus/queue_config"
|
|
7
8
|
module Cuniculus
|
8
9
|
class Config
|
9
10
|
ENFORCED_CONN_OPTS = {
|
10
|
-
threaded: false # No need for a reader thread, since this connection is only used for
|
11
|
+
threaded: false, # No need for a reader thread, since this connection is only used for declaring exchanges and queues.
|
12
|
+
automatically_recover: false,
|
13
|
+
log_level: ::Logger::ERROR
|
11
14
|
}.freeze
|
12
15
|
|
13
|
-
|
14
|
-
|
16
|
+
attr_reader(
|
17
|
+
:dead_queue_ttl,
|
18
|
+
:exchange_name,
|
19
|
+
:opts,
|
20
|
+
:pub_pool_size,
|
21
|
+
:pub_reconnect_attempts,
|
22
|
+
:pub_reconnect_delay,
|
23
|
+
:pub_reconnect_delay_max,
|
24
|
+
:pub_shutdown_grace_period,
|
25
|
+
:queues
|
26
|
+
)
|
27
|
+
|
28
|
+
attr_accessor :rabbitmq_opts
|
15
29
|
|
16
30
|
def initialize
|
17
31
|
@opts = {}
|
32
|
+
|
33
|
+
# ---- Default values
|
18
34
|
@queues = { "cun_default" => QueueConfig.new({ "name" => "cun_default" }) }
|
19
35
|
@rabbitmq_opts = {
|
20
36
|
host: "127.0.0.1",
|
@@ -23,11 +39,51 @@ module Cuniculus
|
|
23
39
|
pass: "guest",
|
24
40
|
vhost: "/"
|
25
41
|
}
|
26
|
-
@exchange_name =
|
27
|
-
@pub_thr_pool_size = 5
|
42
|
+
@exchange_name = Cuniculus::CUNICULUS_EXCHANGE
|
28
43
|
@dead_queue_ttl = 1000 * 60 * 60 * 24 * 180 # 180 days
|
44
|
+
@pub_reconnect_attempts = :infinite
|
45
|
+
@pub_reconnect_delay = 1.5
|
46
|
+
@pub_reconnect_delay_max = 10
|
47
|
+
@pub_shutdown_grace_period = 50
|
48
|
+
@pub_pool_size = 5
|
49
|
+
## ---- End of default values
|
29
50
|
end
|
30
51
|
|
52
|
+
def dead_queue_ttl=(ttl)
|
53
|
+
raise Cuniculus::ConfigError, "dead_queue_ttl should be a positive integer, given #{ttl.inspect}" if ttl.to_i <= 0
|
54
|
+
@dead_queue_ttl = ttl
|
55
|
+
end
|
56
|
+
|
57
|
+
def exchange_name=(xname)
|
58
|
+
raise Cuniculus::ConfigError, "exchange_name should not be blank" if xname.to_s.empty?
|
59
|
+
@exchange_name = xname
|
60
|
+
end
|
61
|
+
|
62
|
+
def pub_pool_size=(pool_size)
|
63
|
+
raise Cuniculus::ConfigError, "pub_pool_size should be a positive integer, given #{pool_size.inspect}" if pool_size.to_i <= 0
|
64
|
+
@pub_pool_size = pool_size
|
65
|
+
end
|
66
|
+
|
67
|
+
def pub_reconnect_attempts=(attempts)
|
68
|
+
raise Cuniculus::ConfigError, "pub_reconnect_attempts should be either :infinite or a non-negative integer, was given #{attempts.inspect}" if attempts != :infinite && attempts.to_i < 0
|
69
|
+
@pub_reconnect_attempts = attempts
|
70
|
+
end
|
71
|
+
|
72
|
+
def pub_reconnect_delay=(delay)
|
73
|
+
raise Cuniculus::ConfigError, "pub_reconnect_delay should be a non-negative integer, was given #{delay.inspect}" if delay.to_i < 0
|
74
|
+
@pub_reconnect_delay = delay
|
75
|
+
end
|
76
|
+
|
77
|
+
def pub_reconnect_delay_max=(delay)
|
78
|
+
raise Cuniculus::ConfigError, "pub_reconnect_delay_max should be a non-negative integer, was given #{delay.inspect}" if delay.to_i < 0
|
79
|
+
@pub_reconnect_delay_max = delay
|
80
|
+
end
|
81
|
+
|
82
|
+
def pub_shutdown_grace_period=(period)
|
83
|
+
raise Cuniculus::ConfigError, "pub_shutdown_grace_period should be a non-negative integer, was given #{period.inspect}" if period.to_i < 0
|
84
|
+
@pub_shutdown_grace_period = period
|
85
|
+
end
|
86
|
+
|
31
87
|
# Configure an additional queue
|
32
88
|
#
|
33
89
|
# Note that a single call to `add_queue` might lead to the creation of multiple queues on RabbitMQ: one base queue, and an additional queue for every retry attempt.
|
@@ -62,6 +118,9 @@ module Cuniculus
|
|
62
118
|
declare_exchanges!(ch)
|
63
119
|
declare_dead_queue!(ch)
|
64
120
|
@queues.each_value { |q| q.declare!(ch) }
|
121
|
+
conn.close unless conn.closed?
|
122
|
+
rescue Bunny::TCPConnectionFailed => ex
|
123
|
+
raise Cuniculus.convert_exception_class(ex, Cuniculus::RMQConnectionError)
|
65
124
|
end
|
66
125
|
|
67
126
|
# Specify if the default queue `cun_default` should be created.
|
@@ -87,8 +146,7 @@ module Cuniculus
|
|
87
146
|
arguments: {
|
88
147
|
"x-message-ttl" => dead_queue_ttl
|
89
148
|
}
|
90
|
-
).
|
91
|
-
bind(Cuniculus::CUNICULUS_DLX_EXCHANGE)
|
149
|
+
).bind(Cuniculus::CUNICULUS_DLX_EXCHANGE)
|
92
150
|
end
|
93
151
|
end
|
94
152
|
end
|
data/lib/cuniculus/core.rb
CHANGED
@@ -0,0 +1,156 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "bunny"
|
4
|
+
require "cuniculus/pub_worker"
|
5
|
+
|
6
|
+
module Cuniculus
|
7
|
+
# The dispatcher forwards jobs to a worker pool to be published to RabbitMQ.
|
8
|
+
# It holds a RabbitMQ session and, when it receives information from one of its workers
|
9
|
+
# that a network exception occurred, tries to reestablish the connection and restarts
|
10
|
+
# the pool.
|
11
|
+
#
|
12
|
+
# The dispatcher background thread, which monitors for connection errors, is started
|
13
|
+
# whenever the first job is enqueued by a {Cuniculus::Worker}.
|
14
|
+
class Dispatcher
|
15
|
+
ENFORCED_CONN_OPTS = {
|
16
|
+
threaded: false, # No need for a reader thread, since this connection is only used for publishing
|
17
|
+
automatically_recover: false,
|
18
|
+
logger: ::Logger.new(IO::NULL)
|
19
|
+
}.freeze
|
20
|
+
RECOVERABLE_ERRORS = [AMQ::Protocol::Error, ::Bunny::Exception, Errno::ECONNRESET].freeze
|
21
|
+
|
22
|
+
attr_reader :dispatcher_chan, :job_queue, :reconnect_attempts, :reconnect_delay, :reconnect_delay_max, :shutdown_grace_period
|
23
|
+
|
24
|
+
# Instantiates a dispatcher using the passed {Cuniculus::Config}.
|
25
|
+
#
|
26
|
+
# @param config [Cuniculus::Config]
|
27
|
+
def initialize(config)
|
28
|
+
@config = config
|
29
|
+
@conn = nil
|
30
|
+
@job_queue = Queue.new
|
31
|
+
@dispatcher_chan = Queue.new
|
32
|
+
@shutdown = false
|
33
|
+
@workers = config.pub_pool_size.times.map do |i|
|
34
|
+
Cuniculus::PubWorker.new(config, @job_queue, @dispatcher_chan)
|
35
|
+
end
|
36
|
+
@reconnect_attempts = config.pub_reconnect_attempts
|
37
|
+
@reconnect_delay = config.pub_reconnect_delay
|
38
|
+
@reconnect_delay_max = config.pub_reconnect_delay_max
|
39
|
+
@shutdown_grace_period = config.pub_shutdown_grace_period
|
40
|
+
@thread = nil
|
41
|
+
@shutdown = false
|
42
|
+
end
|
43
|
+
|
44
|
+
def describe(log_level = Logger::DEBUG)
|
45
|
+
Cuniculus.logger.info @thread&.backtrace
|
46
|
+
@workers.each do |w|
|
47
|
+
Cuniculus.logger.log(log_level, w.instance_variable_get(:@thread)&.backtrace)
|
48
|
+
end
|
49
|
+
end
|
50
|
+
|
51
|
+
|
52
|
+
# Starts a thread responsible for reestablishing lost RabbitMQ connections and
|
53
|
+
# restarting {Cuniculus::PubWorker}s.
|
54
|
+
#
|
55
|
+
# It keeps track of the last time it had to reconnect, in case it receives outdated
|
56
|
+
# messages of failed connections from workers.
|
57
|
+
#
|
58
|
+
# PubWorkers communicate to it through its `dispatcher_chan` queue.
|
59
|
+
# Depending on the content fetched from the dispatcher channel, it takes different actions:
|
60
|
+
# - when a :shutdown message is received, it waits until current jobs are finished (up to the configured `shutdown_grace_period`) and stops its background thread.
|
61
|
+
# - when a timestamp is received that is smaller than the last reconnect timestamp, the message is ignored
|
62
|
+
# - when the timestamp is larger than the last reconnect timestamp, it tries to reestablish the connection to RabbitMQ and restarts its workers.
|
63
|
+
#
|
64
|
+
# Note that the first time the dispatcher is started, it sends a message to its own background thread with a timestamp to trigger the first connection.
|
65
|
+
def start!
|
66
|
+
return if @shutdown || @thread&.alive?
|
67
|
+
@thread = Thread.new do
|
68
|
+
last_connect_time = 0
|
69
|
+
loop do
|
70
|
+
disconnect_time = @dispatcher_chan.pop
|
71
|
+
break if disconnect_time == :shutdown
|
72
|
+
if disconnect_time > last_connect_time
|
73
|
+
recover_from_net_error
|
74
|
+
last_connect_time = Cuniculus.mark_time
|
75
|
+
end
|
76
|
+
end
|
77
|
+
end
|
78
|
+
@conn = ::Bunny.new(@config.rabbitmq_opts.merge(ENFORCED_CONN_OPTS).merge(session_error_handler: @thread))
|
79
|
+
@dispatcher_chan << Cuniculus.mark_time
|
80
|
+
end
|
81
|
+
|
82
|
+
# Whether its background thread is running.
|
83
|
+
#
|
84
|
+
# @return [Boolean]
|
85
|
+
def alive?
|
86
|
+
@thread&.alive? || false
|
87
|
+
end
|
88
|
+
|
89
|
+
# Starts connection to RabbitMQ followed by starting the workers background threads.
|
90
|
+
#
|
91
|
+
# if it fails to connect, it keeps retrying for a certain number of attempts, defined by
|
92
|
+
# {Config.pub_reconnect_attempts}. For unlimited retries, this value should be set to `:infinite`.
|
93
|
+
#
|
94
|
+
# The time between reconnect attempts follows an exponential backoff formula:
|
95
|
+
#
|
96
|
+
# ```
|
97
|
+
# t = delay * 2^(n-1)
|
98
|
+
# ```
|
99
|
+
#
|
100
|
+
# where n is the attempt number, and delay is defined by {Config.pub_reconnect_delay}.
|
101
|
+
#
|
102
|
+
# If {Config.pub_reconnect_delay_max} is defined, it works as a cap for the above time.
|
103
|
+
# @return [void]
|
104
|
+
def recover_from_net_error
|
105
|
+
attempt = 0
|
106
|
+
begin
|
107
|
+
@conn.start
|
108
|
+
Cuniculus.logger.info("Connection established")
|
109
|
+
|
110
|
+
@workers.each { |w| w.start!(@conn) }
|
111
|
+
rescue *RECOVERABLE_ERRORS => ex
|
112
|
+
handle_error(Cuniculus.convert_exception_class(ex, Cuniculus::RMQConnectionError))
|
113
|
+
sleep_time = @shutdown ? 1 : [(reconnect_delay * 2**(attempt-1)), reconnect_delay_max].min
|
114
|
+
sleep sleep_time
|
115
|
+
attempt += 1
|
116
|
+
|
117
|
+
retry if @shutdown && attempt <= reconnect_delay_max
|
118
|
+
retry if reconnect_attempts == :infinite || attempt <= reconnect_attempts
|
119
|
+
end
|
120
|
+
end
|
121
|
+
|
122
|
+
# Shutdown workers, giving them time to conclude outstanding tasks.
|
123
|
+
#
|
124
|
+
# Shutdown is forced after {Config.pub_shutdown_grace_period} seconds.
|
125
|
+
#
|
126
|
+
# @return [void]
|
127
|
+
def shutdown
|
128
|
+
Cuniculus.logger.info("Cuniculus: Shutting down dispatcher")
|
129
|
+
@shutdown = true
|
130
|
+
alive_size = @workers.size
|
131
|
+
shutdown_t0 = Cuniculus.mark_time
|
132
|
+
|
133
|
+
sleep 1 until Cuniculus.mark_time - shutdown_t0 > shutdown_grace_period || @job_queue.empty?
|
134
|
+
|
135
|
+
until Cuniculus.mark_time - shutdown_t0 > shutdown_grace_period || (alive_size = @workers.select(&:alive?).size) == 0
|
136
|
+
sleep 1
|
137
|
+
alive_size.times { @job_queue << :shutdown }
|
138
|
+
end
|
139
|
+
|
140
|
+
@dispatcher_chan << :shutdown
|
141
|
+
alive_size = @workers.select(&:alive?).size
|
142
|
+
return unless alive_size > 0
|
143
|
+
|
144
|
+
Cuniculus.logger.warn("Cuniculus: Forcing shutdown with #{alive_size} workers remaining")
|
145
|
+
describe
|
146
|
+
end
|
147
|
+
|
148
|
+
private
|
149
|
+
|
150
|
+
def handle_error(e)
|
151
|
+
Cuniculus.logger.error("#{e.class.name}: #{e.message}")
|
152
|
+
Cuniculus.logger.error(e.backtrace.join("\n")) unless e.backtrace.nil?
|
153
|
+
end
|
154
|
+
end
|
155
|
+
end
|
156
|
+
|
data/lib/cuniculus/exceptions.rb
CHANGED
@@ -0,0 +1,83 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "bunny"
|
4
|
+
|
5
|
+
module Cuniculus
|
6
|
+
# Each PubWorker maintains a background thread in a loop, fetching jobs reaching
|
7
|
+
# its job queue and publishing the payloads to RabbitMQ. They are not instantiated
|
8
|
+
# directly, but are rather created and managed by a {Cuniculus::Dispatcher}.
|
9
|
+
class PubWorker
|
10
|
+
def initialize(config, job_queue, dispatcher_chan)
|
11
|
+
@config = config
|
12
|
+
@job_queue = job_queue
|
13
|
+
@dispatcher_chan = dispatcher_chan
|
14
|
+
@mutex = Mutex.new
|
15
|
+
@thread = nil
|
16
|
+
end
|
17
|
+
|
18
|
+
# Declares exchanges, and starts a background thread that consumes and publishes messages.
|
19
|
+
#
|
20
|
+
# If the connection to RabbitMQ it receives is not established, or if it fails to declare
|
21
|
+
# the exchanges, the background thread is not started and a message is sent to the
|
22
|
+
# dispatcher channel with the current timestamp. The dispatcher is then responsible for
|
23
|
+
# trying to set the connection up again and starting each of its workers.
|
24
|
+
#
|
25
|
+
# @param conn [::Bunny::Session] Connection to RabbitMQ. Expected to be open at this stage.
|
26
|
+
def start!(conn)
|
27
|
+
return @dispatcher_chan << Cuniculus.mark_time unless conn.open?
|
28
|
+
|
29
|
+
@channel = sync { conn.create_channel }
|
30
|
+
@x = sync { @channel.direct(Cuniculus::CUNICULUS_EXCHANGE, { durable: true }) }
|
31
|
+
@dlx = sync { @channel.fanout(Cuniculus::CUNICULUS_DLX_EXCHANGE, { durable: true }) }
|
32
|
+
@thread = Thread.new { run }
|
33
|
+
rescue Bunny::Exception
|
34
|
+
@dispatcher_chan << Cuniculus.mark_time
|
35
|
+
end
|
36
|
+
|
37
|
+
# Whether the background thread is running.
|
38
|
+
#
|
39
|
+
# @return [Boolean]
|
40
|
+
def alive?
|
41
|
+
@thread&.alive? || false
|
42
|
+
end
|
43
|
+
|
44
|
+
private
|
45
|
+
|
46
|
+
# Starts the job consuming loop. This is used internally by `start!` and runs in
|
47
|
+
# a background thread. Messages are published to RabbitMQ.
|
48
|
+
#
|
49
|
+
# The loop is finished if the message `:shutdown` is retrieved from the job queue or
|
50
|
+
# if an exception happens while trying to publish a message to RabbitMQ. In the
|
51
|
+
# latter case, the job is reinserted into the job queue, and a message with the timestamp
|
52
|
+
# is sent into the dispatcher channel, so that it can try restart the connection
|
53
|
+
# and the workers again.
|
54
|
+
def run
|
55
|
+
loop do
|
56
|
+
case msg = @job_queue.pop
|
57
|
+
when :shutdown
|
58
|
+
break
|
59
|
+
else
|
60
|
+
xname, payload, routing_key = msg
|
61
|
+
exchange = if xname == CUNICULUS_DLX_EXCHANGE
|
62
|
+
sync { @dlx }
|
63
|
+
else
|
64
|
+
sync { @x }
|
65
|
+
end
|
66
|
+
begin
|
67
|
+
publish_time = Cuniculus.mark_time
|
68
|
+
exchange.publish(payload, { routing_key: routing_key, persistent: true })
|
69
|
+
rescue *::Cuniculus::Dispatcher::RECOVERABLE_ERRORS
|
70
|
+
@job_queue << [xname, payload, routing_key]
|
71
|
+
@dispatcher_chan << publish_time
|
72
|
+
break
|
73
|
+
end
|
74
|
+
end
|
75
|
+
end
|
76
|
+
sync { @channel.close unless @channel.closed? }
|
77
|
+
end
|
78
|
+
|
79
|
+
def sync(&block)
|
80
|
+
@mutex.synchronize(&block)
|
81
|
+
end
|
82
|
+
end
|
83
|
+
end
|
data/lib/cuniculus/version.rb
CHANGED
@@ -6,11 +6,11 @@ module Cuniculus
|
|
6
6
|
|
7
7
|
# The minor version of Cuniculus. Bumped for every non-patch level
|
8
8
|
# release.
|
9
|
-
MINOR =
|
9
|
+
MINOR = 2
|
10
10
|
|
11
11
|
# The tiny version of Cuniculus. Usually 0, only bumped for bugfix
|
12
12
|
# releases that fix regressions from previous versions.
|
13
|
-
TINY =
|
13
|
+
TINY = 1
|
14
14
|
|
15
15
|
# The version of Cuniculus you are using, as a string (e.g. "2.11.0")
|
16
16
|
VERSION = [MAJOR, MINOR, TINY].join(".").freeze
|
data/lib/cuniculus/worker.rb
CHANGED
@@ -1,82 +1,66 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
require "cuniculus/core"
|
4
|
-
require "cuniculus/
|
4
|
+
require "cuniculus/exceptions"
|
5
5
|
|
6
6
|
module Cuniculus
|
7
7
|
module Worker
|
8
|
-
|
9
|
-
|
8
|
+
DEFAULT_OPTS = { queue: "cun_default" }.freeze
|
9
|
+
VALID_OPT_KEYS = %i[queue].freeze
|
10
10
|
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
# When reading the options, a subclass will call the singleton method cun_opts.
|
15
|
-
# If the subclass doesn't redefine this method via a call to cuniculus_options,
|
16
|
-
# it will still use the definition from its parent class.
|
17
|
-
base.define_singleton_method("cun_opts=") do |opts|
|
18
|
-
singleton_class.class_eval do
|
19
|
-
define_method("cun_opts") { opts }
|
20
|
-
end
|
21
|
-
end
|
11
|
+
def self.extended(base)
|
12
|
+
base.instance_variable_set(:@cun_opts, DEFAULT_OPTS)
|
13
|
+
super
|
22
14
|
end
|
23
15
|
|
24
|
-
|
25
|
-
|
26
|
-
|
16
|
+
def inherited(mod)
|
17
|
+
mod.instance_variable_set(:@cun_opts, @cun_opts)
|
18
|
+
super
|
19
|
+
end
|
27
20
|
|
28
|
-
|
29
|
-
#
|
30
|
-
# @return opts [Hash] hash with current values
|
31
|
-
def cun_opts
|
32
|
-
DEFAULT_OPTS
|
33
|
-
end
|
21
|
+
attr_reader :cun_opts
|
34
22
|
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
|
54
|
-
|
55
|
-
|
23
|
+
# Worker-specific options for running cuniculus.
|
24
|
+
#
|
25
|
+
# Note that options set on a worker class are inherited by its subclasses.
|
26
|
+
#
|
27
|
+
# @param opts [Hash]
|
28
|
+
# @option opts [String] "queue" ("cun_default") Name of the underlying RabbitMQ queue.
|
29
|
+
#
|
30
|
+
# @example Change the queue name of a worker
|
31
|
+
# class MyWorker
|
32
|
+
# include Cuniculus::Worker
|
33
|
+
#
|
34
|
+
# cuniculus_options queue: "critical"
|
35
|
+
#
|
36
|
+
# def perform
|
37
|
+
# # run the task
|
38
|
+
# end
|
39
|
+
# end
|
40
|
+
def cuniculus_options(opts)
|
41
|
+
opts = validate_opts!(opts)
|
42
|
+
@cun_opts = opts
|
43
|
+
end
|
56
44
|
|
57
|
-
|
58
|
-
|
59
|
-
|
60
|
-
|
61
|
-
|
62
|
-
|
63
|
-
end
|
45
|
+
def validate_opts!(opts)
|
46
|
+
raise Cuniculus::WorkerOptionsError, "Argument passed to 'cuniculus_options' should be a Hash" unless opts.is_a?(Hash)
|
47
|
+
invalid_keys = opts.keys - VALID_OPT_KEYS
|
48
|
+
raise Cuniculus::WorkerOptionsError, "Invalid keys passed to 'cuniculus_options': #{invalid_keys.inspect}" unless invalid_keys.empty?
|
49
|
+
opts
|
50
|
+
end
|
64
51
|
|
65
|
-
|
66
|
-
|
67
|
-
|
52
|
+
def perform_async(*args)
|
53
|
+
publish({ "class" => self, "args" => args })
|
54
|
+
end
|
68
55
|
|
69
|
-
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
|
74
|
-
end
|
75
|
-
end
|
56
|
+
def publish(item)
|
57
|
+
routing_key = cun_opts[:queue]
|
58
|
+
payload = normalize_item(item)
|
59
|
+
Cuniculus.enqueue [Cuniculus::CUNICULUS_EXCHANGE, payload, routing_key]
|
60
|
+
end
|
76
61
|
|
77
|
-
|
78
|
-
|
79
|
-
end
|
62
|
+
def normalize_item(item)
|
63
|
+
Cuniculus.dump_job(item)
|
80
64
|
end
|
81
65
|
end
|
82
66
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: cuniculus
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1
|
4
|
+
version: 0.2.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Marcelo Pereira
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-07-26 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bunny
|
@@ -24,20 +24,6 @@ dependencies:
|
|
24
24
|
- - ">="
|
25
25
|
- !ruby/object:Gem::Version
|
26
26
|
version: 2.15.0
|
27
|
-
- !ruby/object:Gem::Dependency
|
28
|
-
name: connection_pool
|
29
|
-
requirement: !ruby/object:Gem::Requirement
|
30
|
-
requirements:
|
31
|
-
- - ">="
|
32
|
-
- !ruby/object:Gem::Version
|
33
|
-
version: 2.2.2
|
34
|
-
type: :runtime
|
35
|
-
prerelease: false
|
36
|
-
version_requirements: !ruby/object:Gem::Requirement
|
37
|
-
requirements:
|
38
|
-
- - ">="
|
39
|
-
- !ruby/object:Gem::Version
|
40
|
-
version: 2.2.2
|
41
27
|
- !ruby/object:Gem::Dependency
|
42
28
|
name: pry
|
43
29
|
requirement: !ruby/object:Gem::Requirement
|
@@ -182,13 +168,14 @@ files:
|
|
182
168
|
- lib/cuniculus/config.rb
|
183
169
|
- lib/cuniculus/consumer.rb
|
184
170
|
- lib/cuniculus/core.rb
|
171
|
+
- lib/cuniculus/dispatcher.rb
|
185
172
|
- lib/cuniculus/exceptions.rb
|
186
173
|
- lib/cuniculus/job_queue.rb
|
187
174
|
- lib/cuniculus/logger.rb
|
188
175
|
- lib/cuniculus/plugins.rb
|
189
176
|
- lib/cuniculus/plugins/health_check.rb
|
177
|
+
- lib/cuniculus/pub_worker.rb
|
190
178
|
- lib/cuniculus/queue_config.rb
|
191
|
-
- lib/cuniculus/rmq_pool.rb
|
192
179
|
- lib/cuniculus/supervisor.rb
|
193
180
|
- lib/cuniculus/version.rb
|
194
181
|
- lib/cuniculus/worker.rb
|
@@ -207,7 +194,7 @@ rdoc_options:
|
|
207
194
|
- "--title"
|
208
195
|
- 'Cuniculus: Background job processing with RabbitMQ'
|
209
196
|
- "--main"
|
210
|
-
- README.
|
197
|
+
- README.md
|
211
198
|
require_paths:
|
212
199
|
- lib
|
213
200
|
required_ruby_version: !ruby/object:Gem::Requirement
|
data/lib/cuniculus/rmq_pool.rb
DELETED
@@ -1,44 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
require "bunny"
|
4
|
-
require "connection_pool"
|
5
|
-
require "cuniculus/core"
|
6
|
-
require "cuniculus/config"
|
7
|
-
|
8
|
-
module Cuniculus
|
9
|
-
class RMQPool
|
10
|
-
ENFORCED_CONN_OPTS = {
|
11
|
-
threaded: false # No need for a reader thread, since this connection is only used for publishing
|
12
|
-
}.freeze
|
13
|
-
|
14
|
-
class << self
|
15
|
-
def configure(config)
|
16
|
-
@config = config
|
17
|
-
end
|
18
|
-
|
19
|
-
def config
|
20
|
-
@config ||= Cuniculus::Config.new
|
21
|
-
end
|
22
|
-
|
23
|
-
def init!
|
24
|
-
@conn = ::Bunny.new(@config.rabbitmq_opts.merge(ENFORCED_CONN_OPTS))
|
25
|
-
@conn.start
|
26
|
-
@channel_pool = ConnectionPool.new(timeout: 1, size: @config.pub_thr_pool_size) do
|
27
|
-
ch = @conn.create_channel
|
28
|
-
ch.direct(Cuniculus::CUNICULUS_EXCHANGE, { durable: true })
|
29
|
-
ch.fanout(Cuniculus::CUNICULUS_DLX_EXCHANGE, { durable: true })
|
30
|
-
ch
|
31
|
-
end
|
32
|
-
end
|
33
|
-
|
34
|
-
def with_exchange(&block)
|
35
|
-
init! unless @channel_pool
|
36
|
-
@channel_pool.with do |ch|
|
37
|
-
block.call(ch.exchanges["cuniculus"])
|
38
|
-
ensure
|
39
|
-
ch.open if ch.closed?
|
40
|
-
end
|
41
|
-
end
|
42
|
-
end
|
43
|
-
end
|
44
|
-
end
|