cuniculus 0.0.1 → 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/LICENSE +1 -1
- data/README.md +67 -2
- data/lib/cuniculus.rb +2 -0
- data/lib/cuniculus/config.rb +33 -0
- data/lib/cuniculus/consumer.rb +0 -1
- data/lib/cuniculus/core.rb +1 -1
- data/lib/cuniculus/exceptions.rb +20 -5
- data/lib/cuniculus/job_queue.rb +1 -1
- data/lib/cuniculus/plugins/health_check.rb +41 -38
- data/lib/cuniculus/queue_config.rb +19 -14
- data/lib/cuniculus/supervisor.rb +2 -2
- data/lib/cuniculus/version.rb +2 -2
- data/lib/cuniculus/worker.rb +53 -1
- metadata +44 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: ca0cc8cbcc9cc9607488d5c7c88903548fe4106799e6517e82b2efb5824a9971
|
4
|
+
data.tar.gz: 6a0589c882ae8de39a0192e2e5b54c537a709fbe6ca496ce3b999d3a03ff6da1
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 73fc5ea97169bdbdf5a72b95dec76e2b529a13c6b308a3c3c9c947d891bc3845d48ad5b4344820d0f1fed8cbed8b0bb1ffac3a4c3ea043ca0cb3c1737eb8014d
|
7
|
+
data.tar.gz: 7dbdd0c30c1dc808457fc408506aade75b65de83f5318799f8b7a5aab4c0891ed71fbb927e58eaa70b099442790b1c1d20b06d34d12a53967def89052f068887
|
data/LICENSE
CHANGED
data/README.md
CHANGED
@@ -2,6 +2,36 @@
|
|
2
2
|
|
3
3
|
Ruby job queue backed by RabbitMQ. The word _cuniculus_ comes from the scientific name of the European rabbit (Oryctolagus cuniculus).
|
4
4
|
|
5
|
+
## Benchmarks
|
6
|
+
|
7
|
+
The following measurements were performed with the `bin/run_benchmarks` utility, with different command parameters. Run it with `-h` to see its usage.
|
8
|
+
|
9
|
+
To simulate network latency, [Toxiproxy](https://github.com/Shopify/toxiproxy) was used. It needs to be started with `toxiproxy-server` before running the benchmarks.
|
10
|
+
|
11
|
+
Network latency (_ms_) | Prefetch count | Throughput (_jobs/s_) | Average latency (_ms_)
|
12
|
+
----------------------:|---------------------:|----------------------:|----------------------:
|
13
|
+
1 | 65535 (max. allowed) | 10225 | 2
|
14
|
+
10 | 65535 (max. allowed) | 9990 | 13
|
15
|
+
1 | 50 | 8051 | 2
|
16
|
+
10 | 50 | 2500 | 13
|
17
|
+
100 | 50 | 481 | 103
|
18
|
+
1 | 25 | 7824 | 2
|
19
|
+
10 | 25 | 1824 | 13
|
20
|
+
50 | 25 | 469 | 53
|
21
|
+
1 | 10 (default) | 5266 | 2
|
22
|
+
10 | 10 (default) | 807 | 13
|
23
|
+
1 | 1 | 481 | 2
|
24
|
+
10 | 1 | 81 | 13
|
25
|
+
|
26
|
+
Additional benchmark parameters:
|
27
|
+
- throughput was measured by consuming 100k jobs;
|
28
|
+
- job latency was averaged over 200 samples;
|
29
|
+
- Ruby 2.7.2 was used.
|
30
|
+
|
31
|
+
Several remarks can be made:
|
32
|
+
- Higher prefetch counts lead to higher throughput, but there are downsides of having it too high; see [this reference](https://www.cloudamqp.com/blog/2017-12-29-part1-rabbitmq-best-practice.html#prefetch) on how to properly tune it.
|
33
|
+
- Network latency has a severe impact on the throughput, and the effect is larger the smaller the prefetch count is.
|
34
|
+
|
5
35
|
## Getting started
|
6
36
|
|
7
37
|
```sh
|
@@ -18,6 +48,8 @@ require 'cuniculus/worker'
|
|
18
48
|
class MyWorker
|
19
49
|
include Cuniculus::Worker
|
20
50
|
|
51
|
+
# the queue name is not explicitly given, so "cun_default" is used.
|
52
|
+
|
21
53
|
def perform(arg1, arg2)
|
22
54
|
puts "Processing:"
|
23
55
|
puts "arg1: #{arg1.inspect}"
|
@@ -52,6 +84,8 @@ There is also a more complete example in the Cuniculus repository itself. To run
|
|
52
84
|
bin/cuniculus -I examples/ -r example/init_cuniculus.rb
|
53
85
|
```
|
54
86
|
|
87
|
+
The `-I examples` option adds the `examples/` directory into the load path, and `-r example/init_cuniculus.rb` requires `init_cuniculus.rb` prior to starting the consumer. The latter is where configurations such as that described in the next section should be.
|
88
|
+
|
55
89
|
## Configuration
|
56
90
|
|
57
91
|
Configuration is done through code, using `Cuniculus.configure`.
|
@@ -75,6 +109,20 @@ Cuniculus.configure do |cfg|
|
|
75
109
|
cfg.rabbitmq_opts = rabbitmq_conn
|
76
110
|
cfg.pub_thr_pool_size = 5 # Only affects job producers
|
77
111
|
cfg.dead_queue_ttl = 1000 * 60 * 60 * 24 * 30 # keep failed jobs for 30 days
|
112
|
+
cfg.add_queue({ name: "critical", durable: true, max_retry: 10, prefetch_count: 1})
|
113
|
+
end
|
114
|
+
```
|
115
|
+
|
116
|
+
To configure the queue used by a worker, used `cuniculus_options`:
|
117
|
+
|
118
|
+
```ruby
|
119
|
+
class MyWorker
|
120
|
+
include Cuniculus::Worker
|
121
|
+
|
122
|
+
cuniculus_options queue: "critical"
|
123
|
+
def perform
|
124
|
+
# code
|
125
|
+
end
|
78
126
|
end
|
79
127
|
```
|
80
128
|
|
@@ -92,15 +140,32 @@ The method expects a block that will receive an exception, and run in the scope
|
|
92
140
|
|
93
141
|
## Retry mechanism
|
94
142
|
|
95
|
-
|
143
|
+
Retries are enabled by default (with 8 retries) with an exponential backoff, meaning the time between retries increases the more failures happen. The formula for calculating the times between retries can be found in {Cuniculus::QueueConfig}, namely in the `x-message-ttl` line. As an example, the time between the 7th and 8th retries is roughly 29 days.
|
144
|
+
|
145
|
+
Given a declared queue in the configuration, Cuniculus starts on RabbitMQ the corresponding base queue, in addition to its retry queues. As an example, let's consider the default queue `cun_default`: Cuniculus declares a `cun_default` queue, together with some `cun_default_{n}` queues used for job retries.
|
146
|
+
|
96
147
|
When a job raises an exception, it is placed into the `cun_default_1` queue for the first retry. It stays there for some pre-defined time, and then gets moved back into the `cun_default` queue for execution.
|
97
148
|
|
98
149
|
If it fails again, it gets moved to `cun_default_2`, where it stays for a longer period until it's moved back directly into the `cun_default` queue again.
|
99
150
|
|
100
|
-
This goes on until there are no more retry attempts, in which case the job gets moved into the `cun_dead` queue. It can be then only be moved back into the `cun_default` queue manually; otherwise it is discarded after some time, defined as the
|
151
|
+
This goes on until there are no more retry attempts, in which case the job gets moved into the `cun_dead` queue. It can be then only be moved back into the `cun_default` queue manually; otherwise it is discarded after some time, defined as the {Cuniculus::Config.dead_queue_ttl}, in milliseconds (by default, 180 days).
|
101
152
|
|
102
153
|
Note that if a job cannot even be parsed, it is moved straight to the dead queue, as there's no point in retrying.
|
103
154
|
|
155
|
+
## Health check plugin
|
156
|
+
|
157
|
+
Cuniculus ships with a health check plugin. When enabled, a Rack server is started (therefore the [Rack](https://github.com/rack/rack) gem is required, as well as the used handler), which responds with `200 OK` upon receiving a request in the configured port and path.
|
158
|
+
|
159
|
+
Enable it with `Cuniculus.plugin(:health_check)`, which binds the server to `0.0.0.0:3000`, listening on the `/healthcheck` path. To configure the server, pass additional options:
|
160
|
+
|
161
|
+
```ruby
|
162
|
+
Cuniculus.plugin(:health_check, { "bind_to" => "127.0.0.1", "port" => 3003, "path" => "ping" })
|
163
|
+
```
|
164
|
+
|
165
|
+
Check {Cuniculus::Plugins::HealthCheck} for further details.
|
166
|
+
|
167
|
+
_Note that the default handler "webrick" is not bundled by default with Ruby 3 and needs to be installed separately, if it is to be used._
|
168
|
+
|
104
169
|
## How it works
|
105
170
|
|
106
171
|
Cuniculus code and conventions are very much inspired by another Ruby job queue library: [Sidekiq](https://github.com/mperham/sidekiq).
|
data/lib/cuniculus.rb
CHANGED
@@ -14,12 +14,14 @@ require "cuniculus/supervisor"
|
|
14
14
|
module Cuniculus
|
15
15
|
|
16
16
|
# Configure Cuniculus.
|
17
|
+
# Check {Cuniculus::Config} for the available options.
|
17
18
|
#
|
18
19
|
# @yield [Cuniculus::Config]
|
19
20
|
#
|
20
21
|
# @example Change RabbitMQ connection details.
|
21
22
|
# Cuniculus.configure do |cfg|
|
22
23
|
# cfg.rabbitmq_opts = { host: 'rmq.mycompany.com', user: 'guest', pass: 'guest' }
|
24
|
+
# cfg.add_queue({ name: "new_queue", max_retry: 4 })
|
23
25
|
# end
|
24
26
|
def self.configure
|
25
27
|
cfg = Cuniculus::Config.new
|
data/lib/cuniculus/config.rb
CHANGED
@@ -1,6 +1,7 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
require "cuniculus/core"
|
4
|
+
require "cuniculus/exceptions"
|
4
5
|
require "cuniculus/queue_config"
|
5
6
|
|
6
7
|
module Cuniculus
|
@@ -23,9 +24,37 @@ module Cuniculus
|
|
23
24
|
vhost: "/"
|
24
25
|
}
|
25
26
|
@exchange_name = "cuniculus"
|
27
|
+
@pub_thr_pool_size = 5
|
26
28
|
@dead_queue_ttl = 1000 * 60 * 60 * 24 * 180 # 180 days
|
27
29
|
end
|
28
30
|
|
31
|
+
# Configure an additional queue
|
32
|
+
#
|
33
|
+
# Note that a single call to `add_queue` might lead to the creation of multiple queues on RabbitMQ: one base queue, and an additional queue for every retry attempt.
|
34
|
+
# For example, with a queue named `"test"` with `max_retry` set to `4`, 5 queues are created in RabbitMQ.
|
35
|
+
#
|
36
|
+
# For tuning `prefetch_count`, refer to [this guide](https://www.cloudamqp.com/blog/2017-12-29-part1-rabbitmq-best-practice.html#prefetch).
|
37
|
+
#
|
38
|
+
# If a queue already exists in RabbitMQ, and an attempt is done to add it again through `add_queue`, nothing happens, except if the options passed to `add_queue` conflict with the existing queue. For example if a queue exists that is durable, and `add_queue` is called with `"durable" => false`, a `Cuniculus::RMQQueueConfigurationConflict` is raised. To redeclare a queue with conflicting configurations, the original queue has first to be removed from RabbitMQ manually. This can be done, for example, through the management console.
|
39
|
+
#
|
40
|
+
# @param qopts [Hash] Queue config options.
|
41
|
+
# @option qopts [String] "name" Name of the queue.
|
42
|
+
# @option qopts [Boolean] "durable" (true) Whether queue is declared as durable in RabbitMQ. Jobs in non-durable queues may be lost if the RabbitMQ goes down.
|
43
|
+
# @option qopts [Integer] "max_retry" (8) Number of retries for failed jobs in this queue.
|
44
|
+
# @option qopts [Integer] "prefetch_count" (10) Prefetch count used when consuming jobs from this queue.
|
45
|
+
# @option qopts [Integer] "thread_pool_size" (5) Thread pool size for receiving jobs.
|
46
|
+
#
|
47
|
+
# @example Add queue named "critical"
|
48
|
+
# Cuniculus.configure do |cfg|
|
49
|
+
# cfg.add_queue({ name: "critical", max_retry: 10 })
|
50
|
+
# end
|
51
|
+
def add_queue(qopts)
|
52
|
+
qopts = qopts.transform_keys(&:to_s)
|
53
|
+
qname = qopts["name"].to_s
|
54
|
+
raise Cuniculus::ConfigError, "Missing 'name' key in queue configuration hash" if qname.strip.empty?
|
55
|
+
@queues[qname] = QueueConfig.new(qopts)
|
56
|
+
end
|
57
|
+
|
29
58
|
def declare!
|
30
59
|
conn = ::Bunny.new(rabbitmq_opts.merge(ENFORCED_CONN_OPTS))
|
31
60
|
conn.start
|
@@ -35,6 +64,10 @@ module Cuniculus
|
|
35
64
|
@queues.each_value { |q| q.declare!(ch) }
|
36
65
|
end
|
37
66
|
|
67
|
+
# Specify if the default queue `cun_default` should be created.
|
68
|
+
# `cun_default` is used by workers that don't explicitly specify a queue with `cuniculus_options queue: "another_queue"`.
|
69
|
+
#
|
70
|
+
# @param bool [Boolean] If false, queue `cun_default` is not created. Defaults to `true`.
|
38
71
|
def default_queue=(bool)
|
39
72
|
@queues.delete("cun_default") unless bool
|
40
73
|
end
|
data/lib/cuniculus/consumer.rb
CHANGED
@@ -17,7 +17,6 @@ module Cuniculus
|
|
17
17
|
|
18
18
|
def start
|
19
19
|
@exchange = channel.direct(Cuniculus::CUNICULUS_EXCHANGE, { durable: true })
|
20
|
-
# channel.direct(Cuniculus::CUNICULUS_DLX_EXCHANGE, { durable: true })
|
21
20
|
@job_queue = queue_config.declare!(channel)
|
22
21
|
@_consumer = job_queue.subscribe(manual_ack: true, block: false) do |delivery_info, properties, payload|
|
23
22
|
run_job(delivery_info, properties, payload)
|
data/lib/cuniculus/core.rb
CHANGED
@@ -23,7 +23,7 @@ module Cuniculus
|
|
23
23
|
# the message and backtrace of `exception`.
|
24
24
|
#
|
25
25
|
# @param exception [Exception] The exception being wrapped
|
26
|
-
# @param [Cuniculus::Error] The subclass of `Cuniculus::Error`
|
26
|
+
# @param klass [Cuniculus::Error] The subclass of `Cuniculus::Error`
|
27
27
|
#
|
28
28
|
# @return [Cuniculus::Error] An instance of the input `Cuniculus::Error`
|
29
29
|
def convert_exception_class(exception, klass)
|
data/lib/cuniculus/exceptions.rb
CHANGED
@@ -5,12 +5,14 @@ module Cuniculus
|
|
5
5
|
#
|
6
6
|
# * `Cuniculus::Error`: Default exception raised by Cuniculus.
|
7
7
|
# All exceptions classes defined by Cuniculus descend from this class.
|
8
|
-
# * `Cuniculus::
|
9
|
-
#
|
8
|
+
# * `Cuniculus::BadlyFormattedPayload`: A Cuniculus consumer received an
|
9
|
+
# improperly formatted job message.
|
10
|
+
# * `Cuniculus::ConfigError`: Incorrect configuration passed to Cuniculus.
|
11
|
+
# * `Cuniculus::RMQConnectionError`: Unable to connect to RabbitMQ.
|
12
|
+
# * `Cuniculus::RMQQueueConfigurationConflict`: The queue configuration
|
10
13
|
# given to Cuniculus conflicts with the current configuration of the same
|
11
14
|
# existing queue in RabbitMQ.
|
12
|
-
# * `Cuniculus::
|
13
|
-
# improperly formatted job message.
|
15
|
+
# * `Cuniculus::WorkerOptionsError`: Invalid options passed to cuniculus_options.
|
14
16
|
|
15
17
|
class Error < ::StandardError
|
16
18
|
# If the Cuniculus exception wraps an underlying exception, the latter
|
@@ -25,6 +27,18 @@ module Cuniculus
|
|
25
27
|
end
|
26
28
|
end
|
27
29
|
|
30
|
+
# Dev note:
|
31
|
+
# As explained [here](https://github.com/jeremyevans/sequel/commit/24681efad0fec48195e43801c224bf18cdc8be13#diff-64cd7b67eccdc6dfa69c23b3b19f34e318f9e6827c5dee5f6e845b2993ab035c), empty classes created
|
32
|
+
# with `Class.new` require about 200 bytes less memory than ones created as `class MyClass; end`.
|
33
|
+
# The call to `name` is used so that the names of such classes are cached before runtime.
|
34
|
+
(
|
35
|
+
BadlyFormattedPayload = Class.new(Error)
|
36
|
+
).name
|
37
|
+
|
38
|
+
(
|
39
|
+
ConfigError = Class.new(Error)
|
40
|
+
).name
|
41
|
+
|
28
42
|
(
|
29
43
|
RMQConnectionError = Class.new(Error)
|
30
44
|
).name
|
@@ -33,7 +47,8 @@ module Cuniculus
|
|
33
47
|
RMQQueueConfigurationConflict = Class.new(Error)
|
34
48
|
).name
|
35
49
|
|
50
|
+
|
36
51
|
(
|
37
|
-
|
52
|
+
WorkerOptionsError = Class.new(Error)
|
38
53
|
).name
|
39
54
|
end
|
data/lib/cuniculus/job_queue.rb
CHANGED
@@ -2,11 +2,12 @@
|
|
2
2
|
|
3
3
|
require "socket"
|
4
4
|
require "thread"
|
5
|
+
require "rack"
|
5
6
|
|
6
7
|
module Cuniculus
|
7
8
|
module Plugins
|
8
|
-
# The HealthCheck plugin starts a
|
9
|
-
# It currently does not perform any additional checks returns '200 OK' regardless of whether
|
9
|
+
# The HealthCheck plugin starts a Rack server after consumers are initialized, for health probing.
|
10
|
+
# It currently does not perform any additional checks and returns '200 OK' regardless of whether
|
10
11
|
# - the node can connect to RabbitMQ;
|
11
12
|
# - consumers are stuck.
|
12
13
|
#
|
@@ -17,23 +18,23 @@ module Cuniculus
|
|
17
18
|
# Cuniculus.plugin(:health_check)
|
18
19
|
# ```
|
19
20
|
#
|
20
|
-
# Options may be passed as well
|
21
|
+
# Options may be passed as well:
|
21
22
|
# ```ruby
|
22
23
|
# opts = {
|
23
24
|
# "bind_to" => "127.0.0.1", # Default: "0.0.0.0"
|
24
25
|
# "port" => 8080 # Default: 3000
|
26
|
+
# "path" => "alive" # Default: "healtcheck"
|
25
27
|
# }
|
26
28
|
# Cuniculus.plugin(:health_check, opts)
|
27
29
|
# ```
|
28
|
-
# This starts the server bound to 127.0.0.1 and port 8080.
|
29
|
-
#
|
30
|
-
# Note that the request path is not considered. The server responds with 200 to any path.
|
30
|
+
# This starts the server bound to 127.0.0.1 and port 8080, and responds on path "alive".
|
31
|
+
# The server responds with 404 when requests are made to different paths.
|
31
32
|
module HealthCheck
|
32
|
-
HEALTH_CHECK_RESPONSE = "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 2\r\nConnection: close\r\n\r\nOK"
|
33
|
-
|
34
33
|
DEFAULTS = {
|
35
34
|
"bind_to" => "0.0.0.0",
|
35
|
+
"path" => "healthcheck",
|
36
36
|
"port" => 3000,
|
37
|
+
"quiet" => false,
|
37
38
|
"server" => "webrick",
|
38
39
|
"block" => nil
|
39
40
|
}.freeze
|
@@ -42,15 +43,19 @@ module Cuniculus
|
|
42
43
|
|
43
44
|
# Configure `health_check` plugin
|
44
45
|
#
|
45
|
-
# @param plugins_cfg [Hash] Global plugin config hash, passed by Cuniculus. This should not be
|
46
|
+
# @param plugins_cfg [Hash] Global plugin config hash, passed by Cuniculus. This should not be modified by plugin users.
|
46
47
|
# @param opts [Hash] Plugin specific options.
|
47
|
-
# @option opts [String] "bind_to"
|
48
|
-
# @option opts [
|
48
|
+
# @option opts [String] "bind_to" ("0.0.0.0") IP address to bind to.
|
49
|
+
# @option opts [String] "path" ("healthcheck") Request path to respond to. Requests to other paths will get a 404 response.
|
50
|
+
# @option opts [Numeric] "port" (3000) Port number to bind to.
|
51
|
+
# @option opts [Boolean] "quiet" (false) Disable server logging to STDOUT and STDERR.
|
52
|
+
# @option opts [String] "server" ("webrick") Rack server handler to use .
|
49
53
|
def self.configure(plugins_cfg, opts = {}, &block)
|
54
|
+
opts = opts.transform_keys(&:to_s)
|
50
55
|
invalid_opts = opts.keys - DEFAULTS.keys
|
51
56
|
raise Cuniculus::Error, "Invalid option keys for :health_check plugin: #{invalid_opts}" unless invalid_opts.empty?
|
52
57
|
|
53
|
-
plugins_cfg[OPTS_KEY] = h = opts.slice("bind_to", "port", "server")
|
58
|
+
plugins_cfg[OPTS_KEY] = h = opts.slice("bind_to", "path", "port", "quiet", "server")
|
54
59
|
h["block"] = block if block
|
55
60
|
DEFAULTS.each do |k, v|
|
56
61
|
h[k] = v if v && !h.key?(k)
|
@@ -58,45 +63,43 @@ module Cuniculus
|
|
58
63
|
end
|
59
64
|
|
60
65
|
module SupervisorMethods
|
66
|
+
def initialize(config)
|
67
|
+
super(config)
|
68
|
+
hc_plugin_opts = config.opts[OPTS_KEY]
|
69
|
+
@hc_server = Rack::Handler.get(hc_plugin_opts["server"])
|
70
|
+
@hc_rack_app = build_rack_app(hc_plugin_opts)
|
71
|
+
end
|
72
|
+
|
61
73
|
def start
|
62
|
-
|
63
|
-
start_health_check_server(hc_rd)
|
74
|
+
start_health_check_server
|
64
75
|
super
|
65
76
|
end
|
66
77
|
|
67
78
|
def stop
|
68
|
-
@
|
79
|
+
@hc_server.shutdown
|
69
80
|
super
|
70
81
|
end
|
71
82
|
|
72
83
|
|
73
84
|
private
|
74
85
|
|
75
|
-
def
|
76
|
-
|
77
|
-
|
78
|
-
|
79
|
-
|
80
|
-
|
81
|
-
|
82
|
-
@hc_thread = Thread.new do
|
83
|
-
sock = nil
|
84
|
-
done = false
|
85
|
-
loop do
|
86
|
-
begin
|
87
|
-
break if done
|
88
|
-
sock = server.accept_nonblock
|
89
|
-
rescue IO::WaitReadable, Errno::EINTR
|
90
|
-
io = IO.select([server, pipe_reader])
|
91
|
-
done = true if io.first.include?(pipe_reader)
|
92
|
-
retry
|
93
|
-
end
|
94
|
-
|
95
|
-
sock.print HEALTH_CHECK_RESPONSE
|
96
|
-
sock.shutdown
|
86
|
+
def build_rack_app(opts)
|
87
|
+
app = ::Object.new
|
88
|
+
app.define_singleton_method(:call) do |env|
|
89
|
+
if Rack::Request.new(env).path == "/#{opts['path']}"
|
90
|
+
[200, {}, ["OK"]]
|
91
|
+
else
|
92
|
+
[404, {}, ["Not Found"]]
|
97
93
|
end
|
94
|
+
end
|
95
|
+
app
|
96
|
+
end
|
98
97
|
|
99
|
-
|
98
|
+
def start_health_check_server
|
99
|
+
opts = config.opts[OPTS_KEY]
|
100
|
+
Thread.new do
|
101
|
+
access_log = opts["quiet"] ? [] : nil
|
102
|
+
@hc_server.run(@hc_rack_app, AccessLog: access_log, Port: opts["port"], Host: opts["bind_to"])
|
100
103
|
end
|
101
104
|
end
|
102
105
|
end
|
@@ -6,27 +6,32 @@ require "cuniculus/job_queue"
|
|
6
6
|
|
7
7
|
module Cuniculus
|
8
8
|
class QueueConfig
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
@
|
9
|
+
DEFAULT_MAX_RETRY = 8
|
10
|
+
DEFAULT_PREFETCH_COUNT = 10
|
11
|
+
DEFAULT_QUEUE_NAME = "cun_default"
|
12
|
+
DEFAULT_THREAD_POOL_SIZE = 5
|
13
|
+
|
14
|
+
attr_reader :durable, :max_retry, :name, :prefetch_count, :thread_pool_size
|
15
|
+
|
16
|
+
def initialize(opts = {})
|
17
|
+
opts = opts.transform_keys(&:to_s)
|
18
|
+
@durable = read_opt(opts["durable"], true)
|
19
|
+
@name = read_opt(opts["name"], DEFAULT_QUEUE_NAME)
|
20
|
+
@max_retry = read_opt(opts["max_retry"], DEFAULT_MAX_RETRY)
|
21
|
+
@prefetch_count = read_opt(opts["prefetch_count"], DEFAULT_PREFETCH_COUNT)
|
22
|
+
@thread_pool_size = read_opt(opts["thread_pool_size"], DEFAULT_THREAD_POOL_SIZE)
|
23
|
+
freeze
|
19
24
|
end
|
20
25
|
|
21
|
-
def read_opt(
|
22
|
-
|
26
|
+
def read_opt(val, default)
|
27
|
+
val.nil? ? default : val
|
23
28
|
end
|
24
29
|
|
25
30
|
def declare!(channel)
|
26
31
|
queue_name = name
|
27
32
|
base_q = channel.queue(
|
28
33
|
queue_name,
|
29
|
-
durable:
|
34
|
+
durable: durable,
|
30
35
|
exclusive: false,
|
31
36
|
arguments: { "x-dead-letter-exchange" => Cuniculus::CUNICULUS_DLX_EXCHANGE }
|
32
37
|
)
|
@@ -38,7 +43,7 @@ module Cuniculus
|
|
38
43
|
|
39
44
|
q = channel.queue(
|
40
45
|
queue_name,
|
41
|
-
durable:
|
46
|
+
durable: durable,
|
42
47
|
exclusive: false,
|
43
48
|
arguments: {
|
44
49
|
"x-dead-letter-exchange" => Cuniculus::CUNICULUS_EXCHANGE,
|
data/lib/cuniculus/supervisor.rb
CHANGED
@@ -37,9 +37,9 @@ module Cuniculus
|
|
37
37
|
|
38
38
|
def create_consumers(conn, queues)
|
39
39
|
consumers = []
|
40
|
-
consumer_pool_size = 5
|
41
40
|
queues.each do |_name, q_cfg|
|
42
|
-
ch = conn.create_channel(nil,
|
41
|
+
ch = conn.create_channel(nil, q_cfg.thread_pool_size)
|
42
|
+
ch.prefetch(q_cfg.prefetch_count) if q_cfg.prefetch_count
|
43
43
|
consumers << Cuniculus::Consumer.new(q_cfg, ch)
|
44
44
|
end
|
45
45
|
consumers
|
data/lib/cuniculus/version.rb
CHANGED
@@ -6,11 +6,11 @@ module Cuniculus
|
|
6
6
|
|
7
7
|
# The minor version of Cuniculus. Bumped for every non-patch level
|
8
8
|
# release.
|
9
|
-
MINOR =
|
9
|
+
MINOR = 1
|
10
10
|
|
11
11
|
# The tiny version of Cuniculus. Usually 0, only bumped for bugfix
|
12
12
|
# releases that fix regressions from previous versions.
|
13
|
-
TINY =
|
13
|
+
TINY = 0
|
14
14
|
|
15
15
|
# The version of Cuniculus you are using, as a string (e.g. "2.11.0")
|
16
16
|
VERSION = [MAJOR, MINOR, TINY].join(".").freeze
|
data/lib/cuniculus/worker.rb
CHANGED
@@ -7,15 +7,67 @@ module Cuniculus
|
|
7
7
|
module Worker
|
8
8
|
def self.included(base)
|
9
9
|
base.extend(ClassMethods)
|
10
|
+
|
11
|
+
# Dev note:
|
12
|
+
# The point here is to allow options set via cuniculus_options to be
|
13
|
+
# inherited by subclasses.
|
14
|
+
# When reading the options, a subclass will call the singleton method cun_opts.
|
15
|
+
# If the subclass doesn't redefine this method via a call to cuniculus_options,
|
16
|
+
# it will still use the definition from its parent class.
|
17
|
+
base.define_singleton_method("cun_opts=") do |opts|
|
18
|
+
singleton_class.class_eval do
|
19
|
+
define_method("cun_opts") { opts }
|
20
|
+
end
|
21
|
+
end
|
10
22
|
end
|
11
23
|
|
12
24
|
module ClassMethods
|
25
|
+
DEFAULT_OPTS = { "queue" => "cun_default" }.freeze
|
26
|
+
VALID_OPT_KEYS = %w[queue].freeze
|
27
|
+
|
28
|
+
# Read-only cuniculus option values
|
29
|
+
#
|
30
|
+
# @return opts [Hash] hash with current values
|
31
|
+
def cun_opts
|
32
|
+
DEFAULT_OPTS
|
33
|
+
end
|
34
|
+
|
35
|
+
# Worker-specific options for running cuniculus.
|
36
|
+
#
|
37
|
+
# Note that options set on a worker class are inherited by its subclasses.
|
38
|
+
#
|
39
|
+
# @param opts [Hash]
|
40
|
+
# @option opts [String] "queue" ("cun_default") Name of the underlying RabbitMQ queue.
|
41
|
+
#
|
42
|
+
# @example Change the queue name of a worker
|
43
|
+
# class MyWorker
|
44
|
+
# include Cuniculus::Worker
|
45
|
+
#
|
46
|
+
# cuniculus_options queue: "critical"
|
47
|
+
#
|
48
|
+
# def perform
|
49
|
+
# # run the task
|
50
|
+
# end
|
51
|
+
# end
|
52
|
+
def cuniculus_options(opts)
|
53
|
+
opts = validate_opts!(opts)
|
54
|
+
self.cun_opts = opts
|
55
|
+
end
|
56
|
+
|
57
|
+
def validate_opts!(opts)
|
58
|
+
raise WorkerOptionsError, "Argument passed to 'cuniculus_options' should be a Hash" unless opts.is_a?(Hash)
|
59
|
+
opts = opts.transform_keys(&:to_s)
|
60
|
+
invalid_keys = opts.keys - VALID_OPT_KEYS
|
61
|
+
raise WorkerOptionsError, "Invalid keys passed to 'cuniculus_options': #{invalid_keys.inspect}" unless invalid_keys.empty?
|
62
|
+
opts
|
63
|
+
end
|
64
|
+
|
13
65
|
def perform_async(*args)
|
14
66
|
publish({ "class" => self, "args" => args })
|
15
67
|
end
|
16
68
|
|
17
69
|
def publish(item)
|
18
|
-
routing_key = "
|
70
|
+
routing_key = cun_opts["queue"]
|
19
71
|
payload = normalize_item(item)
|
20
72
|
Cuniculus::RMQPool.with_exchange do |x|
|
21
73
|
x.publish(payload, { routing_key: routing_key, persistent: true })
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: cuniculus
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0
|
4
|
+
version: 0.1.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Marcelo Pereira
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-03-07 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bunny
|
@@ -52,6 +52,20 @@ dependencies:
|
|
52
52
|
- - ">="
|
53
53
|
- !ruby/object:Gem::Version
|
54
54
|
version: '0'
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: rack
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - ">="
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '0'
|
62
|
+
type: :development
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - ">="
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '0'
|
55
69
|
- !ruby/object:Gem::Dependency
|
56
70
|
name: redcarpet
|
57
71
|
requirement: !ruby/object:Gem::Requirement
|
@@ -94,6 +108,20 @@ dependencies:
|
|
94
108
|
- - ">="
|
95
109
|
- !ruby/object:Gem::Version
|
96
110
|
version: '0'
|
111
|
+
- !ruby/object:Gem::Dependency
|
112
|
+
name: toxiproxy
|
113
|
+
requirement: !ruby/object:Gem::Requirement
|
114
|
+
requirements:
|
115
|
+
- - ">="
|
116
|
+
- !ruby/object:Gem::Version
|
117
|
+
version: '0'
|
118
|
+
type: :development
|
119
|
+
prerelease: false
|
120
|
+
version_requirements: !ruby/object:Gem::Requirement
|
121
|
+
requirements:
|
122
|
+
- - ">="
|
123
|
+
- !ruby/object:Gem::Version
|
124
|
+
version: '0'
|
97
125
|
- !ruby/object:Gem::Dependency
|
98
126
|
name: warning
|
99
127
|
requirement: !ruby/object:Gem::Requirement
|
@@ -108,6 +136,20 @@ dependencies:
|
|
108
136
|
- - ">="
|
109
137
|
- !ruby/object:Gem::Version
|
110
138
|
version: '0'
|
139
|
+
- !ruby/object:Gem::Dependency
|
140
|
+
name: webrick
|
141
|
+
requirement: !ruby/object:Gem::Requirement
|
142
|
+
requirements:
|
143
|
+
- - ">="
|
144
|
+
- !ruby/object:Gem::Version
|
145
|
+
version: '0'
|
146
|
+
type: :development
|
147
|
+
prerelease: false
|
148
|
+
version_requirements: !ruby/object:Gem::Requirement
|
149
|
+
requirements:
|
150
|
+
- - ">="
|
151
|
+
- !ruby/object:Gem::Version
|
152
|
+
version: '0'
|
111
153
|
- !ruby/object:Gem::Dependency
|
112
154
|
name: yard
|
113
155
|
requirement: !ruby/object:Gem::Requirement
|