cuniculus 0.0.1 → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/LICENSE +1 -1
- data/README.md +67 -2
- data/lib/cuniculus.rb +2 -0
- data/lib/cuniculus/config.rb +33 -0
- data/lib/cuniculus/consumer.rb +0 -1
- data/lib/cuniculus/core.rb +1 -1
- data/lib/cuniculus/exceptions.rb +20 -5
- data/lib/cuniculus/job_queue.rb +1 -1
- data/lib/cuniculus/plugins/health_check.rb +41 -38
- data/lib/cuniculus/queue_config.rb +19 -14
- data/lib/cuniculus/supervisor.rb +2 -2
- data/lib/cuniculus/version.rb +2 -2
- data/lib/cuniculus/worker.rb +53 -1
- metadata +44 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: ca0cc8cbcc9cc9607488d5c7c88903548fe4106799e6517e82b2efb5824a9971
|
4
|
+
data.tar.gz: 6a0589c882ae8de39a0192e2e5b54c537a709fbe6ca496ce3b999d3a03ff6da1
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 73fc5ea97169bdbdf5a72b95dec76e2b529a13c6b308a3c3c9c947d891bc3845d48ad5b4344820d0f1fed8cbed8b0bb1ffac3a4c3ea043ca0cb3c1737eb8014d
|
7
|
+
data.tar.gz: 7dbdd0c30c1dc808457fc408506aade75b65de83f5318799f8b7a5aab4c0891ed71fbb927e58eaa70b099442790b1c1d20b06d34d12a53967def89052f068887
|
data/LICENSE
CHANGED
data/README.md
CHANGED
@@ -2,6 +2,36 @@
|
|
2
2
|
|
3
3
|
Ruby job queue backed by RabbitMQ. The word _cuniculus_ comes from the scientific name of the European rabbit (Oryctolagus cuniculus).
|
4
4
|
|
5
|
+
## Benchmarks
|
6
|
+
|
7
|
+
The following measurements were performed with the `bin/run_benchmarks` utility, with different command parameters. Run it with `-h` to see its usage.
|
8
|
+
|
9
|
+
To simulate network latency, [Toxiproxy](https://github.com/Shopify/toxiproxy) was used. It needs to be started with `toxiproxy-server` before running the benchmarks.
|
10
|
+
|
11
|
+
Network latency (_ms_) | Prefetch count | Throughput (_jobs/s_) | Average latency (_ms_)
|
12
|
+
----------------------:|---------------------:|----------------------:|----------------------:
|
13
|
+
1 | 65535 (max. allowed) | 10225 | 2
|
14
|
+
10 | 65535 (max. allowed) | 9990 | 13
|
15
|
+
1 | 50 | 8051 | 2
|
16
|
+
10 | 50 | 2500 | 13
|
17
|
+
100 | 50 | 481 | 103
|
18
|
+
1 | 25 | 7824 | 2
|
19
|
+
10 | 25 | 1824 | 13
|
20
|
+
50 | 25 | 469 | 53
|
21
|
+
1 | 10 (default) | 5266 | 2
|
22
|
+
10 | 10 (default) | 807 | 13
|
23
|
+
1 | 1 | 481 | 2
|
24
|
+
10 | 1 | 81 | 13
|
25
|
+
|
26
|
+
Additional benchmark parameters:
|
27
|
+
- throughput was measured by consuming 100k jobs;
|
28
|
+
- job latency was averaged over 200 samples;
|
29
|
+
- Ruby 2.7.2 was used.
|
30
|
+
|
31
|
+
Several remarks can be made:
|
32
|
+
- Higher prefetch counts lead to higher throughput, but there are downsides of having it too high; see [this reference](https://www.cloudamqp.com/blog/2017-12-29-part1-rabbitmq-best-practice.html#prefetch) on how to properly tune it.
|
33
|
+
- Network latency has a severe impact on the throughput, and the effect is larger the smaller the prefetch count is.
|
34
|
+
|
5
35
|
## Getting started
|
6
36
|
|
7
37
|
```sh
|
@@ -18,6 +48,8 @@ require 'cuniculus/worker'
|
|
18
48
|
class MyWorker
|
19
49
|
include Cuniculus::Worker
|
20
50
|
|
51
|
+
# the queue name is not explicitly given, so "cun_default" is used.
|
52
|
+
|
21
53
|
def perform(arg1, arg2)
|
22
54
|
puts "Processing:"
|
23
55
|
puts "arg1: #{arg1.inspect}"
|
@@ -52,6 +84,8 @@ There is also a more complete example in the Cuniculus repository itself. To run
|
|
52
84
|
bin/cuniculus -I examples/ -r example/init_cuniculus.rb
|
53
85
|
```
|
54
86
|
|
87
|
+
The `-I examples` option adds the `examples/` directory into the load path, and `-r example/init_cuniculus.rb` requires `init_cuniculus.rb` prior to starting the consumer. The latter is where configurations such as that described in the next section should be.
|
88
|
+
|
55
89
|
## Configuration
|
56
90
|
|
57
91
|
Configuration is done through code, using `Cuniculus.configure`.
|
@@ -75,6 +109,20 @@ Cuniculus.configure do |cfg|
|
|
75
109
|
cfg.rabbitmq_opts = rabbitmq_conn
|
76
110
|
cfg.pub_thr_pool_size = 5 # Only affects job producers
|
77
111
|
cfg.dead_queue_ttl = 1000 * 60 * 60 * 24 * 30 # keep failed jobs for 30 days
|
112
|
+
cfg.add_queue({ name: "critical", durable: true, max_retry: 10, prefetch_count: 1})
|
113
|
+
end
|
114
|
+
```
|
115
|
+
|
116
|
+
To configure the queue used by a worker, used `cuniculus_options`:
|
117
|
+
|
118
|
+
```ruby
|
119
|
+
class MyWorker
|
120
|
+
include Cuniculus::Worker
|
121
|
+
|
122
|
+
cuniculus_options queue: "critical"
|
123
|
+
def perform
|
124
|
+
# code
|
125
|
+
end
|
78
126
|
end
|
79
127
|
```
|
80
128
|
|
@@ -92,15 +140,32 @@ The method expects a block that will receive an exception, and run in the scope
|
|
92
140
|
|
93
141
|
## Retry mechanism
|
94
142
|
|
95
|
-
|
143
|
+
Retries are enabled by default (with 8 retries) with an exponential backoff, meaning the time between retries increases the more failures happen. The formula for calculating the times between retries can be found in {Cuniculus::QueueConfig}, namely in the `x-message-ttl` line. As an example, the time between the 7th and 8th retries is roughly 29 days.
|
144
|
+
|
145
|
+
Given a declared queue in the configuration, Cuniculus starts on RabbitMQ the corresponding base queue, in addition to its retry queues. As an example, let's consider the default queue `cun_default`: Cuniculus declares a `cun_default` queue, together with some `cun_default_{n}` queues used for job retries.
|
146
|
+
|
96
147
|
When a job raises an exception, it is placed into the `cun_default_1` queue for the first retry. It stays there for some pre-defined time, and then gets moved back into the `cun_default` queue for execution.
|
97
148
|
|
98
149
|
If it fails again, it gets moved to `cun_default_2`, where it stays for a longer period until it's moved back directly into the `cun_default` queue again.
|
99
150
|
|
100
|
-
This goes on until there are no more retry attempts, in which case the job gets moved into the `cun_dead` queue. It can be then only be moved back into the `cun_default` queue manually; otherwise it is discarded after some time, defined as the
|
151
|
+
This goes on until there are no more retry attempts, in which case the job gets moved into the `cun_dead` queue. It can be then only be moved back into the `cun_default` queue manually; otherwise it is discarded after some time, defined as the {Cuniculus::Config.dead_queue_ttl}, in milliseconds (by default, 180 days).
|
101
152
|
|
102
153
|
Note that if a job cannot even be parsed, it is moved straight to the dead queue, as there's no point in retrying.
|
103
154
|
|
155
|
+
## Health check plugin
|
156
|
+
|
157
|
+
Cuniculus ships with a health check plugin. When enabled, a Rack server is started (therefore the [Rack](https://github.com/rack/rack) gem is required, as well as the used handler), which responds with `200 OK` upon receiving a request in the configured port and path.
|
158
|
+
|
159
|
+
Enable it with `Cuniculus.plugin(:health_check)`, which binds the server to `0.0.0.0:3000`, listening on the `/healthcheck` path. To configure the server, pass additional options:
|
160
|
+
|
161
|
+
```ruby
|
162
|
+
Cuniculus.plugin(:health_check, { "bind_to" => "127.0.0.1", "port" => 3003, "path" => "ping" })
|
163
|
+
```
|
164
|
+
|
165
|
+
Check {Cuniculus::Plugins::HealthCheck} for further details.
|
166
|
+
|
167
|
+
_Note that the default handler "webrick" is not bundled by default with Ruby 3 and needs to be installed separately, if it is to be used._
|
168
|
+
|
104
169
|
## How it works
|
105
170
|
|
106
171
|
Cuniculus code and conventions are very much inspired by another Ruby job queue library: [Sidekiq](https://github.com/mperham/sidekiq).
|
data/lib/cuniculus.rb
CHANGED
@@ -14,12 +14,14 @@ require "cuniculus/supervisor"
|
|
14
14
|
module Cuniculus
|
15
15
|
|
16
16
|
# Configure Cuniculus.
|
17
|
+
# Check {Cuniculus::Config} for the available options.
|
17
18
|
#
|
18
19
|
# @yield [Cuniculus::Config]
|
19
20
|
#
|
20
21
|
# @example Change RabbitMQ connection details.
|
21
22
|
# Cuniculus.configure do |cfg|
|
22
23
|
# cfg.rabbitmq_opts = { host: 'rmq.mycompany.com', user: 'guest', pass: 'guest' }
|
24
|
+
# cfg.add_queue({ name: "new_queue", max_retry: 4 })
|
23
25
|
# end
|
24
26
|
def self.configure
|
25
27
|
cfg = Cuniculus::Config.new
|
data/lib/cuniculus/config.rb
CHANGED
@@ -1,6 +1,7 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
require "cuniculus/core"
|
4
|
+
require "cuniculus/exceptions"
|
4
5
|
require "cuniculus/queue_config"
|
5
6
|
|
6
7
|
module Cuniculus
|
@@ -23,9 +24,37 @@ module Cuniculus
|
|
23
24
|
vhost: "/"
|
24
25
|
}
|
25
26
|
@exchange_name = "cuniculus"
|
27
|
+
@pub_thr_pool_size = 5
|
26
28
|
@dead_queue_ttl = 1000 * 60 * 60 * 24 * 180 # 180 days
|
27
29
|
end
|
28
30
|
|
31
|
+
# Configure an additional queue
|
32
|
+
#
|
33
|
+
# Note that a single call to `add_queue` might lead to the creation of multiple queues on RabbitMQ: one base queue, and an additional queue for every retry attempt.
|
34
|
+
# For example, with a queue named `"test"` with `max_retry` set to `4`, 5 queues are created in RabbitMQ.
|
35
|
+
#
|
36
|
+
# For tuning `prefetch_count`, refer to [this guide](https://www.cloudamqp.com/blog/2017-12-29-part1-rabbitmq-best-practice.html#prefetch).
|
37
|
+
#
|
38
|
+
# If a queue already exists in RabbitMQ, and an attempt is done to add it again through `add_queue`, nothing happens, except if the options passed to `add_queue` conflict with the existing queue. For example if a queue exists that is durable, and `add_queue` is called with `"durable" => false`, a `Cuniculus::RMQQueueConfigurationConflict` is raised. To redeclare a queue with conflicting configurations, the original queue has first to be removed from RabbitMQ manually. This can be done, for example, through the management console.
|
39
|
+
#
|
40
|
+
# @param qopts [Hash] Queue config options.
|
41
|
+
# @option qopts [String] "name" Name of the queue.
|
42
|
+
# @option qopts [Boolean] "durable" (true) Whether queue is declared as durable in RabbitMQ. Jobs in non-durable queues may be lost if the RabbitMQ goes down.
|
43
|
+
# @option qopts [Integer] "max_retry" (8) Number of retries for failed jobs in this queue.
|
44
|
+
# @option qopts [Integer] "prefetch_count" (10) Prefetch count used when consuming jobs from this queue.
|
45
|
+
# @option qopts [Integer] "thread_pool_size" (5) Thread pool size for receiving jobs.
|
46
|
+
#
|
47
|
+
# @example Add queue named "critical"
|
48
|
+
# Cuniculus.configure do |cfg|
|
49
|
+
# cfg.add_queue({ name: "critical", max_retry: 10 })
|
50
|
+
# end
|
51
|
+
def add_queue(qopts)
|
52
|
+
qopts = qopts.transform_keys(&:to_s)
|
53
|
+
qname = qopts["name"].to_s
|
54
|
+
raise Cuniculus::ConfigError, "Missing 'name' key in queue configuration hash" if qname.strip.empty?
|
55
|
+
@queues[qname] = QueueConfig.new(qopts)
|
56
|
+
end
|
57
|
+
|
29
58
|
def declare!
|
30
59
|
conn = ::Bunny.new(rabbitmq_opts.merge(ENFORCED_CONN_OPTS))
|
31
60
|
conn.start
|
@@ -35,6 +64,10 @@ module Cuniculus
|
|
35
64
|
@queues.each_value { |q| q.declare!(ch) }
|
36
65
|
end
|
37
66
|
|
67
|
+
# Specify if the default queue `cun_default` should be created.
|
68
|
+
# `cun_default` is used by workers that don't explicitly specify a queue with `cuniculus_options queue: "another_queue"`.
|
69
|
+
#
|
70
|
+
# @param bool [Boolean] If false, queue `cun_default` is not created. Defaults to `true`.
|
38
71
|
def default_queue=(bool)
|
39
72
|
@queues.delete("cun_default") unless bool
|
40
73
|
end
|
data/lib/cuniculus/consumer.rb
CHANGED
@@ -17,7 +17,6 @@ module Cuniculus
|
|
17
17
|
|
18
18
|
def start
|
19
19
|
@exchange = channel.direct(Cuniculus::CUNICULUS_EXCHANGE, { durable: true })
|
20
|
-
# channel.direct(Cuniculus::CUNICULUS_DLX_EXCHANGE, { durable: true })
|
21
20
|
@job_queue = queue_config.declare!(channel)
|
22
21
|
@_consumer = job_queue.subscribe(manual_ack: true, block: false) do |delivery_info, properties, payload|
|
23
22
|
run_job(delivery_info, properties, payload)
|
data/lib/cuniculus/core.rb
CHANGED
@@ -23,7 +23,7 @@ module Cuniculus
|
|
23
23
|
# the message and backtrace of `exception`.
|
24
24
|
#
|
25
25
|
# @param exception [Exception] The exception being wrapped
|
26
|
-
# @param [Cuniculus::Error] The subclass of `Cuniculus::Error`
|
26
|
+
# @param klass [Cuniculus::Error] The subclass of `Cuniculus::Error`
|
27
27
|
#
|
28
28
|
# @return [Cuniculus::Error] An instance of the input `Cuniculus::Error`
|
29
29
|
def convert_exception_class(exception, klass)
|
data/lib/cuniculus/exceptions.rb
CHANGED
@@ -5,12 +5,14 @@ module Cuniculus
|
|
5
5
|
#
|
6
6
|
# * `Cuniculus::Error`: Default exception raised by Cuniculus.
|
7
7
|
# All exceptions classes defined by Cuniculus descend from this class.
|
8
|
-
# * `Cuniculus::
|
9
|
-
#
|
8
|
+
# * `Cuniculus::BadlyFormattedPayload`: A Cuniculus consumer received an
|
9
|
+
# improperly formatted job message.
|
10
|
+
# * `Cuniculus::ConfigError`: Incorrect configuration passed to Cuniculus.
|
11
|
+
# * `Cuniculus::RMQConnectionError`: Unable to connect to RabbitMQ.
|
12
|
+
# * `Cuniculus::RMQQueueConfigurationConflict`: The queue configuration
|
10
13
|
# given to Cuniculus conflicts with the current configuration of the same
|
11
14
|
# existing queue in RabbitMQ.
|
12
|
-
# * `Cuniculus::
|
13
|
-
# improperly formatted job message.
|
15
|
+
# * `Cuniculus::WorkerOptionsError`: Invalid options passed to cuniculus_options.
|
14
16
|
|
15
17
|
class Error < ::StandardError
|
16
18
|
# If the Cuniculus exception wraps an underlying exception, the latter
|
@@ -25,6 +27,18 @@ module Cuniculus
|
|
25
27
|
end
|
26
28
|
end
|
27
29
|
|
30
|
+
# Dev note:
|
31
|
+
# As explained [here](https://github.com/jeremyevans/sequel/commit/24681efad0fec48195e43801c224bf18cdc8be13#diff-64cd7b67eccdc6dfa69c23b3b19f34e318f9e6827c5dee5f6e845b2993ab035c), empty classes created
|
32
|
+
# with `Class.new` require about 200 bytes less memory than ones created as `class MyClass; end`.
|
33
|
+
# The call to `name` is used so that the names of such classes are cached before runtime.
|
34
|
+
(
|
35
|
+
BadlyFormattedPayload = Class.new(Error)
|
36
|
+
).name
|
37
|
+
|
38
|
+
(
|
39
|
+
ConfigError = Class.new(Error)
|
40
|
+
).name
|
41
|
+
|
28
42
|
(
|
29
43
|
RMQConnectionError = Class.new(Error)
|
30
44
|
).name
|
@@ -33,7 +47,8 @@ module Cuniculus
|
|
33
47
|
RMQQueueConfigurationConflict = Class.new(Error)
|
34
48
|
).name
|
35
49
|
|
50
|
+
|
36
51
|
(
|
37
|
-
|
52
|
+
WorkerOptionsError = Class.new(Error)
|
38
53
|
).name
|
39
54
|
end
|
data/lib/cuniculus/job_queue.rb
CHANGED
@@ -2,11 +2,12 @@
|
|
2
2
|
|
3
3
|
require "socket"
|
4
4
|
require "thread"
|
5
|
+
require "rack"
|
5
6
|
|
6
7
|
module Cuniculus
|
7
8
|
module Plugins
|
8
|
-
# The HealthCheck plugin starts a
|
9
|
-
# It currently does not perform any additional checks returns '200 OK' regardless of whether
|
9
|
+
# The HealthCheck plugin starts a Rack server after consumers are initialized, for health probing.
|
10
|
+
# It currently does not perform any additional checks and returns '200 OK' regardless of whether
|
10
11
|
# - the node can connect to RabbitMQ;
|
11
12
|
# - consumers are stuck.
|
12
13
|
#
|
@@ -17,23 +18,23 @@ module Cuniculus
|
|
17
18
|
# Cuniculus.plugin(:health_check)
|
18
19
|
# ```
|
19
20
|
#
|
20
|
-
# Options may be passed as well
|
21
|
+
# Options may be passed as well:
|
21
22
|
# ```ruby
|
22
23
|
# opts = {
|
23
24
|
# "bind_to" => "127.0.0.1", # Default: "0.0.0.0"
|
24
25
|
# "port" => 8080 # Default: 3000
|
26
|
+
# "path" => "alive" # Default: "healtcheck"
|
25
27
|
# }
|
26
28
|
# Cuniculus.plugin(:health_check, opts)
|
27
29
|
# ```
|
28
|
-
# This starts the server bound to 127.0.0.1 and port 8080.
|
29
|
-
#
|
30
|
-
# Note that the request path is not considered. The server responds with 200 to any path.
|
30
|
+
# This starts the server bound to 127.0.0.1 and port 8080, and responds on path "alive".
|
31
|
+
# The server responds with 404 when requests are made to different paths.
|
31
32
|
module HealthCheck
|
32
|
-
HEALTH_CHECK_RESPONSE = "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 2\r\nConnection: close\r\n\r\nOK"
|
33
|
-
|
34
33
|
DEFAULTS = {
|
35
34
|
"bind_to" => "0.0.0.0",
|
35
|
+
"path" => "healthcheck",
|
36
36
|
"port" => 3000,
|
37
|
+
"quiet" => false,
|
37
38
|
"server" => "webrick",
|
38
39
|
"block" => nil
|
39
40
|
}.freeze
|
@@ -42,15 +43,19 @@ module Cuniculus
|
|
42
43
|
|
43
44
|
# Configure `health_check` plugin
|
44
45
|
#
|
45
|
-
# @param plugins_cfg [Hash] Global plugin config hash, passed by Cuniculus. This should not be
|
46
|
+
# @param plugins_cfg [Hash] Global plugin config hash, passed by Cuniculus. This should not be modified by plugin users.
|
46
47
|
# @param opts [Hash] Plugin specific options.
|
47
|
-
# @option opts [String] "bind_to"
|
48
|
-
# @option opts [
|
48
|
+
# @option opts [String] "bind_to" ("0.0.0.0") IP address to bind to.
|
49
|
+
# @option opts [String] "path" ("healthcheck") Request path to respond to. Requests to other paths will get a 404 response.
|
50
|
+
# @option opts [Numeric] "port" (3000) Port number to bind to.
|
51
|
+
# @option opts [Boolean] "quiet" (false) Disable server logging to STDOUT and STDERR.
|
52
|
+
# @option opts [String] "server" ("webrick") Rack server handler to use .
|
49
53
|
def self.configure(plugins_cfg, opts = {}, &block)
|
54
|
+
opts = opts.transform_keys(&:to_s)
|
50
55
|
invalid_opts = opts.keys - DEFAULTS.keys
|
51
56
|
raise Cuniculus::Error, "Invalid option keys for :health_check plugin: #{invalid_opts}" unless invalid_opts.empty?
|
52
57
|
|
53
|
-
plugins_cfg[OPTS_KEY] = h = opts.slice("bind_to", "port", "server")
|
58
|
+
plugins_cfg[OPTS_KEY] = h = opts.slice("bind_to", "path", "port", "quiet", "server")
|
54
59
|
h["block"] = block if block
|
55
60
|
DEFAULTS.each do |k, v|
|
56
61
|
h[k] = v if v && !h.key?(k)
|
@@ -58,45 +63,43 @@ module Cuniculus
|
|
58
63
|
end
|
59
64
|
|
60
65
|
module SupervisorMethods
|
66
|
+
def initialize(config)
|
67
|
+
super(config)
|
68
|
+
hc_plugin_opts = config.opts[OPTS_KEY]
|
69
|
+
@hc_server = Rack::Handler.get(hc_plugin_opts["server"])
|
70
|
+
@hc_rack_app = build_rack_app(hc_plugin_opts)
|
71
|
+
end
|
72
|
+
|
61
73
|
def start
|
62
|
-
|
63
|
-
start_health_check_server(hc_rd)
|
74
|
+
start_health_check_server
|
64
75
|
super
|
65
76
|
end
|
66
77
|
|
67
78
|
def stop
|
68
|
-
@
|
79
|
+
@hc_server.shutdown
|
69
80
|
super
|
70
81
|
end
|
71
82
|
|
72
83
|
|
73
84
|
private
|
74
85
|
|
75
|
-
def
|
76
|
-
|
77
|
-
|
78
|
-
|
79
|
-
|
80
|
-
|
81
|
-
|
82
|
-
@hc_thread = Thread.new do
|
83
|
-
sock = nil
|
84
|
-
done = false
|
85
|
-
loop do
|
86
|
-
begin
|
87
|
-
break if done
|
88
|
-
sock = server.accept_nonblock
|
89
|
-
rescue IO::WaitReadable, Errno::EINTR
|
90
|
-
io = IO.select([server, pipe_reader])
|
91
|
-
done = true if io.first.include?(pipe_reader)
|
92
|
-
retry
|
93
|
-
end
|
94
|
-
|
95
|
-
sock.print HEALTH_CHECK_RESPONSE
|
96
|
-
sock.shutdown
|
86
|
+
def build_rack_app(opts)
|
87
|
+
app = ::Object.new
|
88
|
+
app.define_singleton_method(:call) do |env|
|
89
|
+
if Rack::Request.new(env).path == "/#{opts['path']}"
|
90
|
+
[200, {}, ["OK"]]
|
91
|
+
else
|
92
|
+
[404, {}, ["Not Found"]]
|
97
93
|
end
|
94
|
+
end
|
95
|
+
app
|
96
|
+
end
|
98
97
|
|
99
|
-
|
98
|
+
def start_health_check_server
|
99
|
+
opts = config.opts[OPTS_KEY]
|
100
|
+
Thread.new do
|
101
|
+
access_log = opts["quiet"] ? [] : nil
|
102
|
+
@hc_server.run(@hc_rack_app, AccessLog: access_log, Port: opts["port"], Host: opts["bind_to"])
|
100
103
|
end
|
101
104
|
end
|
102
105
|
end
|
@@ -6,27 +6,32 @@ require "cuniculus/job_queue"
|
|
6
6
|
|
7
7
|
module Cuniculus
|
8
8
|
class QueueConfig
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
@
|
9
|
+
DEFAULT_MAX_RETRY = 8
|
10
|
+
DEFAULT_PREFETCH_COUNT = 10
|
11
|
+
DEFAULT_QUEUE_NAME = "cun_default"
|
12
|
+
DEFAULT_THREAD_POOL_SIZE = 5
|
13
|
+
|
14
|
+
attr_reader :durable, :max_retry, :name, :prefetch_count, :thread_pool_size
|
15
|
+
|
16
|
+
def initialize(opts = {})
|
17
|
+
opts = opts.transform_keys(&:to_s)
|
18
|
+
@durable = read_opt(opts["durable"], true)
|
19
|
+
@name = read_opt(opts["name"], DEFAULT_QUEUE_NAME)
|
20
|
+
@max_retry = read_opt(opts["max_retry"], DEFAULT_MAX_RETRY)
|
21
|
+
@prefetch_count = read_opt(opts["prefetch_count"], DEFAULT_PREFETCH_COUNT)
|
22
|
+
@thread_pool_size = read_opt(opts["thread_pool_size"], DEFAULT_THREAD_POOL_SIZE)
|
23
|
+
freeze
|
19
24
|
end
|
20
25
|
|
21
|
-
def read_opt(
|
22
|
-
|
26
|
+
def read_opt(val, default)
|
27
|
+
val.nil? ? default : val
|
23
28
|
end
|
24
29
|
|
25
30
|
def declare!(channel)
|
26
31
|
queue_name = name
|
27
32
|
base_q = channel.queue(
|
28
33
|
queue_name,
|
29
|
-
durable:
|
34
|
+
durable: durable,
|
30
35
|
exclusive: false,
|
31
36
|
arguments: { "x-dead-letter-exchange" => Cuniculus::CUNICULUS_DLX_EXCHANGE }
|
32
37
|
)
|
@@ -38,7 +43,7 @@ module Cuniculus
|
|
38
43
|
|
39
44
|
q = channel.queue(
|
40
45
|
queue_name,
|
41
|
-
durable:
|
46
|
+
durable: durable,
|
42
47
|
exclusive: false,
|
43
48
|
arguments: {
|
44
49
|
"x-dead-letter-exchange" => Cuniculus::CUNICULUS_EXCHANGE,
|
data/lib/cuniculus/supervisor.rb
CHANGED
@@ -37,9 +37,9 @@ module Cuniculus
|
|
37
37
|
|
38
38
|
def create_consumers(conn, queues)
|
39
39
|
consumers = []
|
40
|
-
consumer_pool_size = 5
|
41
40
|
queues.each do |_name, q_cfg|
|
42
|
-
ch = conn.create_channel(nil,
|
41
|
+
ch = conn.create_channel(nil, q_cfg.thread_pool_size)
|
42
|
+
ch.prefetch(q_cfg.prefetch_count) if q_cfg.prefetch_count
|
43
43
|
consumers << Cuniculus::Consumer.new(q_cfg, ch)
|
44
44
|
end
|
45
45
|
consumers
|
data/lib/cuniculus/version.rb
CHANGED
@@ -6,11 +6,11 @@ module Cuniculus
|
|
6
6
|
|
7
7
|
# The minor version of Cuniculus. Bumped for every non-patch level
|
8
8
|
# release.
|
9
|
-
MINOR =
|
9
|
+
MINOR = 1
|
10
10
|
|
11
11
|
# The tiny version of Cuniculus. Usually 0, only bumped for bugfix
|
12
12
|
# releases that fix regressions from previous versions.
|
13
|
-
TINY =
|
13
|
+
TINY = 0
|
14
14
|
|
15
15
|
# The version of Cuniculus you are using, as a string (e.g. "2.11.0")
|
16
16
|
VERSION = [MAJOR, MINOR, TINY].join(".").freeze
|
data/lib/cuniculus/worker.rb
CHANGED
@@ -7,15 +7,67 @@ module Cuniculus
|
|
7
7
|
module Worker
|
8
8
|
def self.included(base)
|
9
9
|
base.extend(ClassMethods)
|
10
|
+
|
11
|
+
# Dev note:
|
12
|
+
# The point here is to allow options set via cuniculus_options to be
|
13
|
+
# inherited by subclasses.
|
14
|
+
# When reading the options, a subclass will call the singleton method cun_opts.
|
15
|
+
# If the subclass doesn't redefine this method via a call to cuniculus_options,
|
16
|
+
# it will still use the definition from its parent class.
|
17
|
+
base.define_singleton_method("cun_opts=") do |opts|
|
18
|
+
singleton_class.class_eval do
|
19
|
+
define_method("cun_opts") { opts }
|
20
|
+
end
|
21
|
+
end
|
10
22
|
end
|
11
23
|
|
12
24
|
module ClassMethods
|
25
|
+
DEFAULT_OPTS = { "queue" => "cun_default" }.freeze
|
26
|
+
VALID_OPT_KEYS = %w[queue].freeze
|
27
|
+
|
28
|
+
# Read-only cuniculus option values
|
29
|
+
#
|
30
|
+
# @return opts [Hash] hash with current values
|
31
|
+
def cun_opts
|
32
|
+
DEFAULT_OPTS
|
33
|
+
end
|
34
|
+
|
35
|
+
# Worker-specific options for running cuniculus.
|
36
|
+
#
|
37
|
+
# Note that options set on a worker class are inherited by its subclasses.
|
38
|
+
#
|
39
|
+
# @param opts [Hash]
|
40
|
+
# @option opts [String] "queue" ("cun_default") Name of the underlying RabbitMQ queue.
|
41
|
+
#
|
42
|
+
# @example Change the queue name of a worker
|
43
|
+
# class MyWorker
|
44
|
+
# include Cuniculus::Worker
|
45
|
+
#
|
46
|
+
# cuniculus_options queue: "critical"
|
47
|
+
#
|
48
|
+
# def perform
|
49
|
+
# # run the task
|
50
|
+
# end
|
51
|
+
# end
|
52
|
+
def cuniculus_options(opts)
|
53
|
+
opts = validate_opts!(opts)
|
54
|
+
self.cun_opts = opts
|
55
|
+
end
|
56
|
+
|
57
|
+
def validate_opts!(opts)
|
58
|
+
raise WorkerOptionsError, "Argument passed to 'cuniculus_options' should be a Hash" unless opts.is_a?(Hash)
|
59
|
+
opts = opts.transform_keys(&:to_s)
|
60
|
+
invalid_keys = opts.keys - VALID_OPT_KEYS
|
61
|
+
raise WorkerOptionsError, "Invalid keys passed to 'cuniculus_options': #{invalid_keys.inspect}" unless invalid_keys.empty?
|
62
|
+
opts
|
63
|
+
end
|
64
|
+
|
13
65
|
def perform_async(*args)
|
14
66
|
publish({ "class" => self, "args" => args })
|
15
67
|
end
|
16
68
|
|
17
69
|
def publish(item)
|
18
|
-
routing_key = "
|
70
|
+
routing_key = cun_opts["queue"]
|
19
71
|
payload = normalize_item(item)
|
20
72
|
Cuniculus::RMQPool.with_exchange do |x|
|
21
73
|
x.publish(payload, { routing_key: routing_key, persistent: true })
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: cuniculus
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0
|
4
|
+
version: 0.1.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Marcelo Pereira
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-03-07 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bunny
|
@@ -52,6 +52,20 @@ dependencies:
|
|
52
52
|
- - ">="
|
53
53
|
- !ruby/object:Gem::Version
|
54
54
|
version: '0'
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: rack
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - ">="
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '0'
|
62
|
+
type: :development
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - ">="
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '0'
|
55
69
|
- !ruby/object:Gem::Dependency
|
56
70
|
name: redcarpet
|
57
71
|
requirement: !ruby/object:Gem::Requirement
|
@@ -94,6 +108,20 @@ dependencies:
|
|
94
108
|
- - ">="
|
95
109
|
- !ruby/object:Gem::Version
|
96
110
|
version: '0'
|
111
|
+
- !ruby/object:Gem::Dependency
|
112
|
+
name: toxiproxy
|
113
|
+
requirement: !ruby/object:Gem::Requirement
|
114
|
+
requirements:
|
115
|
+
- - ">="
|
116
|
+
- !ruby/object:Gem::Version
|
117
|
+
version: '0'
|
118
|
+
type: :development
|
119
|
+
prerelease: false
|
120
|
+
version_requirements: !ruby/object:Gem::Requirement
|
121
|
+
requirements:
|
122
|
+
- - ">="
|
123
|
+
- !ruby/object:Gem::Version
|
124
|
+
version: '0'
|
97
125
|
- !ruby/object:Gem::Dependency
|
98
126
|
name: warning
|
99
127
|
requirement: !ruby/object:Gem::Requirement
|
@@ -108,6 +136,20 @@ dependencies:
|
|
108
136
|
- - ">="
|
109
137
|
- !ruby/object:Gem::Version
|
110
138
|
version: '0'
|
139
|
+
- !ruby/object:Gem::Dependency
|
140
|
+
name: webrick
|
141
|
+
requirement: !ruby/object:Gem::Requirement
|
142
|
+
requirements:
|
143
|
+
- - ">="
|
144
|
+
- !ruby/object:Gem::Version
|
145
|
+
version: '0'
|
146
|
+
type: :development
|
147
|
+
prerelease: false
|
148
|
+
version_requirements: !ruby/object:Gem::Requirement
|
149
|
+
requirements:
|
150
|
+
- - ">="
|
151
|
+
- !ruby/object:Gem::Version
|
152
|
+
version: '0'
|
111
153
|
- !ruby/object:Gem::Dependency
|
112
154
|
name: yard
|
113
155
|
requirement: !ruby/object:Gem::Requirement
|