cuniculus 0.0.1 → 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: a706ee37ab059f8897e2aba8cd1ab3a6dc15cdde2eabc78d98ddecd5113afb2a
4
- data.tar.gz: 02bc1b2265e346485e99f583ed3f3c3a619335238d7f1552d809490dea3d5a0d
3
+ metadata.gz: ca0cc8cbcc9cc9607488d5c7c88903548fe4106799e6517e82b2efb5824a9971
4
+ data.tar.gz: 6a0589c882ae8de39a0192e2e5b54c537a709fbe6ca496ce3b999d3a03ff6da1
5
5
  SHA512:
6
- metadata.gz: bc3a8d8a26b299ffab344a642fc539e762fd03f3922bd165ff0611d44a0ecd62a39cd4a242fe529bc283e063764a15e399c2f4ad1dbbb4112c97863953f032ab
7
- data.tar.gz: efc582168bc82aa2b5a66ecb42f26dc1d4fc36a73f95a9f46fb9028114a0781d0ad0f4cc10dc4fb41ff0f32f93550c8f745b80fb1e6a93c16a502e7f19e4a73e
6
+ metadata.gz: 73fc5ea97169bdbdf5a72b95dec76e2b529a13c6b308a3c3c9c947d891bc3845d48ad5b4344820d0f1fed8cbed8b0bb1ffac3a4c3ea043ca0cb3c1737eb8014d
7
+ data.tar.gz: 7dbdd0c30c1dc808457fc408506aade75b65de83f5318799f8b7a5aab4c0891ed71fbb927e58eaa70b099442790b1c1d20b06d34d12a53967def89052f068887
data/LICENSE CHANGED
@@ -1,6 +1,6 @@
1
1
  BSD 2-Clause License
2
2
 
3
- Copyright (c) 2020, Marcelo
3
+ Copyright (c) 2021, Marcelo
4
4
  All rights reserved.
5
5
 
6
6
  Redistribution and use in source and binary forms, with or without
data/README.md CHANGED
@@ -2,6 +2,36 @@
2
2
 
3
3
  Ruby job queue backed by RabbitMQ. The word _cuniculus_ comes from the scientific name of the European rabbit (Oryctolagus cuniculus).
4
4
 
5
+ ## Benchmarks
6
+
7
+ The following measurements were performed with the `bin/run_benchmarks` utility, with different command parameters. Run it with `-h` to see its usage.
8
+
9
+ To simulate network latency, [Toxiproxy](https://github.com/Shopify/toxiproxy) was used. It needs to be started with `toxiproxy-server` before running the benchmarks.
10
+
11
+ Network latency (_ms_) | Prefetch count | Throughput (_jobs/s_) | Average latency (_ms_)
12
+ ----------------------:|---------------------:|----------------------:|----------------------:
13
+ 1 | 65535 (max. allowed) | 10225 | 2
14
+ 10 | 65535 (max. allowed) | 9990 | 13
15
+ 1 | 50 | 8051 | 2
16
+ 10 | 50 | 2500 | 13
17
+ 100 | 50 | 481 | 103
18
+ 1 | 25 | 7824 | 2
19
+ 10 | 25 | 1824 | 13
20
+ 50 | 25 | 469 | 53
21
+ 1 | 10 (default) | 5266 | 2
22
+ 10 | 10 (default) | 807 | 13
23
+ 1 | 1 | 481 | 2
24
+ 10 | 1 | 81 | 13
25
+
26
+ Additional benchmark parameters:
27
+ - throughput was measured by consuming 100k jobs;
28
+ - job latency was averaged over 200 samples;
29
+ - Ruby 2.7.2 was used.
30
+
31
+ Several remarks can be made:
32
+ - Higher prefetch counts lead to higher throughput, but there are downsides of having it too high; see [this reference](https://www.cloudamqp.com/blog/2017-12-29-part1-rabbitmq-best-practice.html#prefetch) on how to properly tune it.
33
+ - Network latency has a severe impact on the throughput, and the effect is larger the smaller the prefetch count is.
34
+
5
35
  ## Getting started
6
36
 
7
37
  ```sh
@@ -18,6 +48,8 @@ require 'cuniculus/worker'
18
48
  class MyWorker
19
49
  include Cuniculus::Worker
20
50
 
51
+ # the queue name is not explicitly given, so "cun_default" is used.
52
+
21
53
  def perform(arg1, arg2)
22
54
  puts "Processing:"
23
55
  puts "arg1: #{arg1.inspect}"
@@ -52,6 +84,8 @@ There is also a more complete example in the Cuniculus repository itself. To run
52
84
  bin/cuniculus -I examples/ -r example/init_cuniculus.rb
53
85
  ```
54
86
 
87
+ The `-I examples` option adds the `examples/` directory into the load path, and `-r example/init_cuniculus.rb` requires `init_cuniculus.rb` prior to starting the consumer. The latter is where configurations such as that described in the next section should be.
88
+
55
89
  ## Configuration
56
90
 
57
91
  Configuration is done through code, using `Cuniculus.configure`.
@@ -75,6 +109,20 @@ Cuniculus.configure do |cfg|
75
109
  cfg.rabbitmq_opts = rabbitmq_conn
76
110
  cfg.pub_thr_pool_size = 5 # Only affects job producers
77
111
  cfg.dead_queue_ttl = 1000 * 60 * 60 * 24 * 30 # keep failed jobs for 30 days
112
+ cfg.add_queue({ name: "critical", durable: true, max_retry: 10, prefetch_count: 1})
113
+ end
114
+ ```
115
+
116
+ To configure the queue used by a worker, used `cuniculus_options`:
117
+
118
+ ```ruby
119
+ class MyWorker
120
+ include Cuniculus::Worker
121
+
122
+ cuniculus_options queue: "critical"
123
+ def perform
124
+ # code
125
+ end
78
126
  end
79
127
  ```
80
128
 
@@ -92,15 +140,32 @@ The method expects a block that will receive an exception, and run in the scope
92
140
 
93
141
  ## Retry mechanism
94
142
 
95
- Cuniculus declares a `cun_default` queue, together with some `cun_default_{n}` queues used for job retries.
143
+ Retries are enabled by default (with 8 retries) with an exponential backoff, meaning the time between retries increases the more failures happen. The formula for calculating the times between retries can be found in {Cuniculus::QueueConfig}, namely in the `x-message-ttl` line. As an example, the time between the 7th and 8th retries is roughly 29 days.
144
+
145
+ Given a declared queue in the configuration, Cuniculus starts on RabbitMQ the corresponding base queue, in addition to its retry queues. As an example, let's consider the default queue `cun_default`: Cuniculus declares a `cun_default` queue, together with some `cun_default_{n}` queues used for job retries.
146
+
96
147
  When a job raises an exception, it is placed into the `cun_default_1` queue for the first retry. It stays there for some pre-defined time, and then gets moved back into the `cun_default` queue for execution.
97
148
 
98
149
  If it fails again, it gets moved to `cun_default_2`, where it stays for a longer period until it's moved back directly into the `cun_default` queue again.
99
150
 
100
- This goes on until there are no more retry attempts, in which case the job gets moved into the `cun_dead` queue. It can be then only be moved back into the `cun_default` queue manually; otherwise it is discarded after some time, defined as the `dead_queue_ttl`, in milliseconds (by default, 180 days).
151
+ This goes on until there are no more retry attempts, in which case the job gets moved into the `cun_dead` queue. It can be then only be moved back into the `cun_default` queue manually; otherwise it is discarded after some time, defined as the {Cuniculus::Config.dead_queue_ttl}, in milliseconds (by default, 180 days).
101
152
 
102
153
  Note that if a job cannot even be parsed, it is moved straight to the dead queue, as there's no point in retrying.
103
154
 
155
+ ## Health check plugin
156
+
157
+ Cuniculus ships with a health check plugin. When enabled, a Rack server is started (therefore the [Rack](https://github.com/rack/rack) gem is required, as well as the used handler), which responds with `200 OK` upon receiving a request in the configured port and path.
158
+
159
+ Enable it with `Cuniculus.plugin(:health_check)`, which binds the server to `0.0.0.0:3000`, listening on the `/healthcheck` path. To configure the server, pass additional options:
160
+
161
+ ```ruby
162
+ Cuniculus.plugin(:health_check, { "bind_to" => "127.0.0.1", "port" => 3003, "path" => "ping" })
163
+ ```
164
+
165
+ Check {Cuniculus::Plugins::HealthCheck} for further details.
166
+
167
+ _Note that the default handler "webrick" is not bundled by default with Ruby 3 and needs to be installed separately, if it is to be used._
168
+
104
169
  ## How it works
105
170
 
106
171
  Cuniculus code and conventions are very much inspired by another Ruby job queue library: [Sidekiq](https://github.com/mperham/sidekiq).
data/lib/cuniculus.rb CHANGED
@@ -14,12 +14,14 @@ require "cuniculus/supervisor"
14
14
  module Cuniculus
15
15
 
16
16
  # Configure Cuniculus.
17
+ # Check {Cuniculus::Config} for the available options.
17
18
  #
18
19
  # @yield [Cuniculus::Config]
19
20
  #
20
21
  # @example Change RabbitMQ connection details.
21
22
  # Cuniculus.configure do |cfg|
22
23
  # cfg.rabbitmq_opts = { host: 'rmq.mycompany.com', user: 'guest', pass: 'guest' }
24
+ # cfg.add_queue({ name: "new_queue", max_retry: 4 })
23
25
  # end
24
26
  def self.configure
25
27
  cfg = Cuniculus::Config.new
@@ -1,6 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  require "cuniculus/core"
4
+ require "cuniculus/exceptions"
4
5
  require "cuniculus/queue_config"
5
6
 
6
7
  module Cuniculus
@@ -23,9 +24,37 @@ module Cuniculus
23
24
  vhost: "/"
24
25
  }
25
26
  @exchange_name = "cuniculus"
27
+ @pub_thr_pool_size = 5
26
28
  @dead_queue_ttl = 1000 * 60 * 60 * 24 * 180 # 180 days
27
29
  end
28
30
 
31
+ # Configure an additional queue
32
+ #
33
+ # Note that a single call to `add_queue` might lead to the creation of multiple queues on RabbitMQ: one base queue, and an additional queue for every retry attempt.
34
+ # For example, with a queue named `"test"` with `max_retry` set to `4`, 5 queues are created in RabbitMQ.
35
+ #
36
+ # For tuning `prefetch_count`, refer to [this guide](https://www.cloudamqp.com/blog/2017-12-29-part1-rabbitmq-best-practice.html#prefetch).
37
+ #
38
+ # If a queue already exists in RabbitMQ, and an attempt is done to add it again through `add_queue`, nothing happens, except if the options passed to `add_queue` conflict with the existing queue. For example if a queue exists that is durable, and `add_queue` is called with `"durable" => false`, a `Cuniculus::RMQQueueConfigurationConflict` is raised. To redeclare a queue with conflicting configurations, the original queue has first to be removed from RabbitMQ manually. This can be done, for example, through the management console.
39
+ #
40
+ # @param qopts [Hash] Queue config options.
41
+ # @option qopts [String] "name" Name of the queue.
42
+ # @option qopts [Boolean] "durable" (true) Whether queue is declared as durable in RabbitMQ. Jobs in non-durable queues may be lost if the RabbitMQ goes down.
43
+ # @option qopts [Integer] "max_retry" (8) Number of retries for failed jobs in this queue.
44
+ # @option qopts [Integer] "prefetch_count" (10) Prefetch count used when consuming jobs from this queue.
45
+ # @option qopts [Integer] "thread_pool_size" (5) Thread pool size for receiving jobs.
46
+ #
47
+ # @example Add queue named "critical"
48
+ # Cuniculus.configure do |cfg|
49
+ # cfg.add_queue({ name: "critical", max_retry: 10 })
50
+ # end
51
+ def add_queue(qopts)
52
+ qopts = qopts.transform_keys(&:to_s)
53
+ qname = qopts["name"].to_s
54
+ raise Cuniculus::ConfigError, "Missing 'name' key in queue configuration hash" if qname.strip.empty?
55
+ @queues[qname] = QueueConfig.new(qopts)
56
+ end
57
+
29
58
  def declare!
30
59
  conn = ::Bunny.new(rabbitmq_opts.merge(ENFORCED_CONN_OPTS))
31
60
  conn.start
@@ -35,6 +64,10 @@ module Cuniculus
35
64
  @queues.each_value { |q| q.declare!(ch) }
36
65
  end
37
66
 
67
+ # Specify if the default queue `cun_default` should be created.
68
+ # `cun_default` is used by workers that don't explicitly specify a queue with `cuniculus_options queue: "another_queue"`.
69
+ #
70
+ # @param bool [Boolean] If false, queue `cun_default` is not created. Defaults to `true`.
38
71
  def default_queue=(bool)
39
72
  @queues.delete("cun_default") unless bool
40
73
  end
@@ -17,7 +17,6 @@ module Cuniculus
17
17
 
18
18
  def start
19
19
  @exchange = channel.direct(Cuniculus::CUNICULUS_EXCHANGE, { durable: true })
20
- # channel.direct(Cuniculus::CUNICULUS_DLX_EXCHANGE, { durable: true })
21
20
  @job_queue = queue_config.declare!(channel)
22
21
  @_consumer = job_queue.subscribe(manual_ack: true, block: false) do |delivery_info, properties, payload|
23
22
  run_job(delivery_info, properties, payload)
@@ -23,7 +23,7 @@ module Cuniculus
23
23
  # the message and backtrace of `exception`.
24
24
  #
25
25
  # @param exception [Exception] The exception being wrapped
26
- # @param [Cuniculus::Error] The subclass of `Cuniculus::Error`
26
+ # @param klass [Cuniculus::Error] The subclass of `Cuniculus::Error`
27
27
  #
28
28
  # @return [Cuniculus::Error] An instance of the input `Cuniculus::Error`
29
29
  def convert_exception_class(exception, klass)
@@ -5,12 +5,14 @@ module Cuniculus
5
5
  #
6
6
  # * `Cuniculus::Error`: Default exception raised by Cuniculus.
7
7
  # All exceptions classes defined by Cuniculus descend from this class.
8
- # * `Cuniculus::RMQConnectionError`: Raised when unable to connect to RabbitMQ.
9
- # * `Cuniculus::RMQQueueConfigurationConflict`: Raised when the queue configuration
8
+ # * `Cuniculus::BadlyFormattedPayload`: A Cuniculus consumer received an
9
+ # improperly formatted job message.
10
+ # * `Cuniculus::ConfigError`: Incorrect configuration passed to Cuniculus.
11
+ # * `Cuniculus::RMQConnectionError`: Unable to connect to RabbitMQ.
12
+ # * `Cuniculus::RMQQueueConfigurationConflict`: The queue configuration
10
13
  # given to Cuniculus conflicts with the current configuration of the same
11
14
  # existing queue in RabbitMQ.
12
- # * `Cuniculus::BadlyFormattedPayload`: Raised when Cuniculus consumer receives an
13
- # improperly formatted job message.
15
+ # * `Cuniculus::WorkerOptionsError`: Invalid options passed to cuniculus_options.
14
16
 
15
17
  class Error < ::StandardError
16
18
  # If the Cuniculus exception wraps an underlying exception, the latter
@@ -25,6 +27,18 @@ module Cuniculus
25
27
  end
26
28
  end
27
29
 
30
+ # Dev note:
31
+ # As explained [here](https://github.com/jeremyevans/sequel/commit/24681efad0fec48195e43801c224bf18cdc8be13#diff-64cd7b67eccdc6dfa69c23b3b19f34e318f9e6827c5dee5f6e845b2993ab035c), empty classes created
32
+ # with `Class.new` require about 200 bytes less memory than ones created as `class MyClass; end`.
33
+ # The call to `name` is used so that the names of such classes are cached before runtime.
34
+ (
35
+ BadlyFormattedPayload = Class.new(Error)
36
+ ).name
37
+
38
+ (
39
+ ConfigError = Class.new(Error)
40
+ ).name
41
+
28
42
  (
29
43
  RMQConnectionError = Class.new(Error)
30
44
  ).name
@@ -33,7 +47,8 @@ module Cuniculus
33
47
  RMQQueueConfigurationConflict = Class.new(Error)
34
48
  ).name
35
49
 
50
+
36
51
  (
37
- BadlyFormattedPayload = Class.new(Error)
52
+ WorkerOptionsError = Class.new(Error)
38
53
  ).name
39
54
  end
@@ -5,7 +5,7 @@ module Cuniculus
5
5
  class JobQueue
6
6
  extend Forwardable
7
7
 
8
- def_delegators :@base_queue, :subscribe
8
+ def_delegators :@base_queue, :message_count, :subscribe
9
9
 
10
10
  def initialize(base_queue, retry_queue_names)
11
11
  @base_queue = base_queue
@@ -2,11 +2,12 @@
2
2
 
3
3
  require "socket"
4
4
  require "thread"
5
+ require "rack"
5
6
 
6
7
  module Cuniculus
7
8
  module Plugins
8
- # The HealthCheck plugin starts a TCP server together with consumers for health probing.
9
- # It currently does not perform any additional checks returns '200 OK' regardless of whether
9
+ # The HealthCheck plugin starts a Rack server after consumers are initialized, for health probing.
10
+ # It currently does not perform any additional checks and returns '200 OK' regardless of whether
10
11
  # - the node can connect to RabbitMQ;
11
12
  # - consumers are stuck.
12
13
  #
@@ -17,23 +18,23 @@ module Cuniculus
17
18
  # Cuniculus.plugin(:health_check)
18
19
  # ```
19
20
  #
20
- # Options may be passed as well (use `String` keys):
21
+ # Options may be passed as well:
21
22
  # ```ruby
22
23
  # opts = {
23
24
  # "bind_to" => "127.0.0.1", # Default: "0.0.0.0"
24
25
  # "port" => 8080 # Default: 3000
26
+ # "path" => "alive" # Default: "healtcheck"
25
27
  # }
26
28
  # Cuniculus.plugin(:health_check, opts)
27
29
  # ```
28
- # This starts the server bound to 127.0.0.1 and port 8080.
29
- #
30
- # Note that the request path is not considered. The server responds with 200 to any path.
30
+ # This starts the server bound to 127.0.0.1 and port 8080, and responds on path "alive".
31
+ # The server responds with 404 when requests are made to different paths.
31
32
  module HealthCheck
32
- HEALTH_CHECK_RESPONSE = "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 2\r\nConnection: close\r\n\r\nOK"
33
-
34
33
  DEFAULTS = {
35
34
  "bind_to" => "0.0.0.0",
35
+ "path" => "healthcheck",
36
36
  "port" => 3000,
37
+ "quiet" => false,
37
38
  "server" => "webrick",
38
39
  "block" => nil
39
40
  }.freeze
@@ -42,15 +43,19 @@ module Cuniculus
42
43
 
43
44
  # Configure `health_check` plugin
44
45
  #
45
- # @param plugins_cfg [Hash] Global plugin config hash, passed by Cuniculus. This should not be used by plugin users.
46
+ # @param plugins_cfg [Hash] Global plugin config hash, passed by Cuniculus. This should not be modified by plugin users.
46
47
  # @param opts [Hash] Plugin specific options.
47
- # @option opts [String] "bind_to" IP address to bind to (default: "0.0.0.0")
48
- # @option opts [Numeric] "port" Port number to bind to (default: 3000)
48
+ # @option opts [String] "bind_to" ("0.0.0.0") IP address to bind to.
49
+ # @option opts [String] "path" ("healthcheck") Request path to respond to. Requests to other paths will get a 404 response.
50
+ # @option opts [Numeric] "port" (3000) Port number to bind to.
51
+ # @option opts [Boolean] "quiet" (false) Disable server logging to STDOUT and STDERR.
52
+ # @option opts [String] "server" ("webrick") Rack server handler to use .
49
53
  def self.configure(plugins_cfg, opts = {}, &block)
54
+ opts = opts.transform_keys(&:to_s)
50
55
  invalid_opts = opts.keys - DEFAULTS.keys
51
56
  raise Cuniculus::Error, "Invalid option keys for :health_check plugin: #{invalid_opts}" unless invalid_opts.empty?
52
57
 
53
- plugins_cfg[OPTS_KEY] = h = opts.slice("bind_to", "port", "server")
58
+ plugins_cfg[OPTS_KEY] = h = opts.slice("bind_to", "path", "port", "quiet", "server")
54
59
  h["block"] = block if block
55
60
  DEFAULTS.each do |k, v|
56
61
  h[k] = v if v && !h.key?(k)
@@ -58,45 +63,43 @@ module Cuniculus
58
63
  end
59
64
 
60
65
  module SupervisorMethods
66
+ def initialize(config)
67
+ super(config)
68
+ hc_plugin_opts = config.opts[OPTS_KEY]
69
+ @hc_server = Rack::Handler.get(hc_plugin_opts["server"])
70
+ @hc_rack_app = build_rack_app(hc_plugin_opts)
71
+ end
72
+
61
73
  def start
62
- hc_rd, @hc_wr = IO.pipe
63
- start_health_check_server(hc_rd)
74
+ start_health_check_server
64
75
  super
65
76
  end
66
77
 
67
78
  def stop
68
- @hc_wr << "a"
79
+ @hc_server.shutdown
69
80
  super
70
81
  end
71
82
 
72
83
 
73
84
  private
74
85
 
75
- def start_health_check_server(pipe_reader)
76
- opts = config.opts[OPTS_KEY]
77
- server = ::TCPServer.new(opts["bind_to"], opts["port"])
78
-
79
- # If port was assigned by OS (when 'port' option was given as 0),
80
- # now override input value with it.
81
- opts["port"] = server.addr[1]
82
- @hc_thread = Thread.new do
83
- sock = nil
84
- done = false
85
- loop do
86
- begin
87
- break if done
88
- sock = server.accept_nonblock
89
- rescue IO::WaitReadable, Errno::EINTR
90
- io = IO.select([server, pipe_reader])
91
- done = true if io.first.include?(pipe_reader)
92
- retry
93
- end
94
-
95
- sock.print HEALTH_CHECK_RESPONSE
96
- sock.shutdown
86
+ def build_rack_app(opts)
87
+ app = ::Object.new
88
+ app.define_singleton_method(:call) do |env|
89
+ if Rack::Request.new(env).path == "/#{opts['path']}"
90
+ [200, {}, ["OK"]]
91
+ else
92
+ [404, {}, ["Not Found"]]
97
93
  end
94
+ end
95
+ app
96
+ end
98
97
 
99
- sock&.close if sock && !sock.closed?
98
+ def start_health_check_server
99
+ opts = config.opts[OPTS_KEY]
100
+ Thread.new do
101
+ access_log = opts["quiet"] ? [] : nil
102
+ @hc_server.run(@hc_rack_app, AccessLog: access_log, Port: opts["port"], Host: opts["bind_to"])
100
103
  end
101
104
  end
102
105
  end
@@ -6,27 +6,32 @@ require "cuniculus/job_queue"
6
6
 
7
7
  module Cuniculus
8
8
  class QueueConfig
9
- OPTS = {}.freeze
10
-
11
- DEFAULT_MAX_RETRY = 4
12
-
13
- attr_reader :max_retry, :name, :thread_pool_size
14
-
15
- def initialize(opts = OPTS)
16
- @name = read_opt(opts, "name") || "cun_default"
17
- @max_retry = read_opt(opts, "max_retry") || DEFAULT_MAX_RETRY
18
- @thread_pool_size = read_opt(opts, "thread_pool_size")
9
+ DEFAULT_MAX_RETRY = 8
10
+ DEFAULT_PREFETCH_COUNT = 10
11
+ DEFAULT_QUEUE_NAME = "cun_default"
12
+ DEFAULT_THREAD_POOL_SIZE = 5
13
+
14
+ attr_reader :durable, :max_retry, :name, :prefetch_count, :thread_pool_size
15
+
16
+ def initialize(opts = {})
17
+ opts = opts.transform_keys(&:to_s)
18
+ @durable = read_opt(opts["durable"], true)
19
+ @name = read_opt(opts["name"], DEFAULT_QUEUE_NAME)
20
+ @max_retry = read_opt(opts["max_retry"], DEFAULT_MAX_RETRY)
21
+ @prefetch_count = read_opt(opts["prefetch_count"], DEFAULT_PREFETCH_COUNT)
22
+ @thread_pool_size = read_opt(opts["thread_pool_size"], DEFAULT_THREAD_POOL_SIZE)
23
+ freeze
19
24
  end
20
25
 
21
- def read_opt(opts, key)
22
- opts[key.to_s] || opts[key.to_sym]
26
+ def read_opt(val, default)
27
+ val.nil? ? default : val
23
28
  end
24
29
 
25
30
  def declare!(channel)
26
31
  queue_name = name
27
32
  base_q = channel.queue(
28
33
  queue_name,
29
- durable: true,
34
+ durable: durable,
30
35
  exclusive: false,
31
36
  arguments: { "x-dead-letter-exchange" => Cuniculus::CUNICULUS_DLX_EXCHANGE }
32
37
  )
@@ -38,7 +43,7 @@ module Cuniculus
38
43
 
39
44
  q = channel.queue(
40
45
  queue_name,
41
- durable: true,
46
+ durable: durable,
42
47
  exclusive: false,
43
48
  arguments: {
44
49
  "x-dead-letter-exchange" => Cuniculus::CUNICULUS_EXCHANGE,
@@ -37,9 +37,9 @@ module Cuniculus
37
37
 
38
38
  def create_consumers(conn, queues)
39
39
  consumers = []
40
- consumer_pool_size = 5
41
40
  queues.each do |_name, q_cfg|
42
- ch = conn.create_channel(nil, consumer_pool_size)
41
+ ch = conn.create_channel(nil, q_cfg.thread_pool_size)
42
+ ch.prefetch(q_cfg.prefetch_count) if q_cfg.prefetch_count
43
43
  consumers << Cuniculus::Consumer.new(q_cfg, ch)
44
44
  end
45
45
  consumers
@@ -6,11 +6,11 @@ module Cuniculus
6
6
 
7
7
  # The minor version of Cuniculus. Bumped for every non-patch level
8
8
  # release.
9
- MINOR = 0
9
+ MINOR = 1
10
10
 
11
11
  # The tiny version of Cuniculus. Usually 0, only bumped for bugfix
12
12
  # releases that fix regressions from previous versions.
13
- TINY = 1
13
+ TINY = 0
14
14
 
15
15
  # The version of Cuniculus you are using, as a string (e.g. "2.11.0")
16
16
  VERSION = [MAJOR, MINOR, TINY].join(".").freeze
@@ -7,15 +7,67 @@ module Cuniculus
7
7
  module Worker
8
8
  def self.included(base)
9
9
  base.extend(ClassMethods)
10
+
11
+ # Dev note:
12
+ # The point here is to allow options set via cuniculus_options to be
13
+ # inherited by subclasses.
14
+ # When reading the options, a subclass will call the singleton method cun_opts.
15
+ # If the subclass doesn't redefine this method via a call to cuniculus_options,
16
+ # it will still use the definition from its parent class.
17
+ base.define_singleton_method("cun_opts=") do |opts|
18
+ singleton_class.class_eval do
19
+ define_method("cun_opts") { opts }
20
+ end
21
+ end
10
22
  end
11
23
 
12
24
  module ClassMethods
25
+ DEFAULT_OPTS = { "queue" => "cun_default" }.freeze
26
+ VALID_OPT_KEYS = %w[queue].freeze
27
+
28
+ # Read-only cuniculus option values
29
+ #
30
+ # @return opts [Hash] hash with current values
31
+ def cun_opts
32
+ DEFAULT_OPTS
33
+ end
34
+
35
+ # Worker-specific options for running cuniculus.
36
+ #
37
+ # Note that options set on a worker class are inherited by its subclasses.
38
+ #
39
+ # @param opts [Hash]
40
+ # @option opts [String] "queue" ("cun_default") Name of the underlying RabbitMQ queue.
41
+ #
42
+ # @example Change the queue name of a worker
43
+ # class MyWorker
44
+ # include Cuniculus::Worker
45
+ #
46
+ # cuniculus_options queue: "critical"
47
+ #
48
+ # def perform
49
+ # # run the task
50
+ # end
51
+ # end
52
+ def cuniculus_options(opts)
53
+ opts = validate_opts!(opts)
54
+ self.cun_opts = opts
55
+ end
56
+
57
+ def validate_opts!(opts)
58
+ raise WorkerOptionsError, "Argument passed to 'cuniculus_options' should be a Hash" unless opts.is_a?(Hash)
59
+ opts = opts.transform_keys(&:to_s)
60
+ invalid_keys = opts.keys - VALID_OPT_KEYS
61
+ raise WorkerOptionsError, "Invalid keys passed to 'cuniculus_options': #{invalid_keys.inspect}" unless invalid_keys.empty?
62
+ opts
63
+ end
64
+
13
65
  def perform_async(*args)
14
66
  publish({ "class" => self, "args" => args })
15
67
  end
16
68
 
17
69
  def publish(item)
18
- routing_key = "cun_default"
70
+ routing_key = cun_opts["queue"]
19
71
  payload = normalize_item(item)
20
72
  Cuniculus::RMQPool.with_exchange do |x|
21
73
  x.publish(payload, { routing_key: routing_key, persistent: true })
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: cuniculus
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.1
4
+ version: 0.1.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Marcelo Pereira
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2021-02-14 00:00:00.000000000 Z
11
+ date: 2021-03-07 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bunny
@@ -52,6 +52,20 @@ dependencies:
52
52
  - - ">="
53
53
  - !ruby/object:Gem::Version
54
54
  version: '0'
55
+ - !ruby/object:Gem::Dependency
56
+ name: rack
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - ">="
60
+ - !ruby/object:Gem::Version
61
+ version: '0'
62
+ type: :development
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - ">="
67
+ - !ruby/object:Gem::Version
68
+ version: '0'
55
69
  - !ruby/object:Gem::Dependency
56
70
  name: redcarpet
57
71
  requirement: !ruby/object:Gem::Requirement
@@ -94,6 +108,20 @@ dependencies:
94
108
  - - ">="
95
109
  - !ruby/object:Gem::Version
96
110
  version: '0'
111
+ - !ruby/object:Gem::Dependency
112
+ name: toxiproxy
113
+ requirement: !ruby/object:Gem::Requirement
114
+ requirements:
115
+ - - ">="
116
+ - !ruby/object:Gem::Version
117
+ version: '0'
118
+ type: :development
119
+ prerelease: false
120
+ version_requirements: !ruby/object:Gem::Requirement
121
+ requirements:
122
+ - - ">="
123
+ - !ruby/object:Gem::Version
124
+ version: '0'
97
125
  - !ruby/object:Gem::Dependency
98
126
  name: warning
99
127
  requirement: !ruby/object:Gem::Requirement
@@ -108,6 +136,20 @@ dependencies:
108
136
  - - ">="
109
137
  - !ruby/object:Gem::Version
110
138
  version: '0'
139
+ - !ruby/object:Gem::Dependency
140
+ name: webrick
141
+ requirement: !ruby/object:Gem::Requirement
142
+ requirements:
143
+ - - ">="
144
+ - !ruby/object:Gem::Version
145
+ version: '0'
146
+ type: :development
147
+ prerelease: false
148
+ version_requirements: !ruby/object:Gem::Requirement
149
+ requirements:
150
+ - - ">="
151
+ - !ruby/object:Gem::Version
152
+ version: '0'
111
153
  - !ruby/object:Gem::Dependency
112
154
  name: yard
113
155
  requirement: !ruby/object:Gem::Requirement