cuniculus 0.0.1 → 0.1.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: a706ee37ab059f8897e2aba8cd1ab3a6dc15cdde2eabc78d98ddecd5113afb2a
4
- data.tar.gz: 02bc1b2265e346485e99f583ed3f3c3a619335238d7f1552d809490dea3d5a0d
3
+ metadata.gz: ca0cc8cbcc9cc9607488d5c7c88903548fe4106799e6517e82b2efb5824a9971
4
+ data.tar.gz: 6a0589c882ae8de39a0192e2e5b54c537a709fbe6ca496ce3b999d3a03ff6da1
5
5
  SHA512:
6
- metadata.gz: bc3a8d8a26b299ffab344a642fc539e762fd03f3922bd165ff0611d44a0ecd62a39cd4a242fe529bc283e063764a15e399c2f4ad1dbbb4112c97863953f032ab
7
- data.tar.gz: efc582168bc82aa2b5a66ecb42f26dc1d4fc36a73f95a9f46fb9028114a0781d0ad0f4cc10dc4fb41ff0f32f93550c8f745b80fb1e6a93c16a502e7f19e4a73e
6
+ metadata.gz: 73fc5ea97169bdbdf5a72b95dec76e2b529a13c6b308a3c3c9c947d891bc3845d48ad5b4344820d0f1fed8cbed8b0bb1ffac3a4c3ea043ca0cb3c1737eb8014d
7
+ data.tar.gz: 7dbdd0c30c1dc808457fc408506aade75b65de83f5318799f8b7a5aab4c0891ed71fbb927e58eaa70b099442790b1c1d20b06d34d12a53967def89052f068887
data/LICENSE CHANGED
@@ -1,6 +1,6 @@
1
1
  BSD 2-Clause License
2
2
 
3
- Copyright (c) 2020, Marcelo
3
+ Copyright (c) 2021, Marcelo
4
4
  All rights reserved.
5
5
 
6
6
  Redistribution and use in source and binary forms, with or without
data/README.md CHANGED
@@ -2,6 +2,36 @@
2
2
 
3
3
  Ruby job queue backed by RabbitMQ. The word _cuniculus_ comes from the scientific name of the European rabbit (Oryctolagus cuniculus).
4
4
 
5
+ ## Benchmarks
6
+
7
+ The following measurements were performed with the `bin/run_benchmarks` utility, with different command parameters. Run it with `-h` to see its usage.
8
+
9
+ To simulate network latency, [Toxiproxy](https://github.com/Shopify/toxiproxy) was used. It needs to be started with `toxiproxy-server` before running the benchmarks.
10
+
11
+ Network latency (_ms_) | Prefetch count | Throughput (_jobs/s_) | Average latency (_ms_)
12
+ ----------------------:|---------------------:|----------------------:|----------------------:
13
+ 1 | 65535 (max. allowed) | 10225 | 2
14
+ 10 | 65535 (max. allowed) | 9990 | 13
15
+ 1 | 50 | 8051 | 2
16
+ 10 | 50 | 2500 | 13
17
+ 100 | 50 | 481 | 103
18
+ 1 | 25 | 7824 | 2
19
+ 10 | 25 | 1824 | 13
20
+ 50 | 25 | 469 | 53
21
+ 1 | 10 (default) | 5266 | 2
22
+ 10 | 10 (default) | 807 | 13
23
+ 1 | 1 | 481 | 2
24
+ 10 | 1 | 81 | 13
25
+
26
+ Additional benchmark parameters:
27
+ - throughput was measured by consuming 100k jobs;
28
+ - job latency was averaged over 200 samples;
29
+ - Ruby 2.7.2 was used.
30
+
31
+ Several remarks can be made:
32
+ - Higher prefetch counts lead to higher throughput, but there are downsides of having it too high; see [this reference](https://www.cloudamqp.com/blog/2017-12-29-part1-rabbitmq-best-practice.html#prefetch) on how to properly tune it.
33
+ - Network latency has a severe impact on the throughput, and the effect is larger the smaller the prefetch count is.
34
+
5
35
  ## Getting started
6
36
 
7
37
  ```sh
@@ -18,6 +48,8 @@ require 'cuniculus/worker'
18
48
  class MyWorker
19
49
  include Cuniculus::Worker
20
50
 
51
+ # the queue name is not explicitly given, so "cun_default" is used.
52
+
21
53
  def perform(arg1, arg2)
22
54
  puts "Processing:"
23
55
  puts "arg1: #{arg1.inspect}"
@@ -52,6 +84,8 @@ There is also a more complete example in the Cuniculus repository itself. To run
52
84
  bin/cuniculus -I examples/ -r example/init_cuniculus.rb
53
85
  ```
54
86
 
87
+ The `-I examples` option adds the `examples/` directory into the load path, and `-r example/init_cuniculus.rb` requires `init_cuniculus.rb` prior to starting the consumer. The latter is where configurations such as that described in the next section should be.
88
+
55
89
  ## Configuration
56
90
 
57
91
  Configuration is done through code, using `Cuniculus.configure`.
@@ -75,6 +109,20 @@ Cuniculus.configure do |cfg|
75
109
  cfg.rabbitmq_opts = rabbitmq_conn
76
110
  cfg.pub_thr_pool_size = 5 # Only affects job producers
77
111
  cfg.dead_queue_ttl = 1000 * 60 * 60 * 24 * 30 # keep failed jobs for 30 days
112
+ cfg.add_queue({ name: "critical", durable: true, max_retry: 10, prefetch_count: 1})
113
+ end
114
+ ```
115
+
116
+ To configure the queue used by a worker, used `cuniculus_options`:
117
+
118
+ ```ruby
119
+ class MyWorker
120
+ include Cuniculus::Worker
121
+
122
+ cuniculus_options queue: "critical"
123
+ def perform
124
+ # code
125
+ end
78
126
  end
79
127
  ```
80
128
 
@@ -92,15 +140,32 @@ The method expects a block that will receive an exception, and run in the scope
92
140
 
93
141
  ## Retry mechanism
94
142
 
95
- Cuniculus declares a `cun_default` queue, together with some `cun_default_{n}` queues used for job retries.
143
+ Retries are enabled by default (with 8 retries) with an exponential backoff, meaning the time between retries increases the more failures happen. The formula for calculating the times between retries can be found in {Cuniculus::QueueConfig}, namely in the `x-message-ttl` line. As an example, the time between the 7th and 8th retries is roughly 29 days.
144
+
145
+ Given a declared queue in the configuration, Cuniculus starts on RabbitMQ the corresponding base queue, in addition to its retry queues. As an example, let's consider the default queue `cun_default`: Cuniculus declares a `cun_default` queue, together with some `cun_default_{n}` queues used for job retries.
146
+
96
147
  When a job raises an exception, it is placed into the `cun_default_1` queue for the first retry. It stays there for some pre-defined time, and then gets moved back into the `cun_default` queue for execution.
97
148
 
98
149
  If it fails again, it gets moved to `cun_default_2`, where it stays for a longer period until it's moved back directly into the `cun_default` queue again.
99
150
 
100
- This goes on until there are no more retry attempts, in which case the job gets moved into the `cun_dead` queue. It can be then only be moved back into the `cun_default` queue manually; otherwise it is discarded after some time, defined as the `dead_queue_ttl`, in milliseconds (by default, 180 days).
151
+ This goes on until there are no more retry attempts, in which case the job gets moved into the `cun_dead` queue. It can be then only be moved back into the `cun_default` queue manually; otherwise it is discarded after some time, defined as the {Cuniculus::Config.dead_queue_ttl}, in milliseconds (by default, 180 days).
101
152
 
102
153
  Note that if a job cannot even be parsed, it is moved straight to the dead queue, as there's no point in retrying.
103
154
 
155
+ ## Health check plugin
156
+
157
+ Cuniculus ships with a health check plugin. When enabled, a Rack server is started (therefore the [Rack](https://github.com/rack/rack) gem is required, as well as the used handler), which responds with `200 OK` upon receiving a request in the configured port and path.
158
+
159
+ Enable it with `Cuniculus.plugin(:health_check)`, which binds the server to `0.0.0.0:3000`, listening on the `/healthcheck` path. To configure the server, pass additional options:
160
+
161
+ ```ruby
162
+ Cuniculus.plugin(:health_check, { "bind_to" => "127.0.0.1", "port" => 3003, "path" => "ping" })
163
+ ```
164
+
165
+ Check {Cuniculus::Plugins::HealthCheck} for further details.
166
+
167
+ _Note that the default handler "webrick" is not bundled by default with Ruby 3 and needs to be installed separately, if it is to be used._
168
+
104
169
  ## How it works
105
170
 
106
171
  Cuniculus code and conventions are very much inspired by another Ruby job queue library: [Sidekiq](https://github.com/mperham/sidekiq).
data/lib/cuniculus.rb CHANGED
@@ -14,12 +14,14 @@ require "cuniculus/supervisor"
14
14
  module Cuniculus
15
15
 
16
16
  # Configure Cuniculus.
17
+ # Check {Cuniculus::Config} for the available options.
17
18
  #
18
19
  # @yield [Cuniculus::Config]
19
20
  #
20
21
  # @example Change RabbitMQ connection details.
21
22
  # Cuniculus.configure do |cfg|
22
23
  # cfg.rabbitmq_opts = { host: 'rmq.mycompany.com', user: 'guest', pass: 'guest' }
24
+ # cfg.add_queue({ name: "new_queue", max_retry: 4 })
23
25
  # end
24
26
  def self.configure
25
27
  cfg = Cuniculus::Config.new
@@ -1,6 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  require "cuniculus/core"
4
+ require "cuniculus/exceptions"
4
5
  require "cuniculus/queue_config"
5
6
 
6
7
  module Cuniculus
@@ -23,9 +24,37 @@ module Cuniculus
23
24
  vhost: "/"
24
25
  }
25
26
  @exchange_name = "cuniculus"
27
+ @pub_thr_pool_size = 5
26
28
  @dead_queue_ttl = 1000 * 60 * 60 * 24 * 180 # 180 days
27
29
  end
28
30
 
31
+ # Configure an additional queue
32
+ #
33
+ # Note that a single call to `add_queue` might lead to the creation of multiple queues on RabbitMQ: one base queue, and an additional queue for every retry attempt.
34
+ # For example, with a queue named `"test"` with `max_retry` set to `4`, 5 queues are created in RabbitMQ.
35
+ #
36
+ # For tuning `prefetch_count`, refer to [this guide](https://www.cloudamqp.com/blog/2017-12-29-part1-rabbitmq-best-practice.html#prefetch).
37
+ #
38
+ # If a queue already exists in RabbitMQ, and an attempt is done to add it again through `add_queue`, nothing happens, except if the options passed to `add_queue` conflict with the existing queue. For example if a queue exists that is durable, and `add_queue` is called with `"durable" => false`, a `Cuniculus::RMQQueueConfigurationConflict` is raised. To redeclare a queue with conflicting configurations, the original queue has first to be removed from RabbitMQ manually. This can be done, for example, through the management console.
39
+ #
40
+ # @param qopts [Hash] Queue config options.
41
+ # @option qopts [String] "name" Name of the queue.
42
+ # @option qopts [Boolean] "durable" (true) Whether queue is declared as durable in RabbitMQ. Jobs in non-durable queues may be lost if the RabbitMQ goes down.
43
+ # @option qopts [Integer] "max_retry" (8) Number of retries for failed jobs in this queue.
44
+ # @option qopts [Integer] "prefetch_count" (10) Prefetch count used when consuming jobs from this queue.
45
+ # @option qopts [Integer] "thread_pool_size" (5) Thread pool size for receiving jobs.
46
+ #
47
+ # @example Add queue named "critical"
48
+ # Cuniculus.configure do |cfg|
49
+ # cfg.add_queue({ name: "critical", max_retry: 10 })
50
+ # end
51
+ def add_queue(qopts)
52
+ qopts = qopts.transform_keys(&:to_s)
53
+ qname = qopts["name"].to_s
54
+ raise Cuniculus::ConfigError, "Missing 'name' key in queue configuration hash" if qname.strip.empty?
55
+ @queues[qname] = QueueConfig.new(qopts)
56
+ end
57
+
29
58
  def declare!
30
59
  conn = ::Bunny.new(rabbitmq_opts.merge(ENFORCED_CONN_OPTS))
31
60
  conn.start
@@ -35,6 +64,10 @@ module Cuniculus
35
64
  @queues.each_value { |q| q.declare!(ch) }
36
65
  end
37
66
 
67
+ # Specify if the default queue `cun_default` should be created.
68
+ # `cun_default` is used by workers that don't explicitly specify a queue with `cuniculus_options queue: "another_queue"`.
69
+ #
70
+ # @param bool [Boolean] If false, queue `cun_default` is not created. Defaults to `true`.
38
71
  def default_queue=(bool)
39
72
  @queues.delete("cun_default") unless bool
40
73
  end
@@ -17,7 +17,6 @@ module Cuniculus
17
17
 
18
18
  def start
19
19
  @exchange = channel.direct(Cuniculus::CUNICULUS_EXCHANGE, { durable: true })
20
- # channel.direct(Cuniculus::CUNICULUS_DLX_EXCHANGE, { durable: true })
21
20
  @job_queue = queue_config.declare!(channel)
22
21
  @_consumer = job_queue.subscribe(manual_ack: true, block: false) do |delivery_info, properties, payload|
23
22
  run_job(delivery_info, properties, payload)
@@ -23,7 +23,7 @@ module Cuniculus
23
23
  # the message and backtrace of `exception`.
24
24
  #
25
25
  # @param exception [Exception] The exception being wrapped
26
- # @param [Cuniculus::Error] The subclass of `Cuniculus::Error`
26
+ # @param klass [Cuniculus::Error] The subclass of `Cuniculus::Error`
27
27
  #
28
28
  # @return [Cuniculus::Error] An instance of the input `Cuniculus::Error`
29
29
  def convert_exception_class(exception, klass)
@@ -5,12 +5,14 @@ module Cuniculus
5
5
  #
6
6
  # * `Cuniculus::Error`: Default exception raised by Cuniculus.
7
7
  # All exceptions classes defined by Cuniculus descend from this class.
8
- # * `Cuniculus::RMQConnectionError`: Raised when unable to connect to RabbitMQ.
9
- # * `Cuniculus::RMQQueueConfigurationConflict`: Raised when the queue configuration
8
+ # * `Cuniculus::BadlyFormattedPayload`: A Cuniculus consumer received an
9
+ # improperly formatted job message.
10
+ # * `Cuniculus::ConfigError`: Incorrect configuration passed to Cuniculus.
11
+ # * `Cuniculus::RMQConnectionError`: Unable to connect to RabbitMQ.
12
+ # * `Cuniculus::RMQQueueConfigurationConflict`: The queue configuration
10
13
  # given to Cuniculus conflicts with the current configuration of the same
11
14
  # existing queue in RabbitMQ.
12
- # * `Cuniculus::BadlyFormattedPayload`: Raised when Cuniculus consumer receives an
13
- # improperly formatted job message.
15
+ # * `Cuniculus::WorkerOptionsError`: Invalid options passed to cuniculus_options.
14
16
 
15
17
  class Error < ::StandardError
16
18
  # If the Cuniculus exception wraps an underlying exception, the latter
@@ -25,6 +27,18 @@ module Cuniculus
25
27
  end
26
28
  end
27
29
 
30
+ # Dev note:
31
+ # As explained [here](https://github.com/jeremyevans/sequel/commit/24681efad0fec48195e43801c224bf18cdc8be13#diff-64cd7b67eccdc6dfa69c23b3b19f34e318f9e6827c5dee5f6e845b2993ab035c), empty classes created
32
+ # with `Class.new` require about 200 bytes less memory than ones created as `class MyClass; end`.
33
+ # The call to `name` is used so that the names of such classes are cached before runtime.
34
+ (
35
+ BadlyFormattedPayload = Class.new(Error)
36
+ ).name
37
+
38
+ (
39
+ ConfigError = Class.new(Error)
40
+ ).name
41
+
28
42
  (
29
43
  RMQConnectionError = Class.new(Error)
30
44
  ).name
@@ -33,7 +47,8 @@ module Cuniculus
33
47
  RMQQueueConfigurationConflict = Class.new(Error)
34
48
  ).name
35
49
 
50
+
36
51
  (
37
- BadlyFormattedPayload = Class.new(Error)
52
+ WorkerOptionsError = Class.new(Error)
38
53
  ).name
39
54
  end
@@ -5,7 +5,7 @@ module Cuniculus
5
5
  class JobQueue
6
6
  extend Forwardable
7
7
 
8
- def_delegators :@base_queue, :subscribe
8
+ def_delegators :@base_queue, :message_count, :subscribe
9
9
 
10
10
  def initialize(base_queue, retry_queue_names)
11
11
  @base_queue = base_queue
@@ -2,11 +2,12 @@
2
2
 
3
3
  require "socket"
4
4
  require "thread"
5
+ require "rack"
5
6
 
6
7
  module Cuniculus
7
8
  module Plugins
8
- # The HealthCheck plugin starts a TCP server together with consumers for health probing.
9
- # It currently does not perform any additional checks returns '200 OK' regardless of whether
9
+ # The HealthCheck plugin starts a Rack server after consumers are initialized, for health probing.
10
+ # It currently does not perform any additional checks and returns '200 OK' regardless of whether
10
11
  # - the node can connect to RabbitMQ;
11
12
  # - consumers are stuck.
12
13
  #
@@ -17,23 +18,23 @@ module Cuniculus
17
18
  # Cuniculus.plugin(:health_check)
18
19
  # ```
19
20
  #
20
- # Options may be passed as well (use `String` keys):
21
+ # Options may be passed as well:
21
22
  # ```ruby
22
23
  # opts = {
23
24
  # "bind_to" => "127.0.0.1", # Default: "0.0.0.0"
24
25
  # "port" => 8080 # Default: 3000
26
+ # "path" => "alive" # Default: "healtcheck"
25
27
  # }
26
28
  # Cuniculus.plugin(:health_check, opts)
27
29
  # ```
28
- # This starts the server bound to 127.0.0.1 and port 8080.
29
- #
30
- # Note that the request path is not considered. The server responds with 200 to any path.
30
+ # This starts the server bound to 127.0.0.1 and port 8080, and responds on path "alive".
31
+ # The server responds with 404 when requests are made to different paths.
31
32
  module HealthCheck
32
- HEALTH_CHECK_RESPONSE = "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 2\r\nConnection: close\r\n\r\nOK"
33
-
34
33
  DEFAULTS = {
35
34
  "bind_to" => "0.0.0.0",
35
+ "path" => "healthcheck",
36
36
  "port" => 3000,
37
+ "quiet" => false,
37
38
  "server" => "webrick",
38
39
  "block" => nil
39
40
  }.freeze
@@ -42,15 +43,19 @@ module Cuniculus
42
43
 
43
44
  # Configure `health_check` plugin
44
45
  #
45
- # @param plugins_cfg [Hash] Global plugin config hash, passed by Cuniculus. This should not be used by plugin users.
46
+ # @param plugins_cfg [Hash] Global plugin config hash, passed by Cuniculus. This should not be modified by plugin users.
46
47
  # @param opts [Hash] Plugin specific options.
47
- # @option opts [String] "bind_to" IP address to bind to (default: "0.0.0.0")
48
- # @option opts [Numeric] "port" Port number to bind to (default: 3000)
48
+ # @option opts [String] "bind_to" ("0.0.0.0") IP address to bind to.
49
+ # @option opts [String] "path" ("healthcheck") Request path to respond to. Requests to other paths will get a 404 response.
50
+ # @option opts [Numeric] "port" (3000) Port number to bind to.
51
+ # @option opts [Boolean] "quiet" (false) Disable server logging to STDOUT and STDERR.
52
+ # @option opts [String] "server" ("webrick") Rack server handler to use .
49
53
  def self.configure(plugins_cfg, opts = {}, &block)
54
+ opts = opts.transform_keys(&:to_s)
50
55
  invalid_opts = opts.keys - DEFAULTS.keys
51
56
  raise Cuniculus::Error, "Invalid option keys for :health_check plugin: #{invalid_opts}" unless invalid_opts.empty?
52
57
 
53
- plugins_cfg[OPTS_KEY] = h = opts.slice("bind_to", "port", "server")
58
+ plugins_cfg[OPTS_KEY] = h = opts.slice("bind_to", "path", "port", "quiet", "server")
54
59
  h["block"] = block if block
55
60
  DEFAULTS.each do |k, v|
56
61
  h[k] = v if v && !h.key?(k)
@@ -58,45 +63,43 @@ module Cuniculus
58
63
  end
59
64
 
60
65
  module SupervisorMethods
66
+ def initialize(config)
67
+ super(config)
68
+ hc_plugin_opts = config.opts[OPTS_KEY]
69
+ @hc_server = Rack::Handler.get(hc_plugin_opts["server"])
70
+ @hc_rack_app = build_rack_app(hc_plugin_opts)
71
+ end
72
+
61
73
  def start
62
- hc_rd, @hc_wr = IO.pipe
63
- start_health_check_server(hc_rd)
74
+ start_health_check_server
64
75
  super
65
76
  end
66
77
 
67
78
  def stop
68
- @hc_wr << "a"
79
+ @hc_server.shutdown
69
80
  super
70
81
  end
71
82
 
72
83
 
73
84
  private
74
85
 
75
- def start_health_check_server(pipe_reader)
76
- opts = config.opts[OPTS_KEY]
77
- server = ::TCPServer.new(opts["bind_to"], opts["port"])
78
-
79
- # If port was assigned by OS (when 'port' option was given as 0),
80
- # now override input value with it.
81
- opts["port"] = server.addr[1]
82
- @hc_thread = Thread.new do
83
- sock = nil
84
- done = false
85
- loop do
86
- begin
87
- break if done
88
- sock = server.accept_nonblock
89
- rescue IO::WaitReadable, Errno::EINTR
90
- io = IO.select([server, pipe_reader])
91
- done = true if io.first.include?(pipe_reader)
92
- retry
93
- end
94
-
95
- sock.print HEALTH_CHECK_RESPONSE
96
- sock.shutdown
86
+ def build_rack_app(opts)
87
+ app = ::Object.new
88
+ app.define_singleton_method(:call) do |env|
89
+ if Rack::Request.new(env).path == "/#{opts['path']}"
90
+ [200, {}, ["OK"]]
91
+ else
92
+ [404, {}, ["Not Found"]]
97
93
  end
94
+ end
95
+ app
96
+ end
98
97
 
99
- sock&.close if sock && !sock.closed?
98
+ def start_health_check_server
99
+ opts = config.opts[OPTS_KEY]
100
+ Thread.new do
101
+ access_log = opts["quiet"] ? [] : nil
102
+ @hc_server.run(@hc_rack_app, AccessLog: access_log, Port: opts["port"], Host: opts["bind_to"])
100
103
  end
101
104
  end
102
105
  end
@@ -6,27 +6,32 @@ require "cuniculus/job_queue"
6
6
 
7
7
  module Cuniculus
8
8
  class QueueConfig
9
- OPTS = {}.freeze
10
-
11
- DEFAULT_MAX_RETRY = 4
12
-
13
- attr_reader :max_retry, :name, :thread_pool_size
14
-
15
- def initialize(opts = OPTS)
16
- @name = read_opt(opts, "name") || "cun_default"
17
- @max_retry = read_opt(opts, "max_retry") || DEFAULT_MAX_RETRY
18
- @thread_pool_size = read_opt(opts, "thread_pool_size")
9
+ DEFAULT_MAX_RETRY = 8
10
+ DEFAULT_PREFETCH_COUNT = 10
11
+ DEFAULT_QUEUE_NAME = "cun_default"
12
+ DEFAULT_THREAD_POOL_SIZE = 5
13
+
14
+ attr_reader :durable, :max_retry, :name, :prefetch_count, :thread_pool_size
15
+
16
+ def initialize(opts = {})
17
+ opts = opts.transform_keys(&:to_s)
18
+ @durable = read_opt(opts["durable"], true)
19
+ @name = read_opt(opts["name"], DEFAULT_QUEUE_NAME)
20
+ @max_retry = read_opt(opts["max_retry"], DEFAULT_MAX_RETRY)
21
+ @prefetch_count = read_opt(opts["prefetch_count"], DEFAULT_PREFETCH_COUNT)
22
+ @thread_pool_size = read_opt(opts["thread_pool_size"], DEFAULT_THREAD_POOL_SIZE)
23
+ freeze
19
24
  end
20
25
 
21
- def read_opt(opts, key)
22
- opts[key.to_s] || opts[key.to_sym]
26
+ def read_opt(val, default)
27
+ val.nil? ? default : val
23
28
  end
24
29
 
25
30
  def declare!(channel)
26
31
  queue_name = name
27
32
  base_q = channel.queue(
28
33
  queue_name,
29
- durable: true,
34
+ durable: durable,
30
35
  exclusive: false,
31
36
  arguments: { "x-dead-letter-exchange" => Cuniculus::CUNICULUS_DLX_EXCHANGE }
32
37
  )
@@ -38,7 +43,7 @@ module Cuniculus
38
43
 
39
44
  q = channel.queue(
40
45
  queue_name,
41
- durable: true,
46
+ durable: durable,
42
47
  exclusive: false,
43
48
  arguments: {
44
49
  "x-dead-letter-exchange" => Cuniculus::CUNICULUS_EXCHANGE,
@@ -37,9 +37,9 @@ module Cuniculus
37
37
 
38
38
  def create_consumers(conn, queues)
39
39
  consumers = []
40
- consumer_pool_size = 5
41
40
  queues.each do |_name, q_cfg|
42
- ch = conn.create_channel(nil, consumer_pool_size)
41
+ ch = conn.create_channel(nil, q_cfg.thread_pool_size)
42
+ ch.prefetch(q_cfg.prefetch_count) if q_cfg.prefetch_count
43
43
  consumers << Cuniculus::Consumer.new(q_cfg, ch)
44
44
  end
45
45
  consumers
@@ -6,11 +6,11 @@ module Cuniculus
6
6
 
7
7
  # The minor version of Cuniculus. Bumped for every non-patch level
8
8
  # release.
9
- MINOR = 0
9
+ MINOR = 1
10
10
 
11
11
  # The tiny version of Cuniculus. Usually 0, only bumped for bugfix
12
12
  # releases that fix regressions from previous versions.
13
- TINY = 1
13
+ TINY = 0
14
14
 
15
15
  # The version of Cuniculus you are using, as a string (e.g. "2.11.0")
16
16
  VERSION = [MAJOR, MINOR, TINY].join(".").freeze
@@ -7,15 +7,67 @@ module Cuniculus
7
7
  module Worker
8
8
  def self.included(base)
9
9
  base.extend(ClassMethods)
10
+
11
+ # Dev note:
12
+ # The point here is to allow options set via cuniculus_options to be
13
+ # inherited by subclasses.
14
+ # When reading the options, a subclass will call the singleton method cun_opts.
15
+ # If the subclass doesn't redefine this method via a call to cuniculus_options,
16
+ # it will still use the definition from its parent class.
17
+ base.define_singleton_method("cun_opts=") do |opts|
18
+ singleton_class.class_eval do
19
+ define_method("cun_opts") { opts }
20
+ end
21
+ end
10
22
  end
11
23
 
12
24
  module ClassMethods
25
+ DEFAULT_OPTS = { "queue" => "cun_default" }.freeze
26
+ VALID_OPT_KEYS = %w[queue].freeze
27
+
28
+ # Read-only cuniculus option values
29
+ #
30
+ # @return opts [Hash] hash with current values
31
+ def cun_opts
32
+ DEFAULT_OPTS
33
+ end
34
+
35
+ # Worker-specific options for running cuniculus.
36
+ #
37
+ # Note that options set on a worker class are inherited by its subclasses.
38
+ #
39
+ # @param opts [Hash]
40
+ # @option opts [String] "queue" ("cun_default") Name of the underlying RabbitMQ queue.
41
+ #
42
+ # @example Change the queue name of a worker
43
+ # class MyWorker
44
+ # include Cuniculus::Worker
45
+ #
46
+ # cuniculus_options queue: "critical"
47
+ #
48
+ # def perform
49
+ # # run the task
50
+ # end
51
+ # end
52
+ def cuniculus_options(opts)
53
+ opts = validate_opts!(opts)
54
+ self.cun_opts = opts
55
+ end
56
+
57
+ def validate_opts!(opts)
58
+ raise WorkerOptionsError, "Argument passed to 'cuniculus_options' should be a Hash" unless opts.is_a?(Hash)
59
+ opts = opts.transform_keys(&:to_s)
60
+ invalid_keys = opts.keys - VALID_OPT_KEYS
61
+ raise WorkerOptionsError, "Invalid keys passed to 'cuniculus_options': #{invalid_keys.inspect}" unless invalid_keys.empty?
62
+ opts
63
+ end
64
+
13
65
  def perform_async(*args)
14
66
  publish({ "class" => self, "args" => args })
15
67
  end
16
68
 
17
69
  def publish(item)
18
- routing_key = "cun_default"
70
+ routing_key = cun_opts["queue"]
19
71
  payload = normalize_item(item)
20
72
  Cuniculus::RMQPool.with_exchange do |x|
21
73
  x.publish(payload, { routing_key: routing_key, persistent: true })
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: cuniculus
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.1
4
+ version: 0.1.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Marcelo Pereira
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2021-02-14 00:00:00.000000000 Z
11
+ date: 2021-03-07 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bunny
@@ -52,6 +52,20 @@ dependencies:
52
52
  - - ">="
53
53
  - !ruby/object:Gem::Version
54
54
  version: '0'
55
+ - !ruby/object:Gem::Dependency
56
+ name: rack
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - ">="
60
+ - !ruby/object:Gem::Version
61
+ version: '0'
62
+ type: :development
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - ">="
67
+ - !ruby/object:Gem::Version
68
+ version: '0'
55
69
  - !ruby/object:Gem::Dependency
56
70
  name: redcarpet
57
71
  requirement: !ruby/object:Gem::Requirement
@@ -94,6 +108,20 @@ dependencies:
94
108
  - - ">="
95
109
  - !ruby/object:Gem::Version
96
110
  version: '0'
111
+ - !ruby/object:Gem::Dependency
112
+ name: toxiproxy
113
+ requirement: !ruby/object:Gem::Requirement
114
+ requirements:
115
+ - - ">="
116
+ - !ruby/object:Gem::Version
117
+ version: '0'
118
+ type: :development
119
+ prerelease: false
120
+ version_requirements: !ruby/object:Gem::Requirement
121
+ requirements:
122
+ - - ">="
123
+ - !ruby/object:Gem::Version
124
+ version: '0'
97
125
  - !ruby/object:Gem::Dependency
98
126
  name: warning
99
127
  requirement: !ruby/object:Gem::Requirement
@@ -108,6 +136,20 @@ dependencies:
108
136
  - - ">="
109
137
  - !ruby/object:Gem::Version
110
138
  version: '0'
139
+ - !ruby/object:Gem::Dependency
140
+ name: webrick
141
+ requirement: !ruby/object:Gem::Requirement
142
+ requirements:
143
+ - - ">="
144
+ - !ruby/object:Gem::Version
145
+ version: '0'
146
+ type: :development
147
+ prerelease: false
148
+ version_requirements: !ruby/object:Gem::Requirement
149
+ requirements:
150
+ - - ">="
151
+ - !ruby/object:Gem::Version
152
+ version: '0'
111
153
  - !ruby/object:Gem::Dependency
112
154
  name: yard
113
155
  requirement: !ruby/object:Gem::Requirement