hutch 0.25.0.pre.rc1-java → 2.0.0.rc1-java
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +5 -5
- data/CHANGELOG.md +402 -5
- data/LICENSE +1 -0
- data/README.md +157 -47
- data/bin/ci/before_build.sh +20 -0
- data/bin/ci/before_build_docker.sh +15 -0
- data/bin/ci/install_on_debian.sh +46 -0
- data/lib/hutch/acknowledgements/nack_on_all_failures.rb +1 -1
- data/lib/hutch/adapters/bunny.rb +29 -1
- data/lib/hutch/adapters/march_hare.rb +24 -0
- data/lib/hutch/broker.rb +122 -36
- data/lib/hutch/cli.rb +23 -12
- data/lib/hutch/config.rb +45 -13
- data/lib/hutch/consumer.rb +63 -4
- data/lib/hutch/error_handlers/airbrake.rb +20 -2
- data/lib/hutch/error_handlers/base.rb +15 -0
- data/lib/hutch/error_handlers/bugsnag.rb +30 -0
- data/lib/hutch/error_handlers/honeybadger.rb +11 -3
- data/lib/hutch/error_handlers/logger.rb +7 -2
- data/lib/hutch/error_handlers/rollbar.rb +28 -0
- data/lib/hutch/error_handlers/sentry.rb +13 -11
- data/lib/hutch/error_handlers.rb +2 -1
- data/lib/hutch/publisher.rb +1 -1
- data/lib/hutch/serializers/json.rb +3 -3
- data/lib/hutch/tracers/datadog.rb +18 -0
- data/lib/hutch/tracers.rb +1 -1
- data/lib/hutch/version.rb +1 -2
- data/lib/hutch/waiter.rb +1 -1
- data/lib/hutch/worker.rb +74 -8
- data/lib/hutch.rb +8 -4
- data/lib/yard-settings/yard-settings.rb +2 -2
- data/spec/hutch/broker_spec.rb +222 -10
- data/spec/hutch/cli_spec.rb +25 -9
- data/spec/hutch/config_spec.rb +84 -12
- data/spec/hutch/consumer_spec.rb +93 -4
- data/spec/hutch/error_handlers/airbrake_spec.rb +23 -1
- data/spec/hutch/error_handlers/bugsnag_spec.rb +56 -0
- data/spec/hutch/error_handlers/honeybadger_spec.rb +22 -1
- data/spec/hutch/error_handlers/logger_spec.rb +12 -1
- data/spec/hutch/error_handlers/rollbar_spec.rb +45 -0
- data/spec/hutch/error_handlers/sentry_spec.rb +27 -2
- data/spec/hutch/message_spec.rb +1 -1
- data/spec/hutch/tracers/datadog_spec.rb +46 -0
- data/spec/hutch/waiter_spec.rb +2 -2
- data/spec/hutch/worker_spec.rb +139 -25
- data/spec/integration/channel_recovery_spec.rb +187 -0
- data/spec/integration/publish_consume_spec.rb +47 -0
- metadata +35 -61
- data/.gitignore +0 -10
- data/.rspec +0 -1
- data/.travis.yml +0 -19
- data/.yardopts +0 -5
- data/Gemfile +0 -33
- data/Guardfile +0 -14
- data/Rakefile +0 -21
- data/examples/consumer.rb +0 -13
- data/examples/producer.rb +0 -10
- data/hutch.gemspec +0 -29
- data/lib/hutch/error_handlers/opbeat.rb +0 -24
- data/lib/hutch/tracers/opbeat.rb +0 -37
- data/spec/hutch/error_handlers/opbeat_spec.rb +0 -22
- data/spec/spec_helper.rb +0 -42
- data/spec/tracers/opbeat_spec.rb +0 -44
data/README.md
CHANGED
|
@@ -1,9 +1,5 @@
|
|
|
1
|
-
](http://badge.fury.io/rb/hutch)
|
|
4
|
-
[](https://travis-ci.org/gocardless/hutch)
|
|
5
|
-
[](https://gemnasium.com/gocardless/hutch)
|
|
6
|
-
[](https://codeclimate.com/github/gocardless/hutch)
|
|
1
|
+
[](https://badge.fury.io/rb/hutch)
|
|
2
|
+
[](https://github.com/ruby-amqp/hutch/actions/workflows/test.yml)
|
|
7
3
|
|
|
8
4
|
Hutch is a Ruby library for enabling asynchronous inter-service communication
|
|
9
5
|
in a service-oriented architecture, using RabbitMQ.
|
|
@@ -25,6 +21,8 @@ gem install hutch
|
|
|
25
21
|
- [Message Processing Tracers](#message-processing-tracers)
|
|
26
22
|
- [Running Hutch](#running-hutch)
|
|
27
23
|
- [Loading Consumers](#loading-consumers)
|
|
24
|
+
- [Consumer Groups](#consumer-groups)
|
|
25
|
+
- [Loading Consumers Manually (One-by-One)](#loading-consumers-manually-one-by-one)
|
|
28
26
|
- [Stopping Hutch](#stopping-hutch)
|
|
29
27
|
- [Producers](#producers)
|
|
30
28
|
- [Producer Configuration](#producer-configuration)
|
|
@@ -40,8 +38,8 @@ gem install hutch
|
|
|
40
38
|
|
|
41
39
|
## Requirements
|
|
42
40
|
|
|
43
|
-
- Hutch requires Ruby 2
|
|
44
|
-
- Hutch requires RabbitMQ 3.
|
|
41
|
+
- Hutch requires Ruby 3.2+ or JRuby 10+
|
|
42
|
+
- Hutch requires RabbitMQ 3.13 or later
|
|
45
43
|
|
|
46
44
|
## Overview
|
|
47
45
|
|
|
@@ -54,13 +52,14 @@ With Hutch, consumers are stored in separate files and include the `Hutch::Consu
|
|
|
54
52
|
They are then loaded by a command line runner which connects to RabbitMQ, sets up queues and bindings,
|
|
55
53
|
and so on. Publishers connect to RabbitMQ via `Hutch.connect` and publish using `Hutch.publish`.
|
|
56
54
|
|
|
57
|
-
Hutch uses [Bunny](
|
|
55
|
+
Hutch uses [Bunny](https://github.com/ruby-amqp/bunny) or [March Hare](https://github.com/ruby-amqp/march_hare)
|
|
58
56
|
(on JRuby) under the hood.
|
|
59
57
|
|
|
60
58
|
### Project Maturity
|
|
61
59
|
|
|
62
|
-
Hutch is a
|
|
63
|
-
|
|
60
|
+
Hutch is a mature project that was originally extracted from production systems
|
|
61
|
+
at [GoCardless](https://gocardless.com) in 2013 and is now maintained by its contributors
|
|
62
|
+
and users.
|
|
64
63
|
|
|
65
64
|
## Consumers
|
|
66
65
|
|
|
@@ -80,7 +79,7 @@ message[:id] # => "02ABCXYZ"
|
|
|
80
79
|
To subscribe to a topic, pass a routing key to `consume` in the class
|
|
81
80
|
definition. To bind to multiple routing keys, simply pass extra routing keys
|
|
82
81
|
in as additional arguments. Refer to the [RabbitMQ docs on topic exchanges
|
|
83
|
-
](
|
|
82
|
+
](https://www.rabbitmq.com/tutorials/tutorial-five-ruby) for more information
|
|
84
83
|
about how to use routing keys. Here's an example consumer:
|
|
85
84
|
|
|
86
85
|
```ruby
|
|
@@ -109,6 +108,28 @@ class FailedPaymentConsumer
|
|
|
109
108
|
end
|
|
110
109
|
```
|
|
111
110
|
|
|
111
|
+
It is possible to set some custom options to consumer's queue explicitly.
|
|
112
|
+
This example sets the consumer's queue as a
|
|
113
|
+
[quorum queue](https://www.rabbitmq.com/docs/quorum-queues)
|
|
114
|
+
and to operate in the [lazy mode](https://www.rabbitmq.com/docs/lazy-queues).
|
|
115
|
+
The `initial_group_size`
|
|
116
|
+
[argument](https://www.rabbitmq.com/docs/quorum-queues#replication-factor) is
|
|
117
|
+
optional.
|
|
118
|
+
|
|
119
|
+
```ruby
|
|
120
|
+
class FailedPaymentConsumer
|
|
121
|
+
include Hutch::Consumer
|
|
122
|
+
consume 'gc.ps.payment.failed'
|
|
123
|
+
queue_name 'failed_payments'
|
|
124
|
+
lazy_queue
|
|
125
|
+
quorum_queue initial_group_size: 3
|
|
126
|
+
|
|
127
|
+
def process(message)
|
|
128
|
+
mark_payment_as_failed(message[:id])
|
|
129
|
+
end
|
|
130
|
+
end
|
|
131
|
+
```
|
|
132
|
+
|
|
112
133
|
You can also set custom arguments per consumer. This example declares a consumer with
|
|
113
134
|
a maximum length of 10 messages:
|
|
114
135
|
|
|
@@ -121,11 +142,11 @@ end
|
|
|
121
142
|
```
|
|
122
143
|
|
|
123
144
|
This sets the `x-max-length` header. For more details, see the [RabbitMQ
|
|
124
|
-
documentation on Queue Length Limit](https://www.rabbitmq.com/maxlength
|
|
125
|
-
about custom queue arguments, consult the [RabbitMQ documentation on AMQP Protocol Extensions](https://www.rabbitmq.com/extensions
|
|
145
|
+
documentation on Queue Length Limit](https://www.rabbitmq.com/docs/maxlength). To find out more
|
|
146
|
+
about custom queue arguments, consult the [RabbitMQ documentation on AMQP Protocol Extensions](https://www.rabbitmq.com/docs/extensions).
|
|
126
147
|
|
|
127
148
|
Consumers can write to Hutch's log by calling the logger method. The logger method returns
|
|
128
|
-
a [Logger object](
|
|
149
|
+
a [Logger object](https://docs.ruby-lang.org/en/3.0/Logger.html).
|
|
129
150
|
|
|
130
151
|
```ruby
|
|
131
152
|
class FailedPaymentConsumer
|
|
@@ -153,7 +174,7 @@ client_logger = Logger.new("/path/to/bunny.log")
|
|
|
153
174
|
Hutch::Config.set(:client_logger, client_logger)
|
|
154
175
|
```
|
|
155
176
|
|
|
156
|
-
See this [RabbitMQ tutorial on topic exchanges](
|
|
177
|
+
See this [RabbitMQ tutorial on topic exchanges](https://www.rabbitmq.com/tutorials/tutorial-five-ruby)
|
|
157
178
|
to learn more.
|
|
158
179
|
|
|
159
180
|
### Message Processing Tracers
|
|
@@ -166,7 +187,13 @@ This will enable NewRelic custom instrumentation:
|
|
|
166
187
|
Hutch::Config.set(:tracer, Hutch::Tracers::NewRelic)
|
|
167
188
|
```
|
|
168
189
|
|
|
169
|
-
|
|
190
|
+
And this will enable Datadog custom instrumentation:
|
|
191
|
+
|
|
192
|
+
```ruby
|
|
193
|
+
Hutch::Config.set(:tracer, Hutch::Tracers::Datadog)
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
Batteries included!
|
|
170
197
|
|
|
171
198
|
## Running Hutch
|
|
172
199
|
|
|
@@ -179,7 +206,7 @@ usage: hutch [options]
|
|
|
179
206
|
--mq-host HOST Set the RabbitMQ host
|
|
180
207
|
--mq-port PORT Set the RabbitMQ port
|
|
181
208
|
-t, --[no-]mq-tls Use TLS for the AMQP connection
|
|
182
|
-
--mq-tls-cert FILE Certificate
|
|
209
|
+
--mq-tls-cert FILE Certificate for TLS client verification
|
|
183
210
|
--mq-tls-key FILE Private key for TLS client verification
|
|
184
211
|
--mq-exchange EXCHANGE Set the RabbitMQ exchange
|
|
185
212
|
--mq-vhost VHOST Set the RabbitMQ vhost
|
|
@@ -215,17 +242,59 @@ in the config file, allowing for easy customization.
|
|
|
215
242
|
|
|
216
243
|
Using Hutch with a Rails app is simple. Either start Hutch in the working
|
|
217
244
|
directory of a Rails app, or pass the path to a Rails app in with the
|
|
218
|
-
`--require` option. Consumers defined in Rails apps should be placed
|
|
245
|
+
`--require` option. Consumers defined in Rails apps should be placed within
|
|
219
246
|
the `app/consumers/` directory, to allow them to be auto-loaded when Rails
|
|
220
247
|
boots.
|
|
221
248
|
|
|
249
|
+
If you're using the Zeitwerk autoloader (the default since Rails 6)
|
|
250
|
+
and the consumers are not loaded in the development environment, you will need to
|
|
251
|
+
trigger eager loading in an initializer with
|
|
252
|
+
|
|
253
|
+
```ruby
|
|
254
|
+
::Zeitwerk::Loader.eager_load_all
|
|
255
|
+
```
|
|
256
|
+
|
|
257
|
+
or with something more specific like
|
|
258
|
+
|
|
259
|
+
```ruby
|
|
260
|
+
autoloader = Rails.autoloaders.main
|
|
261
|
+
|
|
262
|
+
Dir.glob(File.join('app/consumers', '*_consumer.rb')).each do |consumer|
|
|
263
|
+
autoloader.preload(consumer)
|
|
264
|
+
end
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
### Consumer Groups
|
|
268
|
+
|
|
269
|
+
It is possible to load only a subset of consumers. This is done by defining a consumer
|
|
270
|
+
group under the `consumer_groups` configuration key:
|
|
271
|
+
|
|
272
|
+
``` yaml
|
|
273
|
+
consumer_groups:
|
|
274
|
+
payments:
|
|
275
|
+
- DepositConsumer
|
|
276
|
+
- CashoutConsumer
|
|
277
|
+
notification:
|
|
278
|
+
- EmailNotificationConsumer
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
To only load a group of consumers, use the `--only-group` option:
|
|
282
|
+
|
|
283
|
+
``` shell
|
|
284
|
+
hutch --only-group=payments --config=/path/to/hutch.yaml
|
|
285
|
+
```
|
|
286
|
+
|
|
287
|
+
### Loading Consumers Manually (One-by-One)
|
|
288
|
+
|
|
222
289
|
To require files that define consumers manually, simply pass each file as an
|
|
223
290
|
option to `--require`. Hutch will automatically detect whether you've provided
|
|
224
291
|
a Rails app or a standard file, and take the appropriate behaviour:
|
|
225
292
|
|
|
226
293
|
```bash
|
|
227
|
-
|
|
228
|
-
|
|
294
|
+
# loads a rails app
|
|
295
|
+
hutch --require path/to/rails-app
|
|
296
|
+
# loads a ruby file
|
|
297
|
+
hutch --require path/to/file.rb
|
|
229
298
|
```
|
|
230
299
|
|
|
231
300
|
### Stopping Hutch
|
|
@@ -240,8 +309,6 @@ kill -SIGTERM 456 # or kill -15 456
|
|
|
240
309
|
kill -SIGQUIT 789 # or kill -3 789
|
|
241
310
|
```
|
|
242
311
|
|
|
243
|
-

|
|
244
|
-
|
|
245
312
|
## Producers
|
|
246
313
|
|
|
247
314
|
Hutch includes a `publish` method for sending messages to Hutch consumers. When
|
|
@@ -265,7 +332,7 @@ Hutch::Config.set(:mq_exchange, 'name')
|
|
|
265
332
|
### Publisher Confirms
|
|
266
333
|
|
|
267
334
|
For maximum message reliability when producing messages, you can force Hutch to use
|
|
268
|
-
[Publisher Confirms](https://www.rabbitmq.com/confirms
|
|
335
|
+
[Publisher Confirms](https://www.rabbitmq.com/docs/confirms) and wait for a confirmation
|
|
269
336
|
after every message published. This is the safest possible option for publishers
|
|
270
337
|
but also results in a **significant throughput drop**.
|
|
271
338
|
|
|
@@ -285,25 +352,27 @@ send messages to Hutch.
|
|
|
285
352
|
- Hutch works with topic exchanges, check the producer is also using topic
|
|
286
353
|
exchanges.
|
|
287
354
|
- Use message routing keys that match those used in your Hutch consumers.
|
|
288
|
-
- Be sure your exchanges are marked as durable
|
|
289
|
-
|
|
290
|
-
- Publish messages as persistent
|
|
291
|
-
- Using publisher confirms is highly recommended
|
|
355
|
+
- Be sure your exchanges are marked as durable, by passing `durable: true`
|
|
356
|
+
to the exchange creation method
|
|
357
|
+
- Publish messages as persistent
|
|
358
|
+
- Using publisher confirms is highly recommended
|
|
292
359
|
|
|
293
360
|
Here's an example of a well-behaved publisher, minus publisher confirms:
|
|
294
361
|
|
|
295
362
|
```ruby
|
|
296
|
-
|
|
297
|
-
channel = AMQP::Channel.new(connection)
|
|
298
|
-
exchange = channel.topic(config[:exchange], durable: true)
|
|
363
|
+
require 'bunny'
|
|
299
364
|
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
end
|
|
303
|
-
```
|
|
365
|
+
conn = Bunny.new(host: config[:host])
|
|
366
|
+
conn.start
|
|
304
367
|
|
|
305
|
-
|
|
306
|
-
|
|
368
|
+
ch = conn.create_channel
|
|
369
|
+
exchange = ch.topic(config[:exchange], durable: true)
|
|
370
|
+
|
|
371
|
+
message = JSON.dump({ subject: 'Test', id: 'abc' })
|
|
372
|
+
exchange.publish(message, routing_key: 'test', persistent: true)
|
|
373
|
+
|
|
374
|
+
conn.close
|
|
375
|
+
```
|
|
307
376
|
|
|
308
377
|
## Configuration
|
|
309
378
|
|
|
@@ -316,7 +385,7 @@ Known configuration parameters are:
|
|
|
316
385
|
* `mq_host`: RabbitMQ hostname (default: `localhost`)
|
|
317
386
|
* `mq_port`: RabbitMQ port (default: `5672`)
|
|
318
387
|
* `mq_vhost`: vhost to use (default: `/`)
|
|
319
|
-
* `mq_username`: username to use (default: `guest`, only
|
|
388
|
+
* `mq_username`: username to use (default: `guest`, can only connect from localhost)
|
|
320
389
|
* `mq_password`: password to use (default: `guest`)
|
|
321
390
|
* `mq_tls`: should TLS be used? (default: `false`)
|
|
322
391
|
* `mq_tls_cert`: path to client TLS certificate (public key)
|
|
@@ -337,11 +406,15 @@ Known configuration parameters are:
|
|
|
337
406
|
* `error_acknowledgements`: a chain of responsibility of objects that acknowledge/reject/requeue messages when an
|
|
338
407
|
exception happens, see classes in `Hutch::Acknowledgements`.
|
|
339
408
|
* `mq_exchange`: exchange to use for publishing (default: `hutch`)
|
|
340
|
-
* `
|
|
409
|
+
* `mq_client_properties`: Bunny's [client properties](https://www.rabbitmq.com/docs/connections#capabilities) (default: `{}`)
|
|
410
|
+
* `heartbeat`: [RabbitMQ heartbeat timeout](https://www.rabbitmq.com/docs/heartbeats) (default: `30`)
|
|
341
411
|
* `connection_timeout`: Bunny's socket open timeout (default: `11`)
|
|
342
412
|
* `read_timeout`: Bunny's socket read timeout (default: `11`)
|
|
343
413
|
* `write_timeout`: Bunny's socket write timeout (default: `11`)
|
|
414
|
+
* `automatically_recover`: Bunny's enable/disable network recovery (default: `true`)
|
|
415
|
+
* `network_recovery_interval`: Bunny's reconnect interval (default: `1`)
|
|
344
416
|
* `tracer`: tracer to use to track message processing
|
|
417
|
+
* `namespace`: A namespace string to help group your queues (default: `nil`)
|
|
345
418
|
|
|
346
419
|
### Environment variables
|
|
347
420
|
|
|
@@ -400,6 +473,13 @@ Generate with
|
|
|
400
473
|
<td><tt>HUTCH_MQ_EXCHANGE</tt></td>
|
|
401
474
|
<td><p>RabbitMQ Exchange to use for publishing</p></td>
|
|
402
475
|
</tr>
|
|
476
|
+
<tr>
|
|
477
|
+
<td><tt>mq_exchange_type</tt></td>
|
|
478
|
+
<td>topic</td>
|
|
479
|
+
<td>String</td>
|
|
480
|
+
<td><tt>HUTCH_MQ_EXCHANGE_TYPE</tt></td>
|
|
481
|
+
<td><p>RabbitMQ Exchange type to use for publishing</p></td>
|
|
482
|
+
</tr>
|
|
403
483
|
<tr>
|
|
404
484
|
<td><tt>mq_vhost</tt></td>
|
|
405
485
|
<td>/</td>
|
|
@@ -454,7 +534,7 @@ Generate with
|
|
|
454
534
|
<td>30</td>
|
|
455
535
|
<td>Number</td>
|
|
456
536
|
<td><tt>HUTCH_HEARTBEAT</tt></td>
|
|
457
|
-
<td><p><a href="
|
|
537
|
+
<td><p><a href="https://www.rabbitmq.com/docs/heartbeats">RabbitMQ heartbeat timeout</a></p></td>
|
|
458
538
|
</tr>
|
|
459
539
|
<tr>
|
|
460
540
|
<td><tt>channel_prefetch</tt></td>
|
|
@@ -463,6 +543,13 @@ Generate with
|
|
|
463
543
|
<td><tt>HUTCH_CHANNEL_PREFETCH</tt></td>
|
|
464
544
|
<td><p>The <tt>basic.qos</tt> prefetch value to use.</p></td>
|
|
465
545
|
</tr>
|
|
546
|
+
<tr>
|
|
547
|
+
<td><tt>connection_name</tt></td>
|
|
548
|
+
<td>nil</td>
|
|
549
|
+
<td>String</td>
|
|
550
|
+
<td><tt>HUTCH_CONNECTION_NAME</tt></td>
|
|
551
|
+
<td><p><a href="https://www.rabbitmq.com/docs/connections#client-provided-names">Client-Provided Connection Name</a></p></td>
|
|
552
|
+
</tr>
|
|
466
553
|
<tr>
|
|
467
554
|
<td><tt>connection_timeout</tt></td>
|
|
468
555
|
<td>11</td>
|
|
@@ -484,12 +571,26 @@ Generate with
|
|
|
484
571
|
<td><tt>HUTCH_WRITE_TIMEOUT</tt></td>
|
|
485
572
|
<td><p>Bunny's socket write timeout</p></td>
|
|
486
573
|
</tr>
|
|
574
|
+
<tr>
|
|
575
|
+
<td><tt>automatically_recover</tt></td>
|
|
576
|
+
<td>true</td>
|
|
577
|
+
<td>Boolean</td>
|
|
578
|
+
<td><tt>HUTCH_AUTOMATICALLY_RECOVER</tt></td>
|
|
579
|
+
<td><p>Bunny's enable/disable network recovery</p></td>
|
|
580
|
+
</tr>
|
|
581
|
+
<tr>
|
|
582
|
+
<td><tt>network_recovery_interval</tt></td>
|
|
583
|
+
<td>1</td>
|
|
584
|
+
<td>Number</td>
|
|
585
|
+
<td><tt>HUTCH_NETWORK_RECOVERY_INTERVAL</tt></td>
|
|
586
|
+
<td><p>Bunny's reconnect interval</p></td>
|
|
587
|
+
</tr>
|
|
487
588
|
<tr>
|
|
488
589
|
<td><tt>graceful_exit_timeout</tt></td>
|
|
489
590
|
<td>11</td>
|
|
490
591
|
<td>Number</td>
|
|
491
592
|
<td><tt>HUTCH_GRACEFUL_EXIT_TIMEOUT</tt></td>
|
|
492
|
-
<td><p>
|
|
593
|
+
<td><p>Timeout (in seconds) for consumer threads to finish before being killed during graceful shutdown</p></td>
|
|
493
594
|
</tr>
|
|
494
595
|
<tr>
|
|
495
596
|
<td><tt>consumer_pool_size</tt></td>
|
|
@@ -545,7 +646,7 @@ Generate with
|
|
|
545
646
|
<td>false</td>
|
|
546
647
|
<td>Boolean</td>
|
|
547
648
|
<td><tt>HUTCH_FORCE_PUBLISHER_CONFIRMS</tt></td>
|
|
548
|
-
<td><p>Enables publisher confirms, forces Hutch::Broker#wait_for_confirms for</p></td>
|
|
649
|
+
<td><p>Enables publisher confirms, forces Hutch::Broker#wait_for_confirms for every publish</p></td>
|
|
549
650
|
</tr>
|
|
550
651
|
<tr>
|
|
551
652
|
<td><tt>enable_http_api_use</tt></td>
|
|
@@ -568,10 +669,19 @@ Generate with
|
|
|
568
669
|
<td><tt>HUTCH_CONSUMER_TAG_PREFIX</tt></td>
|
|
569
670
|
<td><p>Prefix displayed on the consumers tags.</p></td>
|
|
570
671
|
</tr>
|
|
672
|
+
<tr>
|
|
673
|
+
<td><tt>namespace</tt></td>
|
|
674
|
+
<td>nil</td>
|
|
675
|
+
<td>String</td>
|
|
676
|
+
<td><tt>HUTCH_NAMESPACE</tt></td>
|
|
677
|
+
<td><p>A namespace to help group your queues</p></td>
|
|
678
|
+
</tr>
|
|
679
|
+
<tr>
|
|
680
|
+
<td><tt>group</tt></td>
|
|
681
|
+
<td>''</td>
|
|
682
|
+
<td>String</td>
|
|
683
|
+
<td><tt>HUTCH_GROUP</tt></td>
|
|
684
|
+
<td><p>Consumer group to load (see Consumer Groups)</p></td>
|
|
685
|
+
</tr>
|
|
571
686
|
</tbody>
|
|
572
687
|
</table>
|
|
573
|
-
|
|
574
|
-
|
|
575
|
-
---
|
|
576
|
-
|
|
577
|
-
GoCardless ♥ open source. If you do too, come [join us](https://gocardless.com/jobs/backend_developer).
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
#!/usr/bin/env sh
|
|
2
|
+
|
|
3
|
+
CTL=${BUNNY_RABBITMQCTL:-"sudo rabbitmqctl"}
|
|
4
|
+
PLUGINS=${BUNNY_RABBITMQ_PLUGINS:-"sudo rabbitmq-plugins"}
|
|
5
|
+
|
|
6
|
+
echo "Will use rabbitmqctl at ${CTL}"
|
|
7
|
+
echo "Will use rabbitmq-plugins at ${PLUGINS}"
|
|
8
|
+
|
|
9
|
+
$PLUGINS enable rabbitmq_management
|
|
10
|
+
|
|
11
|
+
sleep 3
|
|
12
|
+
|
|
13
|
+
# guest:guest has full access to /
|
|
14
|
+
$CTL add_vhost /
|
|
15
|
+
$CTL add_user guest guest
|
|
16
|
+
$CTL set_permissions -p / guest ".*" ".*" ".*"
|
|
17
|
+
|
|
18
|
+
# Reduce retention policy for faster publishing of stats
|
|
19
|
+
$CTL eval 'supervisor2:terminate_child(rabbit_mgmt_sup_sup, rabbit_mgmt_sup), application:set_env(rabbitmq_management, sample_retention_policies, [{global, [{605, 1}]}, {basic, [{605, 1}]}, {detailed, [{10, 1}]}]), rabbit_mgmt_sup_sup:start_child().' || true
|
|
20
|
+
$CTL eval 'supervisor2:terminate_child(rabbit_mgmt_agent_sup_sup, rabbit_mgmt_agent_sup), application:set_env(rabbitmq_management_agent, sample_retention_policies, [{global, [{605, 1}]}, {basic, [{605, 1}]}, {detailed, [{10, 1}]}]), rabbit_mgmt_agent_sup_sup:start_child().' || true
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
#!/bin/sh
|
|
2
|
+
|
|
3
|
+
CTL=${MARCH_HARE_RABBITMQCTL:="docker exec rabbitmq rabbitmqctl"}
|
|
4
|
+
|
|
5
|
+
# guest:guest has full access to /
|
|
6
|
+
|
|
7
|
+
$CTL add_vhost /
|
|
8
|
+
# $CTL add_user guest guest # already exists
|
|
9
|
+
$CTL set_permissions -p / guest ".*" ".*" ".*"
|
|
10
|
+
|
|
11
|
+
# Reduce retention policy for faster publishing of stats
|
|
12
|
+
$CTL eval 'supervisor2:terminate_child(rabbit_mgmt_sup_sup, rabbit_mgmt_sup), application:set_env(rabbitmq_management, sample_retention_policies, [{global, [{605, 1}]}, {basic, [{605, 1}]}, {detailed, [{10, 1}]}]), rabbit_mgmt_sup_sup:start_child().' || true
|
|
13
|
+
$CTL eval 'supervisor2:terminate_child(rabbit_mgmt_agent_sup_sup, rabbit_mgmt_agent_sup), application:set_env(rabbitmq_management_agent, sample_retention_policies, [{global, [{605, 1}]}, {basic, [{605, 1}]}, {detailed, [{10, 1}]}]), rabbit_mgmt_agent_sup_sup:start_child().' || true
|
|
14
|
+
|
|
15
|
+
sleep 3
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
#!/bin/sh
|
|
2
|
+
|
|
3
|
+
sudo apt-get install curl gnupg debian-keyring debian-archive-keyring apt-transport-https -y
|
|
4
|
+
|
|
5
|
+
## Team RabbitMQ's main signing key
|
|
6
|
+
sudo apt-key adv --keyserver "hkps://keys.openpgp.org" --recv-keys "0x0A9AF2115F4687BD29803A206B73A36E6026DFCA"
|
|
7
|
+
## Launchpad PPA that provides modern Erlang releases
|
|
8
|
+
sudo apt-key adv --keyserver "keyserver.ubuntu.com" --recv-keys "F77F1EDA57EBB1CC"
|
|
9
|
+
## PackageCloud RabbitMQ repository
|
|
10
|
+
sudo apt-key adv --keyserver "keyserver.ubuntu.com" --recv-keys "F6609E60DC62814E"
|
|
11
|
+
|
|
12
|
+
## Add apt repositories maintained by Team RabbitMQ
|
|
13
|
+
sudo tee /etc/apt/sources.list.d/rabbitmq.list <<EOF
|
|
14
|
+
## Provides modern Erlang/OTP releases
|
|
15
|
+
##
|
|
16
|
+
## "bionic" as distribution name should work for any reasonably recent Ubuntu or Debian release.
|
|
17
|
+
## See the release to distribution mapping table in RabbitMQ doc guides to learn more.
|
|
18
|
+
deb http://ppa.launchpad.net/rabbitmq/rabbitmq-erlang/ubuntu bionic main
|
|
19
|
+
deb-src http://ppa.launchpad.net/rabbitmq/rabbitmq-erlang/ubuntu bionic main
|
|
20
|
+
|
|
21
|
+
## Provides RabbitMQ
|
|
22
|
+
##
|
|
23
|
+
## "bionic" as distribution name should work for any reasonably recent Ubuntu or Debian release.
|
|
24
|
+
## See the release to distribution mapping table in RabbitMQ doc guides to learn more.
|
|
25
|
+
deb https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu/ bionic main
|
|
26
|
+
deb-src https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu/ bionic main
|
|
27
|
+
EOF
|
|
28
|
+
|
|
29
|
+
## Update package indices
|
|
30
|
+
sudo apt-get update -y
|
|
31
|
+
|
|
32
|
+
## Install Erlang packages
|
|
33
|
+
sudo apt-get install -y erlang-base \
|
|
34
|
+
erlang-asn1 erlang-crypto erlang-eldap erlang-ftp erlang-inets \
|
|
35
|
+
erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key \
|
|
36
|
+
erlang-runtime-tools erlang-snmp erlang-ssl \
|
|
37
|
+
erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl
|
|
38
|
+
|
|
39
|
+
## Install rabbitmq-server and its dependencies
|
|
40
|
+
sudo apt-get install rabbitmq-server -y --fix-missing
|
|
41
|
+
|
|
42
|
+
sudo service rabbitmq-server start
|
|
43
|
+
|
|
44
|
+
sudo rabbitmqctl await_startup --timeout 120
|
|
45
|
+
|
|
46
|
+
until sudo lsof -i:5672; do echo "Waiting for RabbitMQ to start..."; sleep 1; done
|
data/lib/hutch/adapters/bunny.rb
CHANGED
|
@@ -1,17 +1,21 @@
|
|
|
1
1
|
require 'bunny'
|
|
2
2
|
require 'forwardable'
|
|
3
3
|
|
|
4
|
+
require 'hutch/logging'
|
|
5
|
+
|
|
4
6
|
module Hutch
|
|
5
7
|
module Adapters
|
|
6
8
|
class BunnyAdapter
|
|
7
9
|
extend Forwardable
|
|
10
|
+
include Logging
|
|
8
11
|
|
|
9
12
|
DEFAULT_VHOST = Bunny::Session::DEFAULT_VHOST
|
|
10
13
|
|
|
11
14
|
ConnectionRefused = Bunny::TCPConnectionFailed
|
|
15
|
+
ChannelAlreadyClosed = Bunny::ChannelAlreadyClosed
|
|
12
16
|
PreconditionFailed = Bunny::PreconditionFailed
|
|
13
17
|
|
|
14
|
-
def_delegators :@connection, :start, :disconnect, :close, :create_channel, :open
|
|
18
|
+
def_delegators :@connection, :start, :disconnect, :close, :create_channel, :open?, :recover_channel_topology
|
|
15
19
|
|
|
16
20
|
def initialize(opts={})
|
|
17
21
|
@connection = Bunny.new(opts)
|
|
@@ -25,9 +29,33 @@ module Hutch
|
|
|
25
29
|
ch.prefetch(prefetch) if prefetch
|
|
26
30
|
end
|
|
27
31
|
|
|
32
|
+
def queue_exists?(name)
|
|
33
|
+
@connection.queue_exists?(name)
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
def install_channel_recovery(ch)
|
|
37
|
+
# A consumer timeout invalidates the delivery tag, so a handler that
|
|
38
|
+
# was still running acknowledges an unknown tag and closes the channel.
|
|
39
|
+
ch.on_error do |channel, close|
|
|
40
|
+
next unless close.delivery_ack_timeout? || close.unknown_delivery_tag?
|
|
41
|
+
|
|
42
|
+
begin
|
|
43
|
+
channel.reopen
|
|
44
|
+
recover_channel_topology(channel)
|
|
45
|
+
logger.warn "recovered consumer channel closed with '#{close.reply_text}'"
|
|
46
|
+
rescue => ex
|
|
47
|
+
logger.error "channel recovery failed: #{ex.class}: #{ex.message}"
|
|
48
|
+
end
|
|
49
|
+
end
|
|
50
|
+
end
|
|
51
|
+
|
|
28
52
|
def current_timestamp
|
|
29
53
|
Time.now.to_i
|
|
30
54
|
end
|
|
55
|
+
|
|
56
|
+
def self.new_exchange(ch, exchange_type, exchange_name, exchange_options)
|
|
57
|
+
Bunny::Exchange.new(ch, exchange_type, exchange_name, exchange_options)
|
|
58
|
+
end
|
|
31
59
|
end
|
|
32
60
|
end
|
|
33
61
|
end
|
|
@@ -9,6 +9,7 @@ module Hutch
|
|
|
9
9
|
DEFAULT_VHOST = "/"
|
|
10
10
|
|
|
11
11
|
ConnectionRefused = MarchHare::ConnectionRefused
|
|
12
|
+
ChannelAlreadyClosed = MarchHare::ChannelAlreadyClosed
|
|
12
13
|
PreconditionFailed = MarchHare::PreconditionFailed
|
|
13
14
|
|
|
14
15
|
def_delegators :@connection, :start, :disconnect, :close, :open?
|
|
@@ -25,6 +26,25 @@ module Hutch
|
|
|
25
26
|
ch.prefetch = prefetch if prefetch
|
|
26
27
|
end
|
|
27
28
|
|
|
29
|
+
# `MarchHare::Session` has no `queue_exists?` counterpart, and a failed
|
|
30
|
+
# passive declare closes the channel, hence the throwaway one.
|
|
31
|
+
def queue_exists?(name)
|
|
32
|
+
ch = @connection.create_channel
|
|
33
|
+
begin
|
|
34
|
+
ch.queue(name, passive: true)
|
|
35
|
+
true
|
|
36
|
+
rescue MarchHare::NotFound
|
|
37
|
+
false
|
|
38
|
+
ensure
|
|
39
|
+
ch.close rescue nil
|
|
40
|
+
end
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
# MarchHare::Channel has no on_error callback, and neither Channel#reopen
|
|
44
|
+
# nor Session#recover_channel_topology has a `march_hare` counterpart.
|
|
45
|
+
def install_channel_recovery(ch)
|
|
46
|
+
end
|
|
47
|
+
|
|
28
48
|
def create_channel(n = nil, consumer_pool_size = 1, consumer_pool_abort_on_exception = false)
|
|
29
49
|
@connection.create_channel(n)
|
|
30
50
|
end
|
|
@@ -32,6 +52,10 @@ module Hutch
|
|
|
32
52
|
def current_timestamp
|
|
33
53
|
Time.now
|
|
34
54
|
end
|
|
55
|
+
|
|
56
|
+
def self.new_exchange(ch, exchange_type, exchange_name, exchange_options)
|
|
57
|
+
MarchHare::Exchange.new(ch, exchange_name, exchange_options.merge(type: exchange_type)).tap(&:declare!)
|
|
58
|
+
end
|
|
35
59
|
end
|
|
36
60
|
end
|
|
37
61
|
end
|