govuk_message_queue_consumer 4.2.0 → 5.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +7 -0
- data/README.md +40 -71
- data/lib/govuk_message_queue_consumer/consumer.rb +12 -26
- data/lib/govuk_message_queue_consumer/message_consumer.rb +2 -16
- data/lib/govuk_message_queue_consumer/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 9b19c5a4d8e4c1f6369d9630e64f7b31bd416fbd523512364f3bae8c3c538b5c
|
4
|
+
data.tar.gz: 92c67e9448299357417ac9db221e6c233f1ae38686b5cca704b90dc0d73fccdf
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: c28069a62e0823f858427c5c01eb6de8a8c58fe93afbbbc0c97295a4993993a1c535aeac8ac7f9e470a3d1ee83512b54932479f4ea75cac52bab8bbb53fe6b62
|
7
|
+
data.tar.gz: c76d78e878780c608af295ff60f7ec900da3fa6a262e2a8a26ce7c3e27ff57808bd3a7df23b679b9d0c032a5d58de730d6d91c1272f6a3f1dd269de35612c001
|
data/CHANGELOG.md
CHANGED
@@ -1,3 +1,10 @@
|
|
1
|
+
# 5.0.0
|
2
|
+
|
3
|
+
- BREAKING: remove disused support for statsd. No clients in alphagov use the statsd functionality any more, so this is only theoretically breaking.
|
4
|
+
- Allow clients to specify a Bunny worker thread pool size of greater than 1. The default behaviour remains unchanged.
|
5
|
+
- Allow clients to specify "client prefetch" to allow more than one unacked message on a channel at a time. The default behaviour remains unchanged.
|
6
|
+
- Clean up some disused remnants of the batch consumer feature. The feature was removed in 4.0.0.
|
7
|
+
|
1
8
|
# 4.2.0
|
2
9
|
|
3
10
|
* Drop support for Ruby 3.0. The minimum required Ruby version is now 3.1.4.
|
data/README.md
CHANGED
@@ -1,40 +1,40 @@
|
|
1
1
|
# GOV.UK Message Queue Consumer
|
2
|
-
[](https://badge.fury.io/rb/govuk_message_queue_consumer)
|
3
|
-
|
4
|
-
Standardises the way GOV.UK consumes messages from [RabbitMQ](https://www.rabbitmq.com/).
|
5
|
-
RabbitMQ is a messaging framework that allows applications to broadcast messages
|
6
|
-
that can be picked up by other applications.
|
7
2
|
|
8
|
-
|
9
|
-
the content-items it receives, so that applications such as
|
10
|
-
[email-alert-service](https://github.com/alphagov/email-alert-service) can be
|
11
|
-
notified of changes in content.
|
3
|
+
[](https://badge.fury.io/rb/govuk_message_queue_consumer)
|
12
4
|
|
13
|
-
|
5
|
+
govuk_message_queue_consumer is a wrapper around the
|
6
|
+
[Bunny](https://github.com/ruby-amqp/bunny) gem for communicating with
|
7
|
+
[RabbitMQ](https://www.rabbitmq.com/). The user of govuk_message_queue_consumer
|
8
|
+
supplies some configuration and a class that processes messages.
|
14
9
|
|
15
|
-
|
10
|
+
RabbitMQ is a multi-producer, multi-consumer message queue that allows
|
11
|
+
applications to subscribe to notifications published by other applications.
|
16
12
|
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
13
|
+
GOV.UK [publishing-api](https://github.com/alphagov/publishing-api) publishes
|
14
|
+
a message to RabbitMQ when a ContentItem is added or changed. Other
|
15
|
+
applications (consumers) subscribe to these messages so that they can perform
|
16
|
+
actions such as emailing users or updating a search index.
|
21
17
|
|
22
|
-
|
18
|
+
Several GOV.UK applications use govuk_message_queue_consumer:
|
23
19
|
|
24
|
-
|
20
|
+
- [Content Data API](https://github.com/alphagov/content-data-api)
|
21
|
+
- [Email Alert Service](https://github.com/alphagov/email-alert-service/)
|
22
|
+
- [Search API](https://github.com/alphagov/search-api)
|
23
|
+
- [Search API v2](https://github.com/alphagov/search-api-v2)
|
25
24
|
|
26
25
|
## Technical documentation
|
27
26
|
|
28
|
-
|
29
|
-
|
30
|
-
supplying the configuration and a class that processes messages.
|
27
|
+
You can browse the [API documentation on
|
28
|
+
rubydoc.info](http://www.rubydoc.info/gems/govuk_message_queue_consumer/GovukMessageQueueConsumer/Consumer#initialize-instance_method).
|
31
29
|
|
32
|
-
|
33
|
-
pull request with the version number incremented.
|
30
|
+
## Release a new version
|
34
31
|
|
35
|
-
|
32
|
+
To release a new version, increment the [version
|
33
|
+
number](/lib/govuk_message_queue_consumer/version.rb) and raise a pull request.
|
36
34
|
|
37
|
-
|
35
|
+
The [CI GitHub Actions
|
36
|
+
workflow](https://github.com/alphagov/govuk_message_queue_consumer/actions/workflows/ci.yml)
|
37
|
+
should automatically build and push the new release when you merge the PR.
|
38
38
|
|
39
39
|
## Usage
|
40
40
|
|
@@ -55,13 +55,15 @@ namespace :message_queue do
|
|
55
55
|
end
|
56
56
|
```
|
57
57
|
|
58
|
-
|
58
|
+
See the [API documentation](http://www.rubydoc.info/gems/govuk_message_queue_consumer/GovukMessageQueueConsumer/Consumer#initialize-instance_method) for the full list of parameters.
|
59
59
|
|
60
|
-
|
60
|
+
`GovukMessageQueueConsumer::Consumer` expects the [`RABBITMQ_URL` environment
|
61
61
|
variable](https://github.com/ruby-amqp/bunny/blob/066496d/docs/guides/connecting.md#paas-environments)
|
62
62
|
to be set to an AMQP connection string, for example:
|
63
63
|
|
64
|
-
|
64
|
+
```sh
|
65
|
+
RABBITMQ_URL=amqp://mrbean:hunter2@rabbitmq.example.com:5672
|
66
|
+
```
|
65
67
|
|
66
68
|
The GOV.UK-specific environment variables `RABBITMQ_HOSTS`, `RABBITMQ_VHOST`,
|
67
69
|
`RABBITMQ_USER` and `RABBITMQ_PASSWORD` are deprecated. Support for these will
|
@@ -78,21 +80,13 @@ class MyProcessor
|
|
78
80
|
end
|
79
81
|
```
|
80
82
|
|
81
|
-
|
83
|
+
You can start the worker by running the `message_queue:consumer` Rake task.
|
82
84
|
|
83
|
-
```
|
84
|
-
|
85
|
-
worker: bundle exec rake message_queue:consumer
|
86
|
-
```
|
87
|
-
|
88
|
-
Because you need the environment variables when running the consumer, you should use
|
89
|
-
`govuk_setenv` to run your app in development:
|
90
|
-
|
91
|
-
```
|
92
|
-
$ govuk_setenv app-name bundle exec rake message_queue:consumer
|
85
|
+
```sh
|
86
|
+
bundle exec rake message_queue:consumer
|
93
87
|
```
|
94
88
|
|
95
|
-
###
|
89
|
+
### Process a message
|
96
90
|
|
97
91
|
Once you receive a message, you *must* tell RabbitMQ once you've processed it. This
|
98
92
|
is called _acking_. You can also _discard_ the message, or _retry_ it.
|
@@ -116,34 +110,12 @@ class MyProcessor
|
|
116
110
|
end
|
117
111
|
```
|
118
112
|
|
119
|
-
###
|
113
|
+
### Test your processor
|
120
114
|
|
121
|
-
|
122
|
-
|
123
|
-
- `your_queue_name.started` - message picked up from the your_queue_name
|
124
|
-
- `your_queue_name.retried` - message has been retried
|
125
|
-
- `your_queue_name.acked` - message has been processed and acked
|
126
|
-
- `your_queue_name.discarded` - message has been discarded
|
127
|
-
- `your_queue_name.uncaught_exception` - an uncaught exception occured during processing
|
128
|
-
|
129
|
-
Remember to use a namespace for the `Statsd` client:
|
115
|
+
govuk_message_queue_consumer provides a test helper for your processor.
|
130
116
|
|
131
117
|
```ruby
|
132
|
-
|
133
|
-
statsd_client.namespace = "govuk.app.my_app_name"
|
134
|
-
|
135
|
-
GovukMessageQueueConsumer::Consumer.new(
|
136
|
-
statsd_client: statsd_client
|
137
|
-
# ... other setup code omitted
|
138
|
-
).run
|
139
|
-
```
|
140
|
-
|
141
|
-
### Testing your processor
|
142
|
-
|
143
|
-
This gem provides a test helper for your processor.
|
144
|
-
|
145
|
-
```ruby
|
146
|
-
# eg. spec/queue_consumers/my_processor_spec.rb
|
118
|
+
# e.g. spec/queue_consumers/my_processor_spec.rb
|
147
119
|
require 'test_helper'
|
148
120
|
require 'govuk_message_queue_consumer/test_helpers'
|
149
121
|
|
@@ -172,10 +144,9 @@ it "acks incoming messages" do
|
|
172
144
|
end
|
173
145
|
```
|
174
146
|
|
175
|
-
For more test cases [see the spec for the mock itself](/spec/mock_message_spec.rb).
|
176
|
-
|
147
|
+
For more test cases [see the spec for the mock itself](/spec/govuk_message_queue_consumer/test_helpers/mock_message_spec.rb).
|
177
148
|
|
178
|
-
###
|
149
|
+
### Run the test suite
|
179
150
|
|
180
151
|
```bash
|
181
152
|
bundle exec rake spec
|
@@ -184,10 +155,8 @@ bundle exec rake spec
|
|
184
155
|
## Further reading
|
185
156
|
|
186
157
|
- [Bunny](https://github.com/ruby-amqp/bunny) is the RabbitMQ client we use.
|
187
|
-
- [The Bunny Guides](
|
188
|
-
AMQP concepts
|
189
|
-
- The [Developer Docs](https://docs.publishing.service.gov.uk/manual/rabbitmq.html)
|
190
|
-
documents the usage of "heartbeat" messages, which this gem also supports.
|
158
|
+
- [The Bunny Guides](https://github.com/ruby-amqp/bunny/tree/main/docs/guides) explain
|
159
|
+
AMQP concepts.
|
191
160
|
|
192
161
|
## Licence
|
193
162
|
|
@@ -1,15 +1,5 @@
|
|
1
1
|
module GovukMessageQueueConsumer
|
2
2
|
class Consumer
|
3
|
-
HANDLE_BATCHES = false
|
4
|
-
# Only fetch one message at a time on the channel.
|
5
|
-
#
|
6
|
-
# By default, queues will grab messages eagerly, which reduces latency.
|
7
|
-
# However, that also means that if multiple workers are running one worker
|
8
|
-
# can starve another of work. We're not expecting a high throughput on this
|
9
|
-
# queue, and a small bit of latency isn't a problem, so we fetch one at a
|
10
|
-
# time to share the work evenly.
|
11
|
-
NUMBER_OF_MESSAGES_TO_PREFETCH = 1
|
12
|
-
|
13
3
|
def self.default_connection_from_env
|
14
4
|
# https://github.com/ruby-amqp/bunny/blob/066496d/docs/guides/connecting.md#paas-environments
|
15
5
|
if !ENV["RABBITMQ_URL"].to_s.empty?
|
@@ -22,17 +12,23 @@ module GovukMessageQueueConsumer
|
|
22
12
|
# Create a new consumer
|
23
13
|
#
|
24
14
|
# @param queue_name [String] Your queue name. This is specific to your application,
|
25
|
-
# and should already exist and have a binding
|
15
|
+
# and should already exist and have a binding
|
16
|
+
# configured via Terraform.
|
26
17
|
# @param processor [Object] An object that responds to `process`
|
27
18
|
# @param rabbitmq_connection [Object] A Bunny connection object derived from `Bunny.new`
|
28
|
-
# @param statsd_client [Statsd] An instance of the Statsd class
|
29
19
|
# @param logger [Object] A Logger object for emitting errors (to stderr by default)
|
30
|
-
|
20
|
+
# @param worker_threads [Number] Size of the worker thread pool. Defaults to 1.
|
21
|
+
# @param prefetch [Number] Maximum number of unacked messages to allow on
|
22
|
+
# the channel. See
|
23
|
+
# https://www.rabbitmq.com/docs/consumer-prefetch
|
24
|
+
# Defaults to 1.
|
25
|
+
def initialize(queue_name:, processor:, rabbitmq_connection: Consumer.default_connection_from_env, logger: Logger.new($stderr), worker_threads: 1, prefetch: 1)
|
31
26
|
@queue_name = queue_name
|
32
27
|
@processor = processor
|
33
28
|
@rabbitmq_connection = rabbitmq_connection
|
34
|
-
@statsd_client = statsd_client
|
35
29
|
@logger = logger
|
30
|
+
@worker_threads = worker_threads
|
31
|
+
@prefetch = prefetch
|
36
32
|
end
|
37
33
|
|
38
34
|
def run(subscribe_opts: {})
|
@@ -41,11 +37,8 @@ module GovukMessageQueueConsumer
|
|
41
37
|
subscribe_opts = { block: true, manual_ack: true }.merge(subscribe_opts)
|
42
38
|
queue.subscribe(subscribe_opts) do |delivery_info, headers, payload|
|
43
39
|
message = Message.new(payload, headers, delivery_info)
|
44
|
-
@statsd_client.increment("#{@queue_name}.started")
|
45
40
|
message_consumer.process(message)
|
46
|
-
@statsd_client.increment("#{@queue_name}.#{message.status}")
|
47
41
|
rescue StandardError => e
|
48
|
-
@statsd_client.increment("#{@queue_name}.uncaught_exception")
|
49
42
|
GovukError.notify(e) if defined?(GovukError)
|
50
43
|
@logger.error "Uncaught exception in processor: \n\n #{e.class}: #{e.message}\n\n#{e.backtrace.join("\n")}"
|
51
44
|
exit(1) # Ensure rabbitmq requeues outstanding messages
|
@@ -58,12 +51,6 @@ module GovukMessageQueueConsumer
|
|
58
51
|
|
59
52
|
private
|
60
53
|
|
61
|
-
class NullStatsd
|
62
|
-
def increment(_key); end
|
63
|
-
|
64
|
-
def count(_key, _volume); end
|
65
|
-
end
|
66
|
-
|
67
54
|
def message_consumer
|
68
55
|
@message_consumer ||= MessageConsumer.new(
|
69
56
|
processors: [
|
@@ -71,19 +58,18 @@ module GovukMessageQueueConsumer
|
|
71
58
|
JSONProcessor.new,
|
72
59
|
@processor,
|
73
60
|
],
|
74
|
-
handle_batches: self.class::HANDLE_BATCHES,
|
75
61
|
)
|
76
62
|
end
|
77
63
|
|
78
64
|
def queue
|
79
65
|
@queue ||= begin
|
80
|
-
channel.prefetch(
|
66
|
+
channel.prefetch(@prefetch)
|
81
67
|
channel.queue(@queue_name, no_declare: true)
|
82
68
|
end
|
83
69
|
end
|
84
70
|
|
85
71
|
def channel
|
86
|
-
@channel ||= @rabbitmq_connection.create_channel
|
72
|
+
@channel ||= @rabbitmq_connection.create_channel(nil, @worker_threads)
|
87
73
|
end
|
88
74
|
end
|
89
75
|
end
|
@@ -1,26 +1,12 @@
|
|
1
1
|
module GovukMessageQueueConsumer
|
2
2
|
class MessageConsumer
|
3
|
-
def initialize(processors
|
3
|
+
def initialize(processors:)
|
4
4
|
@processors = processors
|
5
|
-
@handle_batches = handle_batches
|
6
5
|
end
|
7
6
|
|
8
7
|
def process(records)
|
9
8
|
@processors.inject(Array(records)) do |remaining_records, processor|
|
10
|
-
|
11
|
-
processor.process(remaining_records)
|
12
|
-
else
|
13
|
-
remaining_records.select { |record| processor.process(record) }
|
14
|
-
end
|
15
|
-
end
|
16
|
-
end
|
17
|
-
|
18
|
-
def handles_batches?(processor)
|
19
|
-
case processor
|
20
|
-
when HeartbeatProcessor, JSONProcessor
|
21
|
-
false
|
22
|
-
else
|
23
|
-
@handle_batches
|
9
|
+
remaining_records.select { |record| processor.process(record) }
|
24
10
|
end
|
25
11
|
end
|
26
12
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: govuk_message_queue_consumer
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 5.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- GOV.UK Dev
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2024-04-
|
11
|
+
date: 2024-04-24 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bunny
|