govuk_message_queue_consumer 4.2.0 → 5.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 0afd12a5a1835a2c0b084462729846009ff2db28ac4315297cdbc1120beb0a3a
4
- data.tar.gz: '0567349e9ff36b486384d21280dec6102d92285dd6971e75dd548d892b9e5383'
3
+ metadata.gz: 9b19c5a4d8e4c1f6369d9630e64f7b31bd416fbd523512364f3bae8c3c538b5c
4
+ data.tar.gz: 92c67e9448299357417ac9db221e6c233f1ae38686b5cca704b90dc0d73fccdf
5
5
  SHA512:
6
- metadata.gz: 1061fe3d8164cb1d33cff988224cae58b3a8b2b3a8fc71d21657371e106b5c831ab51625f8cc7f6e1db4b32f717cc7806d34fa3b7cf0ec8422a8bf4b6bcaca00
7
- data.tar.gz: 860de89747df830547fac9ee2702a13adc36d218c23cfbb430a6f9992fe8cd69865e97bc9597f9e603e1bd4a325fd357421fee86786eb2078d3db3445a9b1c0e
6
+ metadata.gz: c28069a62e0823f858427c5c01eb6de8a8c58fe93afbbbc0c97295a4993993a1c535aeac8ac7f9e470a3d1ee83512b54932479f4ea75cac52bab8bbb53fe6b62
7
+ data.tar.gz: c76d78e878780c608af295ff60f7ec900da3fa6a262e2a8a26ce7c3e27ff57808bd3a7df23b679b9d0c032a5d58de730d6d91c1272f6a3f1dd269de35612c001
data/CHANGELOG.md CHANGED
@@ -1,3 +1,10 @@
1
+ # 5.0.0
2
+
3
+ - BREAKING: remove disused support for statsd. No clients in alphagov use the statsd functionality any more, so this is only theoretically breaking.
4
+ - Allow clients to specify a Bunny worker thread pool size of greater than 1. The default behaviour remains unchanged.
5
+ - Allow clients to specify "client prefetch" to allow more than one unacked message on a channel at a time. The default behaviour remains unchanged.
6
+ - Clean up some disused remnants of the batch consumer feature. The feature was removed in 4.0.0.
7
+
1
8
  # 4.2.0
2
9
 
3
10
  * Drop support for Ruby 3.0. The minimum required Ruby version is now 3.1.4.
data/README.md CHANGED
@@ -1,40 +1,40 @@
1
1
  # GOV.UK Message Queue Consumer
2
- [![Gem Version](https://badge.fury.io/rb/govuk_message_queue_consumer.svg)](https://badge.fury.io/rb/govuk_message_queue_consumer)
3
-
4
- Standardises the way GOV.UK consumes messages from [RabbitMQ](https://www.rabbitmq.com/).
5
- RabbitMQ is a messaging framework that allows applications to broadcast messages
6
- that can be picked up by other applications.
7
2
 
8
- On GOV.UK, [publishing-api](https://github.com/alphagov/publishing-api) publishes
9
- the content-items it receives, so that applications such as
10
- [email-alert-service](https://github.com/alphagov/email-alert-service) can be
11
- notified of changes in content.
3
+ [![Gem Version](https://badge.fury.io/rb/govuk_message_queue_consumer.svg)](https://badge.fury.io/rb/govuk_message_queue_consumer)
12
4
 
13
- For detailed documentation, check out the [gem documentation on rubydoc.info](http://www.rubydoc.info/gems/govuk_message_queue_consumer/GovukMessageQueueConsumer/Consumer#initialize-instance_method).
5
+ govuk_message_queue_consumer is a wrapper around the
6
+ [Bunny](https://github.com/ruby-amqp/bunny) gem for communicating with
7
+ [RabbitMQ](https://www.rabbitmq.com/). The user of govuk_message_queue_consumer
8
+ supplies some configuration and a class that processes messages.
14
9
 
15
- This gem is used by:
10
+ RabbitMQ is a multi-producer, multi-consumer message queue that allows
11
+ applications to subscribe to notifications published by other applications.
16
12
 
17
- - [Content Data API](https://github.com/alphagov/content-data-api).
18
- - [Email Alert Service](https://github.com/alphagov/email-alert-service/).
19
- - [Search API](https://github.com/alphagov/search-api).
20
- - [Search API v2](https://github.com/alphagov/search-api-v2).
13
+ GOV.UK [publishing-api](https://github.com/alphagov/publishing-api) publishes
14
+ a message to RabbitMQ when a ContentItem is added or changed. Other
15
+ applications (consumers) subscribe to these messages so that they can perform
16
+ actions such as emailing users or updating a search index.
21
17
 
22
- ## Overview of RabbitMQ
18
+ Several GOV.UK applications use govuk_message_queue_consumer:
23
19
 
24
- To see an overview of RabbitMQ and how we use it, see [here](https://docs.publishing.service.gov.uk/manual/rabbitmq.html#overview).
20
+ - [Content Data API](https://github.com/alphagov/content-data-api)
21
+ - [Email Alert Service](https://github.com/alphagov/email-alert-service/)
22
+ - [Search API](https://github.com/alphagov/search-api)
23
+ - [Search API v2](https://github.com/alphagov/search-api-v2)
25
24
 
26
25
  ## Technical documentation
27
26
 
28
- This is a ruby gem that deals with the boiler plate code of communicating with
29
- [RabbitMQ](https://www.rabbitmq.com/). The user of this gem is left the task of
30
- supplying the configuration and a class that processes messages.
27
+ You can browse the [API documentation on
28
+ rubydoc.info](http://www.rubydoc.info/gems/govuk_message_queue_consumer/GovukMessageQueueConsumer/Consumer#initialize-instance_method).
31
29
 
32
- The gem is automatically released by Jenkins. To release a new version, raise a
33
- pull request with the version number incremented.
30
+ ## Release a new version
34
31
 
35
- ### Dependencies
32
+ To release a new version, increment the [version
33
+ number](/lib/govuk_message_queue_consumer/version.rb) and raise a pull request.
36
34
 
37
- - The [Bunny](https://github.com/ruby-amqp/bunny) gem: to interact with RabbitMQ.
35
+ The [CI GitHub Actions
36
+ workflow](https://github.com/alphagov/govuk_message_queue_consumer/actions/workflows/ci.yml)
37
+ should automatically build and push the new release when you merge the PR.
38
38
 
39
39
  ## Usage
40
40
 
@@ -55,13 +55,15 @@ namespace :message_queue do
55
55
  end
56
56
  ```
57
57
 
58
- More options are [documented here](http://www.rubydoc.info/gems/govuk_message_queue_consumer/GovukMessageQueueConsumer/Consumer#initialize-instance_method).
58
+ See the [API documentation](http://www.rubydoc.info/gems/govuk_message_queue_consumer/GovukMessageQueueConsumer/Consumer#initialize-instance_method) for the full list of parameters.
59
59
 
60
- The consumer expects the [`RABBITMQ_URL` environment
60
+ `GovukMessageQueueConsumer::Consumer` expects the [`RABBITMQ_URL` environment
61
61
  variable](https://github.com/ruby-amqp/bunny/blob/066496d/docs/guides/connecting.md#paas-environments)
62
62
  to be set to an AMQP connection string, for example:
63
63
 
64
- `RABBITMQ_URL=amqp://mrbean:hunter2@rabbitmq.example.com:5672`
64
+ ```sh
65
+ RABBITMQ_URL=amqp://mrbean:hunter2@rabbitmq.example.com:5672
66
+ ```
65
67
 
66
68
  The GOV.UK-specific environment variables `RABBITMQ_HOSTS`, `RABBITMQ_VHOST`,
67
69
  `RABBITMQ_USER` and `RABBITMQ_PASSWORD` are deprecated. Support for these will
@@ -78,21 +80,13 @@ class MyProcessor
78
80
  end
79
81
  ```
80
82
 
81
- The worker should also be added to the Procfile to run in production:
83
+ You can start the worker by running the `message_queue:consumer` Rake task.
82
84
 
83
- ```
84
- # Procfile
85
- worker: bundle exec rake message_queue:consumer
86
- ```
87
-
88
- Because you need the environment variables when running the consumer, you should use
89
- `govuk_setenv` to run your app in development:
90
-
91
- ```
92
- $ govuk_setenv app-name bundle exec rake message_queue:consumer
85
+ ```sh
86
+ bundle exec rake message_queue:consumer
93
87
  ```
94
88
 
95
- ### Processing a message
89
+ ### Process a message
96
90
 
97
91
  Once you receive a message, you *must* tell RabbitMQ once you've processed it. This
98
92
  is called _acking_. You can also _discard_ the message, or _retry_ it.
@@ -116,34 +110,12 @@ class MyProcessor
116
110
  end
117
111
  ```
118
112
 
119
- ### Statsd integration
113
+ ### Test your processor
120
114
 
121
- You can pass a `statsd_client` to the `GovukMessageQueueConsumer::Consumer` initializer. The consumer will emit counters to statsd with these keys:
122
-
123
- - `your_queue_name.started` - message picked up from the your_queue_name
124
- - `your_queue_name.retried` - message has been retried
125
- - `your_queue_name.acked` - message has been processed and acked
126
- - `your_queue_name.discarded` - message has been discarded
127
- - `your_queue_name.uncaught_exception` - an uncaught exception occured during processing
128
-
129
- Remember to use a namespace for the `Statsd` client:
115
+ govuk_message_queue_consumer provides a test helper for your processor.
130
116
 
131
117
  ```ruby
132
- statsd_client = Statsd.new("localhost")
133
- statsd_client.namespace = "govuk.app.my_app_name"
134
-
135
- GovukMessageQueueConsumer::Consumer.new(
136
- statsd_client: statsd_client
137
- # ... other setup code omitted
138
- ).run
139
- ```
140
-
141
- ### Testing your processor
142
-
143
- This gem provides a test helper for your processor.
144
-
145
- ```ruby
146
- # eg. spec/queue_consumers/my_processor_spec.rb
118
+ # e.g. spec/queue_consumers/my_processor_spec.rb
147
119
  require 'test_helper'
148
120
  require 'govuk_message_queue_consumer/test_helpers'
149
121
 
@@ -172,10 +144,9 @@ it "acks incoming messages" do
172
144
  end
173
145
  ```
174
146
 
175
- For more test cases [see the spec for the mock itself](/spec/mock_message_spec.rb).
176
-
147
+ For more test cases [see the spec for the mock itself](/spec/govuk_message_queue_consumer/test_helpers/mock_message_spec.rb).
177
148
 
178
- ### Running the test suite
149
+ ### Run the test suite
179
150
 
180
151
  ```bash
181
152
  bundle exec rake spec
@@ -184,10 +155,8 @@ bundle exec rake spec
184
155
  ## Further reading
185
156
 
186
157
  - [Bunny](https://github.com/ruby-amqp/bunny) is the RabbitMQ client we use.
187
- - [The Bunny Guides](http://rubybunny.info/articles/guides.html) explain all
188
- AMQP concepts really well.
189
- - The [Developer Docs](https://docs.publishing.service.gov.uk/manual/rabbitmq.html)
190
- documents the usage of "heartbeat" messages, which this gem also supports.
158
+ - [The Bunny Guides](https://github.com/ruby-amqp/bunny/tree/main/docs/guides) explain
159
+ AMQP concepts.
191
160
 
192
161
  ## Licence
193
162
 
@@ -1,15 +1,5 @@
1
1
  module GovukMessageQueueConsumer
2
2
  class Consumer
3
- HANDLE_BATCHES = false
4
- # Only fetch one message at a time on the channel.
5
- #
6
- # By default, queues will grab messages eagerly, which reduces latency.
7
- # However, that also means that if multiple workers are running one worker
8
- # can starve another of work. We're not expecting a high throughput on this
9
- # queue, and a small bit of latency isn't a problem, so we fetch one at a
10
- # time to share the work evenly.
11
- NUMBER_OF_MESSAGES_TO_PREFETCH = 1
12
-
13
3
  def self.default_connection_from_env
14
4
  # https://github.com/ruby-amqp/bunny/blob/066496d/docs/guides/connecting.md#paas-environments
15
5
  if !ENV["RABBITMQ_URL"].to_s.empty?
@@ -22,17 +12,23 @@ module GovukMessageQueueConsumer
22
12
  # Create a new consumer
23
13
  #
24
14
  # @param queue_name [String] Your queue name. This is specific to your application,
25
- # and should already exist and have a binding via puppet
15
+ # and should already exist and have a binding
16
+ # configured via Terraform.
26
17
  # @param processor [Object] An object that responds to `process`
27
18
  # @param rabbitmq_connection [Object] A Bunny connection object derived from `Bunny.new`
28
- # @param statsd_client [Statsd] An instance of the Statsd class
29
19
  # @param logger [Object] A Logger object for emitting errors (to stderr by default)
30
- def initialize(queue_name:, processor:, rabbitmq_connection: Consumer.default_connection_from_env, statsd_client: NullStatsd.new, logger: Logger.new($stderr))
20
+ # @param worker_threads [Number] Size of the worker thread pool. Defaults to 1.
21
+ # @param prefetch [Number] Maximum number of unacked messages to allow on
22
+ # the channel. See
23
+ # https://www.rabbitmq.com/docs/consumer-prefetch
24
+ # Defaults to 1.
25
+ def initialize(queue_name:, processor:, rabbitmq_connection: Consumer.default_connection_from_env, logger: Logger.new($stderr), worker_threads: 1, prefetch: 1)
31
26
  @queue_name = queue_name
32
27
  @processor = processor
33
28
  @rabbitmq_connection = rabbitmq_connection
34
- @statsd_client = statsd_client
35
29
  @logger = logger
30
+ @worker_threads = worker_threads
31
+ @prefetch = prefetch
36
32
  end
37
33
 
38
34
  def run(subscribe_opts: {})
@@ -41,11 +37,8 @@ module GovukMessageQueueConsumer
41
37
  subscribe_opts = { block: true, manual_ack: true }.merge(subscribe_opts)
42
38
  queue.subscribe(subscribe_opts) do |delivery_info, headers, payload|
43
39
  message = Message.new(payload, headers, delivery_info)
44
- @statsd_client.increment("#{@queue_name}.started")
45
40
  message_consumer.process(message)
46
- @statsd_client.increment("#{@queue_name}.#{message.status}")
47
41
  rescue StandardError => e
48
- @statsd_client.increment("#{@queue_name}.uncaught_exception")
49
42
  GovukError.notify(e) if defined?(GovukError)
50
43
  @logger.error "Uncaught exception in processor: \n\n #{e.class}: #{e.message}\n\n#{e.backtrace.join("\n")}"
51
44
  exit(1) # Ensure rabbitmq requeues outstanding messages
@@ -58,12 +51,6 @@ module GovukMessageQueueConsumer
58
51
 
59
52
  private
60
53
 
61
- class NullStatsd
62
- def increment(_key); end
63
-
64
- def count(_key, _volume); end
65
- end
66
-
67
54
  def message_consumer
68
55
  @message_consumer ||= MessageConsumer.new(
69
56
  processors: [
@@ -71,19 +58,18 @@ module GovukMessageQueueConsumer
71
58
  JSONProcessor.new,
72
59
  @processor,
73
60
  ],
74
- handle_batches: self.class::HANDLE_BATCHES,
75
61
  )
76
62
  end
77
63
 
78
64
  def queue
79
65
  @queue ||= begin
80
- channel.prefetch(self.class::NUMBER_OF_MESSAGES_TO_PREFETCH)
66
+ channel.prefetch(@prefetch)
81
67
  channel.queue(@queue_name, no_declare: true)
82
68
  end
83
69
  end
84
70
 
85
71
  def channel
86
- @channel ||= @rabbitmq_connection.create_channel
72
+ @channel ||= @rabbitmq_connection.create_channel(nil, @worker_threads)
87
73
  end
88
74
  end
89
75
  end
@@ -1,26 +1,12 @@
1
1
  module GovukMessageQueueConsumer
2
2
  class MessageConsumer
3
- def initialize(processors:, handle_batches:)
3
+ def initialize(processors:)
4
4
  @processors = processors
5
- @handle_batches = handle_batches
6
5
  end
7
6
 
8
7
  def process(records)
9
8
  @processors.inject(Array(records)) do |remaining_records, processor|
10
- if handles_batches?(processor)
11
- processor.process(remaining_records)
12
- else
13
- remaining_records.select { |record| processor.process(record) }
14
- end
15
- end
16
- end
17
-
18
- def handles_batches?(processor)
19
- case processor
20
- when HeartbeatProcessor, JSONProcessor
21
- false
22
- else
23
- @handle_batches
9
+ remaining_records.select { |record| processor.process(record) }
24
10
  end
25
11
  end
26
12
  end
@@ -1,3 +1,3 @@
1
1
  module GovukMessageQueueConsumer
2
- VERSION = "4.2.0".freeze
2
+ VERSION = "5.0.0".freeze
3
3
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: govuk_message_queue_consumer
3
3
  version: !ruby/object:Gem::Version
4
- version: 4.2.0
4
+ version: 5.0.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - GOV.UK Dev
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2024-04-15 00:00:00.000000000 Z
11
+ date: 2024-04-24 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bunny