govuk_message_queue_consumer 4.2.0 → 5.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0afd12a5a1835a2c0b084462729846009ff2db28ac4315297cdbc1120beb0a3a
-  data.tar.gz: '0567349e9ff36b486384d21280dec6102d92285dd6971e75dd548d892b9e5383'
+  metadata.gz: 9b19c5a4d8e4c1f6369d9630e64f7b31bd416fbd523512364f3bae8c3c538b5c
+  data.tar.gz: 92c67e9448299357417ac9db221e6c233f1ae38686b5cca704b90dc0d73fccdf
 SHA512:
-  metadata.gz: 1061fe3d8164cb1d33cff988224cae58b3a8b2b3a8fc71d21657371e106b5c831ab51625f8cc7f6e1db4b32f717cc7806d34fa3b7cf0ec8422a8bf4b6bcaca00
-  data.tar.gz: 860de89747df830547fac9ee2702a13adc36d218c23cfbb430a6f9992fe8cd69865e97bc9597f9e603e1bd4a325fd357421fee86786eb2078d3db3445a9b1c0e
+  metadata.gz: c28069a62e0823f858427c5c01eb6de8a8c58fe93afbbbc0c97295a4993993a1c535aeac8ac7f9e470a3d1ee83512b54932479f4ea75cac52bab8bbb53fe6b62
+  data.tar.gz: c76d78e878780c608af295ff60f7ec900da3fa6a262e2a8a26ce7c3e27ff57808bd3a7df23b679b9d0c032a5d58de730d6d91c1272f6a3f1dd269de35612c001

data/CHANGELOG.md

@@ -1,3 +1,10 @@
+# 5.0.0
+
+- BREAKING: remove disused support for statsd. No clients in alphagov use the statsd functionality any more, so this is only theoretically breaking.
+- Allow clients to specify a Bunny worker thread pool size of greater than 1. The default behaviour remains unchanged.
+- Allow clients to specify "client prefetch" to allow more than one unacked message on a channel at a time. The default behaviour remains unchanged.
+- Clean up some disused remnants of the batch consumer feature. The feature was removed in 4.0.0.
+
 # 4.2.0
 
 * Drop support for Ruby 3.0. The minimum required Ruby version is now 3.1.4.

data/README.md

@@ -1,40 +1,40 @@
 # GOV.UK Message Queue Consumer
-[![Gem Version](https://badge.fury.io/rb/govuk_message_queue_consumer.svg)](https://badge.fury.io/rb/govuk_message_queue_consumer)
-
-Standardises the way GOV.UK consumes messages from [RabbitMQ](https://www.rabbitmq.com/).
-RabbitMQ is a messaging framework that allows applications to broadcast messages
-that can be picked up by other applications.
 
-On GOV.UK, [publishing-api](https://github.com/alphagov/publishing-api) publishes
-the content-items it receives, so that applications such as
-[email-alert-service](https://github.com/alphagov/email-alert-service) can be
-notified of changes in content.
+[![Gem Version](https://badge.fury.io/rb/govuk_message_queue_consumer.svg)](https://badge.fury.io/rb/govuk_message_queue_consumer)
 
-For detailed documentation, check out the [gem documentation on rubydoc.info](http://www.rubydoc.info/gems/govuk_message_queue_consumer/GovukMessageQueueConsumer/Consumer#initialize-instance_method).
+govuk_message_queue_consumer is a wrapper around the
+[Bunny](https://github.com/ruby-amqp/bunny) gem for communicating with
+[RabbitMQ](https://www.rabbitmq.com/). The user of govuk_message_queue_consumer
+supplies some configuration and a class that processes messages.
 
-This gem is used by:
+RabbitMQ is a multi-producer, multi-consumer message queue that allows
+applications to subscribe to notifications published by other applications.
 
-- [Content Data API](https://github.com/alphagov/content-data-api).
-- [Email Alert Service](https://github.com/alphagov/email-alert-service/).
-- [Search API](https://github.com/alphagov/search-api).
-- [Search API v2](https://github.com/alphagov/search-api-v2).
+GOV.UK [publishing-api](https://github.com/alphagov/publishing-api) publishes
+a message to RabbitMQ when a ContentItem is added or changed. Other
+applications (consumers) subscribe to these messages so that they can perform
+actions such as emailing users or updating a search index.
 
-## Overview of RabbitMQ
+Several GOV.UK applications use govuk_message_queue_consumer:
 
-To see an overview of RabbitMQ and how we use it, see [here](https://docs.publishing.service.gov.uk/manual/rabbitmq.html#overview).
+- [Content Data API](https://github.com/alphagov/content-data-api)
+- [Email Alert Service](https://github.com/alphagov/email-alert-service/)
+- [Search API](https://github.com/alphagov/search-api)
+- [Search API v2](https://github.com/alphagov/search-api-v2)
 
 ## Technical documentation
 
-This is a ruby gem that deals with the boiler plate code of communicating with
-[RabbitMQ](https://www.rabbitmq.com/). The user of this gem is left the task of
-supplying the configuration and a class that processes messages.
+You can browse the [API documentation on
+rubydoc.info](http://www.rubydoc.info/gems/govuk_message_queue_consumer/GovukMessageQueueConsumer/Consumer#initialize-instance_method).
 
-The gem is automatically released by Jenkins. To release a new version, raise a
-pull request with the version number incremented.
+## Release a new version
 
-### Dependencies
+To release a new version, increment the [version
+number](/lib/govuk_message_queue_consumer/version.rb) and raise a pull request.
 
-- The [Bunny](https://github.com/ruby-amqp/bunny) gem: to interact with RabbitMQ.
+The [CI GitHub Actions
+workflow](https://github.com/alphagov/govuk_message_queue_consumer/actions/workflows/ci.yml)
+should automatically build and push the new release when you merge the PR.
 
 ## Usage
 
@@ -55,13 +55,15 @@ namespace :message_queue do
 end
 ```
 
-More options are [documented here](http://www.rubydoc.info/gems/govuk_message_queue_consumer/GovukMessageQueueConsumer/Consumer#initialize-instance_method).
+See the [API documentation](http://www.rubydoc.info/gems/govuk_message_queue_consumer/GovukMessageQueueConsumer/Consumer#initialize-instance_method) for the full list of parameters.
 
-The consumer expects the [`RABBITMQ_URL` environment
+`GovukMessageQueueConsumer::Consumer` expects the [`RABBITMQ_URL` environment
 variable](https://github.com/ruby-amqp/bunny/blob/066496d/docs/guides/connecting.md#paas-environments)
 to be set to an AMQP connection string, for example:
 
-`RABBITMQ_URL=amqp://mrbean:hunter2@rabbitmq.example.com:5672`
+```sh
+RABBITMQ_URL=amqp://mrbean:hunter2@rabbitmq.example.com:5672
+```
 
 The GOV.UK-specific environment variables `RABBITMQ_HOSTS`, `RABBITMQ_VHOST`,
 `RABBITMQ_USER` and `RABBITMQ_PASSWORD` are deprecated. Support for these will
@@ -78,21 +80,13 @@ class MyProcessor
 end
 ```
 
-The worker should also be added to the Procfile to run in production:
+You can start the worker by running the `message_queue:consumer` Rake task.
 
-```
-# Procfile
-worker: bundle exec rake message_queue:consumer
-```
-
-Because you need the environment variables when running the consumer, you should use
-`govuk_setenv` to run your app in development:
-
-```
-$ govuk_setenv app-name bundle exec rake message_queue:consumer
+```sh
+bundle exec rake message_queue:consumer
 ```
 
-### Processing a message
+### Process a message
 
 Once you receive a message, you *must* tell RabbitMQ once you've processed it. This
 is called _acking_. You can also _discard_ the message, or _retry_ it.
@@ -116,34 +110,12 @@ class MyProcessor
 end
 ```
 
-### Statsd integration
+### Test your processor
 
-You can pass a `statsd_client` to the `GovukMessageQueueConsumer::Consumer` initializer. The consumer will emit counters to statsd with these keys:
-
-- `your_queue_name.started` - message picked up from the your_queue_name
-- `your_queue_name.retried` - message has been retried
-- `your_queue_name.acked` - message has been processed and acked
-- `your_queue_name.discarded` - message has been discarded
-- `your_queue_name.uncaught_exception` - an uncaught exception occured during processing
-
-Remember to use a namespace for the `Statsd` client:
+govuk_message_queue_consumer provides a test helper for your processor.
 
 ```ruby
-statsd_client = Statsd.new("localhost")
-statsd_client.namespace = "govuk.app.my_app_name"
-
-GovukMessageQueueConsumer::Consumer.new(
-  statsd_client: statsd_client
-  # ... other setup code omitted
-).run
-```
-
-### Testing your processor
-
-This gem provides a test helper for your processor.
-
-```ruby
-# eg. spec/queue_consumers/my_processor_spec.rb
+# e.g. spec/queue_consumers/my_processor_spec.rb
 require 'test_helper'
 require 'govuk_message_queue_consumer/test_helpers'
 
@@ -172,10 +144,9 @@ it "acks incoming messages" do
 end
 ```
 
-For more test cases [see the spec for the mock itself](/spec/mock_message_spec.rb).
-
+For more test cases [see the spec for the mock itself](/spec/govuk_message_queue_consumer/test_helpers/mock_message_spec.rb).
 
-### Running the test suite
+### Run the test suite
 
 ```bash
 bundle exec rake spec
@@ -184,10 +155,8 @@ bundle exec rake spec
 ## Further reading
 
 - [Bunny](https://github.com/ruby-amqp/bunny) is the RabbitMQ client we use.
-- [The Bunny Guides](http://rubybunny.info/articles/guides.html) explain all
-  AMQP concepts really well.
-- The [Developer Docs](https://docs.publishing.service.gov.uk/manual/rabbitmq.html)
-  documents the usage of "heartbeat" messages, which this gem also supports.
+- [The Bunny Guides](https://github.com/ruby-amqp/bunny/tree/main/docs/guides) explain
+  AMQP concepts.
 
 ## Licence
 

data/lib/govuk_message_queue_consumer/consumer.rb

@@ -1,15 +1,5 @@
 module GovukMessageQueueConsumer
   class Consumer
-    HANDLE_BATCHES = false
-    # Only fetch one message at a time on the channel.
-    #
-    # By default, queues will grab messages eagerly, which reduces latency.
-    # However, that also means that if multiple workers are running one worker
-    # can starve another of work.  We're not expecting a high throughput on this
-    # queue, and a small bit of latency isn't a problem, so we fetch one at a
-    # time to share the work evenly.
-    NUMBER_OF_MESSAGES_TO_PREFETCH = 1
-
     def self.default_connection_from_env
       # https://github.com/ruby-amqp/bunny/blob/066496d/docs/guides/connecting.md#paas-environments
       if !ENV["RABBITMQ_URL"].to_s.empty?
@@ -22,17 +12,23 @@ module GovukMessageQueueConsumer
     # Create a new consumer
     #
     # @param queue_name [String] Your queue name. This is specific to your application,
-    #                            and should already exist and have a binding via puppet
+    #                            and should already exist and have a binding
+    #                            configured via Terraform.
     # @param processor [Object] An object that responds to `process`
     # @param rabbitmq_connection [Object] A Bunny connection object derived from `Bunny.new`
-    # @param statsd_client [Statsd] An instance of the Statsd class
     # @param logger [Object] A Logger object for emitting errors (to stderr by default)
-    def initialize(queue_name:, processor:, rabbitmq_connection: Consumer.default_connection_from_env, statsd_client: NullStatsd.new, logger: Logger.new($stderr))
+    # @param worker_threads [Number] Size of the worker thread pool. Defaults to 1.
+    # @param prefetch [Number] Maximum number of unacked messages to allow on
+    #                          the channel. See
+    #                          https://www.rabbitmq.com/docs/consumer-prefetch
+    #                          Defaults to 1.
+    def initialize(queue_name:, processor:, rabbitmq_connection: Consumer.default_connection_from_env, logger: Logger.new($stderr), worker_threads: 1, prefetch: 1)
       @queue_name = queue_name
       @processor = processor
       @rabbitmq_connection = rabbitmq_connection
-      @statsd_client = statsd_client
       @logger = logger
+      @worker_threads = worker_threads
+      @prefetch = prefetch
     end
 
     def run(subscribe_opts: {})
@@ -41,11 +37,8 @@ module GovukMessageQueueConsumer
       subscribe_opts = { block: true, manual_ack: true }.merge(subscribe_opts)
       queue.subscribe(subscribe_opts) do |delivery_info, headers, payload|
         message = Message.new(payload, headers, delivery_info)
-        @statsd_client.increment("#{@queue_name}.started")
         message_consumer.process(message)
-        @statsd_client.increment("#{@queue_name}.#{message.status}")
       rescue StandardError => e
-        @statsd_client.increment("#{@queue_name}.uncaught_exception")
         GovukError.notify(e) if defined?(GovukError)
         @logger.error "Uncaught exception in processor: \n\n #{e.class}: #{e.message}\n\n#{e.backtrace.join("\n")}"
         exit(1) # Ensure rabbitmq requeues outstanding messages
@@ -58,12 +51,6 @@ module GovukMessageQueueConsumer
 
   private
 
-    class NullStatsd
-      def increment(_key); end
-
-      def count(_key, _volume); end
-    end
-
     def message_consumer
       @message_consumer ||= MessageConsumer.new(
         processors: [
@@ -71,19 +58,18 @@ module GovukMessageQueueConsumer
           JSONProcessor.new,
           @processor,
         ],
-        handle_batches: self.class::HANDLE_BATCHES,
       )
     end
 
     def queue
       @queue ||= begin
-        channel.prefetch(self.class::NUMBER_OF_MESSAGES_TO_PREFETCH)
+        channel.prefetch(@prefetch)
         channel.queue(@queue_name, no_declare: true)
       end
     end
 
     def channel
-      @channel ||= @rabbitmq_connection.create_channel
+      @channel ||= @rabbitmq_connection.create_channel(nil, @worker_threads)
     end
   end
 end

data/lib/govuk_message_queue_consumer/message_consumer.rb

@@ -1,26 +1,12 @@
 module GovukMessageQueueConsumer
   class MessageConsumer
-    def initialize(processors:, handle_batches:)
+    def initialize(processors:)
       @processors = processors
-      @handle_batches = handle_batches
     end
 
     def process(records)
       @processors.inject(Array(records)) do |remaining_records, processor|
-        if handles_batches?(processor)
-          processor.process(remaining_records)
-        else
-          remaining_records.select { |record| processor.process(record) }
-        end
-      end
-    end
-
-    def handles_batches?(processor)
-      case processor
-      when HeartbeatProcessor, JSONProcessor
-        false
-      else
-        @handle_batches
+        remaining_records.select { |record| processor.process(record) }
       end
     end
   end

data/lib/govuk_message_queue_consumer/version.rb

@@ -1,3 +1,3 @@
 module GovukMessageQueueConsumer
-  VERSION = "4.2.0".freeze
+  VERSION = "5.0.0".freeze
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: govuk_message_queue_consumer
 version: !ruby/object:Gem::Version
-  version: 4.2.0
+  version: 5.0.0
 platform: ruby
 authors:
 - GOV.UK Dev
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2024-04-15 00:00:00.000000000 Z
+date: 2024-04-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bunny