publishing_platform_message_queue_consumer 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: dcb24019c8122307ea1065e358a617790aedfd2b286e8830b46861518a7d8cf9
+  data.tar.gz: 79abd82a0b158c72d7323655df80a73a26408421aeb81ccbf31ba7cf0a2960fb
+SHA512:
+  metadata.gz: 167902fc49cd11027155662b8ab3f3f3792bdc9e7fc40ea75689bec513a38a0da1af9f02a82bf19786b6015d0b6182f2a7e535b126a0d048de96f9891316d375
+  data.tar.gz: 393f90f3603f6ab1e43595bfe863f00b0fd3ce841a8447a8e4a3e5820dac7051490bb0e278a90b76345a5280343437b5cb591036a4a10491bfdbb0c60b98be43

data/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2025 Publishing Platform
+Copyright (C) 2015 Crown copyright (Government Digital Service)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,133 @@
+# Publishing Platform Message Queue Consumer
+publishing_platform_message_queue_consumer is a wrapper around the
+[Bunny](https://github.com/ruby-amqp/bunny) gem for communicating with
+[RabbitMQ](https://www.rabbitmq.com/). The user of publishing_platform_message_queue_consumer
+supplies some configuration and a class that processes messages.
+
+RabbitMQ is a multi-producer, multi-consumer message queue that allows
+applications to subscribe to notifications published by other applications.
+
+Publishing Platform [publishing-api](https://github.com/publishing-platform/publishing-api) publishes
+a message to RabbitMQ when a ContentItem is added or changed. Other
+applications (consumers) subscribe to these messages so that they can perform
+actions such as emailing users or updating a search index.
+
+## Usage
+
+[Add the gem to your Gemfile](https://rubygems.org/gems/publishing_platform_message_queue_consumer).
+
+Add a rake task like the following example:
+
+```ruby
+# lib/tasks/message_queue.rake
+namespace :message_queue do
+  desc "Run worker to consume messages from rabbitmq"
+  task consumer: :environment do
+    PublishingPlatformMessageQueueConsumer::Consumer.new(
+      queue_name: "some-queue",
+      processor: MyProcessor.new,
+    ).run
+  end
+end
+```
+
+`PublishingPlatformMessageQueueConsumer::Consumer` expects the [`RABBITMQ_URL` environment
+variable](https://github.com/ruby-amqp/bunny/blob/066496d/docs/guides/connecting.md#paas-environments)
+to be set to an AMQP connection string, for example:
+
+```sh
+RABBITMQ_URL=amqp://mrbean:hunter2@rabbitmq.example.com:5672
+```
+
+Define a class that will process the messages:
+
+```ruby
+# eg. app/queue_consumers/my_processor.rb
+class MyProcessor
+  def process(message)
+    # do something cool
+  end
+end
+```
+
+You can start the worker by running the `message_queue:consumer` Rake task.
+
+```sh
+bundle exec rake message_queue:consumer
+```
+
+### Process a message
+
+Once you receive a message, you *must* tell RabbitMQ once you've processed it. This
+is called _acking_. You can also _discard_ the message, or _retry_ it.
+
+```ruby
+class MyProcessor
+  def process(message)
+    result = do_something_with(message)
+
+    if result.ok?
+      # Ack the message when it has been processed correctly.
+      message.ack
+    elsif result.failed_temporarily?
+      # Retry the message to make RabbitMQ send the message again later.
+      message.retry
+    elsif result.failed_permanently?
+      # Discard the message when it can't be processed.
+      message.discard
+    end
+  end
+end
+
+### Test your processor
+
+publishing_platform_message_queue_consumer provides a test helper for your processor.
+
+```ruby
+# e.g. spec/queue_consumers/my_processor_spec.rb
+require 'test_helper'
+require 'publishing_platform_message_queue_consumer/test_helpers'
+
+describe MyProcessor do
+  it_behaves_like "a message queue processor"
+end
+```
+
+This will verify that your processor class implements the correct methods. You
+should add your own tests to verify its behaviour.
+
+You can use `PublishingPlatformMessageQueueConsumer::MockMessage` to test the processor
+behaviour. When using the mock, you can verify it acknowledged, retried or
+discarded. For example, with `MyProcessor` above:
+
+```ruby
+it "acks incoming messages" do
+  message = PublishingPlatformMessageQueueConsumer::MockMessage.new
+
+  MyProcessor.new.process(message)
+
+  expect(message).to be_acked
+end
+```
+
+For more test cases [see the spec for the mock itself](/spec/publishing_platform_message_queue_consumer/test_helpers/mock_message_spec.rb).
+
+### Run the test suite
+
+```bash
+bundle exec rake spec
+```
+
+## Further reading
+
+- [Bunny](https://github.com/ruby-amqp/bunny) is the RabbitMQ client we use.
+- [The Bunny Guides](https://github.com/ruby-amqp/bunny/tree/main/docs/guides) explain
+  AMQP concepts.
+
+## Licence
+
+[MIT License](LICENCE)
+
+## Versioning policy
+
+We follow [Semantic versioning](http://semver.org/spec/v2.0.0.html).

data/lib/publishing_platform_message_queue_consumer/consumer.rb

@@ -0,0 +1,66 @@
+module PublishingPlatformMessageQueueConsumer
+  class Consumer
+    # Create a new consumer
+    #
+    # @param queue_name [String] Your queue name. This is specific to your application,
+    #                            and should already exist and have a binding
+    #                            configured via Terraform.
+    # @param processor [Object] An object that responds to `process`
+    # @param rabbitmq_connection [Object] A Bunny connection object derived from `Bunny.new`
+    # @param logger [Object] A Logger object for emitting errors (to stderr by default)
+    # @param worker_threads [Number] Size of the worker thread pool. Defaults to 1.
+    # @param prefetch [Number] Maximum number of unacked messages to allow on
+    #                          the channel. See
+    #                          https://www.rabbitmq.com/docs/consumer-prefetch
+    #                          Defaults to 1.
+    def initialize(queue_name:, processor:, rabbitmq_connection: Bunny.new, logger: Logger.new($stderr), worker_threads: 1, prefetch: 1)
+      @queue_name = queue_name
+      @processor = processor
+      @rabbitmq_connection = rabbitmq_connection
+      @logger = logger
+      @worker_threads = worker_threads
+      @prefetch = prefetch
+    end
+
+    def run(subscribe_opts: {})
+      @rabbitmq_connection.start
+
+      subscribe_opts = { block: true, manual_ack: true }.merge(subscribe_opts)
+      queue.subscribe(subscribe_opts) do |delivery_info, headers, payload|
+        message = Message.new(payload, headers, delivery_info)
+        message_consumer.process(message)
+      rescue StandardError => e
+        PublishingPlatformError.notify(e) if defined?(PublishingPlatformError)
+        @logger.error "Uncaught exception in processor: \n\n #{e.class}: #{e.message}\n\n#{e.backtrace.join("\n")}"
+        exit(1) # Ensure rabbitmq requeues outstanding messages
+      end
+    rescue SignalException => e
+      PublishingPlatformError.notify(e) if defined?(PublishingPlatformError) && e.message != "SIGTERM"
+
+      exit
+    end
+
+  private
+
+    def message_consumer
+      @message_consumer ||= MessageConsumer.new(
+        processors: [
+          HeartbeatProcessor.new,
+          JSONProcessor.new,
+          @processor,
+        ],
+      )
+    end
+
+    def queue
+      @queue ||= begin
+        channel.prefetch(@prefetch)
+        channel.queue(@queue_name, no_declare: true)
+      end
+    end
+
+    def channel
+      @channel ||= @rabbitmq_connection.create_channel(nil, @worker_threads)
+    end
+  end
+end

data/lib/publishing_platform_message_queue_consumer/heartbeat_processor.rb

@@ -0,0 +1,13 @@
+module PublishingPlatformMessageQueueConsumer
+  class HeartbeatProcessor
+    def process(message)
+      # Ignore heartbeat messages
+      if message.headers.content_type == "application/x-heartbeat"
+        message.ack
+        return false
+      end
+
+      true
+    end
+  end
+end

data/lib/publishing_platform_message_queue_consumer/json_processor.rb

@@ -0,0 +1,17 @@
+module PublishingPlatformMessageQueueConsumer
+  class JSONProcessor
+    JSON_FORMAT = "application/json".freeze
+
+    def process(message)
+      if message.headers.content_type == JSON_FORMAT
+        message.payload = JSON.parse(message.payload)
+      end
+
+      true
+    rescue JSON::ParserError => e
+      PublishingPlatformError.notify(e) if defined?(PublishingPlatformError)
+      message.discard
+      false
+    end
+  end
+end

data/lib/publishing_platform_message_queue_consumer/message.rb

@@ -0,0 +1,28 @@
+module PublishingPlatformMessageQueueConsumer
+  # Client code will receive an instance of this
+  class Message
+    attr_accessor :delivery_info, :headers, :payload, :status
+
+    def initialize(payload, headers, delivery_info)
+      @payload = payload
+      @headers = headers
+      @delivery_info = delivery_info
+      @status = :status
+    end
+
+    def ack
+      @delivery_info.channel.ack(@delivery_info.delivery_tag)
+      @status = :acked
+    end
+
+    def retry
+      @delivery_info.channel.reject(@delivery_info.delivery_tag, true)
+      @status = :retried
+    end
+
+    def discard
+      @delivery_info.channel.reject(@delivery_info.delivery_tag, false)
+      @status = :discarded
+    end
+  end
+end

data/lib/publishing_platform_message_queue_consumer/message_consumer.rb

@@ -0,0 +1,13 @@
+module PublishingPlatformMessageQueueConsumer
+  class MessageConsumer
+    def initialize(processors:)
+      @processors = processors
+    end
+
+    def process(message)
+      @processors.each do |processor|
+        break unless processor.process(message)
+      end
+    end
+  end
+end

data/lib/publishing_platform_message_queue_consumer/test_helpers/mock_message.rb

@@ -0,0 +1,27 @@
+require "ostruct"
+
+module PublishingPlatformMessageQueueConsumer
+  class MockMessage < Message
+    attr_reader :acked, :retried, :discarded, :payload, :header, :delivery_info
+
+    alias_method :acked?, :acked
+    alias_method :discarded?, :discarded
+    alias_method :retried?, :retried
+
+    def initialize(payload = {}, headers = {}, delivery_info = {})
+      super(payload, OpenStruct.new(headers), OpenStruct.new(delivery_info))
+    end
+
+    def ack
+      @acked = true
+    end
+
+    def retry
+      @retried = true
+    end
+
+    def discard
+      @discarded = true
+    end
+  end
+end

data/lib/publishing_platform_message_queue_consumer/test_helpers/shared_examples.rb

@@ -0,0 +1,11 @@
+if defined?(RSpec)
+  RSpec.shared_examples "a message queue processor" do
+    it "implements #process" do
+      expect(subject).to respond_to(:process)
+    end
+
+    it "accepts 1 argument for #process" do
+      expect(subject.method(:process).arity).to eq(1)
+    end
+  end
+end

data/lib/publishing_platform_message_queue_consumer/test_helpers.rb

@@ -0,0 +1,2 @@
+require "publishing_platform_message_queue_consumer/test_helpers/shared_examples"
+require "publishing_platform_message_queue_consumer/test_helpers/mock_message"

data/lib/publishing_platform_message_queue_consumer/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module PublishingPlatformMessageQueueConsumer
+  VERSION = "0.1.0"
+end

data/lib/publishing_platform_message_queue_consumer.rb

@@ -0,0 +1,9 @@
+require "bunny"
+require "json"
+
+require "publishing_platform_message_queue_consumer/version"
+require "publishing_platform_message_queue_consumer/heartbeat_processor"
+require "publishing_platform_message_queue_consumer/json_processor"
+require "publishing_platform_message_queue_consumer/message"
+require "publishing_platform_message_queue_consumer/message_consumer"
+require "publishing_platform_message_queue_consumer/consumer"

metadata

@@ -0,0 +1,106 @@
+--- !ruby/object:Gem::Specification
+name: publishing_platform_message_queue_consumer
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Publishing Platform
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bunny
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.24'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.24'
+- !ruby/object:Gem::Dependency
+  name: bunny-mock
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: publishing_platform_rubocop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: simplecov
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Avoid writing boilerplate code in order to consume messages from an AMQP
+  message queue. Plug in queue configuration, and how to process each message.
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- LICENSE
+- README.md
+- lib/publishing_platform_message_queue_consumer.rb
+- lib/publishing_platform_message_queue_consumer/consumer.rb
+- lib/publishing_platform_message_queue_consumer/heartbeat_processor.rb
+- lib/publishing_platform_message_queue_consumer/json_processor.rb
+- lib/publishing_platform_message_queue_consumer/message.rb
+- lib/publishing_platform_message_queue_consumer/message_consumer.rb
+- lib/publishing_platform_message_queue_consumer/test_helpers.rb
+- lib/publishing_platform_message_queue_consumer/test_helpers/mock_message.rb
+- lib/publishing_platform_message_queue_consumer/test_helpers/shared_examples.rb
+- lib/publishing_platform_message_queue_consumer/version.rb
+licenses:
+- MIT
+metadata: {}
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.1'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.6.9
+specification_version: 4
+summary: AMQP message queue consumption with Publishing Platform conventions
+test_files: []