govuk_message_queue_consumer 1.0.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +132 -37
- data/lib/govuk_message_queue_consumer/consumer.rb +41 -21
- data/lib/govuk_message_queue_consumer/json_processor.rb +20 -0
- data/lib/govuk_message_queue_consumer/message.rb +5 -9
- data/lib/govuk_message_queue_consumer/rabbitmq_config.rb +29 -0
- data/lib/govuk_message_queue_consumer/test_helpers/mock_message.rb +27 -0
- data/lib/govuk_message_queue_consumer/test_helpers/shared_examples.rb +8 -6
- data/lib/govuk_message_queue_consumer/test_helpers/test_consumer.rb +12 -0
- data/lib/govuk_message_queue_consumer/test_helpers.rb +2 -0
- data/lib/govuk_message_queue_consumer/version.rb +1 -1
- data/lib/govuk_message_queue_consumer.rb +2 -0
- data/spec/consumer_spec.rb +12 -25
- data/spec/json_processor_spec.rb +33 -0
- data/spec/message_spec.rb +4 -6
- data/spec/mock_message_spec.rb +43 -0
- data/spec/rabbitmq_config_spec.rb +26 -0
- data/spec/spec_helper.rb +5 -17
- metadata +12 -16
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: f3d0a2e8f69b4d555db92e1251f26b8238f1ad63
|
4
|
+
data.tar.gz: c97edff469050cfbf09031625ff3e22f2c573bee
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 0ad7430c3ed9f12498a91b90c30050737809f0e2c0234e0892a417d7813004c306d68c95992c04def1fa23b987bc6f5652671472685a8a95dcc257f5ec5c15be
|
7
|
+
data.tar.gz: b719dadfe75138257d573f91ea758ebf6878ac076971a76dd49012b8de6dd2a35c9c68b559609e1a44148c6ced539c9d8d510c7ad59c95e38241a1903eebd0c6
|
data/README.md
CHANGED
@@ -1,73 +1,140 @@
|
|
1
|
-
# GOV.UK Message
|
1
|
+
# GOV.UK Message Queue Consumer
|
2
2
|
|
3
|
-
|
3
|
+
Standardises the way GOV.UK consumes messages from [RabbitMQ](https://www.rabbitmq.com/).
|
4
|
+
RabbitMQ is a messaging framework that allows applications to broadcast messages
|
5
|
+
that can be picked up by other applications.
|
4
6
|
|
7
|
+
On GOV.UK, [publishing-api](https://github.com/alphagov/publishing-api) publishes
|
8
|
+
the content-items it receives, so that applications such as
|
9
|
+
[email-alert-service](https://github.com/alphagov/email-alert-service) can be
|
10
|
+
notified of changes in content.
|
5
11
|
|
6
12
|
## Nomenclature
|
7
13
|
|
8
|
-
|
9
|
-
|
10
|
-
|
14
|
+

|
15
|
+
|
16
|
+
- **Producer**: an application that sends messages RabbitMQ. On GOV.UK this could
|
17
|
+
be [publishing-api](https://github.com/alphagov/publishing-api).
|
18
|
+
- **Message**: an object sent over RabbitMQ. It consists of a _payload_ and
|
19
|
+
_headers_. In the case of the publishing-api the payload is a
|
20
|
+
[content item](https://github.com/alphagov/govuk-content-schemas).
|
21
|
+
- **Consumer**: the app that receives the messages and does something with them.
|
22
|
+
On GOV.UK, these could be [email-alert-service](https://github.com/alphagov/email-alert-service)
|
23
|
+
and [content-register](https://github.com/alphagov/content-register).
|
24
|
+
- **Exchange**: in RabbitMQ's model, producers send messages to an _exchange_.
|
25
|
+
Consumers can create a Queue that listens to the exchange, instead of
|
26
|
+
subscribing to the exchange directly. This is done so that the queue can buffer
|
27
|
+
any messages and we can make sure all messages get delivered to the consumer.
|
28
|
+
- **Queue**: a queue listens to an exchange. In most cases the queue will listen
|
29
|
+
to all messages, but it's also possible to listen to a specific pattern.
|
30
|
+
- **Processor**: the specific class that processes a message.
|
11
31
|
|
12
32
|
## Technical documentation
|
13
33
|
|
14
|
-
This is a ruby gem that deals with the boiler plate code of
|
15
|
-
|
16
|
-
|
17
|
-
The user of this gem is left the task of supplying their rabbitmq infrastructure
|
18
|
-
configuration and an instance of a class that processes messages.
|
34
|
+
This is a ruby gem that deals with the boiler plate code of communicating with
|
35
|
+
[RabbitMQ](https://www.rabbitmq.com/). The user of this gem is left the task of
|
36
|
+
supplying the configuration and a class that processes messages.
|
19
37
|
|
20
|
-
|
21
|
-
|
38
|
+
This gem is auto-released with [gem_publisher](https://github.com/alphagov/gem_publisher).
|
39
|
+
To release a new version, simply raise a pull request with the version number
|
40
|
+
incremented.
|
22
41
|
|
23
42
|
### Dependencies
|
24
43
|
|
25
|
-
-
|
26
|
-
|
44
|
+
- The [Bunny](https://github.com/ruby-amqp/bunny) gem: to interact with RabbitMQ.
|
45
|
+
|
46
|
+
## Usage
|
27
47
|
|
48
|
+
For an example on how to implement a message queue consumer, see [alphagov/panopticon#307](https://github.com/alphagov/panopticon/pull/307/files).
|
28
49
|
|
29
|
-
|
50
|
+
Add the gem:
|
51
|
+
|
52
|
+
```ruby
|
53
|
+
# Gemfile
|
54
|
+
gem "govuk_message_queue_consumer", "~> 2.0.0"
|
55
|
+
```
|
30
56
|
|
31
|
-
|
57
|
+
Add a rake task like the following example:
|
32
58
|
|
33
59
|
```ruby
|
60
|
+
# lib/tasks/message_queue.rake
|
34
61
|
namespace :message_queue do
|
35
62
|
desc "Run worker to consume messages from rabbitmq"
|
36
63
|
task consumer: :environment do
|
37
|
-
|
38
|
-
|
39
|
-
|
64
|
+
GovukMessageQueueConsumer::Consumer.new(
|
65
|
+
queue_name: "some-queue",
|
66
|
+
exchange_name: "some-exchange",
|
67
|
+
processor: MyProcessor.new
|
68
|
+
).run
|
40
69
|
end
|
41
70
|
end
|
42
71
|
```
|
43
72
|
|
44
|
-
|
73
|
+
The consumer expects a number of environment variables to be present. On GOV.UK,
|
74
|
+
these should be set up in puppet.
|
75
|
+
|
76
|
+
```
|
77
|
+
RABBITMQ_HOSTS=rabbitmq1.example.com,rabbitmq2.example.com
|
78
|
+
RABBITMQ_VHOST=/
|
79
|
+
RABBITMQ_USER=a_user
|
80
|
+
RABBITMQ_PASSWORD=a_super_secret
|
81
|
+
```
|
82
|
+
|
83
|
+
Define a class that will process the messages:
|
45
84
|
|
46
85
|
```ruby
|
47
|
-
#
|
48
|
-
config = {
|
49
|
-
host: 'localhost',
|
50
|
-
port: 5672,
|
51
|
-
user: rabbitmq_user,
|
52
|
-
pass: rabbitmq_pass,
|
53
|
-
recover_from_connection_close: true,
|
54
|
-
exchange: my_exchange,
|
55
|
-
queue: my_queue,
|
56
|
-
}
|
57
|
-
|
58
|
-
# example message processor
|
86
|
+
# eg. app/queue_consumers/my_processor.rb
|
59
87
|
class MyProcessor
|
60
88
|
def process(message)
|
61
|
-
|
89
|
+
# do something cool
|
62
90
|
end
|
63
91
|
end
|
64
92
|
```
|
65
93
|
|
66
|
-
|
94
|
+
The worker should also be added to the Procfile to run in production:
|
95
|
+
|
96
|
+
```
|
97
|
+
# Procfile
|
98
|
+
worker: bundle exec rake message_queue:consumer
|
99
|
+
```
|
100
|
+
|
101
|
+
Because you need the environment variables when running the consumer, you should use
|
102
|
+
`govuk_setenv` to run your app in development:
|
103
|
+
|
104
|
+
```
|
105
|
+
$ govuk_setenv app-name bundle exec rake message_queue:consumer
|
106
|
+
```
|
107
|
+
|
108
|
+
### Processing a message
|
109
|
+
|
110
|
+
Once you receive a message, you *must* tell RabbitMQ once you've processed it. This
|
111
|
+
is called _acking_. You can also _discard_ the message, or _retry_ it.
|
112
|
+
|
113
|
+
```ruby
|
114
|
+
class MyProcessor
|
115
|
+
def process(message)
|
116
|
+
result = do_something_with(message)
|
117
|
+
|
118
|
+
if result.ok?
|
119
|
+
# Ack the message when it has been processed correctly.
|
120
|
+
message.ack
|
121
|
+
elsif result.failed_temporarily?
|
122
|
+
# Retry the message to make RabbitMQ send the message again later.
|
123
|
+
message.retry
|
124
|
+
elsif result.failed_permanently?
|
125
|
+
# Discard the message when it can't be processed.
|
126
|
+
message.discard
|
127
|
+
end
|
128
|
+
end
|
129
|
+
end
|
130
|
+
```
|
131
|
+
|
132
|
+
### Testing your processor
|
67
133
|
|
68
134
|
This gem provides a test helper for your processor.
|
69
135
|
|
70
136
|
```ruby
|
137
|
+
# eg. spec/queue_consumers/my_processor_spec.rb
|
71
138
|
require 'test_helper'
|
72
139
|
require 'govuk_message_queue_consumer/test_helpers'
|
73
140
|
|
@@ -76,7 +143,28 @@ describe MyProcessor do
|
|
76
143
|
end
|
77
144
|
```
|
78
145
|
|
79
|
-
This will verify that your processor class implements the correct methods. You
|
146
|
+
This will verify that your processor class implements the correct methods. You
|
147
|
+
should add your own tests to verify its behaviour.
|
148
|
+
|
149
|
+
You can use `GovukMessageQueueConsumer::MockMessage` to test the processor
|
150
|
+
behaviour. When using the mock, you can verify it acknowledged, retried or
|
151
|
+
discarded. For example, with `MyProcessor` above:
|
152
|
+
|
153
|
+
```ruby
|
154
|
+
it "acks incoming messages" do
|
155
|
+
message = GovukMessageQueueConsumer::MockMessage.new
|
156
|
+
|
157
|
+
MyProcessor.new.process(message)
|
158
|
+
|
159
|
+
expect(message).to be_acked
|
160
|
+
|
161
|
+
# or if you use minitest:
|
162
|
+
assert message.acked?
|
163
|
+
end
|
164
|
+
```
|
165
|
+
|
166
|
+
For more test cases [see the spec for the mock itself](/spec/mock_message_spec.rb).
|
167
|
+
|
80
168
|
|
81
169
|
### Running the test suite
|
82
170
|
|
@@ -84,12 +172,19 @@ This will verify that your processor class implements the correct methods. You s
|
|
84
172
|
bundle exec rake spec
|
85
173
|
```
|
86
174
|
|
175
|
+
## Further reading
|
176
|
+
|
177
|
+
- [Bunny](https://github.com/ruby-amqp/bunny) is the RabbitMQ client we use.
|
178
|
+
- [The Bunny Guides](http://rubybunny.info/articles/guides.html) explain all
|
179
|
+
AMQP concepts really well.
|
180
|
+
- The [Opsmanual](https://github.gds/pages/gds/opsmanual/2nd-line/nagios.html?highlight=rabbitmq#rabbitmq-checks)
|
181
|
+
documents the usage of "heartbeat" messages, which this gem also supports.
|
87
182
|
|
88
183
|
## Licence
|
89
184
|
|
90
185
|
[MIT License](LICENCE)
|
91
186
|
|
92
|
-
|
93
187
|
## Versioning policy
|
94
188
|
|
95
|
-
[Semantic versioning](http://semver.org/spec/v2.0.0.html)
|
189
|
+
We follow [Semantic versioning](http://semver.org/spec/v2.0.0.html). Check the
|
190
|
+
[CHANGELOG](CHANGELOG.md) for changes.
|
@@ -1,22 +1,28 @@
|
|
1
|
-
require 'active_support/core_ext/hash/indifferent_access'
|
2
1
|
require 'bunny'
|
3
2
|
|
4
3
|
module GovukMessageQueueConsumer
|
5
4
|
class Consumer
|
6
|
-
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
5
|
+
# Only fetch one message at a time on the channel.
|
6
|
+
#
|
7
|
+
# By default, queues will grab messages eagerly, which reduces latency.
|
8
|
+
# However, that also means that if multiple workers are running one worker
|
9
|
+
# can starve another of work. We're not expecting a high throughput on this
|
10
|
+
# queue, and a small bit of latency isn't a problem, so we fetch one at a
|
11
|
+
# time to share the work evenly.
|
12
|
+
NUMBER_OF_MESSAGES_TO_PREFETCH = 1
|
13
|
+
|
14
|
+
def initialize(queue_name:, exchange_name:, processor:, routing_key: '#')
|
15
|
+
@queue_name = queue_name
|
16
|
+
@exchange_name = exchange_name
|
17
|
+
@processor = processor
|
18
|
+
@routing_key = routing_key
|
14
19
|
end
|
15
20
|
|
16
21
|
def run
|
17
|
-
queue.subscribe(:
|
22
|
+
queue.subscribe(block: true, manual_ack: true) do |delivery_info, headers, payload|
|
18
23
|
begin
|
19
|
-
|
24
|
+
message = Message.new(payload, headers, delivery_info)
|
25
|
+
processor_chain.process(message)
|
20
26
|
rescue Exception => e
|
21
27
|
$stderr.puts "rabbitmq_consumer: aborting due to unhandled exception in processor #{e.class}: #{e.message}"
|
22
28
|
exit(1) # ensure rabbitmq requeues outstanding messages
|
@@ -24,21 +30,35 @@ module GovukMessageQueueConsumer
|
|
24
30
|
end
|
25
31
|
end
|
26
32
|
|
27
|
-
|
33
|
+
private
|
34
|
+
|
35
|
+
def processor_chain
|
36
|
+
@processor_chain ||= HeartbeatProcessor.new(JSONProcessor.new(@processor))
|
37
|
+
end
|
28
38
|
|
29
39
|
def queue
|
30
|
-
@queue ||=
|
40
|
+
@queue ||= begin
|
41
|
+
channel.prefetch(NUMBER_OF_MESSAGES_TO_PREFETCH)
|
42
|
+
queue = channel.queue(@queue_name, durable: true)
|
43
|
+
queue.bind(exchange, routing_key: @routing_key)
|
44
|
+
queue
|
45
|
+
end
|
46
|
+
end
|
47
|
+
|
48
|
+
def exchange
|
49
|
+
@exchange ||= channel.topic(@exchange_name, passive: true)
|
50
|
+
end
|
51
|
+
|
52
|
+
def channel
|
53
|
+
@channel ||= connection.create_channel
|
31
54
|
end
|
32
55
|
|
33
|
-
def
|
34
|
-
@
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
exchange = @channel.topic(exchange_name, :passive => true)
|
39
|
-
queue.bind(exchange, :routing_key => routing_key)
|
56
|
+
def connection
|
57
|
+
@connection ||= begin
|
58
|
+
new_connection = Bunny.new(RabbitMQConfig.new.from_environment)
|
59
|
+
new_connection.start
|
60
|
+
new_connection
|
40
61
|
end
|
41
|
-
queue
|
42
62
|
end
|
43
63
|
end
|
44
64
|
end
|
@@ -0,0 +1,20 @@
|
|
1
|
+
module GovukMessageQueueConsumer
|
2
|
+
class JSONProcessor
|
3
|
+
JSON_FORMAT = "application/json".freeze
|
4
|
+
|
5
|
+
def initialize(next_processor)
|
6
|
+
@next_processor = next_processor
|
7
|
+
end
|
8
|
+
|
9
|
+
def process(message)
|
10
|
+
if message.headers.content_type == JSON_FORMAT
|
11
|
+
message.payload = JSON.parse(message.payload)
|
12
|
+
end
|
13
|
+
|
14
|
+
@next_processor.process(message)
|
15
|
+
rescue JSON::ParserError => e
|
16
|
+
Airbrake.notify_or_ignore(e) if defined?(Airbrake)
|
17
|
+
message.discard
|
18
|
+
end
|
19
|
+
end
|
20
|
+
end
|
@@ -3,16 +3,12 @@ require 'json'
|
|
3
3
|
module GovukMessageQueueConsumer
|
4
4
|
# Client code will receive an instance of this
|
5
5
|
class Message
|
6
|
-
|
7
|
-
@delivery_info = delivery_info
|
8
|
-
@headers = headers
|
9
|
-
@body = payload
|
10
|
-
end
|
6
|
+
attr_accessor :delivery_info, :headers, :payload
|
11
7
|
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
@
|
8
|
+
def initialize(payload, headers, delivery_info)
|
9
|
+
@payload = payload
|
10
|
+
@headers = headers
|
11
|
+
@delivery_info = delivery_info
|
16
12
|
end
|
17
13
|
|
18
14
|
def ack
|
@@ -0,0 +1,29 @@
|
|
1
|
+
class RabbitMQConfig
|
2
|
+
class ConfigurationError < StandardError
|
3
|
+
end
|
4
|
+
|
5
|
+
def from_environment
|
6
|
+
{
|
7
|
+
hosts: fetch("RABBITMQ_HOSTS").split(','),
|
8
|
+
vhost: fetch("RABBITMQ_VHOST"),
|
9
|
+
user: fetch("RABBITMQ_USER"),
|
10
|
+
pass: fetch("RABBITMQ_PASSWORD"),
|
11
|
+
recover_from_connection_close: true,
|
12
|
+
}
|
13
|
+
end
|
14
|
+
|
15
|
+
private
|
16
|
+
|
17
|
+
def fetch(variable_name)
|
18
|
+
ENV[variable_name] || raise_error(variable_name)
|
19
|
+
end
|
20
|
+
|
21
|
+
def raise_error(variable_name)
|
22
|
+
raise ConfigurationError, <<-err
|
23
|
+
The environment variable #{variable_name} is not set. If you are in test
|
24
|
+
mode, make sure you set the correct vars in your helpers. If you get this
|
25
|
+
error in development, make sure you run rails or rake with `govuk_setenv`
|
26
|
+
and puppet is up to date.
|
27
|
+
err
|
28
|
+
end
|
29
|
+
end
|
@@ -0,0 +1,27 @@
|
|
1
|
+
module GovukMessageQueueConsumer
|
2
|
+
class MockMessage < Message
|
3
|
+
attr_reader :acked, :retried, :discarded
|
4
|
+
|
5
|
+
alias :acked? :acked
|
6
|
+
alias :discarded? :discarded
|
7
|
+
alias :retried? :retried
|
8
|
+
|
9
|
+
def initialize(payload = {}, headers = {}, delivery_info = {})
|
10
|
+
@payload = payload
|
11
|
+
@headers = OpenStruct.new(headers)
|
12
|
+
@delivery_info = OpenStruct.new(delivery_info)
|
13
|
+
end
|
14
|
+
|
15
|
+
def ack
|
16
|
+
@acked = true
|
17
|
+
end
|
18
|
+
|
19
|
+
def retry
|
20
|
+
@retried = true
|
21
|
+
end
|
22
|
+
|
23
|
+
def discard
|
24
|
+
@discarded = true
|
25
|
+
end
|
26
|
+
end
|
27
|
+
end
|
@@ -1,9 +1,11 @@
|
|
1
|
-
RSpec
|
2
|
-
|
3
|
-
|
4
|
-
|
1
|
+
if defined?(RSpec)
|
2
|
+
RSpec.shared_examples "a message queue processor" do
|
3
|
+
it "implements #process" do
|
4
|
+
expect(subject).to respond_to(:process)
|
5
|
+
end
|
5
6
|
|
6
|
-
|
7
|
-
|
7
|
+
it "accepts 1 argument for #process" do
|
8
|
+
expect(subject.method(:process).arity).to eq(1)
|
9
|
+
end
|
8
10
|
end
|
9
11
|
end
|
@@ -1,4 +1,6 @@
|
|
1
1
|
require 'govuk_message_queue_consumer/version'
|
2
2
|
require 'govuk_message_queue_consumer/heartbeat_processor'
|
3
|
+
require 'govuk_message_queue_consumer/json_processor'
|
3
4
|
require 'govuk_message_queue_consumer/message'
|
4
5
|
require 'govuk_message_queue_consumer/consumer'
|
6
|
+
require 'govuk_message_queue_consumer/rabbitmq_config'
|
data/spec/consumer_spec.rb
CHANGED
@@ -1,48 +1,35 @@
|
|
1
1
|
require_relative 'spec_helper'
|
2
2
|
|
3
3
|
describe Consumer do
|
4
|
-
|
5
|
-
let(:queue) { instance_double('Bunny::Queue', bind: nil, subscribe: message_values) }
|
4
|
+
let(:queue) { instance_double('Bunny::Queue', bind: nil, subscribe: '') }
|
6
5
|
let(:channel) { instance_double('Bunny::Channel', queue: queue, prefetch: nil, topic: nil) }
|
7
6
|
let(:rabbitmq_connecton) { instance_double("Bunny::Session", start: nil, create_channel: channel) }
|
8
7
|
let(:client_processor) { instance_double('Client::Processor') }
|
9
8
|
|
10
9
|
before do
|
10
|
+
stub_environment_variables!
|
11
11
|
allow(Bunny).to receive(:new).and_return(rabbitmq_connecton)
|
12
12
|
end
|
13
13
|
|
14
|
-
describe "
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
it "passes the client processor to the Heartbeat Processor" do
|
19
|
-
expect(HeartbeatProcessor).to receive(:new).with(client_processor)
|
20
|
-
|
21
|
-
Consumer.new(rabbitmq_config, client_processor)
|
22
|
-
end
|
23
|
-
|
24
|
-
it "connects to rabbitmq" do
|
25
|
-
expected_options = rabbitmq_config['connection'].symbolize_keys # Bunny requires the keys to be symbols
|
26
|
-
expect(Bunny).to receive(:new).with(expected_options).and_return(rabbitmq_connecton)
|
27
|
-
expect(rabbitmq_connecton).to receive(:start)
|
14
|
+
describe "running the consumer" do
|
15
|
+
it "binds the queue to the all-routing key" do
|
16
|
+
expect(queue).to receive(:bind).with(nil, { routing_key: "#" })
|
28
17
|
|
29
|
-
Consumer.new(
|
18
|
+
Consumer.new(queue_name: "some-queue", exchange_name: "my-exchange", processor: client_processor).run
|
30
19
|
end
|
31
|
-
end
|
32
20
|
|
33
|
-
|
34
|
-
|
35
|
-
expect(queue).to receive(:bind)
|
21
|
+
it "binds the queue to a custom routing key" do
|
22
|
+
expect(queue).to receive(:bind).with(nil, { routing_key: "*.major" })
|
36
23
|
|
37
|
-
Consumer.new(
|
24
|
+
Consumer.new(queue_name: "some-queue", exchange_name: "my-exchange", processor: client_processor, routing_key: "*.major").run
|
38
25
|
end
|
39
26
|
|
40
27
|
it "calls the heartbeat processor when subscribing to messages" do
|
41
|
-
expect(queue).to receive(:subscribe).and_yield(
|
42
|
-
expect(Message).to receive(:new).with(
|
28
|
+
expect(queue).to receive(:subscribe).and_yield(:delivery_info_object, :headers, "payload")
|
29
|
+
expect(Message).to receive(:new).with("payload", :headers, :delivery_info_object)
|
43
30
|
expect_any_instance_of(HeartbeatProcessor).to receive(:process)
|
44
31
|
|
45
|
-
Consumer.new(
|
32
|
+
Consumer.new(queue_name: "some-queue", exchange_name: "my-exchange", processor: client_processor).run
|
46
33
|
end
|
47
34
|
end
|
48
35
|
end
|
@@ -0,0 +1,33 @@
|
|
1
|
+
require_relative 'spec_helper'
|
2
|
+
|
3
|
+
describe JSONProcessor do
|
4
|
+
describe "#process" do
|
5
|
+
it "parses the payload string" do
|
6
|
+
next_processor = double("next_processor", process: "ha")
|
7
|
+
message = MockMessage.new('{"some":"json"}', { content_type: "application/json" })
|
8
|
+
|
9
|
+
JSONProcessor.new(next_processor).process(message)
|
10
|
+
|
11
|
+
expect(message.payload).to eql({ "some" => "json" })
|
12
|
+
expect(next_processor).to have_received(:process)
|
13
|
+
end
|
14
|
+
|
15
|
+
it "discards messages with JSON errors" do
|
16
|
+
message = MockMessage.new('{"some" "json"}', { content_type: "application/json" })
|
17
|
+
|
18
|
+
JSONProcessor.new(double).process(message)
|
19
|
+
|
20
|
+
expect(message).to be_discarded
|
21
|
+
end
|
22
|
+
|
23
|
+
it "doesn't parse non-JSON message" do
|
24
|
+
next_processor = double("next_processor", process: "ha")
|
25
|
+
message = MockMessage.new('<SomeXML></SomeXML>', { content_type: "application/xml" })
|
26
|
+
|
27
|
+
JSONProcessor.new(next_processor).process(message)
|
28
|
+
|
29
|
+
expect(message.payload).to eql('<SomeXML></SomeXML>')
|
30
|
+
expect(next_processor).to have_received(:process)
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
data/spec/message_spec.rb
CHANGED
@@ -4,25 +4,23 @@ describe Message do
|
|
4
4
|
let(:mock_channel) { instance_double("Channel") }
|
5
5
|
let(:delivery_info) { instance_double("DeliveryInfo", :channel => mock_channel, :delivery_tag => "a_tag") }
|
6
6
|
let(:headers) { instance_double("Headers") }
|
7
|
-
let(:
|
8
|
-
let(:message) { Message.new(delivery_info, headers, body) }
|
9
|
-
|
10
|
-
it "json decodes the body" do
|
11
|
-
expect(message.body_data).to eq("foo" => "bar")
|
12
|
-
end
|
7
|
+
let(:message) { Message.new({ "a" => "payload" }, headers, delivery_info) }
|
13
8
|
|
14
9
|
it "ack sends an ack to the channel" do
|
15
10
|
expect(mock_channel).to receive(:ack).with("a_tag")
|
11
|
+
|
16
12
|
message.ack
|
17
13
|
end
|
18
14
|
|
19
15
|
it "retry sends a reject to the channel with requeue set" do
|
20
16
|
expect(mock_channel).to receive(:reject).with("a_tag", true)
|
17
|
+
|
21
18
|
message.retry
|
22
19
|
end
|
23
20
|
|
24
21
|
it "reject sends a reject to the channel without requeue set" do
|
25
22
|
expect(mock_channel).to receive(:reject).with("a_tag", false)
|
23
|
+
|
26
24
|
message.discard
|
27
25
|
end
|
28
26
|
end
|
@@ -0,0 +1,43 @@
|
|
1
|
+
require_relative 'spec_helper'
|
2
|
+
require_relative '../lib/govuk_message_queue_consumer/test_helpers/mock_message'
|
3
|
+
|
4
|
+
describe GovukMessageQueueConsumer::MockMessage do
|
5
|
+
describe '#methods' do
|
6
|
+
it "implements the same methods as Message" do
|
7
|
+
mock = MockMessage.new
|
8
|
+
real = Message.new(double, double, double)
|
9
|
+
|
10
|
+
expect(real.methods - mock.methods).to be_empty
|
11
|
+
end
|
12
|
+
end
|
13
|
+
|
14
|
+
describe '#ack' do
|
15
|
+
it "marks the message as acked" do
|
16
|
+
message = MockMessage.new
|
17
|
+
|
18
|
+
message.ack
|
19
|
+
|
20
|
+
expect(message).to be_acked
|
21
|
+
end
|
22
|
+
end
|
23
|
+
|
24
|
+
describe '#retry' do
|
25
|
+
it "marks the message as retried" do
|
26
|
+
message = MockMessage.new
|
27
|
+
|
28
|
+
message.retry
|
29
|
+
|
30
|
+
expect(message).to be_retried
|
31
|
+
end
|
32
|
+
end
|
33
|
+
|
34
|
+
describe '#discard' do
|
35
|
+
it "marks the message as discarded" do
|
36
|
+
message = MockMessage.new
|
37
|
+
|
38
|
+
message.discard
|
39
|
+
|
40
|
+
expect(message).to be_discarded
|
41
|
+
end
|
42
|
+
end
|
43
|
+
end
|
@@ -0,0 +1,26 @@
|
|
1
|
+
require 'spec_helper'
|
2
|
+
|
3
|
+
RSpec.describe RabbitMQConfig do
|
4
|
+
describe ".from_environment" do
|
5
|
+
it "connects to rabbitmq with the correct environment variables" do
|
6
|
+
ENV["RABBITMQ_HOSTS"] = "server-one,server-two"
|
7
|
+
ENV["RABBITMQ_VHOST"] = "/"
|
8
|
+
ENV["RABBITMQ_USER"] = "my_user"
|
9
|
+
ENV["RABBITMQ_PASSWORD"] = "my_pass"
|
10
|
+
|
11
|
+
expect(RabbitMQConfig.new.from_environment).to eql({
|
12
|
+
hosts: ["server-one", "server-two"],
|
13
|
+
vhost: "/",
|
14
|
+
user: "my_user",
|
15
|
+
pass: "my_pass",
|
16
|
+
recover_from_connection_close: true,
|
17
|
+
})
|
18
|
+
end
|
19
|
+
|
20
|
+
it "provides a friendly error message when a variable is missing" do
|
21
|
+
ENV["RABBITMQ_HOSTS"] = nil
|
22
|
+
|
23
|
+
expect { RabbitMQConfig.new.from_environment }.to raise_error(RabbitMQConfig::ConfigurationError)
|
24
|
+
end
|
25
|
+
end
|
26
|
+
end
|
data/spec/spec_helper.rb
CHANGED
@@ -3,23 +3,11 @@ require_relative '../lib/govuk_message_queue_consumer'
|
|
3
3
|
include GovukMessageQueueConsumer
|
4
4
|
|
5
5
|
module TestHelpers
|
6
|
-
def
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
"vhost" => "/",
|
12
|
-
"user" => "a_user",
|
13
|
-
"pass" => "super secret",
|
14
|
-
"recover_from_connection_close" => true,
|
15
|
-
},
|
16
|
-
"queue" => "content_register",
|
17
|
-
"exchange" => "published_documents",
|
18
|
-
}
|
19
|
-
end
|
20
|
-
|
21
|
-
def message_values
|
22
|
-
[:delivery_info1, :headers1, "message1_body"]
|
6
|
+
def stub_environment_variables!
|
7
|
+
ENV["RABBITMQ_HOSTS"] ||= ""
|
8
|
+
ENV["RABBITMQ_VHOST"] ||= "/"
|
9
|
+
ENV["RABBITMQ_USER"] ||= "/"
|
10
|
+
ENV["RABBITMQ_PASSWORD"] ||= "/"
|
23
11
|
end
|
24
12
|
end
|
25
13
|
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: govuk_message_queue_consumer
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 2.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- GOV.UK Dev
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2015-
|
11
|
+
date: 2015-11-03 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bunny
|
@@ -24,20 +24,6 @@ dependencies:
|
|
24
24
|
- - "~>"
|
25
25
|
- !ruby/object:Gem::Version
|
26
26
|
version: 2.2.0
|
27
|
-
- !ruby/object:Gem::Dependency
|
28
|
-
name: activesupport
|
29
|
-
requirement: !ruby/object:Gem::Requirement
|
30
|
-
requirements:
|
31
|
-
- - "~>"
|
32
|
-
- !ruby/object:Gem::Version
|
33
|
-
version: '4.0'
|
34
|
-
type: :runtime
|
35
|
-
prerelease: false
|
36
|
-
version_requirements: !ruby/object:Gem::Requirement
|
37
|
-
requirements:
|
38
|
-
- - "~>"
|
39
|
-
- !ruby/object:Gem::Version
|
40
|
-
version: '4.0'
|
41
27
|
- !ruby/object:Gem::Dependency
|
42
28
|
name: gem_publisher
|
43
29
|
requirement: !ruby/object:Gem::Requirement
|
@@ -93,13 +79,20 @@ files:
|
|
93
79
|
- lib/govuk_message_queue_consumer.rb
|
94
80
|
- lib/govuk_message_queue_consumer/consumer.rb
|
95
81
|
- lib/govuk_message_queue_consumer/heartbeat_processor.rb
|
82
|
+
- lib/govuk_message_queue_consumer/json_processor.rb
|
96
83
|
- lib/govuk_message_queue_consumer/message.rb
|
84
|
+
- lib/govuk_message_queue_consumer/rabbitmq_config.rb
|
97
85
|
- lib/govuk_message_queue_consumer/test_helpers.rb
|
86
|
+
- lib/govuk_message_queue_consumer/test_helpers/mock_message.rb
|
98
87
|
- lib/govuk_message_queue_consumer/test_helpers/shared_examples.rb
|
88
|
+
- lib/govuk_message_queue_consumer/test_helpers/test_consumer.rb
|
99
89
|
- lib/govuk_message_queue_consumer/version.rb
|
100
90
|
- spec/consumer_spec.rb
|
101
91
|
- spec/heartbeat_processor_spec.rb
|
92
|
+
- spec/json_processor_spec.rb
|
102
93
|
- spec/message_spec.rb
|
94
|
+
- spec/mock_message_spec.rb
|
95
|
+
- spec/rabbitmq_config_spec.rb
|
103
96
|
- spec/spec_helper.rb
|
104
97
|
homepage: https://github.com/alphagov/govuk_message_queue_consumer
|
105
98
|
licenses: []
|
@@ -129,3 +122,6 @@ test_files:
|
|
129
122
|
- spec/spec_helper.rb
|
130
123
|
- spec/message_spec.rb
|
131
124
|
- spec/consumer_spec.rb
|
125
|
+
- spec/mock_message_spec.rb
|
126
|
+
- spec/json_processor_spec.rb
|
127
|
+
- spec/rabbitmq_config_spec.rb
|