govuk_message_queue_consumer 1.0.0 → 2.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +132 -37
- data/lib/govuk_message_queue_consumer/consumer.rb +41 -21
- data/lib/govuk_message_queue_consumer/json_processor.rb +20 -0
- data/lib/govuk_message_queue_consumer/message.rb +5 -9
- data/lib/govuk_message_queue_consumer/rabbitmq_config.rb +29 -0
- data/lib/govuk_message_queue_consumer/test_helpers/mock_message.rb +27 -0
- data/lib/govuk_message_queue_consumer/test_helpers/shared_examples.rb +8 -6
- data/lib/govuk_message_queue_consumer/test_helpers/test_consumer.rb +12 -0
- data/lib/govuk_message_queue_consumer/test_helpers.rb +2 -0
- data/lib/govuk_message_queue_consumer/version.rb +1 -1
- data/lib/govuk_message_queue_consumer.rb +2 -0
- data/spec/consumer_spec.rb +12 -25
- data/spec/json_processor_spec.rb +33 -0
- data/spec/message_spec.rb +4 -6
- data/spec/mock_message_spec.rb +43 -0
- data/spec/rabbitmq_config_spec.rb +26 -0
- data/spec/spec_helper.rb +5 -17
- metadata +12 -16
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: f3d0a2e8f69b4d555db92e1251f26b8238f1ad63
|
4
|
+
data.tar.gz: c97edff469050cfbf09031625ff3e22f2c573bee
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 0ad7430c3ed9f12498a91b90c30050737809f0e2c0234e0892a417d7813004c306d68c95992c04def1fa23b987bc6f5652671472685a8a95dcc257f5ec5c15be
|
7
|
+
data.tar.gz: b719dadfe75138257d573f91ea758ebf6878ac076971a76dd49012b8de6dd2a35c9c68b559609e1a44148c6ced539c9d8d510c7ad59c95e38241a1903eebd0c6
|
data/README.md
CHANGED
@@ -1,73 +1,140 @@
|
|
1
|
-
# GOV.UK Message
|
1
|
+
# GOV.UK Message Queue Consumer
|
2
2
|
|
3
|
-
|
3
|
+
Standardises the way GOV.UK consumes messages from [RabbitMQ](https://www.rabbitmq.com/).
|
4
|
+
RabbitMQ is a messaging framework that allows applications to broadcast messages
|
5
|
+
that can be picked up by other applications.
|
4
6
|
|
7
|
+
On GOV.UK, [publishing-api](https://github.com/alphagov/publishing-api) publishes
|
8
|
+
the content-items it receives, so that applications such as
|
9
|
+
[email-alert-service](https://github.com/alphagov/email-alert-service) can be
|
10
|
+
notified of changes in content.
|
5
11
|
|
6
12
|
## Nomenclature
|
7
13
|
|
8
|
-
|
9
|
-
|
10
|
-
|
14
|
+
![A graph showing the message flow](docs/graph.png)
|
15
|
+
|
16
|
+
- **Producer**: an application that sends messages RabbitMQ. On GOV.UK this could
|
17
|
+
be [publishing-api](https://github.com/alphagov/publishing-api).
|
18
|
+
- **Message**: an object sent over RabbitMQ. It consists of a _payload_ and
|
19
|
+
_headers_. In the case of the publishing-api the payload is a
|
20
|
+
[content item](https://github.com/alphagov/govuk-content-schemas).
|
21
|
+
- **Consumer**: the app that receives the messages and does something with them.
|
22
|
+
On GOV.UK, these could be [email-alert-service](https://github.com/alphagov/email-alert-service)
|
23
|
+
and [content-register](https://github.com/alphagov/content-register).
|
24
|
+
- **Exchange**: in RabbitMQ's model, producers send messages to an _exchange_.
|
25
|
+
Consumers can create a Queue that listens to the exchange, instead of
|
26
|
+
subscribing to the exchange directly. This is done so that the queue can buffer
|
27
|
+
any messages and we can make sure all messages get delivered to the consumer.
|
28
|
+
- **Queue**: a queue listens to an exchange. In most cases the queue will listen
|
29
|
+
to all messages, but it's also possible to listen to a specific pattern.
|
30
|
+
- **Processor**: the specific class that processes a message.
|
11
31
|
|
12
32
|
## Technical documentation
|
13
33
|
|
14
|
-
This is a ruby gem that deals with the boiler plate code of
|
15
|
-
|
16
|
-
|
17
|
-
The user of this gem is left the task of supplying their rabbitmq infrastructure
|
18
|
-
configuration and an instance of a class that processes messages.
|
34
|
+
This is a ruby gem that deals with the boiler plate code of communicating with
|
35
|
+
[RabbitMQ](https://www.rabbitmq.com/). The user of this gem is left the task of
|
36
|
+
supplying the configuration and a class that processes messages.
|
19
37
|
|
20
|
-
|
21
|
-
|
38
|
+
This gem is auto-released with [gem_publisher](https://github.com/alphagov/gem_publisher).
|
39
|
+
To release a new version, simply raise a pull request with the version number
|
40
|
+
incremented.
|
22
41
|
|
23
42
|
### Dependencies
|
24
43
|
|
25
|
-
-
|
26
|
-
|
44
|
+
- The [Bunny](https://github.com/ruby-amqp/bunny) gem: to interact with RabbitMQ.
|
45
|
+
|
46
|
+
## Usage
|
27
47
|
|
48
|
+
For an example on how to implement a message queue consumer, see [alphagov/panopticon#307](https://github.com/alphagov/panopticon/pull/307/files).
|
28
49
|
|
29
|
-
|
50
|
+
Add the gem:
|
51
|
+
|
52
|
+
```ruby
|
53
|
+
# Gemfile
|
54
|
+
gem "govuk_message_queue_consumer", "~> 2.0.0"
|
55
|
+
```
|
30
56
|
|
31
|
-
|
57
|
+
Add a rake task like the following example:
|
32
58
|
|
33
59
|
```ruby
|
60
|
+
# lib/tasks/message_queue.rake
|
34
61
|
namespace :message_queue do
|
35
62
|
desc "Run worker to consume messages from rabbitmq"
|
36
63
|
task consumer: :environment do
|
37
|
-
|
38
|
-
|
39
|
-
|
64
|
+
GovukMessageQueueConsumer::Consumer.new(
|
65
|
+
queue_name: "some-queue",
|
66
|
+
exchange_name: "some-exchange",
|
67
|
+
processor: MyProcessor.new
|
68
|
+
).run
|
40
69
|
end
|
41
70
|
end
|
42
71
|
```
|
43
72
|
|
44
|
-
|
73
|
+
The consumer expects a number of environment variables to be present. On GOV.UK,
|
74
|
+
these should be set up in puppet.
|
75
|
+
|
76
|
+
```
|
77
|
+
RABBITMQ_HOSTS=rabbitmq1.example.com,rabbitmq2.example.com
|
78
|
+
RABBITMQ_VHOST=/
|
79
|
+
RABBITMQ_USER=a_user
|
80
|
+
RABBITMQ_PASSWORD=a_super_secret
|
81
|
+
```
|
82
|
+
|
83
|
+
Define a class that will process the messages:
|
45
84
|
|
46
85
|
```ruby
|
47
|
-
#
|
48
|
-
config = {
|
49
|
-
host: 'localhost',
|
50
|
-
port: 5672,
|
51
|
-
user: rabbitmq_user,
|
52
|
-
pass: rabbitmq_pass,
|
53
|
-
recover_from_connection_close: true,
|
54
|
-
exchange: my_exchange,
|
55
|
-
queue: my_queue,
|
56
|
-
}
|
57
|
-
|
58
|
-
# example message processor
|
86
|
+
# eg. app/queue_consumers/my_processor.rb
|
59
87
|
class MyProcessor
|
60
88
|
def process(message)
|
61
|
-
|
89
|
+
# do something cool
|
62
90
|
end
|
63
91
|
end
|
64
92
|
```
|
65
93
|
|
66
|
-
|
94
|
+
The worker should also be added to the Procfile to run in production:
|
95
|
+
|
96
|
+
```
|
97
|
+
# Procfile
|
98
|
+
worker: bundle exec rake message_queue:consumer
|
99
|
+
```
|
100
|
+
|
101
|
+
Because you need the environment variables when running the consumer, you should use
|
102
|
+
`govuk_setenv` to run your app in development:
|
103
|
+
|
104
|
+
```
|
105
|
+
$ govuk_setenv app-name bundle exec rake message_queue:consumer
|
106
|
+
```
|
107
|
+
|
108
|
+
### Processing a message
|
109
|
+
|
110
|
+
Once you receive a message, you *must* tell RabbitMQ once you've processed it. This
|
111
|
+
is called _acking_. You can also _discard_ the message, or _retry_ it.
|
112
|
+
|
113
|
+
```ruby
|
114
|
+
class MyProcessor
|
115
|
+
def process(message)
|
116
|
+
result = do_something_with(message)
|
117
|
+
|
118
|
+
if result.ok?
|
119
|
+
# Ack the message when it has been processed correctly.
|
120
|
+
message.ack
|
121
|
+
elsif result.failed_temporarily?
|
122
|
+
# Retry the message to make RabbitMQ send the message again later.
|
123
|
+
message.retry
|
124
|
+
elsif result.failed_permanently?
|
125
|
+
# Discard the message when it can't be processed.
|
126
|
+
message.discard
|
127
|
+
end
|
128
|
+
end
|
129
|
+
end
|
130
|
+
```
|
131
|
+
|
132
|
+
### Testing your processor
|
67
133
|
|
68
134
|
This gem provides a test helper for your processor.
|
69
135
|
|
70
136
|
```ruby
|
137
|
+
# eg. spec/queue_consumers/my_processor_spec.rb
|
71
138
|
require 'test_helper'
|
72
139
|
require 'govuk_message_queue_consumer/test_helpers'
|
73
140
|
|
@@ -76,7 +143,28 @@ describe MyProcessor do
|
|
76
143
|
end
|
77
144
|
```
|
78
145
|
|
79
|
-
This will verify that your processor class implements the correct methods. You
|
146
|
+
This will verify that your processor class implements the correct methods. You
|
147
|
+
should add your own tests to verify its behaviour.
|
148
|
+
|
149
|
+
You can use `GovukMessageQueueConsumer::MockMessage` to test the processor
|
150
|
+
behaviour. When using the mock, you can verify it acknowledged, retried or
|
151
|
+
discarded. For example, with `MyProcessor` above:
|
152
|
+
|
153
|
+
```ruby
|
154
|
+
it "acks incoming messages" do
|
155
|
+
message = GovukMessageQueueConsumer::MockMessage.new
|
156
|
+
|
157
|
+
MyProcessor.new.process(message)
|
158
|
+
|
159
|
+
expect(message).to be_acked
|
160
|
+
|
161
|
+
# or if you use minitest:
|
162
|
+
assert message.acked?
|
163
|
+
end
|
164
|
+
```
|
165
|
+
|
166
|
+
For more test cases [see the spec for the mock itself](/spec/mock_message_spec.rb).
|
167
|
+
|
80
168
|
|
81
169
|
### Running the test suite
|
82
170
|
|
@@ -84,12 +172,19 @@ This will verify that your processor class implements the correct methods. You s
|
|
84
172
|
bundle exec rake spec
|
85
173
|
```
|
86
174
|
|
175
|
+
## Further reading
|
176
|
+
|
177
|
+
- [Bunny](https://github.com/ruby-amqp/bunny) is the RabbitMQ client we use.
|
178
|
+
- [The Bunny Guides](http://rubybunny.info/articles/guides.html) explain all
|
179
|
+
AMQP concepts really well.
|
180
|
+
- The [Opsmanual](https://github.gds/pages/gds/opsmanual/2nd-line/nagios.html?highlight=rabbitmq#rabbitmq-checks)
|
181
|
+
documents the usage of "heartbeat" messages, which this gem also supports.
|
87
182
|
|
88
183
|
## Licence
|
89
184
|
|
90
185
|
[MIT License](LICENCE)
|
91
186
|
|
92
|
-
|
93
187
|
## Versioning policy
|
94
188
|
|
95
|
-
[Semantic versioning](http://semver.org/spec/v2.0.0.html)
|
189
|
+
We follow [Semantic versioning](http://semver.org/spec/v2.0.0.html). Check the
|
190
|
+
[CHANGELOG](CHANGELOG.md) for changes.
|
@@ -1,22 +1,28 @@
|
|
1
|
-
require 'active_support/core_ext/hash/indifferent_access'
|
2
1
|
require 'bunny'
|
3
2
|
|
4
3
|
module GovukMessageQueueConsumer
|
5
4
|
class Consumer
|
6
|
-
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
5
|
+
# Only fetch one message at a time on the channel.
|
6
|
+
#
|
7
|
+
# By default, queues will grab messages eagerly, which reduces latency.
|
8
|
+
# However, that also means that if multiple workers are running one worker
|
9
|
+
# can starve another of work. We're not expecting a high throughput on this
|
10
|
+
# queue, and a small bit of latency isn't a problem, so we fetch one at a
|
11
|
+
# time to share the work evenly.
|
12
|
+
NUMBER_OF_MESSAGES_TO_PREFETCH = 1
|
13
|
+
|
14
|
+
def initialize(queue_name:, exchange_name:, processor:, routing_key: '#')
|
15
|
+
@queue_name = queue_name
|
16
|
+
@exchange_name = exchange_name
|
17
|
+
@processor = processor
|
18
|
+
@routing_key = routing_key
|
14
19
|
end
|
15
20
|
|
16
21
|
def run
|
17
|
-
queue.subscribe(:
|
22
|
+
queue.subscribe(block: true, manual_ack: true) do |delivery_info, headers, payload|
|
18
23
|
begin
|
19
|
-
|
24
|
+
message = Message.new(payload, headers, delivery_info)
|
25
|
+
processor_chain.process(message)
|
20
26
|
rescue Exception => e
|
21
27
|
$stderr.puts "rabbitmq_consumer: aborting due to unhandled exception in processor #{e.class}: #{e.message}"
|
22
28
|
exit(1) # ensure rabbitmq requeues outstanding messages
|
@@ -24,21 +30,35 @@ module GovukMessageQueueConsumer
|
|
24
30
|
end
|
25
31
|
end
|
26
32
|
|
27
|
-
|
33
|
+
private
|
34
|
+
|
35
|
+
def processor_chain
|
36
|
+
@processor_chain ||= HeartbeatProcessor.new(JSONProcessor.new(@processor))
|
37
|
+
end
|
28
38
|
|
29
39
|
def queue
|
30
|
-
@queue ||=
|
40
|
+
@queue ||= begin
|
41
|
+
channel.prefetch(NUMBER_OF_MESSAGES_TO_PREFETCH)
|
42
|
+
queue = channel.queue(@queue_name, durable: true)
|
43
|
+
queue.bind(exchange, routing_key: @routing_key)
|
44
|
+
queue
|
45
|
+
end
|
46
|
+
end
|
47
|
+
|
48
|
+
def exchange
|
49
|
+
@exchange ||= channel.topic(@exchange_name, passive: true)
|
50
|
+
end
|
51
|
+
|
52
|
+
def channel
|
53
|
+
@channel ||= connection.create_channel
|
31
54
|
end
|
32
55
|
|
33
|
-
def
|
34
|
-
@
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
exchange = @channel.topic(exchange_name, :passive => true)
|
39
|
-
queue.bind(exchange, :routing_key => routing_key)
|
56
|
+
def connection
|
57
|
+
@connection ||= begin
|
58
|
+
new_connection = Bunny.new(RabbitMQConfig.new.from_environment)
|
59
|
+
new_connection.start
|
60
|
+
new_connection
|
40
61
|
end
|
41
|
-
queue
|
42
62
|
end
|
43
63
|
end
|
44
64
|
end
|
@@ -0,0 +1,20 @@
|
|
1
|
+
module GovukMessageQueueConsumer
|
2
|
+
class JSONProcessor
|
3
|
+
JSON_FORMAT = "application/json".freeze
|
4
|
+
|
5
|
+
def initialize(next_processor)
|
6
|
+
@next_processor = next_processor
|
7
|
+
end
|
8
|
+
|
9
|
+
def process(message)
|
10
|
+
if message.headers.content_type == JSON_FORMAT
|
11
|
+
message.payload = JSON.parse(message.payload)
|
12
|
+
end
|
13
|
+
|
14
|
+
@next_processor.process(message)
|
15
|
+
rescue JSON::ParserError => e
|
16
|
+
Airbrake.notify_or_ignore(e) if defined?(Airbrake)
|
17
|
+
message.discard
|
18
|
+
end
|
19
|
+
end
|
20
|
+
end
|
@@ -3,16 +3,12 @@ require 'json'
|
|
3
3
|
module GovukMessageQueueConsumer
|
4
4
|
# Client code will receive an instance of this
|
5
5
|
class Message
|
6
|
-
|
7
|
-
@delivery_info = delivery_info
|
8
|
-
@headers = headers
|
9
|
-
@body = payload
|
10
|
-
end
|
6
|
+
attr_accessor :delivery_info, :headers, :payload
|
11
7
|
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
@
|
8
|
+
def initialize(payload, headers, delivery_info)
|
9
|
+
@payload = payload
|
10
|
+
@headers = headers
|
11
|
+
@delivery_info = delivery_info
|
16
12
|
end
|
17
13
|
|
18
14
|
def ack
|
@@ -0,0 +1,29 @@
|
|
1
|
+
class RabbitMQConfig
|
2
|
+
class ConfigurationError < StandardError
|
3
|
+
end
|
4
|
+
|
5
|
+
def from_environment
|
6
|
+
{
|
7
|
+
hosts: fetch("RABBITMQ_HOSTS").split(','),
|
8
|
+
vhost: fetch("RABBITMQ_VHOST"),
|
9
|
+
user: fetch("RABBITMQ_USER"),
|
10
|
+
pass: fetch("RABBITMQ_PASSWORD"),
|
11
|
+
recover_from_connection_close: true,
|
12
|
+
}
|
13
|
+
end
|
14
|
+
|
15
|
+
private
|
16
|
+
|
17
|
+
def fetch(variable_name)
|
18
|
+
ENV[variable_name] || raise_error(variable_name)
|
19
|
+
end
|
20
|
+
|
21
|
+
def raise_error(variable_name)
|
22
|
+
raise ConfigurationError, <<-err
|
23
|
+
The environment variable #{variable_name} is not set. If you are in test
|
24
|
+
mode, make sure you set the correct vars in your helpers. If you get this
|
25
|
+
error in development, make sure you run rails or rake with `govuk_setenv`
|
26
|
+
and puppet is up to date.
|
27
|
+
err
|
28
|
+
end
|
29
|
+
end
|
@@ -0,0 +1,27 @@
|
|
1
|
+
module GovukMessageQueueConsumer
|
2
|
+
class MockMessage < Message
|
3
|
+
attr_reader :acked, :retried, :discarded
|
4
|
+
|
5
|
+
alias :acked? :acked
|
6
|
+
alias :discarded? :discarded
|
7
|
+
alias :retried? :retried
|
8
|
+
|
9
|
+
def initialize(payload = {}, headers = {}, delivery_info = {})
|
10
|
+
@payload = payload
|
11
|
+
@headers = OpenStruct.new(headers)
|
12
|
+
@delivery_info = OpenStruct.new(delivery_info)
|
13
|
+
end
|
14
|
+
|
15
|
+
def ack
|
16
|
+
@acked = true
|
17
|
+
end
|
18
|
+
|
19
|
+
def retry
|
20
|
+
@retried = true
|
21
|
+
end
|
22
|
+
|
23
|
+
def discard
|
24
|
+
@discarded = true
|
25
|
+
end
|
26
|
+
end
|
27
|
+
end
|
@@ -1,9 +1,11 @@
|
|
1
|
-
RSpec
|
2
|
-
|
3
|
-
|
4
|
-
|
1
|
+
if defined?(RSpec)
|
2
|
+
RSpec.shared_examples "a message queue processor" do
|
3
|
+
it "implements #process" do
|
4
|
+
expect(subject).to respond_to(:process)
|
5
|
+
end
|
5
6
|
|
6
|
-
|
7
|
-
|
7
|
+
it "accepts 1 argument for #process" do
|
8
|
+
expect(subject.method(:process).arity).to eq(1)
|
9
|
+
end
|
8
10
|
end
|
9
11
|
end
|
@@ -1,4 +1,6 @@
|
|
1
1
|
require 'govuk_message_queue_consumer/version'
|
2
2
|
require 'govuk_message_queue_consumer/heartbeat_processor'
|
3
|
+
require 'govuk_message_queue_consumer/json_processor'
|
3
4
|
require 'govuk_message_queue_consumer/message'
|
4
5
|
require 'govuk_message_queue_consumer/consumer'
|
6
|
+
require 'govuk_message_queue_consumer/rabbitmq_config'
|
data/spec/consumer_spec.rb
CHANGED
@@ -1,48 +1,35 @@
|
|
1
1
|
require_relative 'spec_helper'
|
2
2
|
|
3
3
|
describe Consumer do
|
4
|
-
|
5
|
-
let(:queue) { instance_double('Bunny::Queue', bind: nil, subscribe: message_values) }
|
4
|
+
let(:queue) { instance_double('Bunny::Queue', bind: nil, subscribe: '') }
|
6
5
|
let(:channel) { instance_double('Bunny::Channel', queue: queue, prefetch: nil, topic: nil) }
|
7
6
|
let(:rabbitmq_connecton) { instance_double("Bunny::Session", start: nil, create_channel: channel) }
|
8
7
|
let(:client_processor) { instance_double('Client::Processor') }
|
9
8
|
|
10
9
|
before do
|
10
|
+
stub_environment_variables!
|
11
11
|
allow(Bunny).to receive(:new).and_return(rabbitmq_connecton)
|
12
12
|
end
|
13
13
|
|
14
|
-
describe "
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
it "passes the client processor to the Heartbeat Processor" do
|
19
|
-
expect(HeartbeatProcessor).to receive(:new).with(client_processor)
|
20
|
-
|
21
|
-
Consumer.new(rabbitmq_config, client_processor)
|
22
|
-
end
|
23
|
-
|
24
|
-
it "connects to rabbitmq" do
|
25
|
-
expected_options = rabbitmq_config['connection'].symbolize_keys # Bunny requires the keys to be symbols
|
26
|
-
expect(Bunny).to receive(:new).with(expected_options).and_return(rabbitmq_connecton)
|
27
|
-
expect(rabbitmq_connecton).to receive(:start)
|
14
|
+
describe "running the consumer" do
|
15
|
+
it "binds the queue to the all-routing key" do
|
16
|
+
expect(queue).to receive(:bind).with(nil, { routing_key: "#" })
|
28
17
|
|
29
|
-
Consumer.new(
|
18
|
+
Consumer.new(queue_name: "some-queue", exchange_name: "my-exchange", processor: client_processor).run
|
30
19
|
end
|
31
|
-
end
|
32
20
|
|
33
|
-
|
34
|
-
|
35
|
-
expect(queue).to receive(:bind)
|
21
|
+
it "binds the queue to a custom routing key" do
|
22
|
+
expect(queue).to receive(:bind).with(nil, { routing_key: "*.major" })
|
36
23
|
|
37
|
-
Consumer.new(
|
24
|
+
Consumer.new(queue_name: "some-queue", exchange_name: "my-exchange", processor: client_processor, routing_key: "*.major").run
|
38
25
|
end
|
39
26
|
|
40
27
|
it "calls the heartbeat processor when subscribing to messages" do
|
41
|
-
expect(queue).to receive(:subscribe).and_yield(
|
42
|
-
expect(Message).to receive(:new).with(
|
28
|
+
expect(queue).to receive(:subscribe).and_yield(:delivery_info_object, :headers, "payload")
|
29
|
+
expect(Message).to receive(:new).with("payload", :headers, :delivery_info_object)
|
43
30
|
expect_any_instance_of(HeartbeatProcessor).to receive(:process)
|
44
31
|
|
45
|
-
Consumer.new(
|
32
|
+
Consumer.new(queue_name: "some-queue", exchange_name: "my-exchange", processor: client_processor).run
|
46
33
|
end
|
47
34
|
end
|
48
35
|
end
|
@@ -0,0 +1,33 @@
|
|
1
|
+
require_relative 'spec_helper'
|
2
|
+
|
3
|
+
describe JSONProcessor do
|
4
|
+
describe "#process" do
|
5
|
+
it "parses the payload string" do
|
6
|
+
next_processor = double("next_processor", process: "ha")
|
7
|
+
message = MockMessage.new('{"some":"json"}', { content_type: "application/json" })
|
8
|
+
|
9
|
+
JSONProcessor.new(next_processor).process(message)
|
10
|
+
|
11
|
+
expect(message.payload).to eql({ "some" => "json" })
|
12
|
+
expect(next_processor).to have_received(:process)
|
13
|
+
end
|
14
|
+
|
15
|
+
it "discards messages with JSON errors" do
|
16
|
+
message = MockMessage.new('{"some" "json"}', { content_type: "application/json" })
|
17
|
+
|
18
|
+
JSONProcessor.new(double).process(message)
|
19
|
+
|
20
|
+
expect(message).to be_discarded
|
21
|
+
end
|
22
|
+
|
23
|
+
it "doesn't parse non-JSON message" do
|
24
|
+
next_processor = double("next_processor", process: "ha")
|
25
|
+
message = MockMessage.new('<SomeXML></SomeXML>', { content_type: "application/xml" })
|
26
|
+
|
27
|
+
JSONProcessor.new(next_processor).process(message)
|
28
|
+
|
29
|
+
expect(message.payload).to eql('<SomeXML></SomeXML>')
|
30
|
+
expect(next_processor).to have_received(:process)
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
data/spec/message_spec.rb
CHANGED
@@ -4,25 +4,23 @@ describe Message do
|
|
4
4
|
let(:mock_channel) { instance_double("Channel") }
|
5
5
|
let(:delivery_info) { instance_double("DeliveryInfo", :channel => mock_channel, :delivery_tag => "a_tag") }
|
6
6
|
let(:headers) { instance_double("Headers") }
|
7
|
-
let(:
|
8
|
-
let(:message) { Message.new(delivery_info, headers, body) }
|
9
|
-
|
10
|
-
it "json decodes the body" do
|
11
|
-
expect(message.body_data).to eq("foo" => "bar")
|
12
|
-
end
|
7
|
+
let(:message) { Message.new({ "a" => "payload" }, headers, delivery_info) }
|
13
8
|
|
14
9
|
it "ack sends an ack to the channel" do
|
15
10
|
expect(mock_channel).to receive(:ack).with("a_tag")
|
11
|
+
|
16
12
|
message.ack
|
17
13
|
end
|
18
14
|
|
19
15
|
it "retry sends a reject to the channel with requeue set" do
|
20
16
|
expect(mock_channel).to receive(:reject).with("a_tag", true)
|
17
|
+
|
21
18
|
message.retry
|
22
19
|
end
|
23
20
|
|
24
21
|
it "reject sends a reject to the channel without requeue set" do
|
25
22
|
expect(mock_channel).to receive(:reject).with("a_tag", false)
|
23
|
+
|
26
24
|
message.discard
|
27
25
|
end
|
28
26
|
end
|
@@ -0,0 +1,43 @@
|
|
1
|
+
require_relative 'spec_helper'
|
2
|
+
require_relative '../lib/govuk_message_queue_consumer/test_helpers/mock_message'
|
3
|
+
|
4
|
+
describe GovukMessageQueueConsumer::MockMessage do
|
5
|
+
describe '#methods' do
|
6
|
+
it "implements the same methods as Message" do
|
7
|
+
mock = MockMessage.new
|
8
|
+
real = Message.new(double, double, double)
|
9
|
+
|
10
|
+
expect(real.methods - mock.methods).to be_empty
|
11
|
+
end
|
12
|
+
end
|
13
|
+
|
14
|
+
describe '#ack' do
|
15
|
+
it "marks the message as acked" do
|
16
|
+
message = MockMessage.new
|
17
|
+
|
18
|
+
message.ack
|
19
|
+
|
20
|
+
expect(message).to be_acked
|
21
|
+
end
|
22
|
+
end
|
23
|
+
|
24
|
+
describe '#retry' do
|
25
|
+
it "marks the message as retried" do
|
26
|
+
message = MockMessage.new
|
27
|
+
|
28
|
+
message.retry
|
29
|
+
|
30
|
+
expect(message).to be_retried
|
31
|
+
end
|
32
|
+
end
|
33
|
+
|
34
|
+
describe '#discard' do
|
35
|
+
it "marks the message as discarded" do
|
36
|
+
message = MockMessage.new
|
37
|
+
|
38
|
+
message.discard
|
39
|
+
|
40
|
+
expect(message).to be_discarded
|
41
|
+
end
|
42
|
+
end
|
43
|
+
end
|
@@ -0,0 +1,26 @@
|
|
1
|
+
require 'spec_helper'
|
2
|
+
|
3
|
+
RSpec.describe RabbitMQConfig do
|
4
|
+
describe ".from_environment" do
|
5
|
+
it "connects to rabbitmq with the correct environment variables" do
|
6
|
+
ENV["RABBITMQ_HOSTS"] = "server-one,server-two"
|
7
|
+
ENV["RABBITMQ_VHOST"] = "/"
|
8
|
+
ENV["RABBITMQ_USER"] = "my_user"
|
9
|
+
ENV["RABBITMQ_PASSWORD"] = "my_pass"
|
10
|
+
|
11
|
+
expect(RabbitMQConfig.new.from_environment).to eql({
|
12
|
+
hosts: ["server-one", "server-two"],
|
13
|
+
vhost: "/",
|
14
|
+
user: "my_user",
|
15
|
+
pass: "my_pass",
|
16
|
+
recover_from_connection_close: true,
|
17
|
+
})
|
18
|
+
end
|
19
|
+
|
20
|
+
it "provides a friendly error message when a variable is missing" do
|
21
|
+
ENV["RABBITMQ_HOSTS"] = nil
|
22
|
+
|
23
|
+
expect { RabbitMQConfig.new.from_environment }.to raise_error(RabbitMQConfig::ConfigurationError)
|
24
|
+
end
|
25
|
+
end
|
26
|
+
end
|
data/spec/spec_helper.rb
CHANGED
@@ -3,23 +3,11 @@ require_relative '../lib/govuk_message_queue_consumer'
|
|
3
3
|
include GovukMessageQueueConsumer
|
4
4
|
|
5
5
|
module TestHelpers
|
6
|
-
def
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
"vhost" => "/",
|
12
|
-
"user" => "a_user",
|
13
|
-
"pass" => "super secret",
|
14
|
-
"recover_from_connection_close" => true,
|
15
|
-
},
|
16
|
-
"queue" => "content_register",
|
17
|
-
"exchange" => "published_documents",
|
18
|
-
}
|
19
|
-
end
|
20
|
-
|
21
|
-
def message_values
|
22
|
-
[:delivery_info1, :headers1, "message1_body"]
|
6
|
+
def stub_environment_variables!
|
7
|
+
ENV["RABBITMQ_HOSTS"] ||= ""
|
8
|
+
ENV["RABBITMQ_VHOST"] ||= "/"
|
9
|
+
ENV["RABBITMQ_USER"] ||= "/"
|
10
|
+
ENV["RABBITMQ_PASSWORD"] ||= "/"
|
23
11
|
end
|
24
12
|
end
|
25
13
|
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: govuk_message_queue_consumer
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 2.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- GOV.UK Dev
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2015-
|
11
|
+
date: 2015-11-03 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bunny
|
@@ -24,20 +24,6 @@ dependencies:
|
|
24
24
|
- - "~>"
|
25
25
|
- !ruby/object:Gem::Version
|
26
26
|
version: 2.2.0
|
27
|
-
- !ruby/object:Gem::Dependency
|
28
|
-
name: activesupport
|
29
|
-
requirement: !ruby/object:Gem::Requirement
|
30
|
-
requirements:
|
31
|
-
- - "~>"
|
32
|
-
- !ruby/object:Gem::Version
|
33
|
-
version: '4.0'
|
34
|
-
type: :runtime
|
35
|
-
prerelease: false
|
36
|
-
version_requirements: !ruby/object:Gem::Requirement
|
37
|
-
requirements:
|
38
|
-
- - "~>"
|
39
|
-
- !ruby/object:Gem::Version
|
40
|
-
version: '4.0'
|
41
27
|
- !ruby/object:Gem::Dependency
|
42
28
|
name: gem_publisher
|
43
29
|
requirement: !ruby/object:Gem::Requirement
|
@@ -93,13 +79,20 @@ files:
|
|
93
79
|
- lib/govuk_message_queue_consumer.rb
|
94
80
|
- lib/govuk_message_queue_consumer/consumer.rb
|
95
81
|
- lib/govuk_message_queue_consumer/heartbeat_processor.rb
|
82
|
+
- lib/govuk_message_queue_consumer/json_processor.rb
|
96
83
|
- lib/govuk_message_queue_consumer/message.rb
|
84
|
+
- lib/govuk_message_queue_consumer/rabbitmq_config.rb
|
97
85
|
- lib/govuk_message_queue_consumer/test_helpers.rb
|
86
|
+
- lib/govuk_message_queue_consumer/test_helpers/mock_message.rb
|
98
87
|
- lib/govuk_message_queue_consumer/test_helpers/shared_examples.rb
|
88
|
+
- lib/govuk_message_queue_consumer/test_helpers/test_consumer.rb
|
99
89
|
- lib/govuk_message_queue_consumer/version.rb
|
100
90
|
- spec/consumer_spec.rb
|
101
91
|
- spec/heartbeat_processor_spec.rb
|
92
|
+
- spec/json_processor_spec.rb
|
102
93
|
- spec/message_spec.rb
|
94
|
+
- spec/mock_message_spec.rb
|
95
|
+
- spec/rabbitmq_config_spec.rb
|
103
96
|
- spec/spec_helper.rb
|
104
97
|
homepage: https://github.com/alphagov/govuk_message_queue_consumer
|
105
98
|
licenses: []
|
@@ -129,3 +122,6 @@ test_files:
|
|
129
122
|
- spec/spec_helper.rb
|
130
123
|
- spec/message_spec.rb
|
131
124
|
- spec/consumer_spec.rb
|
125
|
+
- spec/mock_message_spec.rb
|
126
|
+
- spec/json_processor_spec.rb
|
127
|
+
- spec/rabbitmq_config_spec.rb
|