amqp 0.8.0.rc13 → 0.8.0.rc14
Sign up to get free protection for your applications and to get access to all the features.
- data/.rspec +2 -1
- data/.travis.yml +8 -2
- data/.yardopts +1 -0
- data/CHANGELOG +9 -0
- data/Gemfile +17 -11
- data/README.md +26 -16
- data/amqp.gemspec +2 -2
- data/bin/ci/before_build.sh +21 -0
- data/docs/08Migration.textile +199 -5
- data/docs/AMQP091ModelExplained.textile +322 -0
- data/docs/Bindings.textile +24 -4
- data/docs/Clustering.textile +1 -1
- data/docs/ConnectingToTheBroker.textile +98 -82
- data/docs/ConnectionEncryptionWithTLS.textile +65 -5
- data/docs/DocumentationGuidesIndex.textile +93 -13
- data/docs/Durability.textile +1 -1
- data/docs/ErrorHandling.textile +458 -94
- data/docs/Exchanges.textile +901 -87
- data/docs/GettingStarted.textile +278 -143
- data/docs/PatternsAndUseCases.textile +420 -0
- data/docs/Queues.textile +730 -178
- data/docs/RabbitMQVersions.textile +18 -3
- data/docs/RunningTests.textile +1 -1
- data/docs/TestingWithEventedSpec.textile +121 -0
- data/docs/Troubleshooting.textile +15 -1
- data/docs/VendorSpecificExtensions.textile +1 -1
- data/docs/diagrams/001_hello_world_example_routing.png +0 -0
- data/docs/diagrams/002_blabbr_example_routing.png +0 -0
- data/docs/diagrams/003_weathr_example_routing.png +0 -0
- data/docs/diagrams/004_fanout_exchange.png +0 -0
- data/docs/diagrams/005_direct_exchange.png +0 -0
- data/docs/diagrams/redhat/direct_exchange.png +0 -0
- data/docs/diagrams/redhat/fanout_exchange.png +0 -0
- data/docs/diagrams/redhat/topic_exchange.png +0 -0
- data/examples/error_handling/automatic_recovery_of_channel_and_queues.rb +50 -0
- data/examples/error_handling/automatically_recovering_hello_world_consumer.rb +51 -0
- data/examples/error_handling/automatically_recovering_hello_world_consumer_that_uses_a_server_named_queue.rb +51 -0
- data/examples/error_handling/basic_connection_failover.rb +22 -0
- data/examples/error_handling/channel_level_exception.rb +9 -2
- data/examples/error_handling/connection_level_exception.rb +8 -1
- data/examples/error_handling/connection_level_exception_with_objects.rb +49 -0
- data/examples/error_handling/connection_loss_handler.rb +1 -5
- data/examples/error_handling/hello_world_producer.rb +43 -0
- data/examples/error_handling/insufficient_permissions.rb +54 -0
- data/examples/error_handling/manual_connection_and_channel_recovery.rb +71 -0
- data/examples/error_handling/queue_exclusivity_violation.rb +41 -0
- data/examples/error_handling/queue_name_violation.rb +31 -0
- data/examples/exchanges/autodeletion_of_exchanges.rb +1 -4
- data/examples/guides/queues/01a_declaring_a_server_named_queue_using_queue_constructor.rb +7 -8
- data/examples/guides/queues/01b_declaring_a_queue_using_queue_constructor.rb +7 -8
- data/examples/guides/queues/02a_declaring_a_durable_shared_queue.rb +5 -8
- data/examples/guides/queues/02b_declaring_a_durable_shared_queue.rb +5 -8
- data/examples/guides/queues/03a_declaring_a_temporary_exclusive_queue.rb +7 -8
- data/examples/guides/queues/04_bind_a_queue_using_exchange_instance.rb +9 -10
- data/examples/guides/queues/05_bind_a_queue_using_exchange_name.rb +8 -10
- data/examples/guides/queues/06_subscribe_to_receive_messages.rb +10 -12
- data/examples/guides/queues/07_fetch_a_message_from_the_queue.rb +14 -14
- data/examples/guides/queues/08_unsubscribing_a_consumer.rb +13 -16
- data/examples/guides/queues/09_unbinding_from_exchange.rb +16 -22
- data/examples/guides/queues/10_purge_a_queue.rb +13 -18
- data/examples/guides/queues/11_deleting_a_queue.rb +14 -19
- data/examples/guides/queues/12_objects_that_consume_messages.rb +69 -0
- data/examples/guides/queues/13_objects_that_consume_messages_take_two.rb +89 -0
- data/examples/hello_world.rb +1 -3
- data/examples/hello_world_with_an_empty_string.rb +5 -6
- data/examples/inspecting_server_information.rb +45 -0
- data/examples/issues/issue_93.rb +23 -0
- data/examples/issues/issue_94.rb +23 -0
- data/examples/patterns/command/consumer.rb +45 -0
- data/examples/patterns/command/producer.rb +26 -0
- data/examples/patterns/request_reply/client.rb +29 -0
- data/examples/patterns/request_reply/server.rb +26 -0
- data/examples/publishing/publishing_a_one_off_message.rb +6 -4
- data/examples/publishing/returned_messages.rb +2 -10
- data/examples/queues/accessing_message_metadata.rb +15 -13
- data/examples/queues/queue_status.rb +12 -15
- data/examples/routing/fanout_routing.rb +33 -0
- data/examples/routing/headers_routing.rb +17 -15
- data/examples/routing/round_robin_with_direct_exchange.rb +39 -0
- data/examples/routing/round_robin_with_the_default_exchange.rb +38 -0
- data/examples/routing/unroutable_mandatory_message_is_returned.rb +33 -0
- data/examples/routing/weather_updates.rb +15 -20
- data/examples/tls/using_tls.rb +41 -0
- data/lib/amqp/bit_set.rb +80 -0
- data/lib/amqp/broker.rb +72 -0
- data/lib/amqp/channel.rb +93 -13
- data/lib/amqp/client.rb +11 -22
- data/lib/amqp/compatibility/ruby187_patchlevel_check.rb +2 -0
- data/lib/amqp/connection.rb +2 -3
- data/lib/amqp/consumer.rb +208 -0
- data/lib/amqp/deprecated/fork.rb +2 -0
- data/lib/amqp/deprecated/mq.rb +2 -0
- data/lib/amqp/exchange.rb +6 -4
- data/lib/amqp/extensions/rabbitmq.rb +3 -1
- data/lib/amqp/header.rb +76 -14
- data/lib/amqp/int_allocator.rb +96 -0
- data/lib/amqp/logger.rb +2 -0
- data/lib/amqp/queue.rb +242 -86
- data/lib/amqp/rpc.rb +2 -0
- data/lib/amqp/session.rb +169 -9
- data/lib/amqp/utilities/event_loop_helper.rb +2 -0
- data/lib/amqp/utilities/server_type.rb +2 -0
- data/lib/amqp/version.rb +2 -2
- data/lib/mq.rb +4 -2
- data/lib/mq/logger.rb +3 -1
- data/lib/mq/rpc.rb +3 -1
- data/spec/integration/authentication_spec.rb +17 -10
- data/spec/integration/automatic_binding_for_default_direct_exchange_spec.rb +1 -1
- data/spec/integration/automatic_recovery_predicate_spec.rb +68 -0
- data/spec/integration/basic_get_spec.rb +2 -1
- data/spec/integration/{extensions/basic_return_spec.rb → basic_return_spec.rb} +2 -1
- data/spec/integration/channel_level_exception_handling_spec.rb +53 -0
- data/spec/integration/connection_level_exception_handling_spec.rb +49 -0
- data/spec/integration/declare_and_immediately_bind_a_server_named_queue_spec.rb +38 -17
- data/spec/integration/declare_one_hundred_server_named_queues_spec.rb +44 -0
- data/spec/integration/direct_exchange_routing_spec.rb +125 -0
- data/spec/integration/exchange_declaration_spec.rb +75 -46
- data/spec/integration/extensions/rabbitmq/publisher_confirmations_spec.rb +180 -0
- data/spec/integration/{workload_distribution_spec.rb → fanout_exchange_routing_spec.rb} +10 -9
- data/spec/integration/headers_exchange_routing_spec.rb +269 -0
- data/spec/integration/hello_world_spec.rb +77 -0
- data/spec/integration/immediate_messages_spec.rb +59 -0
- data/spec/integration/mandatory_messages_spec.rb +52 -0
- data/spec/integration/message_metadata_access_spec.rb +106 -0
- data/spec/integration/multiple_consumers_per_queue_spec.rb +319 -0
- data/spec/integration/ordering_of_published_messages_spec.rb +96 -0
- data/spec/integration/queue_declaration_spec.rb +8 -8
- data/spec/integration/queue_status_spec.rb +66 -0
- data/spec/integration/recovery/per_channel_automatic_recovery_on_graceful_broker_shutdown_spec.rb +76 -0
- data/spec/integration/recovery/per_channel_automatic_recovery_spec.rb +72 -0
- data/spec/integration/redelivery_of_unacknowledged_messages_spec.rb +96 -0
- data/spec/integration/regressions/concurrent_publishing_on_the_same_channel_spec.rb +91 -0
- data/spec/integration/regressions/empty_message_body_spec.rb +56 -0
- data/spec/integration/regressions/issue66_spec.rb +2 -1
- data/spec/integration/reply_queue_communication_spec.rb +2 -1
- data/spec/integration/store_and_forward_spec.rb +4 -3
- data/spec/integration/topic_subscription_spec.rb +2 -1
- data/spec/integration/tx_commit_spec.rb +124 -0
- data/spec/integration/tx_rollback_spec.rb +167 -0
- data/spec/spec_helper.rb +44 -71
- data/spec/unit/amqp/bit_set_spec.rb +127 -0
- data/spec/unit/amqp/channel_id_allocation_spec.rb +40 -0
- data/spec/unit/amqp/connection_spec.rb +4 -2
- data/spec/unit/amqp/int_allocator_spec.rb +116 -0
- metadata +92 -26
- data/CONTRIBUTORS +0 -29
- data/docs/Routing.textile +0 -30
- data/examples/real-world/task-queue/README.textile +0 -3
- data/examples/real-world/task-queue/consumer.rb +0 -27
- data/examples/real-world/task-queue/producer.rb +0 -22
- data/spec/unit/amqp/basic_spec.rb +0 -39
- data/tasks.rb +0 -4
@@ -0,0 +1,420 @@
|
|
1
|
+
# @title Ruby AMQP gem: Patterns and Use Cases
|
2
|
+
|
3
|
+
|
4
|
+
h1. Patterns and Use Cases
|
5
|
+
|
6
|
+
|
7
|
+
h2. About this guide
|
8
|
+
|
9
|
+
This guide explains typical messaging patterns and use cases. It only covers the most common scenarios. For comprehensive list
|
10
|
+
of messaging patterns, consult books on this subject, for example, "Enterprise Integration Patterns":http://www.eaipatterns.com.
|
11
|
+
|
12
|
+
|
13
|
+
h2. Covered versions
|
14
|
+
|
15
|
+
This guide covers "Ruby amqp gem":http://github.com/ruby-amqp/amqp v0.8.0 and later.
|
16
|
+
|
17
|
+
|
18
|
+
|
19
|
+
h2. Introduction
|
20
|
+
|
21
|
+
Messaging patterns are a lot like object-oriented design patterns: they are generalized reusable solutions to specific problems. They are not
|
22
|
+
recipes, however, and their exact implementation may and will vary from application to application. Just like OO design patterns,
|
23
|
+
they can classified:
|
24
|
+
|
25
|
+
* Message construction patterns describe form, content and purpose of messages.
|
26
|
+
* Message routing patterns outline how messages can be directed from producers to consumers.
|
27
|
+
* Message transformation patterns change message content or metadata.
|
28
|
+
|
29
|
+
There are other, more specialized group of messaging patterns that are out of scope of this guide.
|
30
|
+
|
31
|
+
This guide demonstrates implementation of several common routing patterns plus explains how built-in AMQP 0.9.1 features
|
32
|
+
can be used to implement message construction and message transformation patterns.
|
33
|
+
|
34
|
+
TBD
|
35
|
+
|
36
|
+
|
37
|
+
|
38
|
+
h2. Request/Reply pattern
|
39
|
+
|
40
|
+
h3. Description & Use cases
|
41
|
+
|
42
|
+
Request/Reply is a simple way of integration when one application issues a request and another application responds
|
43
|
+
to it. This pattern is often referred to as"Remote Procedure Call", even when it is not entirely correct. Request/Reply
|
44
|
+
pattern is a 1:1 communication pattern.
|
45
|
+
|
46
|
+
Some examples of Request/Reply pattern are:
|
47
|
+
|
48
|
+
* The 1st application requests a document that the 2nd application generates or loads and returns.
|
49
|
+
* End-user application issues a search request and another application returns results back.
|
50
|
+
* One application requests a progress report from another application.
|
51
|
+
|
52
|
+
h3. AMQP-based implementation
|
53
|
+
|
54
|
+
Implementation of Request/Reply pattern on top of AMQP 0.9.1 involves two messages: a request (Req) and a response (Res).
|
55
|
+
Client app generates a request identifier and sets :message_id attribute on Req. Client also uses a server-named
|
56
|
+
exclusive queue to receive replies and thus sets :reply_to Req attribute to the name of that queue.
|
57
|
+
|
58
|
+
Server app uses a well-known queue name to receive requests and sets :correlation_id to :message_id of the original
|
59
|
+
request message (Req) to make it possible for the client to identify what request this reply is for.
|
60
|
+
|
61
|
+
h4. Request message attributes
|
62
|
+
|
63
|
+
<dl>
|
64
|
+
<dt>:message_id</dt>
|
65
|
+
<dd>Unique message identifier</dd>
|
66
|
+
|
67
|
+
<dt>:reply_to</dt>
|
68
|
+
<dd>Queue name server should send the response to</dd>
|
69
|
+
</dl>
|
70
|
+
|
71
|
+
h4. Response message attributes
|
72
|
+
|
73
|
+
<dl>
|
74
|
+
<dt>:correlation_id</dt>
|
75
|
+
<dd>Identifier of the original request message (set to request's :correlation_id)</dd>
|
76
|
+
|
77
|
+
<dt>:routing_key</dt>
|
78
|
+
<dd>Client's replies queue name (set to request's :reply_to)</dd>
|
79
|
+
</dl>
|
80
|
+
|
81
|
+
|
82
|
+
|
83
|
+
h3. Code example
|
84
|
+
|
85
|
+
h4. Client code
|
86
|
+
|
87
|
+
<pre>
|
88
|
+
<code>
|
89
|
+
require "amqp"
|
90
|
+
|
91
|
+
EventMachine.run do
|
92
|
+
connection = AMQP.connect
|
93
|
+
channel = AMQP::Channel.new(connection)
|
94
|
+
|
95
|
+
replies_queue = channel.queue("", :exclusive => true, :auto_delete => true)
|
96
|
+
replies_queue.subscribe do |metadata, payload|
|
97
|
+
puts "[response] Response for #{metadata.correlation_id}: #{payload.inspect}"
|
98
|
+
end
|
99
|
+
|
100
|
+
# request time from a peer every 3 seconds
|
101
|
+
EventMachine.add_periodic_timer(3.0) do
|
102
|
+
puts "[request] Sending a request..."
|
103
|
+
channel.default_exchange.publish("get.time",
|
104
|
+
:routing_key => "amqpgem.examples.services.time",
|
105
|
+
:message_id => Kernel.rand(10101010).to_s,
|
106
|
+
:reply_to => replies_queue.name,
|
107
|
+
:immediate => true)
|
108
|
+
end
|
109
|
+
|
110
|
+
|
111
|
+
Signal.trap("INT") { connection.close { EventMachine.stop } }
|
112
|
+
end
|
113
|
+
</code>
|
114
|
+
</pre>
|
115
|
+
|
116
|
+
|
117
|
+
h4. Server code
|
118
|
+
|
119
|
+
<pre>
|
120
|
+
<code>
|
121
|
+
require "amqp"
|
122
|
+
|
123
|
+
EventMachine.run do
|
124
|
+
connection = AMQP.connect
|
125
|
+
channel = AMQP::Channel.new(connection)
|
126
|
+
|
127
|
+
requests_queue = channel.queue("amqpgem.examples.services.time", :exclusive => true, :auto_delete => true)
|
128
|
+
requests_queue.subscribe(:ack => true) do |metadata, payload|
|
129
|
+
puts "[requests] Got a request #{metadata.message_id}. Sending a reply..."
|
130
|
+
channel.default_exchange.publish(Time.now.to_s,
|
131
|
+
:routing_key => metadata.reply_to,
|
132
|
+
:correlation_id => metadata.message_id,
|
133
|
+
:immediate => true,
|
134
|
+
:mandatory => true)
|
135
|
+
|
136
|
+
metadata.ack
|
137
|
+
end
|
138
|
+
|
139
|
+
|
140
|
+
Signal.trap("INT") { connection.close { EventMachine.stop } }
|
141
|
+
end
|
142
|
+
</code>
|
143
|
+
</pre>
|
144
|
+
|
145
|
+
In the examples above messages are published with the :immediate attribute set. This is not necessary in all
|
146
|
+
cases: sometimes it is OK for requests to sit in the queue without active consumers. Replies, on the other hand,
|
147
|
+
assume an active consumer and existing replies queue, so if routing or immediate delivery do not succeed,
|
148
|
+
server application will log returned messages. More on this in the {file:docs/Exchanges.textile Working With Exchanges} guide.
|
149
|
+
|
150
|
+
|
151
|
+
|
152
|
+
h3. Related patterns
|
153
|
+
|
154
|
+
* Request/Reply
|
155
|
+
* Event
|
156
|
+
* Scatter/Gather
|
157
|
+
* Smart Proxy
|
158
|
+
|
159
|
+
|
160
|
+
|
161
|
+
|
162
|
+
|
163
|
+
h2. Command pattern
|
164
|
+
|
165
|
+
h3. Description & Use cases
|
166
|
+
|
167
|
+
Command pattern is very similar to Request/Reply, except that there is no reply and messages are typed. For example, most modern Web
|
168
|
+
applications have at least one "background task processor" that carries out a number of operations asynchronously,
|
169
|
+
without sending any responses back. Command pattern usually assumes 1:1 communication.
|
170
|
+
|
171
|
+
Some specific examples of Command pattern are:
|
172
|
+
|
173
|
+
* Account termination in a Web app triggers information archiving (or deletion) that is done by a separate app "in the background".
|
174
|
+
* After a document or profile update, a Web app sends out commands to a search indexer application.
|
175
|
+
* Virtual machines control dashboard app sends virtual machine controller application a command to reboot.
|
176
|
+
|
177
|
+
|
178
|
+
h3. AMQP-based implementation
|
179
|
+
|
180
|
+
Implementation of Command pattern on top of AMQP 0.9.1 involves well-known durable queues. Application that issues the command
|
181
|
+
then can use default exchange to publish messages to well-known services directly. Request message :type attribute then indicates
|
182
|
+
command type and message body (or body and headers) carry additional information consumer needs to carry it out.
|
183
|
+
|
184
|
+
h4. Request message attributes
|
185
|
+
|
186
|
+
<dl>
|
187
|
+
<dt>:type</dt>
|
188
|
+
<dd>Message type, as a string. For example: gems.install or commands.shutdown</dd>
|
189
|
+
</dl>
|
190
|
+
|
191
|
+
h3. Code example
|
192
|
+
|
193
|
+
h4. Producer (Sender)
|
194
|
+
|
195
|
+
<pre>
|
196
|
+
<code>
|
197
|
+
require "rubygems"
|
198
|
+
require "amqp"
|
199
|
+
require "yaml"
|
200
|
+
|
201
|
+
t = Thread.new { EventMachine.run }
|
202
|
+
sleep(0.5)
|
203
|
+
|
204
|
+
|
205
|
+
connection = AMQP.connect
|
206
|
+
channel = AMQP::Channel.new(connection, :auto_recovery => true)
|
207
|
+
|
208
|
+
channel.prefetch(1)
|
209
|
+
|
210
|
+
# Acknowledgements are good for letting the server know
|
211
|
+
# that the task is finished. If the consumer doesn't send
|
212
|
+
# the acknowledgement, then the task is considered to be unfinished
|
213
|
+
# and will be requeued when consumer closes AMQP connection (because of a crash, for example).
|
214
|
+
channel.queue("amqpgem.examples.patterns.command", :durable => true, :auto_delete => false).subscribe(:ack => true) do |metadata, payload|
|
215
|
+
case metadata.type
|
216
|
+
when "gems.install"
|
217
|
+
data = YAML.load(payload)
|
218
|
+
puts "[gems.install] Received a 'gems.install' request with #{data.inspect}"
|
219
|
+
|
220
|
+
# just to demonstrate a realistic example
|
221
|
+
shellout = "gem install #{data[:gem]} --version '#{data[:version]}'"
|
222
|
+
puts "[gems.install] Executing #{shellout}"; system(shellout)
|
223
|
+
puts "[gems.install] Done"
|
224
|
+
puts
|
225
|
+
else
|
226
|
+
puts "[commands] Unknown command: #{metadata.type}"
|
227
|
+
end
|
228
|
+
|
229
|
+
# message is processed, acknowledge it so that broker discards it
|
230
|
+
metadata.ack
|
231
|
+
end
|
232
|
+
|
233
|
+
puts "[boot] Ready. Will be publishing commands every 10 seconds."
|
234
|
+
Signal.trap("INT") { connection.close { EventMachine.stop } }
|
235
|
+
t.join
|
236
|
+
</code>
|
237
|
+
</pre>
|
238
|
+
|
239
|
+
|
240
|
+
h4. Consumer (Recipient)
|
241
|
+
|
242
|
+
<pre>
|
243
|
+
<code>
|
244
|
+
require "amqp"
|
245
|
+
require "yaml"
|
246
|
+
|
247
|
+
t = Thread.new { EventMachine.run }
|
248
|
+
sleep(0.5)
|
249
|
+
|
250
|
+
connection = AMQP.connect
|
251
|
+
channel = AMQP::Channel.new(connection)
|
252
|
+
|
253
|
+
# publish new commands every 3 seconds
|
254
|
+
EventMachine.add_periodic_timer(10.0) do
|
255
|
+
puts "Publishing a command (gems.install)"
|
256
|
+
payload = { :gem => "rack", :version => "~> 1.3.0" }.to_yaml
|
257
|
+
|
258
|
+
channel.default_exchange.publish(payload,
|
259
|
+
:type => "gems.install",
|
260
|
+
:routing_key => "amqpgem.examples.patterns.command")
|
261
|
+
end
|
262
|
+
|
263
|
+
puts "[boot] Ready"
|
264
|
+
Signal.trap("INT") { connection.close { EventMachine.stop } }
|
265
|
+
t.join
|
266
|
+
</code>
|
267
|
+
</pre>
|
268
|
+
|
269
|
+
|
270
|
+
h3. Related patterns
|
271
|
+
|
272
|
+
* Event
|
273
|
+
* Request/Reply
|
274
|
+
|
275
|
+
|
276
|
+
|
277
|
+
|
278
|
+
|
279
|
+
h2. Event pattern
|
280
|
+
|
281
|
+
h3. Description & Use cases
|
282
|
+
|
283
|
+
Event pattern is a version of the Command pattern, but with 1 or more receivers (1:N communication).
|
284
|
+
The world we live in is full of events, so applications of this pattern are endless.
|
285
|
+
|
286
|
+
Some specific use cases of Event pattern are
|
287
|
+
|
288
|
+
* Event logging (one application asks event collector to record certain event and possibly take action)
|
289
|
+
* Event propagation in MMO games
|
290
|
+
* Live sport score updates
|
291
|
+
* Various "push notifications" for mobile applications
|
292
|
+
|
293
|
+
|
294
|
+
h3. AMQP-based implementation
|
295
|
+
|
296
|
+
Because Event pattern is a 1:N communication pattern, it typically uses a fanout exchange. Event listeners
|
297
|
+
then use server-named exclusive queues and all bind to that exchange. Event messages use :type message
|
298
|
+
attribute to indicate event type and message body (plus, possibly, message headers) to pass event
|
299
|
+
context information.
|
300
|
+
|
301
|
+
More on fanout exchange type in the {file:docs/Exchanges.textile Working With Exchanges} guide.
|
302
|
+
|
303
|
+
|
304
|
+
h3. Code example
|
305
|
+
|
306
|
+
TBD
|
307
|
+
|
308
|
+
|
309
|
+
h3. Related patterns
|
310
|
+
|
311
|
+
* Command
|
312
|
+
* Publish/Subscribe
|
313
|
+
|
314
|
+
|
315
|
+
|
316
|
+
h2. Document Message pattern
|
317
|
+
|
318
|
+
h3. Description & Use cases
|
319
|
+
|
320
|
+
TBD
|
321
|
+
|
322
|
+
h3. AMQP-based implementation
|
323
|
+
|
324
|
+
TBD
|
325
|
+
|
326
|
+
h3. Code example
|
327
|
+
|
328
|
+
TBD
|
329
|
+
|
330
|
+
|
331
|
+
h2. Competing Consumers pattern
|
332
|
+
|
333
|
+
h3. Description & Use cases
|
334
|
+
|
335
|
+
"Competing Consumers":http://www.eaipatterns.com/CompetingConsumers.html are multiple consumers that process messages from a shared queue.
|
336
|
+
|
337
|
+
TBD
|
338
|
+
|
339
|
+
h3. AMQP-based implementation
|
340
|
+
|
341
|
+
TBD
|
342
|
+
|
343
|
+
h3. Code example
|
344
|
+
|
345
|
+
TBD
|
346
|
+
|
347
|
+
|
348
|
+
|
349
|
+
h2. Publish/Subscribe pattern
|
350
|
+
|
351
|
+
h3. Description & Use cases
|
352
|
+
|
353
|
+
TBD
|
354
|
+
|
355
|
+
h3. AMQP-based implementation
|
356
|
+
|
357
|
+
TBD
|
358
|
+
|
359
|
+
h3. Code example
|
360
|
+
|
361
|
+
TBD
|
362
|
+
|
363
|
+
|
364
|
+
|
365
|
+
h2. Scatter/Gather pattern
|
366
|
+
|
367
|
+
h3. Description & Use cases
|
368
|
+
|
369
|
+
TBD
|
370
|
+
|
371
|
+
h3. AMQP-based implementation
|
372
|
+
|
373
|
+
TBD
|
374
|
+
|
375
|
+
h3. Code example
|
376
|
+
|
377
|
+
TBD
|
378
|
+
|
379
|
+
|
380
|
+
|
381
|
+
h2. Smart Proxy pattern
|
382
|
+
|
383
|
+
h3. Description & Use cases
|
384
|
+
|
385
|
+
TBD
|
386
|
+
|
387
|
+
h3. AMQP-based implementation
|
388
|
+
|
389
|
+
TBD
|
390
|
+
|
391
|
+
h3. Code example
|
392
|
+
|
393
|
+
TBD
|
394
|
+
|
395
|
+
|
396
|
+
|
397
|
+
|
398
|
+
h2. Multistep Processing (Routing Slip) pattern
|
399
|
+
|
400
|
+
h3. Description & Use cases
|
401
|
+
|
402
|
+
TBD
|
403
|
+
|
404
|
+
h3. AMQP-based implementation
|
405
|
+
|
406
|
+
TBD
|
407
|
+
|
408
|
+
h3. Code example
|
409
|
+
|
410
|
+
TBD
|
411
|
+
|
412
|
+
|
413
|
+
|
414
|
+
h2. Tell us what you think!
|
415
|
+
|
416
|
+
Please take a moment and tell us what you think about this guide "on Twitter":http://twitter.com/rubyamqp or "Ruby AMQP mailing list":http://groups.google.com/group/ruby-amqp:
|
417
|
+
what was unclear? what wasn't covered? maybe you don't like guide style or grammar and spelling are incorrect? Readers feedback is
|
418
|
+
key to making documentation better.
|
419
|
+
|
420
|
+
If mailing list communication is not an option for you for some reason, you can "contact guides author directly":mailto:michael@novemberain.com?subject=amqp%20gem%20documentation
|
data/docs/Queues.textile
CHANGED
@@ -5,76 +5,138 @@ h1. Working with queues
|
|
5
5
|
|
6
6
|
h2. About this guide
|
7
7
|
|
8
|
-
This guide covers everything related to queues in AMQP
|
8
|
+
This guide covers everything related to queues in the AMQP v0.9.1 specification, common usage scenarios and how to accomplish typical operations using the
|
9
9
|
amqp gem.
|
10
10
|
|
11
11
|
|
12
|
-
h2.
|
12
|
+
h2. Which versions of the amqp gem does this guide cover?
|
13
13
|
|
14
|
-
This guide covers "Ruby amqp gem":http://github.com/ruby-amqp/amqp
|
14
|
+
This guide covers v0.8.0 and later of the "Ruby amqp gem":http://github.com/ruby-amqp/amqp.
|
15
15
|
|
16
16
|
|
17
|
-
|
18
|
-
h2. Queues in AMQP 0.9.1, briefly
|
17
|
+
h2. Queues in AMQP v0.9.1 - overview
|
19
18
|
|
20
19
|
h3. What are AMQP queues?
|
21
20
|
|
22
|
-
|
21
|
+
_Queues_ store and forward messages to consumers. They are similar to mailboxes in SMTP.
|
23
22
|
Messages flow from producing applications to {file:docs/Exchanges.textile exchanges} that route them
|
24
|
-
to queues and finally queues deliver
|
23
|
+
to queues and finally queues deliver the messages to consumer applications (or consumer applications fetch messages as needed).
|
25
24
|
|
26
25
|
Note that unlike some other messaging protocols/systems, messages are not delivered directly
|
27
26
|
to queues. They are delivered to exchanges that route messages to queues using rules
|
28
|
-
|
27
|
+
known as _bindings_.
|
29
28
|
|
30
29
|
AMQP is a programmable protocol, so queues and bindings alike are declared by applications.
|
31
30
|
|
32
31
|
|
33
32
|
h3. Concept of bindings
|
34
33
|
|
35
|
-
|
36
|
-
Learn more about bindings in {file:docs/Bindings.textile Bindings guide}.
|
34
|
+
A _binding_ is an association between a queue and an exchange. Queues must be bound to at least one exchange in order to receive messages from publishers.
|
35
|
+
Learn more about bindings in the {file:docs/Bindings.textile Bindings guide}.
|
37
36
|
|
38
37
|
|
39
|
-
h3.
|
38
|
+
h3. Queue attributes
|
40
39
|
|
41
40
|
Queues have several attributes associated with them:
|
42
41
|
|
43
42
|
* Name
|
44
43
|
* Exclusivity
|
45
44
|
* Durability
|
46
|
-
* Whether queue is auto-deleted when no longer used
|
47
|
-
* Other metadata (
|
45
|
+
* Whether the queue is auto-deleted when no longer used
|
46
|
+
* Other metadata (sometimes called _X-arguments_)
|
48
47
|
|
49
|
-
These attributes define how queues can be used, what their
|
48
|
+
These attributes define how queues can be used, what their life-cycle is like and other aspects of queue
|
50
49
|
behavior.
|
51
50
|
|
52
|
-
amqp gem represents queues as instances of {AMQP::Queue}.
|
51
|
+
The amqp gem represents queues as instances of {AMQP::Queue}.
|
53
52
|
|
54
|
-
h2. Queue names. Server-named queues. Predefined queues.
|
55
53
|
|
56
|
-
|
57
|
-
path segments are separated by a slash (/), although it may be almost any string, with some limitations (see below).
|
58
|
-
Applications may pick queue names or ask broker to generate a name for them. To do so, pass *empty string* as queue name argument.
|
54
|
+
h2. Queue names and declaring queues
|
59
55
|
|
60
|
-
|
56
|
+
Every AMQP queue has a name that identifies it. Queue names often contain several segments separated by a dot ".", in a similar fashion to URI
|
57
|
+
path segments being separated by a slash "/", although almost any string can represent a segment (with some limitations - see below).
|
61
58
|
|
62
|
-
|
59
|
+
Before a queue can be used, it has to be *declared*. Declaring a queue will cause it to be created if it does not already exist. The declaration will have no effect if the queue does already exist
|
60
|
+
and its attributes are the *same as those in the declaration*. When the existing queue attributes are not the same as those in the declaration a channel-level exception is raised. This case is explained later in this
|
61
|
+
guide.
|
62
|
+
|
63
|
+
h3. Explicitly named queues
|
64
|
+
|
65
|
+
Applications may pick queue names or ask the broker to generate a name for them.
|
66
|
+
|
67
|
+
To declare a queue with a particular name, for example, "images.resize", pass it to the Queue class constructor:
|
63
68
|
|
64
|
-
|
69
|
+
<pre>
|
70
|
+
<code>
|
71
|
+
queue = AMQP::Queue.new(channel, "images.resize", :auto_delete => true)
|
72
|
+
</code>
|
73
|
+
</pre>
|
65
74
|
|
75
|
+
Full example:
|
66
76
|
<script src="https://gist.github.com/998721.js"> </script>
|
67
77
|
|
68
|
-
Queue names starting with 'amq.' are reserved for internal use by the broker. Attempts to declare queue with a name that violates this
|
69
|
-
rule will result in AMQP::IncompatibleOptionsError to be thrown (when queue is re-declared on the same channel object) or channel-level exception
|
70
|
-
(when originally queue was declared on one channel and re-declaration with different attributes happens on another channel).
|
71
|
-
Learn more in Error handling and recovery section below.
|
72
78
|
|
79
|
+
h3. Server-named queues
|
80
|
+
|
81
|
+
To ask an AMQP broker to generate a unique queue name for you, pass an *empty string* as the queue name argument:
|
82
|
+
|
83
|
+
<pre>
|
84
|
+
<code>
|
85
|
+
AMQP::Queue.new(channel, "", :auto_delete => true) do |queue, declare_ok|
|
86
|
+
puts "#{queue.name} is ready to go. AMQP method: #{declare_ok.inspect}"
|
87
|
+
end
|
88
|
+
</code>
|
89
|
+
</pre>
|
90
|
+
|
91
|
+
Full example:
|
92
|
+
<script src="https://gist.github.com/998720.js"> </script>
|
93
|
+
|
94
|
+
The amqp gem allows server-named queues to be declared without callbacks:
|
95
|
+
|
96
|
+
<pre>
|
97
|
+
<code>
|
98
|
+
queue = AMQP::Queue.new(channel, "", :auto_delete => true)
|
99
|
+
</code>
|
100
|
+
</pre>
|
101
|
+
|
102
|
+
In this case, as soon as the AMQP broker reply (`queue.declare-ok` AMQP method) arrives, the queue object name will
|
103
|
+
be assigned to the value that the broker generated. Many AMQP operations require a queue name, so before an
|
104
|
+
{AMQP::Queue} instance receives its name, those operations are delayed. This example demonstrates this:
|
105
|
+
|
106
|
+
<pre>
|
107
|
+
<code>
|
108
|
+
queue = channel.queue("")
|
109
|
+
queue.bind("builds").subscribe do |metadata, payload|
|
110
|
+
# message handling implementation...
|
111
|
+
end
|
112
|
+
</code>
|
113
|
+
</pre>
|
114
|
+
|
115
|
+
In this example, binding will be performed as soon as the queue has received its name generated by the broker.
|
116
|
+
If a particular piece of code relies on the queue name being available immediately a callback should be used.
|
117
|
+
|
118
|
+
|
119
|
+
h3. Reserved queue name prefix
|
120
|
+
|
121
|
+
Queue names starting with "amq." are reserved for internal use by the broker. Attempts to declare a queue with a name
|
122
|
+
that violates this rule will result in a channel-level exception with reply code 403 (ACCESS_REFUSED) and a reply
|
123
|
+
message similar to this:
|
73
124
|
|
125
|
+
<pre>ACCESS_REFUSED - queue name 'amq.queue' contains reserved prefix 'amq.*'</pre>
|
74
126
|
|
75
|
-
h2. Queue life-cycle patterns.
|
76
127
|
|
77
|
-
|
128
|
+
h3. Queue re-declaration with different attributes
|
129
|
+
|
130
|
+
When queue declaration attributes are different from those that the queue already has, a channel-level exception with
|
131
|
+
code 406 (PRECONDITION_FAILED) will be raised. The reply text will be similar to this:
|
132
|
+
|
133
|
+
<pre>PRECONDITION_FAILED - parameters for queue 'amqpgem.examples.channel_exception' in vhost '/' not equivalent</pre>
|
134
|
+
|
135
|
+
|
136
|
+
|
137
|
+
h2. Queue life-cycle patterns
|
138
|
+
|
139
|
+
According to the AMQP v0.9.1 specification, there are two common message queue life-cycle patterns:
|
78
140
|
|
79
141
|
* Durable message queues that are shared by many consumers and have an independent existence: i.e. they
|
80
142
|
will continue to exist and collect messages whether or not there are consumers to receive them.
|
@@ -84,40 +146,56 @@ To quote AMQP 0.9.1 spec, there are two common message queue life-cycle patterns
|
|
84
146
|
There are some variations of these, such as shared message queues that are deleted when the last of
|
85
147
|
many consumers disconnects.
|
86
148
|
|
87
|
-
|
88
|
-
|
89
|
-
|
90
|
-
|
91
|
-
|
92
|
-
|
93
|
-
|
94
|
-
|
149
|
+
Let us examine the example of a well-known service like an event collector (event logger). A logger is
|
150
|
+
usually up and running regardless of the existence of services that want to log anything at a particular
|
151
|
+
point in time. Other applications know which queues to use in order to communicate with the logger and can
|
152
|
+
rely on those queues being available and able to survive broker restarts. In this case, explicitly named durable
|
153
|
+
queues are optimal and the coupling that is created between applications is not an issue.
|
154
|
+
|
155
|
+
Another example of a well-known long-lived service is a distributed metadata/directory/locking server like
|
156
|
+
"Apache Zookeeper":http://zookeeper.apache.org, "Google's Chubby":http://labs.google.com/papers/chubby.html or DNS. Services like this benefit from using well-known, not server-generated,
|
157
|
+
queue names and so do any other applications that use them.
|
95
158
|
|
96
|
-
|
97
|
-
|
98
|
-
queue names
|
99
|
-
topic or fanout exchanges to receive relevant messages
|
159
|
+
A different sort of scenario is in "a cloud setting" when some kind of worker/instance might start and
|
160
|
+
stop at any time so that other applications cannot rely on it being available. In this case, it is possible
|
161
|
+
to use well-known queue names, but a much better solution is to use server-generated, short-lived queues
|
162
|
+
that are bound to topic or fanout exchanges in order to receive relevant messages.
|
100
163
|
|
101
|
-
Imagine a service that processes an endless stream of events
|
102
|
-
|
103
|
-
Those new instances want to subscribe to receive messages to process but the rest of the system
|
104
|
-
know anything about them
|
105
|
-
from a shared stream and are
|
106
|
-
message consumers to
|
164
|
+
Imagine a service that processes an endless stream of events - Twitter is one example. When traffic
|
165
|
+
increases, development operations may start additional application instances in the cloud to handle the load.
|
166
|
+
Those new instances want to subscribe to receive messages to process, but the rest of the system does not
|
167
|
+
know anything about them and cannot rely on them being online or try to address them directly. The new instances
|
168
|
+
process events from a shared stream and are the same as their peers. In a case like this, there is no reason for
|
169
|
+
message consumers not to use queue names generated by the broker.
|
107
170
|
|
108
|
-
In general, use of explicitly named or server-named queues depends on messaging pattern your application
|
109
|
-
{http://www.eaipatterns.com/ Enterprise Integration
|
110
|
-
RabbitMQ FAQ also has a section on {http://www.rabbitmq.com/faq.html#scenarios use cases}.
|
171
|
+
In general, use of explicitly named or server-named queues depends on the messaging pattern that your application
|
172
|
+
needs. {http://www.eaipatterns.com/ Enterprise Integration Patterns} discusses many messaging patterns in depth and
|
173
|
+
the RabbitMQ FAQ also has a section on {http://www.rabbitmq.com/faq.html#scenarios use cases}.
|
111
174
|
|
112
175
|
|
113
176
|
|
114
177
|
h2. Declaring a durable shared queue
|
115
178
|
|
116
|
-
To declare a durable shared queue, you pass queue name that is a non-blank string and use :durable option:
|
179
|
+
To declare a durable shared queue, you pass a queue name that is a non-blank string and use the ":durable" option:
|
117
180
|
|
181
|
+
<pre>
|
182
|
+
<code>
|
183
|
+
queue = AMQP::Queue.new(channel, "images.resize", :durable => true)
|
184
|
+
</code>
|
185
|
+
</pre>
|
186
|
+
|
187
|
+
Full example:
|
118
188
|
<script src="https://gist.github.com/998723.js"> </script>
|
119
189
|
|
120
|
-
the same
|
190
|
+
the same example rewritten to use {AMQP::Channel#queue}:
|
191
|
+
|
192
|
+
<pre>
|
193
|
+
<code>
|
194
|
+
channel.queue("images.resize", :durable => true) do |queue, declare_ok|
|
195
|
+
puts "#{queue.name} is ready to go."
|
196
|
+
end
|
197
|
+
</code>
|
198
|
+
</pre>
|
121
199
|
|
122
200
|
<script src="https://gist.github.com/998724.js"> </script>
|
123
201
|
|
@@ -125,26 +203,72 @@ the same piece of code that uses {AMQP::Channel#queue} for convenience:
|
|
125
203
|
|
126
204
|
h2. Declaring a temporary exclusive queue
|
127
205
|
|
128
|
-
To declare a server-named, exclusive, auto-deleted queue, pass "" (empty string) as queue name and
|
129
|
-
use :exclusive and :auto_delete options:
|
206
|
+
To declare a server-named, exclusive, auto-deleted queue, pass "" (empty string) as the queue name and
|
207
|
+
use the ":exclusive" and ":auto_delete" options:
|
208
|
+
|
209
|
+
<pre>
|
210
|
+
<code>
|
211
|
+
AMQP::Queue.new(channel, "", :auto_delete => true, :exclusive => true) do |queue, declare_ok|
|
212
|
+
puts "#{queue.name} is ready to go."
|
213
|
+
end
|
214
|
+
</code>
|
215
|
+
</pre>
|
216
|
+
|
217
|
+
Full example:
|
130
218
|
|
131
219
|
<script src="https://gist.github.com/998725.js"> </script>
|
132
220
|
|
133
|
-
|
221
|
+
The same example can be rewritten to use {AMQP::Channel#queue}:
|
222
|
+
|
223
|
+
<pre>
|
224
|
+
<code>
|
225
|
+
channel.queue("", :auto_delete => true, :exclusive => true) do |queue, declare_ok|
|
226
|
+
puts "#{queue.name} is ready to go."
|
227
|
+
end
|
228
|
+
</code>
|
229
|
+
</pre>
|
230
|
+
|
231
|
+
Full example:
|
134
232
|
|
135
233
|
<script src="https://gist.github.com/998726.js"> </script>
|
136
234
|
|
235
|
+
Exclusive queues may only be accessed by the current connection and are deleted when that connection closes.
|
236
|
+
The declaration of an exclusive queue by other connections is not allowed and will result in a channel-level exception
|
237
|
+
with the code 405 (RESOURCE_LOCKED) and a reply message similar to
|
238
|
+
|
239
|
+
<pre>RESOURCE_LOCKED - cannot obtain exclusive access to locked queue 'amqpgem.examples.queue' in vhost '/'</pre>
|
240
|
+
|
241
|
+
The following example demonstrates this:
|
242
|
+
<script src="https://gist.github.com/1008529.js"> </script>
|
243
|
+
|
137
244
|
|
138
245
|
|
139
246
|
h2. Binding queues to exchanges
|
140
247
|
|
141
|
-
In order to receive messages, a queue needs to be bound to at least one exchange. Most of the time binding is explcit
|
142
|
-
To bind a queue to an exchange, use {AMQP::Queue#bind}
|
248
|
+
In order to receive messages, a queue needs to be bound to at least one exchange. Most of the time binding is explcit
|
249
|
+
(done by applications). To bind a queue to an exchange, use {AMQP::Queue#bind} where the argument passed can be
|
250
|
+
either an {AMQP::Exchange} instance or a string.
|
143
251
|
|
252
|
+
<pre>
|
253
|
+
<code>
|
254
|
+
queue.bind(exchange) do |bind_ok|
|
255
|
+
puts "Just bound #{queue.name} to #{exchange.name}"
|
256
|
+
end
|
257
|
+
</code>
|
258
|
+
</pre>
|
259
|
+
|
260
|
+
Full example:
|
144
261
|
<script src="https://gist.github.com/998727.js"> </script>
|
145
262
|
|
146
|
-
|
263
|
+
The same example using a string without callback:
|
147
264
|
|
265
|
+
<pre>
|
266
|
+
<code>
|
267
|
+
queue.bind("amq.fanout")
|
268
|
+
</code>
|
269
|
+
</pre>
|
270
|
+
|
271
|
+
Full example:
|
148
272
|
<script src="https://gist.github.com/998729.js"> </script>
|
149
273
|
|
150
274
|
|
@@ -152,24 +276,35 @@ or an exchange name given as a string:
|
|
152
276
|
|
153
277
|
h2. Subscribing to receive messages ("push API")
|
154
278
|
|
155
|
-
To
|
156
|
-
Then when a message arrives, message header and body (
|
279
|
+
To set up a queue subscription to enable an application to receive messages as they arrive in a queue, one uses the
|
280
|
+
{AMQP::Queue#subscribe} method. Then when a message arrives, the message header (metadata) and body (payload) are
|
281
|
+
passed to the handler:
|
282
|
+
|
283
|
+
<pre>
|
284
|
+
<code>
|
285
|
+
queue.subscribe do |metadata, payload|
|
286
|
+
puts "Received a message: #{payload.inspect}."
|
287
|
+
end
|
288
|
+
</code>
|
289
|
+
</pre>
|
157
290
|
|
291
|
+
Full example:
|
158
292
|
<script src="https://gist.github.com/998731.js"> </script>
|
159
293
|
|
160
|
-
Subscriptions for message delivery are usually referred to as
|
161
|
-
Consumers last as long as the channel they were declared on, or until the
|
294
|
+
Subscriptions for message delivery are usually referred to as _consumers_ in the AMQP v0.9.1 specification, client
|
295
|
+
library documentation and books. Consumers last as long as the channel that they were declared on, or until the
|
296
|
+
client cancels them (unsubscribes).
|
162
297
|
|
163
|
-
Consumers are identified by <i>consumer tags</i>. If you need to obtain consumer tag of a
|
164
|
-
use {AMQP::Queue#consumer_tag}.
|
298
|
+
Consumers are identified by <i>consumer tags</i>. If you need to obtain the consumer tag of a subscribed
|
299
|
+
queue then use {AMQP::Queue#consumer_tag}.
|
165
300
|
|
166
301
|
|
167
302
|
h3. Accessing message metadata
|
168
303
|
|
169
|
-
`header` object in the example above provides access to message metadata and delivery information:
|
304
|
+
The `header` object in the example above provides access to message metadata and delivery information:
|
170
305
|
|
171
306
|
* Message content type
|
172
|
-
* Message content
|
307
|
+
* Message content encoding
|
173
308
|
* Message routing key
|
174
309
|
* Message delivery mode (persistent or not)
|
175
310
|
* Consumer tag this delivery is for
|
@@ -180,16 +315,59 @@ h3. Accessing message metadata
|
|
180
315
|
|
181
316
|
and so on. An example to demonstrate how to access some of those attributes:
|
182
317
|
|
318
|
+
<pre>
|
319
|
+
<code>
|
320
|
+
# producer
|
321
|
+
exchange.publish("Hello, world!",
|
322
|
+
:app_id => "amqpgem.example",
|
323
|
+
:priority => 8,
|
324
|
+
:type => "kinda.checkin",
|
325
|
+
# headers table keys can be anything
|
326
|
+
:headers => {
|
327
|
+
:coordinates => {
|
328
|
+
:latitude => 59.35,
|
329
|
+
:longitude => 18.066667
|
330
|
+
},
|
331
|
+
:participants => 11,
|
332
|
+
:venue => "Stockholm"
|
333
|
+
},
|
334
|
+
:timestamp => Time.now.to_i)
|
335
|
+
</code>
|
336
|
+
</pre>
|
337
|
+
|
338
|
+
<pre>
|
339
|
+
<code>
|
340
|
+
# consumer
|
341
|
+
queue.subscribe do |metadata, payload|
|
342
|
+
puts "metadata.routing_key : #{metadata.routing_key}"
|
343
|
+
puts "metadata.content_type: #{metadata.content_type}"
|
344
|
+
puts "metadata.priority : #{metadata.priority}"
|
345
|
+
puts "metadata.headers : #{metadata.headers.inspect}"
|
346
|
+
puts "metadata.timestamp : #{metadata.timestamp.inspect}"
|
347
|
+
puts "metadata.type : #{metadata.type}"
|
348
|
+
puts "metadata.delivery_tag: #{metadata.delivery_tag}"
|
349
|
+
puts "metadata.redelivered : #{metadata.redelivered?}"
|
350
|
+
|
351
|
+
puts "metadata.app_id : #{metadata.app_id}"
|
352
|
+
puts "metadata.exchange : #{metadata.exchange}"
|
353
|
+
puts
|
354
|
+
puts "Received a message: #{payload}."
|
355
|
+
end
|
356
|
+
</code>
|
357
|
+
</pre>
|
358
|
+
|
359
|
+
Full example:
|
183
360
|
<script src="https://gist.github.com/998739.js"> </script>
|
184
361
|
|
185
362
|
|
186
363
|
h3. Exclusive consumers
|
187
364
|
|
188
|
-
Consumers can request exclusive access to the queue (meaning only this consumer can access the queue). This is useful
|
189
|
-
queue to be temporarily accessible by just one application (or thread, or process).
|
190
|
-
|
365
|
+
Consumers can request exclusive access to the queue (meaning only this consumer can access the queue). This is useful
|
366
|
+
when you want a long-lived shared queue to be temporarily accessible by just one application (or thread, or process).
|
367
|
+
If the application employing the exclusive consumer crashes or loses the TCP connection to the broker, then the
|
368
|
+
channel is closed and the exclusive consumer is cancelled.
|
191
369
|
|
192
|
-
To exclusively receive messages from the queue, pass :exclusive option to {AMQP::Queue#subscribe}:
|
370
|
+
To exclusively receive messages from the queue, pass the ":exclusive" option to {AMQP::Queue#subscribe}:
|
193
371
|
|
194
372
|
<pre>
|
195
373
|
<code>
|
@@ -202,24 +380,90 @@ end
|
|
202
380
|
TBD: describe what happens when exclusivity property is violated and how to handle it.
|
203
381
|
|
204
382
|
|
383
|
+
h3. Using multiple consumers per queue
|
384
|
+
|
385
|
+
Historically, amqp gem versions before v0.8.0.RC14 (current master branch in the repository) have had a "one consumer
|
386
|
+
per Queue instance" limitation. Previously, to work around this problem, application developers had to open multiple
|
387
|
+
channels and work with multiple queue instances on different channels. This is not very convenient and is surprising
|
388
|
+
for developers familiar with AMQP clients for other languages.
|
389
|
+
|
390
|
+
With more and more Ruby implementations dropping the "GIL":http://en.wikipedia.org/wiki/Global_Interpreter_Lock,
|
391
|
+
load balancing between multiple consumers in the same queue in the same OS process has become more and more common.
|
392
|
+
In certain cases, even applications that do not need any concurrency benefit from having multiple consumers on the
|
393
|
+
same queue in the same process.
|
394
|
+
|
395
|
+
Starting from amqp gem v0.8.0.RC14, it is possible to add any number of consumers by instantiating {AMQP::Consumer} directly:
|
396
|
+
|
397
|
+
<pre>
|
398
|
+
<code>
|
399
|
+
# non-exclusive consumer, consumer tag is generated
|
400
|
+
consumer1 = AMQP::Consumer.new(channel, queue)
|
401
|
+
|
402
|
+
# non-exclusive consumer, consumer tag is explicitly given
|
403
|
+
consumer2 = AMQP::Consumer.new(channel, queue, "#{queue.name}-consumer-#{rand}-#{Time.now}")
|
404
|
+
|
405
|
+
# exclusive consumer, consumer tag is generated
|
406
|
+
consumer3 = AMQP::Consumer.new(channel, queue, nil, true)
|
407
|
+
</code>
|
408
|
+
</pre>
|
409
|
+
|
410
|
+
Instantiated consumers do not begin consuming messages immediately. This is because in certain cases, it is useful to
|
411
|
+
add a consumer but make it active at a later time. To consume messages, use the {AMQP::Consumer#consume} method in
|
412
|
+
combination with {AMQP::Consumer#on_delivery}:
|
413
|
+
|
414
|
+
<pre>
|
415
|
+
<code>
|
416
|
+
consumer1.consume.on_delivery do |metadata, payload|
|
417
|
+
@consumer1_mailbox << payload
|
418
|
+
end
|
419
|
+
</code>
|
420
|
+
</pre>
|
421
|
+
|
422
|
+
{AMQP::Consumer#on_delivery} takes a block that is used exactly like the block passed to {AMQP::Queue#subscribe}.
|
423
|
+
In fact, {AMQP::Queue#subscribe} uses {AMQP::Consumer} under the hood, adding a _default consumer_ to the queue.
|
424
|
+
|
425
|
+
<span class="note">
|
426
|
+
Default consumers do not have any special properties, they just provide a convenient way for application developers
|
427
|
+
to register multiple consumers and a means of preserving backwards compatibility. Application developers are
|
428
|
+
always free to use AMQP::Consumer instances directly, or intermix them with AMQP::Queue#subscribe.
|
429
|
+
</span>
|
430
|
+
|
431
|
+
Most of the public API methods on {AMQP::Consumer} return self, so it is possible to use method chaining extensively.
|
432
|
+
An example from "amqp gem spec suite":https://github.com/ruby-amqp/amqp/tree/master/spec:
|
433
|
+
|
434
|
+
<pre>
|
435
|
+
<code>
|
436
|
+
consumer1 = AMQP::Consumer.new(@channel, @queue).consume.on_delivery { |metadata, payload| mailbox1 << payload }
|
437
|
+
consumer2 = AMQP::Consumer.new(@channel, @queue).consume.on_delivery { |metadata, payload| mailbox2 << payload }
|
438
|
+
</code>
|
439
|
+
</pre>
|
440
|
+
|
441
|
+
To cancel a particular consumer, use {AMQP::Consumer#cancel} method. To cancel a default queue consumer, use {AMQP::Queue#unsubscribe}.
|
442
|
+
|
443
|
+
|
205
444
|
h3. Message acknowledgements
|
206
445
|
|
207
|
-
Consumer applications
|
208
|
-
crash.
|
209
|
-
|
446
|
+
Consumer applications - applications that receive and process messages - may occasionally fail to process individual
|
447
|
+
messages, or will just crash. There is also the possibility of network issues causing problems. This raises a
|
448
|
+
question - "When should the AMQP broker remove messages from queues?" The AMQP v0.9.1 specification proposes
|
449
|
+
two choices:
|
210
450
|
|
211
451
|
* After broker sends a message to an application (using either basic.deliver or basic.get-ok methods).
|
212
452
|
* After the application sends back an acknowledgement (using basic.ack AMQP method).
|
213
453
|
|
214
|
-
The former
|
215
|
-
|
216
|
-
|
454
|
+
The former choice is called the *automatic acknowledgement model*, while the latter is called the *explicit
|
455
|
+
acknowledgement model*. With the explicit model, the application chooses when it is time to send an acknowledgement.
|
456
|
+
It can be right after receiving a message, or after persisting it to a data store before processing, or after fully
|
457
|
+
processing the message (for example, successfully fetching a Web page, processing and storing it into some persistent
|
458
|
+
data store).
|
217
459
|
|
218
|
-
If a consumer dies without sending an
|
219
|
-
until at least one consumer is registered for the same queue
|
460
|
+
If a consumer dies without sending an acknowledgement, the AMQP broker will redeliver it to another consumer, or, if
|
461
|
+
none are available at the time, the broker will wait until at least one consumer is registered for the same queue
|
462
|
+
before attempting redelivery.
|
220
463
|
|
221
|
-
|
222
|
-
To switch to the *explicit* model, :ack option should
|
464
|
+
The acknowledgement model is chosen when a new consumer is registered for a queue. By default,
|
465
|
+
{AMQP::Queue#subscribe} will use the *automatic* model. To switch to the *explicit* model, the ":ack" option should
|
466
|
+
be used:
|
223
467
|
|
224
468
|
<pre>
|
225
469
|
<code>
|
@@ -229,16 +473,16 @@ end
|
|
229
473
|
</code>
|
230
474
|
</pre>
|
231
475
|
|
232
|
-
To demonstrate how redelivery works,
|
476
|
+
To demonstrate how redelivery works, let us have a look at the following code example:
|
233
477
|
|
234
478
|
<script src="https://gist.github.com/999396.js"> </script>
|
235
479
|
|
236
|
-
So what is going on here? This example uses 3 AMQP connections to imitate 3 applications, 1 producer and two
|
237
|
-
channel:
|
480
|
+
So what is going on here? This example uses 3 AMQP connections to imitate 3 applications, 1 producer and two
|
481
|
+
consumers. Each AMQP connection opens a single channel:
|
238
482
|
|
239
483
|
<pre>
|
240
484
|
<code>
|
241
|
-
# open
|
485
|
+
# open three connections to imitate three apps
|
242
486
|
connection1 = AMQP.connect
|
243
487
|
connection2 = AMQP.connect
|
244
488
|
connection3 = AMQP.connect
|
@@ -260,9 +504,10 @@ channel3.on_error(&channel_exception_handler)
|
|
260
504
|
</code>
|
261
505
|
</pre>
|
262
506
|
|
263
|
-
|
264
|
-
|
265
|
-
|
507
|
+
The consumers share a queue and the producer publishes messages to the queue periodically using an `amq.direct`
|
508
|
+
exchange. Both "applications" subscribe to receive messages using the explicit acknowledgement model. The AMQP broker
|
509
|
+
by default will send each message to the next consumer in sequence (this kind of load balancing is known as
|
510
|
+
*round-robin*). This means that some messages will be delivered to consumer #1 and some to consumer #2.
|
266
511
|
|
267
512
|
<pre>
|
268
513
|
<code>
|
@@ -271,7 +516,7 @@ exchange = channel3.direct("amq.direct")
|
|
271
516
|
# ...
|
272
517
|
|
273
518
|
queue1 = channel1.queue("amqpgem.examples.acknowledgements.explicit", :auto_delete => false)
|
274
|
-
# purge the queue so that we
|
519
|
+
# purge the queue so that we do not get any redeliveries from previous runs
|
275
520
|
queue1.purge
|
276
521
|
queue1.bind(exchange).subscribe(:ack => true) do |metadata, payload|
|
277
522
|
# do some work
|
@@ -285,7 +530,7 @@ queue1.bind(exchange).subscribe(:ack => true) do |metadata, payload|
|
|
285
530
|
else
|
286
531
|
# odd messages are not ack-ed and will remain in the queue for redelivery
|
287
532
|
# when app #1 connection is closed (either properly or due to a crash)
|
288
|
-
puts "[consumer1] Got message ##{metadata.headers['i']},
|
533
|
+
puts "[consumer1] Got message ##{metadata.headers['i']}, SKIPPED"
|
289
534
|
end
|
290
535
|
end
|
291
536
|
|
@@ -298,21 +543,22 @@ end
|
|
298
543
|
</code>
|
299
544
|
</pre>
|
300
545
|
|
301
|
-
To demonstrate message redelivery we make consumer #1 randomly select
|
302
|
-
a crash). When that happens, AMQP broker redelivers unacknowledged messages to
|
303
|
-
this example closes all outstanding
|
546
|
+
To demonstrate message redelivery we make consumer #1 randomly select which messages to acknowledge. After 4 seconds
|
547
|
+
we disconnect it (to imitate a crash). When that happens, the AMQP broker redelivers unacknowledged messages to
|
548
|
+
consumer #2 which acknowledges them unconditionally. After 10 seconds, this example closes all outstanding
|
549
|
+
connections and exits.
|
304
550
|
|
305
|
-
An
|
551
|
+
An extract of output produced by this example:
|
306
552
|
|
307
553
|
<pre>
|
308
554
|
=> Subscribing for messages using explicit acknowledgements model
|
309
555
|
|
310
556
|
[consumer2] Received Message #0, redelivered = false, ack-ed
|
311
|
-
[consumer1] Got message #1,
|
312
|
-
[consumer1] Got message #2,
|
557
|
+
[consumer1] Got message #1, SKIPPED
|
558
|
+
[consumer1] Got message #2, SKIPPED
|
313
559
|
[consumer1] Got message #3, ack-ed
|
314
560
|
[consumer2] Received Message #4, redelivered = false, ack-ed
|
315
|
-
[consumer1] Got message #5,
|
561
|
+
[consumer1] Got message #5, SKIPPED
|
316
562
|
[consumer2] Received Message #6, redelivered = false, ack-ed
|
317
563
|
[consumer2] Received Message #7, redelivered = false, ack-ed
|
318
564
|
[consumer2] Received Message #8, redelivered = false, ack-ed
|
@@ -352,10 +598,10 @@ An example output produced by this example:
|
|
352
598
|
As we can see, consumer #1 did not acknowledge 3 messages (labelled 1, 2 and 5):
|
353
599
|
|
354
600
|
<pre>
|
355
|
-
[consumer1] Got message #1,
|
356
|
-
[consumer1] Got message #2,
|
601
|
+
[consumer1] Got message #1, SKIPPED
|
602
|
+
[consumer1] Got message #2, SKIPPED
|
357
603
|
...
|
358
|
-
[consumer1] Got message #5,
|
604
|
+
[consumer1] Got message #5, SKIPPED
|
359
605
|
</pre>
|
360
606
|
|
361
607
|
and then, once consumer #1 had "crashed", those messages were immediately redelivered to the consumer #2:
|
@@ -367,7 +613,7 @@ Connection 1 is now closed (we pretend that it has crashed)
|
|
367
613
|
[consumer2] Received Message #2, redelivered = true, ack-ed
|
368
614
|
</pre>
|
369
615
|
|
370
|
-
To acknowledge a message
|
616
|
+
To acknowledge a message use {AMQP::Channel#acknowledge}:
|
371
617
|
|
372
618
|
<pre>
|
373
619
|
<code>
|
@@ -375,13 +621,14 @@ channel1.acknowledge(metadata.delivery_tag, false)
|
|
375
621
|
</code>
|
376
622
|
</pre>
|
377
623
|
|
378
|
-
{AMQP::Channel#acknowledge} takes two arguments: message *delivery tag* and a flag that indicates whether or not we
|
379
|
-
multiple messages at once. Delivery tag is simply a channel-specific increasing number that
|
624
|
+
{AMQP::Channel#acknowledge} takes two arguments: message *delivery tag* and a flag that indicates whether or not we
|
625
|
+
want to acknowledge multiple messages at once. Delivery tag is simply a channel-specific increasing number that
|
626
|
+
the server uses to identify deliveries.
|
380
627
|
|
381
|
-
When acknowledging multiple messages at once, the delivery tag is treated as "up to and including". For example,
|
382
|
-
tag = 5 that would mean "acknowledge messages 1, 2, 3, 4 and 5".
|
628
|
+
When acknowledging multiple messages at once, the delivery tag is treated as "up to and including". For example, if
|
629
|
+
delivery tag = 5 that would mean "acknowledge messages 1, 2, 3, 4 and 5".
|
383
630
|
|
384
|
-
As a shortcut, it is possible to acknowledge messages using {AMQP::Header#ack} method:
|
631
|
+
As a shortcut, it is possible to acknowledge messages using the {AMQP::Header#ack} method:
|
385
632
|
|
386
633
|
<pre>
|
387
634
|
<code>
|
@@ -392,48 +639,49 @@ end
|
|
392
639
|
</pre>
|
393
640
|
|
394
641
|
<span class="note">
|
395
|
-
Acknowledgements are channel-specific. Applications must not receive messages on one channel and acknowledge them on
|
642
|
+
Acknowledgements are channel-specific. Applications must not receive messages on one channel and acknowledge them on
|
643
|
+
another.
|
396
644
|
</span>
|
397
645
|
|
398
646
|
<span class="note">
|
399
|
-
A message MUST not be acknowledged more than once. Doing so will result in a channel-level exception
|
400
|
-
with an error message like this: «PRECONDITION_FAILED - unknown delivery tag»
|
647
|
+
A message MUST not be acknowledged more than once. Doing so will result in a channel-level exception
|
648
|
+
(PRECONDITION_FAILED) with an error message like this: «PRECONDITION_FAILED - unknown delivery tag»
|
401
649
|
</span>
|
402
650
|
|
403
651
|
|
404
652
|
|
405
|
-
h3. Rejecting messages
|
653
|
+
h3. Rejecting messages
|
406
654
|
|
407
|
-
When a consumer application receives a message, processing of that message may or may not succeed.
|
408
|
-
indicate
|
409
|
-
|
655
|
+
When a consumer application receives a message, processing of that message may or may not succeed. An application can
|
656
|
+
indicate to the broker that message processing has failed (or cannot be accomplished at the time) by rejecting a
|
657
|
+
message. When rejecting a message, an application can ask the broker to discard or requeue it.
|
410
658
|
|
411
|
-
To reject a message
|
659
|
+
To reject a message use the {AMQP::Channel#reject} method:
|
412
660
|
|
413
661
|
<pre>
|
414
662
|
<code>
|
415
663
|
queue.bind(exchange).subscribe do |metadata, payload|
|
416
|
-
# reject but
|
664
|
+
# reject but do not requeue (simply discard)
|
417
665
|
channel.reject(metadata.delivery_tag)
|
418
666
|
end
|
419
667
|
</code>
|
420
668
|
</pre>
|
421
669
|
|
422
|
-
in the example above messages are rejected without requeueing (broker will simply discard them). To requeue
|
423
|
-
use the second argument {AMQP::Channel#reject} takes:
|
670
|
+
in the example above, messages are rejected without requeueing (broker will simply discard them). To requeue a
|
671
|
+
rejected message, use the second argument that {AMQP::Channel#reject} takes:
|
424
672
|
|
425
673
|
<pre>
|
426
674
|
<code>
|
427
675
|
queue.bind(exchange).subscribe do |metadata, payload|
|
428
|
-
# reject
|
676
|
+
# reject and requeue
|
429
677
|
channel.reject(metadata.delivery_tag, true)
|
430
678
|
end
|
431
679
|
</code>
|
432
680
|
</pre>
|
433
681
|
|
434
682
|
<span class="note">
|
435
|
-
When there is only one consumer on a queue, make sure you
|
436
|
-
message from
|
683
|
+
When there is only one consumer on a queue, make sure you do not create infinite message delivery loops by rejecting
|
684
|
+
and requeueing a message from the same consumer over and over again.
|
437
685
|
</span>
|
438
686
|
|
439
687
|
Another way to reject a message is by using {AMQP::Header#reject}:
|
@@ -441,7 +689,7 @@ Another way to reject a message is by using {AMQP::Header#reject}:
|
|
441
689
|
<pre>
|
442
690
|
<code>
|
443
691
|
queue.bind(exchange).subscribe do |metadata, payload|
|
444
|
-
# reject but
|
692
|
+
# reject but do not requeue (simply discard)
|
445
693
|
metadata.reject
|
446
694
|
end
|
447
695
|
</code>
|
@@ -450,66 +698,73 @@ end
|
|
450
698
|
<pre>
|
451
699
|
<code>
|
452
700
|
queue.bind(exchange).subscribe do |metadata, payload|
|
453
|
-
# reject
|
701
|
+
# reject and requeue
|
454
702
|
metadata.reject(true)
|
455
703
|
end
|
456
704
|
</code>
|
457
705
|
</pre>
|
458
706
|
|
459
707
|
|
460
|
-
h3. Negative acknowledgements
|
708
|
+
h3. Negative acknowledgements
|
461
709
|
|
462
|
-
Messages are rejected with `basic.reject` AMQP method. There is one limitation `basic.reject` has: there
|
463
|
-
messages,
|
464
|
-
an AMQP
|
465
|
-
|
710
|
+
Messages are rejected with the `basic.reject` AMQP method. There is one limitation that `basic.reject` has: there
|
711
|
+
is no way to reject multiple messages, as you can do with acknowledgements. However, if you are using
|
712
|
+
"RabbitMQ":http://rabbitmq.com, then there is a solution. RabbitMQ provides an AMQP v0.9.1 extension known as
|
713
|
+
"negative acknowledgements":http://www.rabbitmq.com/extensions.html#negative-acknowledgements (nacks) and
|
714
|
+
the amqp gem supports this extension. For more information, please refer to the
|
715
|
+
{file:docs/VendorSpecificExtensions.textile Vendor-specific Extensions guide}.
|
466
716
|
|
467
717
|
|
468
|
-
h3. QoS
|
718
|
+
h3. QoS - Prefetching messages
|
469
719
|
|
470
|
-
For cases when multiple consumers share a queue, it is useful to be able to specify how many messages each consumer
|
471
|
-
|
472
|
-
to be published in batches
|
720
|
+
For cases when multiple consumers share a queue, it is useful to be able to specify how many messages each consumer
|
721
|
+
can be sent at once before sending the next acknowledgement. This can be used as a simple load balancing technique or
|
722
|
+
to improve throughput if messages tend to be published in batches. For example, if a producing application sends
|
723
|
+
messages every minute because of the nature of the work it is doing.
|
473
724
|
|
474
|
-
Imagine a
|
475
|
-
calculates how many tweets
|
725
|
+
Imagine a website that takes data from social media sources like Twitter or Facebook during the Champions League
|
726
|
+
final (or the Superbowl), and then calculates how many tweets mention a particular team during the last minute.
|
727
|
+
The site could be structured as 3 applications:
|
476
728
|
|
477
729
|
* A crawler that uses streaming APIs to fetch tweets/statuses, normalizes them and sends them in JSON for processing by other applications ("app A").
|
478
730
|
* A calculator that detects what team is mentioned in a message, updates statistics and pushes an update to the Web UI once a minute ("app B").
|
479
731
|
* A Web UI that fans visit to see the stats ("app C").
|
480
732
|
|
481
|
-
In this imaginary example, tweets per second rate will vary but to improve throughput of the system and
|
482
|
-
|
483
|
-
|
484
|
-
|
733
|
+
In this imaginary example, the "tweets per second" rate will vary, but to improve the throughput of the system and
|
734
|
+
to decrease the maximum number of messages that the AMQP broker has to hold in memory at once, applications can be
|
735
|
+
designed in such a way that application "app B", the "calculator", receives 5000 messages and then acknowledges them
|
736
|
+
all at once. The broker will not send message 5001 unless it receives an acknowledgement.
|
485
737
|
|
486
|
-
In AMQP parlance this is know as *QoS* or *message prefetching*. Prefetching is configured on per-channel
|
487
|
-
basis. To configure prefetching per channel, use
|
488
|
-
section:
|
738
|
+
In AMQP parlance this is know as *QoS* or *message prefetching*. Prefetching is configured on a per-channel
|
739
|
+
(typically) or per-connection (rarely used) basis. To configure prefetching per channel, use
|
740
|
+
the {AMQP::Channel#prefetch} method. Let us return to the example we used in the "Message acknowledgements" section:
|
489
741
|
|
490
742
|
<pre>
|
491
743
|
<code>
|
492
|
-
#
|
493
|
-
#
|
494
|
-
#
|
744
|
+
# app #1 will be given up to 3 messages at a time. If it does not
|
745
|
+
# send an ack after receiving the messages, then the messages will
|
746
|
+
# be routed to app #2.
|
495
747
|
channel1.prefetch(3)
|
496
748
|
|
497
|
-
# app #2 processes messages one-by-one and has to send
|
749
|
+
# app #2 processes messages one-by-one and has to send an ack after receiving each message
|
498
750
|
channel2.prefetch(1)
|
499
751
|
</code>
|
500
752
|
</pre>
|
501
753
|
|
502
|
-
In that example, one consumer prefetches 3 messages and another consumer prefetches just 1. If we take a look at the output that example produces,
|
503
|
-
|
754
|
+
In that example, one consumer prefetches 3 messages and another consumer prefetches just 1. If we take a look at the output that the example produces, we will see that `consumer1` fetched 4 messages and acknowledged 1. After that,
|
755
|
+
all subsequent messages were delivered to `consumer2`:
|
504
756
|
|
505
757
|
<pre>
|
506
758
|
[consumer2] Received Message #0, redelivered = false, ack-ed
|
507
|
-
[consumer1] Got message #1,
|
508
|
-
[consumer1] Got message #2,
|
759
|
+
[consumer1] Got message #1, SKIPPED
|
760
|
+
[consumer1] Got message #2, SKIPPED
|
509
761
|
[consumer1] Got message #3, ack-ed
|
510
762
|
[consumer2] Received Message #4, redelivered = false, ack-ed
|
511
|
-
[consumer1] Got message #5,
|
512
|
-
---
|
763
|
+
[consumer1] Got message #5, SKIPPED
|
764
|
+
---
|
765
|
+
by now consumer 1 has received 3 messages it did not acknowledge.
|
766
|
+
With prefetch = 3, AMQP broker will not send it any more messages until consumer 1 sends an ack
|
767
|
+
---
|
513
768
|
[consumer2] Received Message #6, redelivered = false, ack-ed
|
514
769
|
[consumer2] Received Message #7, redelivered = false, ack-ed
|
515
770
|
[consumer2] Received Message #8, redelivered = false, ack-ed
|
@@ -519,68 +774,367 @@ we will see that `consumer1` fetched 4 messages and acknowledged 1. After that,
|
|
519
774
|
</pre>
|
520
775
|
|
521
776
|
<span class="note">
|
522
|
-
|
777
|
+
The prefetching setting is ignored for consumers that do not use explicit acknowledgements.
|
523
778
|
</span>
|
524
779
|
|
525
780
|
|
781
|
+
h2. How message acknowledgements relate to transactions and Publisher Confirms
|
782
|
+
|
783
|
+
In cases where you cannot afford to lose a single message, AMQP v0.9.1 applications can use one or a combination of
|
784
|
+
the following protocol features:
|
785
|
+
|
786
|
+
* Publisher confirms (a RabbitMQ-specific extension to AMQP v0.9.1)
|
787
|
+
* Publishing messages as immediate
|
788
|
+
* Transactions (noticeable overhead)
|
789
|
+
|
790
|
+
This topic is covered in depth in the {file:docs/Exchanges.textile Working With Exchanges} guide.
|
791
|
+
In this guide, we will only mention how message acknowledgements are related to AMQP transactions and the Publisher
|
792
|
+
Confirms extension.
|
793
|
+
|
794
|
+
Let us consider a publisher application (P) that communications with a consumer (C) using AMQP v0.9.1. Their
|
795
|
+
communication can be graphically represented like this:
|
796
|
+
|
797
|
+
<pre>
|
798
|
+
----- ----- -----
|
799
|
+
| | S1 | | S2 | |
|
800
|
+
| P | ====> | B | ====> | C |
|
801
|
+
| | | | | |
|
802
|
+
----- ----- -----
|
803
|
+
</pre>
|
804
|
+
|
805
|
+
We have two network segments, S1 and S2. Each of them may fail. P is concerned with making sure that
|
806
|
+
messages cross S1, while broker (B) and C are concerned with ensuring that messages cross S2 and are only removed
|
807
|
+
from the queue when they are processed successfully.
|
808
|
+
|
809
|
+
Message acknowledgements cover reliable delivery over S2 as well as successful processing. For S1, P has to use
|
810
|
+
transactions (a heavyweight solution) or the more lightweight Publisher Confirms RabbitMQ extension.
|
811
|
+
|
526
812
|
|
527
813
|
h2. Fetching messages when needed ("pull API")
|
528
814
|
|
529
|
-
AMQP
|
530
|
-
{AMQP::Queue#pop}:
|
815
|
+
The AMQP v0.9.1 specification also provides a way for applications to fetch (pull) messages from the queue only
|
816
|
+
when necessary. For that, use {AMQP::Queue#pop}:
|
817
|
+
|
818
|
+
<pre>
|
819
|
+
<code>
|
820
|
+
queue.pop do |metadata, payload|
|
821
|
+
if payload
|
822
|
+
puts "Fetched a message: #{payload.inspect}, content_type: #{metadata.content_type}. Shutting down..."
|
823
|
+
else
|
824
|
+
puts "No messages in the queue"
|
825
|
+
end
|
826
|
+
end
|
827
|
+
</code>
|
828
|
+
</pre>
|
531
829
|
|
830
|
+
Full example:
|
532
831
|
<script src="https://gist.github.com/998732.js"> </script>
|
533
832
|
|
534
|
-
|
833
|
+
If the queue is empty, then the `payload` argument will be nil, otherwise arguments are identical to those of
|
834
|
+
the {AMQP::Queue#subscribe} callback.
|
535
835
|
|
536
836
|
|
537
837
|
h2. Unsubscribing from messages
|
538
838
|
|
539
|
-
Sometimes it is necessary to unsubscribe from messages without deleting a queue. To do that, use
|
839
|
+
Sometimes it is necessary to unsubscribe from messages without deleting a queue. To do that, use
|
840
|
+
the {AMQP::Queue#unsubscribe} method:
|
841
|
+
|
842
|
+
<pre>
|
843
|
+
<code>
|
844
|
+
queue.unsubscribe
|
845
|
+
</code>
|
846
|
+
</pre>
|
847
|
+
|
848
|
+
By default {AMQP::Queue#unsubscribe} uses the ":noack" option to inform the broker that there is no need to send
|
849
|
+
a confirmation. In other words, it does not expect you to pass in a callback, because the consumer tag on the queue
|
850
|
+
instance and the registered callback for messages are cleared immediately.
|
851
|
+
|
852
|
+
If an application needs to execute a piece of code after the broker response arrives, {AMQP::Queue#unsubscribe} takes
|
853
|
+
an optional callback:
|
540
854
|
|
855
|
+
<pre>
|
856
|
+
<code>
|
857
|
+
queue.unsubscribe do |unbind_ok|
|
858
|
+
# server response arrived, handle it if necessary...
|
859
|
+
end
|
860
|
+
</code>
|
861
|
+
</pre>
|
862
|
+
|
863
|
+
Full example:
|
541
864
|
<script src="https://gist.github.com/998734.js"> </script>
|
542
865
|
|
543
|
-
|
544
|
-
|
545
|
-
|
866
|
+
In AMQP parlance, unsubscribing from messages is often referred to as "cancelling a consumer". Once a consumer is
|
867
|
+
cancelled, messages will no longer be delivered to it, however, due to the asynchronous nature of the protocol,
|
868
|
+
it is possible for "in flight" messages to be received after this call completes.
|
869
|
+
|
870
|
+
Fetching messages with {AMQP::Queue#pop} is still possible even after a consumer is cancelled.
|
546
871
|
|
547
872
|
|
548
873
|
h2. Unbinding queues from exchanges
|
549
874
|
|
550
|
-
To unbind queue from exchange
|
875
|
+
To unbind a queue from an exchange use {AMQP::Queue#unbind}:
|
551
876
|
|
877
|
+
<pre>
|
878
|
+
<code>
|
879
|
+
queue.unbind(exchange)
|
880
|
+
</code>
|
881
|
+
</pre>
|
882
|
+
|
883
|
+
Full example:
|
552
884
|
<script src="https://gist.github.com/998742.js"> </script>
|
553
885
|
|
554
|
-
Note that
|
886
|
+
Note that trying to unbind a queue from an exchange that the queue was never bound to will result in a
|
887
|
+
channel-level exception.
|
888
|
+
|
889
|
+
|
890
|
+
h2. Querying the number of messages in a queue
|
891
|
+
|
892
|
+
It is possible to query the number of messages sitting in the queue by declaring the queue with the ":passive"
|
893
|
+
attribute set. The response (`queue.declare-ok` AMQP method) will include the number of messages along with other
|
894
|
+
attributes. However, the amqp gem provides a convenience method, {AMQP::Queue#status}:
|
895
|
+
|
896
|
+
<pre>
|
897
|
+
<code>
|
898
|
+
queue.status do |number_of_messages, number_of_consumers|
|
899
|
+
puts
|
900
|
+
puts "# of messages in the queue #{queue.name} = #{number_of_messages}"
|
901
|
+
puts
|
902
|
+
end
|
903
|
+
</code>
|
904
|
+
</pre>
|
905
|
+
|
906
|
+
Full example:
|
907
|
+
<script src="https://gist.github.com/1068363.js"> </script>
|
908
|
+
|
909
|
+
|
910
|
+
h2. Querying the number of consumers on a queue
|
911
|
+
|
912
|
+
It is possible to query the number of consumers on a queue by declaring the queue with the ":passive" attribute set.
|
913
|
+
The response (`queue.declare-ok` AMQP method) will include the number of consumers along with other attributes.
|
914
|
+
However, the amqp gem provides a convenience method, {AMQP::Queue#status}:
|
915
|
+
|
916
|
+
<pre>
|
917
|
+
<code>
|
918
|
+
queue.status do |number_of_messages, number_of_consumers|
|
919
|
+
puts
|
920
|
+
puts "# of consumers on the queue #{queue.name} = #{number_of_consumers}"
|
921
|
+
puts
|
922
|
+
end
|
923
|
+
</code>
|
924
|
+
</pre>
|
925
|
+
|
926
|
+
Full example:
|
927
|
+
<script src="https://gist.github.com/1068377.js"> </script>
|
555
928
|
|
556
929
|
|
557
930
|
h2. Purging queues
|
558
931
|
|
559
|
-
It is possible to purge a queue (remove all messages from it) using {AMQP::Queue#purge}:
|
932
|
+
It is possible to purge a queue (remove all of the messages from it) using {AMQP::Queue#purge}:
|
560
933
|
|
561
|
-
<
|
934
|
+
<pre>
|
935
|
+
<code>
|
936
|
+
queue.purge
|
937
|
+
</code>
|
938
|
+
</pre>
|
562
939
|
|
940
|
+
This method takes an optional callback. However, remember that this operation is performed asynchronously.
|
941
|
+
To run a piece of code when the AMQP broker confirms that a queue has been purged, use a callback that
|
942
|
+
{AMQP::Queue#purge} takes:
|
563
943
|
|
564
|
-
|
944
|
+
<pre>
|
945
|
+
<code>
|
946
|
+
queue.purge do |_|
|
947
|
+
puts "Purged #{queue.name}"
|
948
|
+
end
|
949
|
+
</code>
|
950
|
+
</pre>
|
565
951
|
|
952
|
+
Full example:
|
953
|
+
<script src="https://gist.github.com/998743.js"> </script>
|
954
|
+
|
955
|
+
Note that this example purges a newly declared queue with a unique server-generated name. When a queue is declared,
|
956
|
+
it is empty, so for server-named queues, there is no need to purge them before they are used.
|
566
957
|
|
567
958
|
|
568
959
|
h2. Deleting queues
|
569
960
|
|
570
|
-
To delete a queue, use {AMQP::Queue#delete}
|
961
|
+
To delete a queue, use {AMQP::Queue#delete}. When a queue is deleted, all of the messages in it are deleted as well.
|
962
|
+
|
963
|
+
<pre>
|
964
|
+
<code>
|
965
|
+
queue.delete
|
966
|
+
</code>
|
967
|
+
</pre>
|
968
|
+
|
969
|
+
This method takes an optional callback. However, remember that this operation is performed asynchronously.
|
970
|
+
To run a piece of code when the AMQP broker confirms that a queue has been deleted, use a callback that
|
971
|
+
{AMQP::Queue#delete} takes:
|
972
|
+
|
973
|
+
<pre>
|
974
|
+
<code>
|
975
|
+
queue.delete do |_|
|
976
|
+
puts "Deleted #{queue.name}"
|
977
|
+
end
|
978
|
+
</code>
|
979
|
+
</pre>
|
571
980
|
|
981
|
+
Full example:
|
572
982
|
<script src="https://gist.github.com/998744.js"> </script>
|
573
983
|
|
574
|
-
This method takes a callback but it is optional. However, remember that this operation is performed asynchronously.
|
575
984
|
|
576
|
-
|
985
|
+
h2. Objects as message consumers and unit testing consumers in isolation
|
577
986
|
|
987
|
+
Since Ruby is a genuine object-oriented language, it is important to demonstrate how the Ruby amqp gem can be
|
988
|
+
integrated into rich object-oriented code. This part of the guide focuses on queues and the problems/solutions
|
989
|
+
concerning consumer applications (applications that primarily receive and process messages, as opposed to producers
|
990
|
+
that publish them).
|
578
991
|
|
992
|
+
An {AMQP::Queue#subscribe} callback does not have to be a block. It can be any Ruby object that responds to the
|
993
|
+
`call` method. A common technique is to combine {http://rubydoc.info/stdlib/core/1.8.7/Object:method Object#method}
|
994
|
+
and {http://rubydoc.info/stdlib/core/1.8.7/Method:to_proc Method#to_proc} and use object methods as message handlers.
|
579
995
|
|
580
|
-
|
996
|
+
An example to demonstrate this technique:
|
997
|
+
|
998
|
+
<pre>
|
999
|
+
<code>
|
1000
|
+
class Consumer
|
1001
|
+
|
1002
|
+
#
|
1003
|
+
# API
|
1004
|
+
#
|
1005
|
+
|
1006
|
+
def initialize(channel, queue_name = AMQ::Protocol::EMPTY_STRING)
|
1007
|
+
@queue_name = queue_name
|
1008
|
+
|
1009
|
+
@channel = channel
|
1010
|
+
# Consumer#handle_channel_exception will handle channel
|
1011
|
+
# exceptions. Keep in mind that you can only register one error handler,
|
1012
|
+
# so the last one registered "wins".
|
1013
|
+
@channel.on_error(&method(:handle_channel_exception))
|
1014
|
+
end # initialize
|
1015
|
+
|
1016
|
+
def start
|
1017
|
+
@queue = @channel.queue(@queue_name, :exclusive => true)
|
1018
|
+
# #handle_message method will be handling messages routed to @queue
|
1019
|
+
@queue.subscribe(&method(:handle_message))
|
1020
|
+
end # start
|
1021
|
+
|
1022
|
+
|
1023
|
+
|
1024
|
+
#
|
1025
|
+
# Implementation
|
1026
|
+
#
|
1027
|
+
|
1028
|
+
def handle_message(metadata, payload)
|
1029
|
+
puts "Received a message: #{payload}, content_type = #{metadata.content_type}"
|
1030
|
+
end # handle_message(metadata, payload)
|
1031
|
+
|
1032
|
+
def handle_channel_exception(channel, channel_close)
|
1033
|
+
puts "Oops... a channel-level exception: code = #{channel_close.reply_code}, message = #{channel_close.reply_text}"
|
1034
|
+
end # handle_channel_exception(channel, channel_close)
|
1035
|
+
end
|
1036
|
+
</code>
|
1037
|
+
</pre>
|
1038
|
+
|
1039
|
+
Full example:
|
1040
|
+
|
1041
|
+
<script src="https://gist.github.com/1009425.js"> </script>
|
1042
|
+
|
1043
|
+
|
1044
|
+
In this example, `Consumer` instances have to be instantiated with an {AMQP::Channel} instance. If the message
|
1045
|
+
handling was done by an aggregated object, it would completely separate the handling logic and would be make it
|
1046
|
+
easy to unit test in isolation:
|
1047
|
+
|
1048
|
+
<pre>
|
1049
|
+
<code>
|
1050
|
+
class Consumer
|
1051
|
+
|
1052
|
+
#
|
1053
|
+
# API
|
1054
|
+
#
|
1055
|
+
|
1056
|
+
def handle_message(metadata, payload)
|
1057
|
+
puts "Received a message: #{payload}, content_type = #{metadata.content_type}"
|
1058
|
+
end # handle_message(metadata, payload)
|
1059
|
+
end
|
1060
|
+
|
1061
|
+
|
1062
|
+
class Worker
|
1063
|
+
|
1064
|
+
#
|
1065
|
+
# API
|
1066
|
+
#
|
581
1067
|
|
582
|
-
See {file:docs/Durability.textile Durability guide}
|
583
1068
|
|
1069
|
+
def initialize(channel, queue_name = AMQ::Protocol::EMPTY_STRING, consumer = Consumer.new)
|
1070
|
+
@queue_name = queue_name
|
1071
|
+
|
1072
|
+
@channel = channel
|
1073
|
+
@channel.on_error(&method(:handle_channel_exception))
|
1074
|
+
|
1075
|
+
@consumer = consumer
|
1076
|
+
end # initialize
|
1077
|
+
|
1078
|
+
def start
|
1079
|
+
@queue = @channel.queue(@queue_name, :exclusive => true)
|
1080
|
+
@queue.subscribe(&@consumer.method(:handle_message))
|
1081
|
+
end # start
|
1082
|
+
|
1083
|
+
|
1084
|
+
#
|
1085
|
+
# Implementation
|
1086
|
+
#
|
1087
|
+
|
1088
|
+
def handle_channel_exception(channel, channel_close)
|
1089
|
+
puts "Oops... a channel-level exception: code = #{channel_close.reply_code}, message = #{channel_close.reply_text}"
|
1090
|
+
end # handle_channel_exception(channel, channel_close)
|
1091
|
+
end
|
1092
|
+
</code>
|
1093
|
+
</pre>
|
1094
|
+
|
1095
|
+
Full example:
|
1096
|
+
<script src="https://gist.github.com/1009447.js"> </script>
|
1097
|
+
|
1098
|
+
|
1099
|
+
Note that the `Consumer` class demonstrated above can be easily tested in isolation without spinning up any AMQP
|
1100
|
+
connections:
|
1101
|
+
|
1102
|
+
<pre>
|
1103
|
+
<code>
|
1104
|
+
require "ostruct"
|
1105
|
+
require "json"
|
1106
|
+
|
1107
|
+
# RSpec example
|
1108
|
+
describe Consumer do
|
1109
|
+
describe "when a new message arrives" do
|
1110
|
+
subject { described_class.new }
|
1111
|
+
|
1112
|
+
let(:metadata) do
|
1113
|
+
o = OpenStruct.new
|
1114
|
+
|
1115
|
+
o.content_type = "application/json"
|
1116
|
+
o
|
1117
|
+
end
|
1118
|
+
let(:payload) { JSON.encode({ :command => "reload_config" }) }
|
1119
|
+
|
1120
|
+
it "does some useful work" do
|
1121
|
+
# check preconditions here if necessary
|
1122
|
+
|
1123
|
+
subject.handle_message(metadata, payload)
|
1124
|
+
|
1125
|
+
# add your code expectations here
|
1126
|
+
end
|
1127
|
+
end
|
1128
|
+
end
|
1129
|
+
</code>
|
1130
|
+
</pre>
|
1131
|
+
|
1132
|
+
TBD
|
1133
|
+
|
1134
|
+
|
1135
|
+
h2. Queue durability vs message durability
|
1136
|
+
|
1137
|
+
See {file:docs/Durability.textile Durability guide}
|
584
1138
|
|
585
1139
|
|
586
1140
|
h2. Error handling and recovery
|
@@ -588,34 +1142,32 @@ h2. Error handling and recovery
|
|
588
1142
|
See {file:docs/ErrorHandling.textile Error handling and recovery guide}
|
589
1143
|
|
590
1144
|
|
591
|
-
|
592
1145
|
h2. Vendor-specific extensions related to queues
|
593
1146
|
|
594
1147
|
See {file:docs/VendorSpecificExtensions.textile Vendor-specific Extensions guide}
|
595
1148
|
|
596
1149
|
|
597
|
-
|
598
1150
|
h2. What to read next
|
599
1151
|
|
600
|
-
|
601
|
-
topics. Guides related to this one are
|
1152
|
+
The documentation is organized as several {file:docs/DocumentationGuidesIndex.textile documentation guides},
|
1153
|
+
covering all kinds of topics. Guides related to this one are:
|
602
1154
|
|
603
|
-
* {file:docs/Exchanges.textile Exchanges}
|
1155
|
+
* {file:docs/Exchanges.textile Working With Exchanges}
|
604
1156
|
* {file:docs/Bindings.textile Bindings}
|
605
1157
|
* {file:docs/ErrorHandling.textile Error handling and recovery}
|
606
1158
|
|
607
|
-
RabbitMQ implements a number of extensions to AMQP
|
608
|
-
|
609
|
-
|
1159
|
+
RabbitMQ implements a number of extensions to AMQP v0.9.1 functionality that are covered in the
|
1160
|
+
{file:docs/VendorSpecificExtensions.textile Vendor-specific Extensions guide}. At least one extension,
|
1161
|
+
per-queue messages time-to-live (TTL), is related to this guide and can be used with the amqp gem v0.8.0 and later.
|
610
1162
|
|
611
1163
|
|
612
1164
|
h2. Tell us what you think!
|
613
1165
|
|
614
|
-
Please take a moment
|
615
|
-
what was unclear
|
616
|
-
key to making documentation better.
|
1166
|
+
Please take a moment to tell us what you think about this guide "on Twitter":http://twitter.com/rubyamqp or the "Ruby AMQP mailing list":http://groups.google.com/group/ruby-amqp.
|
1167
|
+
Let us know what was unclear or what has not been covered. Maybe you do not like the guide style or grammar or discover spelling mistakes. Reader feedback is
|
1168
|
+
key to making the documentation better.
|
617
1169
|
|
618
|
-
If
|
1170
|
+
If, for some reason, you cannot use the communication channels mentioned above, you can "contact the author of the guides directly":mailto:michael@novemberain.com?subject=amqp%20gem%20documentation
|
619
1171
|
|
620
1172
|
|
621
1173
|
<div id="disqus_thread"></div>
|