async-rabbitmq 0.7.1 → 0.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +75 -0
- data/README.md +48 -2
- data/lib/async_rabbitmq/channel.rb +334 -14
- data/lib/async_rabbitmq/notifier.rb +2 -0
- data/lib/async_rabbitmq/session.rb +9 -2
- data/lib/async_rabbitmq/topology_registry.rb +38 -2
- data/lib/async_rabbitmq/version.rb +1 -1
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 91a60c68246f2c413fadd2c183b925062b38a4e36ac1e8b1e4a21eff2ca49c97
|
|
4
|
+
data.tar.gz: 2d3a70204f101c856539b921255d541d8566af490ea468f57753ee22f1e69708
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 869b317e73b1f566e37a04cc7e22e5534f26cfbc3168d11af3b2477ffda77d8f3759270e249b2bab480b94c0c0cf7d8f3d1610fd3bd425b493bfbd9844c69a9f
|
|
7
|
+
data.tar.gz: e5f6720a22811c30c2fbb600faf55c266a6fe298694d203a5e7499bd3218dc74f6471687624cd5ef564f410043009b8bf352102eaf3c0acc0ab59d2dd96049a2
|
data/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,81 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
|
4
4
|
|
|
5
|
+
## [0.8.1] - 2026-11-15
|
|
6
|
+
|
|
7
|
+
### Fixed
|
|
8
|
+
|
|
9
|
+
- A consumer the broker cancelled is no longer given up on when the connection goes at the same
|
|
10
|
+
moment. Deleting a quorum queue is both events at once: the consumer is cancelled, and the
|
|
11
|
+
re-subscribe is answered with a connection-level `541 INTERNAL_ERROR ... noproc` that takes the
|
|
12
|
+
connection down. 0.8.0's recovery loop stopped at the first sight of a session that was not
|
|
13
|
+
open, and nothing else would have brought that consumer back - it is no longer in the channel's
|
|
14
|
+
registry, so the reconnect does not re-register it either. It now waits through however many
|
|
15
|
+
reconnects it takes, and gives up only when the channel or the session is finished for good.
|
|
16
|
+
- A queue whose consumer the broker cancelled is held out of topology recovery until the client
|
|
17
|
+
has established whether it still exists. Without that the two race: a reconnect re-declares the
|
|
18
|
+
whole registry, so a queue deleted a moment earlier came back - and a queue the broker hands
|
|
19
|
+
back with a consumer attached cannot be deleted at all, which is the bug 0.8.0 set out to fix.
|
|
20
|
+
Measured against RabbitMQ 4.4, deleting a quorum queue reached exactly that state.
|
|
21
|
+
- A connection lost mid-re-subscribe is no longer reported as `consumer.recovery_failed`. It has
|
|
22
|
+
its own event and its own recovery, both already running, and an application alarming on
|
|
23
|
+
recovery failures was told twice about one outage, in the words of the more serious of the two.
|
|
24
|
+
- A `404` that says the queue's process is stopped, rather than that there is no such queue, is no
|
|
25
|
+
longer read as a deletion. A classic queue on a stopped node answers `NOT_FOUND - queue 'q' in
|
|
26
|
+
vhost '/' process is stopped by supervisor` and comes back with its node; the client forgot it,
|
|
27
|
+
so the consumer was never restored and the queue was never re-declared either. The probe now
|
|
28
|
+
answers `:deleted`, `:available`, `:unavailable` or `:unreachable`, and only `:available` is followed by an
|
|
29
|
+
attempt to consume - a `basic_consume` the broker answers with a 404 closes the channel and
|
|
30
|
+
takes the consumers still working on it down with it.
|
|
31
|
+
- `on_cancel` now runs to completion before the client decides what became of the queue. The two
|
|
32
|
+
ran alongside each other, and the documented way to keep a queue - re-declare it in `on_cancel` -
|
|
33
|
+
is an RPC that yields, so the probe sometimes got its answer in first and concluded the queue
|
|
34
|
+
was deleted. The callback is still off the dispatch fiber, so an RPC in it still gets its reply
|
|
35
|
+
and an exception in it still cannot stop the channel.
|
|
36
|
+
- A cancelled queue is released back to topology recovery only on a definite answer that it
|
|
37
|
+
still exists. A probe the broker could not answer is not evidence the queue is there, and
|
|
38
|
+
treating it as such re-declared a queue that had just been deleted - the one thing this
|
|
39
|
+
handling exists to prevent. The queue now stays held back until the broker says `:deleted`
|
|
40
|
+
or `:available`, retrying with the same backoff the recovering path uses.
|
|
41
|
+
- The retry backoff no longer grows for a failure that was the connection's rather than the
|
|
42
|
+
queue's, and resets once the connection is back. A two-second blip could otherwise leave a
|
|
43
|
+
healthy queue unconsumed for the next thirty seconds, since each unanswerable probe doubled the
|
|
44
|
+
delay. The queue probe now answers `:unreachable` - no answer at all - separately from
|
|
45
|
+
`:unavailable`, which is the broker saying the queue is not being served.
|
|
46
|
+
|
|
47
|
+
## [0.8.0] - 2026-11-14
|
|
48
|
+
|
|
49
|
+
### Fixed
|
|
50
|
+
|
|
51
|
+
- A server-sent `basic.cancel` no longer leaves the queue in the topology registry. The broker
|
|
52
|
+
cancels a consumer when it can no longer serve the queue, and one of the ways that happens is
|
|
53
|
+
somebody deleting the queue - after which the next connection recovery re-declared it and the
|
|
54
|
+
deletion never stuck. The client now checks, passively, whether the queue is really gone and
|
|
55
|
+
forgets it, with its bindings, if it is. Only `NOT_FOUND` counts as gone: a replicated queue
|
|
56
|
+
mid-election is unavailable, not deleted, and is left in the registry to be re-declared.
|
|
57
|
+
- `on_cancel` and `on_error` no longer run on the channel's dispatch fiber. The obvious thing to
|
|
58
|
+
write in either is an RPC - re-consume, reopen - and an RPC on that fiber waits for a reply only
|
|
59
|
+
that fiber can deliver: fifteen seconds to `rpc_timeout`, then a channel resync. An exception out
|
|
60
|
+
of either escaped `dispatch_loop`, whose rescue sits outside its loop, so the channel stopped
|
|
61
|
+
delivering anything at all, confirms and RPC replies included. Both now run in a task of their
|
|
62
|
+
own and an exception is logged. Bunny dispatches cancel notifications to its consumer work pool
|
|
63
|
+
and the Java client to its work service, for the same reasons. `on_error` also runs after the
|
|
64
|
+
close has released the fibers it stranded, not before.
|
|
65
|
+
- The log line for a broker-sent cancel names the queue, not just the consumer tag.
|
|
66
|
+
|
|
67
|
+
### Added
|
|
68
|
+
|
|
69
|
+
- `recover_cancelled_consumers:` on `Session.new` and `Session#open_channel`, with
|
|
70
|
+
`Channel#recover_cancelled_consumers=` and `Channel#recovers_cancelled_consumers?`. Off by
|
|
71
|
+
default, as it is in Bunny. When on, a consumer the broker cancels is re-subscribed under the
|
|
72
|
+
same tag, backing off from 1s to 30s for as long as the queue exists and the channel is open.
|
|
73
|
+
Recovery declares nothing: it passively checks that the queue is still there and re-consumes, so
|
|
74
|
+
a queue somebody deleted is never handed back. Re-declaring is the application's call.
|
|
75
|
+
- `consumer.recovered` and `consumer.recovery_failed` events. `consumer.recovery_failed` carries
|
|
76
|
+
`reason: :queue_deleted` - emitted whether or not recovery is enabled, since that is the point
|
|
77
|
+
at which a consumer is known not to be coming back - or `reason: :error` with the exception, once
|
|
78
|
+
per episode rather than once per attempt.
|
|
79
|
+
|
|
5
80
|
## [0.7.1] - 2026-10-18
|
|
6
81
|
|
|
7
82
|
### Added
|
data/README.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# async-rabbitmq
|
|
2
2
|
|
|
3
3
|
A fiber-native RabbitMQ (AMQP 0-9-1) client for Ruby, built on the amazing work
|
|
4
|
-
of [Bunny](https://github.com/ruby-amqp/bunny).
|
|
4
|
+
of [Bunny](https://github.com/ruby-amqp/bunny) and [Async](https://github.com/socketry/async).
|
|
5
5
|
|
|
6
6
|
## Installing
|
|
7
7
|
|
|
@@ -466,7 +466,51 @@ starts), `on_blocked` / `on_unblocked` (connection.blocked),
|
|
|
466
466
|
`Channel#on_return` (mandatory messages the broker could not route),
|
|
467
467
|
`on_cancel` (broker cancelled a consumer), `on_error` (broker closed the
|
|
468
468
|
channel). A broker-closed channel can be reopened in place with
|
|
469
|
-
`Channel#reopen`.
|
|
469
|
+
`Channel#reopen`. Every one of these runs in a task of its own, so a callback
|
|
470
|
+
may issue RPCs — re-consume, reopen, declare — and a callback that raises is
|
|
471
|
+
logged rather than taken out on the channel.
|
|
472
|
+
|
|
473
|
+
### A consumer the broker cancels
|
|
474
|
+
|
|
475
|
+
RabbitMQ cancels a consumer when it can no longer serve the queue to it. The
|
|
476
|
+
connection is untouched, so none of the recovery above runs — a `basic.cancel`
|
|
477
|
+
frame is the only notice there is, and without `on_cancel` the worker goes on
|
|
478
|
+
publishing and heartbeating while consuming nothing.
|
|
479
|
+
|
|
480
|
+
Nothing is re-subscribed by default, which matches Bunny, the Java client and
|
|
481
|
+
the .NET client. `recover_cancelled_consumers: true`, on the session or on one
|
|
482
|
+
channel, re-subscribes under the same consumer tag, backing off from 1s to 30s
|
|
483
|
+
for as long as the queue exists, through however many reconnects that takes. It
|
|
484
|
+
gives up only when the channel or the session is finished for good.
|
|
485
|
+
|
|
486
|
+
**Recovery never declares anything.** One of the reasons a consumer gets
|
|
487
|
+
cancelled is somebody deleting the queue, and a client that answered that by
|
|
488
|
+
re-declaring would hand back a queue nobody could get rid of while a consumer
|
|
489
|
+
was attached. So the client asks the broker, on a channel of its own, whether
|
|
490
|
+
the queue is still there, and re-consumes only if it is. Re-declaring is the
|
|
491
|
+
application's call, and `on_cancel` is where to make it: that callback is run
|
|
492
|
+
to completion before the client goes asking, so a queue declared there is
|
|
493
|
+
there when it looks.
|
|
494
|
+
|
|
495
|
+
Three things follow from that, and each of them is a bug in some client:
|
|
496
|
+
|
|
497
|
+
- **The queue is held out of topology recovery until the client knows.**
|
|
498
|
+
Otherwise the two race — a reconnect re-declares the whole registry — and the
|
|
499
|
+
queue somebody deleted comes back. Deleting a quorum queue is exactly that
|
|
500
|
+
race arriving as one event: the cancel, and then a connection-level `541
|
|
501
|
+
INTERNAL_ERROR ... noproc` in answer to the re-subscribe.
|
|
502
|
+
- **The question is asked before every attempt, never the attempt instead of
|
|
503
|
+
the question.** A `basic_consume` the broker answers `404` closes the
|
|
504
|
+
channel, and takes the consumers still working on it down with it.
|
|
505
|
+
- **`NOT_FOUND` is two answers.** `no queue 'q' in vhost '/'` is a deletion;
|
|
506
|
+
`queue 'q' in vhost '/' process is stopped by supervisor`, which is what a
|
|
507
|
+
classic queue on a stopped node gives, is a queue that comes back with its
|
|
508
|
+
node. The second is waited for, not forgotten.
|
|
509
|
+
|
|
510
|
+
```ruby
|
|
511
|
+
session = AsyncRabbitMQ::Session.new(..., recover_cancelled_consumers: true)
|
|
512
|
+
channel.on_cancel { |tag| logger.warn("consumer #{tag} cancelled by the broker") }
|
|
513
|
+
```
|
|
470
514
|
|
|
471
515
|
## Errors
|
|
472
516
|
|
|
@@ -520,6 +564,8 @@ logged and skipped; it never breaks the connection.
|
|
|
520
564
|
| `channel.rpc` | `channel`, `method` (`"queue.declare-ok"`), `duration` |
|
|
521
565
|
| `consumer.registered` | `channel`, `queue`, `consumer_tag`, `manual_ack` |
|
|
522
566
|
| `consumer.cancelled` | `channel`, `consumer_tag`, `queue`, `reason` (`:client`, `:broker`) |
|
|
567
|
+
| `consumer.recovered` | `channel`, `consumer_tag`, `queue`, `attempts` |
|
|
568
|
+
| `consumer.recovery_failed` | `channel`, `consumer_tag`, `queue`, `reason` (`:queue_deleted`, `:error`), `error` |
|
|
523
569
|
| `message.published` | `channel`, `exchange`, `routing_key`, `count`, `bytes`, `delivery_tag` |
|
|
524
570
|
| `message.confirmed` | `channel`, `delivery_tag`, `multiple`, `acked` |
|
|
525
571
|
| `message.returned` | `channel`, `exchange`, `routing_key`, `code`, `text`, `bytes` |
|
|
@@ -71,6 +71,21 @@ module AsyncRabbitMQ
|
|
|
71
71
|
# after an RPC timeout, used when the session has no rpc_timeout set.
|
|
72
72
|
RESYNC_TIMEOUT = 5
|
|
73
73
|
|
|
74
|
+
# First wait before re-subscribing a consumer the broker cancelled, and the
|
|
75
|
+
# ceiling the doubling stops at. Only used when the channel was asked to
|
|
76
|
+
# recover cancelled consumers; see #recovers_cancelled_consumers?.
|
|
77
|
+
RECOVER_CONSUMER_DELAY = 1
|
|
78
|
+
RECOVER_CONSUMER_MAX_DELAY = 30
|
|
79
|
+
|
|
80
|
+
# A queue that is still declared and a queue that is gone both answer
|
|
81
|
+
# NOT_FOUND to a passive declare, and only the reply text tells them apart.
|
|
82
|
+
# Measured on RabbitMQ 4.4: a deleted queue gives "no queue 'q' in vhost
|
|
83
|
+
# '/'", while a classic queue whose node has been stopped gives "queue 'q'
|
|
84
|
+
# in vhost '/' process is stopped by supervisor" and comes back with its
|
|
85
|
+
# node. Older brokers say "home node ... is down or inaccessible" for the
|
|
86
|
+
# same thing. See #queue_missing?.
|
|
87
|
+
QUEUE_UNAVAILABLE = /process is stopped|is down or inaccessible/
|
|
88
|
+
|
|
74
89
|
# Pushed onto the work queue to retire one handler worker. Compared by
|
|
75
90
|
# identity, so it can never collide with a delivery.
|
|
76
91
|
WORKER_STOP = Object.new.freeze
|
|
@@ -94,7 +109,8 @@ module AsyncRabbitMQ
|
|
|
94
109
|
# concurrently on this channel (Bunny-parity: default 1 = serialized).
|
|
95
110
|
# Resizable at runtime via #pool_size=; basic_qos adjusts it automatically
|
|
96
111
|
# when prefetch_count > 0 so the two stay coupled.
|
|
97
|
-
def initialize(channel_id, session, frame_io, frame_max:, logger:, pool_size: 1, rpc_timeout: nil
|
|
112
|
+
def initialize(channel_id, session, frame_io, frame_max:, logger:, pool_size: 1, rpc_timeout: nil,
|
|
113
|
+
recover_cancelled_consumers: false)
|
|
98
114
|
@channel_id = channel_id
|
|
99
115
|
@session = session
|
|
100
116
|
@notifier = session.respond_to?(:notifier) ? session.notifier : nil
|
|
@@ -124,6 +140,7 @@ module AsyncRabbitMQ
|
|
|
124
140
|
@on_cancel = nil
|
|
125
141
|
@on_error = nil
|
|
126
142
|
@on_handler_error = nil
|
|
143
|
+
@recover_cancelled_consumers = recover_cancelled_consumers
|
|
127
144
|
@on_error_disposition = :dead_letter
|
|
128
145
|
@resyncing = false
|
|
129
146
|
@resync_condition = nil
|
|
@@ -647,10 +664,33 @@ module AsyncRabbitMQ
|
|
|
647
664
|
# Register a callback invoked when the broker cancels a consumer server-side
|
|
648
665
|
# (e.g. the queue is deleted or an HA failover occurs).
|
|
649
666
|
# The block receives the consumer_tag that was cancelled.
|
|
667
|
+
#
|
|
668
|
+
# The callback runs in a task of its own, so it may issue RPCs - a
|
|
669
|
+
# re-consume, a declare - and an exception out of it is logged rather than
|
|
670
|
+
# taken out on the channel.
|
|
650
671
|
def on_cancel(&block)
|
|
651
672
|
@on_cancel = block
|
|
652
673
|
end
|
|
653
674
|
|
|
675
|
+
# Re-subscribe a consumer the broker cancels, under the same consumer tag.
|
|
676
|
+
#
|
|
677
|
+
# Off by default, as it is in Bunny, and for a reason worth stating: the
|
|
678
|
+
# broker cancels a consumer when it can no longer serve the queue, and one
|
|
679
|
+
# of the ways that happens is somebody deleting the queue. A client that
|
|
680
|
+
# recovers by re-declaring would hand back a queue nobody can get rid of
|
|
681
|
+
# while a consumer is attached. So recovery here never declares anything.
|
|
682
|
+
# It passively checks that the queue is still there and re-consumes if it
|
|
683
|
+
# is; if the queue has gone it stops, and forgets the queue so that a later
|
|
684
|
+
# connection recovery does not re-declare it either.
|
|
685
|
+
#
|
|
686
|
+
# Retries back off from RECOVER_CONSUMER_DELAY to RECOVER_CONSUMER_MAX_DELAY
|
|
687
|
+
# and continue for as long as the queue exists and the channel is open.
|
|
688
|
+
attr_writer :recover_cancelled_consumers
|
|
689
|
+
|
|
690
|
+
def recovers_cancelled_consumers?
|
|
691
|
+
@recover_cancelled_consumers
|
|
692
|
+
end
|
|
693
|
+
|
|
654
694
|
# Register a callback invoked when the broker closes this channel due to an
|
|
655
695
|
# error (e.g. 404 queue not found). The block receives the channel and the
|
|
656
696
|
# AMQ::Protocol::Channel::Close method frame.
|
|
@@ -1273,18 +1313,7 @@ module AsyncRabbitMQ
|
|
|
1273
1313
|
handle_channel_close(method)
|
|
1274
1314
|
|
|
1275
1315
|
when AMQ::Protocol::Basic::Cancel
|
|
1276
|
-
|
|
1277
|
-
# Remove from @consumers so deliveries are no longer dispatched to a dead block.
|
|
1278
|
-
entry = @consumers.delete(method.consumer_tag)
|
|
1279
|
-
topology&.delete_consumer(method.consumer_tag)
|
|
1280
|
-
if entry
|
|
1281
|
-
@logger.warn("Channel #{@channel_id}: broker cancelled consumer #{method.consumer_tag}")
|
|
1282
|
-
instrument("consumer.cancelled") do
|
|
1283
|
-
{ channel: @channel_id, consumer_tag: method.consumer_tag, queue: entry[:queue_name], reason: :broker }
|
|
1284
|
-
end
|
|
1285
|
-
@on_cancel&.call(method.consumer_tag)
|
|
1286
|
-
end
|
|
1287
|
-
wake_each_waiters(method.consumer_tag)
|
|
1316
|
+
handle_broker_cancel(method.consumer_tag)
|
|
1288
1317
|
# No CancelOk to send for server-initiated cancel (no-wait is implicit).
|
|
1289
1318
|
|
|
1290
1319
|
when AMQ::Protocol::Channel::Flow
|
|
@@ -1354,7 +1383,6 @@ module AsyncRabbitMQ
|
|
|
1354
1383
|
forget_consumers_in_topology
|
|
1355
1384
|
@session.channel_closed(@channel_id)
|
|
1356
1385
|
instrument("channel.closed") { { channel: @channel_id, reason: :broker, code: code, text: text } }
|
|
1357
|
-
@on_error&.call(self, method)
|
|
1358
1386
|
|
|
1359
1387
|
error = if FrameIO::SOFT_ERROR_CODES.include?(code)
|
|
1360
1388
|
ChannelError.new(code: code, text: text, channel_id: @channel_id, close_method: method)
|
|
@@ -1369,6 +1397,298 @@ module AsyncRabbitMQ
|
|
|
1369
1397
|
@queue&.push(nil)
|
|
1370
1398
|
stop_workers(discard: true)
|
|
1371
1399
|
wake_each_waiters
|
|
1400
|
+
# Last, and in a task of its own. The lines above exist to release fibers
|
|
1401
|
+
# this close has stranded; a hook that is slow, or that raises, must not
|
|
1402
|
+
# come between them and the fibers waiting on them. The natural thing to
|
|
1403
|
+
# write in this hook is a reopen, which is an RPC, and an RPC on the
|
|
1404
|
+
# dispatch fiber waits for a reply only the dispatch fiber can deliver.
|
|
1405
|
+
notify_app("on_error") { @on_error.call(self, method) } if @on_error
|
|
1406
|
+
end
|
|
1407
|
+
|
|
1408
|
+
# An application callback never runs on the dispatch fiber. Two reasons,
|
|
1409
|
+
# both load-bearing. An RPC issued from a callback waits for a reply that
|
|
1410
|
+
# only this fiber delivers, so it deadlocks until rpc_timeout and then
|
|
1411
|
+
# drags the channel through a resync - and a re-consume is the obvious
|
|
1412
|
+
# thing to write in on_cancel. And an exception out of a callback escapes
|
|
1413
|
+
# #handle_message, whose rescue sits outside dispatch_loop's `loop`, so the
|
|
1414
|
+
# loop ends and the channel stops delivering anything at all: no
|
|
1415
|
+
# deliveries, no confirms, no RPC replies, with one line to say so.
|
|
1416
|
+
#
|
|
1417
|
+
# Bunny hands cancel notifications to its consumer work pool and the Java
|
|
1418
|
+
# client to its work service, for the same reason.
|
|
1419
|
+
def notify_app(name, &block)
|
|
1420
|
+
@session.spawn_background { run_callback(name, &block) }
|
|
1421
|
+
end
|
|
1422
|
+
|
|
1423
|
+
# The body of #notify_app, for a caller that is already off the dispatch
|
|
1424
|
+
# fiber and has something to do after the callback.
|
|
1425
|
+
def run_callback(name)
|
|
1426
|
+
yield
|
|
1427
|
+
rescue *PROPAGATE_FROM_HANDLER
|
|
1428
|
+
raise
|
|
1429
|
+
rescue Exception => e # rubocop:disable Lint/RescueException
|
|
1430
|
+
@logger.error("Channel #{@channel_id}: #{name} callback raised #{e.class}: #{e.message}")
|
|
1431
|
+
end
|
|
1432
|
+
|
|
1433
|
+
# The broker cancels a consumer when it can no longer serve the queue to
|
|
1434
|
+
# it: the queue was deleted, or the node holding a classic queue stopped.
|
|
1435
|
+
# On a three-node RabbitMQ 4.4.0 cluster, moving a quorum queue's
|
|
1436
|
+
# leadership did not do it - not a graceful stop of the leader's node, not
|
|
1437
|
+
# removing the member on the consuming node, not losing the majority, and
|
|
1438
|
+
# not cutting the leader's node off the network until the queue re-elected.
|
|
1439
|
+
# Other versions are reported to cancel on unavailability; it makes no
|
|
1440
|
+
# difference here, since the handling is the same whatever the cause. The
|
|
1441
|
+
# connection is untouched either way, so connection recovery never runs:
|
|
1442
|
+
# this frame is the only notice there is.
|
|
1443
|
+
#
|
|
1444
|
+
# The consumer goes from @consumers first, so no delivery is dispatched to
|
|
1445
|
+
# a block that is no longer subscribed, and everything beyond that
|
|
1446
|
+
# bookkeeping happens off this fiber.
|
|
1447
|
+
def handle_broker_cancel(tag)
|
|
1448
|
+
entry = @consumers.delete(tag)
|
|
1449
|
+
topology&.delete_consumer(tag)
|
|
1450
|
+
return wake_each_waiters(tag) unless entry
|
|
1451
|
+
|
|
1452
|
+
# Held back from topology recovery from this moment, before anything can
|
|
1453
|
+
# reconnect, because one of the reasons a consumer is cancelled is the
|
|
1454
|
+
# queue being deleted and a reconnect re-declares the registry wholesale.
|
|
1455
|
+
# Settled by #reconcile_cancelled_consumer, either way.
|
|
1456
|
+
topology&.suspend_queue(entry[:queue_name])
|
|
1457
|
+
@logger.warn("Channel #{@channel_id}: broker cancelled consumer #{tag} on #{entry[:queue_name]}")
|
|
1458
|
+
instrument("consumer.cancelled") do
|
|
1459
|
+
{ channel: @channel_id, consumer_tag: tag, queue: entry[:queue_name], reason: :broker }
|
|
1460
|
+
end
|
|
1461
|
+
# The callback first and the reconcile after it, in one task rather than
|
|
1462
|
+
# two. on_cancel is where an application re-declares a queue it wants
|
|
1463
|
+
# back, and a reconcile running alongside it sometimes asked the broker
|
|
1464
|
+
# before that declare had landed and concluded the queue was gone. Still
|
|
1465
|
+
# off the dispatch fiber, so an RPC in the callback can get its reply and
|
|
1466
|
+
# an exception in it cannot stop the channel.
|
|
1467
|
+
@session.spawn_background do
|
|
1468
|
+
run_callback("on_cancel") { @on_cancel.call(tag) } if @on_cancel
|
|
1469
|
+
reconcile_cancelled_consumer(tag, entry)
|
|
1470
|
+
end
|
|
1471
|
+
end
|
|
1472
|
+
|
|
1473
|
+
# Why the broker cancelled is not on the wire, and the reasons want
|
|
1474
|
+
# opposite handling: a queue that is merely not being served should be
|
|
1475
|
+
# consumed from again, while a queue somebody deleted must be left deleted.
|
|
1476
|
+
# A passive declare is the only way to tell them apart. Spring AMQP's
|
|
1477
|
+
# listener container probes the same way, and for the same reason declares
|
|
1478
|
+
# nothing.
|
|
1479
|
+
#
|
|
1480
|
+
# Runs whether or not this channel recovers cancelled consumers, because
|
|
1481
|
+
# forgetting a deleted queue is not a recovery feature: a queue left in the
|
|
1482
|
+
# topology registry is re-declared on the next connection recovery, and
|
|
1483
|
+
# then the deletion never sticks.
|
|
1484
|
+
def reconcile_cancelled_consumer(tag, entry)
|
|
1485
|
+
queue = entry[:queue_name]
|
|
1486
|
+
return recover_cancelled_consumer(tag, entry) if recovers_cancelled_consumers?
|
|
1487
|
+
|
|
1488
|
+
# Not putting the consumer back, but a queue that has been deleted must
|
|
1489
|
+
# still leave the registry or the next reconnect declares it again.
|
|
1490
|
+
# Nothing is asked of a connection that is not there: a question put to a
|
|
1491
|
+
# session that is down cannot be answered, and guessing here is what puts
|
|
1492
|
+
# a deleted queue back.
|
|
1493
|
+
#
|
|
1494
|
+
# So the queue stays suspended until the broker gives a definite answer,
|
|
1495
|
+
# and only :available releases it. Releasing on a probe that could not be
|
|
1496
|
+
# answered is how a deleted queue comes back - it is not evidence the
|
|
1497
|
+
# queue is there, and 46_broker_amnesia_spec is the case that proves it.
|
|
1498
|
+
delay = RECOVER_CONSUMER_DELAY
|
|
1499
|
+
|
|
1500
|
+
loop do
|
|
1501
|
+
case wait_for_usable_channel
|
|
1502
|
+
when nil then return release_cancelled_consumer(tag, queue)
|
|
1503
|
+
when :reconnected then delay = RECOVER_CONSUMER_DELAY
|
|
1504
|
+
end
|
|
1505
|
+
|
|
1506
|
+
case queue_status(queue)
|
|
1507
|
+
when :deleted then return forget_deleted_queue(tag, queue)
|
|
1508
|
+
when :available then return release_cancelled_consumer(tag, queue)
|
|
1509
|
+
end
|
|
1510
|
+
|
|
1511
|
+
# :unavailable - the broker says the queue is not being served - and
|
|
1512
|
+
# :unreachable - no answer at all - both mean ask again, never "the
|
|
1513
|
+
# queue is there". Sleeping on either is also what keeps this from
|
|
1514
|
+
# spinning when the probe cannot be answered.
|
|
1515
|
+
sleep delay
|
|
1516
|
+
delay = [delay * 2, RECOVER_CONSUMER_MAX_DELAY].min
|
|
1517
|
+
end
|
|
1518
|
+
end
|
|
1519
|
+
|
|
1520
|
+
# Wait until this channel can carry a consumer again, through however many
|
|
1521
|
+
# reconnects that takes. :ready if it already could, :reconnected if it had
|
|
1522
|
+
# to wait for one, nil if it never will.
|
|
1523
|
+
#
|
|
1524
|
+
# Giving up on a connection loss instead would lose the consumer for good:
|
|
1525
|
+
# it is no longer in @consumers, so the reconnect does not re-register it,
|
|
1526
|
+
# and this loop is the only thing that would have brought it back. That is
|
|
1527
|
+
# precisely the case a quorum queue produces - the re-subscribe is answered
|
|
1528
|
+
# with a connection-level 541, not a channel error.
|
|
1529
|
+
#
|
|
1530
|
+
# Over for good when the channel is closed, which means the application
|
|
1531
|
+
# closed it or recovery could not reopen it and its consumers belong to
|
|
1532
|
+
# whoever does that, or when the session has stopped trying.
|
|
1533
|
+
def wait_for_usable_channel
|
|
1534
|
+
waited = false
|
|
1535
|
+
loop do
|
|
1536
|
+
return waited ? :reconnected : :ready if open? && @session.open?
|
|
1537
|
+
return nil if closed? || @session.closed?
|
|
1538
|
+
|
|
1539
|
+
waited = true
|
|
1540
|
+
sleep RECOVER_CONSUMER_DELAY
|
|
1541
|
+
end
|
|
1542
|
+
end
|
|
1543
|
+
|
|
1544
|
+
# Let the queue be recovered again and release anyone in #each. For a
|
|
1545
|
+
# consumer that will not be followed up - this channel does not recover
|
|
1546
|
+
# cancelled consumers, or there is nothing left to recover it onto - the
|
|
1547
|
+
# queue must not stay held back: it was never established that it is gone,
|
|
1548
|
+
# and a session opened again would not re-declare it.
|
|
1549
|
+
def release_cancelled_consumer(tag, queue)
|
|
1550
|
+
topology&.resume_queue(queue)
|
|
1551
|
+
wake_each_waiters(tag)
|
|
1552
|
+
end
|
|
1553
|
+
|
|
1554
|
+
# Re-consume under the same tag, for as long as the queue is there and this
|
|
1555
|
+
# channel can carry a consumer. Nothing is declared and nothing is bound:
|
|
1556
|
+
# whatever the queue is, it is the one that was there before.
|
|
1557
|
+
def recover_cancelled_consumer(tag, entry)
|
|
1558
|
+
queue = entry[:queue_name]
|
|
1559
|
+
delay = RECOVER_CONSUMER_DELAY
|
|
1560
|
+
failures = 0
|
|
1561
|
+
|
|
1562
|
+
loop do
|
|
1563
|
+
# Every round starts with a connection that works and a fresh answer
|
|
1564
|
+
# about the queue. A queue deleted while we were retrying has to stop
|
|
1565
|
+
# the loop, and a basic_consume the broker answers with a 404 closes
|
|
1566
|
+
# this channel and takes the consumers still working on it down too -
|
|
1567
|
+
# so the question is asked on a channel of its own first, and only an
|
|
1568
|
+
# answer of "it is there" is followed by an attempt to consume.
|
|
1569
|
+
case wait_for_usable_channel
|
|
1570
|
+
when nil then return release_cancelled_consumer(tag, queue)
|
|
1571
|
+
when :reconnected
|
|
1572
|
+
# The connection was what was wrong, and it is back: start the
|
|
1573
|
+
# backoff again rather than sleeping out a delay earned by an outage
|
|
1574
|
+
# that is over. A two-second blip should not leave a healthy queue
|
|
1575
|
+
# unconsumed for the next thirty.
|
|
1576
|
+
delay = RECOVER_CONSUMER_DELAY
|
|
1577
|
+
end
|
|
1578
|
+
|
|
1579
|
+
case queue_status(queue)
|
|
1580
|
+
when :deleted
|
|
1581
|
+
return forget_deleted_queue(tag, queue)
|
|
1582
|
+
when :unreachable
|
|
1583
|
+
# No answer rather than an answer: the connection is what is wrong,
|
|
1584
|
+
# not the queue, so the backoff a queue earns does not apply. Go back
|
|
1585
|
+
# and wait for the connection instead of sleeping out a delay that
|
|
1586
|
+
# doubles every time the broker cannot be reached.
|
|
1587
|
+
delay = RECOVER_CONSUMER_DELAY
|
|
1588
|
+
when :available
|
|
1589
|
+
begin
|
|
1590
|
+
basic_consume(queue, consumer_tag: tag, manual_ack: entry[:manual_ack], exclusive: entry[:exclusive],
|
|
1591
|
+
arguments: entry[:arguments] || {}, on_error: entry[:on_error], &entry[:block])
|
|
1592
|
+
# Consuming again, so the queue is there and recovery can have it
|
|
1593
|
+
# back (#handle_broker_cancel held it).
|
|
1594
|
+
topology&.resume_queue(queue)
|
|
1595
|
+
@logger.info("Channel #{@channel_id}: re-subscribed consumer #{tag} to #{queue}")
|
|
1596
|
+
instrument("consumer.recovered") do
|
|
1597
|
+
{ channel: @channel_id, consumer_tag: tag, queue: queue, attempts: failures + 1 }
|
|
1598
|
+
end
|
|
1599
|
+
return
|
|
1600
|
+
rescue *PROPAGATE_FROM_HANDLER
|
|
1601
|
+
raise
|
|
1602
|
+
rescue ConnectionError, NotOpenError => e
|
|
1603
|
+
# The connection went, not the subscription. That has its own
|
|
1604
|
+
# reporting and its own recovery, both already running, so it is
|
|
1605
|
+
# not counted as a failure to re-subscribe: an application alarming
|
|
1606
|
+
# on consumer.recovery_failed would otherwise be told twice about
|
|
1607
|
+
# one event, and told it in the words of the more serious of the
|
|
1608
|
+
# two.
|
|
1609
|
+
@logger.debug("Channel #{@channel_id}: re-subscribe of #{tag} interrupted by " \
|
|
1610
|
+
"#{e.class}; waiting for the connection")
|
|
1611
|
+
rescue ChannelError => e
|
|
1612
|
+
# The queue went between the question and the answer. The broker
|
|
1613
|
+
# has closed this channel saying so, so there is nothing left to
|
|
1614
|
+
# retry on.
|
|
1615
|
+
return forget_deleted_queue(tag, queue) if queue_missing?(e)
|
|
1616
|
+
|
|
1617
|
+
failures += 1
|
|
1618
|
+
note_recovery_failure(tag, queue, e, failures)
|
|
1619
|
+
rescue => e
|
|
1620
|
+
failures += 1
|
|
1621
|
+
note_recovery_failure(tag, queue, e, failures)
|
|
1622
|
+
end
|
|
1623
|
+
end
|
|
1624
|
+
# :unavailable falls through to the backoff and is asked again.
|
|
1625
|
+
|
|
1626
|
+
sleep delay
|
|
1627
|
+
delay = [delay * 2, RECOVER_CONSUMER_MAX_DELAY].min
|
|
1628
|
+
end
|
|
1629
|
+
end
|
|
1630
|
+
|
|
1631
|
+
# One line per episode at warn, the rest at debug: a broker that is away
|
|
1632
|
+
# for an hour would otherwise write a warning every attempt for the whole
|
|
1633
|
+
# hour, and an application counting warnings would read one outage as
|
|
1634
|
+
# hundreds.
|
|
1635
|
+
def note_recovery_failure(tag, queue, error, failures)
|
|
1636
|
+
if failures == 1
|
|
1637
|
+
@logger.warn("Channel #{@channel_id}: could not re-subscribe #{tag} to #{queue}, " \
|
|
1638
|
+
"retrying: #{error.class}: #{error.message}")
|
|
1639
|
+
instrument("consumer.recovery_failed") do
|
|
1640
|
+
{ channel: @channel_id, consumer_tag: tag, queue: queue, reason: :error, error: error }
|
|
1641
|
+
end
|
|
1642
|
+
else
|
|
1643
|
+
@logger.debug("Channel #{@channel_id}: re-subscribe attempt #{failures} for #{tag} failed: #{error.message}")
|
|
1644
|
+
end
|
|
1645
|
+
end
|
|
1646
|
+
|
|
1647
|
+
# The queue behind a cancelled consumer has gone. Drop it from the topology
|
|
1648
|
+
# registry, with its bindings, so a later connection recovery does not
|
|
1649
|
+
# re-declare a queue somebody deleted on purpose - the "deleted queue comes
|
|
1650
|
+
# back" bug the Java client has been fixing instances of for years.
|
|
1651
|
+
def forget_deleted_queue(tag, queue)
|
|
1652
|
+
topology&.delete_queue(queue)
|
|
1653
|
+
@logger.warn("Channel #{@channel_id}: queue #{queue} no longer exists; consumer #{tag} will not be recovered")
|
|
1654
|
+
instrument("consumer.recovery_failed") do
|
|
1655
|
+
{ channel: @channel_id, consumer_tag: tag, queue: queue, reason: :queue_deleted }
|
|
1656
|
+
end
|
|
1657
|
+
wake_each_waiters(tag)
|
|
1658
|
+
end
|
|
1659
|
+
|
|
1660
|
+
# Is the queue gone, there, or merely not being served right now? A passive
|
|
1661
|
+
# declare answers it, on a channel of its own because a passive declare
|
|
1662
|
+
# that fails closes the channel it ran on - which would take the queue's
|
|
1663
|
+
# fellow consumers down with it.
|
|
1664
|
+
#
|
|
1665
|
+
# Four answers, because conflating any two of them does harm. :deleted must
|
|
1666
|
+
# be forgotten and :unavailable must be waited for, and telling those apart
|
|
1667
|
+
# is the whole point. :unreachable - no answer at all, because the
|
|
1668
|
+
# connection is in trouble - is separated from :unavailable so that the
|
|
1669
|
+
# caller does not spend a backoff earned by a queue on a connection that is
|
|
1670
|
+
# simply down, and is never taken for :available, because a basic_consume
|
|
1671
|
+
# on a queue that is not there closes the channel.
|
|
1672
|
+
def queue_status(name)
|
|
1673
|
+
@session.with_channel { |probe| probe.queue(name, passive: true) }
|
|
1674
|
+
:available
|
|
1675
|
+
rescue ChannelError => e
|
|
1676
|
+
queue_missing?(e) ? :deleted : :unavailable
|
|
1677
|
+
rescue *PROPAGATE_FROM_HANDLER
|
|
1678
|
+
raise
|
|
1679
|
+
rescue => e
|
|
1680
|
+
@logger.debug("Channel #{@channel_id}: could not check whether #{name} still exists: #{e.class}: #{e.message}")
|
|
1681
|
+
:unreachable
|
|
1682
|
+
end
|
|
1683
|
+
|
|
1684
|
+
# Gone, rather than merely not being served. Taking a queue whose node is
|
|
1685
|
+
# down for a deleted one forgets a queue that is coming back: its consumer
|
|
1686
|
+
# is never restored, and it is never re-declared either. Any 404 the
|
|
1687
|
+
# QUEUE_UNAVAILABLE wordings do not cover reads as a deletion, which is the
|
|
1688
|
+
# safe way round - an unmatched wording costs only what forgetting it costs
|
|
1689
|
+
# today.
|
|
1690
|
+
def queue_missing?(error)
|
|
1691
|
+
error.code == 404 && !QUEUE_UNAVAILABLE.match?(error.text.to_s)
|
|
1372
1692
|
end
|
|
1373
1693
|
|
|
1374
1694
|
# Release fibers blocked in #each for one consumer, or for all of them.
|
|
@@ -161,6 +161,7 @@ module AsyncRabbitMQ
|
|
|
161
161
|
recovery_interval: RECOVERY_INITIAL,
|
|
162
162
|
recovery_max_interval: RECOVERY_MAX,
|
|
163
163
|
recover_topology: true,
|
|
164
|
+
recover_cancelled_consumers: false,
|
|
164
165
|
topology_recovery_filter: nil,
|
|
165
166
|
instrumenter: nil,
|
|
166
167
|
notifier: nil,
|
|
@@ -194,6 +195,9 @@ module AsyncRabbitMQ
|
|
|
194
195
|
@recovery_interval = recovery_interval
|
|
195
196
|
@recovery_max_interval = recovery_max_interval
|
|
196
197
|
@recover_topology = recover_topology
|
|
198
|
+
# The default every channel opened on this session takes; see
|
|
199
|
+
# Channel#recovers_cancelled_consumers? for why it is off.
|
|
200
|
+
@recover_cancelled_consumers = recover_cancelled_consumers
|
|
197
201
|
@topology = TopologyRegistry.new(filter: topology_recovery_filter)
|
|
198
202
|
@logger = logger
|
|
199
203
|
@notifier = notifier || Notifier.new(logger: logger)
|
|
@@ -354,12 +358,15 @@ module AsyncRabbitMQ
|
|
|
354
358
|
# Open a new channel. Returns an AsyncRabbitMQ::Channel.
|
|
355
359
|
# +pool_size+ bounds concurrent consumer-handler fibers on the channel
|
|
356
360
|
# (Bunny-parity: default 1). basic_qos will auto-adjust it to prefetch_count.
|
|
357
|
-
|
|
361
|
+
# +recover_cancelled_consumers+ defaults to the session's setting; see
|
|
362
|
+
# Channel#recovers_cancelled_consumers?.
|
|
363
|
+
def open_channel(pool_size: 1, recover_cancelled_consumers: @recover_cancelled_consumers)
|
|
358
364
|
raise NotOpenError, "Session is not open" unless open?
|
|
359
365
|
channel = @channel_mutex.acquire do
|
|
360
366
|
channel_id = next_channel_id
|
|
361
367
|
Channel.new(channel_id, self, @frame_io, frame_max: @negotiated_fm || @frame_max, logger: @logger,
|
|
362
|
-
pool_size: pool_size, rpc_timeout: @rpc_timeout
|
|
368
|
+
pool_size: pool_size, rpc_timeout: @rpc_timeout,
|
|
369
|
+
recover_cancelled_consumers: recover_cancelled_consumers).tap { |ch| @channels[channel_id] = ch }
|
|
363
370
|
end
|
|
364
371
|
channel.open
|
|
365
372
|
channel
|
|
@@ -38,6 +38,7 @@ module AsyncRabbitMQ
|
|
|
38
38
|
@queue_bindings = []
|
|
39
39
|
@exchange_bindings = []
|
|
40
40
|
@consumers = {} # consumer tag => queue name
|
|
41
|
+
@suspended = Set.new # queue names held back from recovery
|
|
41
42
|
end
|
|
42
43
|
|
|
43
44
|
# --- recording -----------------------------------------------------------
|
|
@@ -59,6 +60,9 @@ module AsyncRabbitMQ
|
|
|
59
60
|
|
|
60
61
|
def record_queue(channel_id, name, durable:, exclusive:, auto_delete:, arguments:, server_named:, object: nil)
|
|
61
62
|
existing = @queues[name]
|
|
63
|
+
# Declared again by the application, so whatever doubt a cancelled
|
|
64
|
+
# consumer raised about it is settled.
|
|
65
|
+
@suspended.delete(name)
|
|
62
66
|
rec = RecordedQueue.new(channel_id: channel_id, name: name, durable: durable, exclusive: exclusive,
|
|
63
67
|
auto_delete: auto_delete, arguments: arguments.dup, server_named: server_named,
|
|
64
68
|
objects: existing ? existing.objects : [])
|
|
@@ -68,15 +72,45 @@ module AsyncRabbitMQ
|
|
|
68
72
|
|
|
69
73
|
def delete_queue(name)
|
|
70
74
|
@queues.delete(name)
|
|
75
|
+
@suspended.delete(name)
|
|
71
76
|
gone, @queue_bindings = @queue_bindings.partition { |b| b.queue == name }
|
|
72
77
|
gone.each { |b| prune_auto_delete_exchange(b.exchange) }
|
|
73
78
|
end
|
|
74
79
|
|
|
80
|
+
# --- suspension ----------------------------------------------------------
|
|
81
|
+
#
|
|
82
|
+
# A queue is suspended while the client is working out whether it is still
|
|
83
|
+
# there, which is what a consumer the broker cancelled leaves it not
|
|
84
|
+
# knowing. Recovery skips a suspended queue and its bindings.
|
|
85
|
+
#
|
|
86
|
+
# The alternative - keep recovering it until the client is sure - puts a
|
|
87
|
+
# deleted queue back, because the deletion and the reconnect race: the node
|
|
88
|
+
# that was serving the queue is often the one that just went, and a
|
|
89
|
+
# reconnect re-declares the whole registry before anything can ask the new
|
|
90
|
+
# connection whether the queue survived. A queue the broker then hands back
|
|
91
|
+
# with a consumer attached cannot be deleted at all. Holding it back costs
|
|
92
|
+
# nothing by comparison: the worst case is a queue that was fine and is not
|
|
93
|
+
# re-declared for the few seconds it takes to establish that.
|
|
94
|
+
|
|
95
|
+
def suspend_queue(name)
|
|
96
|
+
@suspended << name if @queues.key?(name)
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
def resume_queue(name)
|
|
100
|
+
@suspended.delete(name)
|
|
101
|
+
end
|
|
102
|
+
|
|
103
|
+
# Queue names currently held back, for inspection.
|
|
104
|
+
def suspended_queues
|
|
105
|
+
@suspended.dup
|
|
106
|
+
end
|
|
107
|
+
|
|
75
108
|
# A server-named queue was re-declared under a different name.
|
|
76
109
|
def rename_queue(old_name, new_name)
|
|
77
110
|
rec = @queues.delete(old_name) or return
|
|
78
111
|
rec.name = new_name
|
|
79
112
|
@queues[new_name] = rec
|
|
113
|
+
@suspended << new_name if @suspended.delete?(old_name)
|
|
80
114
|
@queue_bindings.each { |b| b.queue = new_name if b.queue == old_name }
|
|
81
115
|
@consumers.transform_values! { |queue| queue == old_name ? new_name : queue }
|
|
82
116
|
rec.objects.each { |q| q.update_name_to(new_name) }
|
|
@@ -135,11 +169,11 @@ module AsyncRabbitMQ
|
|
|
135
169
|
end
|
|
136
170
|
|
|
137
171
|
def queues
|
|
138
|
-
apply(:filter_queues, @queues.
|
|
172
|
+
apply(:filter_queues, @queues.each_value.reject { |q| @suspended.include?(q.name) })
|
|
139
173
|
end
|
|
140
174
|
|
|
141
175
|
def queue_bindings
|
|
142
|
-
apply(:filter_queue_bindings, @queue_bindings.
|
|
176
|
+
apply(:filter_queue_bindings, @queue_bindings.reject { |b| @suspended.include?(b.queue) })
|
|
143
177
|
end
|
|
144
178
|
|
|
145
179
|
def exchange_bindings
|
|
@@ -159,6 +193,7 @@ module AsyncRabbitMQ
|
|
|
159
193
|
@queue_bindings.clear
|
|
160
194
|
@exchange_bindings.clear
|
|
161
195
|
@consumers.clear
|
|
196
|
+
@suspended.clear
|
|
162
197
|
end
|
|
163
198
|
|
|
164
199
|
# Forget only what was tied to the connection that just went: exclusive
|
|
@@ -174,6 +209,7 @@ module AsyncRabbitMQ
|
|
|
174
209
|
|
|
175
210
|
dropped_queues.each { |name| @queues.delete(name) }
|
|
176
211
|
dropped_exchanges.each { |name| @exchanges.delete(name) }
|
|
212
|
+
@suspended.subtract(dropped_queues)
|
|
177
213
|
|
|
178
214
|
@queue_bindings.reject! do |b|
|
|
179
215
|
dropped_queues.include?(b.queue) || dropped_exchanges.include?(b.exchange)
|