async-rabbitmq 0.8.0 → 0.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +42 -0
- data/README.md +26 -14
- data/lib/async_rabbitmq/channel.rb +215 -65
- data/lib/async_rabbitmq/topology_registry.rb +38 -2
- data/lib/async_rabbitmq/version.rb +1 -1
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 91a60c68246f2c413fadd2c183b925062b38a4e36ac1e8b1e4a21eff2ca49c97
|
|
4
|
+
data.tar.gz: 2d3a70204f101c856539b921255d541d8566af490ea468f57753ee22f1e69708
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 869b317e73b1f566e37a04cc7e22e5534f26cfbc3168d11af3b2477ffda77d8f3759270e249b2bab480b94c0c0cf7d8f3d1610fd3bd425b493bfbd9844c69a9f
|
|
7
|
+
data.tar.gz: e5f6720a22811c30c2fbb600faf55c266a6fe298694d203a5e7499bd3218dc74f6471687624cd5ef564f410043009b8bf352102eaf3c0acc0ab59d2dd96049a2
|
data/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,48 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
|
4
4
|
|
|
5
|
+
## [0.8.1] - 2026-11-15
|
|
6
|
+
|
|
7
|
+
### Fixed
|
|
8
|
+
|
|
9
|
+
- A consumer the broker cancelled is no longer given up on when the connection goes at the same
|
|
10
|
+
moment. Deleting a quorum queue is both events at once: the consumer is cancelled, and the
|
|
11
|
+
re-subscribe is answered with a connection-level `541 INTERNAL_ERROR ... noproc` that takes the
|
|
12
|
+
connection down. 0.8.0's recovery loop stopped at the first sight of a session that was not
|
|
13
|
+
open, and nothing else would have brought that consumer back - it is no longer in the channel's
|
|
14
|
+
registry, so the reconnect does not re-register it either. It now waits through however many
|
|
15
|
+
reconnects it takes, and gives up only when the channel or the session is finished for good.
|
|
16
|
+
- A queue whose consumer the broker cancelled is held out of topology recovery until the client
|
|
17
|
+
has established whether it still exists. Without that the two race: a reconnect re-declares the
|
|
18
|
+
whole registry, so a queue deleted a moment earlier came back - and a queue the broker hands
|
|
19
|
+
back with a consumer attached cannot be deleted at all, which is the bug 0.8.0 set out to fix.
|
|
20
|
+
Measured against RabbitMQ 4.4, deleting a quorum queue reached exactly that state.
|
|
21
|
+
- A connection lost mid-re-subscribe is no longer reported as `consumer.recovery_failed`. It has
|
|
22
|
+
its own event and its own recovery, both already running, and an application alarming on
|
|
23
|
+
recovery failures was told twice about one outage, in the words of the more serious of the two.
|
|
24
|
+
- A `404` that says the queue's process is stopped, rather than that there is no such queue, is no
|
|
25
|
+
longer read as a deletion. A classic queue on a stopped node answers `NOT_FOUND - queue 'q' in
|
|
26
|
+
vhost '/' process is stopped by supervisor` and comes back with its node; the client forgot it,
|
|
27
|
+
so the consumer was never restored and the queue was never re-declared either. The probe now
|
|
28
|
+
answers `:deleted`, `:available`, `:unavailable` or `:unreachable`, and only `:available` is followed by an
|
|
29
|
+
attempt to consume - a `basic_consume` the broker answers with a 404 closes the channel and
|
|
30
|
+
takes the consumers still working on it down with it.
|
|
31
|
+
- `on_cancel` now runs to completion before the client decides what became of the queue. The two
|
|
32
|
+
ran alongside each other, and the documented way to keep a queue - re-declare it in `on_cancel` -
|
|
33
|
+
is an RPC that yields, so the probe sometimes got its answer in first and concluded the queue
|
|
34
|
+
was deleted. The callback is still off the dispatch fiber, so an RPC in it still gets its reply
|
|
35
|
+
and an exception in it still cannot stop the channel.
|
|
36
|
+
- A cancelled queue is released back to topology recovery only on a definite answer that it
|
|
37
|
+
still exists. A probe the broker could not answer is not evidence the queue is there, and
|
|
38
|
+
treating it as such re-declared a queue that had just been deleted - the one thing this
|
|
39
|
+
handling exists to prevent. The queue now stays held back until the broker says `:deleted`
|
|
40
|
+
or `:available`, retrying with the same backoff the recovering path uses.
|
|
41
|
+
- The retry backoff no longer grows for a failure that was the connection's rather than the
|
|
42
|
+
queue's, and resets once the connection is back. A two-second blip could otherwise leave a
|
|
43
|
+
healthy queue unconsumed for the next thirty seconds, since each unanswerable probe doubled the
|
|
44
|
+
delay. The queue probe now answers `:unreachable` - no answer at all - separately from
|
|
45
|
+
`:unavailable`, which is the broker saying the queue is not being served.
|
|
46
|
+
|
|
5
47
|
## [0.8.0] - 2026-11-14
|
|
6
48
|
|
|
7
49
|
### Fixed
|
data/README.md
CHANGED
|
@@ -472,28 +472,40 @@ logged rather than taken out on the channel.
|
|
|
472
472
|
|
|
473
473
|
### A consumer the broker cancels
|
|
474
474
|
|
|
475
|
-
RabbitMQ cancels a consumer when it can no longer serve the queue to it
|
|
476
|
-
|
|
477
|
-
|
|
478
|
-
|
|
479
|
-
worker goes on publishing and heartbeating while consuming nothing.
|
|
475
|
+
RabbitMQ cancels a consumer when it can no longer serve the queue to it. The
|
|
476
|
+
connection is untouched, so none of the recovery above runs — a `basic.cancel`
|
|
477
|
+
frame is the only notice there is, and without `on_cancel` the worker goes on
|
|
478
|
+
publishing and heartbeating while consuming nothing.
|
|
480
479
|
|
|
481
480
|
Nothing is re-subscribed by default, which matches Bunny, the Java client and
|
|
482
481
|
the .NET client. `recover_cancelled_consumers: true`, on the session or on one
|
|
483
482
|
channel, re-subscribes under the same consumer tag, backing off from 1s to 30s
|
|
484
|
-
for as long as the queue exists
|
|
483
|
+
for as long as the queue exists, through however many reconnects that takes. It
|
|
484
|
+
gives up only when the channel or the session is finished for good.
|
|
485
485
|
|
|
486
486
|
**Recovery never declares anything.** One of the reasons a consumer gets
|
|
487
487
|
cancelled is somebody deleting the queue, and a client that answered that by
|
|
488
488
|
re-declaring would hand back a queue nobody could get rid of while a consumer
|
|
489
|
-
was attached. So the client
|
|
490
|
-
and re-consumes only if it is.
|
|
491
|
-
|
|
492
|
-
|
|
493
|
-
|
|
494
|
-
|
|
495
|
-
|
|
496
|
-
|
|
489
|
+
was attached. So the client asks the broker, on a channel of its own, whether
|
|
490
|
+
the queue is still there, and re-consumes only if it is. Re-declaring is the
|
|
491
|
+
application's call, and `on_cancel` is where to make it: that callback is run
|
|
492
|
+
to completion before the client goes asking, so a queue declared there is
|
|
493
|
+
there when it looks.
|
|
494
|
+
|
|
495
|
+
Three things follow from that, and each of them is a bug in some client:
|
|
496
|
+
|
|
497
|
+
- **The queue is held out of topology recovery until the client knows.**
|
|
498
|
+
Otherwise the two race — a reconnect re-declares the whole registry — and the
|
|
499
|
+
queue somebody deleted comes back. Deleting a quorum queue is exactly that
|
|
500
|
+
race arriving as one event: the cancel, and then a connection-level `541
|
|
501
|
+
INTERNAL_ERROR ... noproc` in answer to the re-subscribe.
|
|
502
|
+
- **The question is asked before every attempt, never the attempt instead of
|
|
503
|
+
the question.** A `basic_consume` the broker answers `404` closes the
|
|
504
|
+
channel, and takes the consumers still working on it down with it.
|
|
505
|
+
- **`NOT_FOUND` is two answers.** `no queue 'q' in vhost '/'` is a deletion;
|
|
506
|
+
`queue 'q' in vhost '/' process is stopped by supervisor`, which is what a
|
|
507
|
+
classic queue on a stopped node gives, is a queue that comes back with its
|
|
508
|
+
node. The second is waited for, not forgotten.
|
|
497
509
|
|
|
498
510
|
```ruby
|
|
499
511
|
session = AsyncRabbitMQ::Session.new(..., recover_cancelled_consumers: true)
|
|
@@ -77,6 +77,15 @@ module AsyncRabbitMQ
|
|
|
77
77
|
RECOVER_CONSUMER_DELAY = 1
|
|
78
78
|
RECOVER_CONSUMER_MAX_DELAY = 30
|
|
79
79
|
|
|
80
|
+
# A queue that is still declared and a queue that is gone both answer
|
|
81
|
+
# NOT_FOUND to a passive declare, and only the reply text tells them apart.
|
|
82
|
+
# Measured on RabbitMQ 4.4: a deleted queue gives "no queue 'q' in vhost
|
|
83
|
+
# '/'", while a classic queue whose node has been stopped gives "queue 'q'
|
|
84
|
+
# in vhost '/' process is stopped by supervisor" and comes back with its
|
|
85
|
+
# node. Older brokers say "home node ... is down or inaccessible" for the
|
|
86
|
+
# same thing. See #queue_missing?.
|
|
87
|
+
QUEUE_UNAVAILABLE = /process is stopped|is down or inaccessible/
|
|
88
|
+
|
|
80
89
|
# Pushed onto the work queue to retire one handler worker. Compared by
|
|
81
90
|
# identity, so it can never collide with a delivery.
|
|
82
91
|
WORKER_STOP = Object.new.freeze
|
|
@@ -1407,20 +1416,30 @@ module AsyncRabbitMQ
|
|
|
1407
1416
|
#
|
|
1408
1417
|
# Bunny hands cancel notifications to its consumer work pool and the Java
|
|
1409
1418
|
# client to its work service, for the same reason.
|
|
1410
|
-
def notify_app(name)
|
|
1411
|
-
@session.spawn_background
|
|
1412
|
-
|
|
1413
|
-
|
|
1414
|
-
|
|
1415
|
-
|
|
1416
|
-
|
|
1417
|
-
|
|
1419
|
+
def notify_app(name, &block)
|
|
1420
|
+
@session.spawn_background { run_callback(name, &block) }
|
|
1421
|
+
end
|
|
1422
|
+
|
|
1423
|
+
# The body of #notify_app, for a caller that is already off the dispatch
|
|
1424
|
+
# fiber and has something to do after the callback.
|
|
1425
|
+
def run_callback(name)
|
|
1426
|
+
yield
|
|
1427
|
+
rescue *PROPAGATE_FROM_HANDLER
|
|
1428
|
+
raise
|
|
1429
|
+
rescue Exception => e # rubocop:disable Lint/RescueException
|
|
1430
|
+
@logger.error("Channel #{@channel_id}: #{name} callback raised #{e.class}: #{e.message}")
|
|
1418
1431
|
end
|
|
1419
1432
|
|
|
1420
1433
|
# The broker cancels a consumer when it can no longer serve the queue to
|
|
1421
|
-
# it: the queue was deleted, the node holding
|
|
1422
|
-
#
|
|
1423
|
-
#
|
|
1434
|
+
# it: the queue was deleted, or the node holding a classic queue stopped.
|
|
1435
|
+
# On a three-node RabbitMQ 4.4.0 cluster, moving a quorum queue's
|
|
1436
|
+
# leadership did not do it - not a graceful stop of the leader's node, not
|
|
1437
|
+
# removing the member on the consuming node, not losing the majority, and
|
|
1438
|
+
# not cutting the leader's node off the network until the queue re-elected.
|
|
1439
|
+
# Other versions are reported to cancel on unavailability; it makes no
|
|
1440
|
+
# difference here, since the handling is the same whatever the cause. The
|
|
1441
|
+
# connection is untouched either way, so connection recovery never runs:
|
|
1442
|
+
# this frame is the only notice there is.
|
|
1424
1443
|
#
|
|
1425
1444
|
# The consumer goes from @consumers first, so no delivery is dispatched to
|
|
1426
1445
|
# a block that is no longer subscribed, and everything beyond that
|
|
@@ -1430,20 +1449,33 @@ module AsyncRabbitMQ
|
|
|
1430
1449
|
topology&.delete_consumer(tag)
|
|
1431
1450
|
return wake_each_waiters(tag) unless entry
|
|
1432
1451
|
|
|
1452
|
+
# Held back from topology recovery from this moment, before anything can
|
|
1453
|
+
# reconnect, because one of the reasons a consumer is cancelled is the
|
|
1454
|
+
# queue being deleted and a reconnect re-declares the registry wholesale.
|
|
1455
|
+
# Settled by #reconcile_cancelled_consumer, either way.
|
|
1456
|
+
topology&.suspend_queue(entry[:queue_name])
|
|
1433
1457
|
@logger.warn("Channel #{@channel_id}: broker cancelled consumer #{tag} on #{entry[:queue_name]}")
|
|
1434
1458
|
instrument("consumer.cancelled") do
|
|
1435
1459
|
{ channel: @channel_id, consumer_tag: tag, queue: entry[:queue_name], reason: :broker }
|
|
1436
1460
|
end
|
|
1437
|
-
|
|
1438
|
-
|
|
1461
|
+
# The callback first and the reconcile after it, in one task rather than
|
|
1462
|
+
# two. on_cancel is where an application re-declares a queue it wants
|
|
1463
|
+
# back, and a reconcile running alongside it sometimes asked the broker
|
|
1464
|
+
# before that declare had landed and concluded the queue was gone. Still
|
|
1465
|
+
# off the dispatch fiber, so an RPC in the callback can get its reply and
|
|
1466
|
+
# an exception in it cannot stop the channel.
|
|
1467
|
+
@session.spawn_background do
|
|
1468
|
+
run_callback("on_cancel") { @on_cancel.call(tag) } if @on_cancel
|
|
1469
|
+
reconcile_cancelled_consumer(tag, entry)
|
|
1470
|
+
end
|
|
1439
1471
|
end
|
|
1440
1472
|
|
|
1441
|
-
# Why the broker cancelled is not on the wire, and the
|
|
1442
|
-
# opposite handling: a queue that
|
|
1443
|
-
#
|
|
1444
|
-
#
|
|
1445
|
-
#
|
|
1446
|
-
#
|
|
1473
|
+
# Why the broker cancelled is not on the wire, and the reasons want
|
|
1474
|
+
# opposite handling: a queue that is merely not being served should be
|
|
1475
|
+
# consumed from again, while a queue somebody deleted must be left deleted.
|
|
1476
|
+
# A passive declare is the only way to tell them apart. Spring AMQP's
|
|
1477
|
+
# listener container probes the same way, and for the same reason declares
|
|
1478
|
+
# nothing.
|
|
1447
1479
|
#
|
|
1448
1480
|
# Runs whether or not this channel recovers cancelled consumers, because
|
|
1449
1481
|
# forgetting a deleted queue is not a recovery feature: a queue left in the
|
|
@@ -1451,10 +1483,72 @@ module AsyncRabbitMQ
|
|
|
1451
1483
|
# then the deletion never sticks.
|
|
1452
1484
|
def reconcile_cancelled_consumer(tag, entry)
|
|
1453
1485
|
queue = entry[:queue_name]
|
|
1454
|
-
return
|
|
1455
|
-
|
|
1486
|
+
return recover_cancelled_consumer(tag, entry) if recovers_cancelled_consumers?
|
|
1487
|
+
|
|
1488
|
+
# Not putting the consumer back, but a queue that has been deleted must
|
|
1489
|
+
# still leave the registry or the next reconnect declares it again.
|
|
1490
|
+
# Nothing is asked of a connection that is not there: a question put to a
|
|
1491
|
+
# session that is down cannot be answered, and guessing here is what puts
|
|
1492
|
+
# a deleted queue back.
|
|
1493
|
+
#
|
|
1494
|
+
# So the queue stays suspended until the broker gives a definite answer,
|
|
1495
|
+
# and only :available releases it. Releasing on a probe that could not be
|
|
1496
|
+
# answered is how a deleted queue comes back - it is not evidence the
|
|
1497
|
+
# queue is there, and 46_broker_amnesia_spec is the case that proves it.
|
|
1498
|
+
delay = RECOVER_CONSUMER_DELAY
|
|
1499
|
+
|
|
1500
|
+
loop do
|
|
1501
|
+
case wait_for_usable_channel
|
|
1502
|
+
when nil then return release_cancelled_consumer(tag, queue)
|
|
1503
|
+
when :reconnected then delay = RECOVER_CONSUMER_DELAY
|
|
1504
|
+
end
|
|
1505
|
+
|
|
1506
|
+
case queue_status(queue)
|
|
1507
|
+
when :deleted then return forget_deleted_queue(tag, queue)
|
|
1508
|
+
when :available then return release_cancelled_consumer(tag, queue)
|
|
1509
|
+
end
|
|
1510
|
+
|
|
1511
|
+
# :unavailable - the broker says the queue is not being served - and
|
|
1512
|
+
# :unreachable - no answer at all - both mean ask again, never "the
|
|
1513
|
+
# queue is there". Sleeping on either is also what keeps this from
|
|
1514
|
+
# spinning when the probe cannot be answered.
|
|
1515
|
+
sleep delay
|
|
1516
|
+
delay = [delay * 2, RECOVER_CONSUMER_MAX_DELAY].min
|
|
1517
|
+
end
|
|
1518
|
+
end
|
|
1519
|
+
|
|
1520
|
+
# Wait until this channel can carry a consumer again, through however many
|
|
1521
|
+
# reconnects that takes. :ready if it already could, :reconnected if it had
|
|
1522
|
+
# to wait for one, nil if it never will.
|
|
1523
|
+
#
|
|
1524
|
+
# Giving up on a connection loss instead would lose the consumer for good:
|
|
1525
|
+
# it is no longer in @consumers, so the reconnect does not re-register it,
|
|
1526
|
+
# and this loop is the only thing that would have brought it back. That is
|
|
1527
|
+
# precisely the case a quorum queue produces - the re-subscribe is answered
|
|
1528
|
+
# with a connection-level 541, not a channel error.
|
|
1529
|
+
#
|
|
1530
|
+
# Over for good when the channel is closed, which means the application
|
|
1531
|
+
# closed it or recovery could not reopen it and its consumers belong to
|
|
1532
|
+
# whoever does that, or when the session has stopped trying.
|
|
1533
|
+
def wait_for_usable_channel
|
|
1534
|
+
waited = false
|
|
1535
|
+
loop do
|
|
1536
|
+
return waited ? :reconnected : :ready if open? && @session.open?
|
|
1537
|
+
return nil if closed? || @session.closed?
|
|
1456
1538
|
|
|
1457
|
-
|
|
1539
|
+
waited = true
|
|
1540
|
+
sleep RECOVER_CONSUMER_DELAY
|
|
1541
|
+
end
|
|
1542
|
+
end
|
|
1543
|
+
|
|
1544
|
+
# Let the queue be recovered again and release anyone in #each. For a
|
|
1545
|
+
# consumer that will not be followed up - this channel does not recover
|
|
1546
|
+
# cancelled consumers, or there is nothing left to recover it onto - the
|
|
1547
|
+
# queue must not stay held back: it was never established that it is gone,
|
|
1548
|
+
# and a session opened again would not re-declare it.
|
|
1549
|
+
def release_cancelled_consumer(tag, queue)
|
|
1550
|
+
topology&.resume_queue(queue)
|
|
1551
|
+
wake_each_waiters(tag)
|
|
1458
1552
|
end
|
|
1459
1553
|
|
|
1460
1554
|
# Re-consume under the same tag, for as long as the queue is there and this
|
|
@@ -1466,43 +1560,87 @@ module AsyncRabbitMQ
|
|
|
1466
1560
|
failures = 0
|
|
1467
1561
|
|
|
1468
1562
|
loop do
|
|
1469
|
-
#
|
|
1470
|
-
#
|
|
1471
|
-
|
|
1563
|
+
# Every round starts with a connection that works and a fresh answer
|
|
1564
|
+
# about the queue. A queue deleted while we were retrying has to stop
|
|
1565
|
+
# the loop, and a basic_consume the broker answers with a 404 closes
|
|
1566
|
+
# this channel and takes the consumers still working on it down too -
|
|
1567
|
+
# so the question is asked on a channel of its own first, and only an
|
|
1568
|
+
# answer of "it is there" is followed by an attempt to consume.
|
|
1569
|
+
case wait_for_usable_channel
|
|
1570
|
+
when nil then return release_cancelled_consumer(tag, queue)
|
|
1571
|
+
when :reconnected
|
|
1572
|
+
# The connection was what was wrong, and it is back: start the
|
|
1573
|
+
# backoff again rather than sleeping out a delay earned by an outage
|
|
1574
|
+
# that is over. A two-second blip should not leave a healthy queue
|
|
1575
|
+
# unconsumed for the next thirty.
|
|
1576
|
+
delay = RECOVER_CONSUMER_DELAY
|
|
1577
|
+
end
|
|
1472
1578
|
|
|
1473
|
-
|
|
1474
|
-
|
|
1475
|
-
|
|
1476
|
-
|
|
1477
|
-
|
|
1478
|
-
|
|
1479
|
-
|
|
1480
|
-
|
|
1481
|
-
|
|
1482
|
-
|
|
1483
|
-
|
|
1484
|
-
|
|
1485
|
-
|
|
1486
|
-
|
|
1487
|
-
|
|
1488
|
-
|
|
1489
|
-
|
|
1490
|
-
|
|
1491
|
-
|
|
1492
|
-
instrument("consumer.recovery_failed") do
|
|
1493
|
-
{ channel: @channel_id, consumer_tag: tag, queue: queue, reason: :error, error: e }
|
|
1579
|
+
case queue_status(queue)
|
|
1580
|
+
when :deleted
|
|
1581
|
+
return forget_deleted_queue(tag, queue)
|
|
1582
|
+
when :unreachable
|
|
1583
|
+
# No answer rather than an answer: the connection is what is wrong,
|
|
1584
|
+
# not the queue, so the backoff a queue earns does not apply. Go back
|
|
1585
|
+
# and wait for the connection instead of sleeping out a delay that
|
|
1586
|
+
# doubles every time the broker cannot be reached.
|
|
1587
|
+
delay = RECOVER_CONSUMER_DELAY
|
|
1588
|
+
when :available
|
|
1589
|
+
begin
|
|
1590
|
+
basic_consume(queue, consumer_tag: tag, manual_ack: entry[:manual_ack], exclusive: entry[:exclusive],
|
|
1591
|
+
arguments: entry[:arguments] || {}, on_error: entry[:on_error], &entry[:block])
|
|
1592
|
+
# Consuming again, so the queue is there and recovery can have it
|
|
1593
|
+
# back (#handle_broker_cancel held it).
|
|
1594
|
+
topology&.resume_queue(queue)
|
|
1595
|
+
@logger.info("Channel #{@channel_id}: re-subscribed consumer #{tag} to #{queue}")
|
|
1596
|
+
instrument("consumer.recovered") do
|
|
1597
|
+
{ channel: @channel_id, consumer_tag: tag, queue: queue, attempts: failures + 1 }
|
|
1494
1598
|
end
|
|
1495
|
-
|
|
1496
|
-
|
|
1599
|
+
return
|
|
1600
|
+
rescue *PROPAGATE_FROM_HANDLER
|
|
1601
|
+
raise
|
|
1602
|
+
rescue ConnectionError, NotOpenError => e
|
|
1603
|
+
# The connection went, not the subscription. That has its own
|
|
1604
|
+
# reporting and its own recovery, both already running, so it is
|
|
1605
|
+
# not counted as a failure to re-subscribe: an application alarming
|
|
1606
|
+
# on consumer.recovery_failed would otherwise be told twice about
|
|
1607
|
+
# one event, and told it in the words of the more serious of the
|
|
1608
|
+
# two.
|
|
1609
|
+
@logger.debug("Channel #{@channel_id}: re-subscribe of #{tag} interrupted by " \
|
|
1610
|
+
"#{e.class}; waiting for the connection")
|
|
1611
|
+
rescue ChannelError => e
|
|
1612
|
+
# The queue went between the question and the answer. The broker
|
|
1613
|
+
# has closed this channel saying so, so there is nothing left to
|
|
1614
|
+
# retry on.
|
|
1615
|
+
return forget_deleted_queue(tag, queue) if queue_missing?(e)
|
|
1616
|
+
|
|
1617
|
+
failures += 1
|
|
1618
|
+
note_recovery_failure(tag, queue, e, failures)
|
|
1619
|
+
rescue => e
|
|
1620
|
+
failures += 1
|
|
1621
|
+
note_recovery_failure(tag, queue, e, failures)
|
|
1497
1622
|
end
|
|
1498
1623
|
end
|
|
1624
|
+
# :unavailable falls through to the backoff and is asked again.
|
|
1499
1625
|
|
|
1500
1626
|
sleep delay
|
|
1501
1627
|
delay = [delay * 2, RECOVER_CONSUMER_MAX_DELAY].min
|
|
1502
|
-
|
|
1503
|
-
|
|
1504
|
-
|
|
1505
|
-
|
|
1628
|
+
end
|
|
1629
|
+
end
|
|
1630
|
+
|
|
1631
|
+
# One line per episode at warn, the rest at debug: a broker that is away
|
|
1632
|
+
# for an hour would otherwise write a warning every attempt for the whole
|
|
1633
|
+
# hour, and an application counting warnings would read one outage as
|
|
1634
|
+
# hundreds.
|
|
1635
|
+
def note_recovery_failure(tag, queue, error, failures)
|
|
1636
|
+
if failures == 1
|
|
1637
|
+
@logger.warn("Channel #{@channel_id}: could not re-subscribe #{tag} to #{queue}, " \
|
|
1638
|
+
"retrying: #{error.class}: #{error.message}")
|
|
1639
|
+
instrument("consumer.recovery_failed") do
|
|
1640
|
+
{ channel: @channel_id, consumer_tag: tag, queue: queue, reason: :error, error: error }
|
|
1641
|
+
end
|
|
1642
|
+
else
|
|
1643
|
+
@logger.debug("Channel #{@channel_id}: re-subscribe attempt #{failures} for #{tag} failed: #{error.message}")
|
|
1506
1644
|
end
|
|
1507
1645
|
end
|
|
1508
1646
|
|
|
@@ -1519,26 +1657,38 @@ module AsyncRabbitMQ
|
|
|
1519
1657
|
wake_each_waiters(tag)
|
|
1520
1658
|
end
|
|
1521
1659
|
|
|
1522
|
-
#
|
|
1523
|
-
# of its own because a passive declare
|
|
1524
|
-
# on - which would take the queue's
|
|
1660
|
+
# Is the queue gone, there, or merely not being served right now? A passive
|
|
1661
|
+
# declare answers it, on a channel of its own because a passive declare
|
|
1662
|
+
# that fails closes the channel it ran on - which would take the queue's
|
|
1663
|
+
# fellow consumers down with it.
|
|
1525
1664
|
#
|
|
1526
|
-
#
|
|
1527
|
-
#
|
|
1528
|
-
#
|
|
1529
|
-
#
|
|
1530
|
-
#
|
|
1531
|
-
#
|
|
1532
|
-
|
|
1665
|
+
# Four answers, because conflating any two of them does harm. :deleted must
|
|
1666
|
+
# be forgotten and :unavailable must be waited for, and telling those apart
|
|
1667
|
+
# is the whole point. :unreachable - no answer at all, because the
|
|
1668
|
+
# connection is in trouble - is separated from :unavailable so that the
|
|
1669
|
+
# caller does not spend a backoff earned by a queue on a connection that is
|
|
1670
|
+
# simply down, and is never taken for :available, because a basic_consume
|
|
1671
|
+
# on a queue that is not there closes the channel.
|
|
1672
|
+
def queue_status(name)
|
|
1533
1673
|
@session.with_channel { |probe| probe.queue(name, passive: true) }
|
|
1534
|
-
|
|
1674
|
+
:available
|
|
1535
1675
|
rescue ChannelError => e
|
|
1536
|
-
e
|
|
1676
|
+
queue_missing?(e) ? :deleted : :unavailable
|
|
1537
1677
|
rescue *PROPAGATE_FROM_HANDLER
|
|
1538
1678
|
raise
|
|
1539
1679
|
rescue => e
|
|
1540
1680
|
@logger.debug("Channel #{@channel_id}: could not check whether #{name} still exists: #{e.class}: #{e.message}")
|
|
1541
|
-
|
|
1681
|
+
:unreachable
|
|
1682
|
+
end
|
|
1683
|
+
|
|
1684
|
+
# Gone, rather than merely not being served. Taking a queue whose node is
|
|
1685
|
+
# down for a deleted one forgets a queue that is coming back: its consumer
|
|
1686
|
+
# is never restored, and it is never re-declared either. Any 404 the
|
|
1687
|
+
# QUEUE_UNAVAILABLE wordings do not cover reads as a deletion, which is the
|
|
1688
|
+
# safe way round - an unmatched wording costs only what forgetting it costs
|
|
1689
|
+
# today.
|
|
1690
|
+
def queue_missing?(error)
|
|
1691
|
+
error.code == 404 && !QUEUE_UNAVAILABLE.match?(error.text.to_s)
|
|
1542
1692
|
end
|
|
1543
1693
|
|
|
1544
1694
|
# Release fibers blocked in #each for one consumer, or for all of them.
|
|
@@ -38,6 +38,7 @@ module AsyncRabbitMQ
|
|
|
38
38
|
@queue_bindings = []
|
|
39
39
|
@exchange_bindings = []
|
|
40
40
|
@consumers = {} # consumer tag => queue name
|
|
41
|
+
@suspended = Set.new # queue names held back from recovery
|
|
41
42
|
end
|
|
42
43
|
|
|
43
44
|
# --- recording -----------------------------------------------------------
|
|
@@ -59,6 +60,9 @@ module AsyncRabbitMQ
|
|
|
59
60
|
|
|
60
61
|
def record_queue(channel_id, name, durable:, exclusive:, auto_delete:, arguments:, server_named:, object: nil)
|
|
61
62
|
existing = @queues[name]
|
|
63
|
+
# Declared again by the application, so whatever doubt a cancelled
|
|
64
|
+
# consumer raised about it is settled.
|
|
65
|
+
@suspended.delete(name)
|
|
62
66
|
rec = RecordedQueue.new(channel_id: channel_id, name: name, durable: durable, exclusive: exclusive,
|
|
63
67
|
auto_delete: auto_delete, arguments: arguments.dup, server_named: server_named,
|
|
64
68
|
objects: existing ? existing.objects : [])
|
|
@@ -68,15 +72,45 @@ module AsyncRabbitMQ
|
|
|
68
72
|
|
|
69
73
|
def delete_queue(name)
|
|
70
74
|
@queues.delete(name)
|
|
75
|
+
@suspended.delete(name)
|
|
71
76
|
gone, @queue_bindings = @queue_bindings.partition { |b| b.queue == name }
|
|
72
77
|
gone.each { |b| prune_auto_delete_exchange(b.exchange) }
|
|
73
78
|
end
|
|
74
79
|
|
|
80
|
+
# --- suspension ----------------------------------------------------------
|
|
81
|
+
#
|
|
82
|
+
# A queue is suspended while the client is working out whether it is still
|
|
83
|
+
# there, which is what a consumer the broker cancelled leaves it not
|
|
84
|
+
# knowing. Recovery skips a suspended queue and its bindings.
|
|
85
|
+
#
|
|
86
|
+
# The alternative - keep recovering it until the client is sure - puts a
|
|
87
|
+
# deleted queue back, because the deletion and the reconnect race: the node
|
|
88
|
+
# that was serving the queue is often the one that just went, and a
|
|
89
|
+
# reconnect re-declares the whole registry before anything can ask the new
|
|
90
|
+
# connection whether the queue survived. A queue the broker then hands back
|
|
91
|
+
# with a consumer attached cannot be deleted at all. Holding it back costs
|
|
92
|
+
# nothing by comparison: the worst case is a queue that was fine and is not
|
|
93
|
+
# re-declared for the few seconds it takes to establish that.
|
|
94
|
+
|
|
95
|
+
def suspend_queue(name)
|
|
96
|
+
@suspended << name if @queues.key?(name)
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
def resume_queue(name)
|
|
100
|
+
@suspended.delete(name)
|
|
101
|
+
end
|
|
102
|
+
|
|
103
|
+
# Queue names currently held back, for inspection.
|
|
104
|
+
def suspended_queues
|
|
105
|
+
@suspended.dup
|
|
106
|
+
end
|
|
107
|
+
|
|
75
108
|
# A server-named queue was re-declared under a different name.
|
|
76
109
|
def rename_queue(old_name, new_name)
|
|
77
110
|
rec = @queues.delete(old_name) or return
|
|
78
111
|
rec.name = new_name
|
|
79
112
|
@queues[new_name] = rec
|
|
113
|
+
@suspended << new_name if @suspended.delete?(old_name)
|
|
80
114
|
@queue_bindings.each { |b| b.queue = new_name if b.queue == old_name }
|
|
81
115
|
@consumers.transform_values! { |queue| queue == old_name ? new_name : queue }
|
|
82
116
|
rec.objects.each { |q| q.update_name_to(new_name) }
|
|
@@ -135,11 +169,11 @@ module AsyncRabbitMQ
|
|
|
135
169
|
end
|
|
136
170
|
|
|
137
171
|
def queues
|
|
138
|
-
apply(:filter_queues, @queues.
|
|
172
|
+
apply(:filter_queues, @queues.each_value.reject { |q| @suspended.include?(q.name) })
|
|
139
173
|
end
|
|
140
174
|
|
|
141
175
|
def queue_bindings
|
|
142
|
-
apply(:filter_queue_bindings, @queue_bindings.
|
|
176
|
+
apply(:filter_queue_bindings, @queue_bindings.reject { |b| @suspended.include?(b.queue) })
|
|
143
177
|
end
|
|
144
178
|
|
|
145
179
|
def exchange_bindings
|
|
@@ -159,6 +193,7 @@ module AsyncRabbitMQ
|
|
|
159
193
|
@queue_bindings.clear
|
|
160
194
|
@exchange_bindings.clear
|
|
161
195
|
@consumers.clear
|
|
196
|
+
@suspended.clear
|
|
162
197
|
end
|
|
163
198
|
|
|
164
199
|
# Forget only what was tied to the connection that just went: exclusive
|
|
@@ -174,6 +209,7 @@ module AsyncRabbitMQ
|
|
|
174
209
|
|
|
175
210
|
dropped_queues.each { |name| @queues.delete(name) }
|
|
176
211
|
dropped_exchanges.each { |name| @exchanges.delete(name) }
|
|
212
|
+
@suspended.subtract(dropped_queues)
|
|
177
213
|
|
|
178
214
|
@queue_bindings.reject! do |b|
|
|
179
215
|
dropped_queues.include?(b.queue) || dropped_exchanges.include?(b.exchange)
|