async-rabbitmq 0.8.0 → 0.8.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 42cacea6b44ce97ab03abac57c1d7b0b198103ba0225d8c9bc8aa37f6145fbb4
4
- data.tar.gz: ac847f5cd9f87acaa811a539a728b5b5bb6eb3c4694b0408733775e705266676
3
+ metadata.gz: 91a60c68246f2c413fadd2c183b925062b38a4e36ac1e8b1e4a21eff2ca49c97
4
+ data.tar.gz: 2d3a70204f101c856539b921255d541d8566af490ea468f57753ee22f1e69708
5
5
  SHA512:
6
- metadata.gz: 709f0f1339d28fa2ed77d031f375967970cb94d8d879e230bb3c2a223c84a701573ea5ed36b3fd5dbf24eb38ce0acf83d0ec234d23da7478c383621290d48ce9
7
- data.tar.gz: fbec76e2babf6f6f32ca6e7f86eea6fa2644609102a9bc7b936e72a48ee236b9d0f964b021a17846b6aac95c768072f7a943adbadb650272b4c230d7dd93f79d
6
+ metadata.gz: 869b317e73b1f566e37a04cc7e22e5534f26cfbc3168d11af3b2477ffda77d8f3759270e249b2bab480b94c0c0cf7d8f3d1610fd3bd425b493bfbd9844c69a9f
7
+ data.tar.gz: e5f6720a22811c30c2fbb600faf55c266a6fe298694d203a5e7499bd3218dc74f6471687624cd5ef564f410043009b8bf352102eaf3c0acc0ab59d2dd96049a2
data/CHANGELOG.md CHANGED
@@ -2,6 +2,48 @@
2
2
 
3
3
  All notable changes to this project will be documented in this file.
4
4
 
5
+ ## [0.8.1] - 2026-11-15
6
+
7
+ ### Fixed
8
+
9
+ - A consumer the broker cancelled is no longer given up on when the connection goes at the same
10
+ moment. Deleting a quorum queue is both events at once: the consumer is cancelled, and the
11
+ re-subscribe is answered with a connection-level `541 INTERNAL_ERROR ... noproc` that takes the
12
+ connection down. 0.8.0's recovery loop stopped at the first sight of a session that was not
13
+ open, and nothing else would have brought that consumer back - it is no longer in the channel's
14
+ registry, so the reconnect does not re-register it either. It now waits through however many
15
+ reconnects it takes, and gives up only when the channel or the session is finished for good.
16
+ - A queue whose consumer the broker cancelled is held out of topology recovery until the client
17
+ has established whether it still exists. Without that the two race: a reconnect re-declares the
18
+ whole registry, so a queue deleted a moment earlier came back - and a queue the broker hands
19
+ back with a consumer attached cannot be deleted at all, which is the bug 0.8.0 set out to fix.
20
+ Measured against RabbitMQ 4.4, deleting a quorum queue reached exactly that state.
21
+ - A connection lost mid-re-subscribe is no longer reported as `consumer.recovery_failed`. It has
22
+ its own event and its own recovery, both already running, and an application alarming on
23
+ recovery failures was told twice about one outage, in the words of the more serious of the two.
24
+ - A `404` that says the queue's process is stopped, rather than that there is no such queue, is no
25
+ longer read as a deletion. A classic queue on a stopped node answers `NOT_FOUND - queue 'q' in
26
+ vhost '/' process is stopped by supervisor` and comes back with its node; the client forgot it,
27
+ so the consumer was never restored and the queue was never re-declared either. The probe now
28
+ answers `:deleted`, `:available`, `:unavailable` or `:unreachable`, and only `:available` is followed by an
29
+ attempt to consume - a `basic_consume` the broker answers with a 404 closes the channel and
30
+ takes the consumers still working on it down with it.
31
+ - `on_cancel` now runs to completion before the client decides what became of the queue. The two
32
+ ran alongside each other, and the documented way to keep a queue - re-declare it in `on_cancel` -
33
+ is an RPC that yields, so the probe sometimes got its answer in first and concluded the queue
34
+ was deleted. The callback is still off the dispatch fiber, so an RPC in it still gets its reply
35
+ and an exception in it still cannot stop the channel.
36
+ - A cancelled queue is released back to topology recovery only on a definite answer that it
37
+ still exists. A probe the broker could not answer is not evidence the queue is there, and
38
+ treating it as such re-declared a queue that had just been deleted - the one thing this
39
+ handling exists to prevent. The queue now stays held back until the broker says `:deleted`
40
+ or `:available`, retrying with the same backoff the recovering path uses.
41
+ - The retry backoff no longer grows for a failure that was the connection's rather than the
42
+ queue's, and resets once the connection is back. A two-second blip could otherwise leave a
43
+ healthy queue unconsumed for the next thirty seconds, since each unanswerable probe doubled the
44
+ delay. The queue probe now answers `:unreachable` - no answer at all - separately from
45
+ `:unavailable`, which is the broker saying the queue is not being served.
46
+
5
47
  ## [0.8.0] - 2026-11-14
6
48
 
7
49
  ### Fixed
data/README.md CHANGED
@@ -472,28 +472,40 @@ logged rather than taken out on the channel.
472
472
 
473
473
  ### A consumer the broker cancels
474
474
 
475
- RabbitMQ cancels a consumer when it can no longer serve the queue to it: the
476
- queue was deleted, the node holding it failed, a replicated queue's leader
477
- moved. The connection is untouched, so none of the recovery above runs — a
478
- `basic.cancel` frame is the only notice there is, and without `on_cancel` the
479
- worker goes on publishing and heartbeating while consuming nothing.
475
+ RabbitMQ cancels a consumer when it can no longer serve the queue to it. The
476
+ connection is untouched, so none of the recovery above runs — a `basic.cancel`
477
+ frame is the only notice there is, and without `on_cancel` the worker goes on
478
+ publishing and heartbeating while consuming nothing.
480
479
 
481
480
  Nothing is re-subscribed by default, which matches Bunny, the Java client and
482
481
  the .NET client. `recover_cancelled_consumers: true`, on the session or on one
483
482
  channel, re-subscribes under the same consumer tag, backing off from 1s to 30s
484
- for as long as the queue exists and the channel is open.
483
+ for as long as the queue exists, through however many reconnects that takes. It
484
+ gives up only when the channel or the session is finished for good.
485
485
 
486
486
  **Recovery never declares anything.** One of the reasons a consumer gets
487
487
  cancelled is somebody deleting the queue, and a client that answered that by
488
488
  re-declaring would hand back a queue nobody could get rid of while a consumer
489
- was attached. So the client passively checks whether the queue is still there
490
- and re-consumes only if it is. If it has gone, the consumer stays gone and the
491
- queue is dropped from the topology registry too — otherwise the next connection
492
- recovery would re-declare it and the deletion would never stick. Only
493
- `NOT_FOUND` counts as gone; a replicated queue mid-election is unavailable, not
494
- deleted. Re-declaring is the application's call, and `on_cancel` is where to
495
- make it: declare the queue there and the re-subscribe that follows will find
496
- it.
489
+ was attached. So the client asks the broker, on a channel of its own, whether
490
+ the queue is still there, and re-consumes only if it is. Re-declaring is the
491
+ application's call, and `on_cancel` is where to make it: that callback is run
492
+ to completion before the client goes asking, so a queue declared there is
493
+ there when it looks.
494
+
495
+ Three things follow from that, and each of them is a bug in some client:
496
+
497
+ - **The queue is held out of topology recovery until the client knows.**
498
+ Otherwise the two race — a reconnect re-declares the whole registry — and the
499
+ queue somebody deleted comes back. Deleting a quorum queue is exactly that
500
+ race arriving as one event: the cancel, and then a connection-level `541
501
+ INTERNAL_ERROR ... noproc` in answer to the re-subscribe.
502
+ - **The question is asked before every attempt, never the attempt instead of
503
+ the question.** A `basic_consume` the broker answers `404` closes the
504
+ channel, and takes the consumers still working on it down with it.
505
+ - **`NOT_FOUND` is two answers.** `no queue 'q' in vhost '/'` is a deletion;
506
+ `queue 'q' in vhost '/' process is stopped by supervisor`, which is what a
507
+ classic queue on a stopped node gives, is a queue that comes back with its
508
+ node. The second is waited for, not forgotten.
497
509
 
498
510
  ```ruby
499
511
  session = AsyncRabbitMQ::Session.new(..., recover_cancelled_consumers: true)
@@ -77,6 +77,15 @@ module AsyncRabbitMQ
77
77
  RECOVER_CONSUMER_DELAY = 1
78
78
  RECOVER_CONSUMER_MAX_DELAY = 30
79
79
 
80
+ # A queue that is still declared and a queue that is gone both answer
81
+ # NOT_FOUND to a passive declare, and only the reply text tells them apart.
82
+ # Measured on RabbitMQ 4.4: a deleted queue gives "no queue 'q' in vhost
83
+ # '/'", while a classic queue whose node has been stopped gives "queue 'q'
84
+ # in vhost '/' process is stopped by supervisor" and comes back with its
85
+ # node. Older brokers say "home node ... is down or inaccessible" for the
86
+ # same thing. See #queue_missing?.
87
+ QUEUE_UNAVAILABLE = /process is stopped|is down or inaccessible/
88
+
80
89
  # Pushed onto the work queue to retire one handler worker. Compared by
81
90
  # identity, so it can never collide with a delivery.
82
91
  WORKER_STOP = Object.new.freeze
@@ -1407,20 +1416,30 @@ module AsyncRabbitMQ
1407
1416
  #
1408
1417
  # Bunny hands cancel notifications to its consumer work pool and the Java
1409
1418
  # client to its work service, for the same reason.
1410
- def notify_app(name)
1411
- @session.spawn_background do
1412
- yield
1413
- rescue *PROPAGATE_FROM_HANDLER
1414
- raise
1415
- rescue Exception => e # rubocop:disable Lint/RescueException
1416
- @logger.error("Channel #{@channel_id}: #{name} callback raised #{e.class}: #{e.message}")
1417
- end
1419
+ def notify_app(name, &block)
1420
+ @session.spawn_background { run_callback(name, &block) }
1421
+ end
1422
+
1423
+ # The body of #notify_app, for a caller that is already off the dispatch
1424
+ # fiber and has something to do after the callback.
1425
+ def run_callback(name)
1426
+ yield
1427
+ rescue *PROPAGATE_FROM_HANDLER
1428
+ raise
1429
+ rescue Exception => e # rubocop:disable Lint/RescueException
1430
+ @logger.error("Channel #{@channel_id}: #{name} callback raised #{e.class}: #{e.message}")
1418
1431
  end
1419
1432
 
1420
1433
  # The broker cancels a consumer when it can no longer serve the queue to
1421
- # it: the queue was deleted, the node holding it failed, a replicated
1422
- # queue's leader moved. The connection is untouched, so connection
1423
- # recovery never runs - this frame is the only notice there is.
1434
+ # it: the queue was deleted, or the node holding a classic queue stopped.
1435
+ # On a three-node RabbitMQ 4.4.0 cluster, moving a quorum queue's
1436
+ # leadership did not do it - not a graceful stop of the leader's node, not
1437
+ # removing the member on the consuming node, not losing the majority, and
1438
+ # not cutting the leader's node off the network until the queue re-elected.
1439
+ # Other versions are reported to cancel on unavailability; it makes no
1440
+ # difference here, since the handling is the same whatever the cause. The
1441
+ # connection is untouched either way, so connection recovery never runs:
1442
+ # this frame is the only notice there is.
1424
1443
  #
1425
1444
  # The consumer goes from @consumers first, so no delivery is dispatched to
1426
1445
  # a block that is no longer subscribed, and everything beyond that
@@ -1430,20 +1449,33 @@ module AsyncRabbitMQ
1430
1449
  topology&.delete_consumer(tag)
1431
1450
  return wake_each_waiters(tag) unless entry
1432
1451
 
1452
+ # Held back from topology recovery from this moment, before anything can
1453
+ # reconnect, because one of the reasons a consumer is cancelled is the
1454
+ # queue being deleted and a reconnect re-declares the registry wholesale.
1455
+ # Settled by #reconcile_cancelled_consumer, either way.
1456
+ topology&.suspend_queue(entry[:queue_name])
1433
1457
  @logger.warn("Channel #{@channel_id}: broker cancelled consumer #{tag} on #{entry[:queue_name]}")
1434
1458
  instrument("consumer.cancelled") do
1435
1459
  { channel: @channel_id, consumer_tag: tag, queue: entry[:queue_name], reason: :broker }
1436
1460
  end
1437
- notify_app("on_cancel") { @on_cancel.call(tag) } if @on_cancel
1438
- @session.spawn_background { reconcile_cancelled_consumer(tag, entry) }
1461
+ # The callback first and the reconcile after it, in one task rather than
1462
+ # two. on_cancel is where an application re-declares a queue it wants
1463
+ # back, and a reconcile running alongside it sometimes asked the broker
1464
+ # before that declare had landed and concluded the queue was gone. Still
1465
+ # off the dispatch fiber, so an RPC in the callback can get its reply and
1466
+ # an exception in it cannot stop the channel.
1467
+ @session.spawn_background do
1468
+ run_callback("on_cancel") { @on_cancel.call(tag) } if @on_cancel
1469
+ reconcile_cancelled_consumer(tag, entry)
1470
+ end
1439
1471
  end
1440
1472
 
1441
- # Why the broker cancelled is not on the wire, and the two reasons want
1442
- # opposite handling: a queue that moved or was briefly unavailable should
1443
- # be consumed from again, while a queue somebody deleted must be left
1444
- # deleted. A passive declare is the only way to tell them apart. Spring
1445
- # AMQP's listener container probes the same way, and for the same reason
1446
- # declares nothing.
1473
+ # Why the broker cancelled is not on the wire, and the reasons want
1474
+ # opposite handling: a queue that is merely not being served should be
1475
+ # consumed from again, while a queue somebody deleted must be left deleted.
1476
+ # A passive declare is the only way to tell them apart. Spring AMQP's
1477
+ # listener container probes the same way, and for the same reason declares
1478
+ # nothing.
1447
1479
  #
1448
1480
  # Runs whether or not this channel recovers cancelled consumers, because
1449
1481
  # forgetting a deleted queue is not a recovery feature: a queue left in the
@@ -1451,10 +1483,72 @@ module AsyncRabbitMQ
1451
1483
  # then the deletion never sticks.
1452
1484
  def reconcile_cancelled_consumer(tag, entry)
1453
1485
  queue = entry[:queue_name]
1454
- return forget_deleted_queue(tag, queue) if queue_deleted?(queue)
1455
- return wake_each_waiters(tag) unless recovers_cancelled_consumers?
1486
+ return recover_cancelled_consumer(tag, entry) if recovers_cancelled_consumers?
1487
+
1488
+ # Not putting the consumer back, but a queue that has been deleted must
1489
+ # still leave the registry or the next reconnect declares it again.
1490
+ # Nothing is asked of a connection that is not there: a question put to a
1491
+ # session that is down cannot be answered, and guessing here is what puts
1492
+ # a deleted queue back.
1493
+ #
1494
+ # So the queue stays suspended until the broker gives a definite answer,
1495
+ # and only :available releases it. Releasing on a probe that could not be
1496
+ # answered is how a deleted queue comes back - it is not evidence the
1497
+ # queue is there, and 46_broker_amnesia_spec is the case that proves it.
1498
+ delay = RECOVER_CONSUMER_DELAY
1499
+
1500
+ loop do
1501
+ case wait_for_usable_channel
1502
+ when nil then return release_cancelled_consumer(tag, queue)
1503
+ when :reconnected then delay = RECOVER_CONSUMER_DELAY
1504
+ end
1505
+
1506
+ case queue_status(queue)
1507
+ when :deleted then return forget_deleted_queue(tag, queue)
1508
+ when :available then return release_cancelled_consumer(tag, queue)
1509
+ end
1510
+
1511
+ # :unavailable - the broker says the queue is not being served - and
1512
+ # :unreachable - no answer at all - both mean ask again, never "the
1513
+ # queue is there". Sleeping on either is also what keeps this from
1514
+ # spinning when the probe cannot be answered.
1515
+ sleep delay
1516
+ delay = [delay * 2, RECOVER_CONSUMER_MAX_DELAY].min
1517
+ end
1518
+ end
1519
+
1520
+ # Wait until this channel can carry a consumer again, through however many
1521
+ # reconnects that takes. :ready if it already could, :reconnected if it had
1522
+ # to wait for one, nil if it never will.
1523
+ #
1524
+ # Giving up on a connection loss instead would lose the consumer for good:
1525
+ # it is no longer in @consumers, so the reconnect does not re-register it,
1526
+ # and this loop is the only thing that would have brought it back. That is
1527
+ # precisely the case a quorum queue produces - the re-subscribe is answered
1528
+ # with a connection-level 541, not a channel error.
1529
+ #
1530
+ # Over for good when the channel is closed, which means the application
1531
+ # closed it or recovery could not reopen it and its consumers belong to
1532
+ # whoever does that, or when the session has stopped trying.
1533
+ def wait_for_usable_channel
1534
+ waited = false
1535
+ loop do
1536
+ return waited ? :reconnected : :ready if open? && @session.open?
1537
+ return nil if closed? || @session.closed?
1456
1538
 
1457
- recover_cancelled_consumer(tag, entry)
1539
+ waited = true
1540
+ sleep RECOVER_CONSUMER_DELAY
1541
+ end
1542
+ end
1543
+
1544
+ # Let the queue be recovered again and release anyone in #each. For a
1545
+ # consumer that will not be followed up - this channel does not recover
1546
+ # cancelled consumers, or there is nothing left to recover it onto - the
1547
+ # queue must not stay held back: it was never established that it is gone,
1548
+ # and a session opened again would not re-declare it.
1549
+ def release_cancelled_consumer(tag, queue)
1550
+ topology&.resume_queue(queue)
1551
+ wake_each_waiters(tag)
1458
1552
  end
1459
1553
 
1460
1554
  # Re-consume under the same tag, for as long as the queue is there and this
@@ -1466,43 +1560,87 @@ module AsyncRabbitMQ
1466
1560
  failures = 0
1467
1561
 
1468
1562
  loop do
1469
- # The channel is gone, or the connection is: either way the consumers
1470
- # on it are somebody else's problem now (#reopen, or recovery).
1471
- return wake_each_waiters(tag) unless open? && @session.open?
1563
+ # Every round starts with a connection that works and a fresh answer
1564
+ # about the queue. A queue deleted while we were retrying has to stop
1565
+ # the loop, and a basic_consume the broker answers with a 404 closes
1566
+ # this channel and takes the consumers still working on it down too -
1567
+ # so the question is asked on a channel of its own first, and only an
1568
+ # answer of "it is there" is followed by an attempt to consume.
1569
+ case wait_for_usable_channel
1570
+ when nil then return release_cancelled_consumer(tag, queue)
1571
+ when :reconnected
1572
+ # The connection was what was wrong, and it is back: start the
1573
+ # backoff again rather than sleeping out a delay earned by an outage
1574
+ # that is over. A two-second blip should not leave a healthy queue
1575
+ # unconsumed for the next thirty.
1576
+ delay = RECOVER_CONSUMER_DELAY
1577
+ end
1472
1578
 
1473
- begin
1474
- basic_consume(queue, consumer_tag: tag, manual_ack: entry[:manual_ack], exclusive: entry[:exclusive],
1475
- arguments: entry[:arguments] || {}, on_error: entry[:on_error], &entry[:block])
1476
- @logger.info("Channel #{@channel_id}: re-subscribed consumer #{tag} to #{queue}")
1477
- instrument("consumer.recovered") do
1478
- { channel: @channel_id, consumer_tag: tag, queue: queue, attempts: failures + 1 }
1479
- end
1480
- return
1481
- rescue *PROPAGATE_FROM_HANDLER
1482
- raise
1483
- rescue => e
1484
- failures += 1
1485
- # One line per episode at warn, the rest at debug: a broker that is
1486
- # away for an hour would otherwise write a warning every attempt for
1487
- # the whole hour, and an application counting warnings would read
1488
- # one outage as hundreds.
1489
- if failures == 1
1490
- @logger.warn("Channel #{@channel_id}: could not re-subscribe #{tag} to #{queue}, " \
1491
- "retrying: #{e.class}: #{e.message}")
1492
- instrument("consumer.recovery_failed") do
1493
- { channel: @channel_id, consumer_tag: tag, queue: queue, reason: :error, error: e }
1579
+ case queue_status(queue)
1580
+ when :deleted
1581
+ return forget_deleted_queue(tag, queue)
1582
+ when :unreachable
1583
+ # No answer rather than an answer: the connection is what is wrong,
1584
+ # not the queue, so the backoff a queue earns does not apply. Go back
1585
+ # and wait for the connection instead of sleeping out a delay that
1586
+ # doubles every time the broker cannot be reached.
1587
+ delay = RECOVER_CONSUMER_DELAY
1588
+ when :available
1589
+ begin
1590
+ basic_consume(queue, consumer_tag: tag, manual_ack: entry[:manual_ack], exclusive: entry[:exclusive],
1591
+ arguments: entry[:arguments] || {}, on_error: entry[:on_error], &entry[:block])
1592
+ # Consuming again, so the queue is there and recovery can have it
1593
+ # back (#handle_broker_cancel held it).
1594
+ topology&.resume_queue(queue)
1595
+ @logger.info("Channel #{@channel_id}: re-subscribed consumer #{tag} to #{queue}")
1596
+ instrument("consumer.recovered") do
1597
+ { channel: @channel_id, consumer_tag: tag, queue: queue, attempts: failures + 1 }
1494
1598
  end
1495
- else
1496
- @logger.debug("Channel #{@channel_id}: re-subscribe attempt #{failures} for #{tag} failed: #{e.message}")
1599
+ return
1600
+ rescue *PROPAGATE_FROM_HANDLER
1601
+ raise
1602
+ rescue ConnectionError, NotOpenError => e
1603
+ # The connection went, not the subscription. That has its own
1604
+ # reporting and its own recovery, both already running, so it is
1605
+ # not counted as a failure to re-subscribe: an application alarming
1606
+ # on consumer.recovery_failed would otherwise be told twice about
1607
+ # one event, and told it in the words of the more serious of the
1608
+ # two.
1609
+ @logger.debug("Channel #{@channel_id}: re-subscribe of #{tag} interrupted by " \
1610
+ "#{e.class}; waiting for the connection")
1611
+ rescue ChannelError => e
1612
+ # The queue went between the question and the answer. The broker
1613
+ # has closed this channel saying so, so there is nothing left to
1614
+ # retry on.
1615
+ return forget_deleted_queue(tag, queue) if queue_missing?(e)
1616
+
1617
+ failures += 1
1618
+ note_recovery_failure(tag, queue, e, failures)
1619
+ rescue => e
1620
+ failures += 1
1621
+ note_recovery_failure(tag, queue, e, failures)
1497
1622
  end
1498
1623
  end
1624
+ # :unavailable falls through to the backoff and is asked again.
1499
1625
 
1500
1626
  sleep delay
1501
1627
  delay = [delay * 2, RECOVER_CONSUMER_MAX_DELAY].min
1502
- # Re-checked every round, not just once: a queue deleted while we were
1503
- # retrying has to stop the loop, and it is also what keeps an endless
1504
- # retry endless only while there is something to consume from.
1505
- return forget_deleted_queue(tag, queue) if queue_deleted?(queue)
1628
+ end
1629
+ end
1630
+
1631
+ # One line per episode at warn, the rest at debug: a broker that is away
1632
+ # for an hour would otherwise write a warning every attempt for the whole
1633
+ # hour, and an application counting warnings would read one outage as
1634
+ # hundreds.
1635
+ def note_recovery_failure(tag, queue, error, failures)
1636
+ if failures == 1
1637
+ @logger.warn("Channel #{@channel_id}: could not re-subscribe #{tag} to #{queue}, " \
1638
+ "retrying: #{error.class}: #{error.message}")
1639
+ instrument("consumer.recovery_failed") do
1640
+ { channel: @channel_id, consumer_tag: tag, queue: queue, reason: :error, error: error }
1641
+ end
1642
+ else
1643
+ @logger.debug("Channel #{@channel_id}: re-subscribe attempt #{failures} for #{tag} failed: #{error.message}")
1506
1644
  end
1507
1645
  end
1508
1646
 
@@ -1519,26 +1657,38 @@ module AsyncRabbitMQ
1519
1657
  wake_each_waiters(tag)
1520
1658
  end
1521
1659
 
1522
- # Deleted, as opposed to merely unreachable. A passive declare, on a channel
1523
- # of its own because a passive declare that fails closes the channel it ran
1524
- # on - which would take the queue's fellow consumers down with it.
1660
+ # Is the queue gone, there, or merely not being served right now? A passive
1661
+ # declare answers it, on a channel of its own because a passive declare
1662
+ # that fails closes the channel it ran on - which would take the queue's
1663
+ # fellow consumers down with it.
1525
1664
  #
1526
- # Only NOT_FOUND counts. A replicated queue mid-election is the very case
1527
- # this whole path exists for and it does not answer 404, so treating every
1528
- # channel error as a deletion would forget a queue that is coming back and
1529
- # stop topology recovery ever re-declaring it. Anything inconclusive - a
1530
- # connection that went while we were asking, too - reads as "still there",
1531
- # which costs a retry and never costs a queue.
1532
- def queue_deleted?(name)
1665
+ # Four answers, because conflating any two of them does harm. :deleted must
1666
+ # be forgotten and :unavailable must be waited for, and telling those apart
1667
+ # is the whole point. :unreachable - no answer at all, because the
1668
+ # connection is in trouble - is separated from :unavailable so that the
1669
+ # caller does not spend a backoff earned by a queue on a connection that is
1670
+ # simply down, and is never taken for :available, because a basic_consume
1671
+ # on a queue that is not there closes the channel.
1672
+ def queue_status(name)
1533
1673
  @session.with_channel { |probe| probe.queue(name, passive: true) }
1534
- false
1674
+ :available
1535
1675
  rescue ChannelError => e
1536
- e.code == 404
1676
+ queue_missing?(e) ? :deleted : :unavailable
1537
1677
  rescue *PROPAGATE_FROM_HANDLER
1538
1678
  raise
1539
1679
  rescue => e
1540
1680
  @logger.debug("Channel #{@channel_id}: could not check whether #{name} still exists: #{e.class}: #{e.message}")
1541
- false
1681
+ :unreachable
1682
+ end
1683
+
1684
+ # Gone, rather than merely not being served. Taking a queue whose node is
1685
+ # down for a deleted one forgets a queue that is coming back: its consumer
1686
+ # is never restored, and it is never re-declared either. Any 404 the
1687
+ # QUEUE_UNAVAILABLE wordings do not cover reads as a deletion, which is the
1688
+ # safe way round - an unmatched wording costs only what forgetting it costs
1689
+ # today.
1690
+ def queue_missing?(error)
1691
+ error.code == 404 && !QUEUE_UNAVAILABLE.match?(error.text.to_s)
1542
1692
  end
1543
1693
 
1544
1694
  # Release fibers blocked in #each for one consumer, or for all of them.
@@ -38,6 +38,7 @@ module AsyncRabbitMQ
38
38
  @queue_bindings = []
39
39
  @exchange_bindings = []
40
40
  @consumers = {} # consumer tag => queue name
41
+ @suspended = Set.new # queue names held back from recovery
41
42
  end
42
43
 
43
44
  # --- recording -----------------------------------------------------------
@@ -59,6 +60,9 @@ module AsyncRabbitMQ
59
60
 
60
61
  def record_queue(channel_id, name, durable:, exclusive:, auto_delete:, arguments:, server_named:, object: nil)
61
62
  existing = @queues[name]
63
+ # Declared again by the application, so whatever doubt a cancelled
64
+ # consumer raised about it is settled.
65
+ @suspended.delete(name)
62
66
  rec = RecordedQueue.new(channel_id: channel_id, name: name, durable: durable, exclusive: exclusive,
63
67
  auto_delete: auto_delete, arguments: arguments.dup, server_named: server_named,
64
68
  objects: existing ? existing.objects : [])
@@ -68,15 +72,45 @@ module AsyncRabbitMQ
68
72
 
69
73
  def delete_queue(name)
70
74
  @queues.delete(name)
75
+ @suspended.delete(name)
71
76
  gone, @queue_bindings = @queue_bindings.partition { |b| b.queue == name }
72
77
  gone.each { |b| prune_auto_delete_exchange(b.exchange) }
73
78
  end
74
79
 
80
+ # --- suspension ----------------------------------------------------------
81
+ #
82
+ # A queue is suspended while the client is working out whether it is still
83
+ # there, which is what a consumer the broker cancelled leaves it not
84
+ # knowing. Recovery skips a suspended queue and its bindings.
85
+ #
86
+ # The alternative - keep recovering it until the client is sure - puts a
87
+ # deleted queue back, because the deletion and the reconnect race: the node
88
+ # that was serving the queue is often the one that just went, and a
89
+ # reconnect re-declares the whole registry before anything can ask the new
90
+ # connection whether the queue survived. A queue the broker then hands back
91
+ # with a consumer attached cannot be deleted at all. Holding it back costs
92
+ # nothing by comparison: the worst case is a queue that was fine and is not
93
+ # re-declared for the few seconds it takes to establish that.
94
+
95
+ def suspend_queue(name)
96
+ @suspended << name if @queues.key?(name)
97
+ end
98
+
99
+ def resume_queue(name)
100
+ @suspended.delete(name)
101
+ end
102
+
103
+ # Queue names currently held back, for inspection.
104
+ def suspended_queues
105
+ @suspended.dup
106
+ end
107
+
75
108
  # A server-named queue was re-declared under a different name.
76
109
  def rename_queue(old_name, new_name)
77
110
  rec = @queues.delete(old_name) or return
78
111
  rec.name = new_name
79
112
  @queues[new_name] = rec
113
+ @suspended << new_name if @suspended.delete?(old_name)
80
114
  @queue_bindings.each { |b| b.queue = new_name if b.queue == old_name }
81
115
  @consumers.transform_values! { |queue| queue == old_name ? new_name : queue }
82
116
  rec.objects.each { |q| q.update_name_to(new_name) }
@@ -135,11 +169,11 @@ module AsyncRabbitMQ
135
169
  end
136
170
 
137
171
  def queues
138
- apply(:filter_queues, @queues.values)
172
+ apply(:filter_queues, @queues.each_value.reject { |q| @suspended.include?(q.name) })
139
173
  end
140
174
 
141
175
  def queue_bindings
142
- apply(:filter_queue_bindings, @queue_bindings.dup)
176
+ apply(:filter_queue_bindings, @queue_bindings.reject { |b| @suspended.include?(b.queue) })
143
177
  end
144
178
 
145
179
  def exchange_bindings
@@ -159,6 +193,7 @@ module AsyncRabbitMQ
159
193
  @queue_bindings.clear
160
194
  @exchange_bindings.clear
161
195
  @consumers.clear
196
+ @suspended.clear
162
197
  end
163
198
 
164
199
  # Forget only what was tied to the connection that just went: exclusive
@@ -174,6 +209,7 @@ module AsyncRabbitMQ
174
209
 
175
210
  dropped_queues.each { |name| @queues.delete(name) }
176
211
  dropped_exchanges.each { |name| @exchanges.delete(name) }
212
+ @suspended.subtract(dropped_queues)
177
213
 
178
214
  @queue_bindings.reject! do |b|
179
215
  dropped_queues.include?(b.queue) || dropped_exchanges.include?(b.exchange)
@@ -1,3 +1,3 @@
1
1
  module AsyncRabbitMQ
2
- VERSION = "0.8.0"
2
+ VERSION = "0.8.1"
3
3
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: async-rabbitmq
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.8.0
4
+ version: 0.8.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Russell Penney