async-rabbitmq 0.7.1 → 0.8.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 12f881a8b2274539cd1e93e8ea827140887f0a392952dcd123270acaf5d4aea3
4
- data.tar.gz: 94ad7522a6b7c33d5cde8b892c0629b5db4961b9eb289c4efc6ab6c3f7286c66
3
+ metadata.gz: 91a60c68246f2c413fadd2c183b925062b38a4e36ac1e8b1e4a21eff2ca49c97
4
+ data.tar.gz: 2d3a70204f101c856539b921255d541d8566af490ea468f57753ee22f1e69708
5
5
  SHA512:
6
- metadata.gz: ed9504ed356545629ff40c48ee8e3d7dadd6482538fff0bd3539c2c48dcb020c3b5949538dd1382a4b9b4f5bd650986eda6782efe7b9ce10545e047474788cae
7
- data.tar.gz: cca32432f4ca23fc9b32150ce48684d9fde94feb8a0f9b9927e7186d4de59e6ee32677d3a437549559013e9c94a8350e5ea40eeb1c4f748ef9418470af2c593e
6
+ metadata.gz: 869b317e73b1f566e37a04cc7e22e5534f26cfbc3168d11af3b2477ffda77d8f3759270e249b2bab480b94c0c0cf7d8f3d1610fd3bd425b493bfbd9844c69a9f
7
+ data.tar.gz: e5f6720a22811c30c2fbb600faf55c266a6fe298694d203a5e7499bd3218dc74f6471687624cd5ef564f410043009b8bf352102eaf3c0acc0ab59d2dd96049a2
data/CHANGELOG.md CHANGED
@@ -2,6 +2,81 @@
2
2
 
3
3
  All notable changes to this project will be documented in this file.
4
4
 
5
+ ## [0.8.1] - 2026-11-15
6
+
7
+ ### Fixed
8
+
9
+ - A consumer the broker cancelled is no longer given up on when the connection goes at the same
10
+ moment. Deleting a quorum queue is both events at once: the consumer is cancelled, and the
11
+ re-subscribe is answered with a connection-level `541 INTERNAL_ERROR ... noproc` that takes the
12
+ connection down. 0.8.0's recovery loop stopped at the first sight of a session that was not
13
+ open, and nothing else would have brought that consumer back - it is no longer in the channel's
14
+ registry, so the reconnect does not re-register it either. It now waits through however many
15
+ reconnects it takes, and gives up only when the channel or the session is finished for good.
16
+ - A queue whose consumer the broker cancelled is held out of topology recovery until the client
17
+ has established whether it still exists. Without that the two race: a reconnect re-declares the
18
+ whole registry, so a queue deleted a moment earlier came back - and a queue the broker hands
19
+ back with a consumer attached cannot be deleted at all, which is the bug 0.8.0 set out to fix.
20
+ Measured against RabbitMQ 4.4, deleting a quorum queue reached exactly that state.
21
+ - A connection lost mid-re-subscribe is no longer reported as `consumer.recovery_failed`. It has
22
+ its own event and its own recovery, both already running, and an application alarming on
23
+ recovery failures was told twice about one outage, in the words of the more serious of the two.
24
+ - A `404` that says the queue's process is stopped, rather than that there is no such queue, is no
25
+ longer read as a deletion. A classic queue on a stopped node answers `NOT_FOUND - queue 'q' in
26
+ vhost '/' process is stopped by supervisor` and comes back with its node; the client forgot it,
27
+ so the consumer was never restored and the queue was never re-declared either. The probe now
28
+ answers `:deleted`, `:available`, `:unavailable` or `:unreachable`, and only `:available` is followed by an
29
+ attempt to consume - a `basic_consume` the broker answers with a 404 closes the channel and
30
+ takes the consumers still working on it down with it.
31
+ - `on_cancel` now runs to completion before the client decides what became of the queue. The two
32
+ ran alongside each other, and the documented way to keep a queue - re-declare it in `on_cancel` -
33
+ is an RPC that yields, so the probe sometimes got its answer in first and concluded the queue
34
+ was deleted. The callback is still off the dispatch fiber, so an RPC in it still gets its reply
35
+ and an exception in it still cannot stop the channel.
36
+ - A cancelled queue is released back to topology recovery only on a definite answer that it
37
+ still exists. A probe the broker could not answer is not evidence the queue is there, and
38
+ treating it as such re-declared a queue that had just been deleted - the one thing this
39
+ handling exists to prevent. The queue now stays held back until the broker says `:deleted`
40
+ or `:available`, retrying with the same backoff the recovering path uses.
41
+ - The retry backoff no longer grows for a failure that was the connection's rather than the
42
+ queue's, and resets once the connection is back. A two-second blip could otherwise leave a
43
+ healthy queue unconsumed for the next thirty seconds, since each unanswerable probe doubled the
44
+ delay. The queue probe now answers `:unreachable` - no answer at all - separately from
45
+ `:unavailable`, which is the broker saying the queue is not being served.
46
+
47
+ ## [0.8.0] - 2026-11-14
48
+
49
+ ### Fixed
50
+
51
+ - A server-sent `basic.cancel` no longer leaves the queue in the topology registry. The broker
52
+ cancels a consumer when it can no longer serve the queue, and one of the ways that happens is
53
+ somebody deleting the queue - after which the next connection recovery re-declared it and the
54
+ deletion never stuck. The client now checks, passively, whether the queue is really gone and
55
+ forgets it, with its bindings, if it is. Only `NOT_FOUND` counts as gone: a replicated queue
56
+ mid-election is unavailable, not deleted, and is left in the registry to be re-declared.
57
+ - `on_cancel` and `on_error` no longer run on the channel's dispatch fiber. The obvious thing to
58
+ write in either is an RPC - re-consume, reopen - and an RPC on that fiber waits for a reply only
59
+ that fiber can deliver: fifteen seconds to `rpc_timeout`, then a channel resync. An exception out
60
+ of either escaped `dispatch_loop`, whose rescue sits outside its loop, so the channel stopped
61
+ delivering anything at all, confirms and RPC replies included. Both now run in a task of their
62
+ own and an exception is logged. Bunny dispatches cancel notifications to its consumer work pool
63
+ and the Java client to its work service, for the same reasons. `on_error` also runs after the
64
+ close has released the fibers it stranded, not before.
65
+ - The log line for a broker-sent cancel names the queue, not just the consumer tag.
66
+
67
+ ### Added
68
+
69
+ - `recover_cancelled_consumers:` on `Session.new` and `Session#open_channel`, with
70
+ `Channel#recover_cancelled_consumers=` and `Channel#recovers_cancelled_consumers?`. Off by
71
+ default, as it is in Bunny. When on, a consumer the broker cancels is re-subscribed under the
72
+ same tag, backing off from 1s to 30s for as long as the queue exists and the channel is open.
73
+ Recovery declares nothing: it passively checks that the queue is still there and re-consumes, so
74
+ a queue somebody deleted is never handed back. Re-declaring is the application's call.
75
+ - `consumer.recovered` and `consumer.recovery_failed` events. `consumer.recovery_failed` carries
76
+ `reason: :queue_deleted` - emitted whether or not recovery is enabled, since that is the point
77
+ at which a consumer is known not to be coming back - or `reason: :error` with the exception, once
78
+ per episode rather than once per attempt.
79
+
5
80
  ## [0.7.1] - 2026-10-18
6
81
 
7
82
  ### Added
data/README.md CHANGED
@@ -1,7 +1,7 @@
1
1
  # async-rabbitmq
2
2
 
3
3
  A fiber-native RabbitMQ (AMQP 0-9-1) client for Ruby, built on the amazing work
4
- of [Bunny](https://github.com/ruby-amqp/bunny).
4
+ of [Bunny](https://github.com/ruby-amqp/bunny) and [Async](https://github.com/socketry/async).
5
5
 
6
6
  ## Installing
7
7
 
@@ -466,7 +466,51 @@ starts), `on_blocked` / `on_unblocked` (connection.blocked),
466
466
  `Channel#on_return` (mandatory messages the broker could not route),
467
467
  `on_cancel` (broker cancelled a consumer), `on_error` (broker closed the
468
468
  channel). A broker-closed channel can be reopened in place with
469
- `Channel#reopen`.
469
+ `Channel#reopen`. Every one of these runs in a task of its own, so a callback
470
+ may issue RPCs — re-consume, reopen, declare — and a callback that raises is
471
+ logged rather than taken out on the channel.
472
+
473
+ ### A consumer the broker cancels
474
+
475
+ RabbitMQ cancels a consumer when it can no longer serve the queue to it. The
476
+ connection is untouched, so none of the recovery above runs — a `basic.cancel`
477
+ frame is the only notice there is, and without `on_cancel` the worker goes on
478
+ publishing and heartbeating while consuming nothing.
479
+
480
+ Nothing is re-subscribed by default, which matches Bunny, the Java client and
481
+ the .NET client. `recover_cancelled_consumers: true`, on the session or on one
482
+ channel, re-subscribes under the same consumer tag, backing off from 1s to 30s
483
+ for as long as the queue exists, through however many reconnects that takes. It
484
+ gives up only when the channel or the session is finished for good.
485
+
486
+ **Recovery never declares anything.** One of the reasons a consumer gets
487
+ cancelled is somebody deleting the queue, and a client that answered that by
488
+ re-declaring would hand back a queue nobody could get rid of while a consumer
489
+ was attached. So the client asks the broker, on a channel of its own, whether
490
+ the queue is still there, and re-consumes only if it is. Re-declaring is the
491
+ application's call, and `on_cancel` is where to make it: that callback is run
492
+ to completion before the client goes asking, so a queue declared there is
493
+ there when it looks.
494
+
495
+ Three things follow from that, and each of them is a bug in some client:
496
+
497
+ - **The queue is held out of topology recovery until the client knows.**
498
+ Otherwise the two race — a reconnect re-declares the whole registry — and the
499
+ queue somebody deleted comes back. Deleting a quorum queue is exactly that
500
+ race arriving as one event: the cancel, and then a connection-level `541
501
+ INTERNAL_ERROR ... noproc` in answer to the re-subscribe.
502
+ - **The question is asked before every attempt, never the attempt instead of
503
+ the question.** A `basic_consume` the broker answers `404` closes the
504
+ channel, and takes the consumers still working on it down with it.
505
+ - **`NOT_FOUND` is two answers.** `no queue 'q' in vhost '/'` is a deletion;
506
+ `queue 'q' in vhost '/' process is stopped by supervisor`, which is what a
507
+ classic queue on a stopped node gives, is a queue that comes back with its
508
+ node. The second is waited for, not forgotten.
509
+
510
+ ```ruby
511
+ session = AsyncRabbitMQ::Session.new(..., recover_cancelled_consumers: true)
512
+ channel.on_cancel { |tag| logger.warn("consumer #{tag} cancelled by the broker") }
513
+ ```
470
514
 
471
515
  ## Errors
472
516
 
@@ -520,6 +564,8 @@ logged and skipped; it never breaks the connection.
520
564
  | `channel.rpc` | `channel`, `method` (`"queue.declare-ok"`), `duration` |
521
565
  | `consumer.registered` | `channel`, `queue`, `consumer_tag`, `manual_ack` |
522
566
  | `consumer.cancelled` | `channel`, `consumer_tag`, `queue`, `reason` (`:client`, `:broker`) |
567
+ | `consumer.recovered` | `channel`, `consumer_tag`, `queue`, `attempts` |
568
+ | `consumer.recovery_failed` | `channel`, `consumer_tag`, `queue`, `reason` (`:queue_deleted`, `:error`), `error` |
523
569
  | `message.published` | `channel`, `exchange`, `routing_key`, `count`, `bytes`, `delivery_tag` |
524
570
  | `message.confirmed` | `channel`, `delivery_tag`, `multiple`, `acked` |
525
571
  | `message.returned` | `channel`, `exchange`, `routing_key`, `code`, `text`, `bytes` |
@@ -71,6 +71,21 @@ module AsyncRabbitMQ
71
71
  # after an RPC timeout, used when the session has no rpc_timeout set.
72
72
  RESYNC_TIMEOUT = 5
73
73
 
74
+ # First wait before re-subscribing a consumer the broker cancelled, and the
75
+ # ceiling the doubling stops at. Only used when the channel was asked to
76
+ # recover cancelled consumers; see #recovers_cancelled_consumers?.
77
+ RECOVER_CONSUMER_DELAY = 1
78
+ RECOVER_CONSUMER_MAX_DELAY = 30
79
+
80
+ # A queue that is still declared and a queue that is gone both answer
81
+ # NOT_FOUND to a passive declare, and only the reply text tells them apart.
82
+ # Measured on RabbitMQ 4.4: a deleted queue gives "no queue 'q' in vhost
83
+ # '/'", while a classic queue whose node has been stopped gives "queue 'q'
84
+ # in vhost '/' process is stopped by supervisor" and comes back with its
85
+ # node. Older brokers say "home node ... is down or inaccessible" for the
86
+ # same thing. See #queue_missing?.
87
+ QUEUE_UNAVAILABLE = /process is stopped|is down or inaccessible/
88
+
74
89
  # Pushed onto the work queue to retire one handler worker. Compared by
75
90
  # identity, so it can never collide with a delivery.
76
91
  WORKER_STOP = Object.new.freeze
@@ -94,7 +109,8 @@ module AsyncRabbitMQ
94
109
  # concurrently on this channel (Bunny-parity: default 1 = serialized).
95
110
  # Resizable at runtime via #pool_size=; basic_qos adjusts it automatically
96
111
  # when prefetch_count > 0 so the two stay coupled.
97
- def initialize(channel_id, session, frame_io, frame_max:, logger:, pool_size: 1, rpc_timeout: nil)
112
+ def initialize(channel_id, session, frame_io, frame_max:, logger:, pool_size: 1, rpc_timeout: nil,
113
+ recover_cancelled_consumers: false)
98
114
  @channel_id = channel_id
99
115
  @session = session
100
116
  @notifier = session.respond_to?(:notifier) ? session.notifier : nil
@@ -124,6 +140,7 @@ module AsyncRabbitMQ
124
140
  @on_cancel = nil
125
141
  @on_error = nil
126
142
  @on_handler_error = nil
143
+ @recover_cancelled_consumers = recover_cancelled_consumers
127
144
  @on_error_disposition = :dead_letter
128
145
  @resyncing = false
129
146
  @resync_condition = nil
@@ -647,10 +664,33 @@ module AsyncRabbitMQ
647
664
  # Register a callback invoked when the broker cancels a consumer server-side
648
665
  # (e.g. the queue is deleted or an HA failover occurs).
649
666
  # The block receives the consumer_tag that was cancelled.
667
+ #
668
+ # The callback runs in a task of its own, so it may issue RPCs - a
669
+ # re-consume, a declare - and an exception out of it is logged rather than
670
+ # taken out on the channel.
650
671
  def on_cancel(&block)
651
672
  @on_cancel = block
652
673
  end
653
674
 
675
+ # Re-subscribe a consumer the broker cancels, under the same consumer tag.
676
+ #
677
+ # Off by default, as it is in Bunny, and for a reason worth stating: the
678
+ # broker cancels a consumer when it can no longer serve the queue, and one
679
+ # of the ways that happens is somebody deleting the queue. A client that
680
+ # recovers by re-declaring would hand back a queue nobody can get rid of
681
+ # while a consumer is attached. So recovery here never declares anything.
682
+ # It passively checks that the queue is still there and re-consumes if it
683
+ # is; if the queue has gone it stops, and forgets the queue so that a later
684
+ # connection recovery does not re-declare it either.
685
+ #
686
+ # Retries back off from RECOVER_CONSUMER_DELAY to RECOVER_CONSUMER_MAX_DELAY
687
+ # and continue for as long as the queue exists and the channel is open.
688
+ attr_writer :recover_cancelled_consumers
689
+
690
+ def recovers_cancelled_consumers?
691
+ @recover_cancelled_consumers
692
+ end
693
+
654
694
  # Register a callback invoked when the broker closes this channel due to an
655
695
  # error (e.g. 404 queue not found). The block receives the channel and the
656
696
  # AMQ::Protocol::Channel::Close method frame.
@@ -1273,18 +1313,7 @@ module AsyncRabbitMQ
1273
1313
  handle_channel_close(method)
1274
1314
 
1275
1315
  when AMQ::Protocol::Basic::Cancel
1276
- # Server-initiated consumer cancel (e.g. queue deleted, HA failover).
1277
- # Remove from @consumers so deliveries are no longer dispatched to a dead block.
1278
- entry = @consumers.delete(method.consumer_tag)
1279
- topology&.delete_consumer(method.consumer_tag)
1280
- if entry
1281
- @logger.warn("Channel #{@channel_id}: broker cancelled consumer #{method.consumer_tag}")
1282
- instrument("consumer.cancelled") do
1283
- { channel: @channel_id, consumer_tag: method.consumer_tag, queue: entry[:queue_name], reason: :broker }
1284
- end
1285
- @on_cancel&.call(method.consumer_tag)
1286
- end
1287
- wake_each_waiters(method.consumer_tag)
1316
+ handle_broker_cancel(method.consumer_tag)
1288
1317
  # No CancelOk to send for server-initiated cancel (no-wait is implicit).
1289
1318
 
1290
1319
  when AMQ::Protocol::Channel::Flow
@@ -1354,7 +1383,6 @@ module AsyncRabbitMQ
1354
1383
  forget_consumers_in_topology
1355
1384
  @session.channel_closed(@channel_id)
1356
1385
  instrument("channel.closed") { { channel: @channel_id, reason: :broker, code: code, text: text } }
1357
- @on_error&.call(self, method)
1358
1386
 
1359
1387
  error = if FrameIO::SOFT_ERROR_CODES.include?(code)
1360
1388
  ChannelError.new(code: code, text: text, channel_id: @channel_id, close_method: method)
@@ -1369,6 +1397,298 @@ module AsyncRabbitMQ
1369
1397
  @queue&.push(nil)
1370
1398
  stop_workers(discard: true)
1371
1399
  wake_each_waiters
1400
+ # Last, and in a task of its own. The lines above exist to release fibers
1401
+ # this close has stranded; a hook that is slow, or that raises, must not
1402
+ # come between them and the fibers waiting on them. The natural thing to
1403
+ # write in this hook is a reopen, which is an RPC, and an RPC on the
1404
+ # dispatch fiber waits for a reply only the dispatch fiber can deliver.
1405
+ notify_app("on_error") { @on_error.call(self, method) } if @on_error
1406
+ end
1407
+
1408
+ # An application callback never runs on the dispatch fiber. Two reasons,
1409
+ # both load-bearing. An RPC issued from a callback waits for a reply that
1410
+ # only this fiber delivers, so it deadlocks until rpc_timeout and then
1411
+ # drags the channel through a resync - and a re-consume is the obvious
1412
+ # thing to write in on_cancel. And an exception out of a callback escapes
1413
+ # #handle_message, whose rescue sits outside dispatch_loop's `loop`, so the
1414
+ # loop ends and the channel stops delivering anything at all: no
1415
+ # deliveries, no confirms, no RPC replies, with one line to say so.
1416
+ #
1417
+ # Bunny hands cancel notifications to its consumer work pool and the Java
1418
+ # client to its work service, for the same reason.
1419
+ def notify_app(name, &block)
1420
+ @session.spawn_background { run_callback(name, &block) }
1421
+ end
1422
+
1423
+ # The body of #notify_app, for a caller that is already off the dispatch
1424
+ # fiber and has something to do after the callback.
1425
+ def run_callback(name)
1426
+ yield
1427
+ rescue *PROPAGATE_FROM_HANDLER
1428
+ raise
1429
+ rescue Exception => e # rubocop:disable Lint/RescueException
1430
+ @logger.error("Channel #{@channel_id}: #{name} callback raised #{e.class}: #{e.message}")
1431
+ end
1432
+
1433
+ # The broker cancels a consumer when it can no longer serve the queue to
1434
+ # it: the queue was deleted, or the node holding a classic queue stopped.
1435
+ # On a three-node RabbitMQ 4.4.0 cluster, moving a quorum queue's
1436
+ # leadership did not do it - not a graceful stop of the leader's node, not
1437
+ # removing the member on the consuming node, not losing the majority, and
1438
+ # not cutting the leader's node off the network until the queue re-elected.
1439
+ # Other versions are reported to cancel on unavailability; it makes no
1440
+ # difference here, since the handling is the same whatever the cause. The
1441
+ # connection is untouched either way, so connection recovery never runs:
1442
+ # this frame is the only notice there is.
1443
+ #
1444
+ # The consumer goes from @consumers first, so no delivery is dispatched to
1445
+ # a block that is no longer subscribed, and everything beyond that
1446
+ # bookkeeping happens off this fiber.
1447
+ def handle_broker_cancel(tag)
1448
+ entry = @consumers.delete(tag)
1449
+ topology&.delete_consumer(tag)
1450
+ return wake_each_waiters(tag) unless entry
1451
+
1452
+ # Held back from topology recovery from this moment, before anything can
1453
+ # reconnect, because one of the reasons a consumer is cancelled is the
1454
+ # queue being deleted and a reconnect re-declares the registry wholesale.
1455
+ # Settled by #reconcile_cancelled_consumer, either way.
1456
+ topology&.suspend_queue(entry[:queue_name])
1457
+ @logger.warn("Channel #{@channel_id}: broker cancelled consumer #{tag} on #{entry[:queue_name]}")
1458
+ instrument("consumer.cancelled") do
1459
+ { channel: @channel_id, consumer_tag: tag, queue: entry[:queue_name], reason: :broker }
1460
+ end
1461
+ # The callback first and the reconcile after it, in one task rather than
1462
+ # two. on_cancel is where an application re-declares a queue it wants
1463
+ # back, and a reconcile running alongside it sometimes asked the broker
1464
+ # before that declare had landed and concluded the queue was gone. Still
1465
+ # off the dispatch fiber, so an RPC in the callback can get its reply and
1466
+ # an exception in it cannot stop the channel.
1467
+ @session.spawn_background do
1468
+ run_callback("on_cancel") { @on_cancel.call(tag) } if @on_cancel
1469
+ reconcile_cancelled_consumer(tag, entry)
1470
+ end
1471
+ end
1472
+
1473
+ # Why the broker cancelled is not on the wire, and the reasons want
1474
+ # opposite handling: a queue that is merely not being served should be
1475
+ # consumed from again, while a queue somebody deleted must be left deleted.
1476
+ # A passive declare is the only way to tell them apart. Spring AMQP's
1477
+ # listener container probes the same way, and for the same reason declares
1478
+ # nothing.
1479
+ #
1480
+ # Runs whether or not this channel recovers cancelled consumers, because
1481
+ # forgetting a deleted queue is not a recovery feature: a queue left in the
1482
+ # topology registry is re-declared on the next connection recovery, and
1483
+ # then the deletion never sticks.
1484
+ def reconcile_cancelled_consumer(tag, entry)
1485
+ queue = entry[:queue_name]
1486
+ return recover_cancelled_consumer(tag, entry) if recovers_cancelled_consumers?
1487
+
1488
+ # Not putting the consumer back, but a queue that has been deleted must
1489
+ # still leave the registry or the next reconnect declares it again.
1490
+ # Nothing is asked of a connection that is not there: a question put to a
1491
+ # session that is down cannot be answered, and guessing here is what puts
1492
+ # a deleted queue back.
1493
+ #
1494
+ # So the queue stays suspended until the broker gives a definite answer,
1495
+ # and only :available releases it. Releasing on a probe that could not be
1496
+ # answered is how a deleted queue comes back - it is not evidence the
1497
+ # queue is there, and 46_broker_amnesia_spec is the case that proves it.
1498
+ delay = RECOVER_CONSUMER_DELAY
1499
+
1500
+ loop do
1501
+ case wait_for_usable_channel
1502
+ when nil then return release_cancelled_consumer(tag, queue)
1503
+ when :reconnected then delay = RECOVER_CONSUMER_DELAY
1504
+ end
1505
+
1506
+ case queue_status(queue)
1507
+ when :deleted then return forget_deleted_queue(tag, queue)
1508
+ when :available then return release_cancelled_consumer(tag, queue)
1509
+ end
1510
+
1511
+ # :unavailable - the broker says the queue is not being served - and
1512
+ # :unreachable - no answer at all - both mean ask again, never "the
1513
+ # queue is there". Sleeping on either is also what keeps this from
1514
+ # spinning when the probe cannot be answered.
1515
+ sleep delay
1516
+ delay = [delay * 2, RECOVER_CONSUMER_MAX_DELAY].min
1517
+ end
1518
+ end
1519
+
1520
+ # Wait until this channel can carry a consumer again, through however many
1521
+ # reconnects that takes. :ready if it already could, :reconnected if it had
1522
+ # to wait for one, nil if it never will.
1523
+ #
1524
+ # Giving up on a connection loss instead would lose the consumer for good:
1525
+ # it is no longer in @consumers, so the reconnect does not re-register it,
1526
+ # and this loop is the only thing that would have brought it back. That is
1527
+ # precisely the case a quorum queue produces - the re-subscribe is answered
1528
+ # with a connection-level 541, not a channel error.
1529
+ #
1530
+ # Over for good when the channel is closed, which means the application
1531
+ # closed it or recovery could not reopen it and its consumers belong to
1532
+ # whoever does that, or when the session has stopped trying.
1533
+ def wait_for_usable_channel
1534
+ waited = false
1535
+ loop do
1536
+ return waited ? :reconnected : :ready if open? && @session.open?
1537
+ return nil if closed? || @session.closed?
1538
+
1539
+ waited = true
1540
+ sleep RECOVER_CONSUMER_DELAY
1541
+ end
1542
+ end
1543
+
1544
+ # Let the queue be recovered again and release anyone in #each. For a
1545
+ # consumer that will not be followed up - this channel does not recover
1546
+ # cancelled consumers, or there is nothing left to recover it onto - the
1547
+ # queue must not stay held back: it was never established that it is gone,
1548
+ # and a session opened again would not re-declare it.
1549
+ def release_cancelled_consumer(tag, queue)
1550
+ topology&.resume_queue(queue)
1551
+ wake_each_waiters(tag)
1552
+ end
1553
+
1554
+ # Re-consume under the same tag, for as long as the queue is there and this
1555
+ # channel can carry a consumer. Nothing is declared and nothing is bound:
1556
+ # whatever the queue is, it is the one that was there before.
1557
+ def recover_cancelled_consumer(tag, entry)
1558
+ queue = entry[:queue_name]
1559
+ delay = RECOVER_CONSUMER_DELAY
1560
+ failures = 0
1561
+
1562
+ loop do
1563
+ # Every round starts with a connection that works and a fresh answer
1564
+ # about the queue. A queue deleted while we were retrying has to stop
1565
+ # the loop, and a basic_consume the broker answers with a 404 closes
1566
+ # this channel and takes the consumers still working on it down too -
1567
+ # so the question is asked on a channel of its own first, and only an
1568
+ # answer of "it is there" is followed by an attempt to consume.
1569
+ case wait_for_usable_channel
1570
+ when nil then return release_cancelled_consumer(tag, queue)
1571
+ when :reconnected
1572
+ # The connection was what was wrong, and it is back: start the
1573
+ # backoff again rather than sleeping out a delay earned by an outage
1574
+ # that is over. A two-second blip should not leave a healthy queue
1575
+ # unconsumed for the next thirty.
1576
+ delay = RECOVER_CONSUMER_DELAY
1577
+ end
1578
+
1579
+ case queue_status(queue)
1580
+ when :deleted
1581
+ return forget_deleted_queue(tag, queue)
1582
+ when :unreachable
1583
+ # No answer rather than an answer: the connection is what is wrong,
1584
+ # not the queue, so the backoff a queue earns does not apply. Go back
1585
+ # and wait for the connection instead of sleeping out a delay that
1586
+ # doubles every time the broker cannot be reached.
1587
+ delay = RECOVER_CONSUMER_DELAY
1588
+ when :available
1589
+ begin
1590
+ basic_consume(queue, consumer_tag: tag, manual_ack: entry[:manual_ack], exclusive: entry[:exclusive],
1591
+ arguments: entry[:arguments] || {}, on_error: entry[:on_error], &entry[:block])
1592
+ # Consuming again, so the queue is there and recovery can have it
1593
+ # back (#handle_broker_cancel held it).
1594
+ topology&.resume_queue(queue)
1595
+ @logger.info("Channel #{@channel_id}: re-subscribed consumer #{tag} to #{queue}")
1596
+ instrument("consumer.recovered") do
1597
+ { channel: @channel_id, consumer_tag: tag, queue: queue, attempts: failures + 1 }
1598
+ end
1599
+ return
1600
+ rescue *PROPAGATE_FROM_HANDLER
1601
+ raise
1602
+ rescue ConnectionError, NotOpenError => e
1603
+ # The connection went, not the subscription. That has its own
1604
+ # reporting and its own recovery, both already running, so it is
1605
+ # not counted as a failure to re-subscribe: an application alarming
1606
+ # on consumer.recovery_failed would otherwise be told twice about
1607
+ # one event, and told it in the words of the more serious of the
1608
+ # two.
1609
+ @logger.debug("Channel #{@channel_id}: re-subscribe of #{tag} interrupted by " \
1610
+ "#{e.class}; waiting for the connection")
1611
+ rescue ChannelError => e
1612
+ # The queue went between the question and the answer. The broker
1613
+ # has closed this channel saying so, so there is nothing left to
1614
+ # retry on.
1615
+ return forget_deleted_queue(tag, queue) if queue_missing?(e)
1616
+
1617
+ failures += 1
1618
+ note_recovery_failure(tag, queue, e, failures)
1619
+ rescue => e
1620
+ failures += 1
1621
+ note_recovery_failure(tag, queue, e, failures)
1622
+ end
1623
+ end
1624
+ # :unavailable falls through to the backoff and is asked again.
1625
+
1626
+ sleep delay
1627
+ delay = [delay * 2, RECOVER_CONSUMER_MAX_DELAY].min
1628
+ end
1629
+ end
1630
+
1631
+ # One line per episode at warn, the rest at debug: a broker that is away
1632
+ # for an hour would otherwise write a warning every attempt for the whole
1633
+ # hour, and an application counting warnings would read one outage as
1634
+ # hundreds.
1635
+ def note_recovery_failure(tag, queue, error, failures)
1636
+ if failures == 1
1637
+ @logger.warn("Channel #{@channel_id}: could not re-subscribe #{tag} to #{queue}, " \
1638
+ "retrying: #{error.class}: #{error.message}")
1639
+ instrument("consumer.recovery_failed") do
1640
+ { channel: @channel_id, consumer_tag: tag, queue: queue, reason: :error, error: error }
1641
+ end
1642
+ else
1643
+ @logger.debug("Channel #{@channel_id}: re-subscribe attempt #{failures} for #{tag} failed: #{error.message}")
1644
+ end
1645
+ end
1646
+
1647
+ # The queue behind a cancelled consumer has gone. Drop it from the topology
1648
+ # registry, with its bindings, so a later connection recovery does not
1649
+ # re-declare a queue somebody deleted on purpose - the "deleted queue comes
1650
+ # back" bug the Java client has been fixing instances of for years.
1651
+ def forget_deleted_queue(tag, queue)
1652
+ topology&.delete_queue(queue)
1653
+ @logger.warn("Channel #{@channel_id}: queue #{queue} no longer exists; consumer #{tag} will not be recovered")
1654
+ instrument("consumer.recovery_failed") do
1655
+ { channel: @channel_id, consumer_tag: tag, queue: queue, reason: :queue_deleted }
1656
+ end
1657
+ wake_each_waiters(tag)
1658
+ end
1659
+
1660
+ # Is the queue gone, there, or merely not being served right now? A passive
1661
+ # declare answers it, on a channel of its own because a passive declare
1662
+ # that fails closes the channel it ran on - which would take the queue's
1663
+ # fellow consumers down with it.
1664
+ #
1665
+ # Four answers, because conflating any two of them does harm. :deleted must
1666
+ # be forgotten and :unavailable must be waited for, and telling those apart
1667
+ # is the whole point. :unreachable - no answer at all, because the
1668
+ # connection is in trouble - is separated from :unavailable so that the
1669
+ # caller does not spend a backoff earned by a queue on a connection that is
1670
+ # simply down, and is never taken for :available, because a basic_consume
1671
+ # on a queue that is not there closes the channel.
1672
+ def queue_status(name)
1673
+ @session.with_channel { |probe| probe.queue(name, passive: true) }
1674
+ :available
1675
+ rescue ChannelError => e
1676
+ queue_missing?(e) ? :deleted : :unavailable
1677
+ rescue *PROPAGATE_FROM_HANDLER
1678
+ raise
1679
+ rescue => e
1680
+ @logger.debug("Channel #{@channel_id}: could not check whether #{name} still exists: #{e.class}: #{e.message}")
1681
+ :unreachable
1682
+ end
1683
+
1684
+ # Gone, rather than merely not being served. Taking a queue whose node is
1685
+ # down for a deleted one forgets a queue that is coming back: its consumer
1686
+ # is never restored, and it is never re-declared either. Any 404 the
1687
+ # QUEUE_UNAVAILABLE wordings do not cover reads as a deletion, which is the
1688
+ # safe way round - an unmatched wording costs only what forgetting it costs
1689
+ # today.
1690
+ def queue_missing?(error)
1691
+ error.code == 404 && !QUEUE_UNAVAILABLE.match?(error.text.to_s)
1372
1692
  end
1373
1693
 
1374
1694
  # Release fibers blocked in #each for one consumer, or for all of them.
@@ -36,6 +36,8 @@ module AsyncRabbitMQ
36
36
  channel.rpc
37
37
  consumer.registered
38
38
  consumer.cancelled
39
+ consumer.recovered
40
+ consumer.recovery_failed
39
41
  message.published
40
42
  message.confirmed
41
43
  message.returned
@@ -161,6 +161,7 @@ module AsyncRabbitMQ
161
161
  recovery_interval: RECOVERY_INITIAL,
162
162
  recovery_max_interval: RECOVERY_MAX,
163
163
  recover_topology: true,
164
+ recover_cancelled_consumers: false,
164
165
  topology_recovery_filter: nil,
165
166
  instrumenter: nil,
166
167
  notifier: nil,
@@ -194,6 +195,9 @@ module AsyncRabbitMQ
194
195
  @recovery_interval = recovery_interval
195
196
  @recovery_max_interval = recovery_max_interval
196
197
  @recover_topology = recover_topology
198
+ # The default every channel opened on this session takes; see
199
+ # Channel#recovers_cancelled_consumers? for why it is off.
200
+ @recover_cancelled_consumers = recover_cancelled_consumers
197
201
  @topology = TopologyRegistry.new(filter: topology_recovery_filter)
198
202
  @logger = logger
199
203
  @notifier = notifier || Notifier.new(logger: logger)
@@ -354,12 +358,15 @@ module AsyncRabbitMQ
354
358
  # Open a new channel. Returns an AsyncRabbitMQ::Channel.
355
359
  # +pool_size+ bounds concurrent consumer-handler fibers on the channel
356
360
  # (Bunny-parity: default 1). basic_qos will auto-adjust it to prefetch_count.
357
- def open_channel(pool_size: 1)
361
+ # +recover_cancelled_consumers+ defaults to the session's setting; see
362
+ # Channel#recovers_cancelled_consumers?.
363
+ def open_channel(pool_size: 1, recover_cancelled_consumers: @recover_cancelled_consumers)
358
364
  raise NotOpenError, "Session is not open" unless open?
359
365
  channel = @channel_mutex.acquire do
360
366
  channel_id = next_channel_id
361
367
  Channel.new(channel_id, self, @frame_io, frame_max: @negotiated_fm || @frame_max, logger: @logger,
362
- pool_size: pool_size, rpc_timeout: @rpc_timeout).tap { |ch| @channels[channel_id] = ch }
368
+ pool_size: pool_size, rpc_timeout: @rpc_timeout,
369
+ recover_cancelled_consumers: recover_cancelled_consumers).tap { |ch| @channels[channel_id] = ch }
363
370
  end
364
371
  channel.open
365
372
  channel
@@ -38,6 +38,7 @@ module AsyncRabbitMQ
38
38
  @queue_bindings = []
39
39
  @exchange_bindings = []
40
40
  @consumers = {} # consumer tag => queue name
41
+ @suspended = Set.new # queue names held back from recovery
41
42
  end
42
43
 
43
44
  # --- recording -----------------------------------------------------------
@@ -59,6 +60,9 @@ module AsyncRabbitMQ
59
60
 
60
61
  def record_queue(channel_id, name, durable:, exclusive:, auto_delete:, arguments:, server_named:, object: nil)
61
62
  existing = @queues[name]
63
+ # Declared again by the application, so whatever doubt a cancelled
64
+ # consumer raised about it is settled.
65
+ @suspended.delete(name)
62
66
  rec = RecordedQueue.new(channel_id: channel_id, name: name, durable: durable, exclusive: exclusive,
63
67
  auto_delete: auto_delete, arguments: arguments.dup, server_named: server_named,
64
68
  objects: existing ? existing.objects : [])
@@ -68,15 +72,45 @@ module AsyncRabbitMQ
68
72
 
69
73
  def delete_queue(name)
70
74
  @queues.delete(name)
75
+ @suspended.delete(name)
71
76
  gone, @queue_bindings = @queue_bindings.partition { |b| b.queue == name }
72
77
  gone.each { |b| prune_auto_delete_exchange(b.exchange) }
73
78
  end
74
79
 
80
+ # --- suspension ----------------------------------------------------------
81
+ #
82
+ # A queue is suspended while the client is working out whether it is still
83
+ # there, which is what a consumer the broker cancelled leaves it not
84
+ # knowing. Recovery skips a suspended queue and its bindings.
85
+ #
86
+ # The alternative - keep recovering it until the client is sure - puts a
87
+ # deleted queue back, because the deletion and the reconnect race: the node
88
+ # that was serving the queue is often the one that just went, and a
89
+ # reconnect re-declares the whole registry before anything can ask the new
90
+ # connection whether the queue survived. A queue the broker then hands back
91
+ # with a consumer attached cannot be deleted at all. Holding it back costs
92
+ # nothing by comparison: the worst case is a queue that was fine and is not
93
+ # re-declared for the few seconds it takes to establish that.
94
+
95
+ def suspend_queue(name)
96
+ @suspended << name if @queues.key?(name)
97
+ end
98
+
99
+ def resume_queue(name)
100
+ @suspended.delete(name)
101
+ end
102
+
103
+ # Queue names currently held back, for inspection.
104
+ def suspended_queues
105
+ @suspended.dup
106
+ end
107
+
75
108
  # A server-named queue was re-declared under a different name.
76
109
  def rename_queue(old_name, new_name)
77
110
  rec = @queues.delete(old_name) or return
78
111
  rec.name = new_name
79
112
  @queues[new_name] = rec
113
+ @suspended << new_name if @suspended.delete?(old_name)
80
114
  @queue_bindings.each { |b| b.queue = new_name if b.queue == old_name }
81
115
  @consumers.transform_values! { |queue| queue == old_name ? new_name : queue }
82
116
  rec.objects.each { |q| q.update_name_to(new_name) }
@@ -135,11 +169,11 @@ module AsyncRabbitMQ
135
169
  end
136
170
 
137
171
  def queues
138
- apply(:filter_queues, @queues.values)
172
+ apply(:filter_queues, @queues.each_value.reject { |q| @suspended.include?(q.name) })
139
173
  end
140
174
 
141
175
  def queue_bindings
142
- apply(:filter_queue_bindings, @queue_bindings.dup)
176
+ apply(:filter_queue_bindings, @queue_bindings.reject { |b| @suspended.include?(b.queue) })
143
177
  end
144
178
 
145
179
  def exchange_bindings
@@ -159,6 +193,7 @@ module AsyncRabbitMQ
159
193
  @queue_bindings.clear
160
194
  @exchange_bindings.clear
161
195
  @consumers.clear
196
+ @suspended.clear
162
197
  end
163
198
 
164
199
  # Forget only what was tied to the connection that just went: exclusive
@@ -174,6 +209,7 @@ module AsyncRabbitMQ
174
209
 
175
210
  dropped_queues.each { |name| @queues.delete(name) }
176
211
  dropped_exchanges.each { |name| @exchanges.delete(name) }
212
+ @suspended.subtract(dropped_queues)
177
213
 
178
214
  @queue_bindings.reject! do |b|
179
215
  dropped_queues.include?(b.queue) || dropped_exchanges.include?(b.exchange)
@@ -1,3 +1,3 @@
1
1
  module AsyncRabbitMQ
2
- VERSION = "0.7.1"
2
+ VERSION = "0.8.1"
3
3
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: async-rabbitmq
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.7.1
4
+ version: 0.8.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Russell Penney