async-rabbitmq 0.4.0 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c8e21899276061f5e0ecd749000e4b7bf5b81932a491a1704e0b2e643cf647b1
-  data.tar.gz: a7fcd5f434e5e677ad7fce840ca388e323264edd8b2347f8ed92ec73a0fd7a3e
+  metadata.gz: 77df2cb6ea955db0392941563a8e6c43cc55ad06d429c1f8d45ea05bd79ecb50
+  data.tar.gz: 1ea2fbdc554c5901638002e7d4c16cbaf5c9af7189434b06fb1003eef2894d14
 SHA512:
-  metadata.gz: 0a41b23ef4e9d1fdad8403e8dbc99783f53205ed6b0fe5cb3e891b79d9763b9a9c0ec05bceda15f78de1fb47e0fb5b65bdbc5b756c2d19cf9c6609d2ee13fd3c
-  data.tar.gz: 4e7795488172face945464fbd3d36c9355b7b6500705238079df4ae38b0c6529c67088c84323354c0772bd31562b9efb7b1927c06872b369addc4d56638c5815
+  metadata.gz: '0197dc20bc43a0863883f0a723a4120844c05bbc1542a42509fa7cb98473414f860d034a65cdf8e645a8a7aec414c6bfa347887b803d191c14745ec4dc4b2681'
+  data.tar.gz: aaa2b3835cfefed5612f5b0e93f68dd3f9985d0cd4436c1201e5d7907b2fe25c194a05249de67680b7531ce3c8aaa22bc8d95a8fa15317539755da9c6eb8f0ba

data/CHANGELOG.md

@@ -2,6 +2,74 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.6.0] - 2026-10-11
+
+### Fixed
+
+- A consumer handler raising outside `StandardError` no longer stops the channel dispatching.
+  `SystemStackError` and `ScriptError` now fail that delivery through the configured error
+  disposition. `Async::Stop`, `SystemExit`, `SignalException` and `NoMemoryError` still propagate.
+  Applies equally to the `on_handler_error` hook and to settling after an error.
+- Handler workers are created on demand instead of one per prefetch slot. `basic_qos(prefetch_count: 500)`
+  previously created 500 fibers and retired all but `pool_size` of them unused: 12.6 MB at prefetch
+  500 across two channels. Concurrency is unchanged.
+
+### Added
+
+- `on_error:` on a consumer, and `Channel#on_error_disposition` for the channel default, choose what
+  happens to a delivery whose handler raised:
+  - `:dead_letter` (default) — `reject(requeue: false)`: one attempt, then the dead-letter route.
+  - `:retry` — `reject(requeue: true)`: counts towards `x-delivery-limit`.
+  - `:release` — `nack(requeue: true)`: requeued without counting towards the limit.
+  - `:leave` — settles nothing; the handler is responsible.
+- `Channel#on_handler_error` receives `(error, delivery_info, queue_name)` and may return a
+  disposition to override the default for that delivery.
+
+## [0.5.0] - 2026-10-09
+
+### Fixed
+
+- **A consumer backlog no longer costs a fiber per queued message.** The dispatch loop started an
+  `Async::Task` per delivery and took the pool semaphore from inside it, so `pool_size` capped how
+  many handlers *ran*, not how many *existed*. A parked task measured ~15.7 KB (the fiber object
+  plus its touched stack pages) against ~48 bytes for a queue entry, and 20,000 backlogged messages
+  cost 315 MB of RSS. Handlers now run in `pool_size` long-lived worker fibers draining a queue of
+  deliveries — the same shape as Bunny's `ConsumerWorkPool`, the .NET client's consumer dispatcher
+  and amqp091-go.
+
+  That queue is deliberately unbounded. The dispatch loop is also what delivers this channel's
+  confirms, close-ok and returns, so blocking it on a full queue would deadlock a handler that
+  publishes and waits for confirms — which is precisely rabbitmq-amqp-java-client#328, where a
+  bounded work pool parked the I/O thread in `LinkedBlockingQueue#put`. Memory is therefore bounded
+  by prefetch on a manual-ack consumer and unbounded on an auto-ack one, as it is in every client,
+  because automatic acknowledgement has no backpressure to offer.
+- **The missing-prefetch warning no longer sends auto-ack consumers to a no-op.** RabbitMQ applies
+  prefetch only to *unacknowledged* messages, so `basic_qos` does nothing at all on a
+  `manual_ack: false` consumer. The warning told you to call it anyway, and then went quiet once you
+  had — silence in the one case where the caller most believes the consumer is bounded and it is
+  not. It now says which of the two cases you are in, and still warns an auto-ack consumer that has
+  set a prefetch.
+
+### Added
+
+- `Channel#backlog` — deliveries handed to the handler workers but not yet picked up, the number
+  Bunny reports as `ConsumerWorkPool#backlog`.
+- `AsyncRabbitMQ.warn_unbounded_consumers` — a deliberate way to silence the unbounded-consumer
+  warning for someone who has read it and accepted the trade, instead of turning the logger down
+  and losing every other warning a channel raises (stale delivery tags, broker-cancelled consumers,
+  handler exceptions). One process-wide setting, read when a consumer starts; defaults to `true`.
+
+### Changed
+
+- A backlog still queued when the channel goes away is now **finished on a graceful `close`** (as
+  Bunny's `ConsumerWorkPool#shutdown` drains) and **discarded when the connection is lost or the
+  channel is reopened**, where the broker requeues the same messages and running the queued handlers
+  too would process each one twice.
+- `Channel#pool_size=` resizes by adding or retiring workers rather than resizing a semaphore.
+  Growing takes effect at once; shrinking retires a worker as soon as the queue drains to the
+  sentinel — immediately when idle, after the current backlog when busy. A running handler is never
+  interrupted.
+
 ## [0.4.0] - 2026-10-07
 
 Reliability review against a payments workload. The two delivery-correctness items

data/README.md

@@ -1,8 +1,53 @@
 # async-rabbitmq
 
-A fiber-native RabbitMQ (AMQP 0-9-1) client for Ruby, built on the
+A fiber-native RabbitMQ (AMQP 0-9-1) client for Ruby, built on the amazing work
+of [Bunny](https://github.com/ruby-amqp/bunny).
+
+## Installing
+
+```ruby
+# Gemfile
+gem "async-rabbitmq", "~> 0.5"
+```
+
+or
+
+```bash
+gem install async-rabbitmq
+```
+
+Requires Ruby 3.4+ and RabbitMQ 3.13+ (tested against 4.x). The gem also
+installs an `async-rabbitmq` command for publishing, consuming, inspecting and
+purging from a terminal.
+
+## Bunny is the gold standard
+
+Bunny is the client Ruby learned RabbitMQ on. Its API is the one Ruby developers
+already know, its defaults are the ones a decade of production traffic settled,
+and its source is the reference this library was written against. None of that
+is improved on here, and none of it is meant to be.
+
+**Parity with Bunny is a design goal**, chosen deliberately over fiber-native
+maximalism: the audience for this gem is people who already know Bunny, and
+surprising them is a bug. Where Bunny has an answer, that is the answer here.
+`pool_size` defaults to 1 and serialises handlers because `ConsumerWorkPool`
+does. A consumer backlog is an unbounded queue drained by a fixed set of workers
+because that is what `ConsumerWorkPool` is. `basic_get` acknowledges manually by
+default because Bunny's does. `spec/integration/25_bunny_api_parity_spec.rb`
+exists to keep it honest.
+
+Where this client does diverge, the README says so and says why - see the two
+publishing differences below, and `basic_qos`, which also resizes the handler
+pool where Bunny's leaves it alone.
+
+If you are not running under the fiber scheduler, use Bunny. It is excellent,
+and this gem has no advantage over it outside an async application.
+
+## What this adds
+
+The runtime, not the API. Built on the
 [async](https://github.com/socketry/async) ecosystem and
-[amq-protocol](https://github.com/ruby-amqp/amq-protocol). No threads: the
+[amq-protocol](https://github.com/ruby-amqp/amq-protocol), with no threads: the
 reader, writer, heartbeat and every consumer handler are fibers, so it fits
 Falcon, async-http and anything else running under the fiber scheduler.
 
@@ -140,11 +185,30 @@ ch.basic_cancel(tag)
 ch.each("q") { |delivery, header, body| ... }   # blocks until the consumer or channel goes away
 ```
 
-Consumer handlers run in their own fibers, at most `pool_size` at a time per
-channel. Set `basic_qos` before `basic_consume`: without a prefetch limit the
-broker sends the whole queue as fast as it can and one fiber is created per
-delivery, so memory tracks queue depth rather than concurrency. The client
-warns once per channel if you don't.
+Each channel runs `pool_size` long-lived handler fibers (default 1, serialized,
+Bunny parity) draining a queue of deliveries, so a backlog costs the messages it
+holds rather than a fiber per message. `ch.backlog` reports how many deliveries
+are waiting, as Bunny's `ConsumerWorkPool#backlog` does.
+
+What bounds that backlog depends on the ack mode. With `manual_ack: true`, set
+`basic_qos` before `basic_consume` and the broker holds everything past the
+prefetch window. With automatic acks there is no backpressure to be had:
+RabbitMQ applies prefetch only to *unacknowledged* messages, so the broker
+treats each one as acked on send and keeps sending whatever you set. The client
+warns once per channel in either case, and says which of the two you are in.
+
+If you have read that warning and accepted the trade - an auto-ack consumer on a
+queue you know stays shallow, say - switch it off during boot:
+
+```ruby
+AsyncRabbitMQ.warn_unbounded_consumers = false
+```
+
+It is deliberately one process-wide setting rather than a per-channel option: an
+application either accepts unbounded consumers or it does not. The point of
+having it at all is that the alternative is turning the logger down, which would
+also lose stale delivery tags, broker-cancelled consumers and handler
+exceptions.
 
 ### Acknowledging, and delivery tags across a reconnect
 
@@ -174,17 +238,98 @@ sent and `false` when the tag belonged to an earlier generation and was
 dropped. A dropped ack is not a lost message: the broker requeued it when the
 channel went, and it is redelivered on the new connection.
 
-When a handler raises, the delivery is nacked with `requeue: false` — so a
-message that always fails dead-letters instead of looping — and
-`on_handler_error` is called. A delivery the handler already settled itself is
-left alone, so acking and then raising in whatever follows is safe:
+### When a handler raises
+
+The delivery is settled for you. What happens to it is the *disposition*, and
+there are three:
+
+| | what it does | when |
+|---|---|---|
+| `:dead_letter` (default) | `reject(requeue: false)` | one attempt, then the queue's dead-letter route. Safe on any queue. |
+| `:retry` | `reject(requeue: true)` | the message failed — hand it back and let the broker count it. See below. |
+| `:release` | `nack(requeue: true)` | *not* the message's fault — hand it back without counting it. |
+| `:leave` | nothing | you settle it yourself. This is Bunny's behaviour. |
+
+`:retry` and `:release` both return the message; the difference is blame.
+RabbitMQ counts a delivery as failed only on `basic.reject` (AMQP 1.0
+`modified` with `delivery-failed`), not on `basic.nack` (AMQP 1.0 `released`).
+So use `:retry` for a message that cannot be processed, and `:release` when the
+message is innocent and you simply cannot handle it now — shutting down,
+shedding load, a downstream outage. Rejecting during a gateway outage would
+walk your whole backlog to the dead-letter queue in `x-delivery-limit`
+attempts; releasing it costs the messages nothing.
+
+The trade is that `:release` has no backstop: nothing advances towards
+`x-delivery-limit`, so a genuinely poison message is redelivered forever. Use it
+when you know the failure is yours, not the message's.
+
+Set it per consumer, or channel-wide for all of them:
+
+```ruby
+ch.on_error_disposition = :retry                       # channel default
+ch.basic_consume("q", manual_ack: true, on_error: :leave) { |d, h, b| ... }
+```
+
+A delivery the handler already settled itself is left alone, so acking and then
+raising in whatever follows is safe.
+
+`on_handler_error` runs **before** the delivery is settled, so what it returns
+can decide this one message's fate. Return a disposition to override; return
+anything else, or nothing, to take the configured default:
 
 ```ruby
 ch.on_handler_error do |error, delivery, queue_name|
   Sentry.capture_exception(error, extra: { queue: queue_name })
+  case error
+  when GatewayDown           then :release      # innocent, do not count it
+  when TransientUpstreamError then :retry       # failed, count towards the limit
+  else                            :dead_letter
+  end
 end
 ```
 
+A hook that raises is logged and the configured default applies, so a broken
+hook cannot strand a delivery.
+
+### Letting the broker do the retrying
+
+`:retry` is built for quorum queues, which count failed deliveries themselves:
+
+```ruby
+ch.quorum_queue("orders.commands", arguments: {
+  "x-delivery-limit"          => 5,          # retire after 5 failures
+  "x-delayed-retry-type"      => "failed",   # back off between attempts
+  "x-delayed-retry-min"       => 1_000,
+  "x-delayed-retry-max"       => 30_000,
+  "x-dead-letter-exchange"    => "",
+  "x-dead-letter-routing-key" => "orders.commands.parked",
+})
+```
+
+The broker spaces the attempts (`min(min_delay × delivery_count, max_delay)`),
+counts them in the `x-delivery-count` header, and dead-letters the message once
+`x-delivery-limit` is reached. There is no second delay queue to declare and no
+attempt counting to write.
+
+Two things that are easy to get wrong:
+
+**It has to be `reject`, not `nack`.** RabbitMQ only treats a delivery as
+*failed* on `basic.reject`; after `basic.nack` the `x-delivery-count` does not
+move, so a nacked message is redelivered forever and never reaches the limit.
+This client always rejects, which is why `:retry` works at all. If you settle
+deliveries yourself, use `basic_reject`.
+
+**Reconnects spend the same budget.** A channel closing with the delivery
+unacknowledged — a disconnect, a recovery, a broker-side channel error — counts
+as a failed delivery too. A message can therefore be dead-lettered by three
+reconnects without your handler ever failing. Size `x-delivery-limit` for
+handler failures *plus* the reconnects you expect, and read the dead-letter
+queue as "failed or unlucky".
+
+Without `x-delivery-limit`, `:retry` is an infinite loop for a message that
+always fails. That is why the default is `:dead_letter`: it is safe on any
+queue, and a default should not depend on configuration the client cannot see.
+
 ### RabbitMQ 4.2+ and transient queues
 
 RabbitMQ 4.2 and later refuse to declare a queue that is both non-durable
@@ -282,8 +427,7 @@ end
 ```
 
 `Async::Condition#wait`, a `sleep`, or your own supervisor task work equally
-well. Versions up to 0.3.0 kept the reactor alive by accident, because those
-tasks were children of whichever task called `connect`.
+well.
 
 ## Connection recovery
 

data/lib/async_rabbitmq/channel.rb

@@ -9,6 +9,51 @@ module AsyncRabbitMQ
   # Duck-typed stream interface: #each (yields deliveries) and #write (publishes).
   # Does NOT inherit Async::IO::Stream — a channel is logical, not physical IO.
   class Channel
+    # What to do with a delivery whose handler raised. Chosen per consumer, or
+    # returned by the on_handler_error hook to decide per message.
+    #
+    # :dead_letter  reject(requeue: false) -- one attempt, then the queue's
+    #               dead-letter route. Safe on any queue, so it is the default:
+    #               :retry on a queue with no delivery limit loops forever, and
+    #               a default must not depend on configuration the client
+    #               cannot see.
+    # :retry        reject(requeue: true) -- hand it back and let the broker
+    #               count. A quorum queue with x-delivery-limit retires it (and
+    #               dead-letters it) once the limit is reached, with
+    #               x-delayed-retry-* spacing the attempts.
+    # :release      nack(requeue: true) -- hand it back WITHOUT counting the
+    #               failure against the message. For a failure that is not the
+    #               message's fault: shutting down, shedding load, a downstream
+    #               outage. On a poison message this loops forever, since
+    #               nothing advances towards x-delivery-limit.
+    # :leave        settle nothing. Bunny's behaviour: the delivery stays
+    #               unacknowledged and it is yours to resolve.
+    #
+    # :retry rejects and :release nacks, and the difference is the whole point.
+    # The broker counts a delivery as failed only on basic.reject (AMQP 1.0
+    # modified/delivery-failed); after basic.nack the x-delivery-count does not
+    # move (AMQP 1.0 released). So reject means "this message failed, hold it
+    # against it" and nack means "have it back, no blame". For a single
+    # delivery the verbs are otherwise equivalent.
+    ERROR_DISPOSITIONS = %i[dead_letter retry release leave].freeze
+
+    # A handler is application code and can raise anything. Catching only
+    # StandardError let a handler raising outside that hierarchy escape and take
+    # the channel's frame dispatch with it, so the catch below is deliberately
+    # as wide as Ruby allows - and these are what it has to put back.
+    #
+    # Async stops a task by raising Async::Stop into it, and Async::Stop is not
+    # a StandardError precisely so that ordinary rescues leave it alone.
+    # Swallowing it would leave handler workers unstoppable, and a channel close
+    # or a reactor shutdown would hang on a parked handler. The signals beside
+    # it are not this delivery's problem either: the process is exiting, was
+    # interrupted, or has run out of memory.
+    #
+    # SystemStackError and ScriptError are deliberately absent. An accidental
+    # infinite recursion in a handler is an ordinary bug and should fail that
+    # one delivery, not silence the channel.
+    PROPAGATE_FROM_HANDLER = [Async::Stop, SystemExit, SignalException, NoMemoryError].freeze
+
     # A message published under confirms whose fate is unknown: the broker
     # neither acked nor nacked it before the connection went. +payload+ and its
     # routing are kept so it can be published again on another connection.
@@ -24,6 +69,10 @@ module AsyncRabbitMQ
     # after an RPC timeout, used when the session has no rpc_timeout set.
     RESYNC_TIMEOUT = 5
 
+    # Pushed onto the work queue to retire one handler worker. Compared by
+    # identity, so it can never collide with a delivery.
+    WORKER_STOP = Object.new.freeze
+
     include Instrumented
 
     attr_reader :channel_id, :pool_size
@@ -73,6 +122,7 @@ module AsyncRabbitMQ
       @on_cancel         = nil
       @on_error          = nil
       @on_handler_error  = nil
+      @on_error_disposition = :dead_letter
       @resyncing         = false
       @resync_condition  = nil
       # Incremented on every reopen (connection recovery or a single-channel
@@ -89,14 +139,27 @@ module AsyncRabbitMQ
       @pending_content   = nil
       @recovered_condition = nil     # fibers parked while the connection recovers
       @pool_size         = validate_pool_size!(pool_size)
-      @pool_sem          = Async::Semaphore.new(@pool_size)
+      @work_queue        = nil       # Async::Queue of [method, header, body, entry]
+      @workers           = []        # long-lived handler fibers draining it
     end
 
-    # Resize the consumer-handler concurrency cap. Shrinking does not evict
-    # already-running handlers; new deliveries park until permits free up.
+    # Resize the consumer-handler concurrency cap by adding or retiring
+    # workers. Growing takes effect at once. Shrinking is a sentinel on the
+    # work queue, so it retires a worker as soon as the queue drains to it:
+    # immediately when idle, after the current backlog when busy. A handler
+    # already running is never interrupted.
     def pool_size=(n)
-      @pool_size      = validate_pool_size!(n)
-      @pool_sem.limit = @pool_size
+      n = validate_pool_size!(n)
+      return if n == @pool_size
+
+      @pool_size = n
+      resize_workers
+    end
+
+    # Deliveries handed to the handler workers but not yet picked up. Bunny
+    # reports the same number as ConsumerWorkPool#backlog.
+    def backlog
+      @work_queue&.size || 0
     end
 
     def open?
@@ -147,6 +210,7 @@ module AsyncRabbitMQ
       instrument("channel.closed") { { channel: @channel_id, reason: :user } }
       @session.channel_closed(@channel_id)
       @queue&.push(nil)  # wake dispatch_loop so it can detect :closed and exit
+      stop_workers
       wake_each_waiters
     end
 
@@ -408,13 +472,19 @@ module AsyncRabbitMQ
     # gated by the channel's pool_size semaphore so at most +pool_size+
     # handlers run concurrently across all consumers on this channel.
     # Returns the consumer tag.
-    def basic_consume(queue_name, consumer_tag: "", manual_ack: false, exclusive: false, arguments: {}, &block)
-      warn_unbounded_prefetch(queue_name)
+    # +on_error+ decides what happens to a delivery whose handler raises; see
+    # ERROR_DISPOSITIONS. nil takes the channel's default, set with
+    # #on_error_disposition=.
+    def basic_consume(queue_name, consumer_tag: "", manual_ack: false, exclusive: false,
+                      arguments: {}, on_error: nil, &block)
+      validate_disposition!(on_error) if on_error
+      warn_unbounded_prefetch(queue_name, manual_ack)
       resp = rpc(
         AMQ::Protocol::Basic::Consume.encode(@channel_id, queue_name, consumer_tag, false, !manual_ack, exclusive, false, arguments),
         AMQ::Protocol::Basic::ConsumeOk
       )
-      @consumers[resp.consumer_tag] = { queue_name: queue_name, block: block, manual_ack: manual_ack }
+      @consumers[resp.consumer_tag] = { queue_name: queue_name, block: block,
+                                        manual_ack: manual_ack, on_error: on_error }
       topology&.record_consumer(resp.consumer_tag, queue_name)
       instrument("consumer.registered") do
         { channel: @channel_id, queue: queue_name, consumer_tag: resp.consumer_tag, manual_ack: manual_ack }
@@ -587,10 +657,23 @@ module AsyncRabbitMQ
     # receives the exception, the Basic::Deliver method frame and the queue
     # name. The delivery has already been nacked (requeue: false) if the
     # consumer was registered with manual_ack: true.
+    # The block receives the exception, the Basic::Deliver frame and the queue
+    # name, and runs *before* the delivery is settled. Return one of
+    # ERROR_DISPOSITIONS to decide this message's fate; return anything else
+    # (or nothing) to take the consumer's or channel's configured default.
     def on_handler_error(&block)
       @on_handler_error = block
     end
 
+    # The channel-wide default disposition for a handler that raises, used by
+    # any consumer that did not set its own. See ERROR_DISPOSITIONS.
+    attr_reader :on_error_disposition
+
+    def on_error_disposition=(value)
+      validate_disposition!(value)
+      @on_error_disposition = value
+    end
+
     # -------------------------------------------------------------------------
     # Flow control
     # -------------------------------------------------------------------------
@@ -667,6 +750,9 @@ module AsyncRabbitMQ
       @confirm_condition = nil
       release_outstanding_slots(error)
       @queue&.push(nil) rescue nil
+      # The connection is gone, so the queued deliveries are too: the broker
+      # requeues whatever was unacknowledged and sends it again on the new one.
+      stop_workers(discard: true)
     rescue => e
       # ignore — best-effort unblock
     end
@@ -832,20 +918,34 @@ module AsyncRabbitMQ
       true
     end
 
-    # Without a prefetch limit the broker sends as fast as it can and the
-    # dispatch loop starts a task per delivery. pool_size caps how many run at
-    # once, not how many exist, so memory grows with the queue depth rather
-    # than with the concurrency. Warned once per channel; basic_qos fixes it.
-    def warn_unbounded_prefetch(queue_name)
+    # Without a bound the broker sends as fast as it can and the backlog is
+    # held in this process. What can bound it depends on the ack mode, so the
+    # advice does too: RabbitMQ applies prefetch only to *unacknowledged*
+    # messages, so on an auto-ack consumer basic_qos never engages and telling
+    # the user to call it would be wrong. Warned once per channel.
+    def warn_unbounded_prefetch(queue_name, manual_ack)
+      return unless AsyncRabbitMQ.warn_unbounded_consumers
       return if @qos_warned
-      return if @prefetch && @prefetch[:count].to_i > 0
+      # A prefetch answers the manual-ack case, so having set one is a reason to
+      # stay quiet there. It does nothing for an auto-ack consumer, so it is not
+      # a reason here: that is precisely the case where the caller believes the
+      # consumer is bounded and it is not.
+      return if manual_ack && @prefetch && @prefetch[:count].to_i > 0
 
       @qos_warned = true
       @logger.warn(
-        "Channel #{@channel_id}: consuming from #{queue_name} without basic_qos. " \
-        "The broker will send the whole queue as fast as it can and one task is " \
-        "created per delivery, so memory tracks queue depth. Call " \
-        "basic_qos(prefetch_count: n) before basic_consume."
+        if manual_ack
+          "Channel #{@channel_id}: consuming from #{queue_name} without basic_qos. " \
+          "The broker will send the whole queue as fast as it can and the backlog is " \
+          "held in this process. Call basic_qos(prefetch_count: n) before " \
+          "basic_consume to bound it."
+        else
+          "Channel #{@channel_id}: consuming from #{queue_name} with automatic acks. " \
+          "The broker treats every message as acknowledged on send, so basic_qos " \
+          "cannot bound this consumer and the whole queue is held in this process. " \
+          "Use manual_ack: true with basic_qos(prefetch_count: n) to get " \
+          "backpressure, or watch #backlog."
+        end
       )
     end
 
@@ -870,33 +970,67 @@ module AsyncRabbitMQ
     # the consumer stops receiving anything at all.
     def handle_consumer_error(error, method, entry)
       @logger.error(
-        "Channel #{@channel_id}: consumer #{method.consumer_tag} raised " \
-        "#{error.class}: #{error.message}"
+        "Channel #{@channel_id}: consumer #{method.consumer_tag} raised "         "#{error.class}: #{error.message}"
       )
       instrument("consumer.error") do
         { channel: @channel_id, consumer_tag: method.consumer_tag, queue: entry[:queue_name],
           error: error.class.name, message: error.message }
       end
 
+      # The hook runs before the delivery is settled, so what it returns can
+      # decide the disposition. Anything outside ERROR_DISPOSITIONS (including
+      # nil, and whatever a hook that was not written to choose happens to
+      # return) falls through to the configured default.
+      chosen = call_handler_error_hook(error, method, entry)
+      disposition = ERROR_DISPOSITIONS.include?(chosen) ? chosen : consumer_disposition(entry)
+      return if disposition == :leave
+
       # Only a manual-ack delivery the handler has not already settled. A
       # handler that acks and then raises in whatever follows would otherwise
-      # be nacking a tag the broker has already resolved: that is a 406, which
-      # closes the channel and takes its consumers with it.
-      if entry[:manual_ack] && unsettled?(method.delivery_tag)
-        begin
-          basic_nack(method.delivery_tag, requeue: false)
-        rescue => e
-          @logger.error("Channel #{@channel_id}: could not nack after consumer error: #{e.class}: #{e.message}")
+      # be rejecting a tag the broker has already resolved: that is a 406,
+      # which closes the channel and takes its consumers with it.
+      return unless entry[:manual_ack] && unsettled?(method.delivery_tag)
+
+      begin
+        if disposition == :release
+          # nack, not reject: released rather than failed, so the broker does
+          # not count it against the message.
+          basic_nack(method.delivery_tag, requeue: true)
+        else
+          basic_reject(method.delivery_tag, requeue: disposition == :retry)
         end
+      rescue *PROPAGATE_FROM_HANDLER
+        raise
+      rescue Exception => e # rubocop:disable Lint/RescueException
+        @logger.error("Channel #{@channel_id}: could not settle after consumer error: #{e.class}: #{e.message}")
       end
+    end
 
-      return unless @on_handler_error
+    # Returns whatever the hook returned, or nil if there is no hook or it
+    # raised. A hook that blows up must not also lose the delivery.
+    def call_handler_error_hook(error, method, entry)
+      return nil unless @on_handler_error
 
-      begin
-        @on_handler_error.call(error, method, entry[:queue_name])
-      rescue => e
-        @logger.error("Channel #{@channel_id}: on_handler_error hook raised #{e.class}: #{e.message}")
-      end
+      @on_handler_error.call(error, method, entry[:queue_name])
+    rescue *PROPAGATE_FROM_HANDLER
+      raise
+    rescue Exception => e # rubocop:disable Lint/RescueException
+      # The hook is the caller's code as much as the handler is, and it runs
+      # while a delivery is still unsettled: letting it escape would strand the
+      # message and kill the worker.
+      @logger.error("Channel #{@channel_id}: on_handler_error hook raised #{e.class}: #{e.message}")
+      nil
+    end
+
+    def consumer_disposition(entry)
+      entry[:on_error] || @on_error_disposition
+    end
+
+    def validate_disposition!(value)
+      return if ERROR_DISPOSITIONS.include?(value)
+
+      raise ArgumentError,
+            "on_error must be one of #{ERROR_DISPOSITIONS.map(&:inspect).join(', ')} (got #{value.inspect})"
     end
 
     # -------------------------------------------------------------------------
@@ -904,17 +1038,130 @@ module AsyncRabbitMQ
     # -------------------------------------------------------------------------
 
     def start_dispatch_task
+      start_workers
       @session.spawn_background { dispatch_loop }
     end
 
+    # Up to pool_size long-lived fibers drain the work queue for the life of
+    # this channel generation. A backlogged delivery costs a 4-element array
+    # (~48 bytes) rather than a parked Async::Task and its fiber stack (~15.7 KB
+    # measured), so memory tracks the messages held, not the fibers holding
+    # them. Bunny's ConsumerWorkPool is the same shape: workers draining an
+    # unbounded ::Queue.
+    #
+    # The queue is made here; the workers are not. See #ensure_worker.
+    def start_workers
+      stop_workers(discard: true)
+      @work_queue = Async::Queue.new
+      @workers    = []
+    end
+
+    # Create a worker on demand, never up front.
+    #
+    # basic_qos couples pool_size to prefetch_count, so creating them eagerly
+    # meant a caller asking for a prefetch of 500 got 500 fibers, and a caller
+    # who then set pool_size back down - the documented escape hatch from that
+    # coupling - retired all but pool_size of them without a single delivery
+    # between them. A fiber's touched stack pages stay in the process RSS after
+    # it exits, measured at ~12 KB apiece: 12.6 MB at prefetch 500 across two
+    # channels, which is the whole of the consumer memory difference the 24-hour
+    # soaks showed.
+    #
+    # Demand never exceeds pool_size, so the concurrency bound is unchanged: a
+    # channel that genuinely runs pool_size handlers at once still gets there,
+    # one delivery at a time.
+    def ensure_worker(queue)
+      return if queue.nil? || @workers.size >= @pool_size
+
+      @workers << @session.spawn_background { worker_loop(queue) }
+    end
+
+    # Retire this generation's workers. The sentinel goes on the back of the
+    # queue, so by default they finish the deliveries already queued before
+    # standing down - Bunny's ConsumerWorkPool#shutdown drains the same way.
+    #
+    # +discard+ throws that backlog away instead, for when the connection
+    # underneath it has gone: the broker requeues whatever was unacknowledged
+    # and sends it again on the new one, so running these handlers too would
+    # process every queued message twice. dequeue(timeout: 0) returns at once
+    # on an empty queue, so the drain cannot suspend us mid-teardown.
+    def stop_workers(discard: false)
+      queue = @work_queue
+      nil while discard && queue&.dequeue(timeout: 0)
+      @workers.size.times { queue&.push(WORKER_STOP) }
+      @workers    = []
+      @work_queue = nil
+    end
+
+    # Only ever shrinks. Growing is left to #ensure_worker, so raising pool_size
+    # costs nothing until there are deliveries that need the extra concurrency.
+    def resize_workers
+      queue  = @work_queue
+      excess = @workers.size - @pool_size
+      return if queue.nil? || !excess.positive?
+
+      excess.times { queue.push(WORKER_STOP) }
+      @workers.pop(excess)
+    end
+
+    # Workers are bound to the queue they were started on, never to @work_queue,
+    # so a worker outliving its generation cannot start draining the next one.
+    #
+    # A worker spawned from dispatch_loop takes its first delivery inside
+    # dispatch_loop's own turn, because Async runs a new task immediately rather
+    # than scheduling it. That is safe: a handler that waits - on a confirm, on
+    # an RPC reply - suspends, and suspending hands control back to
+    # dispatch_loop, so the reply still arrives. The two reentrancy examples in
+    # 57_consumer_backlog_spec.rb pin that. An earlier revision yielded here
+    # first; it was removed because no test could show it preventing anything.
+    def worker_loop(queue)
+      loop do
+        job = queue.dequeue
+        break if job.nil? || job.equal?(WORKER_STOP)
+
+        run_handler(*job)
+      end
+    end
+
+    def run_handler(method, header, body, entry)
+      started = instrument_clock
+      begin
+        entry[:block].call(method, header, body)
+      rescue *PROPAGATE_FROM_HANDLER
+        raise
+      rescue Exception => e # rubocop:disable Lint/RescueException
+        handle_consumer_error(e, method, entry)
+      ensure
+        if started
+          instrument("message.consumed") do
+            { channel: @channel_id, queue: entry[:queue_name], consumer_tag: method.consumer_tag,
+              bytes: body.to_s.bytesize, redelivered: method.redelivered,
+              duration: instrument_elapsed(started) }
+          end
+        end
+      end
+    end
+
     def dispatch_loop
       loop do
         msg = @queue.pop
         break if msg.nil?
         handle_message(msg)
       end
+    rescue *PROPAGATE_FROM_HANDLER
+      raise
     rescue => e
       @logger.error("Channel #{@channel_id} dispatch error: #{e.class}: #{e.message}")
+    rescue Exception => e # rubocop:disable Lint/RescueException
+      # Handlers are contained now, so nothing should reach here. If something
+      # does, this channel is about to stop delivering anything at all - no
+      # confirms, no RPC replies, no close-ok - and the one thing that must not
+      # happen is for it to go quiet without saying so.
+      @logger.error(
+        "Channel #{@channel_id} dispatch died on #{e.class}: #{e.message}. " \
+        "The channel will not deliver anything further."
+      )
+      raise
     end
 
     def handle_message(msg)
@@ -979,23 +1226,17 @@ module AsyncRabbitMQ
         if entry
           stamp_delivery_tag(method)
           @unsettled[method.delivery_tag.to_i] = true if entry[:manual_ack]
-          Async do
-            @pool_sem.acquire do
-              started = instrument_clock
-              begin
-                entry[:block].call(method, header, body)
-              rescue => e
-                handle_consumer_error(e, method, entry)
-              ensure
-                if started
-                  instrument("message.consumed") do
-                    { channel: @channel_id, queue: entry[:queue_name], consumer_tag: method.consumer_tag,
-                      bytes: body.to_s.bytesize, redelivered: method.redelivered,
-                      duration: instrument_elapsed(started) }
-                  end
-                end
-              end
-            end
+          # Hand off without suspending: Async::Queue wraps Thread::Queue,
+          # whose push never blocks. dispatch_loop must stay free, because it
+          # is also what delivers this channel's confirms, close-ok and
+          # returns - a handler that publishes and waits for confirms would
+          # otherwise deadlock against the loop that would have woken it. That
+          # is not hypothetical: it is rabbitmq-amqp-java-client#328, where a
+          # bounded work pool parked the I/O thread in LinkedBlockingQueue#put.
+          # A nil queue means the channel is going away and the delivery is moot.
+          if (queue = @work_queue)
+            queue.push([method, header, body, entry])
+            ensure_worker(queue)
           end
         else
           @logger.warn("Delivery on channel #{@channel_id} for unknown consumer #{method.consumer_tag}")
@@ -1119,6 +1360,7 @@ module AsyncRabbitMQ
       @reply_condition = nil
       # Stop dispatch_loop
       @queue&.push(nil)
+      stop_workers(discard: true)
       wake_each_waiters
     end
 

data/lib/async_rabbitmq/version.rb

@@ -1,3 +1,3 @@
 module AsyncRabbitMQ
-  VERSION = "0.4.0"
+  VERSION = "0.6.0"
 end

data/lib/async_rabbitmq.rb

@@ -1,3 +1,21 @@
+module AsyncRabbitMQ
+  class << self
+    # Whether a channel warns, once, when it starts a consumer whose backlog
+    # nothing bounds - an auto-ack consumer, or a manual-ack one with no
+    # prefetch. True by default.
+    #
+    # Set it false when you have read the warning and accepted the trade, so
+    # that silencing it does not mean turning the logger down and losing every
+    # other warning a channel raises (stale delivery tags, broker-cancelled
+    # consumers, handler exceptions). It is read when a consumer starts, so set
+    # it during boot, before the first #basic_consume.
+    #
+    #   AsyncRabbitMQ.warn_unbounded_consumers = false
+    attr_accessor :warn_unbounded_consumers
+  end
+  self.warn_unbounded_consumers = true
+end
+
 require_relative "async_rabbitmq/version"
 require_relative "async_rabbitmq/log"
 require_relative "async_rabbitmq/errors"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: async-rabbitmq
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.6.0
 platform: ruby
 authors:
 - Russell Penney