async-rabbitmq 0.4.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c8e21899276061f5e0ecd749000e4b7bf5b81932a491a1704e0b2e643cf647b1
-  data.tar.gz: a7fcd5f434e5e677ad7fce840ca388e323264edd8b2347f8ed92ec73a0fd7a3e
+  metadata.gz: 63e32dac7aed43a3627da9fec4b8b2df543592f940f5301eddc4b388869f5728
+  data.tar.gz: '08d691a8d45ec1d1cb4162b6b8e874a3bd1bf52813c9ca9004d2439a0eaa2d67'
 SHA512:
-  metadata.gz: 0a41b23ef4e9d1fdad8403e8dbc99783f53205ed6b0fe5cb3e891b79d9763b9a9c0ec05bceda15f78de1fb47e0fb5b65bdbc5b756c2d19cf9c6609d2ee13fd3c
-  data.tar.gz: 4e7795488172face945464fbd3d36c9355b7b6500705238079df4ae38b0c6529c67088c84323354c0772bd31562b9efb7b1927c06872b369addc4d56638c5815
+  metadata.gz: 0c3b6732b01c0a6014bf6bd420ffd59e8f4093be2c253194d74accf67b57545d5655e5ac70856c18c4dd4c5697aed8e79e30282af00954fd9e482bc04d65d19f
+  data.tar.gz: 64dcd5ac758a50d502fc86d99809a7df1ddb64a19c34251282fc12ccf33ea657011cf86d51c35c1b86e59efd699205ef93e7bad97bf5fc01d1523609559dd50e

data/CHANGELOG.md

@@ -2,6 +2,51 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.5.0] - 2026-10-09
+
+### Fixed
+
+- **A consumer backlog no longer costs a fiber per queued message.** The dispatch loop started an
+  `Async::Task` per delivery and took the pool semaphore from inside it, so `pool_size` capped how
+  many handlers *ran*, not how many *existed*. A parked task measured ~15.7 KB (the fiber object
+  plus its touched stack pages) against ~48 bytes for a queue entry, and 20,000 backlogged messages
+  cost 315 MB of RSS. Handlers now run in `pool_size` long-lived worker fibers draining a queue of
+  deliveries — the same shape as Bunny's `ConsumerWorkPool`, the .NET client's consumer dispatcher
+  and amqp091-go.
+
+  That queue is deliberately unbounded. The dispatch loop is also what delivers this channel's
+  confirms, close-ok and returns, so blocking it on a full queue would deadlock a handler that
+  publishes and waits for confirms — which is precisely rabbitmq-amqp-java-client#328, where a
+  bounded work pool parked the I/O thread in `LinkedBlockingQueue#put`. Memory is therefore bounded
+  by prefetch on a manual-ack consumer and unbounded on an auto-ack one, as it is in every client,
+  because automatic acknowledgement has no backpressure to offer.
+- **The missing-prefetch warning no longer sends auto-ack consumers to a no-op.** RabbitMQ applies
+  prefetch only to *unacknowledged* messages, so `basic_qos` does nothing at all on a
+  `manual_ack: false` consumer. The warning told you to call it anyway, and then went quiet once you
+  had — silence in the one case where the caller most believes the consumer is bounded and it is
+  not. It now says which of the two cases you are in, and still warns an auto-ack consumer that has
+  set a prefetch.
+
+### Added
+
+- `Channel#backlog` — deliveries handed to the handler workers but not yet picked up, the number
+  Bunny reports as `ConsumerWorkPool#backlog`.
+- `AsyncRabbitMQ.warn_unbounded_consumers` — a deliberate way to silence the unbounded-consumer
+  warning for someone who has read it and accepted the trade, instead of turning the logger down
+  and losing every other warning a channel raises (stale delivery tags, broker-cancelled consumers,
+  handler exceptions). One process-wide setting, read when a consumer starts; defaults to `true`.
+
+### Changed
+
+- A backlog still queued when the channel goes away is now **finished on a graceful `close`** (as
+  Bunny's `ConsumerWorkPool#shutdown` drains) and **discarded when the connection is lost or the
+  channel is reopened**, where the broker requeues the same messages and running the queued handlers
+  too would process each one twice.
+- `Channel#pool_size=` resizes by adding or retiring workers rather than resizing a semaphore.
+  Growing takes effect at once; shrinking retires a worker as soon as the queue drains to the
+  sentinel — immediately when idle, after the current backlog when busy. A running handler is never
+  interrupted.
+
 ## [0.4.0] - 2026-10-07
 
 Reliability review against a payments workload. The two delivery-correctness items

data/README.md

@@ -1,8 +1,36 @@
 # async-rabbitmq
 
-A fiber-native RabbitMQ (AMQP 0-9-1) client for Ruby, built on the
+A fiber-native RabbitMQ (AMQP 0-9-1) client for Ruby, built on the amazing work
+of [Bunny](https://github.com/ruby-amqp/bunny).
+
+## Bunny is the gold standard
+
+Bunny is the client Ruby learned RabbitMQ on. Its API is the one Ruby developers
+already know, its defaults are the ones a decade of production traffic settled,
+and its source is the reference this library was written against. None of that
+is improved on here, and none of it is meant to be.
+
+**Parity with Bunny is a design goal**, chosen deliberately over fiber-native
+maximalism: the audience for this gem is people who already know Bunny, and
+surprising them is a bug. Where Bunny has an answer, that is the answer here.
+`pool_size` defaults to 1 and serialises handlers because `ConsumerWorkPool`
+does. A consumer backlog is an unbounded queue drained by a fixed set of workers
+because that is what `ConsumerWorkPool` is. `basic_get` acknowledges manually by
+default because Bunny's does. `spec/integration/25_bunny_api_parity_spec.rb`
+exists to keep it honest.
+
+Where this client does diverge, the README says so and says why - see the two
+publishing differences below, and `basic_qos`, which also resizes the handler
+pool where Bunny's leaves it alone.
+
+If you are not running under the fiber scheduler, use Bunny. It is excellent,
+and this gem has no advantage over it outside an async application.
+
+## What this adds
+
+The runtime, not the API. Built on the
 [async](https://github.com/socketry/async) ecosystem and
-[amq-protocol](https://github.com/ruby-amqp/amq-protocol). No threads: the
+[amq-protocol](https://github.com/ruby-amqp/amq-protocol), with no threads: the
 reader, writer, heartbeat and every consumer handler are fibers, so it fits
 Falcon, async-http and anything else running under the fiber scheduler.
 
@@ -140,11 +168,30 @@ ch.basic_cancel(tag)
 ch.each("q") { |delivery, header, body| ... }   # blocks until the consumer or channel goes away
 ```
 
-Consumer handlers run in their own fibers, at most `pool_size` at a time per
-channel. Set `basic_qos` before `basic_consume`: without a prefetch limit the
-broker sends the whole queue as fast as it can and one fiber is created per
-delivery, so memory tracks queue depth rather than concurrency. The client
-warns once per channel if you don't.
+Each channel runs `pool_size` long-lived handler fibers (default 1, serialized,
+Bunny parity) draining a queue of deliveries, so a backlog costs the messages it
+holds rather than a fiber per message. `ch.backlog` reports how many deliveries
+are waiting, as Bunny's `ConsumerWorkPool#backlog` does.
+
+What bounds that backlog depends on the ack mode. With `manual_ack: true`, set
+`basic_qos` before `basic_consume` and the broker holds everything past the
+prefetch window. With automatic acks there is no backpressure to be had:
+RabbitMQ applies prefetch only to *unacknowledged* messages, so the broker
+treats each one as acked on send and keeps sending whatever you set. The client
+warns once per channel in either case, and says which of the two you are in.
+
+If you have read that warning and accepted the trade - an auto-ack consumer on a
+queue you know stays shallow, say - switch it off during boot:
+
+```ruby
+AsyncRabbitMQ.warn_unbounded_consumers = false
+```
+
+It is deliberately one process-wide setting rather than a per-channel option: an
+application either accepts unbounded consumers or it does not. The point of
+having it at all is that the alternative is turning the logger down, which would
+also lose stale delivery tags, broker-cancelled consumers and handler
+exceptions.
 
 ### Acknowledging, and delivery tags across a reconnect
 
@@ -282,8 +329,7 @@ end
 ```
 
 `Async::Condition#wait`, a `sleep`, or your own supervisor task work equally
-well. Versions up to 0.3.0 kept the reactor alive by accident, because those
-tasks were children of whichever task called `connect`.
+well.
 
 ## Connection recovery
 

data/lib/async_rabbitmq/channel.rb

@@ -24,6 +24,10 @@ module AsyncRabbitMQ
     # after an RPC timeout, used when the session has no rpc_timeout set.
     RESYNC_TIMEOUT = 5
 
+    # Pushed onto the work queue to retire one handler worker. Compared by
+    # identity, so it can never collide with a delivery.
+    WORKER_STOP = Object.new.freeze
+
     include Instrumented
 
     attr_reader :channel_id, :pool_size
@@ -89,14 +93,27 @@ module AsyncRabbitMQ
       @pending_content   = nil
       @recovered_condition = nil     # fibers parked while the connection recovers
       @pool_size         = validate_pool_size!(pool_size)
-      @pool_sem          = Async::Semaphore.new(@pool_size)
+      @work_queue        = nil       # Async::Queue of [method, header, body, entry]
+      @workers           = []        # long-lived handler fibers draining it
     end
 
-    # Resize the consumer-handler concurrency cap. Shrinking does not evict
-    # already-running handlers; new deliveries park until permits free up.
+    # Resize the consumer-handler concurrency cap by adding or retiring
+    # workers. Growing takes effect at once. Shrinking is a sentinel on the
+    # work queue, so it retires a worker as soon as the queue drains to it:
+    # immediately when idle, after the current backlog when busy. A handler
+    # already running is never interrupted.
     def pool_size=(n)
-      @pool_size      = validate_pool_size!(n)
-      @pool_sem.limit = @pool_size
+      n = validate_pool_size!(n)
+      return if n == @pool_size
+
+      @pool_size = n
+      resize_workers
+    end
+
+    # Deliveries handed to the handler workers but not yet picked up. Bunny
+    # reports the same number as ConsumerWorkPool#backlog.
+    def backlog
+      @work_queue&.size || 0
     end
 
     def open?
@@ -147,6 +164,7 @@ module AsyncRabbitMQ
       instrument("channel.closed") { { channel: @channel_id, reason: :user } }
       @session.channel_closed(@channel_id)
       @queue&.push(nil)  # wake dispatch_loop so it can detect :closed and exit
+      stop_workers
       wake_each_waiters
     end
 
@@ -409,7 +427,7 @@ module AsyncRabbitMQ
     # handlers run concurrently across all consumers on this channel.
     # Returns the consumer tag.
     def basic_consume(queue_name, consumer_tag: "", manual_ack: false, exclusive: false, arguments: {}, &block)
-      warn_unbounded_prefetch(queue_name)
+      warn_unbounded_prefetch(queue_name, manual_ack)
       resp = rpc(
         AMQ::Protocol::Basic::Consume.encode(@channel_id, queue_name, consumer_tag, false, !manual_ack, exclusive, false, arguments),
         AMQ::Protocol::Basic::ConsumeOk
@@ -667,6 +685,9 @@ module AsyncRabbitMQ
       @confirm_condition = nil
       release_outstanding_slots(error)
       @queue&.push(nil) rescue nil
+      # The connection is gone, so the queued deliveries are too: the broker
+      # requeues whatever was unacknowledged and sends it again on the new one.
+      stop_workers(discard: true)
     rescue => e
       # ignore — best-effort unblock
     end
@@ -832,20 +853,34 @@ module AsyncRabbitMQ
       true
     end
 
-    # Without a prefetch limit the broker sends as fast as it can and the
-    # dispatch loop starts a task per delivery. pool_size caps how many run at
-    # once, not how many exist, so memory grows with the queue depth rather
-    # than with the concurrency. Warned once per channel; basic_qos fixes it.
-    def warn_unbounded_prefetch(queue_name)
+    # Without a bound the broker sends as fast as it can and the backlog is
+    # held in this process. What can bound it depends on the ack mode, so the
+    # advice does too: RabbitMQ applies prefetch only to *unacknowledged*
+    # messages, so on an auto-ack consumer basic_qos never engages and telling
+    # the user to call it would be wrong. Warned once per channel.
+    def warn_unbounded_prefetch(queue_name, manual_ack)
+      return unless AsyncRabbitMQ.warn_unbounded_consumers
       return if @qos_warned
-      return if @prefetch && @prefetch[:count].to_i > 0
+      # A prefetch answers the manual-ack case, so having set one is a reason to
+      # stay quiet there. It does nothing for an auto-ack consumer, so it is not
+      # a reason here: that is precisely the case where the caller believes the
+      # consumer is bounded and it is not.
+      return if manual_ack && @prefetch && @prefetch[:count].to_i > 0
 
       @qos_warned = true
       @logger.warn(
-        "Channel #{@channel_id}: consuming from #{queue_name} without basic_qos. " \
-        "The broker will send the whole queue as fast as it can and one task is " \
-        "created per delivery, so memory tracks queue depth. Call " \
-        "basic_qos(prefetch_count: n) before basic_consume."
+        if manual_ack
+          "Channel #{@channel_id}: consuming from #{queue_name} without basic_qos. " \
+          "The broker will send the whole queue as fast as it can and the backlog is " \
+          "held in this process. Call basic_qos(prefetch_count: n) before " \
+          "basic_consume to bound it."
+        else
+          "Channel #{@channel_id}: consuming from #{queue_name} with automatic acks. " \
+          "The broker treats every message as acknowledged on send, so basic_qos " \
+          "cannot bound this consumer and the whole queue is held in this process. " \
+          "Use manual_ack: true with basic_qos(prefetch_count: n) to get " \
+          "backpressure, or watch #backlog."
+        end
       )
     end
 
@@ -904,9 +939,80 @@ module AsyncRabbitMQ
     # -------------------------------------------------------------------------
 
     def start_dispatch_task
+      start_workers
       @session.spawn_background { dispatch_loop }
     end
 
+    # One long-lived fiber per pool_size, each draining the work queue for the
+    # life of this channel generation. A backlogged delivery costs a 4-element
+    # array (~48 bytes) rather than a parked Async::Task and its fiber stack
+    # (~15.7 KB measured), so memory tracks the messages held, not the fibers
+    # holding them. Bunny's ConsumerWorkPool is the same shape: fixed workers
+    # draining an unbounded ::Queue.
+    def start_workers
+      stop_workers(discard: true)
+      queue    = @work_queue = Async::Queue.new
+      @workers = Array.new(@pool_size) { @session.spawn_background { worker_loop(queue) } }
+    end
+
+    # Retire this generation's workers. The sentinel goes on the back of the
+    # queue, so by default they finish the deliveries already queued before
+    # standing down - Bunny's ConsumerWorkPool#shutdown drains the same way.
+    #
+    # +discard+ throws that backlog away instead, for when the connection
+    # underneath it has gone: the broker requeues whatever was unacknowledged
+    # and sends it again on the new one, so running these handlers too would
+    # process every queued message twice. dequeue(timeout: 0) returns at once
+    # on an empty queue, so the drain cannot suspend us mid-teardown.
+    def stop_workers(discard: false)
+      queue = @work_queue
+      nil while discard && queue&.dequeue(timeout: 0)
+      @workers.size.times { queue&.push(WORKER_STOP) }
+      @workers    = []
+      @work_queue = nil
+    end
+
+    # Workers are bound to the queue they were started on, never to @work_queue,
+    # so a worker outliving its generation cannot start draining the next one.
+    def resize_workers
+      queue = @work_queue
+      return unless queue
+
+      delta = @pool_size - @workers.size
+      if delta.positive?
+        delta.times { @workers << @session.spawn_background { worker_loop(queue) } }
+      elsif delta.negative?
+        (-delta).times { queue.push(WORKER_STOP) }
+        @workers.pop(-delta)
+      end
+    end
+
+    def worker_loop(queue)
+      loop do
+        job = queue.dequeue
+        break if job.nil? || job.equal?(WORKER_STOP)
+
+        run_handler(*job)
+      end
+    end
+
+    def run_handler(method, header, body, entry)
+      started = instrument_clock
+      begin
+        entry[:block].call(method, header, body)
+      rescue => e
+        handle_consumer_error(e, method, entry)
+      ensure
+        if started
+          instrument("message.consumed") do
+            { channel: @channel_id, queue: entry[:queue_name], consumer_tag: method.consumer_tag,
+              bytes: body.to_s.bytesize, redelivered: method.redelivered,
+              duration: instrument_elapsed(started) }
+          end
+        end
+      end
+    end
+
     def dispatch_loop
       loop do
         msg = @queue.pop
@@ -979,24 +1085,15 @@ module AsyncRabbitMQ
         if entry
           stamp_delivery_tag(method)
           @unsettled[method.delivery_tag.to_i] = true if entry[:manual_ack]
-          Async do
-            @pool_sem.acquire do
-              started = instrument_clock
-              begin
-                entry[:block].call(method, header, body)
-              rescue => e
-                handle_consumer_error(e, method, entry)
-              ensure
-                if started
-                  instrument("message.consumed") do
-                    { channel: @channel_id, queue: entry[:queue_name], consumer_tag: method.consumer_tag,
-                      bytes: body.to_s.bytesize, redelivered: method.redelivered,
-                      duration: instrument_elapsed(started) }
-                  end
-                end
-              end
-            end
-          end
+          # Hand off without suspending: Async::Queue wraps Thread::Queue,
+          # whose push never blocks. dispatch_loop must stay free, because it
+          # is also what delivers this channel's confirms, close-ok and
+          # returns - a handler that publishes and waits for confirms would
+          # otherwise deadlock against the loop that would have woken it. That
+          # is not hypothetical: it is rabbitmq-amqp-java-client#328, where a
+          # bounded work pool parked the I/O thread in LinkedBlockingQueue#put.
+          # A nil queue means the channel is going away and the delivery is moot.
+          @work_queue&.push([method, header, body, entry])
         else
           @logger.warn("Delivery on channel #{@channel_id} for unknown consumer #{method.consumer_tag}")
         end
@@ -1119,6 +1216,7 @@ module AsyncRabbitMQ
       @reply_condition = nil
       # Stop dispatch_loop
       @queue&.push(nil)
+      stop_workers(discard: true)
       wake_each_waiters
     end
 

data/lib/async_rabbitmq/version.rb

@@ -1,3 +1,3 @@
 module AsyncRabbitMQ
-  VERSION = "0.4.0"
+  VERSION = "0.5.0"
 end

data/lib/async_rabbitmq.rb

@@ -1,3 +1,21 @@
+module AsyncRabbitMQ
+  class << self
+    # Whether a channel warns, once, when it starts a consumer whose backlog
+    # nothing bounds - an auto-ack consumer, or a manual-ack one with no
+    # prefetch. True by default.
+    #
+    # Set it false when you have read the warning and accepted the trade, so
+    # that silencing it does not mean turning the logger down and losing every
+    # other warning a channel raises (stale delivery tags, broker-cancelled
+    # consumers, handler exceptions). It is read when a consumer starts, so set
+    # it during boot, before the first #basic_consume.
+    #
+    #   AsyncRabbitMQ.warn_unbounded_consumers = false
+    attr_accessor :warn_unbounded_consumers
+  end
+  self.warn_unbounded_consumers = true
+end
+
 require_relative "async_rabbitmq/version"
 require_relative "async_rabbitmq/log"
 require_relative "async_rabbitmq/errors"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: async-rabbitmq
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.5.0
 platform: ruby
 authors:
 - Russell Penney