flu-rails 8.0.6 → 8.0.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 80a0fc1ea3123f0811eb936297c9aacc868e2b3f8e6d771a53dfd768901c6f83
-  data.tar.gz: 4764f353735e58b7e8a9183190fd104836caa15c7f99ed8df03a26bed5b21565
+  metadata.gz: bbc282f411276d4ad52ca02b624367edafd40b6396eafe57f6c4250c9c18da29
+  data.tar.gz: 28fcf8776ec3aa50318bfbed9d7779482be0f76cd3e51222c965c7ace9acb316
 SHA512:
-  metadata.gz: cfdf59bb4782cad462e48110c90084506141bb4d5b16ee39db055feafc5f925eb3cf705bd772038d1fb67bc4c5f9e40841d5a82959362c3c999adad3951d7175
-  data.tar.gz: 0f888571495893ff2e503b10133364bb5fe556aaf0f48105ab6a4c106b11234cbc5fbb5c59002224542674f73976da593b535d5e90a3485a3558e3fc95e9f9d1
+  metadata.gz: 6085eec4b84cb2a42eee67eebc5eb6752c21a8f1c889d59cb3ce215fa2e91efd7f13b31d816662fb96b3dc8ab30b79b20bf6cb44049ff964277793d2a71ea571
+  data.tar.gz: f6ef03cd47e961a3234e21e5d51bce950da55326e5df21d305c8080be3e31eb7356339b546b68daec73c05ef83ea72752526860d553cbe4ece04474e57c0c4a7

data/CHANGELOG.md

@@ -5,6 +5,32 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+### [8.0.9] - 2026-09-14
+
+* Enforce gem `json < 3`
+
+**Fixed**
+
+* Close the channel of a thread that has ended. Since 8.0.5, each thread publishes on its own Bunny channel, and that channel was never closed when the thread ended. RabbitMQ allows 2047 channels per connection, so a server that keeps creating and ending threads (Puma does, when `min_threads` is lower than `max_threads`) reached that limit after about a day. From then on, every new thread failed to publish with `RuntimeError: Cannot open a channel: max number of channels on connection reached`, while the older threads kept publishing normally, which made the problem look random. The channels of the threads that have ended are now closed each time a new channel is opened.
+* Publish from a fiber on the channel of its thread. The channel was stored in `Thread.current[]`, which is per fiber, not per thread: a publication made from an `Enumerator` or a streaming response opened one more channel, and never closed it either.
+* Publish on the channel Bunny reopens rather than on a new one. After a lost connection, Bunny reopens the connection first, then the channels it had on it. A thread that published in between opened one more channel, or hung for 15 seconds when Bunny was still in the handshake, and the one Bunny was reopening stayed open on the broker with no thread to use it. Publishing in between now raises `Flu::ConnectionLostError`, as it does while the connection is down, and goes on with the reopened channel once Bunny is done.
+
+### [8.0.8] - 2026-08-26
+
+**Fixed**
+
+* Keep the event publisher across code reloads. The railtie re-runs `init` on every reload, and `init` disconnected the publisher it replaced -- but a tracked model publishes through the very publisher it was tracked with, so every model the reload did not reload kept publishing on a connection that had just been closed, and nothing ever reopened it: `Flu::NotConnectedError` ("'connect' was never called, or 'disconnect' was") on every event from then on, until the process was restarted. The publisher now outlives the reloads, and only a change of kind (the real publisher for the dummy one, or the other way round) replaces and disconnects it.
+* Reopen a connection Bunny has stopped recovering. Bunny reopens a connection it lost, but not one that never opened, nor one whose recovery attempts ran out: `publish` only ever reconnected after a fork, so a publisher left with such a connection raised on every event for the rest of the process. It now reopens it on the next publication, and no more than once every five seconds, since a broker that hangs rather than refuses costs a full `connect_timeout` per attempt.
+
+**Changed**
+
+* `connect` no longer retries an unreachable broker forever. It gives up after `max_connect_wait` seconds and raises `Flu::ConnectionLostError`. The railtie calls it from `to_prepare`, which runs on every code reload: waiting on a broker that never answers held the reload interlock, and the request behind it, for good.
+* A broker that cannot be reached at startup no longer keeps the application from booting. `Flu.start` logs the failure, and publishing opens the connection once the broker answers again.
+
+**Added**
+
+* `max_connect_wait` (default `30`), how many seconds the startup retries a broker that does not answer before letting the application boot. `nil` waits for the broker for as long as it takes.
+
 ### [8.0.6] - 2026-08-19
 
 **Fixed**

data/README.md

@@ -16,7 +16,7 @@ For now, events are generated from:
 Add the gem to your project's Gemfile:
 
   ```ruby
-  gem "flu-rails", "8.0.6"
+  gem "flu-rails", "8.0.9"
   ```
 
 Then, create an initializer into your Rails app (`config/initializers/flu-rails.rb`)
@@ -50,7 +50,11 @@ Each configuration is detailed below.
 ### Start up
 
 `flu-rails` starts automatically through its `Railtie`: there is nothing to call by hand.
-Its startup waits until its RabbitMQ exchange is connected.
+Its startup waits for its RabbitMQ exchange to be connected, for at most `max_connect_wait` seconds. A broker that is still not there by then does not keep the application from booting:
+publishing reopens the connection itself once the broker answers again.
+
+It opens one connection per process, and one channel on it per thread that publishes. A fiber publishes on the channel of its thread.
+The channels of the threads that have ended are closed the next time a thread opens one, so a server that keeps creating and ending threads (Puma does, when `min_threads` is lower than `max_threads`) does not accumulate them.
 
 ### Track changes on an ActiveRecord model
 
@@ -170,6 +174,7 @@ All options have a default value. However, all of them can be changed in your in
 | `bunny_options` | `{}` | Hash of symbols | Optional | Additional options to add when connecting the RabbitMQ broker. This overrides the existing options with the same name. | `{ verify_peer: true }` |
 | `max_pending_events` | `1000` | Integer | Optional | An event the broker refused waits for the connection to be back, and is published by the next commit of the thread or at the end of the request. This is how many a thread keeps waiting before handing the oldest to `on_publication_failure`. They are held in memory: a process that dies takes them with it. | `5000` |
 | `on_publication_failure` | `nil` | Lambda | Optional | Called with the event and the error when an event cannot be published, instead of logging it. The transaction the event belongs to has already committed by then, so this is the last chance to keep it: store it and publish it again later. The event is `nil` when it could not even be built. | `lambda { |event, error| OutboxEvent.create!(payload: event&.to_json, error: error.message) }` |
+| `max_connect_wait` | `30` | Integer or `nil` | Optional | How many seconds the startup retries a broker that does not answer before giving up and letting the application boot. Publishing reopens the connection itself once the broker answers again. `nil` waits for the broker for as long as it takes. | `nil` |
 
 ## How to execute tests
 
@@ -315,8 +320,8 @@ scoped RubyGems credential.
 3. Tag the commit and push the tag:
 
 ```
-  $ git tag -a v8.0.6 -m "Version 8.0.6"
-  $ git push origin v8.0.6
+  $ git tag -a v8.0.9 -m "Version 8.0.9"
+  $ git push origin v8.0.9
 ```
 
 The workflow then checks that the tag matches `Flu::VERSION`, runs the tests, builds the gem

data/flu-rails.gemspec

@@ -28,6 +28,9 @@ Gem::Specification.new do |spec|
   spec.add_dependency "activerecord",             "~> 8.0"
   spec.add_dependency "activesupport",            "~> 8.0"
   spec.add_dependency "bunny",                    "~> 3.1"
+  # json 3 refuses the options Faraday 2.14.3 still passes to 'JSON.parse' as a hash.
+  # Fixed on Faraday's main branch, not released yet.
+  spec.add_dependency "json",                     "< 3"
   spec.add_dependency "logger",                   "~> 1.7"
   spec.add_dependency "rabbitmq_http_api_client", "~> 3.2"
 

data/lib/flu-rails/configuration.rb

@@ -20,6 +20,7 @@ module Flu
                   :application_name,
                   :bunny_options,
                   :on_publication_failure,
-                  :max_pending_events
+                  :max_pending_events,
+                  :max_connect_wait
   end
 end

data/lib/flu-rails/event_publisher.rb

@@ -13,11 +13,17 @@ module Flu
     CONNECTION_LOST_MESSAGE = "the connection to RabbitMQ is down. Bunny reopens it in the " \
                               "background when 'automatically_recover' is on, and publishing " \
                               "works again once it has."
+    CONNECTION_FAILED_MESSAGE = "could not reach RabbitMQ within %s seconds. Publishing reopens " \
+                                "the connection itself once the broker answers again."
+    RECONNECTION_INTERVAL = 5
 
     def initialize(configuration)
-      @logger        = configuration.logger
-      @configuration = configuration
-      @mutex         = Mutex.new
+      @logger          = configuration.logger
+      @configuration   = configuration
+      @mutex           = Mutex.new
+      @next_attempt_at = 0
+      @exchanges       = {}
+      @exchanges_mutex = Mutex.new
     end
 
     def publish(event, persistent=true)
@@ -29,19 +35,20 @@ module Flu
       raise ConnectionLostError, CONNECTION_LOST_MESSAGE
     end
 
+    # Retries a broker that is not there yet, for at most 'max_connect_wait' seconds. Waiting on it
+    # forever would hold whatever called it -- the railtie calls it from 'to_prepare', which runs on
+    # every code reload, holding the reload interlock and the request that triggered it.
     def connect
       @mutex.synchronize do
-        unless connected?
-          connected = false
-          while !connected
-            begin
-              connect_to_exchange
-              connected = true
-            rescue Bunny::TCPConnectionFailedForAllHosts
-              @logger.warn("RabbitMQ connection failed, try again in 1 second.")
-              sleep 1
-            end
-          end
+        next if connected?
+        give_up_at = deadline
+        begin
+          connect_to_exchange
+        rescue Bunny::TCPConnectionFailedForAllHosts
+          raise ConnectionLostError, format(CONNECTION_FAILED_MESSAGE, @configuration.max_connect_wait) if expired?(give_up_at)
+          @logger.warn("RabbitMQ connection failed, try again in 1 second.")
+          sleep 1
+          retry
         end
       end
     end
@@ -60,37 +67,130 @@ module Flu
         @connection.close if connected?
         @connection = nil
         @pid        = nil
-        Thread.current[exchange_key] = nil
+        forget_exchanges
       end
     end
 
     private
 
+    def now
+      Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    end
+
+    # A nil 'max_connect_wait' waits on the broker for however long it takes.
+    def deadline
+      @configuration.max_connect_wait.nil? ? nil : now + @configuration.max_connect_wait
+    end
+
+    def expired?(give_up_at)
+      !give_up_at.nil? && now >= give_up_at
+    end
+
+    # A connection Bunny is recovering comes back on its own. One that never opened, or that Bunny
+    # has given up on, comes back from here or not at all.
+    def abandoned?
+      return false if @connection.nil? || @connection.open?
+      !@connection.recovering_from_network_failure? &&
+        (!@connection.automatically_recover? ||
+         @connection.closed? ||
+         @connection.status == :not_connected)
+    end
+
+    # A broker that hangs rather than refuses costs a full Bunny 'connect_timeout' per attempt, so
+    # publishing pays for one at most every 'RECONNECTION_INTERVAL' seconds.
+    def due_for_another_attempt?
+      return false if now < @next_attempt_at
+      @next_attempt_at = now + RECONNECTION_INTERVAL
+      true
+    end
+
+    # Reopening is best effort: the caller is publishing, and an event that cannot go out is
+    # reported through the connection errors below rather than through whatever the broker refused.
+    def reconnect
+      @mutex.synchronize { connect_to_exchange unless connected? }
+    rescue StandardError => error
+      @logger.warn("Could not reopen the connection to RabbitMQ: #{error.class}: #{error.message}")
+    end
+
     # A child inherits the parent's socket but none of the threads Bunny runs on it.
     def forked?
       !@pid.nil? && @pid != Process.pid
     end
 
-    # Per process too: a channel cached before the fork still reports itself open in the child.
-    def exchange_key
-      :"flu_exchange_#{object_id}_#{Process.pid}"
-    end
-
-    # One channel per thread rather than one for the whole publisher. 
-    # Bunny serialises every publication on the channel's own mutex.
+    # One channel per thread, kept in '@exchanges' (thread => exchange).
+    #
+    # Why not 'Thread.current[]': it is per fiber, not per thread, and a channel stored there is
+    # never closed when the thread ends. RabbitMQ allows 2047 channels per connection, so a server
+    # that keeps creating and ending threads hit that limit after a day and could not publish
+    # from any new thread. Now, each time a channel is opened, the channels of the threads that
+    # have ended are closed.
+    #
+    # Until then, '@exchanges' still references the ended threads and what
+    # their thread-local variables hold: a weak reference would lose the channel before it is closed.
     #
-    # No bookkeeping of the channels handed out is needed. 
-    # Closing the connection closes all of them, so a thread holding a closed channel simply opens a new one on its next publication:
-    # 'disconnect' and a reconnection are both covered without reaching into other threads.
+    # A thread whose channel is closed opens a new one on its next publication.
+    # That is what makes 'disconnect' and a reconnection safe.
     # A connection that is down is reported as such rather than left to 'create_channel', which
     # raises a bare 'RuntimeError' the caller has no way to tell from any other.
+    # The connection is checked before the cached channel: Bunny marks the channels open before it
+    # announces it is done, so a thread that read its channel closed, then the announcement, would
+    # open one more.
     def exchange
-      connect if forked?
-      cached = Thread.current[exchange_key]
-      return cached if cached && cached.channel.open?
+      reconnect if forked? || (abandoned? && due_for_another_attempt?)
       raise NotConnectedError, NOT_CONNECTED_MESSAGE if @connection.nil?
       raise ConnectionLostError, CONNECTION_LOST_MESSAGE unless @connection.open?
-      Thread.current[exchange_key] = declare_exchange
+      raise ConnectionLostError, CONNECTION_LOST_MESSAGE if being_reopened?
+      cached = @exchanges_mutex.synchronize { @exchanges[Thread.current] }
+      return cached if cached && cached.channel.open?
+      remember_exchange(declare_exchange)
+    end
+
+    # Bunny reopens the connection first, then the channels it had on it. A channel opened in between
+    # stays open on the broker with no thread to use it, and the session does not show that window:
+    # 'open?' is true as soon as the socket is. Bunny announces each recovery attempt and its
+    # completion instead. The session is kept rather than a flag: a child process inherits the flag
+    # of a connection its parent was reopening, and opens one of its own.
+    # Bunny holds one callback of each: the ones the application gave in 'bunny_options' are called
+    # from here.
+    def hold_publishing_while_bunny_reopens(session, options)
+      started   = options[:recovery_attempt_started]
+      completed = options[:recovery_completed]
+      session.before_recovery_attempt_starts { @being_reopened = session; started&.call }
+      session.after_recovery_completed { @being_reopened = nil if @being_reopened.equal?(session); completed&.call }
+    end
+
+    def being_reopened?
+      @being_reopened&.equal?(@connection)
+    end
+
+    def remember_exchange(exchange)
+      ended = @exchanges_mutex.synchronize do
+        @exchanges[Thread.current] = exchange
+        @exchanges.keys.reject(&:alive?).map { |thread| @exchanges.delete(thread) }
+      end
+
+      unless ended.empty?
+        @logger.debug { "Closing the channels of #{ended.size} threads that have ended." }
+      end
+
+      ended.each { |orphan| close_channel(orphan.channel) }
+      exchange
+    end
+
+    # Forgets the channels without closing them.
+    # Their connection is gone: closed by 'disconnect',
+    # lost, or inherited from a parent process
+    # (closing that one would close the parent's socket).
+    def forget_exchanges
+      @exchanges_mutex.synchronize { @exchanges.clear }
+    end
+
+    # Closing may fail (the broker may have closed the channel already). That is fine: the thread
+    # is gone, and the publication in progress must not fail because of it.
+    def close_channel(channel)
+      channel.close if channel.open?
+    rescue StandardError => error
+      @logger.debug { "Could not close the channel of an ended thread: #{error.class}: #{error.message}" }
     end
 
     def declare_exchange
@@ -109,10 +209,14 @@ module Flu
         automatically_recover: true
       }.merge(@configuration.bunny_options || {})
 
+      # Before 'start': when it fails, the channels of the previous connection must be gone already,
+      # or the forking thread would publish on the parent's channel in a child that has no broker.
+      forget_exchanges
       @connection = Bunny.new(options)
+      hold_publishing_while_bunny_reopens(@connection, options)
       @connection.start
       @pid = Process.pid
-      Thread.current[exchange_key] = declare_exchange
+      remember_exchange(declare_exchange)
     end
   end
 end

data/lib/flu-rails/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Flu
-  VERSION = "8.0.6"
+  VERSION = "8.0.9"
 end

data/lib/flu-rails.rb

@@ -73,10 +73,6 @@ module Flu
     raise "configuration.application_name must not be nil" if @configuration.application_name.nil?
     @logger          = @configuration.logger
     @event_factory   = Flu::EventFactory.new(@configuration)
-    # The railtie re-runs 'init' on every code reload. The publisher being replaced owns an open
-    # connection, its channel and Bunny's heartbeat thread: dropping the reference to it without
-    # closing it leaks all three for the lifetime of the process.
-    @event_publisher&.disconnect
     @event_publisher = create_event_publisher(@configuration)
     extend_models_and_controllers
   end
@@ -90,14 +86,24 @@ module Flu
     end
   end
 
+  # The railtie re-runs 'init' on every code reload. Building a new publisher on each of them left
+  # every reference the application had already taken on the previous one -- a tracked model
+  # publishes through the very publisher it was tracked with -- on a connection 'init' had closed
+  # and that nothing ever reopens. The publisher outlives the reloads, and only a change of kind
+  # replaces it: its connection, its channels and Bunny's heartbeat thread are then closed with it.
   def self.create_event_publisher(configuration)
-    if is_testing_environment?
+    publisher_class = event_publisher_class
+    return @event_publisher if @event_publisher.instance_of?(publisher_class)
+
+    @event_publisher&.disconnect
+    if publisher_class == Flu::Dummy::InMemoryEventPublisher
       logger.info("Loading Flu with a dummy event publisher (this will not connect any exchange)")
-      require_relative "flu-rails/dummy/in_memory_event_publisher"
-      Flu::Dummy::InMemoryEventPublisher.new(@configuration)
-    else
-      Flu::EventPublisher.new(@configuration)
     end
+    publisher_class.new(configuration)
+  end
+
+  def self.event_publisher_class
+    is_testing_environment? ? Flu::Dummy::InMemoryEventPublisher : Flu::EventPublisher
   end
 
   def self.is_testing_environment?
@@ -112,8 +118,13 @@ module Flu
     Flu::CoreExt.extend_controller_classes(@event_factory, @event_publisher, @logger)
   end
 
+  # A broker that is not there yet must not keep the application from booting: publishing reopens
+  # the connection itself once the broker answers again.
   def self.start
-    @event_publisher.connect if config.auto_connect_to_exchange
+    return unless config.auto_connect_to_exchange
+    @event_publisher.connect
+  rescue Flu::ConnectionLostError => error
+    config.logger&.error("Flu could not connect to RabbitMQ: #{error.message}")
   end
 
   def self.load_configuration
@@ -137,6 +148,7 @@ module Flu
       config.bunny_options                  = {}
       config.on_publication_failure         = nil
       config.max_pending_events             = 1000
+      config.max_connect_wait               = 30
     end
   end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: flu-rails
 version: !ruby/object:Gem::Version
-  version: 8.0.6
+  version: 8.0.9
 platform: ruby
 authors:
 - Loïc Vigneron
@@ -68,6 +68,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '3.1'
+- !ruby/object:Gem::Dependency
+  name: json
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '3'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '3'
 - !ruby/object:Gem::Dependency
   name: logger
   requirement: !ruby/object:Gem::Requirement