flu-rails 8.0.5 → 8.0.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 24d179705111c94185f7421297a692e1f962feeef2a9c2132a04e744d70a6503
-  data.tar.gz: 93526441dcfaea7cc3a5be4795c5e86cdb9a4acdbaf95b825e530259b3c89d64
+  metadata.gz: 80a0fc1ea3123f0811eb936297c9aacc868e2b3f8e6d771a53dfd768901c6f83
+  data.tar.gz: 4764f353735e58b7e8a9183190fd104836caa15c7f99ed8df03a26bed5b21565
 SHA512:
-  metadata.gz: bd84b67b67c9f209fbab6c5309d719719e3d9f19df12e1d7816acc6690c084dad052e8590cc39b85cfcc9477b6c187d1ec352a3a8e55a932c5f51d22c062e458
-  data.tar.gz: e7e0eab197a600c06a10c1bb6ec801499cfe3d55327e94232b565830b5ee6fa69c4a5bf7c19ab3082f0492917d83e6c36581ce53600d5cbf9ef0259df1b620b1
+  metadata.gz: cfdf59bb4782cad462e48110c90084506141bb4d5b16ee39db055feafc5f925eb3cf705bd772038d1fb67bc4c5f9e40841d5a82959362c3c999adad3951d7175
+  data.tar.gz: 0f888571495893ff2e503b10133364bb5fe556aaf0f48105ab6a4c106b11234cbc5fbb5c59002224542674f73976da593b535d5e90a3485a3558e3fc95e9f9d1

data/CHANGELOG.md

@@ -5,6 +5,26 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+### [8.0.6] - 2026-08-19
+
+**Fixed**
+
+* Publish the events of a transaction from its commit rather than from each record's `after_commit`, which used to lose events two separate ways: Rails runs the transactional callbacks of a row on one single instance of it and skips the others (which one it picks is what `run_commit_callbacks_on_first_saved_instances_in_transaction` decides, and neither value was safe), and it skips every callback still queued as soon as one of them raises. Events now go out from the commit itself, which nothing can skip, once per recorded change and in the order the records were saved. A non-joinable transaction, such as the one `use_transactional_tests` wraps an example in, is never waited on, so a test suite sees what production does.
+* Keep an event whose publication failed for another attempt rather than giving up on it there and then. It waits in memory, per thread, up to `max_pending_events`, and is published by the next transaction to commit on that thread or at the end of the request or the job, whichever comes first. Nothing is attempted while the publisher reports itself unreachable, and an event still refused after three attempts, or pushed out of a full buffer, is handed to `on_publication_failure`.
+* Build and publish every change on its own, so an event that cannot be built or cannot reach the broker no longer costs the events queued behind it. A failed publication does not fail the transaction it belongs to either -- it has already committed by then.
+* Reconnect to RabbitMQ after a fork. A forked child inherited the parent's `Bunny` connection and published on it, which the broker then ended for both. `EventPublisher` now remembers the pid it connected under, caches channels per process as well as per thread, and drops an inherited connection instead of closing it.
+* Serialize `connect` and `disconnect` on a mutex. Two threads reaching `connect` together both opened a connection, the second overwriting the first, which stayed open and unreachable.
+
+**Added**
+
+* `Flu::ConnectionLostError`, raised by `publish` while the connection to RabbitMQ is down.
+* `on_publication_failure`, called with the event and the error when an event cannot be published, so that an application can keep it rather than read about it in the logs. The event is `nil` when it could not even be built.
+* `max_pending_events` (default `1000`), how many events a thread keeps waiting for the broker to be reachable again before the oldest are handed to `on_publication_failure`.
+
+**Changed**
+
+* Publishing while the connection is down now raises `Flu::ConnectionLostError` instead of the bare `RuntimeError` `Bunny::Session#create_channel` raises ("this connection is not open"), which nothing could tell apart from any other `RuntimeError`. It still fails immediately rather than waiting: Bunny reopens the connection in the background, and publishing works again once it has.
+
 ### [8.0.5] - 2026-08-03
 
 **Fixed**

data/README.md

@@ -16,7 +16,7 @@ For now, events are generated from:
 Add the gem to your project's Gemfile:
 
   ```ruby
-  gem "flu-rails", "8.0.5"
+  gem "flu-rails", "8.0.6"
   ```
 
 Then, create an initializer into your Rails app (`config/initializers/flu-rails.rb`)
@@ -168,6 +168,8 @@ All options have a default value. However, all of them can be changed in your in
 | `default_ignored_request_params` | `[:password, :password_confirmation, :controller, :action]` | Boolean | Optional | By default, all these parameters will be ignored from controller request's `params` when creating an event. Independently of this option, any parameter your Rails application already masks through `config.filter_parameters` (passwords, tokens,...) is replaced with `"[FILTERED]"` in the event too, including inside nested params. | `false` |
 | `application_name` | `Rails.application.class.module_parent_name`, resolved on startup | String | Required | Is used as `emitter` for each event created by `flu-rails`, if not overriden by the `track_met`. | `my_app` |
 | `bunny_options` | `{}` | Hash of symbols | Optional | Additional options to add when connecting the RabbitMQ broker. This overrides the existing options with the same name. | `{ verify_peer: true }` |
+| `max_pending_events` | `1000` | Integer | Optional | An event the broker refused waits for the connection to be back, and is published by the next commit of the thread or at the end of the request. This is how many a thread keeps waiting before handing the oldest to `on_publication_failure`. They are held in memory: a process that dies takes them with it. | `5000` |
+| `on_publication_failure` | `nil` | Lambda | Optional | Called with the event and the error when an event cannot be published, instead of logging it. The transaction the event belongs to has already committed by then, so this is the last chance to keep it: store it and publish it again later. The event is `nil` when it could not even be built. | `lambda { |event, error| OutboxEvent.create!(payload: event&.to_json, error: error.message) }` |
 
 ## How to execute tests
 
@@ -313,8 +315,8 @@ scoped RubyGems credential.
 3. Tag the commit and push the tag:
 
 ```
-  $ git tag -a v8.0.5 -m "Version 8.0.5"
-  $ git push origin v8.0.5
+  $ git tag -a v8.0.6 -m "Version 8.0.6"
+  $ git push origin v8.0.6
 ```
 
 The workflow then checks that the tag matches `Flu::VERSION`, runs the tests, builds the gem

data/lib/flu-rails/active_record_extender.rb

@@ -1,16 +1,23 @@
 # frozen_string_literal: true
 
 require "active_support/core_ext/object/try"
+require "active_support/notifications"
 
 module Flu
   class ActiveRecordExtender
     def self.extend_models(event_factory, event_publisher)
+      publish_on_transaction_commit
+
       ActiveRecord::Base.class_eval do
         unless singleton_class.method_defined?(:flu_is_tracked)
           class_attribute :flu_is_tracked,               instance_accessor: false, default: false
           class_attribute :flu_user_metadata_lambdas,    instance_accessor: false, default: {}.freeze
           class_attribute :flu_ignored_model_changes,    instance_accessor: false, default: [].freeze
           class_attribute :flu_overriden_emitter_lambda, instance_accessor: false, default: nil
+          # Held per model rather than looked up on 'Flu', so that the publication driven by the
+          # transaction reaches the very publisher the model was tracked with.
+          class_attribute :flu_event_factory,            instance_accessor: false, default: nil
+          class_attribute :flu_event_publisher,          instance_accessor: false, default: nil
         end
 
         define_singleton_method(:track_entity_changes) do |options = {}|
@@ -18,6 +25,8 @@ module Flu
           self.flu_user_metadata_lambdas    = options.fetch(:user_metadata, {})
           self.flu_ignored_model_changes    = options.fetch(:ignored_model_changes, []).map(&:to_s)
           self.flu_overriden_emitter_lambda = options.fetch(:emitter, nil)
+          self.flu_event_factory            = event_factory
+          self.flu_event_publisher          = event_publisher
         
           after_create   { flu_track_entity_change(:create, saved_changes, event_factory) }
           after_update   { flu_track_entity_change(:update, saved_changes, event_factory) }
@@ -46,7 +55,8 @@ module Flu
         # and used 'run_callbacks(:commit)' instead, which runs *every* 'after_commit' callback on the record, 
         # including the hoste application's own (mailers, jobs, cache invalidation), not just Flu's.
         define_method(:flu_publish_events!) do
-          flu_commit_changes(event_factory, event_publisher)
+          flu_commit_changes(self.class.flu_event_factory   || event_factory,
+                             self.class.flu_event_publisher || event_publisher)
         end
 
         def flu_add_manual_event(name, data)
@@ -56,24 +66,37 @@ module Flu
             data: data,
             flu_is_a_manual_event: true
           })
+          Flu::TransactionBuffer.current.record(self) if self.class.flu_is_tracked
         end
 
         def flu_changes_as_events(event_factory)
-          flu_changes.select do |data|
-            !data[:changes].try(:empty?) || data[:flu_is_a_manual_event]
-          end.map do |data|
-            if data[:flu_is_a_manual_event]
-              event_factory.build_manual_event(data[:name], data[:data])
-            else
-              event_factory.build_entity_change_event(data)
-            end
+          flu_publishable_changes.map { |change| flu_change_as_event(change, event_factory) }
+        end
+
+        def flu_publishable_changes
+          flu_changes.select do |change|
+            !change[:changes].try(:empty?) || change[:flu_is_a_manual_event]
+          end
+        end
+
+        def flu_change_as_event(change, event_factory)
+          if change[:flu_is_a_manual_event]
+            event_factory.build_manual_event(change[:name], change[:data])
+          else
+            event_factory.build_entity_change_event(change)
           end
         end
 
+        # Every change is built and published on its own: one that cannot be is reported and the next
+        # one goes out all the same, where a raise would take the whole rest of the batch with it.
         def flu_commit_changes(event_factory, event_publisher)
-          flu_changes_as_events(event_factory).each do |event|
+          flu_publishable_changes.each do |change|
+            event = flu_change_as_event(change, event_factory)
             event_publisher.publish(event)
+          rescue StandardError => error
+            Flu.publication_failed(event, error, event_publisher)
           end
+        ensure
           flu_flush_changes
         end
 
@@ -98,10 +121,49 @@ module Flu
                                                                                            self.class.flu_association_columns,
                                                                                            self.class.flu_ignored_model_changes,
                                                                                            self.class.flu_overriden_emitter_lambda)
-            flu_changes.push(data) unless data.nil?
+            return if data.nil?
+            flu_changes.push(data)
+            Flu::TransactionBuffer.current.record(self)
           end
         end
       end
     end
+
+    # The commit of the transaction, unlike the 'after_commit' callbacks that follow it, is a moment
+    # Rails cannot skip: the notification is emitted once the COMMIT is through and before the first
+    # callback runs, so no callback raising afterwards can cost anybody their events.
+    def self.publish_on_transaction_commit
+      unless @subscribed
+        @subscribed = true
+
+        ActiveSupport::Notifications.subscribe("start_transaction.active_record") do |*, payload|
+          TransactionBuffer.current.transaction_started if joinable?(payload)
+        end
+
+        ActiveSupport::Notifications.subscribe("transaction.active_record") do |*, payload|
+          next unless joinable?(payload)
+
+          buffer = TransactionBuffer.current
+          if payload[:outcome] == :commit
+            buffer.transaction_committed.each do |entity|
+              # Nothing may travel from here into the commit that is calling us.
+              entity.flu_commit_changes(entity.class.flu_event_factory, entity.class.flu_event_publisher)
+            rescue StandardError => error
+              Flu.config.logger&.error("Flu could not build the events of #{entity.class}: " \
+                                      "#{error.class}: #{error.message}")
+            end
+            Flu.retry_pending_publications
+          else
+            buffer.transaction_rolled_back.each(&:flu_rollback_changes)
+          end
+        end
+      end
+    end
+
+    # A transaction opened as non-joinable, such as the one 'use_transactional_tests' wraps an
+    # example in, is rolled back rather than committed and only its savepoints publish anything.
+    def self.joinable?(payload)
+      !payload[:transaction].equal?(ActiveRecord::Transaction::NULL_TRANSACTION)
+    end
   end
 end

data/lib/flu-rails/configuration.rb

@@ -18,6 +18,8 @@ module Flu
                   :default_ignored_model_changes,
                   :default_ignored_request_params,
                   :application_name,
-                  :bunny_options
+                  :bunny_options,
+                  :on_publication_failure,
+                  :max_pending_events
   end
 end

data/lib/flu-rails/errors.rb

@@ -6,4 +6,7 @@ module Flu
 
   class NotConnectedError < Error
   end
+
+  class ConnectionLostError < Error
+  end
 end

data/lib/flu-rails/event_publisher.rb

@@ -10,11 +10,14 @@ module Flu
     NOT_CONNECTED_MESSAGE = "no connection to RabbitMQ: 'connect' was never called, or " \
                             "'disconnect' was. The railtie calls it at boot unless " \
                             "'auto_connect_to_exchange' is false."
+    CONNECTION_LOST_MESSAGE = "the connection to RabbitMQ is down. Bunny reopens it in the " \
+                              "background when 'automatically_recover' is on, and publishing " \
+                              "works again once it has."
 
     def initialize(configuration)
       @logger        = configuration.logger
       @configuration = configuration
-      @exchange_key  = :"flu_exchange_#{object_id}" # Channels are cached per thread
+      @mutex         = Mutex.new
     end
 
     def publish(event, persistent=true)
@@ -22,54 +25,72 @@ module Flu
       @logger.debug { "Publishing event with id '#{event.id}' with routing key: #{routing_key}" }
       exchange.publish(event.to_json, routing_key: routing_key, persistent: persistent)
       @logger.debug { "Event published." }
+    rescue Bunny::ConnectionClosedError
+      raise ConnectionLostError, CONNECTION_LOST_MESSAGE
     end
 
     def connect
-      unless connected?
-        connected = false
-        while !connected
-          begin
-            connect_to_exchange
-            connected = true
-          rescue Bunny::TCPConnectionFailedForAllHosts
-            @logger.warn("RabbitMQ connection failed, try again in 1 second.")
-            sleep 1
+      @mutex.synchronize do
+        unless connected?
+          connected = false
+          while !connected
+            begin
+              connect_to_exchange
+              connected = true
+            rescue Bunny::TCPConnectionFailedForAllHosts
+              @logger.warn("RabbitMQ connection failed, try again in 1 second.")
+              sleep 1
+            end
           end
         end
       end
     end
 
     def connected?
-      !@connection.nil? && @connection.open?
+      !forked? && !@connection.nil? && @connection.open?
     end
 
     # Closing the connection closes every channel opened on it, and stops the heartbeat and
     # recovery threads Bunny runs alongside it.
     # The guard is on the connection alone: a connection that was opened before the exchange could
     # be declared still has to be closed.
+    # An inherited connection is dropped rather than closed: its socket is the parent's.
     def disconnect
-      if !@connection.nil? && @connection.open?
-        @connection.close
+      @mutex.synchronize do
+        @connection.close if connected?
+        @connection = nil
+        @pid        = nil
+        Thread.current[exchange_key] = nil
       end
-      @connection = nil
-      Thread.current[@exchange_key] = nil
     end
 
     private
 
+    # A child inherits the parent's socket but none of the threads Bunny runs on it.
+    def forked?
+      !@pid.nil? && @pid != Process.pid
+    end
+
+    # Per process too: a channel cached before the fork still reports itself open in the child.
+    def exchange_key
+      :"flu_exchange_#{object_id}_#{Process.pid}"
+    end
+
     # One channel per thread rather than one for the whole publisher. 
     # Bunny serialises every publication on the channel's own mutex.
     #
     # No bookkeeping of the channels handed out is needed. 
     # Closing the connection closes all of them, so a thread holding a closed channel simply opens a new one on its next publication:
     # 'disconnect' and a reconnection are both covered without reaching into other threads.
-    # Only the absence of a connection is reported here. A connection that exists but is closed or
-    # recovering is Bunny's story to tell, and its own error says more about it than this one could.
+    # A connection that is down is reported as such rather than left to 'create_channel', which
+    # raises a bare 'RuntimeError' the caller has no way to tell from any other.
     def exchange
-      cached = Thread.current[@exchange_key]
+      connect if forked?
+      cached = Thread.current[exchange_key]
       return cached if cached && cached.channel.open?
       raise NotConnectedError, NOT_CONNECTED_MESSAGE if @connection.nil?
-      Thread.current[@exchange_key] = declare_exchange
+      raise ConnectionLostError, CONNECTION_LOST_MESSAGE unless @connection.open?
+      Thread.current[exchange_key] = declare_exchange
     end
 
     def declare_exchange
@@ -90,7 +111,8 @@ module Flu
 
       @connection = Bunny.new(options)
       @connection.start
-      Thread.current[@exchange_key] = declare_exchange
+      @pid = Process.pid
+      Thread.current[exchange_key] = declare_exchange
     end
   end
 end

data/lib/flu-rails/pending_publications.rb

@@ -0,0 +1,67 @@
+# frozen_string_literal: true
+
+module Flu
+  # The events whose publication failed, kept for another attempt.
+  #
+  # A broker that closes a connection takes a few seconds to be usable again, since Bunny reopens it
+  # in the background, and an event published in that window has nowhere to go. Rather than being
+  # given up on there, it waits here for the connection to be back: the next transaction to commit on
+  # the thread comes back for it, and so does the end of the request or the job it belongs to.
+  #
+  # In memory, and per thread: what is waiting here dies with the process. An application that cannot
+  # afford to lose an event keeps it itself, from 'on_publication_failure', which is called for every
+  # event this gives up on.
+  class PendingPublications
+    MAX_ATTEMPTS = 3
+
+    def self.current
+      Thread.current[:flu_pending_publications] ||= new
+    end
+
+    def initialize
+      @entries = []
+    end
+
+    def size
+      @entries.size
+    end
+
+    def push(event, publisher, error)
+      give_up(@entries.shift) while @entries.size >= Flu.config.max_pending_events
+      @entries.push({ event: event, publisher: publisher, error: error, attempts: 1 })
+    end
+
+    # Publishes again what its publisher can reach again, keeps what it cannot, and gives up on what
+    # has been refused MAX_ATTEMPTS times, an event the broker itself rejects being no more publishable
+    # on the tenth attempt than on the first.
+    def drain
+      return if @entries.empty?
+
+      kept = []
+      @entries.each do |entry|
+        next kept.push(entry) unless reachable?(entry[:publisher])
+
+        begin
+          entry[:publisher].publish(entry[:event])
+        rescue StandardError => error
+          entry[:error]     = error
+          entry[:attempts] += 1
+          entry[:attempts] < MAX_ATTEMPTS ? kept.push(entry) : give_up(entry)
+        end
+      end
+      @entries = kept
+    end
+
+    private
+
+    def reachable?(publisher)
+      publisher.connected?
+    rescue StandardError
+      false
+    end
+
+    def give_up(entry)
+      Flu.report_publication_failure(entry[:event], entry[:error])
+    end
+  end
+end

data/lib/flu-rails/railtie.rb

@@ -15,5 +15,11 @@ module Flu
       Flu.init
       Flu.start
     end
+
+    # The end of a request or of a job, which runs even when it is an exception that ended it, is the
+    # last chance to publish what the broker could not take a moment earlier.
+    initializer "flu.retry_pending_publications" do |application|
+      application.executor.to_complete { Flu.retry_pending_publications }
+    end
   end
 end

data/lib/flu-rails/transaction_buffer.rb

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+module Flu
+  # The entities that recorded events in the transaction currently open, so that its commit can
+  # publish them.
+  #
+  # A record hands its events over from 'after_commit', and Rails skips every transactional callback
+  # still to run as soon as one of them raises: it re-commits the rest of the batch with
+  # 'should_run_callbacks: false' (ActiveRecord::ConnectionAdapters::Transaction#commit_records). One
+  # raising callback, in this gem or in the application, therefore drops the events of every record
+  # left in the queue. The commit of the transaction itself is the one moment nothing can skip, and
+  # it is where these are published instead.
+  #
+  # One buffer per thread, one mark per open transaction: what a transaction recorded is what was
+  # pushed past its mark, which is what its rollback discards and what the outermost commit hands
+  # back. A thread holding transactions open on several databases at once shares that one stack, so
+  # the events of the first to commit wait for the last.
+  class TransactionBuffer
+    def self.current
+      Thread.current[:flu_transaction_buffer] ||= new
+    end
+
+    def initialize
+      @entities = []
+      @marks    = []
+    end
+
+    # @return [Boolean] false when no transaction is open, in which case nothing will ever drain the
+    #   buffer and the entity is left to publish its own changes.
+    def record(entity)
+      if @marks.empty?
+        false
+      else
+        @entities.push(entity)
+        true
+      end
+    end
+
+    def transaction_started
+      @marks.push(@entities.size)
+    end
+
+    # @return [Array] the entities to publish: those of the outermost transaction, none otherwise,
+    #   a nested transaction leaving what it recorded to the one it is nested in.
+    def transaction_committed
+      @marks.pop
+      if @marks.empty?
+        committed = @entities
+        @entities = []
+        committed
+      else
+        []
+      end
+    end
+
+    # @return [Array] the entities whose events the rollback discards.
+    def transaction_rolled_back
+      @entities.slice!((@marks.pop || 0)..) || []
+    end
+  end
+end

data/lib/flu-rails/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Flu
-  VERSION = "8.0.5"
+  VERSION = "8.0.6"
 end

data/lib/flu-rails.rb

@@ -10,6 +10,8 @@ require_relative "flu-rails/event_factory"
 require_relative "flu-rails/queue_repository"
 require_relative "flu-rails/configuration"
 require_relative "flu-rails/core_ext"
+require_relative "flu-rails/transaction_buffer"
+require_relative "flu-rails/pending_publications"
 require_relative "flu-rails/event_publisher"
 require_relative "flu-rails/util"
 require_relative "flu-rails/dummy/in_memory_event_publisher"
@@ -36,6 +38,36 @@ module Flu
     @event_publisher
   end
 
+  # Keeps an event whose publication failed for another attempt, a broker being reachable again
+  # within seconds. One that could not even be built is reported at once instead.
+  def self.publication_failed(event, error, event_publisher)
+    if event.nil?
+      report_publication_failure(event, error) 
+    else
+      PendingPublications.current.push(event, event_publisher, error)
+    end
+  end
+
+  # Publishes again what the last attempt could not. Never raises: its callers are a transaction that
+  # has committed and the end of a request, neither of which is a place to fail.
+  def self.retry_pending_publications
+    PendingPublications.current.drain
+  rescue StandardError => error
+    config.logger&.error("Flu could not retry the publications it had kept: #{error.class}: #{error.message}")
+  end
+
+  # @param event [Flu::Event, nil] nil when the event could not even be built.
+  def self.report_publication_failure(event, error)
+    handler = config.on_publication_failure
+    if handler.nil?
+      subject = event.nil? ? "an event it could not build" : "the event '#{event.id}' ('#{event.name}')"
+      config.logger&.error("Flu could not publish #{subject}: #{error.class}: #{error.message}. " \
+                           "The event is lost unless 'on_publication_failure' is configured to keep it.")
+    else
+      handler.call(event, error) 
+    end
+  end
+
   def self.init
     @configuration.application_name ||= default_application_name
     raise "configuration.application_name must not be nil" if @configuration.application_name.nil?
@@ -103,6 +135,8 @@ module Flu
       config.default_ignored_request_params = [:password, :password_confirmation, :controller, :action]
       config.application_name               = nil
       config.bunny_options                  = {}
+      config.on_publication_failure         = nil
+      config.max_pending_events             = 1000
     end
   end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: flu-rails
 version: !ruby/object:Gem::Version
-  version: 8.0.5
+  version: 8.0.6
 platform: ruby
 authors:
 - Loïc Vigneron
@@ -203,8 +203,10 @@ files:
 - lib/flu-rails/event.rb
 - lib/flu-rails/event_factory.rb
 - lib/flu-rails/event_publisher.rb
+- lib/flu-rails/pending_publications.rb
 - lib/flu-rails/queue_repository.rb
 - lib/flu-rails/railtie.rb
+- lib/flu-rails/transaction_buffer.rb
 - lib/flu-rails/util.rb
 - lib/flu-rails/version.rb
 homepage: https://github.com/crepesourcing/flu-rails