dionysus-rb 1.2.0 → 1.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 715bda096eaf87ea615134206607a11d22ff3d673b5cbe9625c584599fb5696b
-  data.tar.gz: d4c5741f509d02f0c5a28a4332f34b08b28f46ca1c7d026d59f59bff2e37638f
+  metadata.gz: 8cd74dab3609a68e4919adb2913c5e2d428b2bd044ec785988eae123d125677b
+  data.tar.gz: 15ae10c89ce12d63f0d452bf8ebfb835a8b3593784048c003a28a093a9c77250
 SHA512:
-  metadata.gz: c101ce0cfa0f3ddb58339bdf28b3b479838fb3b66107d8223d4a795888e5048cc8f044db232fe7ae03899cf558d1b8100a8119119d8a8d3faac3996082671618
-  data.tar.gz: be3c63dddbc4b1c3ae58134b050370c62a0940d3227ca3e1820cce72a1bba36cb932de2906d42a7fe723f811ac30dbc4431f668c5832787fd0a27fb521fc5594
+  metadata.gz: f3a2a0e0179804ff474dc9bb7575ba565e3c4377f6755cfb31aed5c783b7a3915966af0158a50132155a18c588e44495540d47a7463e475d07260f9db4cea6ef
+  data.tar.gz: 2e4450ae8743746067d81ab3c28614c4293fec570fdb550a9240b44f4e8ed097d7f3ecc110e03f51135c8fcd8237835e0a5a3ca90f754009b9146bd2e83e4866

data/CHANGELOG.md

@@ -1,5 +1,28 @@
 ## [Unreleased]
 
+## [1.4.0]
+
+### Fixed
+
+- Re-serialize a record whose row moved while its payload was being built, behind `config.publish_consistent_snapshots` (default `false`), so the payload describes a single moment.
+- The defect, reproduced against a real broker in `spec/torn_payload_reproduction_spec.rb`. A serializer reads a record's own columns and then queries its associations. Anything committed in between lands in the payload beside a timestamp taken before it, so the payload's `updated_at` can predate the records embedded next to it. The reproduction publishes from a record read fresh moments earlier and still emits a payload tens of milliseconds adrift carrying an association row the timestamp predates - no stale object, no query cache, no clock skew and no replica involved.
+- Why that is a lost record rather than a stale field. Consumers rank duplicates and then compare the payload's `updated_at` against what they have stored. A payload whose timestamp predates its own contents loses that comparison and is skipped by `next`, and the has_many records embedded in it are persisted further down the same loop body, so they are dropped with it: the parent keeps its old values and the child rows are never created, with no error and no counter.
+- Measured on the sibling gem's affected topic in production: serialization spans 95 ms at the median and up to 3.5 s, which is ample time for a write to arrive; 141 of 1,476 records receiving more than one message showed `updated_at` moving backwards, and in 69 of those the message that wins deduplication reported a timestamp older than one already published at a lower offset.
+- The check reads the row's timestamp back with the query cache bypassed. Publishing runs inside the request when `publish_after_commit` is on, where the cache is live on that connection, and a cached read would report that nothing had moved and hand back the very payload the check exists to catch.
+- Retries are bounded by `config.max_snapshot_attempts` (default `Producer::MAX_SNAPSHOT_ATTEMPTS`, 3) and the last attempt is published rather than dropped - a payload that may still be torn beats no message at all. Records without an `updated_at`, and anything that is not an `ActiveRecord::Base`, are serialized once as before.
+- The retry is the expensive half, and its cost runs opposite to its benefit. Measured over 5 events on one record with the serializer held at the production median of 95 ms: a quiet record costs nothing beyond one indexed read per message, but a record written every 40 ms pins to the ceiling on every event - 15 serializations instead of 5, and 1.5 s instead of 0.5 s - and still publishes an attempt that may be torn. The hottest records are both the most expensive to guard and the least likely to converge, so `max_snapshot_attempts` is configurable to bound it.
+- Every guarded message increments `dionysus.publish.consistent_snapshot` exactly once, tagged `result:consistent|exhausted|unsupported`, `attempts:N`, `topic` and `model`. `exhausted` is the case worth alerting on: the payload is published with no marker and consumers see an ordinary message, so the counter is the only place that outcome is ever visible. Nothing is emitted while the feature is off, so an empty series reads as "not running" rather than "running clean".
+
+## [1.3.0]
+
+### Fixed
+
+- Read the record and its associations without the query cache while publishing, behind `config.publish_with_uncached_reads` (default `false`). The payload is built from the record plus the associations declared with `with:`. When `publish_after_commit` is on, publishing runs inside the request, where Rails' query cache is live on that connection - so a row read earlier in the request is served from the cache while its associations are queried fresh. The payload then carries an `updated_at` older than the data it contains.
+- Measured in production on the sibling gem over a two and a half hour window on one topic: of 1,476 records that received more than one message, 141 had a payload whose `updated_at` moved backwards as `serialized_at` moved forwards, and in 69 of those the message that wins deduplication carried an `updated_at` lower than one already published at a lower offset. One record published a payload serialized at `09:27:23.662` reporting `updated_at 09:27:22.139557`, when a payload serialized 1.3 seconds earlier had already reported `09:27:22.345092` - a value a row's `updated_at` cannot return to.
+- The consequence is not a stale field but a discarded record. Consumers rank duplicates and then apply `persist_with_dionysus?`, which compares the payload's `updated_at` against what is stored; a payload that loses that comparison is skipped by `next`, and the `has_many` records embedded in it are persisted further down the same loop body, so they are dropped with it. The parent keeps its old values and the new child rows are never created, with no error, no counter and nothing to self-heal it.
+- This is why no ordering over the fields already in the message could fix both known incidents: `updated_at` was in the message all along and is simply wrong. `serialized_at`, added in 1.2.0, correctly identifies the last-serialized message - which is precisely the one the guard rejects when its `updated_at` is stale.
+- Rolling this out needs no consumer changes: the fix is entirely on the producer, and the payload's shape is unchanged.
+
 ## [1.2.0]
 
 ### Fixed

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    dionysus-rb (1.2.0)
+    dionysus-rb (1.4.0)
       activerecord (>= 5)
       activesupport (>= 3.2)
       concurrent-ruby

data/README.md

@@ -337,10 +337,24 @@ Dionysus::Producer.configure do |config|
   config.outbox_worker_publishing_delay = 5 # non required, defaults to 0 a delay in seconds until the outbox record is considered publishable. Check `Publishing records right after the transaction is committed` section for more details.
   config.remove_consecutive_duplicates_before_publishing = true # not required, defaults to false. If set to true, the consecutive duplicates in the publishable batch will be removed and only one message will be published to a given topic. For example, if for whatever reason there are ten messages in a row for a given topic to publish `user_updated` ecent, only the last will be published. Check `Dionysus::Consumer::ParamsBatchTransformations::RemoveDuplicatesStrategy` for exact implementation. To verify if this feature is useful, it's recommended to browse Karafka UI and check messages in the topics if there are any obvious duplicates happening often.
   config.observers_inline_maximum_size = 100 # not required, defaults to 1000. This config setting matters in case there is a huge amount of dependent records (observers). If the threshold is exceeded, the observers will be published via Genesis process to not cause issues like blocking the outbox worker.
+  config.publish_consistent_snapshots = true # not required, defaults to `false`. A serializer reads a record's own columns and then queries its associations, so a write landing in between produces a payload whose `updated_at` predates the records embedded next to it - consumers rank and guard on that timestamp, so such a payload is discarded along with its embedded associations. When enabled, the row's timestamp is read back (bypassing the query cache) after serializing and the payload is re-serialized if the record moved. Emits the `dionysus.publish.consistent_snapshot` counter described below.
+  config.max_snapshot_attempts = 2 # not required, defaults to 3. Only relevant with `publish_consistent_snapshots` enabled. A record written faster than it serializes fails the check on every attempt, so it pays the full serialization cost this many times over and still publishes a payload that may be torn - lower this to bound that cost. Set it to 1 to keep the check and its metric while disabling re-serialization entirely.
   config.sidekiq_queue = :default # required, defaults to :dionysus. The queue will be used for a genesis process
 end
 ```
 
+##### Monitoring consistent snapshots
+
+With `publish_consistent_snapshots` enabled, every guarded message increments `dionysus.publish.consistent_snapshot` exactly once, tagged with `result`, `attempts`, `topic` and `model`. Nothing is emitted when the feature is off, so an empty series means the guard is not running rather than running clean.
+
+| `result` | meaning |
+|---|---|
+| `consistent` | the record had not moved by the time the payload was built. With `attempts:1` the first try was already clean; higher values mean it converged after re-serializing. |
+| `exhausted` | the record was still moving at `max_snapshot_attempts`. The payload is published anyway - a payload that may be torn beats no message - but it carries no marker and consumers see an ordinary message, so this counter is the only place the outcome is visible. |
+| `unsupported` | the records carry no `updated_at`, so the guard could not run on that message at all. |
+
+Watch the `exhausted` rate before and after a rollout: the retry cost scales with write rate multiplied by serialization span, so the hottest records are both the most expensive to guard and the least likely to converge.
+
 ##### DionysusOutbox model
 
 Generate a model for the outbox:

data/lib/dionysus/producer/config.rb

@@ -9,7 +9,8 @@ class Dionysus::Producer::Config
     :transactional_outbox_enabled, :sidekiq_queue, :publisher_service_name,
     :genesis_consistency_safety_delay, :hermes_event_producer, :publish_after_commit, :outbox_worker_publishing_delay,
     :high_priority_sidekiq_queue, :observers_inline_maximum_size, :remove_consecutive_duplicates_before_publishing,
-    :include_serialized_at_in_payload
+    :include_serialized_at_in_payload, :publish_with_uncached_reads, :publish_consistent_snapshots,
+    :max_snapshot_attempts
 
   def self.default_sidekiq_queue
     :dionysus
@@ -97,6 +98,32 @@ class Dionysus::Producer::Config
     @observers_inline_maximum_size || 1000
   end
 
+  # Off by default so it can be switched on per producer, and switched back off in one env change
+  # if the extra reads ever cost more than they are worth.
+  # How many times a payload may be re-serialized before the last attempt is published as it stands.
+  # This is the expensive half of publish_consistent_snapshots: a record written faster than it
+  # serializes fails the check on every attempt, so it pays the full serialization cost this many
+  # times over and still publishes a payload that may be torn. Lower it to bound that cost; 1
+  # disables re-serialization while leaving the check (and its metric) in place.
+  def max_snapshot_attempts
+    @max_snapshot_attempts || Dionysus::Producer::MAX_SNAPSHOT_ATTEMPTS
+  end
+
+  def publish_with_uncached_reads
+    return @publish_with_uncached_reads if defined?(@publish_with_uncached_reads)
+
+    false
+  end
+
+  # Re-serialize when a record moved while its payload was being built, so the payload describes a
+  # single moment. Off by default; the extra read costs one indexed column per message, and the
+  # re-serialization only happens on the records that were actually contended.
+  def publish_consistent_snapshots
+    return @publish_consistent_snapshots if defined?(@publish_consistent_snapshots)
+
+    false
+  end
+
   # Off by default so consumers, which fall back to the offset when the field is absent, can be
   # rolled out first.
   def include_serialized_at_in_payload

data/lib/dionysus/producer/karafka_responder_generator.rb

@@ -49,7 +49,7 @@ class Dionysus::Producer::KarafkaResponderGenerator
               # the offset alone is publish order, and a message published later can carry an
               # earlier snapshot, so consumers need to know when this payload was actually read
               serialized_at = config.include_serialized_at_in_payload ? Time.now.utc : nil
-              payload = serialize_to_payload(records, topic, batch_options)
+              payload = serialize_consistently(records, topic, batch_options)
 
               event_payload = {
                 event: event,
@@ -75,6 +75,73 @@ class Dionysus::Producer::KarafkaResponderGenerator
 
       private
 
+      # A serializer reads a record's own columns and then queries its associations, so anything
+      # committed in between lands in the payload beside a timestamp taken before it - the payload
+      # describes no single moment. Consumers rank on that timestamp and then compare it against
+      # what they have stored, so a payload whose timestamp predates its own contents loses the
+      # comparison and is discarded, taking the has_many records embedded in it along with it.
+      #
+      # Measured on the affected topic in production: serialization spans 95ms at the median and up
+      # to 3.5s, and 4.7% of records receiving more than one message had the surviving message
+      # report a timestamp older than one already published.
+      #
+      # So serialize, then check whether the records moved underneath it. If they did, the payload
+      # is not a snapshot of anything: reload and serialize again. Retries are bounded, and the last
+      # attempt is published rather than dropped - a payload that may be torn still beats no message.
+      define_method :serialize_consistently do |records, current_topic, batch_options|
+        next serialize_to_payload(records, current_topic, batch_options) unless config.publish_consistent_snapshots
+
+        max_attempts = config.max_snapshot_attempts
+        attempts = 0
+        loop do
+          before = snapshot_of(records)
+          payload = serialize_to_payload(records, current_topic, batch_options)
+          attempts += 1
+
+          # nothing to compare against, so the guard did not run on this message at all
+          break instrument_snapshot("unsupported", attempts, records, current_topic, payload) if before.nil?
+          if before == committed_snapshot_of(records)
+            break instrument_snapshot("consistent", attempts, records, current_topic, payload)
+          end
+          if attempts >= max_attempts
+            # published anyway: a payload that may be torn beats no message. Nothing downstream can
+            # tell this apart from a clean one - the payload carries no marker and the consumer sees
+            # an ordinary message - so this counter is the only place the outcome is ever visible.
+            break instrument_snapshot("exhausted", attempts, records, current_topic, payload)
+          end
+
+          records.each { |record| record.reload if record.is_a?(ActiveRecord::Base) && record.persisted? }
+        end
+      end
+
+      # One counter rather than several: the denominator, the retry distribution and the failure
+      # rate all have to come from the same series or none of them can be read as a rate.
+      define_method :instrument_snapshot do |result, attempts, records, current_topic, payload|
+        config.instrumenter.increment("dionysus.publish.consistent_snapshot",
+          tags: ["result:#{result}", "attempts:#{attempts}", "topic:#{current_topic}",
+            "model:#{records.first.class}"])
+        payload
+      end
+
+      # nil means there is nothing to compare - a record without timestamps, or not a record at all
+      define_method :snapshot_of do |records|
+        stamps = records.map { |record| record.updated_at if record.respond_to?(:updated_at) }
+        stamps.any?(&:nil?) ? nil : stamps
+      end
+
+      define_method :committed_snapshot_of do |records|
+        records.map do |record|
+          next record.updated_at unless record.is_a?(ActiveRecord::Base) && record.persisted?
+
+          # publishing runs inside the request when publish_after_commit is on, where the query
+          # cache is live on this connection - a cached read here would report that nothing moved
+          # and hand back the torn payload the check exists to catch
+          record.class.uncached do
+            record.class.where(record.class.primary_key => record.id).pick(:updated_at)
+          end
+        end
+      end
+
       define_method :serialize_to_payload do |records, current_topic, batch_options|
         if batch_options.to_h[:serialize] == false
           records.map(&:as_json)

data/lib/dionysus/producer/outbox/publisher.rb

@@ -11,52 +11,56 @@ class Dionysus::Producer::Outbox::Publisher
   def publish(outbox_record, options = {})
     return if Dionysus::Producer::Suppressor.suppressed?
 
-    instrument("publishing_with_dionysus") do
-      resource_class = outbox_record.resource_class.constantize
-      primary_key = resource_class.primary_key
-      primary_key_value = outbox_record.resource_id
-      topic = outbox_record.topic
-      resource = resource_class.find_by(primary_key => primary_key_value) ||
-        resource_class.new(primary_key => primary_key_value)
-      event_name = outbox_record.event_name
-      if resource.new_record? && outbox_record.created_event?
-        logger.error(
-          "Attempted to publish #{resource.class}, id: #{resource.id} but it was deleted, that should never happen!"
-        )
-        return
-      end
+    with_consistent_reads do
+      instrument("publishing_with_dionysus") do
+        resource_class = outbox_record.resource_class.constantize
+        primary_key = resource_class.primary_key
+        primary_key_value = outbox_record.resource_id
+        topic = outbox_record.topic
+        resource = resource_class.find_by(primary_key => primary_key_value) ||
+          resource_class.new(primary_key => primary_key_value)
+        event_name = outbox_record.event_name
+        if resource.new_record? && outbox_record.created_event?
+          logger.error(
+            "Attempted to publish #{resource.class}, id: #{resource.id} but it was deleted, that should never happen!"
+          )
+          return
+        end
 
-      if resource.new_record? && outbox_record.updated_event?
-        logger.error(
-          "There was an update of #{resource.class}, id: #{resource.id} but it was deleted, that should never happen!"
-        )
-        return
-      end
+        if resource.new_record? && outbox_record.updated_event?
+          logger.error(
+            "There was an update of #{resource.class}, id: #{resource.id} but it was deleted, that should never happen!"
+          )
+          return
+        end
 
-      published_count = publish_for_top_level_resource(outbox_record, resource, event_name, topic, options) +
-        publish_for_dependency(resource, topic, options)
-      handle_nothing_published(resource, event_name, topic) if published_count.zero?
-      published_count
+        published_count = publish_for_top_level_resource(outbox_record, resource, event_name, topic, options) +
+          publish_for_dependency(resource, topic, options)
+        handle_nothing_published(resource, event_name, topic) if published_count.zero?
+        published_count
+      end
     end
   end
 
   def publish_observers(outbox_record)
     return if Dionysus::Producer::Suppressor.suppressed?
 
-    instrument("publishing_observers_with_dionysus") do
-      resource_class = outbox_record.resource_class.constantize
-      primary_key = resource_class.primary_key
-      primary_key_value = outbox_record.resource_id
-      resource = resource_class.find_by(primary_key => primary_key_value) ||
-        resource_class.new(primary_key => primary_key_value)
-      changeset = outbox_record.transformed_changeset
-
-      Dionysus::Producer.observers_with_responders_for(resource,
-        changeset).each do |observers, responder|
-        if observers.count > config.observers_inline_maximum_size
-          execute_genesis_for_observers(observers, responder)
-        else
-          observers.each { |observer_record| publish_observer(observer_record, responder) }
+    with_consistent_reads do
+      instrument("publishing_observers_with_dionysus") do
+        resource_class = outbox_record.resource_class.constantize
+        primary_key = resource_class.primary_key
+        primary_key_value = outbox_record.resource_id
+        resource = resource_class.find_by(primary_key => primary_key_value) ||
+          resource_class.new(primary_key => primary_key_value)
+        changeset = outbox_record.transformed_changeset
+
+        Dionysus::Producer.observers_with_responders_for(resource,
+          changeset).each do |observers, responder|
+          if observers.count > config.observers_inline_maximum_size
+            execute_genesis_for_observers(observers, responder)
+          else
+            observers.each { |observer_record| publish_observer(observer_record, responder) }
+          end
         end
       end
     end
@@ -64,6 +68,18 @@ class Dionysus::Producer::Outbox::Publisher
 
   private
 
+  # The payload is built from the record and its associations. Publishing runs inside the request
+  # when publish_after_commit is on, where Rails' query cache is live on that connection, so a row
+  # read earlier in the request can be served from the cache while its associations are queried
+  # fresh - producing a payload whose updated_at is older than the data it carries. Consumers rank
+  # and guard on updated_at, so such a payload is silently discarded, taking its embedded
+  # associations with it.
+  def with_consistent_reads(&)
+    return yield unless config.publish_with_uncached_reads
+
+    ActiveRecord::Base.uncached(&)
+  end
+
   delegate :instrumenter, :error_handler, to: :config
   delegate :instrument, to: :instrumenter
   delegate :logger, to: Dionysus

data/lib/dionysus/producer.rb

@@ -1,6 +1,11 @@
 # frozen_string_literal: true
 
 class Dionysus::Producer
+  # a contended record can move again while it is being re-serialized; bound the work and publish
+  # the last attempt rather than looping or dropping the message. Overridable per producer through
+  # config.max_snapshot_attempts - this is only the default.
+  MAX_SNAPSHOT_ATTEMPTS = 3
+
   def self.configuration
     @configuration ||= Dionysus::Producer::Config.new
   end

data/lib/dionysus/version.rb

@@ -3,5 +3,5 @@
 module Dionysus
   module Version
   end
-  VERSION = "1.2.0"
+  VERSION = "1.4.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: dionysus-rb
 version: !ruby/object:Gem::Version
-  version: 1.2.0
+  version: 1.4.0
 platform: ruby
 authors:
 - Karol Galanciak
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-08-26 00:00:00.000000000 Z
+date: 2026-08-28 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord