dionysus-rb 1.3.0 → 1.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 968407d443681c04bfc7426b8925d7f63db69a09763357413d01218c03ec3ad8
-  data.tar.gz: baf5e739a77bc2d48ab5197591ddb59c6d970bdaed68479014d5afbec5698f0e
+  metadata.gz: aea1780da49737ea39a412112502ccefec1921bcaa8e4d9e3f4a17c01e359cb1
+  data.tar.gz: 0de793797246fa1aa63209eb9713529df3bf3b51aedca2d03be485ab3b140f43
 SHA512:
-  metadata.gz: 121b9c6af2b90700a7af1aba098282cfe05df901c633dfcc71d7845bda935b5e124b21c8bedab20895e9c5dd11a3dd12c2edd049c1bcd9b984c7df0da8a246b5
-  data.tar.gz: 6594b49acf6f64bb59475bc4a8a3a70c5747a09dcb72d3bc815e544bf3016bf0df732f3b15f0b3b7c8106578ba3c8ed49f4206d5b206e240285d376b6e08b31f
+  metadata.gz: 9ba62b20fdf34088c43f28bfd71e4fa721b4d512125ca030c16c61eb077354aab5751a05c1ce0192708a7fc87ccd3d3af0a63b8a2f06a48ab903858c928d4ce8
+  data.tar.gz: d67700b960048a4ea6b272d20d193ba4232115665a5a72d0b2171cec6ba089c3bb451ab36fd5092ffbec55a763fca95fa7b8b7299b3f44263122e58aea26932f

data/CHANGELOG.md

@@ -1,5 +1,24 @@
 ## [Unreleased]
 
+## [1.4.1]
+- Stamp `serialized_at` when the attempt that is actually published begins, not before the retry loop. 1.4.0 takes the stamp once and then re-serializes up to `max_snapshot_attempts` times, so a message can be published a whole retry cycle after its own stamp - and the payload it publishes is the one read last, not the one the stamp describes. Consumers rank duplicates by `[serialized_at, offset]`, so the freshest payload ended up carrying the oldest stamp and losing its group to a staler sibling. That is the failure `serialized_at` was added to prevent, reintroduced by the guard added to prevent a different one.
+- Measured on the affected topic over 65 minutes: 205 of 6,572 messages (3.12%) carried an `updated_at` LATER than their own `serialized_at`, the worst by 1,225 ms. A timestamp cannot describe a moment after the one it was taken at, so that alone proves the stamp preceded the payload.
+- `serialize_consistently` now returns `[payload, read_at]`. The looseness between stamp and payload is bounded by a single serialization pass again, as it was before 1.4.0, rather than by the whole retry cycle - so contended records, which retry most, are no longer the ones stamped most wrongly.
+- Covered end to end against a real broker in `spec/serialized_at_tracks_published_attempt_spec.rb`: a contended publish retries while a competing publish serializes once and stamps later, both are read back off the topic, and the real `RemoveDuplicatesStrategy` is asked which survives. The spec fails on 1.4.0 on both assertions.
+
+## [1.4.0]
+
+### Fixed
+
+- Re-serialize a record whose row moved while its payload was being built, behind `config.publish_consistent_snapshots` (default `false`), so the payload describes a single moment.
+- The defect, reproduced against a real broker in `spec/torn_payload_reproduction_spec.rb`. A serializer reads a record's own columns and then queries its associations. Anything committed in between lands in the payload beside a timestamp taken before it, so the payload's `updated_at` can predate the records embedded next to it. The reproduction publishes from a record read fresh moments earlier and still emits a payload tens of milliseconds adrift carrying an association row the timestamp predates - no stale object, no query cache, no clock skew and no replica involved.
+- Why that is a lost record rather than a stale field. Consumers rank duplicates and then compare the payload's `updated_at` against what they have stored. A payload whose timestamp predates its own contents loses that comparison and is skipped by `next`, and the has_many records embedded in it are persisted further down the same loop body, so they are dropped with it: the parent keeps its old values and the child rows are never created, with no error and no counter.
+- Measured on the sibling gem's affected topic in production: serialization spans 95 ms at the median and up to 3.5 s, which is ample time for a write to arrive; 141 of 1,476 records receiving more than one message showed `updated_at` moving backwards, and in 69 of those the message that wins deduplication reported a timestamp older than one already published at a lower offset.
+- The check reads the row's timestamp back with the query cache bypassed. Publishing runs inside the request when `publish_after_commit` is on, where the cache is live on that connection, and a cached read would report that nothing had moved and hand back the very payload the check exists to catch.
+- Retries are bounded by `config.max_snapshot_attempts` (default `Producer::MAX_SNAPSHOT_ATTEMPTS`, 3) and the last attempt is published rather than dropped - a payload that may still be torn beats no message at all. Records without an `updated_at`, and anything that is not an `ActiveRecord::Base`, are serialized once as before.
+- The retry is the expensive half, and its cost runs opposite to its benefit. Measured over 5 events on one record with the serializer held at the production median of 95 ms: a quiet record costs nothing beyond one indexed read per message, but a record written every 40 ms pins to the ceiling on every event - 15 serializations instead of 5, and 1.5 s instead of 0.5 s - and still publishes an attempt that may be torn. The hottest records are both the most expensive to guard and the least likely to converge, so `max_snapshot_attempts` is configurable to bound it.
+- Every guarded message increments `dionysus.publish.consistent_snapshot` exactly once, tagged `result:consistent|exhausted|unsupported`, `attempts:N`, `topic` and `model`. `exhausted` is the case worth alerting on: the payload is published with no marker and consumers see an ordinary message, so the counter is the only place that outcome is ever visible. Nothing is emitted while the feature is off, so an empty series reads as "not running" rather than "running clean".
+
 ## [1.3.0]
 
 ### Fixed

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    dionysus-rb (1.3.0)
+    dionysus-rb (1.4.1)
       activerecord (>= 5)
       activesupport (>= 3.2)
       concurrent-ruby

data/README.md

@@ -337,10 +337,24 @@ Dionysus::Producer.configure do |config|
   config.outbox_worker_publishing_delay = 5 # non required, defaults to 0 a delay in seconds until the outbox record is considered publishable. Check `Publishing records right after the transaction is committed` section for more details.
   config.remove_consecutive_duplicates_before_publishing = true # not required, defaults to false. If set to true, the consecutive duplicates in the publishable batch will be removed and only one message will be published to a given topic. For example, if for whatever reason there are ten messages in a row for a given topic to publish `user_updated` ecent, only the last will be published. Check `Dionysus::Consumer::ParamsBatchTransformations::RemoveDuplicatesStrategy` for exact implementation. To verify if this feature is useful, it's recommended to browse Karafka UI and check messages in the topics if there are any obvious duplicates happening often.
   config.observers_inline_maximum_size = 100 # not required, defaults to 1000. This config setting matters in case there is a huge amount of dependent records (observers). If the threshold is exceeded, the observers will be published via Genesis process to not cause issues like blocking the outbox worker.
+  config.publish_consistent_snapshots = true # not required, defaults to `false`. A serializer reads a record's own columns and then queries its associations, so a write landing in between produces a payload whose `updated_at` predates the records embedded next to it - consumers rank and guard on that timestamp, so such a payload is discarded along with its embedded associations. When enabled, the row's timestamp is read back (bypassing the query cache) after serializing and the payload is re-serialized if the record moved. Emits the `dionysus.publish.consistent_snapshot` counter described below.
+  config.max_snapshot_attempts = 2 # not required, defaults to 3. Only relevant with `publish_consistent_snapshots` enabled. A record written faster than it serializes fails the check on every attempt, so it pays the full serialization cost this many times over and still publishes a payload that may be torn - lower this to bound that cost. Set it to 1 to keep the check and its metric while disabling re-serialization entirely.
   config.sidekiq_queue = :default # required, defaults to :dionysus. The queue will be used for a genesis process
 end
 ```
 
+##### Monitoring consistent snapshots
+
+With `publish_consistent_snapshots` enabled, every guarded message increments `dionysus.publish.consistent_snapshot` exactly once, tagged with `result`, `attempts`, `topic` and `model`. Nothing is emitted when the feature is off, so an empty series means the guard is not running rather than running clean.
+
+| `result` | meaning |
+|---|---|
+| `consistent` | the record had not moved by the time the payload was built. With `attempts:1` the first try was already clean; higher values mean it converged after re-serializing. |
+| `exhausted` | the record was still moving at `max_snapshot_attempts`. The payload is published anyway - a payload that may be torn beats no message - but it carries no marker and consumers see an ordinary message, so this counter is the only place the outcome is visible. |
+| `unsupported` | the records carry no `updated_at`, so the guard could not run on that message at all. |
+
+Watch the `exhausted` rate before and after a rollout: the retry cost scales with write rate multiplied by serialization span, so the hottest records are both the most expensive to guard and the least likely to converge.
+
 ##### DionysusOutbox model
 
 Generate a model for the outbox:

data/lib/dionysus/producer/config.rb

@@ -9,7 +9,8 @@ class Dionysus::Producer::Config
     :transactional_outbox_enabled, :sidekiq_queue, :publisher_service_name,
     :genesis_consistency_safety_delay, :hermes_event_producer, :publish_after_commit, :outbox_worker_publishing_delay,
     :high_priority_sidekiq_queue, :observers_inline_maximum_size, :remove_consecutive_duplicates_before_publishing,
-    :include_serialized_at_in_payload, :publish_with_uncached_reads
+    :include_serialized_at_in_payload, :publish_with_uncached_reads, :publish_consistent_snapshots,
+    :max_snapshot_attempts
 
   def self.default_sidekiq_queue
     :dionysus
@@ -99,12 +100,30 @@ class Dionysus::Producer::Config
 
   # Off by default so it can be switched on per producer, and switched back off in one env change
   # if the extra reads ever cost more than they are worth.
+  # How many times a payload may be re-serialized before the last attempt is published as it stands.
+  # This is the expensive half of publish_consistent_snapshots: a record written faster than it
+  # serializes fails the check on every attempt, so it pays the full serialization cost this many
+  # times over and still publishes a payload that may be torn. Lower it to bound that cost; 1
+  # disables re-serialization while leaving the check (and its metric) in place.
+  def max_snapshot_attempts
+    @max_snapshot_attempts || Dionysus::Producer::MAX_SNAPSHOT_ATTEMPTS
+  end
+
   def publish_with_uncached_reads
     return @publish_with_uncached_reads if defined?(@publish_with_uncached_reads)
 
     false
   end
 
+  # Re-serialize when a record moved while its payload was being built, so the payload describes a
+  # single moment. Off by default; the extra read costs one indexed column per message, and the
+  # re-serialization only happens on the records that were actually contended.
+  def publish_consistent_snapshots
+    return @publish_consistent_snapshots if defined?(@publish_consistent_snapshots)
+
+    false
+  end
+
   # Off by default so consumers, which fall back to the offset when the field is absent, can be
   # rolled out first.
   def include_serialized_at_in_payload

data/lib/dionysus/producer/karafka_responder_generator.rb

@@ -46,10 +46,10 @@ class Dionysus::Producer::KarafkaResponderGenerator
 
               record = records.sample
 
-              # the offset alone is publish order, and a message published later can carry an
-              # earlier snapshot, so consumers need to know when this payload was actually read
-              serialized_at = config.include_serialized_at_in_payload ? Time.now.utc : nil
-              payload = serialize_to_payload(records, topic, batch_options)
+              # consumers rank duplicates by this stamp, so it has to describe the attempt actually
+              # published - with retries that is the last one, not the first
+              payload, read_at = serialize_consistently(records, topic, batch_options)
+              serialized_at = config.include_serialized_at_in_payload ? read_at : nil
 
               event_payload = {
                 event: event,
@@ -75,6 +75,80 @@ class Dionysus::Producer::KarafkaResponderGenerator
 
       private
 
+      # A serializer reads a record's own columns and then queries its associations, so anything
+      # committed in between lands in the payload beside a timestamp taken before it - the payload
+      # describes no single moment. Consumers rank on that timestamp and then compare it against
+      # what they have stored, so a payload whose timestamp predates its own contents loses the
+      # comparison and is discarded, taking the has_many records embedded in it along with it.
+      #
+      # Measured on the affected topic in production: serialization spans 95ms at the median and up
+      # to 3.5s, and 4.7% of records receiving more than one message had the surviving message
+      # report a timestamp older than one already published.
+      #
+      # So serialize, then check whether the records moved underneath it. If they did, the payload
+      # is not a snapshot of anything: reload and serialize again. Retries are bounded, and the last
+      # attempt is published rather than dropped - a payload that may be torn still beats no message.
+      # returns [payload, read_at] - read_at is when the attempt that produced this payload began
+      define_method :serialize_consistently do |records, current_topic, batch_options|
+        unless config.publish_consistent_snapshots
+          read_at = Time.now.utc
+          next [serialize_to_payload(records, current_topic, batch_options), read_at]
+        end
+
+        max_attempts = config.max_snapshot_attempts
+        attempts = 0
+        loop do
+          read_at = Time.now.utc
+          before = snapshot_of(records)
+          payload = serialize_to_payload(records, current_topic, batch_options)
+          attempts += 1
+
+          if before.nil?
+            # nothing to compare against, so the guard did not run on this message at all
+            break [instrument_snapshot("unsupported", attempts, records, current_topic, payload), read_at]
+          end
+          if before == committed_snapshot_of(records)
+            break [instrument_snapshot("consistent", attempts, records, current_topic, payload), read_at]
+          end
+          if attempts >= max_attempts
+            # published anyway: a payload that may be torn beats no message. Nothing downstream can
+            # tell this apart from a clean one - the payload carries no marker and the consumer sees
+            # an ordinary message - so this counter is the only place the outcome is ever visible.
+            break [instrument_snapshot("exhausted", attempts, records, current_topic, payload), read_at]
+          end
+
+          records.each { |record| record.reload if record.is_a?(ActiveRecord::Base) && record.persisted? }
+        end
+      end
+
+      # One counter rather than several: the denominator, the retry distribution and the failure
+      # rate all have to come from the same series or none of them can be read as a rate.
+      define_method :instrument_snapshot do |result, attempts, records, current_topic, payload|
+        config.instrumenter.increment("dionysus.publish.consistent_snapshot",
+          tags: ["result:#{result}", "attempts:#{attempts}", "topic:#{current_topic}",
+            "model:#{records.first.class}"])
+        payload
+      end
+
+      # nil means there is nothing to compare - a record without timestamps, or not a record at all
+      define_method :snapshot_of do |records|
+        stamps = records.map { |record| record.updated_at if record.respond_to?(:updated_at) }
+        stamps.any?(&:nil?) ? nil : stamps
+      end
+
+      define_method :committed_snapshot_of do |records|
+        records.map do |record|
+          next record.updated_at unless record.is_a?(ActiveRecord::Base) && record.persisted?
+
+          # publishing runs inside the request when publish_after_commit is on, where the query
+          # cache is live on this connection - a cached read here would report that nothing moved
+          # and hand back the torn payload the check exists to catch
+          record.class.uncached do
+            record.class.where(record.class.primary_key => record.id).pick(:updated_at)
+          end
+        end
+      end
+
       define_method :serialize_to_payload do |records, current_topic, batch_options|
         if batch_options.to_h[:serialize] == false
           records.map(&:as_json)

data/lib/dionysus/producer.rb

@@ -1,6 +1,11 @@
 # frozen_string_literal: true
 
 class Dionysus::Producer
+  # a contended record can move again while it is being re-serialized; bound the work and publish
+  # the last attempt rather than looping or dropping the message. Overridable per producer through
+  # config.max_snapshot_attempts - this is only the default.
+  MAX_SNAPSHOT_ATTEMPTS = 3
+
   def self.configuration
     @configuration ||= Dionysus::Producer::Config.new
   end

data/lib/dionysus/version.rb

@@ -3,5 +3,5 @@
 module Dionysus
   module Version
   end
-  VERSION = "1.3.0"
+  VERSION = "1.4.1"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: dionysus-rb
 version: !ruby/object:Gem::Version
-  version: 1.3.0
+  version: 1.4.1
 platform: ruby
 authors:
 - Karol Galanciak
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-08-27 00:00:00.000000000 Z
+date: 2026-08-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord