dionysus-rb 1.1.0 → 1.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d190bbc3fd4b2ca0aa7c7d1be7c0ddaa6a3b6bdcbd6556b90e4752c2e875bfc6
-  data.tar.gz: 33379f5ccc97bceee7caad1d9b3383e84c3073e48b15bda1d0f592e765c39352
+  metadata.gz: 968407d443681c04bfc7426b8925d7f63db69a09763357413d01218c03ec3ad8
+  data.tar.gz: baf5e739a77bc2d48ab5197591ddb59c6d970bdaed68479014d5afbec5698f0e
 SHA512:
-  metadata.gz: cafb4cb76ad00f5dbbc45f9c3a578284ef554aaf680310286961d1f8251620bc67c10e190f301ac06fca1c8e55be9fab2e75977cc85ecf80f21503504558da20
-  data.tar.gz: 78e8c07d652f3dcc46cb3a527834cb72b2290b119266e4362cde3ddbf1d5679c147cb03cce34acb6682451c6453fb58532faab8a7bf5be43a6db6be791e7e1e2
+  metadata.gz: 121b9c6af2b90700a7af1aba098282cfe05df901c633dfcc71d7845bda935b5e124b21c8bedab20895e9c5dd11a3dd12c2edd049c1bcd9b984c7df0da8a246b5
+  data.tar.gz: 6594b49acf6f64bb59475bc4a8a3a70c5747a09dcb72d3bc815e544bf3016bf0df732f3b15f0b3b7c8106578ba3c8ed49f4206d5b206e240285d376b6e08b31f

data/.rubocop.yml

@@ -170,6 +170,7 @@ Naming/AccessorMethodName:
 RSpec/DescribeClass:
   Exclude:
     - 'spec/integration_spec.rb'
+    - 'spec/dedup_serialization_time_integration_spec.rb'
 
 Style/MultilineBlockChain:
   Exclude:

data/CHANGELOG.md

@@ -1,5 +1,24 @@
 ## [Unreleased]
 
+## [1.3.0]
+
+### Fixed
+
+- Read the record and its associations without the query cache while publishing, behind `config.publish_with_uncached_reads` (default `false`). The payload is built from the record plus the associations declared with `with:`. When `publish_after_commit` is on, publishing runs inside the request, where Rails' query cache is live on that connection - so a row read earlier in the request is served from the cache while its associations are queried fresh. The payload then carries an `updated_at` older than the data it contains.
+- Measured in production on the sibling gem over a two and a half hour window on one topic: of 1,476 records that received more than one message, 141 had a payload whose `updated_at` moved backwards as `serialized_at` moved forwards, and in 69 of those the message that wins deduplication carried an `updated_at` lower than one already published at a lower offset. One record published a payload serialized at `09:27:23.662` reporting `updated_at 09:27:22.139557`, when a payload serialized 1.3 seconds earlier had already reported `09:27:22.345092` - a value a row's `updated_at` cannot return to.
+- The consequence is not a stale field but a discarded record. Consumers rank duplicates and then apply `persist_with_dionysus?`, which compares the payload's `updated_at` against what is stored; a payload that loses that comparison is skipped by `next`, and the `has_many` records embedded in it are persisted further down the same loop body, so they are dropped with it. The parent keeps its old values and the new child rows are never created, with no error, no counter and nothing to self-heal it.
+- This is why no ordering over the fields already in the message could fix both known incidents: `updated_at` was in the message all along and is simply wrong. `serialized_at`, added in 1.2.0, correctly identifies the last-serialized message - which is precisely the one the guard rejects when its `updated_at` is stale.
+- Rolling this out needs no consumer changes: the fix is entirely on the producer, and the payload's shape is unchanged.
+
+## [1.2.0]
+
+### Fixed
+
+- Rank duplicate messages in a consumer batch by when their payload was serialized, not by Kafka offset. The previous release ranked on the highest offset, on the premise that offsets strictly increase in publish order. They do - but publish order is not serialization order: `Publisher#publish` does `find_by` -> `generate_message` -> `responder.call`, so a message produced later can carry an earlier snapshot. Measured in production on one record, eight `updated` events inside 700 ms: the offset the batch ended on was published at `.905` carrying a payload read at `.398194` with `paid_amount 0.0`, while an earlier offset published at `.837` carried a payload read at `.671830` with the real value of `1447.0`. Ranking on offset kept the stale payload and the consumer's value stuck at `0.00` with nothing to self-heal it.
+- Neither field already in the message is a freshness signal. `updated_at` is the record's own timestamp and does not move when only an associated record changes, so two payloads with different computed values can share one `updated_at` - that is the tie the previous release was written for. The offset is publish order, per above. The two known incidents are mirror images: in one the correct message has the higher offset and the older `updated_at`, in the other the lower offset and the newer `updated_at`, so no ordering over the existing fields fixes both.
+- The producer therefore stamps `serialized_at` where the payload is actually built, and the consumer ranks on `[serialized_at, offset]`. Messages without the field fall back to `Time.at(0)` and are ordered among themselves by offset, which is exactly the previous behaviour; a stamped message wins over an unstamped one, which is correct because a stamp can only come from a newer producer.
+- The stamp is gated on `config.include_serialized_at_in_payload`, default `false`, so published envelopes are unchanged until it is switched on. Roll consumers out first - with the fallback that deploy is a no-op - then enable it on the producer.
+
 ## [1.1.0]
 
 ### Fixed

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    dionysus-rb (1.1.0)
+    dionysus-rb (1.3.0)
       activerecord (>= 5)
       activesupport (>= 3.2)
       concurrent-ruby

data/lib/dionysus/consumer/params_batch_transformations/remove_duplicates_strategy.rb

@@ -1,6 +1,10 @@
 # frozen_string_literal: true
 
+require "time"
+
 class Dionysus::Consumer::ParamsBatchTransformations::RemoveDuplicatesStrategy
+  SERIALIZED_AT_FORMAT = %r{\A\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?Z\z}
+
   def call(params_batch)
     return params_batch if duplicates_removal_not_applicable?(params_batch)
 
@@ -12,17 +16,37 @@ class Dionysus::Consumer::ParamsBatchTransformations::RemoveDuplicatesStrategy
   # the idea is following:
   # 1. group messages by event and id - for a given model we can expect unique messages for _created and _deleted events
   #   but we can have multiple _updated events, so this is where we are interested in removing duplicates
-  # 2. from each group keep the message published last (highest offset) - payloads are serialized at
-  #   publish time, so that is the freshest state; "updated_at" does not track it
+  # 2. from each group keep the message whose payload was serialized last, with the offset as a tie-break -
+  #   the offset alone is publish order, and a message published later can carry an earlier snapshot
   # 3. flatten the array of arrays as all groups will have a single item
   def transform_messages_array(params_batch)
     params_batch
       .to_a
       .group_by { |batch| grouping_key_by_event_and_id(batch) }
-      .map { |_, group| group.max_by(&:offset) }
+      .map { |_, group| freshest(group) }
       .flatten
   end
 
+  # a group is ranked by its stamps only when every message in it carries one - during the
+  # producer rollout a batch can hold both, and a stamp says nothing about an unstamped neighbour
+  def freshest(group)
+    stamps = group.map { |message| serialized_at_for(message) }
+
+    return group.max_by(&:offset) if stamps.any?(&:nil?)
+
+    group.zip(stamps).max_by { |message, serialized_at| [serialized_at, message.offset] }.first
+  end
+
+  def serialized_at_for(message)
+    raw = message.payload.fetch("message").first["serialized_at"]
+
+    return unless raw.is_a?(String) && raw.match?(SERIALIZED_AT_FORMAT)
+
+    Time.parse(raw)
+  rescue
+    nil
+  end
+
   def grouping_key_by_event_and_id(batch)
     [
       batch.payload.fetch("message").first.fetch("event"),

data/lib/dionysus/producer/config.rb

@@ -8,7 +8,8 @@ class Dionysus::Producer::Config
     :lock_client, :lock_expiry_time, :error_handler, :outbox_publishing_batch_size,
     :transactional_outbox_enabled, :sidekiq_queue, :publisher_service_name,
     :genesis_consistency_safety_delay, :hermes_event_producer, :publish_after_commit, :outbox_worker_publishing_delay,
-    :high_priority_sidekiq_queue, :observers_inline_maximum_size, :remove_consecutive_duplicates_before_publishing
+    :high_priority_sidekiq_queue, :observers_inline_maximum_size, :remove_consecutive_duplicates_before_publishing,
+    :include_serialized_at_in_payload, :publish_with_uncached_reads
 
   def self.default_sidekiq_queue
     :dionysus
@@ -96,6 +97,22 @@ class Dionysus::Producer::Config
     @observers_inline_maximum_size || 1000
   end
 
+  # Off by default so it can be switched on per producer, and switched back off in one env change
+  # if the extra reads ever cost more than they are worth.
+  def publish_with_uncached_reads
+    return @publish_with_uncached_reads if defined?(@publish_with_uncached_reads)
+
+    false
+  end
+
+  # Off by default so consumers, which fall back to the offset when the field is absent, can be
+  # rolled out first.
+  def include_serialized_at_in_payload
+    return @include_serialized_at_in_payload if defined?(@include_serialized_at_in_payload)
+
+    false
+  end
+
   def remove_consecutive_duplicates_before_publishing
     return @remove_consecutive_duplicates_before_publishing if defined?(@remove_consecutive_duplicates_before_publishing)
 

data/lib/dionysus/producer/karafka_responder_generator.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require "active_support/core_ext/string"
+require "time"
 
 class Dionysus::Producer::KarafkaResponderGenerator
   TOMBSTONE = nil
@@ -45,13 +46,18 @@ class Dionysus::Producer::KarafkaResponderGenerator
 
               record = records.sample
 
+              # the offset alone is publish order, and a message published later can carry an
+              # earlier snapshot, so consumers need to know when this payload was actually read
+              serialized_at = config.include_serialized_at_in_payload ? Time.now.utc : nil
               payload = serialize_to_payload(records, topic, batch_options)
 
-              {
+              event_payload = {
                 event: event,
                 model_name: record.model_name.name,
                 data: payload
               }
+              event_payload[:serialized_at] = serialized_at.iso8601(6) if serialized_at
+              event_payload
             end
             unless genesis_only?(options)
               respond_to topic_name, { message: message }, **final_options

data/lib/dionysus/producer/outbox/publisher.rb

@@ -11,52 +11,56 @@ class Dionysus::Producer::Outbox::Publisher
   def publish(outbox_record, options = {})
     return if Dionysus::Producer::Suppressor.suppressed?
 
-    instrument("publishing_with_dionysus") do
-      resource_class = outbox_record.resource_class.constantize
-      primary_key = resource_class.primary_key
-      primary_key_value = outbox_record.resource_id
-      topic = outbox_record.topic
-      resource = resource_class.find_by(primary_key => primary_key_value) ||
-        resource_class.new(primary_key => primary_key_value)
-      event_name = outbox_record.event_name
-      if resource.new_record? && outbox_record.created_event?
-        logger.error(
-          "Attempted to publish #{resource.class}, id: #{resource.id} but it was deleted, that should never happen!"
-        )
-        return
-      end
+    with_consistent_reads do
+      instrument("publishing_with_dionysus") do
+        resource_class = outbox_record.resource_class.constantize
+        primary_key = resource_class.primary_key
+        primary_key_value = outbox_record.resource_id
+        topic = outbox_record.topic
+        resource = resource_class.find_by(primary_key => primary_key_value) ||
+          resource_class.new(primary_key => primary_key_value)
+        event_name = outbox_record.event_name
+        if resource.new_record? && outbox_record.created_event?
+          logger.error(
+            "Attempted to publish #{resource.class}, id: #{resource.id} but it was deleted, that should never happen!"
+          )
+          return
+        end
 
-      if resource.new_record? && outbox_record.updated_event?
-        logger.error(
-          "There was an update of #{resource.class}, id: #{resource.id} but it was deleted, that should never happen!"
-        )
-        return
-      end
+        if resource.new_record? && outbox_record.updated_event?
+          logger.error(
+            "There was an update of #{resource.class}, id: #{resource.id} but it was deleted, that should never happen!"
+          )
+          return
+        end
 
-      published_count = publish_for_top_level_resource(outbox_record, resource, event_name, topic, options) +
-        publish_for_dependency(resource, topic, options)
-      handle_nothing_published(resource, event_name, topic) if published_count.zero?
-      published_count
+        published_count = publish_for_top_level_resource(outbox_record, resource, event_name, topic, options) +
+          publish_for_dependency(resource, topic, options)
+        handle_nothing_published(resource, event_name, topic) if published_count.zero?
+        published_count
+      end
     end
   end
 
   def publish_observers(outbox_record)
     return if Dionysus::Producer::Suppressor.suppressed?
 
-    instrument("publishing_observers_with_dionysus") do
-      resource_class = outbox_record.resource_class.constantize
-      primary_key = resource_class.primary_key
-      primary_key_value = outbox_record.resource_id
-      resource = resource_class.find_by(primary_key => primary_key_value) ||
-        resource_class.new(primary_key => primary_key_value)
-      changeset = outbox_record.transformed_changeset
-
-      Dionysus::Producer.observers_with_responders_for(resource,
-        changeset).each do |observers, responder|
-        if observers.count > config.observers_inline_maximum_size
-          execute_genesis_for_observers(observers, responder)
-        else
-          observers.each { |observer_record| publish_observer(observer_record, responder) }
+    with_consistent_reads do
+      instrument("publishing_observers_with_dionysus") do
+        resource_class = outbox_record.resource_class.constantize
+        primary_key = resource_class.primary_key
+        primary_key_value = outbox_record.resource_id
+        resource = resource_class.find_by(primary_key => primary_key_value) ||
+          resource_class.new(primary_key => primary_key_value)
+        changeset = outbox_record.transformed_changeset
+
+        Dionysus::Producer.observers_with_responders_for(resource,
+          changeset).each do |observers, responder|
+          if observers.count > config.observers_inline_maximum_size
+            execute_genesis_for_observers(observers, responder)
+          else
+            observers.each { |observer_record| publish_observer(observer_record, responder) }
+          end
         end
       end
     end
@@ -64,6 +68,18 @@ class Dionysus::Producer::Outbox::Publisher
 
   private
 
+  # The payload is built from the record and its associations. Publishing runs inside the request
+  # when publish_after_commit is on, where Rails' query cache is live on that connection, so a row
+  # read earlier in the request can be served from the cache while its associations are queried
+  # fresh - producing a payload whose updated_at is older than the data it carries. Consumers rank
+  # and guard on updated_at, so such a payload is silently discarded, taking its embedded
+  # associations with it.
+  def with_consistent_reads(&)
+    return yield unless config.publish_with_uncached_reads
+
+    ActiveRecord::Base.uncached(&)
+  end
+
   delegate :instrumenter, :error_handler, to: :config
   delegate :instrument, to: :instrumenter
   delegate :logger, to: Dionysus

data/lib/dionysus/version.rb

@@ -3,5 +3,5 @@
 module Dionysus
   module Version
   end
-  VERSION = "1.1.0"
+  VERSION = "1.3.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: dionysus-rb
 version: !ruby/object:Gem::Version
-  version: 1.1.0
+  version: 1.3.0
 platform: ruby
 authors:
 - Karol Galanciak
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-08-18 00:00:00.000000000 Z
+date: 2026-08-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord