dionysus-rb 1.4.0 → 1.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8cd74dab3609a68e4919adb2913c5e2d428b2bd044ec785988eae123d125677b
-  data.tar.gz: 15ae10c89ce12d63f0d452bf8ebfb835a8b3593784048c003a28a093a9c77250
+  metadata.gz: fa1bffce325e2384847246f0125606bb90dc46d3226d32764e3ad650a74dccc4
+  data.tar.gz: 07fec41263e2726ddabdcc2cd8b5fae1dcd7452185013e2e9eda2d68b9ec6007
 SHA512:
-  metadata.gz: f3a2a0e0179804ff474dc9bb7575ba565e3c4377f6755cfb31aed5c783b7a3915966af0158a50132155a18c588e44495540d47a7463e475d07260f9db4cea6ef
-  data.tar.gz: 2e4450ae8743746067d81ab3c28614c4293fec570fdb550a9240b44f4e8ed097d7f3ecc110e03f51135c8fcd8237835e0a5a3ca90f754009b9146bd2e83e4866
+  metadata.gz: 76e8759abcf8533a5f3ef0559b97f3fb7fdcc379552204ef33456820108acd0271e88cc9c66f757f539731dbd0d95e574dd4afaa9e5e537e18468a191ae4c26c
+  data.tar.gz: d7f572eb5e5440bf599a38d33fb31c78d194d226a414f17890f536edf9150e58193242a761f47a46e4073d532c7001e676fe48487812a45cee5d66d711ec910b

data/CHANGELOG.md

@@ -1,5 +1,21 @@
 ## [Unreleased]
 
+## [1.5.0]
+- Publish the survivor of a collapsed run of duplicates once more after a delay, behind `config.republish_deduplicated_records` (default `false`) with `config.republish_deduplicated_records_delay` (default 30 seconds). It does nothing unless `remove_consecutive_duplicates_before_publishing` is also on, since there is no run to collapse otherwise.
+- What a collapsed run means. `DuplicatesFilter` keeps the last record of each consecutive run for the same resource, event and topic. A run longer than one only happens when the record was written again while an earlier message for it was still queued - which is the same window in which a payload can be serialized across a write and end up carrying an `updated_at` older than its own contents. `publish_consistent_snapshots` bounds its retries and publishes the last attempt regardless, so a payload that may still be torn does get published; this schedules one more publish once the writes have settled.
+- Why a republish is sufficient, and why it needs no change on the consumer. The outbox stores a pointer, not a payload: `Outbox::Publisher#publish` re-reads the row with `find_by`, so the scheduled record is serialized fresh at publish time and carries the settled state. The consumer's guard is `event_updated_at >= synced_at`, and `synced_at` is its mirror of the row's `updated_at` as of the last accepted message - a value the row's own `updated_at` can never fall below. A freshly-read republish is therefore accepted by construction, so nothing has to be forced past the guard and no timestamp has to be altered to make it win.
+- Scheduled with `retry_at`, not a future `created_at`. `fetch_publishable` already honours `retry_at` exactly, whereas `created_at` is compared against `Time.current + outbox_worker_publishing_delay` - so a future `created_at` publishes early by the length of that look-ahead window, and would report a negative `publishing_latency`. `retry_at` leaves `created_at` honest; `failed_at` and `error_class` stay `nil`, so a scheduled republish is distinguishable from a failed record awaiting retry.
+- Termination. A scheduled record that comes due on its own is a run of one, so it schedules nothing further. If the record is still being written when it comes due, it collapses again and schedules once more, which is the intended behaviour - the extra load is bounded at one additional record per key per batch, and pile-ups for the same key collapse through the same filter.
+- Observer records are skipped: they carry a changeset and publish through `publish_observers`, a different path.
+- Records that failed to publish are skipped too - they are already retried by `handle_error`'s backoff, so scheduling a republish for them would duplicate that.
+- Operational note: the scheduled records sit unpublished for the delay, so they raise the average and maximum reported by the outbox latency gauges. That is a truthful measurement - those messages really do wait - but any alert threshold on outbox latency should be checked before switching this on.
+
+## [1.4.1]
+- Stamp `serialized_at` when the attempt that is actually published begins, not before the retry loop. 1.4.0 takes the stamp once and then re-serializes up to `max_snapshot_attempts` times, so a message can be published a whole retry cycle after its own stamp - and the payload it publishes is the one read last, not the one the stamp describes. Consumers rank duplicates by `[serialized_at, offset]`, so the freshest payload ended up carrying the oldest stamp and losing its group to a staler sibling. That is the failure `serialized_at` was added to prevent, reintroduced by the guard added to prevent a different one.
+- Measured on the affected topic over 65 minutes: 205 of 6,572 messages (3.12%) carried an `updated_at` LATER than their own `serialized_at`, the worst by 1,225 ms. A timestamp cannot describe a moment after the one it was taken at, so that alone proves the stamp preceded the payload.
+- `serialize_consistently` now returns `[payload, read_at]`. The looseness between stamp and payload is bounded by a single serialization pass again, as it was before 1.4.0, rather than by the whole retry cycle - so contended records, which retry most, are no longer the ones stamped most wrongly.
+- Covered end to end against a real broker in `spec/serialized_at_tracks_published_attempt_spec.rb`: a contended publish retries while a competing publish serializes once and stamps later, both are read back off the topic, and the real `RemoveDuplicatesStrategy` is asked which survives. The spec fails on 1.4.0 on both assertions.
+
 ## [1.4.0]
 
 ### Fixed

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    dionysus-rb (1.4.0)
+    dionysus-rb (1.5.0)
       activerecord (>= 5)
       activesupport (>= 3.2)
       concurrent-ruby

data/README.md

@@ -336,6 +336,8 @@ Dionysus::Producer.configure do |config|
   config.publish_after_commit = true # not required, defaults to `false`. Check `Publishing records right after the transaction is committed` section for more details.
   config.outbox_worker_publishing_delay = 5 # non required, defaults to 0 a delay in seconds until the outbox record is considered publishable. Check `Publishing records right after the transaction is committed` section for more details.
   config.remove_consecutive_duplicates_before_publishing = true # not required, defaults to false. If set to true, the consecutive duplicates in the publishable batch will be removed and only one message will be published to a given topic. For example, if for whatever reason there are ten messages in a row for a given topic to publish `user_updated` ecent, only the last will be published. Check `Dionysus::Consumer::ParamsBatchTransformations::RemoveDuplicatesStrategy` for exact implementation. To verify if this feature is useful, it's recommended to browse Karafka UI and check messages in the topics if there are any obvious duplicates happening often.
+  config.republish_deduplicated_records = true # not required, defaults to false. Only has an effect with `remove_consecutive_duplicates_before_publishing` enabled. When a run of consecutive duplicates collapses, the surviving record is scheduled to publish once more after a delay - a run only collapses when the record was written again while an earlier message for it was still queued, which is the window in which a payload can be serialized across a write. Check `Republishing deduplicated records` section for more details.
+  config.republish_deduplicated_records_delay = 60 # not required, defaults to 30 (seconds). Only relevant with `republish_deduplicated_records` enabled. How long the scheduled republish waits - it has to outlast the burst of writes that produced the duplicates, or the republish is serialized inside the same contended window it exists to escape.
   config.observers_inline_maximum_size = 100 # not required, defaults to 1000. This config setting matters in case there is a huge amount of dependent records (observers). If the threshold is exceeded, the observers will be published via Genesis process to not cause issues like blocking the outbox worker.
   config.publish_consistent_snapshots = true # not required, defaults to `false`. A serializer reads a record's own columns and then queries its associations, so a write landing in between produces a payload whose `updated_at` predates the records embedded next to it - consumers rank and guard on that timestamp, so such a payload is discarded along with its embedded associations. When enabled, the row's timestamp is read back (bypassing the query cache) after serializing and the payload is re-serialized if the record moved. Emits the `dionysus.publish.consistent_snapshot` counter described below.
   config.max_snapshot_attempts = 2 # not required, defaults to 3. Only relevant with `publish_consistent_snapshots` enabled. A record written faster than it serializes fails the check on every attempt, so it pays the full serialization cost this many times over and still publishes a payload that may be torn - lower this to bound that cost. Set it to 1 to keep the check and its metric while disabling re-serialization entirely.
@@ -355,6 +357,28 @@ With `publish_consistent_snapshots` enabled, every guarded message increments `d
 
 Watch the `exhausted` rate before and after a rollout: the retry cost scales with write rate multiplied by serialization span, so the hottest records are both the most expensive to guard and the least likely to converge.
 
+##### Republishing deduplicated records
+
+`remove_consecutive_duplicates_before_publishing` keeps only the last record of each consecutive run for the same resource, event and topic. A run longer than one only happens when the record was written again while an earlier message for it was still queued - the same window in which a payload can be serialized across a write and end up carrying an `updated_at` older than its own contents. `publish_consistent_snapshots` bounds its retries and publishes the last attempt regardless, so a payload that may still be torn does get published.
+
+With `republish_deduplicated_records` enabled, the survivor of every collapsed run is scheduled to publish once more, `republish_deduplicated_records_delay` seconds later:
+
+```ruby
+config.remove_consecutive_duplicates_before_publishing = true
+config.republish_deduplicated_records = true
+config.republish_deduplicated_records_delay = 30
+```
+
+This needs no change on the consumer side. The outbox stores a pointer rather than a payload - `Outbox::Publisher#publish` re-reads the row with `find_by` - so the scheduled record is serialized fresh when it comes due and carries the settled state. The consumer's guard is `event_updated_at >= synced_at`, and `synced_at` is the consumer's mirror of the row's `updated_at` as of the last accepted message, a value the row's own `updated_at` can never fall below. A freshly-read republish is therefore accepted rather than discarded as stale, so nothing has to be forced past the guard and no timestamp has to be altered to make it win.
+
+The republish is scheduled with `retry_at`, not a future `created_at`. `fetch_publishable` honours `retry_at` exactly, whereas `created_at` is compared against `Time.current + outbox_worker_publishing_delay` - a future `created_at` would publish early by the length of that look-ahead window, and would report a negative `publishing_latency`. `failed_at` and `error_class` stay `nil`, so a scheduled republish stays distinguishable from a record awaiting an error retry.
+
+A scheduled record that comes due on its own is a run of one, so it schedules nothing further. If the record is still being written when it comes due it collapses again and schedules once more, which is the intent; the extra load is bounded at one additional record per key per batch, and pile-ups for the same key collapse through the same filter.
+
+Observer records are skipped - they carry a changeset that decides which observers fire, and they publish through `publish_observers`. Records that failed to publish are skipped too, since `handle_error`'s backoff already retries them.
+
+Before enabling this, check any alert threshold on outbox latency. The scheduled records sit unpublished for the delay, so they raise the average and maximum reported by the `"#{namespace}.dionysus.producer.outbox.latency.*"` gauges. That is a truthful measurement - those messages really do wait - but it will move the gauge.
+
 ##### DionysusOutbox model
 
 Generate a model for the outbox:

data/lib/dionysus/producer/config.rb

@@ -10,7 +10,7 @@ class Dionysus::Producer::Config
     :genesis_consistency_safety_delay, :hermes_event_producer, :publish_after_commit, :outbox_worker_publishing_delay,
     :high_priority_sidekiq_queue, :observers_inline_maximum_size, :remove_consecutive_duplicates_before_publishing,
     :include_serialized_at_in_payload, :publish_with_uncached_reads, :publish_consistent_snapshots,
-    :max_snapshot_attempts
+    :max_snapshot_attempts, :republish_deduplicated_records, :republish_deduplicated_records_delay
 
   def self.default_sidekiq_queue
     :dionysus
@@ -137,4 +137,19 @@ class Dionysus::Producer::Config
 
     false
   end
+
+  # Publish the survivor of a collapsed run of duplicates once more, after a delay, so a payload
+  # that may have been serialized across a write is followed by one read after the writes settled.
+  # Off by default; does nothing unless remove_consecutive_duplicates_before_publishing is also on.
+  def republish_deduplicated_records
+    return @republish_deduplicated_records if defined?(@republish_deduplicated_records)
+
+    false
+  end
+
+  # Has to outlast the burst of writes that produced the duplicates, or the republish is serialized
+  # inside the same contended window it exists to escape.
+  def republish_deduplicated_records_delay
+    (@republish_deduplicated_records_delay || 30).to_d.seconds
+  end
 end

data/lib/dionysus/producer/karafka_responder_generator.rb

@@ -46,10 +46,10 @@ class Dionysus::Producer::KarafkaResponderGenerator
 
               record = records.sample
 
-              # the offset alone is publish order, and a message published later can carry an
-              # earlier snapshot, so consumers need to know when this payload was actually read
-              serialized_at = config.include_serialized_at_in_payload ? Time.now.utc : nil
-              payload = serialize_consistently(records, topic, batch_options)
+              # consumers rank duplicates by this stamp, so it has to describe the attempt actually
+              # published - with retries that is the last one, not the first
+              payload, read_at = serialize_consistently(records, topic, batch_options)
+              serialized_at = config.include_serialized_at_in_payload ? read_at : nil
 
               event_payload = {
                 event: event,
@@ -88,26 +88,33 @@ class Dionysus::Producer::KarafkaResponderGenerator
       # So serialize, then check whether the records moved underneath it. If they did, the payload
       # is not a snapshot of anything: reload and serialize again. Retries are bounded, and the last
       # attempt is published rather than dropped - a payload that may be torn still beats no message.
+      # returns [payload, read_at] - read_at is when the attempt that produced this payload began
       define_method :serialize_consistently do |records, current_topic, batch_options|
-        next serialize_to_payload(records, current_topic, batch_options) unless config.publish_consistent_snapshots
+        unless config.publish_consistent_snapshots
+          read_at = Time.now.utc
+          next [serialize_to_payload(records, current_topic, batch_options), read_at]
+        end
 
         max_attempts = config.max_snapshot_attempts
         attempts = 0
         loop do
+          read_at = Time.now.utc
           before = snapshot_of(records)
           payload = serialize_to_payload(records, current_topic, batch_options)
           attempts += 1
 
-          # nothing to compare against, so the guard did not run on this message at all
-          break instrument_snapshot("unsupported", attempts, records, current_topic, payload) if before.nil?
+          if before.nil?
+            # nothing to compare against, so the guard did not run on this message at all
+            break [instrument_snapshot("unsupported", attempts, records, current_topic, payload), read_at]
+          end
           if before == committed_snapshot_of(records)
-            break instrument_snapshot("consistent", attempts, records, current_topic, payload)
+            break [instrument_snapshot("consistent", attempts, records, current_topic, payload), read_at]
           end
           if attempts >= max_attempts
             # published anyway: a payload that may be torn beats no message. Nothing downstream can
             # tell this apart from a clean one - the payload carries no marker and the consumer sees
             # an ordinary message - so this counter is the only place the outcome is ever visible.
-            break instrument_snapshot("exhausted", attempts, records, current_topic, payload)
+            break [instrument_snapshot("exhausted", attempts, records, current_topic, payload), read_at]
           end
 
           records.each { |record| record.reload if record.is_a?(ActiveRecord::Base) && record.persisted? }

data/lib/dionysus/producer/outbox/duplicates_filter.rb

@@ -5,6 +5,10 @@ class Dionysus::Producer::Outbox::DuplicatesFilter
     new(records_to_publish).call
   end
 
+  def self.deduplicated_records(records_to_publish)
+    new(records_to_publish).deduplicated_records
+  end
+
   attr_reader :records_to_publish
   private     :records_to_publish
 
@@ -13,13 +17,23 @@ class Dionysus::Producer::Outbox::DuplicatesFilter
   end
 
   def call
-    records_to_publish
-      .slice_when { |record_1, record_2| generate_uniqueness_key(record_1) != generate_uniqueness_key(record_2) }
-      .flat_map(&:last)
+    consecutive_runs.flat_map(&:last)
+  end
+
+  # The survivor of every run that collapsed something: the record was written again while an
+  # earlier message for it was still queued.
+  def deduplicated_records
+    consecutive_runs.select { |run| run.size > 1 }.map(&:last)
   end
 
   private
 
+  def consecutive_runs
+    @consecutive_runs ||= records_to_publish
+      .slice_when { |record_1, record_2| generate_uniqueness_key(record_1) != generate_uniqueness_key(record_2) }
+      .to_a
+  end
+
   def generate_uniqueness_key(record)
     [record.resource_class, record.resource_id, record.event_name, record.topic]
   end

data/lib/dionysus/producer/outbox/records_processor.rb

@@ -23,6 +23,7 @@ class Dionysus::Producer::Outbox::RecordsProcessor
     end
     published_records = records - failed_records
     mark_as_published(published_records)
+    schedule_republishes(records, failed_records)
     records
   end
 
@@ -36,6 +37,29 @@ class Dionysus::Producer::Outbox::RecordsProcessor
     Dionysus::Producer::Outbox::DuplicatesFilter.call(records)
   end
 
+  # `retry_at` rather than a future `created_at`: it is honoured exactly, whereas `created_at` is
+  # offset by `outbox_worker_publishing_delay` and feeds `publishing_latency`.
+  def schedule_republishes(records, failed_records)
+    return unless config.republish_deduplicated_records
+    return unless config.remove_consecutive_duplicates_before_publishing
+
+    republish_at = Time.current + config.republish_deduplicated_records_delay
+    deduplicated = Dionysus::Producer::Outbox::DuplicatesFilter.deduplicated_records(records)
+
+    (deduplicated - failed_records).each do |record|
+      next if record.observer?
+
+      outbox_model.create!(
+        resource_class: record.resource_class,
+        resource_id: record.resource_id,
+        event_name: record.event_name,
+        partition_key: record.partition_key,
+        topic: record.topic,
+        retry_at: republish_at
+      )
+    end
+  end
+
   def publish(record)
     if record.observer?
       outbox_publisher.publish_observers(record)

data/lib/dionysus/version.rb

@@ -3,5 +3,5 @@
 module Dionysus
   module Version
   end
-  VERSION = "1.4.0"
+  VERSION = "1.5.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: dionysus-rb
 version: !ruby/object:Gem::Version
-  version: 1.4.0
+  version: 1.5.0
 platform: ruby
 authors:
 - Karol Galanciak
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-08-28 00:00:00.000000000 Z
+date: 2026-08-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord