dionysus-rb 1.4.1 → 1.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: aea1780da49737ea39a412112502ccefec1921bcaa8e4d9e3f4a17c01e359cb1
-  data.tar.gz: 0de793797246fa1aa63209eb9713529df3bf3b51aedca2d03be485ab3b140f43
+  metadata.gz: 1b3fb6e16f40d4396b115342f72098bbb2b3f56d8fb630898123e63acce866dc
+  data.tar.gz: ff8f56aa9b8a142131d92b53d83f7014b987439158c39a7518c096f9e05de980
 SHA512:
-  metadata.gz: 9ba62b20fdf34088c43f28bfd71e4fa721b4d512125ca030c16c61eb077354aab5751a05c1ce0192708a7fc87ccd3d3af0a63b8a2f06a48ab903858c928d4ce8
-  data.tar.gz: d67700b960048a4ea6b272d20d193ba4232115665a5a72d0b2171cec6ba089c3bb451ab36fd5092ffbec55a763fca95fa7b8b7299b3f44263122e58aea26932f
+  metadata.gz: 9250e85c200c5190a6f85fef44404fa0576d39621960af375b3cbd3cd7ffe172b19fff8021c4f32b1cd63ac453457c6c3bffd504abac9851091d3f4091530b45
+  data.tar.gz: 28194d92924dbac55258734f66ed3a4090ebe514d2edf8dcfd0368a244c4373111c70be23a32b1d3fb08f271f52ade9434529b4ea6dd91015bee8d0bfaa077b5

data/CHANGELOG.md

@@ -1,5 +1,23 @@
 ## [Unreleased]
 
+## [1.6.0]
+- Judge a payload's embedded children on their own timestamps instead of dropping them with the parent. When `persist_with_dionysus?` rejects a record, `persist` used to `next` past the whole loop body - the child `persist` calls included - so every `has_many` and `has_one` record embedded in that payload was discarded with it, with no error and no counter. The parent is still not written; only the traversal of its children is decoupled from its verdict. Each child re-enters `persist` and is guarded on its own `synced_updated_at || synced_created_at`, so a child the consumer has never seen is created and a child older than the row held locally is still skipped.
+- Reproduced at the consumer, not argued: `spec/embedded_children_persistence_spec.rb` runs a whole batch through the generated consumer with deduplication in place, from two failures observed in production. In the first, all seven money-bearing messages carried an `updated_at` 44 ms older than the zero-money payload already accepted, and the payment embedded in them - a row that did not exist locally - was never created. Three control examples pin the fixture down: the same shape at a timestamp the guard accepts persists the money, creates the embedded payment and calls `resolve_to_many_association`, so a red assertion is a statement about this gem rather than about the payload being malformed.
+- `resolve_to_many_association` and `resolve_to_one_association` are suppressed for the whole subtree beneath a rejected parent. Cancelling or deleting the children absent from the payload's list is how a genuine deletion propagates, and it requires a complete current list - which is exactly what a payload already judged stale cannot provide. Consuming applications implement that hook destructively, several with `destroy_all`, and the old `next` was shielding them by accident; suppressing it deliberately keeps that behaviour rather than changing it.
+- Children beneath a rejected parent are compared strictly, through the new `SynchronizableModel#advances_dionysus_state?`. The ordinary guard accepts a tie, and a purge stamps the cancel column with `update_all` without bumping `synced_updated_at` - so a stale payload that merely ties on a child's timestamp is precisely how a cancelled child would be revived. A tie carries nothing that is not already held, so requiring the child to be demonstrably newer closes that without costing a repair. `persist_with_dionysus?` itself is untouched and behaves as before on the ordinary path. A child the payload carries no timestamp for at all is refused outright for the same reason: a payload already judged stale proves nothing about it, and where `resolve_to_many_association` hard-deletes, applying one would bring back a child a fresher payload had removed.
+- The stale verdict is sticky for the whole subtree, which is a trade rather than a free win: a child that is itself fresh now advances while its own children keep the generation they had, because the purge needs a complete list that only a non-stale ancestor can vouch for. Before this change the child did not advance either, so the subtree stayed internally consistent and equally wrong; a later payload that passes the root guard repairs it. Recorded as a pending example.
+- Not fixed here, and left as a pending example rather than a silent gap: when deduplication keeps a message whose `updated_at` the guard rejects while a sibling in the same group would have been accepted, the parent's own columns are still lost. Recovering that needs the guard - not a payload timestamp - to choose the survivor. Keeping siblings whose `updated_at` looks newer is disproven by the existing torn-payload fixture in the deduplication specs, where the correct payload carries the *lower* `updated_at`; ranking on any single key has a production counterexample, so `RemoveDuplicatesStrategy` is unchanged. A consuming application that needs the parent half today can pass `params_batch_transformation: nil` for the topic, which is covered by an example here.
+
+## [1.5.0]
+- Publish the survivor of a collapsed run of duplicates once more after a delay, behind `config.republish_deduplicated_records` (default `false`) with `config.republish_deduplicated_records_delay` (default 30 seconds). It does nothing unless `remove_consecutive_duplicates_before_publishing` is also on, since there is no run to collapse otherwise.
+- What a collapsed run means. `DuplicatesFilter` keeps the last record of each consecutive run for the same resource, event and topic. A run longer than one only happens when the record was written again while an earlier message for it was still queued - which is the same window in which a payload can be serialized across a write and end up carrying an `updated_at` older than its own contents. `publish_consistent_snapshots` bounds its retries and publishes the last attempt regardless, so a payload that may still be torn does get published; this schedules one more publish once the writes have settled.
+- Why a republish is sufficient, and why it needs no change on the consumer. The outbox stores a pointer, not a payload: `Outbox::Publisher#publish` re-reads the row with `find_by`, so the scheduled record is serialized fresh at publish time and carries the settled state. The consumer's guard is `event_updated_at >= synced_at`, and `synced_at` is its mirror of the row's `updated_at` as of the last accepted message - a value the row's own `updated_at` can never fall below. A freshly-read republish is therefore accepted by construction, so nothing has to be forced past the guard and no timestamp has to be altered to make it win.
+- Scheduled with `retry_at`, not a future `created_at`. `fetch_publishable` already honours `retry_at` exactly, whereas `created_at` is compared against `Time.current + outbox_worker_publishing_delay` - so a future `created_at` publishes early by the length of that look-ahead window, and would report a negative `publishing_latency`. `retry_at` leaves `created_at` honest; `failed_at` and `error_class` stay `nil`, so a scheduled republish is distinguishable from a failed record awaiting retry.
+- Termination. A scheduled record that comes due on its own is a run of one, so it schedules nothing further. If the record is still being written when it comes due, it collapses again and schedules once more, which is the intended behaviour - the extra load is bounded at one additional record per key per batch, and pile-ups for the same key collapse through the same filter.
+- Observer records are skipped: they carry a changeset and publish through `publish_observers`, a different path.
+- Records that failed to publish are skipped too - they are already retried by `handle_error`'s backoff, so scheduling a republish for them would duplicate that.
+- Operational note: the scheduled records sit unpublished for the delay, so they raise the average and maximum reported by the outbox latency gauges. That is a truthful measurement - those messages really do wait - but any alert threshold on outbox latency should be checked before switching this on.
+
 ## [1.4.1]
 - Stamp `serialized_at` when the attempt that is actually published begins, not before the retry loop. 1.4.0 takes the stamp once and then re-serializes up to `max_snapshot_attempts` times, so a message can be published a whole retry cycle after its own stamp - and the payload it publishes is the one read last, not the one the stamp describes. Consumers rank duplicates by `[serialized_at, offset]`, so the freshest payload ended up carrying the oldest stamp and losing its group to a staler sibling. That is the failure `serialized_at` was added to prevent, reintroduced by the guard added to prevent a different one.
 - Measured on the affected topic over 65 minutes: 205 of 6,572 messages (3.12%) carried an `updated_at` LATER than their own `serialized_at`, the worst by 1,225 ms. A timestamp cannot describe a moment after the one it was taken at, so that alone proves the stamp preceded the payload.

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    dionysus-rb (1.4.1)
+    dionysus-rb (1.6.0)
       activerecord (>= 5)
       activesupport (>= 3.2)
       concurrent-ruby

data/README.md

@@ -336,6 +336,8 @@ Dionysus::Producer.configure do |config|
   config.publish_after_commit = true # not required, defaults to `false`. Check `Publishing records right after the transaction is committed` section for more details.
   config.outbox_worker_publishing_delay = 5 # non required, defaults to 0 a delay in seconds until the outbox record is considered publishable. Check `Publishing records right after the transaction is committed` section for more details.
   config.remove_consecutive_duplicates_before_publishing = true # not required, defaults to false. If set to true, the consecutive duplicates in the publishable batch will be removed and only one message will be published to a given topic. For example, if for whatever reason there are ten messages in a row for a given topic to publish `user_updated` ecent, only the last will be published. Check `Dionysus::Consumer::ParamsBatchTransformations::RemoveDuplicatesStrategy` for exact implementation. To verify if this feature is useful, it's recommended to browse Karafka UI and check messages in the topics if there are any obvious duplicates happening often.
+  config.republish_deduplicated_records = true # not required, defaults to false. Only has an effect with `remove_consecutive_duplicates_before_publishing` enabled. When a run of consecutive duplicates collapses, the surviving record is scheduled to publish once more after a delay - a run only collapses when the record was written again while an earlier message for it was still queued, which is the window in which a payload can be serialized across a write. Check `Republishing deduplicated records` section for more details.
+  config.republish_deduplicated_records_delay = 60 # not required, defaults to 30 (seconds). Only relevant with `republish_deduplicated_records` enabled. How long the scheduled republish waits - it has to outlast the burst of writes that produced the duplicates, or the republish is serialized inside the same contended window it exists to escape.
   config.observers_inline_maximum_size = 100 # not required, defaults to 1000. This config setting matters in case there is a huge amount of dependent records (observers). If the threshold is exceeded, the observers will be published via Genesis process to not cause issues like blocking the outbox worker.
   config.publish_consistent_snapshots = true # not required, defaults to `false`. A serializer reads a record's own columns and then queries its associations, so a write landing in between produces a payload whose `updated_at` predates the records embedded next to it - consumers rank and guard on that timestamp, so such a payload is discarded along with its embedded associations. When enabled, the row's timestamp is read back (bypassing the query cache) after serializing and the payload is re-serialized if the record moved. Emits the `dionysus.publish.consistent_snapshot` counter described below.
   config.max_snapshot_attempts = 2 # not required, defaults to 3. Only relevant with `publish_consistent_snapshots` enabled. A record written faster than it serializes fails the check on every attempt, so it pays the full serialization cost this many times over and still publishes a payload that may be torn - lower this to bound that cost. Set it to 1 to keep the check and its metric while disabling re-serialization entirely.
@@ -355,6 +357,28 @@ With `publish_consistent_snapshots` enabled, every guarded message increments `d
 
 Watch the `exhausted` rate before and after a rollout: the retry cost scales with write rate multiplied by serialization span, so the hottest records are both the most expensive to guard and the least likely to converge.
 
+##### Republishing deduplicated records
+
+`remove_consecutive_duplicates_before_publishing` keeps only the last record of each consecutive run for the same resource, event and topic. A run longer than one only happens when the record was written again while an earlier message for it was still queued - the same window in which a payload can be serialized across a write and end up carrying an `updated_at` older than its own contents. `publish_consistent_snapshots` bounds its retries and publishes the last attempt regardless, so a payload that may still be torn does get published.
+
+With `republish_deduplicated_records` enabled, the survivor of every collapsed run is scheduled to publish once more, `republish_deduplicated_records_delay` seconds later:
+
+```ruby
+config.remove_consecutive_duplicates_before_publishing = true
+config.republish_deduplicated_records = true
+config.republish_deduplicated_records_delay = 30
+```
+
+This needs no change on the consumer side. The outbox stores a pointer rather than a payload - `Outbox::Publisher#publish` re-reads the row with `find_by` - so the scheduled record is serialized fresh when it comes due and carries the settled state. The consumer's guard is `event_updated_at >= synced_at`, and `synced_at` is the consumer's mirror of the row's `updated_at` as of the last accepted message, a value the row's own `updated_at` can never fall below. A freshly-read republish is therefore accepted rather than discarded as stale, so nothing has to be forced past the guard and no timestamp has to be altered to make it win.
+
+The republish is scheduled with `retry_at`, not a future `created_at`. `fetch_publishable` honours `retry_at` exactly, whereas `created_at` is compared against `Time.current + outbox_worker_publishing_delay` - a future `created_at` would publish early by the length of that look-ahead window, and would report a negative `publishing_latency`. `failed_at` and `error_class` stay `nil`, so a scheduled republish stays distinguishable from a record awaiting an error retry.
+
+A scheduled record that comes due on its own is a run of one, so it schedules nothing further. If the record is still being written when it comes due it collapses again and schedules once more, which is the intent; the extra load is bounded at one additional record per key per batch, and pile-ups for the same key collapse through the same filter.
+
+Observer records are skipped - they carry a changeset that decides which observers fire, and they publish through `publish_observers`. Records that failed to publish are skipped too, since `handle_error`'s backoff already retries them.
+
+Before enabling this, check any alert threshold on outbox latency. The scheduled records sit unpublished for the delay, so they raise the average and maximum reported by the `"#{namespace}.dionysus.producer.outbox.latency.*"` gauges. That is a truthful measurement - those messages really do wait - but it will move the gauge.
+
 ##### DionysusOutbox model
 
 Generate a model for the outbox:

data/lib/dionysus/consumer/persistor.rb

@@ -9,14 +9,14 @@ class Dionysus::Consumer::Persistor
     @topic = topic
   end
 
-  def persist(dionysus_event, batch_number)
+  def persist(dionysus_event, batch_number, under_stale_parent: false)
     if dionysus_event.generic_event?
       if dionysus_event.created? && topic.options[:import] == true
         persist_via_dionysus_create(dionysus_event, batch_number)
       elsif dionysus_event.destroyed? && topic.options[:import] == true
         persist_via_dionysus_destroy(dionysus_event, batch_number)
       else
-        persist_standard_event(dionysus_event, batch_number)
+        persist_standard_event(dionysus_event, batch_number, under_stale_parent: under_stale_parent)
       end
     else
       log_unknown_event_type(dionysus_event)
@@ -43,12 +43,9 @@ class Dionysus::Consumer::Persistor
     config.model_factory.for_model(dionysus_event.model_name)
   end
 
-  def persist_standard_event(dionysus_event, batch_number)
+  def persist_standard_event(dionysus_event, batch_number, under_stale_parent: false)
     Array.wrap(dionysus_event.transformed_data).each do |deseralized_record|
       model_klass = find_model_klass(dionysus_event) or return
-      attributes = deseralized_record.attributes
-      has_one_relationships = deseralized_record.has_one
-      has_many_relationships = deseralized_record.has_many
       synced_id = deseralized_record.synced_id
 
       if synced_id.nil?
@@ -59,38 +56,53 @@ class Dionysus::Consumer::Persistor
       record = Dionysus::Consumer::SynchronizableModel.new(config,
         model_klass.find_or_initialize_by(config.synced_id_attribute => synced_id))
       event_updated_at = deseralized_record.synced_updated_at || deseralized_record.synced_created_at
+      persistable = persistable?(record, event_updated_at, under_stale_parent)
 
-      next unless record.persist_with_dionysus?(event_updated_at)
+      apply_record(dionysus_event, record, deseralized_record, synced_id, batch_number) if persistable
 
-      record.assign_attributes_from_dionysus(attributes)
-      if dionysus_event.destroyed?
-        instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.remove_with_dionysus") do
-          record.remove_with_dionysus(deseralized_record) if dionysus_event.aggregate_root?
-        end
-      else
-        instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.restore_with_dionysus") do
-          record.restore_with_dionysus if record.restorable?(deseralized_record)
-        end
-      end
+      # children re-enter persist and are guarded on their own timestamp; only the child list is untrusted
+      persist_relationships(dionysus_event, record, deseralized_record, batch_number,
+        under_stale_parent: under_stale_parent || !persistable)
+    end
+  end
 
-      dionysus_event.local_changes[[dionysus_event.model_name, synced_id]] = record.changes if record.changes.present?
+  def persistable?(record, event_updated_at, under_stale_parent)
+    if under_stale_parent
+      record.advances_dionysus_state?(event_updated_at)
+    else
+      record.persist_with_dionysus?(event_updated_at)
+    end
+  end
 
-      instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.save") do
-        record.save unless record.destroyed?
+  def apply_record(dionysus_event, record, deseralized_record, synced_id, batch_number)
+    record.assign_attributes_from_dionysus(deseralized_record.attributes)
+    if dionysus_event.destroyed?
+      instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.remove_with_dionysus") do
+        record.remove_with_dionysus(deseralized_record) if dionysus_event.aggregate_root?
+      end
+    else
+      instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.restore_with_dionysus") do
+        record.restore_with_dionysus if record.restorable?(deseralized_record)
       end
+    end
+
+    dionysus_event.local_changes[[dionysus_event.model_name, synced_id]] = record.changes if record.changes.present?
+
+    instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.save") do
+      record.save unless record.destroyed?
+    end
+  end
 
-      instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.persist_to_many_relationships") do
-        has_many_relationships.each do |relationship_name, relationship_records|
-          persist_to_many_relationship(dionysus_event, relationship_name, record, relationship_records,
-            batch_number)
-        end
+  def persist_relationships(dionysus_event, record, deseralized_record, batch_number, under_stale_parent:)
+    instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.persist_to_many_relationships") do
+      deseralized_record.has_many.each do |relationship|
+        persist_to_many_relationship(dionysus_event, relationship, record, batch_number, under_stale_parent: under_stale_parent)
       end
+    end
 
-      instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.persist_to_one_relationships") do
-        has_one_relationships.each do |relationship_name, relationship_record|
-          persist_to_one_relationship(dionysus_event, relationship_name, record, relationship_record,
-            batch_number)
-        end
+    instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.persist_to_one_relationships") do
+      deseralized_record.has_one.each do |relationship|
+        persist_to_one_relationship(dionysus_event, relationship, record, batch_number, under_stale_parent: under_stale_parent)
       end
     end
   end
@@ -99,7 +111,9 @@ class Dionysus::Consumer::Persistor
     Dionysus.logger.debug("[Dionysus] unknown event type #{dionysus_event.event_name}")
   end
 
-  def persist_to_one_relationship(original_event, relationship_name, parent_model_record, record, batch_number)
+  def persist_to_one_relationship(original_event, relationship, parent_model_record, batch_number,
+    under_stale_parent: false)
+    relationship_name, record = relationship
     instrumentation_arguments = {
       event_name: original_event.event_name,
       parent_model_record: parent_model_record.model_name.to_s,
@@ -115,17 +129,21 @@ class Dionysus::Consumer::Persistor
         relationship_name, records, aggregate_root: false)
 
       instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.persist_to_one_relationship.#{relationship_name}.persist") do
-        persist(dionysus_event, batch_number)
+        persist(dionysus_event, batch_number, under_stale_parent: under_stale_parent)
         original_event.local_changes.merge!(dionysus_event.local_changes)
       end
 
+      return if under_stale_parent
+
       instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.persist_to_one_relationship.#{relationship_name}.resolve_to_one_association") do
         parent_model_record.resolve_to_one_association(relationship_name, record.synced_id)
       end
     end
   end
 
-  def persist_to_many_relationship(original_event, relationship_name, parent_model_record, records, batch_number)
+  def persist_to_many_relationship(original_event, relationship, parent_model_record, batch_number,
+    under_stale_parent: false)
+    relationship_name, records = relationship
     instrumentation_arguments = {
       event_name: original_event.event_name,
       parent_model_record: parent_model_record.model_name.to_s,
@@ -140,10 +158,13 @@ class Dionysus::Consumer::Persistor
         relationship_name, records, aggregate_root: false)
 
       instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.persist_to_many_relationship.#{relationship_name}.persist") do
-        persist(dionysus_event, batch_number)
+        persist(dionysus_event, batch_number, under_stale_parent: under_stale_parent)
         original_event.local_changes.merge!(dionysus_event.local_changes)
       end
 
+      # the purge needs a complete current list, which a payload already judged stale cannot provide
+      return if under_stale_parent
+
       synced_ids_of_related_records = records.map(&:synced_id)
       instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.persist_to_many_relationship.#{relationship_name}.resolve_to_many_association") do
         parent_model_record.resolve_to_many_association(relationship_name, synced_ids_of_related_records)

data/lib/dionysus/consumer/synchronizable_model.rb

@@ -25,6 +25,18 @@ class Dionysus::Consumer::SynchronizableModel < SimpleDelegator
     (synced_at && event_updated_at && event_updated_at >= synced_at) || synced_at.nil? || event_updated_at.nil?
   end
 
+  # A purge stamps the cancel column without bumping synced_updated_at, so under a stale payload a tie
+  # on the child's timestamp is how a cancelled child gets revived. Demand demonstrably newer.
+  # An unstamped child is refused outright: a payload the guard has already rejected proves nothing
+  # about a child it carries no timestamp for, and applying one would resurrect a child that a fresher
+  # payload had already deleted. Refusing it is exactly what the old blanket `next` did.
+  def advances_dionysus_state?(event_updated_at)
+    return false if event_updated_at.nil?
+
+    persist_with_dionysus?(event_updated_at) &&
+      !(synced_at && event_updated_at == synced_at)
+  end
+
   def assign_attributes_from_dionysus(attributes)
     public_send("#{synced_data_attribute}=", attributes)
     reverse_mapping = config.attributes_mapping_for_model(model.model_name).to_a.to_h(&:reverse)

data/lib/dionysus/producer/config.rb

@@ -10,7 +10,7 @@ class Dionysus::Producer::Config
     :genesis_consistency_safety_delay, :hermes_event_producer, :publish_after_commit, :outbox_worker_publishing_delay,
     :high_priority_sidekiq_queue, :observers_inline_maximum_size, :remove_consecutive_duplicates_before_publishing,
     :include_serialized_at_in_payload, :publish_with_uncached_reads, :publish_consistent_snapshots,
-    :max_snapshot_attempts
+    :max_snapshot_attempts, :republish_deduplicated_records, :republish_deduplicated_records_delay
 
   def self.default_sidekiq_queue
     :dionysus
@@ -137,4 +137,19 @@ class Dionysus::Producer::Config
 
     false
   end
+
+  # Publish the survivor of a collapsed run of duplicates once more, after a delay, so a payload
+  # that may have been serialized across a write is followed by one read after the writes settled.
+  # Off by default; does nothing unless remove_consecutive_duplicates_before_publishing is also on.
+  def republish_deduplicated_records
+    return @republish_deduplicated_records if defined?(@republish_deduplicated_records)
+
+    false
+  end
+
+  # Has to outlast the burst of writes that produced the duplicates, or the republish is serialized
+  # inside the same contended window it exists to escape.
+  def republish_deduplicated_records_delay
+    (@republish_deduplicated_records_delay || 30).to_d.seconds
+  end
 end

data/lib/dionysus/producer/outbox/duplicates_filter.rb

@@ -5,6 +5,10 @@ class Dionysus::Producer::Outbox::DuplicatesFilter
     new(records_to_publish).call
   end
 
+  def self.deduplicated_records(records_to_publish)
+    new(records_to_publish).deduplicated_records
+  end
+
   attr_reader :records_to_publish
   private     :records_to_publish
 
@@ -13,13 +17,23 @@ class Dionysus::Producer::Outbox::DuplicatesFilter
   end
 
   def call
-    records_to_publish
-      .slice_when { |record_1, record_2| generate_uniqueness_key(record_1) != generate_uniqueness_key(record_2) }
-      .flat_map(&:last)
+    consecutive_runs.flat_map(&:last)
+  end
+
+  # The survivor of every run that collapsed something: the record was written again while an
+  # earlier message for it was still queued.
+  def deduplicated_records
+    consecutive_runs.select { |run| run.size > 1 }.map(&:last)
   end
 
   private
 
+  def consecutive_runs
+    @consecutive_runs ||= records_to_publish
+      .slice_when { |record_1, record_2| generate_uniqueness_key(record_1) != generate_uniqueness_key(record_2) }
+      .to_a
+  end
+
   def generate_uniqueness_key(record)
     [record.resource_class, record.resource_id, record.event_name, record.topic]
   end

data/lib/dionysus/producer/outbox/records_processor.rb

@@ -23,6 +23,7 @@ class Dionysus::Producer::Outbox::RecordsProcessor
     end
     published_records = records - failed_records
     mark_as_published(published_records)
+    schedule_republishes(records, failed_records)
     records
   end
 
@@ -36,6 +37,29 @@ class Dionysus::Producer::Outbox::RecordsProcessor
     Dionysus::Producer::Outbox::DuplicatesFilter.call(records)
   end
 
+  # `retry_at` rather than a future `created_at`: it is honoured exactly, whereas `created_at` is
+  # offset by `outbox_worker_publishing_delay` and feeds `publishing_latency`.
+  def schedule_republishes(records, failed_records)
+    return unless config.republish_deduplicated_records
+    return unless config.remove_consecutive_duplicates_before_publishing
+
+    republish_at = Time.current + config.republish_deduplicated_records_delay
+    deduplicated = Dionysus::Producer::Outbox::DuplicatesFilter.deduplicated_records(records)
+
+    (deduplicated - failed_records).each do |record|
+      next if record.observer?
+
+      outbox_model.create!(
+        resource_class: record.resource_class,
+        resource_id: record.resource_id,
+        event_name: record.event_name,
+        partition_key: record.partition_key,
+        topic: record.topic,
+        retry_at: republish_at
+      )
+    end
+  end
+
   def publish(record)
     if record.observer?
       outbox_publisher.publish_observers(record)

data/lib/dionysus/version.rb

@@ -3,5 +3,5 @@
 module Dionysus
   module Version
   end
-  VERSION = "1.4.1"
+  VERSION = "1.6.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: dionysus-rb
 version: !ruby/object:Gem::Version
-  version: 1.4.1
+  version: 1.6.0
 platform: ruby
 authors:
 - Karol Galanciak
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-08-31 00:00:00.000000000 Z
+date: 2026-09-01 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord