dionysus-rb 1.5.0 → 1.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fa1bffce325e2384847246f0125606bb90dc46d3226d32764e3ad650a74dccc4
-  data.tar.gz: 07fec41263e2726ddabdcc2cd8b5fae1dcd7452185013e2e9eda2d68b9ec6007
+  metadata.gz: a103f61431f824a5d8769b0d2ed6d3ab6b9493690b3503726ed87d8790da7dd9
+  data.tar.gz: 9d87121dd840665b4bba91771b5f3e7f2a27e753d2dd374b5e461f1db172aa8d
 SHA512:
-  metadata.gz: 76e8759abcf8533a5f3ef0559b97f3fb7fdcc379552204ef33456820108acd0271e88cc9c66f757f539731dbd0d95e574dd4afaa9e5e537e18468a191ae4c26c
-  data.tar.gz: d7f572eb5e5440bf599a38d33fb31c78d194d226a414f17890f536edf9150e58193242a761f47a46e4073d532c7001e676fe48487812a45cee5d66d711ec910b
+  metadata.gz: 784b7e7894cfabc80c1146a9ac93ce7d466a872cb3d86caec664fd1f2c1fa453d125ca2528b03e1a6d55b71fc15f87fdd54eb05cebb32639365c144a7303777d
+  data.tar.gz: b22a989af8d7a2cfe867f6ad1485ebf88f312ba3eb7e0ea0df52663cd60a552cf1576c1c7fc85ee4caffb576397c2684d06f596e8bf993c6d6b3cdf610dcf837

data/CHANGELOG.md

@@ -1,5 +1,16 @@
 ## [Unreleased]
 
+## [1.7.0]
+- Correct a row whose own `updated_at` predates a record embedded in its payload, then serialize again, behind `config.touch_records_behind_their_embedded_records` (default `false`). A serializer reads the parent's columns before the associations, so a child committed in between arrives under a timestamp older than itself, and the consumer discards the payload together with that child.
+
+## [1.6.0]
+- Judge a payload's embedded children on their own timestamps instead of dropping them with the parent. When `persist_with_dionysus?` rejects a record, `persist` used to `next` past the whole loop body - the child `persist` calls included - so every `has_many` and `has_one` record embedded in that payload was discarded with it, with no error and no counter. The parent is still not written; only the traversal of its children is decoupled from its verdict. Each child re-enters `persist` and is guarded on its own `synced_updated_at || synced_created_at`, so a child the consumer has never seen is created and a child older than the row held locally is still skipped.
+- Reproduced at the consumer, not argued: `spec/embedded_children_persistence_spec.rb` runs a whole batch through the generated consumer with deduplication in place, from two failures observed in production. In the first, all seven money-bearing messages carried an `updated_at` 44 ms older than the zero-money payload already accepted, and the payment embedded in them - a row that did not exist locally - was never created. Three control examples pin the fixture down: the same shape at a timestamp the guard accepts persists the money, creates the embedded payment and calls `resolve_to_many_association`, so a red assertion is a statement about this gem rather than about the payload being malformed.
+- `resolve_to_many_association` and `resolve_to_one_association` are suppressed for the whole subtree beneath a rejected parent. Cancelling or deleting the children absent from the payload's list is how a genuine deletion propagates, and it requires a complete current list - which is exactly what a payload already judged stale cannot provide. Consuming applications implement that hook destructively, several with `destroy_all`, and the old `next` was shielding them by accident; suppressing it deliberately keeps that behaviour rather than changing it.
+- Children beneath a rejected parent are compared strictly, through the new `SynchronizableModel#advances_dionysus_state?`. The ordinary guard accepts a tie, and a purge stamps the cancel column with `update_all` without bumping `synced_updated_at` - so a stale payload that merely ties on a child's timestamp is precisely how a cancelled child would be revived. A tie carries nothing that is not already held, so requiring the child to be demonstrably newer closes that without costing a repair. `persist_with_dionysus?` itself is untouched and behaves as before on the ordinary path. A child the payload carries no timestamp for at all is refused outright for the same reason: a payload already judged stale proves nothing about it, and where `resolve_to_many_association` hard-deletes, applying one would bring back a child a fresher payload had removed.
+- The stale verdict is sticky for the whole subtree, which is a trade rather than a free win: a child that is itself fresh now advances while its own children keep the generation they had, because the purge needs a complete list that only a non-stale ancestor can vouch for. Before this change the child did not advance either, so the subtree stayed internally consistent and equally wrong; a later payload that passes the root guard repairs it. Recorded as a pending example.
+- Not fixed here, and left as a pending example rather than a silent gap: when deduplication keeps a message whose `updated_at` the guard rejects while a sibling in the same group would have been accepted, the parent's own columns are still lost. Recovering that needs the guard - not a payload timestamp - to choose the survivor. Keeping siblings whose `updated_at` looks newer is disproven by the existing torn-payload fixture in the deduplication specs, where the correct payload carries the *lower* `updated_at`; ranking on any single key has a production counterexample, so `RemoveDuplicatesStrategy` is unchanged. A consuming application that needs the parent half today can pass `params_batch_transformation: nil` for the topic, which is covered by an example here.
+
 ## [1.5.0]
 - Publish the survivor of a collapsed run of duplicates once more after a delay, behind `config.republish_deduplicated_records` (default `false`) with `config.republish_deduplicated_records_delay` (default 30 seconds). It does nothing unless `remove_consecutive_duplicates_before_publishing` is also on, since there is no run to collapse otherwise.
 - What a collapsed run means. `DuplicatesFilter` keeps the last record of each consecutive run for the same resource, event and topic. A run longer than one only happens when the record was written again while an earlier message for it was still queued - which is the same window in which a payload can be serialized across a write and end up carrying an `updated_at` older than its own contents. `publish_consistent_snapshots` bounds its retries and publishes the last attempt regardless, so a payload that may still be torn does get published; this schedules one more publish once the writes have settled.

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    dionysus-rb (1.5.0)
+    dionysus-rb (1.7.0)
       activerecord (>= 5)
       activesupport (>= 3.2)
       concurrent-ruby

data/lib/dionysus/consumer/persistor.rb

@@ -9,14 +9,14 @@ class Dionysus::Consumer::Persistor
     @topic = topic
   end
 
-  def persist(dionysus_event, batch_number)
+  def persist(dionysus_event, batch_number, under_stale_parent: false)
     if dionysus_event.generic_event?
       if dionysus_event.created? && topic.options[:import] == true
         persist_via_dionysus_create(dionysus_event, batch_number)
       elsif dionysus_event.destroyed? && topic.options[:import] == true
         persist_via_dionysus_destroy(dionysus_event, batch_number)
       else
-        persist_standard_event(dionysus_event, batch_number)
+        persist_standard_event(dionysus_event, batch_number, under_stale_parent: under_stale_parent)
       end
     else
       log_unknown_event_type(dionysus_event)
@@ -43,12 +43,9 @@ class Dionysus::Consumer::Persistor
     config.model_factory.for_model(dionysus_event.model_name)
   end
 
-  def persist_standard_event(dionysus_event, batch_number)
+  def persist_standard_event(dionysus_event, batch_number, under_stale_parent: false)
     Array.wrap(dionysus_event.transformed_data).each do |deseralized_record|
       model_klass = find_model_klass(dionysus_event) or return
-      attributes = deseralized_record.attributes
-      has_one_relationships = deseralized_record.has_one
-      has_many_relationships = deseralized_record.has_many
       synced_id = deseralized_record.synced_id
 
       if synced_id.nil?
@@ -59,38 +56,53 @@ class Dionysus::Consumer::Persistor
       record = Dionysus::Consumer::SynchronizableModel.new(config,
         model_klass.find_or_initialize_by(config.synced_id_attribute => synced_id))
       event_updated_at = deseralized_record.synced_updated_at || deseralized_record.synced_created_at
+      persistable = persistable?(record, event_updated_at, under_stale_parent)
 
-      next unless record.persist_with_dionysus?(event_updated_at)
+      apply_record(dionysus_event, record, deseralized_record, synced_id, batch_number) if persistable
 
-      record.assign_attributes_from_dionysus(attributes)
-      if dionysus_event.destroyed?
-        instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.remove_with_dionysus") do
-          record.remove_with_dionysus(deseralized_record) if dionysus_event.aggregate_root?
-        end
-      else
-        instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.restore_with_dionysus") do
-          record.restore_with_dionysus if record.restorable?(deseralized_record)
-        end
-      end
+      # children re-enter persist and are guarded on their own timestamp; only the child list is untrusted
+      persist_relationships(dionysus_event, record, deseralized_record, batch_number,
+        under_stale_parent: under_stale_parent || !persistable)
+    end
+  end
 
-      dionysus_event.local_changes[[dionysus_event.model_name, synced_id]] = record.changes if record.changes.present?
+  def persistable?(record, event_updated_at, under_stale_parent)
+    if under_stale_parent
+      record.advances_dionysus_state?(event_updated_at)
+    else
+      record.persist_with_dionysus?(event_updated_at)
+    end
+  end
 
-      instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.save") do
-        record.save unless record.destroyed?
+  def apply_record(dionysus_event, record, deseralized_record, synced_id, batch_number)
+    record.assign_attributes_from_dionysus(deseralized_record.attributes)
+    if dionysus_event.destroyed?
+      instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.remove_with_dionysus") do
+        record.remove_with_dionysus(deseralized_record) if dionysus_event.aggregate_root?
+      end
+    else
+      instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.restore_with_dionysus") do
+        record.restore_with_dionysus if record.restorable?(deseralized_record)
       end
+    end
+
+    dionysus_event.local_changes[[dionysus_event.model_name, synced_id]] = record.changes if record.changes.present?
+
+    instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.save") do
+      record.save unless record.destroyed?
+    end
+  end
 
-      instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.persist_to_many_relationships") do
-        has_many_relationships.each do |relationship_name, relationship_records|
-          persist_to_many_relationship(dionysus_event, relationship_name, record, relationship_records,
-            batch_number)
-        end
+  def persist_relationships(dionysus_event, record, deseralized_record, batch_number, under_stale_parent:)
+    instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.persist_to_many_relationships") do
+      deseralized_record.has_many.each do |relationship|
+        persist_to_many_relationship(dionysus_event, relationship, record, batch_number, under_stale_parent: under_stale_parent)
       end
+    end
 
-      instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.persist_to_one_relationships") do
-        has_one_relationships.each do |relationship_name, relationship_record|
-          persist_to_one_relationship(dionysus_event, relationship_name, record, relationship_record,
-            batch_number)
-        end
+    instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.persist_to_one_relationships") do
+      deseralized_record.has_one.each do |relationship|
+        persist_to_one_relationship(dionysus_event, relationship, record, batch_number, under_stale_parent: under_stale_parent)
       end
     end
   end
@@ -99,7 +111,9 @@ class Dionysus::Consumer::Persistor
     Dionysus.logger.debug("[Dionysus] unknown event type #{dionysus_event.event_name}")
   end
 
-  def persist_to_one_relationship(original_event, relationship_name, parent_model_record, record, batch_number)
+  def persist_to_one_relationship(original_event, relationship, parent_model_record, batch_number,
+    under_stale_parent: false)
+    relationship_name, record = relationship
     instrumentation_arguments = {
       event_name: original_event.event_name,
       parent_model_record: parent_model_record.model_name.to_s,
@@ -115,17 +129,21 @@ class Dionysus::Consumer::Persistor
         relationship_name, records, aggregate_root: false)
 
       instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.persist_to_one_relationship.#{relationship_name}.persist") do
-        persist(dionysus_event, batch_number)
+        persist(dionysus_event, batch_number, under_stale_parent: under_stale_parent)
         original_event.local_changes.merge!(dionysus_event.local_changes)
       end
 
+      return if under_stale_parent
+
       instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.persist_to_one_relationship.#{relationship_name}.resolve_to_one_association") do
         parent_model_record.resolve_to_one_association(relationship_name, record.synced_id)
       end
     end
   end
 
-  def persist_to_many_relationship(original_event, relationship_name, parent_model_record, records, batch_number)
+  def persist_to_many_relationship(original_event, relationship, parent_model_record, batch_number,
+    under_stale_parent: false)
+    relationship_name, records = relationship
     instrumentation_arguments = {
       event_name: original_event.event_name,
       parent_model_record: parent_model_record.model_name.to_s,
@@ -140,10 +158,13 @@ class Dionysus::Consumer::Persistor
         relationship_name, records, aggregate_root: false)
 
       instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.persist_to_many_relationship.#{relationship_name}.persist") do
-        persist(dionysus_event, batch_number)
+        persist(dionysus_event, batch_number, under_stale_parent: under_stale_parent)
         original_event.local_changes.merge!(dionysus_event.local_changes)
       end
 
+      # the purge needs a complete current list, which a payload already judged stale cannot provide
+      return if under_stale_parent
+
       synced_ids_of_related_records = records.map(&:synced_id)
       instrument("dionysus.consume.#{topic}.batch_number_#{batch_number}.persist.persist_to_many_relationship.#{relationship_name}.resolve_to_many_association") do
         parent_model_record.resolve_to_many_association(relationship_name, synced_ids_of_related_records)

data/lib/dionysus/consumer/synchronizable_model.rb

@@ -25,6 +25,18 @@ class Dionysus::Consumer::SynchronizableModel < SimpleDelegator
     (synced_at && event_updated_at && event_updated_at >= synced_at) || synced_at.nil? || event_updated_at.nil?
   end
 
+  # A purge stamps the cancel column without bumping synced_updated_at, so under a stale payload a tie
+  # on the child's timestamp is how a cancelled child gets revived. Demand demonstrably newer.
+  # An unstamped child is refused outright: a payload the guard has already rejected proves nothing
+  # about a child it carries no timestamp for, and applying one would resurrect a child that a fresher
+  # payload had already deleted. Refusing it is exactly what the old blanket `next` did.
+  def advances_dionysus_state?(event_updated_at)
+    return false if event_updated_at.nil?
+
+    persist_with_dionysus?(event_updated_at) &&
+      !(synced_at && event_updated_at == synced_at)
+  end
+
   def assign_attributes_from_dionysus(attributes)
     public_send("#{synced_data_attribute}=", attributes)
     reverse_mapping = config.attributes_mapping_for_model(model.model_name).to_a.to_h(&:reverse)

data/lib/dionysus/producer/config.rb

@@ -10,7 +10,8 @@ class Dionysus::Producer::Config
     :genesis_consistency_safety_delay, :hermes_event_producer, :publish_after_commit, :outbox_worker_publishing_delay,
     :high_priority_sidekiq_queue, :observers_inline_maximum_size, :remove_consecutive_duplicates_before_publishing,
     :include_serialized_at_in_payload, :publish_with_uncached_reads, :publish_consistent_snapshots,
-    :max_snapshot_attempts, :republish_deduplicated_records, :republish_deduplicated_records_delay
+    :max_snapshot_attempts, :republish_deduplicated_records, :republish_deduplicated_records_delay,
+    :touch_records_behind_their_embedded_records
 
   def self.default_sidekiq_queue
     :dionysus
@@ -98,6 +99,14 @@ class Dionysus::Producer::Config
     @observers_inline_maximum_size || 1000
   end
 
+  # Corrects a row whose own timestamp predates a record embedded in its payload, then serializes
+  # again. Off by default: it writes to the source table from the publish path.
+  def touch_records_behind_their_embedded_records
+    return @touch_records_behind_their_embedded_records if defined?(@touch_records_behind_their_embedded_records)
+
+    false
+  end
+
   # Off by default so it can be switched on per producer, and switched back off in one env change
   # if the extra reads ever cost more than they are worth.
   # How many times a payload may be re-serialized before the last attempt is published as it stands.

data/lib/dionysus/producer/embedded_timestamp_repair.rb

@@ -0,0 +1,87 @@
+# frozen_string_literal: true
+
+class Dionysus::Producer::EmbeddedTimestampRepair
+  TIMESTAMP_ATTRIBUTE = "updated_at"
+  PRIMARY_KEY_ATTRIBUTE = "id"
+
+  # Takes the block that serializes, so a repaired record can be serialized again in one place.
+  def self.call(records, config, &)
+    new(records, config).call(&)
+  end
+
+  attr_reader :records, :config
+  private :records, :config
+
+  def initialize(records, config)
+    @records = records
+    @config = config
+  end
+
+  def call
+    payload, read_at = yield
+    return [payload, read_at] unless config.touch_records_behind_their_embedded_records
+    return [payload, read_at] unless payload.is_a?(Array)
+    return [payload, read_at] unless any_repaired?(payload)
+
+    yield
+  end
+
+  private
+
+  # Pairing is positional, and the serializer is supplied by the application, so a payload that does
+  # not line up 1:1 could touch one record with another record's children. Fail closed instead.
+  # map, not any? - any? short-circuits and would leave the rest of a batch unrepaired
+  def any_repaired?(payload)
+    return false unless payload.size == Array(records).size
+
+    Array(records).zip(payload).map { |record, record_payload| repaired?(record, record_payload) }.any?
+  end
+
+  # A parent whose own timestamp predates a record embedded in it publishes a payload that describes
+  # no single moment, and the consumer discards it with its children. Correcting the row keeps the
+  # published timestamp equal to the row's own, which is what the republish path relies on.
+  def repaired?(record, record_payload)
+    return false unless repairable?(record, record_payload)
+
+    latest = embedded_timestamps(record_payload).max
+    return false unless latest && latest > record_payload[TIMESTAMP_ATTRIBUTE]
+
+    # WHERE guards the write: a stale read must never move the row backwards. When the row is already
+    # ahead nothing is written, and the reload alone repairs the payload.
+    updated = record.class.where(record.class.primary_key => record.id)
+      .where("#{record.class.quoted_table_name}.#{TIMESTAMP_ATTRIBUTE} < ?", latest)
+      .update_all(TIMESTAMP_ATTRIBUTE => latest)
+    # the row can be deleted between the guard and here, and raising would abort the whole batch
+    record.reload
+    instrument(record, updated)
+    true
+  rescue ActiveRecord::RecordNotFound
+    false
+  end
+
+  def repairable?(record, record_payload)
+    record.is_a?(ActiveRecord::Base) && record.persisted? &&
+      record_payload.is_a?(Hash) && record_payload[PRIMARY_KEY_ATTRIBUTE] == record.id &&
+      time_like?(record_payload[TIMESTAMP_ATTRIBUTE])
+  end
+
+  def embedded_timestamps(record_payload)
+    record_payload.each_value.flat_map do |value|
+      Array.wrap(value).filter_map do |embedded|
+        next unless embedded.is_a?(Hash)
+
+        timestamp = embedded[TIMESTAMP_ATTRIBUTE]
+        timestamp if time_like?(timestamp)
+      end
+    end
+  end
+
+  def instrument(record, updated)
+    config.instrumenter.increment("dionysus.publish.timestamp_repair",
+      tags: ["model:#{record.class}", "outcome:#{updated.zero? ? "stale_read" : "row_behind"}"])
+  end
+
+  def time_like?(value)
+    value.respond_to?(:acts_like_time?) && value.acts_like_time?
+  end
+end

data/lib/dionysus/producer/karafka_responder_generator.rb

@@ -48,7 +48,8 @@ class Dionysus::Producer::KarafkaResponderGenerator
 
               # consumers rank duplicates by this stamp, so it has to describe the attempt actually
               # published - with retries that is the last one, not the first
-              payload, read_at = serialize_consistently(records, topic, batch_options)
+              payload, read_at = Dionysus::Producer::EmbeddedTimestampRepair
+                .call(records, config) { serialize_consistently(records, topic, batch_options) }
               serialized_at = config.include_serialized_at_in_payload ? read_at : nil
 
               event_payload = {

data/lib/dionysus/version.rb

@@ -3,5 +3,5 @@
 module Dionysus
   module Version
   end
-  VERSION = "1.5.0"
+  VERSION = "1.7.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: dionysus-rb
 version: !ruby/object:Gem::Version
-  version: 1.5.0
+  version: 1.7.0
 platform: ruby
 authors:
 - Karol Galanciak
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-08-31 00:00:00.000000000 Z
+date: 2026-09-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -500,6 +500,7 @@ files:
 - lib/dionysus/producer/base_responder.rb
 - lib/dionysus/producer/config.rb
 - lib/dionysus/producer/deleted_record_serializer.rb
+- lib/dionysus/producer/embedded_timestamp_repair.rb
 - lib/dionysus/producer/genesis.rb
 - lib/dionysus/producer/genesis/performed.rb
 - lib/dionysus/producer/genesis/stream_job.rb