dionysus-rb 1.4.1 → 1.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: aea1780da49737ea39a412112502ccefec1921bcaa8e4d9e3f4a17c01e359cb1
-  data.tar.gz: 0de793797246fa1aa63209eb9713529df3bf3b51aedca2d03be485ab3b140f43
+  metadata.gz: fa1bffce325e2384847246f0125606bb90dc46d3226d32764e3ad650a74dccc4
+  data.tar.gz: 07fec41263e2726ddabdcc2cd8b5fae1dcd7452185013e2e9eda2d68b9ec6007
 SHA512:
-  metadata.gz: 9ba62b20fdf34088c43f28bfd71e4fa721b4d512125ca030c16c61eb077354aab5751a05c1ce0192708a7fc87ccd3d3af0a63b8a2f06a48ab903858c928d4ce8
-  data.tar.gz: d67700b960048a4ea6b272d20d193ba4232115665a5a72d0b2171cec6ba089c3bb451ab36fd5092ffbec55a763fca95fa7b8b7299b3f44263122e58aea26932f
+  metadata.gz: 76e8759abcf8533a5f3ef0559b97f3fb7fdcc379552204ef33456820108acd0271e88cc9c66f757f539731dbd0d95e574dd4afaa9e5e537e18468a191ae4c26c
+  data.tar.gz: d7f572eb5e5440bf599a38d33fb31c78d194d226a414f17890f536edf9150e58193242a761f47a46e4073d532c7001e676fe48487812a45cee5d66d711ec910b

data/CHANGELOG.md

@@ -1,5 +1,15 @@
 ## [Unreleased]
 
+## [1.5.0]
+- Publish the survivor of a collapsed run of duplicates once more after a delay, behind `config.republish_deduplicated_records` (default `false`) with `config.republish_deduplicated_records_delay` (default 30 seconds). It does nothing unless `remove_consecutive_duplicates_before_publishing` is also on, since there is no run to collapse otherwise.
+- What a collapsed run means. `DuplicatesFilter` keeps the last record of each consecutive run for the same resource, event and topic. A run longer than one only happens when the record was written again while an earlier message for it was still queued - which is the same window in which a payload can be serialized across a write and end up carrying an `updated_at` older than its own contents. `publish_consistent_snapshots` bounds its retries and publishes the last attempt regardless, so a payload that may still be torn does get published; this schedules one more publish once the writes have settled.
+- Why a republish is sufficient, and why it needs no change on the consumer. The outbox stores a pointer, not a payload: `Outbox::Publisher#publish` re-reads the row with `find_by`, so the scheduled record is serialized fresh at publish time and carries the settled state. The consumer's guard is `event_updated_at >= synced_at`, and `synced_at` is its mirror of the row's `updated_at` as of the last accepted message - a value the row's own `updated_at` can never fall below. A freshly-read republish is therefore accepted by construction, so nothing has to be forced past the guard and no timestamp has to be altered to make it win.
+- Scheduled with `retry_at`, not a future `created_at`. `fetch_publishable` already honours `retry_at` exactly, whereas `created_at` is compared against `Time.current + outbox_worker_publishing_delay` - so a future `created_at` publishes early by the length of that look-ahead window, and would report a negative `publishing_latency`. `retry_at` leaves `created_at` honest; `failed_at` and `error_class` stay `nil`, so a scheduled republish is distinguishable from a failed record awaiting retry.
+- Termination. A scheduled record that comes due on its own is a run of one, so it schedules nothing further. If the record is still being written when it comes due, it collapses again and schedules once more, which is the intended behaviour - the extra load is bounded at one additional record per key per batch, and pile-ups for the same key collapse through the same filter.
+- Observer records are skipped: they carry a changeset and publish through `publish_observers`, a different path.
+- Records that failed to publish are skipped too - they are already retried by `handle_error`'s backoff, so scheduling a republish for them would duplicate that.
+- Operational note: the scheduled records sit unpublished for the delay, so they raise the average and maximum reported by the outbox latency gauges. That is a truthful measurement - those messages really do wait - but any alert threshold on outbox latency should be checked before switching this on.
+
 ## [1.4.1]
 - Stamp `serialized_at` when the attempt that is actually published begins, not before the retry loop. 1.4.0 takes the stamp once and then re-serializes up to `max_snapshot_attempts` times, so a message can be published a whole retry cycle after its own stamp - and the payload it publishes is the one read last, not the one the stamp describes. Consumers rank duplicates by `[serialized_at, offset]`, so the freshest payload ended up carrying the oldest stamp and losing its group to a staler sibling. That is the failure `serialized_at` was added to prevent, reintroduced by the guard added to prevent a different one.
 - Measured on the affected topic over 65 minutes: 205 of 6,572 messages (3.12%) carried an `updated_at` LATER than their own `serialized_at`, the worst by 1,225 ms. A timestamp cannot describe a moment after the one it was taken at, so that alone proves the stamp preceded the payload.

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    dionysus-rb (1.4.1)
+    dionysus-rb (1.5.0)
       activerecord (>= 5)
       activesupport (>= 3.2)
       concurrent-ruby

data/README.md

@@ -336,6 +336,8 @@ Dionysus::Producer.configure do |config|
   config.publish_after_commit = true # not required, defaults to `false`. Check `Publishing records right after the transaction is committed` section for more details.
   config.outbox_worker_publishing_delay = 5 # non required, defaults to 0 a delay in seconds until the outbox record is considered publishable. Check `Publishing records right after the transaction is committed` section for more details.
   config.remove_consecutive_duplicates_before_publishing = true # not required, defaults to false. If set to true, the consecutive duplicates in the publishable batch will be removed and only one message will be published to a given topic. For example, if for whatever reason there are ten messages in a row for a given topic to publish `user_updated` ecent, only the last will be published. Check `Dionysus::Consumer::ParamsBatchTransformations::RemoveDuplicatesStrategy` for exact implementation. To verify if this feature is useful, it's recommended to browse Karafka UI and check messages in the topics if there are any obvious duplicates happening often.
+  config.republish_deduplicated_records = true # not required, defaults to false. Only has an effect with `remove_consecutive_duplicates_before_publishing` enabled. When a run of consecutive duplicates collapses, the surviving record is scheduled to publish once more after a delay - a run only collapses when the record was written again while an earlier message for it was still queued, which is the window in which a payload can be serialized across a write. Check `Republishing deduplicated records` section for more details.
+  config.republish_deduplicated_records_delay = 60 # not required, defaults to 30 (seconds). Only relevant with `republish_deduplicated_records` enabled. How long the scheduled republish waits - it has to outlast the burst of writes that produced the duplicates, or the republish is serialized inside the same contended window it exists to escape.
   config.observers_inline_maximum_size = 100 # not required, defaults to 1000. This config setting matters in case there is a huge amount of dependent records (observers). If the threshold is exceeded, the observers will be published via Genesis process to not cause issues like blocking the outbox worker.
   config.publish_consistent_snapshots = true # not required, defaults to `false`. A serializer reads a record's own columns and then queries its associations, so a write landing in between produces a payload whose `updated_at` predates the records embedded next to it - consumers rank and guard on that timestamp, so such a payload is discarded along with its embedded associations. When enabled, the row's timestamp is read back (bypassing the query cache) after serializing and the payload is re-serialized if the record moved. Emits the `dionysus.publish.consistent_snapshot` counter described below.
   config.max_snapshot_attempts = 2 # not required, defaults to 3. Only relevant with `publish_consistent_snapshots` enabled. A record written faster than it serializes fails the check on every attempt, so it pays the full serialization cost this many times over and still publishes a payload that may be torn - lower this to bound that cost. Set it to 1 to keep the check and its metric while disabling re-serialization entirely.
@@ -355,6 +357,28 @@ With `publish_consistent_snapshots` enabled, every guarded message increments `d
 
 Watch the `exhausted` rate before and after a rollout: the retry cost scales with write rate multiplied by serialization span, so the hottest records are both the most expensive to guard and the least likely to converge.
 
+##### Republishing deduplicated records
+
+`remove_consecutive_duplicates_before_publishing` keeps only the last record of each consecutive run for the same resource, event and topic. A run longer than one only happens when the record was written again while an earlier message for it was still queued - the same window in which a payload can be serialized across a write and end up carrying an `updated_at` older than its own contents. `publish_consistent_snapshots` bounds its retries and publishes the last attempt regardless, so a payload that may still be torn does get published.
+
+With `republish_deduplicated_records` enabled, the survivor of every collapsed run is scheduled to publish once more, `republish_deduplicated_records_delay` seconds later:
+
+```ruby
+config.remove_consecutive_duplicates_before_publishing = true
+config.republish_deduplicated_records = true
+config.republish_deduplicated_records_delay = 30
+```
+
+This needs no change on the consumer side. The outbox stores a pointer rather than a payload - `Outbox::Publisher#publish` re-reads the row with `find_by` - so the scheduled record is serialized fresh when it comes due and carries the settled state. The consumer's guard is `event_updated_at >= synced_at`, and `synced_at` is the consumer's mirror of the row's `updated_at` as of the last accepted message, a value the row's own `updated_at` can never fall below. A freshly-read republish is therefore accepted rather than discarded as stale, so nothing has to be forced past the guard and no timestamp has to be altered to make it win.
+
+The republish is scheduled with `retry_at`, not a future `created_at`. `fetch_publishable` honours `retry_at` exactly, whereas `created_at` is compared against `Time.current + outbox_worker_publishing_delay` - a future `created_at` would publish early by the length of that look-ahead window, and would report a negative `publishing_latency`. `failed_at` and `error_class` stay `nil`, so a scheduled republish stays distinguishable from a record awaiting an error retry.
+
+A scheduled record that comes due on its own is a run of one, so it schedules nothing further. If the record is still being written when it comes due it collapses again and schedules once more, which is the intent; the extra load is bounded at one additional record per key per batch, and pile-ups for the same key collapse through the same filter.
+
+Observer records are skipped - they carry a changeset that decides which observers fire, and they publish through `publish_observers`. Records that failed to publish are skipped too, since `handle_error`'s backoff already retries them.
+
+Before enabling this, check any alert threshold on outbox latency. The scheduled records sit unpublished for the delay, so they raise the average and maximum reported by the `"#{namespace}.dionysus.producer.outbox.latency.*"` gauges. That is a truthful measurement - those messages really do wait - but it will move the gauge.
+
 ##### DionysusOutbox model
 
 Generate a model for the outbox:

data/lib/dionysus/producer/config.rb

@@ -10,7 +10,7 @@ class Dionysus::Producer::Config
     :genesis_consistency_safety_delay, :hermes_event_producer, :publish_after_commit, :outbox_worker_publishing_delay,
     :high_priority_sidekiq_queue, :observers_inline_maximum_size, :remove_consecutive_duplicates_before_publishing,
     :include_serialized_at_in_payload, :publish_with_uncached_reads, :publish_consistent_snapshots,
-    :max_snapshot_attempts
+    :max_snapshot_attempts, :republish_deduplicated_records, :republish_deduplicated_records_delay
 
   def self.default_sidekiq_queue
     :dionysus
@@ -137,4 +137,19 @@ class Dionysus::Producer::Config
 
     false
   end
+
+  # Publish the survivor of a collapsed run of duplicates once more, after a delay, so a payload
+  # that may have been serialized across a write is followed by one read after the writes settled.
+  # Off by default; does nothing unless remove_consecutive_duplicates_before_publishing is also on.
+  def republish_deduplicated_records
+    return @republish_deduplicated_records if defined?(@republish_deduplicated_records)
+
+    false
+  end
+
+  # Has to outlast the burst of writes that produced the duplicates, or the republish is serialized
+  # inside the same contended window it exists to escape.
+  def republish_deduplicated_records_delay
+    (@republish_deduplicated_records_delay || 30).to_d.seconds
+  end
 end

data/lib/dionysus/producer/outbox/duplicates_filter.rb

@@ -5,6 +5,10 @@ class Dionysus::Producer::Outbox::DuplicatesFilter
     new(records_to_publish).call
   end
 
+  def self.deduplicated_records(records_to_publish)
+    new(records_to_publish).deduplicated_records
+  end
+
   attr_reader :records_to_publish
   private     :records_to_publish
 
@@ -13,13 +17,23 @@ class Dionysus::Producer::Outbox::DuplicatesFilter
   end
 
   def call
-    records_to_publish
-      .slice_when { |record_1, record_2| generate_uniqueness_key(record_1) != generate_uniqueness_key(record_2) }
-      .flat_map(&:last)
+    consecutive_runs.flat_map(&:last)
+  end
+
+  # The survivor of every run that collapsed something: the record was written again while an
+  # earlier message for it was still queued.
+  def deduplicated_records
+    consecutive_runs.select { |run| run.size > 1 }.map(&:last)
   end
 
   private
 
+  def consecutive_runs
+    @consecutive_runs ||= records_to_publish
+      .slice_when { |record_1, record_2| generate_uniqueness_key(record_1) != generate_uniqueness_key(record_2) }
+      .to_a
+  end
+
   def generate_uniqueness_key(record)
     [record.resource_class, record.resource_id, record.event_name, record.topic]
   end

data/lib/dionysus/producer/outbox/records_processor.rb

@@ -23,6 +23,7 @@ class Dionysus::Producer::Outbox::RecordsProcessor
     end
     published_records = records - failed_records
     mark_as_published(published_records)
+    schedule_republishes(records, failed_records)
     records
   end
 
@@ -36,6 +37,29 @@ class Dionysus::Producer::Outbox::RecordsProcessor
     Dionysus::Producer::Outbox::DuplicatesFilter.call(records)
   end
 
+  # `retry_at` rather than a future `created_at`: it is honoured exactly, whereas `created_at` is
+  # offset by `outbox_worker_publishing_delay` and feeds `publishing_latency`.
+  def schedule_republishes(records, failed_records)
+    return unless config.republish_deduplicated_records
+    return unless config.remove_consecutive_duplicates_before_publishing
+
+    republish_at = Time.current + config.republish_deduplicated_records_delay
+    deduplicated = Dionysus::Producer::Outbox::DuplicatesFilter.deduplicated_records(records)
+
+    (deduplicated - failed_records).each do |record|
+      next if record.observer?
+
+      outbox_model.create!(
+        resource_class: record.resource_class,
+        resource_id: record.resource_id,
+        event_name: record.event_name,
+        partition_key: record.partition_key,
+        topic: record.topic,
+        retry_at: republish_at
+      )
+    end
+  end
+
   def publish(record)
     if record.observer?
       outbox_publisher.publish_observers(record)

data/lib/dionysus/version.rb

@@ -3,5 +3,5 @@
 module Dionysus
   module Version
   end
-  VERSION = "1.4.1"
+  VERSION = "1.5.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: dionysus-rb
 version: !ruby/object:Gem::Version
-  version: 1.4.1
+  version: 1.5.0
 platform: ruby
 authors:
 - Karol Galanciak