mammoth 0.4.0 → 0.5.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8fdbb7afb90f584e11024c75d7300f54978f5e5b1a3628597bb460f96a306c22
-  data.tar.gz: 9ced2250d7e2f95558a8bcce51809aa4d670d49953d8d970e168a0c48cf90215
+  metadata.gz: c04c055c5cf5faedef6100513bc63a9ada3c69b99b904f701d0be93501aa4ebd
+  data.tar.gz: 15e75ed7ed39723f5f0864253196a830ae0413d6609ae504640be2eca7d6b2da
 SHA512:
-  metadata.gz: cd6b5b3246cf70adb9beb0cc9243e6a35788d20f3442ab406bca66a4cb08bd4a7d80639386c4e42291b518190a0198f960ed9e16cbd165f335a7f3fee10317b7
-  data.tar.gz: 87aa4ee77c38aa69d1a505bf7eb2b838de86504316840dfc26b4e875870333f64da906c85aa180a205e5e3bff4f439e36d2f82ffa79d650117d9f5ccc4c29b69
+  metadata.gz: 1d0b5446e4b5edd10f5d82007dd8ae85e7e9e40f056725ef1faa5e090c2fa844a9007c94c7ac47b41cb7eeec6fe8d9be3eb0dec7685a757da749ff7dbf85ccbd
+  data.tar.gz: 3f3185bc9743f100bab26a62f2fa45f94f04aa3243e787ab05d0b13aafc697a22ab432a84da1e40dfd7214e880bafe727e17c8ca4e15236ca97547f55166687c

data/CHANGELOG.md

@@ -2,6 +2,26 @@
 
 ## Unreleased
 
+## 0.5.1
+
+### Added
+
+- Added benchmark scripts for webhook delivery, webhook fanout, SQLite operational state, observability snapshots, and dead-letter replay.
+- Added shared benchmark helpers for synthetic CDC work items, local HTTP receivers, table output, JSON output, and environment-driven benchmark knobs.
+- Added a benchmark snapshot runner that writes Markdown, JSON, and per-trial logs for publishable benchmark results.
+
+### Changed
+
+- Aligned benchmark documentation across `README.md`, `docs/BENCHMARKS.md`, and `benchmark/README.md` around operator config tuning.
+
+## 0.5.0
+
+### Added
+
+- Added multi-destination webhook fanout through `destinations`, with independent per-destination retry, delivered-ledger, and dead-letter state.
+- Added targeted fanout dead-letter replay so replay sends failed work back only to the original destination.
+- Added Helm rendering for fanout destinations, including secret-backed environment variables for Authorization headers and HMAC signing secrets.
+
 ## 0.4.0
 
 ### Added

data/README.md

@@ -17,7 +17,7 @@ CDC::Core::TransactionEnvelope
       ↓
 Mammoth
       ↓
-Webhook
+Webhook fanout
 ```
 
 🦣 Mammoth is intentionally boring infrastructure. It uses YAML configuration,
@@ -46,6 +46,7 @@ https://kanutocd.github.io/mammoth/Mammoth.html
 - checkpoint persistence
 - dead letter persistence
 - webhook delivery sink
+- webhook fanout to multiple destinations
 - delivery worker with retry, checkpoint, and DLQ handling
 - dead-letter inspection and replay commands
 - CDC-core event serialization boundary
@@ -53,11 +54,11 @@ https://kanutocd.github.io/mammoth/Mammoth.html
 - Docker image support
 - public Helm chart support
 - unit and e2e test tasks
-- Health and metrics endpoints
+- health and metrics endpoints
 
 ## Boundary
 
-Mammoth begins at CDC-core work items and ends at webhook delivery.
+Mammoth begins at CDC-core work items and ends at webhook fanout delivery.
 
 Mammoth does not own pgoutput protocol parsing, value decoding, source
 normalization, ordering policy, or runtime execution. Those belong to the
@@ -103,50 +104,31 @@ Mammoth stores operational memory in SQLite:
 - `schema_migrations`
 - `checkpoints`
 - `dead_letters`
+- `delivered_envelopes`
 
 ## Performance
 
-Mammoth scales downstream delivery throughput using the `cdc-concurrent` runtime while maintaining a single PostgreSQL logical replication stream.
+Mammoth includes local benchmarks for the product surfaces operators tune in
+production:
 
-### Concurrent Delivery Benchmark
+- concurrent delivery runtime
+- real webhook delivery
+- multi-destination webhook fanout
+- SQLite operational state
+- observability snapshots
+- DLQ replay
 
-Environment:
+The historical numbers in [Benchmarks](https://github.com/kanutocd/mammoth/tree/main/docs/BENCHMARKS.md) are retained as a
+snapshot, not a universal performance claim. Re-run the scripts in
+[`benchmark/`](https://github.com/kanutocd/mammoth/tree/main/benchmark) on your own hardware when choosing
+`runtime.concurrency`, `destinations`, SQLite storage, scrape frequency, and
+DLQ replay expectations.
 
-* 10,000 transactions
-* 4 events per transaction
-* 40,000 total events
+Create a publishable benchmark snapshot with:
 
-#### Fast Downstream (10ms)
-
-| Concurrency | Transactions/sec |
-| ----------- | ---------------: |
-| 1           |            96.50 |
-| 25          |          2419.65 |
-
-#### Realistic Webhook (50ms)
-
-| Concurrency | Transactions/sec |
-| ----------- | ---------------: |
-| 1           |            19.85 |
-| 25          |           495.11 |
-
-Observed throughput improved from:
-
-- 96.50 → 2419.65 transactions/sec (10ms sink latency)
-- 19.85 → 495.11 transactions/sec (50ms sink latency)
-
-when increasing delivery concurrency from 1 to 25.
-
-The benchmark demonstrates near-linear scaling of delivery throughput without increasing PostgreSQL replication connections.
-
-See:
-
-- docs/BENCHMARKS.md
-- examples/transaction_webhook
-
-For full benchmark methodology and results see:
-
-See [Benchmarks](docs/BENCHMARKS.md).
+```bash
+bundle exec ruby benchmark/snapshot.rb
+```
 
 ## E2E
 

data/config/mammoth.example.yml

@@ -2,9 +2,10 @@
 
 # Mammoth example configuration.
 #
-# This file is intentionally verbose. It is both a runnable starting point and
-# the primary operator-facing documentation for local development, Kubernetes
-# values, and production deployments.
+# This file is intentionally verbose. It is the primary operator-facing
+# reference for local development, Kubernetes values, and production
+# deployments. Update URLs, environment variables, and PostgreSQL settings
+# before using it as a runtime config.
 #
 # Architecture reminder:
 #
@@ -24,7 +25,7 @@
 mammoth:
   # Human-readable Mammoth instance name.
   #
-  # Used in logs, status output, metrics, and future operational dashboards.
+  # Used in logs, status output, metrics, and operational dashboards.
   # Keep this stable per environment.
   #
   # Examples:
@@ -177,8 +178,7 @@ runtime:
   #
   # concurrent:
   #   Use CDC::Concurrent::ProcessorPool for I/O-heavy delivery workloads such as
-  #   webhook HTTP calls, retry sleeps/backoff, and future multi-destination
-  #   fanout.
+  #   webhook HTTP calls, retry sleeps/backoff, and multi-destination fanout.
   adapter: concurrent
 
   # Number of downstream delivery workers.
@@ -193,6 +193,12 @@ runtime:
   #   webhook-heavy production: 10-50, after soak testing
   concurrency: 1
 
+  # Number of work units submitted to the runtime together.
+  #
+  # Keep this at 1 unless you have validated downstream ordering, retry, and
+  # receiver idempotency behavior under load.
+  batch_size: 1
+
   # Preserve configured ordering while using the runtime.
   #
   # true:
@@ -206,11 +212,17 @@ runtime:
   # Optional runtime timeout in seconds.
   #
   # Leave blank to let destination-specific timeouts control delivery behavior.
-  # Useful later for bounding end-to-end processor execution.
+  # Use this to bound end-to-end processor execution.
   timeout_seconds:
 
+# Single webhook destination shorthand.
+#
+# Use this for one destination. Use destinations below instead when Mammoth
+# should fan out to multiple webhook receivers. Do not configure both unless you
+# intentionally want destinations to replace this shorthand.
 webhook:
-  # Destination name used in logs, dead-letter records, and future metrics.
+  # Destination name used in logs, dead-letter records, delivered ledgers, and
+  # metrics.
   name: primary_webhook
 
   # Destination URL for delivered CDC payloads.
@@ -230,6 +242,7 @@ webhook:
     X-Mammoth-Source: local_mammoth
 
   # Optional HTTP headers whose values are read from environment variables.
+  # In Kubernetes, set these variables from Secrets.
   #
   # Example:
   #   header_env:
@@ -244,6 +257,7 @@ webhook:
   #
   # using the secret from secret_env, then sends the timestamp and signature in
   # configurable headers. The signature value is prefixed with "sha256=".
+  # In Kubernetes, set the secret_env variable from a Secret.
   #
   # signing:
   #   algorithm: hmac_sha256
@@ -251,6 +265,28 @@ webhook:
   #   signature_header: X-Mammoth-Signature
   #   timestamp_header: X-Mammoth-Timestamp
 
+# Optional fanout configuration.
+#
+# Use destinations instead of webhook when one PostgreSQL replication stream
+# should deliver the same CDC work item to multiple webhook receivers. Each
+# destination gets independent retry, delivered-ledger, and dead-letter state.
+# The destinations list replaces the webhook shorthand above.
+#
+# destinations:
+#   - name: primary_webhook
+#     type: webhook
+#     url: https://example.com/webhooks/postgres
+#     timeout_seconds: 5
+#     header_env:
+#       Authorization: MAMMOTH_PRIMARY_WEBHOOK_AUTHORIZATION
+#     signing:
+#       algorithm: hmac_sha256
+#       secret_env: MAMMOTH_PRIMARY_WEBHOOK_SIGNING_SECRET
+#   - name: audit_webhook
+#     type: webhook
+#     url: https://audit.example.com/cdc
+#     timeout_seconds: 5
+
 retry:
   # Maximum delivery attempts before the event/envelope is moved to the dead
   # letter store.
@@ -270,8 +306,8 @@ retry:
 sqlite:
   # Local operational database path.
   #
-  # Mammoth stores checkpoints, delivery attempts, and dead-letter records here.
-  # In Kubernetes, back this path with a PersistentVolumeClaim.
+  # Mammoth stores checkpoints, delivered-envelope ledgers, and dead-letter
+  # records here. In Kubernetes, back this path with a PersistentVolumeClaim.
   path: data/mammoth.db
 
 observability:

data/config/mammoth.schema.json

@@ -8,11 +8,22 @@
     "mammoth",
     "postgres",
     "replication",
-    "webhook",
     "retry",
     "sqlite",
     "logging"
   ],
+  "anyOf": [
+    {
+      "required": [
+        "webhook"
+      ]
+    },
+    {
+      "required": [
+        "destinations"
+      ]
+    }
+  ],
   "properties": {
     "mammoth": {
       "type": "object",
@@ -187,7 +198,93 @@
           "description": "Optional HMAC-SHA256 request signing for webhook receivers."
         }
       },
-      "description": "HTTP webhook destination settings."
+      "description": "HTTP webhook destination settings. This is shorthand for a single webhook destination; use destinations for fanout."
+    },
+    "destinations": {
+      "type": "array",
+      "minItems": 1,
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": [
+          "name",
+          "type",
+          "url",
+          "timeout_seconds"
+        ],
+        "properties": {
+          "name": {
+            "type": "string",
+            "minLength": 1,
+            "description": "Destination name used in delivered ledgers, dead-letter records, logs, and metrics."
+          },
+          "type": {
+            "type": "string",
+            "enum": [
+              "webhook"
+            ],
+            "description": "Destination adapter type. Mammoth OSS 0.5.1 supports webhook destinations."
+          },
+          "url": {
+            "type": "string",
+            "format": "uri",
+            "description": "Webhook URL for delivered CDC payloads. Transaction delivery sends a TransactionEnvelope payload; event delivery sends a single event payload."
+          },
+          "timeout_seconds": {
+            "type": "integer",
+            "minimum": 1,
+            "description": "HTTP timeout in seconds for this destination's delivery attempts."
+          },
+          "headers": {
+            "type": "object",
+            "additionalProperties": {
+              "type": "string"
+            },
+            "description": "Static HTTP headers added to every request for this destination. Do not put secrets here; use header_env for secret-backed values."
+          },
+          "header_env": {
+            "type": "object",
+            "additionalProperties": {
+              "type": "string",
+              "minLength": 1
+            },
+            "description": "HTTP headers whose values are read from environment variables. Keys are header names; values are environment variable names."
+          },
+          "signing": {
+            "type": "object",
+            "additionalProperties": false,
+            "required": [
+              "secret_env"
+            ],
+            "properties": {
+              "algorithm": {
+                "type": "string",
+                "enum": [
+                  "hmac_sha256"
+                ],
+                "description": "Webhook signing algorithm. Mammoth signs '<timestamp>.<body>' using HMAC-SHA256."
+              },
+              "secret_env": {
+                "type": "string",
+                "minLength": 1,
+                "description": "Environment variable containing the webhook signing secret."
+              },
+              "signature_header": {
+                "type": "string",
+                "minLength": 1,
+                "description": "Header name used for the generated signature. Defaults to X-Mammoth-Signature."
+              },
+              "timestamp_header": {
+                "type": "string",
+                "minLength": 1,
+                "description": "Header name used for the signing timestamp. Defaults to X-Mammoth-Timestamp."
+              }
+            },
+            "description": "Optional HMAC-SHA256 request signing for this webhook destination."
+          }
+        }
+      },
+      "description": "Fanout destination list. Each CDC work item is delivered independently to each configured destination."
     },
     "retry": {
       "type": "object",

data/lib/mammoth/application.rb

@@ -21,7 +21,7 @@ module Mammoth
       @sqlite_store = SQLiteStore.connect(config.dig("sqlite", "path")).bootstrap!
       @checkpoint_store = CheckpointStore.new(sqlite_store)
       @consumer = ReplicationConsumer.new(source: source || build_source, delivery_unit: delivery_unit)
-      @delivery_worker = build_delivery_worker(sink: sink || WebhookSink.from_config(config), sleeper: sleeper)
+      @delivery_worker = sink ? build_delivery_worker(sink: sink, sleeper: sleeper) : build_configured_delivery_worker(sleeper:)
     end
 
     # Start the application runtime and deliver consumed CDC work.
@@ -97,6 +97,22 @@ module Mammoth
       )
     end
 
+    def build_configured_delivery_worker(sleeper:)
+      workers = destination_sinks.map { |sink| build_delivery_worker(sink:, sleeper:) }
+      return workers.fetch(0) if workers.one?
+
+      FanoutDeliveryWorker.new(workers)
+    end
+
+    def destination_sinks
+      destinations = config.data["destinations"]
+      return [WebhookSink.from_config(config)] unless destinations
+
+      destinations.map.with_index(1) do |destination, index|
+        WebhookSink.from_destination_config(destination, label: "destinations[#{index - 1}]")
+      end
+    end
+
     def transaction_delivery?
       delivery_unit == :transaction
     end

data/lib/mammoth/configuration.rb

@@ -44,6 +44,7 @@ module Mammoth
       raise ConfigurationError, "configuration must be a YAML mapping" unless data.is_a?(Hash)
 
       validate_schema!
+      validate_destination_names!
       self
     rescue Psych::SyntaxError => e
       raise ConfigurationError, "invalid YAML in #{path}: #{e.message}"
@@ -73,5 +74,16 @@ module Mammoth
     def schema_error_message(errors)
       (["configuration failed schema validation:"] + errors.map { |error| "- #{error}" }).join("\n")
     end
+
+    def validate_destination_names!
+      destinations = data["destinations"]
+      return unless destinations
+
+      names = destinations.map { |destination| destination["name"] }
+      duplicates = names.tally.select { |_name, count| count > 1 }.keys
+      return if duplicates.empty?
+
+      raise ConfigurationError, "destination names must be unique: #{duplicates.join(", ")}"
+    end
   end
 end

data/lib/mammoth/dead_letter_commands.rb

@@ -163,7 +163,23 @@ module Mammoth
 
     def replay_row(row)
       payload = JSON.parse(row.fetch("payload_json"))
-      transaction_payload?(payload) ? worker.deliver_transaction(transaction_envelope(payload)) : worker.deliver(payload)
+      if transaction_payload?(payload)
+        replay_transaction(row.fetch("destination_name"), transaction_envelope(payload))
+      else
+        replay_event(row.fetch("destination_name"), payload)
+      end
+    end
+
+    def replay_event(destination_name, payload)
+      return worker.deliver_to(destination_name, payload) if worker.respond_to?(:deliver_to)
+
+      worker.deliver(payload)
+    end
+
+    def replay_transaction(destination_name, envelope)
+      return worker.deliver_transaction_to(destination_name, envelope) if worker.respond_to?(:deliver_transaction_to)
+
+      worker.deliver_transaction(envelope)
     end
 
     def transaction_payload?(payload)

data/lib/mammoth/fanout_delivery_worker.rb

@@ -0,0 +1,73 @@
+# frozen_string_literal: true
+
+module Mammoth
+  # Delivers one CDC work item to multiple destination-specific workers.
+  class FanoutDeliveryWorker
+    attr_reader :delivery_workers
+
+    # @param delivery_workers [Array<Mammoth::DeliveryWorker>] destination workers
+    def initialize(delivery_workers)
+      raise ConfigurationError, "at least one destination is required" if delivery_workers.empty?
+
+      @delivery_workers = delivery_workers
+    end
+
+    # Deliver an event to every configured destination.
+    #
+    # @param event [Hash, #to_h] normalized event
+    # @return [Hash] aggregate fanout summary
+    def deliver(event)
+      fanout(:deliver, event)
+    end
+
+    # Deliver a transaction envelope to every configured destination.
+    #
+    # @param envelope [#events, #transaction_id] CDC transaction envelope
+    # @return [Hash] aggregate fanout summary
+    def deliver_transaction(envelope)
+      fanout(:deliver_transaction, envelope)
+    end
+
+    # Deliver an event to one configured destination.
+    #
+    # @param destination_name [String] destination name
+    # @param event [Hash, #to_h] normalized event
+    # @return [Hash] destination delivery summary
+    def deliver_to(destination_name, event)
+      worker_for(destination_name).deliver(event)
+    end
+
+    # Deliver a transaction envelope to one configured destination.
+    #
+    # @param destination_name [String] destination name
+    # @param envelope [#events, #transaction_id] CDC transaction envelope
+    # @return [Hash] destination delivery summary
+    def deliver_transaction_to(destination_name, envelope)
+      worker_for(destination_name).deliver_transaction(envelope)
+    end
+
+    private
+
+    def fanout(delivery_method, work)
+      results = delivery_workers.map { |worker| worker.public_send(delivery_method, work) }
+      {
+        status: aggregate_status(results),
+        destinations: results,
+        delivered: results.count { |result| result.fetch(:status) == "delivered" },
+        skipped: results.count { |result| result.fetch(:status) == "skipped" },
+        dead_lettered: results.count { |result| result.fetch(:status) == "dead_lettered" }
+      }
+    end
+
+    def aggregate_status(results)
+      return "fanout_delivered" if results.all? { |result| %w[delivered skipped].include?(result.fetch(:status)) }
+
+      "fanout_partial"
+    end
+
+    def worker_for(destination_name)
+      delivery_workers.find { |worker| worker.send(:destination_name) == destination_name } ||
+        raise(ConfigurationError, "destination not configured: #{destination_name}")
+    end
+  end
+end

data/lib/mammoth/status.rb

@@ -30,7 +30,7 @@ module Mammoth
       puts "Replication publications: #{Array(config.dig("replication", "publications")).join(", ")}"
       puts "Runtime: not started"
       puts "SQLite: #{sqlite_path}"
-      puts "Webhook: #{config.dig("webhook", "name")}"
+      puts "Destinations: #{destination_names.join(", ")}"
       print_store_state if sqlite_store
     end
 
@@ -40,6 +40,13 @@ module Mammoth
       config.dig("sqlite", "path")
     end
 
+    def destination_names
+      destinations = config.data["destinations"]
+      return destinations.map { |destination| destination.fetch("name") } if destinations
+
+      [config.dig("webhook", "name")]
+    end
+
     def print_store_state
       store = sqlite_store.bootstrap!
       puts "Tables: #{store.tables.join(", ")}"

data/lib/mammoth/version.rb

@@ -2,5 +2,5 @@
 
 module Mammoth
   # Current Mammoth gem version.
-  VERSION = "0.4.0"
+  VERSION = "0.5.1"
 end

data/lib/mammoth/webhook_sink.rb

@@ -39,43 +39,52 @@ module Mammoth
       # @param config [Mammoth::Configuration] loaded configuration
       # @return [Mammoth::WebhookSink]
       def from_config(config)
+        from_destination_config(config.data["webhook"], label: "webhook")
+      end
+
+      # Build a sink from one destination configuration entry.
+      #
+      # @param destination [Hash] destination configuration
+      # @param label [String] configuration path used in error messages
+      # @return [Mammoth::WebhookSink]
+      def from_destination_config(destination, label: "destination")
         new(
-          name: config.dig("webhook", "name"),
-          url: config.dig("webhook", "url"),
-          timeout_seconds: config.dig("webhook", "timeout_seconds"),
-          headers: configured_headers(config),
-          signing: configured_signing(config)
+          name: destination.fetch("name"),
+          url: destination.fetch("url"),
+          timeout_seconds: destination.fetch("timeout_seconds"),
+          headers: configured_headers(destination, label: label),
+          signing: configured_signing(destination, label: label)
         )
       end
 
       private
 
-      def configured_headers(config)
-        static_headers = config.dig("webhook", "headers") || {}
-        env_headers = config.dig("webhook", "header_env") || {}
+      def configured_headers(destination, label:)
+        static_headers = destination["headers"] || {}
+        env_headers = destination["header_env"] || {}
 
-        static_headers.merge(resolve_env_headers(env_headers))
+        static_headers.merge(resolve_env_headers(env_headers, label: label))
       end
 
-      def resolve_env_headers(env_headers)
+      def resolve_env_headers(env_headers, label:)
         env_headers.each_with_object(Hash.new) do |(header, env_name), resolved| # rubocop:disable Style/EmptyLiteral
           resolved[header] = ENV.fetch(env_name) do
-            raise ConfigurationError, "webhook.header_env.#{header} references missing environment variable #{env_name}"
+            raise ConfigurationError, "#{label}.header_env.#{header} references missing environment variable #{env_name}"
           end
         end
       end
 
-      def configured_signing(config)
-        signing = config.dig("webhook", "signing")
+      def configured_signing(destination, label:)
+        signing = destination["signing"]
         return unless signing
 
         algorithm = signing.fetch("algorithm", SIGNING_ALGORITHM)
-        raise ConfigurationError, "webhook.signing.algorithm must be #{SIGNING_ALGORITHM}" unless algorithm == SIGNING_ALGORITHM
+        raise ConfigurationError, "#{label}.signing.algorithm must be #{SIGNING_ALGORITHM}" unless algorithm == SIGNING_ALGORITHM
 
         secret_env = signing.fetch("secret_env")
         {
           secret: ENV.fetch(secret_env) do
-            raise ConfigurationError, "webhook.signing.secret_env references missing environment variable #{secret_env}"
+            raise ConfigurationError, "#{label}.signing.secret_env references missing environment variable #{secret_env}"
           end,
           signature_header: signing.fetch("signature_header", "X-Mammoth-Signature"),
           timestamp_header: signing.fetch("timestamp_header", "X-Mammoth-Timestamp")

data/lib/mammoth.rb

@@ -14,6 +14,7 @@ require_relative "mammoth/event_serializer"
 require_relative "mammoth/transaction_envelope_serializer"
 require_relative "mammoth/webhook_sink"
 require_relative "mammoth/delivery_worker"
+require_relative "mammoth/fanout_delivery_worker"
 require_relative "mammoth/delivery_processor"
 require_relative "mammoth/concurrent_delivery_runtime"
 require_relative "mammoth/sources/postgres"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: mammoth
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.5.1
 platform: ruby
 authors:
 - Ken C. Demanawa
@@ -157,6 +157,7 @@ files:
 - lib/mammoth/delivery_worker.rb
 - lib/mammoth/errors.rb
 - lib/mammoth/event_serializer.rb
+- lib/mammoth/fanout_delivery_worker.rb
 - lib/mammoth/observability_server.rb
 - lib/mammoth/observability_snapshot.rb
 - lib/mammoth/replication_consumer.rb