mammoth 1.2.0 → 1.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 9169e3ff1f9fad3a07df87cee57891aea0ca6815f356f7ef998e94572a432ea4
4
- data.tar.gz: 6ee2140075cb22b1fc0aea3977a33adfd1aa49918d184a823ee3cbcab9b87bac
3
+ metadata.gz: 55bd93a58c383e07fe1930ed71f20ea52c05dbfa77a2354e258caddd726865ce
4
+ data.tar.gz: 2b01106f391f82ed0079d7efdf934447cdeb168d2ee5d6029fb8ac7cd8a617b8
5
5
  SHA512:
6
- metadata.gz: 37191be4257dbab756e5e2ba81bae60905df5a921792001daf12b5ece1bcacbf60c353967b6da5c0de4e71311a8b95311ed37e7f8f2587bcb49e7dfbae6a57bd
7
- data.tar.gz: 603479a5b0e7e067743eb6dcb8802ea2077954df50e23a3b29b97bdefb161cb1c47f493dfac919aa27d6a0552f018fcc1a7116e876acca50f71bc716f0d8c539
6
+ metadata.gz: 4359b50a25fada22578bca1ab80c3afcc187ce6d181110d83157d7231bdc9fb5f32ab407b24d39619cd7e63fa8d4a4fc01c14fd27474f640d53c874c5411eff8
7
+ data.tar.gz: 62eb55bba9825caaf491f8ed38ceb9f3da95580a57736c6998bb2349fa1980eb472e2ce7c775972ce7d7d9f8d6fa1ad43daffb22da7847b03c6b898c02e25762
data/CHANGELOG.md CHANGED
@@ -1,6 +1,41 @@
1
+ <!--
2
+ # @title Changelog
3
+ -->
4
+
1
5
  # Changelog
2
6
 
3
- ## Unreleased
7
+ ## 1.4.0 - 2026-07-19
8
+
9
+ ### Added
10
+
11
+ - Declared PostgreSQL 14 through PostgreSQL 18, inclusive, as Mammoth's
12
+ supported server range.
13
+ - Replaced the PostgreSQL 17-only CI gate with real logical-replication E2E
14
+ coverage across PostgreSQL 14, 15, 16, 17, and 18.
15
+ - Documented the maintained-version support policy, version-dependent
16
+ `pg_replication_slots` fields, and PostgreSQL 18-only
17
+ `idle_replication_slot_timeout`.
18
+
19
+ ### Fixed
20
+
21
+ - Query `pg_replication_slots.invalidation_reason` in the E2E helper only on
22
+ PostgreSQL 17 and newer; PostgreSQL 16 does not expose that field.
23
+
24
+ ## 1.3.0 - 2026-07-19
25
+
26
+ ### Added
27
+
28
+ - Added opt-in, destination-scoped payload policies that remove or mask
29
+ selected columns across event data, identity, and changes before webhook
30
+ delivery.
31
+ - Added deterministic policy fingerprints and exact prepared-payload handling
32
+ across signing, retries, dead-letter persistence, and replay.
33
+ - Added a payload-policy projection benchmark and updated the dead-letter replay
34
+ benchmark to measure exact prepared-payload replay without CDC reconstruction
35
+ or policy reapplication.
36
+ - Updated `webhooks-quickstart` to mask `orders.customer_email` by default and
37
+ highlight the applied policy and fingerprint in the Event Console and smoke
38
+ test.
4
39
 
5
40
  ## 1.2.0 - 2026-07-19
6
41
 
data/README.md CHANGED
@@ -1,13 +1,25 @@
1
+ <!--
2
+ # @markup markdown
3
+ # @title Github Repo README
4
+ # @author Ken C. Demanawa
5
+ -->
6
+
1
7
  <p align="center">
2
- <img src="https://raw.githubusercontent.com/kanutocd/mammoth/main/docs/assets/logo/mammoth-horizontal.svg" alt="Mammoth" width="520">
8
+ <picture>
9
+ <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/kanutocd/mammoth/main/docs/branding/logo/exports/png/mammoth-primary-horizontal-reversed-transparent.png">
10
+ <source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/kanutocd/mammoth/main/docs/branding/logo/exports/png/mammoth-primary-horizontal-light.png">
11
+ <img src="https://raw.githubusercontent.com/kanutocd/mammoth/main/docs/branding/logo/exports/png/mammoth-primary-horizontal-light.png" alt="Mammoth" width="620">
12
+ </picture>
3
13
  </p>
4
14
 
5
15
  # Mammoth
6
16
 
7
- [![Gem Version](https://badge.fury.io/rb/mammoth.svg)](https://badge.fury.io/rb/mammoth)
8
- [![CI](https://github.com/kanutocd/mammoth/workflows/CI/badge.svg)](https://github.com/kanutocd/mammoth/actions)
9
- [![Ruby Version](https://img.shields.io/badge/ruby-%3E%3D%204.0-ruby.svg)](https://www.ruby-lang.org/en/)
10
- [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
17
+ [![Gem Version](https://img.shields.io/gem/v/mammoth?logo=rubygems&logoColor=white)](https://rubygems.org/gems/mammoth)
18
+ [![Requires Ruby 4.0+](https://img.shields.io/badge/Requires-Ruby%204.0%2B-CC342D?logo=ruby&logoColor=white)](https://www.ruby-lang.org/)
19
+ [![CI](https://github.com/kanutocd/mammoth/actions/workflows/ci.yml/badge.svg)](https://github.com/kanutocd/mammoth/actions/workflows/ci.yml)
20
+ [![Tested Against PostgreSQL 14–18](https://img.shields.io/badge/Tested%20Against-PostgreSQL%2014--18-336791?logo=postgresql&logoColor=white)](https://github.com/kanutocd/mammoth/blob/main/.github/workflows/ci.yml#L50)
21
+ [![License](https://img.shields.io/badge/License-MIT-22C55E)](LICENSE.txt)
22
+
11
23
 
12
24
  Mammoth is a self-hosted PostgreSQL event relay focused on reliable delivery
13
25
  of database change events.
@@ -28,6 +40,20 @@ Mammoth is intentionally boring infrastructure. It uses YAML configuration,
28
40
  JSON Schema validation, local SQLite operational state, and the CDC Ecosystem's
29
41
  shared vocabulary so operators can inspect, recover, and reason about delivery.
30
42
 
43
+ ## Supported PostgreSQL versions
44
+
45
+ Mammoth supports PostgreSQL 14 through PostgreSQL 18, inclusive. These are the
46
+ PostgreSQL major versions currently maintained by the PostgreSQL community and
47
+ covered by Mammoth's real logical-replication E2E compatibility matrix.
48
+
49
+ Mammoth supports PostgreSQL major versions that are both maintained by the
50
+ PostgreSQL community and included in Mammoth's compatibility test matrix. New
51
+ PostgreSQL majors are unsupported until explicitly tested and documented. EOL
52
+ versions may be removed from the supported range in a subsequent Mammoth minor
53
+ release with release-note notice.
54
+
55
+ PostgreSQL 19 is a development release and is not supported.
56
+
31
57
  ## Documentation
32
58
 
33
59
  Documentation site:
@@ -43,13 +69,18 @@ https://kanutocd.github.io/mammoth/Mammoth.html
43
69
  The recommended first-run experience is the
44
70
  [`webhooks-quickstart`](webhooks-quickstart) stack. One Compose command starts a
45
71
  demo application, logical-replication-enabled PostgreSQL, Mammoth, a signed
46
- webhook receiver with visible retries, and Mammoth health endpoints:
72
+ webhook receiver with visible retries and payload-policy masking, and Mammoth
73
+ health endpoints:
47
74
 
48
75
  ```bash
49
76
  cd webhooks-quickstart
50
77
  docker compose up --build --wait
51
78
  ```
52
79
 
80
+ Released tags default to their matching Mammoth image. When running `main`
81
+ with Unreleased configuration features, follow the quickstart's local-image
82
+ instructions.
83
+
53
84
  Once the flow is visible, follow its
54
85
  [`ADAPTING.md`](webhooks-quickstart/ADAPTING.md) guide to connect Mammoth to an
55
86
  existing PostgreSQL application. Its optional `monitoring` profile adds seeded
@@ -72,6 +103,7 @@ Mammoth 1.x includes:
72
103
  - webhook delivery sink
73
104
  - webhook fanout to multiple destinations
74
105
  - fanout route filters by schema, table, and operation
106
+ - per-destination payload removal and masking policies
75
107
  - per-destination enable/disable and retry policy controls
76
108
  - delivery worker with retry, delivered-ledger, and DLQ handling
77
109
  - contiguous delivery watermark for checkpoint and PostgreSQL acknowledgement
@@ -99,7 +131,7 @@ modes, not isolated API snippets.
99
131
  | --- | --- |
100
132
  | [`live_postgres_webhook`](examples/live_postgres_webhook) | End-to-end PostgreSQL logical replication into webhook delivery. |
101
133
  | [`transaction_webhook`](examples/transaction_webhook) | TransactionEnvelope preservation through the concurrent runtime. |
102
- | [`webhook_fanout`](examples/webhook_fanout) | Routed multi-destination fanout, environment-backed headers, signing, and independent retry policies. |
134
+ | [`webhook_fanout`](examples/webhook_fanout) | Routed multi-destination fanout, destination payload masking, environment-backed headers, signing, and independent retry policies. |
103
135
  | [`ordering`](examples/ordering) | Ordered and throughput-oriented transaction scheduling. |
104
136
  | [`checkpoint_recovery`](examples/checkpoint_recovery) | Durable restart recovery, replay suppression, checkpointing, and acknowledgement. |
105
137
  | [`slot_invalidation_recovery`](examples/slot_invalidation_recovery) | Fail-closed slot invalidation and explicit operator reconciliation. |
@@ -155,9 +187,12 @@ maps the canonical started, succeeded, failed, and skipped notifications to
155
187
  Prometheus counters.
156
188
 
157
189
  `Mammoth::ReplicationConsumer` accepts only exact core events and transaction
158
- envelopes. Operator-facing JSON, such as `deliver-sample` input and persisted
159
- dead letters, is reconstructed by `PersistedPayloadDeserializer` before it
160
- re-enters delivery; stored hashes do not masquerade as live CDC work.
190
+ envelopes. `deliver-sample` input crosses
191
+ `PersistedPayloadDeserializer` before entering the live delivery path, so its
192
+ stored hash does not masquerade as CDC-core work. Dead-letter replay is a
193
+ separate destination boundary: Mammoth sends the exact prepared JSON persisted
194
+ for that destination and does not reconstruct CDC work or reapply the current
195
+ payload policy.
161
196
 
162
197
  ## Extensions
163
198
 
@@ -199,11 +234,18 @@ destinations:
199
234
  schemas: [public]
200
235
  tables: [orders]
201
236
  operations: [insert, update]
237
+ payload_policy:
238
+ rules:
239
+ - columns: [customer_email]
240
+ action: mask
202
241
  retry:
203
242
  max_attempts: 3
204
243
  schedule_seconds: [1, 10]
205
244
  ```
206
245
 
246
+ See [`docs/PAYLOAD-POLICIES.md`](docs/PAYLOAD-POLICIES.md) for deterministic
247
+ data minimization, retry and dead-letter behavior, and policy boundaries.
248
+
207
249
  ## CLI
208
250
 
209
251
  ```bash
@@ -288,6 +288,23 @@ webhook:
288
288
  # signature_header: X-Mammoth-Signature
289
289
  # timestamp_header: X-Mammoth-Timestamp
290
290
 
291
+ # Optional deterministic payload projection and PII redaction.
292
+ #
293
+ # Rules apply after canonical serialization and route evaluation. Matching
294
+ # columns are scrubbed consistently from data, identity, and changes before
295
+ # signing, HTTP delivery, retry, and dead-letter persistence.
296
+ #
297
+ # payload_policy:
298
+ # rules:
299
+ # - schemas: [public]
300
+ # tables: [customers, orders]
301
+ # columns: [customer_email, phone]
302
+ # action: remove
303
+ # - tables: [customers]
304
+ # columns: [external_customer_id]
305
+ # action: mask
306
+ # replacement: "[PRIVATE]"
307
+
291
308
  # Optional fanout configuration.
292
309
  #
293
310
  # Use destinations instead of webhook when one PostgreSQL replication stream
@@ -324,6 +341,14 @@ webhook:
324
341
  # - insert
325
342
  # - update
326
343
  #
344
+ # # Optional destination-specific payload policy. Other destinations still
345
+ # # receive their own independently prepared payloads.
346
+ # payload_policy:
347
+ # rules:
348
+ # - tables: [orders]
349
+ # columns: [customer_email]
350
+ # action: remove
351
+ #
327
352
  # # Optional per-destination retry override. Omit to use top-level retry.
328
353
  # retry:
329
354
  # max_attempts: 3
@@ -4,6 +4,85 @@
4
4
  "title": "Mammoth Configuration",
5
5
  "type": "object",
6
6
  "additionalProperties": false,
7
+ "definitions": {
8
+ "payloadPolicy": {
9
+ "type": "object",
10
+ "additionalProperties": false,
11
+ "required": [
12
+ "rules"
13
+ ],
14
+ "properties": {
15
+ "rules": {
16
+ "type": "array",
17
+ "minItems": 1,
18
+ "items": {
19
+ "type": "object",
20
+ "additionalProperties": false,
21
+ "required": [
22
+ "columns",
23
+ "action"
24
+ ],
25
+ "properties": {
26
+ "schemas": {
27
+ "type": "array",
28
+ "minItems": 1,
29
+ "items": {
30
+ "type": "string",
31
+ "minLength": 1
32
+ },
33
+ "description": "PostgreSQL schemas matched by this rule. Omit to match every schema."
34
+ },
35
+ "tables": {
36
+ "type": "array",
37
+ "minItems": 1,
38
+ "items": {
39
+ "type": "string",
40
+ "minLength": 1
41
+ },
42
+ "description": "PostgreSQL tables matched by this rule. Omit to match every table."
43
+ },
44
+ "operations": {
45
+ "type": "array",
46
+ "minItems": 1,
47
+ "items": {
48
+ "type": "string",
49
+ "enum": [
50
+ "insert",
51
+ "update",
52
+ "delete"
53
+ ]
54
+ },
55
+ "description": "CDC operations matched by this rule. Omit to match every operation."
56
+ },
57
+ "columns": {
58
+ "type": "array",
59
+ "minItems": 1,
60
+ "uniqueItems": true,
61
+ "items": {
62
+ "type": "string",
63
+ "minLength": 1
64
+ },
65
+ "description": "Column names scrubbed from data, identity, and changes."
66
+ },
67
+ "action": {
68
+ "type": "string",
69
+ "enum": [
70
+ "remove",
71
+ "mask"
72
+ ],
73
+ "description": "Remove matching columns or replace their non-null values."
74
+ },
75
+ "replacement": {
76
+ "type": "string",
77
+ "description": "Replacement used by mask. Defaults to [REDACTED]."
78
+ }
79
+ }
80
+ }
81
+ }
82
+ },
83
+ "description": "Deterministic destination payload projection and PII-redaction rules."
84
+ }
85
+ },
7
86
  "required": [
8
87
  "mammoth",
9
88
  "postgres",
@@ -237,6 +316,9 @@
237
316
  }
238
317
  },
239
318
  "description": "Optional HMAC-SHA256 request signing for webhook receivers."
319
+ },
320
+ "payload_policy": {
321
+ "$ref": "#/definitions/payloadPolicy"
240
322
  }
241
323
  },
242
324
  "description": "HTTP webhook destination settings. This is shorthand for a single webhook destination; use destinations for fanout."
@@ -361,6 +443,9 @@
361
443
  },
362
444
  "description": "Optional destination route filter. Event delivery matches one event; transaction delivery matches when any event in the transaction matches and delivers the full envelope."
363
445
  },
446
+ "payload_policy": {
447
+ "$ref": "#/definitions/payloadPolicy"
448
+ },
364
449
  "retry": {
365
450
  "type": "object",
366
451
  "additionalProperties": false,
@@ -30,7 +30,15 @@ module Mammoth
30
30
  @checkpoint_store = @state_adapter.checkpoint_store
31
31
  @consumer = ReplicationConsumer.new(source: source || build_source, delivery_unit: delivery_unit)
32
32
  @progress_coordinator = build_progress_coordinator
33
- @delivery_worker = sink ? build_delivery_worker(sink: sink, sleeper: sleeper) : build_configured_delivery_worker(sleeper:)
33
+ @delivery_worker = if sink
34
+ build_delivery_worker(
35
+ sink: sink,
36
+ sleeper: sleeper,
37
+ delivery_policy: destination_delivery_policy(config.data["webhook"] || {})
38
+ )
39
+ else
40
+ build_configured_delivery_worker(sleeper: sleeper)
41
+ end
34
42
  end
35
43
 
36
44
  # @return [Mammoth::SQLiteStore] underlying SQLite store for compatibility
@@ -155,8 +163,8 @@ module Mammoth
155
163
  def destination_specs
156
164
  destinations = config.data["destinations"]
157
165
  unless destinations
158
- delivery_policy = {} # : Hash[String, untyped]
159
- return [{ sink: WebhookSink.from_config(config), delivery_policy: delivery_policy }]
166
+ webhook = config.data.fetch("webhook")
167
+ return [{ sink: WebhookSink.from_config(config), delivery_policy: destination_delivery_policy(webhook) }]
160
168
  end
161
169
 
162
170
  destinations.map.with_index(1) do |destination, index|
@@ -173,7 +181,8 @@ module Mammoth
173
181
  "enabled" => destination.fetch("enabled", true),
174
182
  "max_attempts" => retry_config.fetch("max_attempts", config.dig("retry", "max_attempts")),
175
183
  "schedule_seconds" => retry_config.fetch("schedule_seconds", config.dig("retry", "schedule_seconds")),
176
- "route_filter" => RouteFilter.new(destination["route"])
184
+ "route_filter" => RouteFilter.new(destination["route"]),
185
+ "payload_policy" => PayloadPolicy.new(destination["payload_policy"])
177
186
  }
178
187
  end
179
188
 
@@ -13,6 +13,7 @@ module Mammoth
13
13
  metrics
14
14
  routing
15
15
  fanout
16
+ payload_policies
16
17
  ].freeze
17
18
 
18
19
  attr_reader :config
@@ -238,38 +238,23 @@ module Mammoth
238
238
 
239
239
  def replay_row(row)
240
240
  payload = JSON.parse(row.fetch("payload_json"))
241
- if transaction_payload?(payload)
242
- replay_transaction(row.fetch("destination_name"), transaction_envelope(payload))
243
- else
244
- replay_event(row.fetch("destination_name"), payload)
245
- end
246
- end
241
+ return replay_payload(row.fetch("destination_name"), payload) if worker.respond_to?(:deliver_payload)
247
242
 
248
- def replay_event(destination_name, payload)
249
- event = PersistedPayloadDeserializer.event(payload)
250
- return worker.deliver_to(destination_name, event) if worker.respond_to?(:deliver_to)
251
-
252
- worker.deliver(event)
243
+ replay_fanout_payload(row.fetch("destination_name"), payload)
253
244
  end
254
245
 
255
- def replay_transaction(destination_name, envelope)
256
- return worker.deliver_transaction_to(destination_name, envelope) if worker.respond_to?(:deliver_transaction_to)
246
+ def replay_payload(_destination_name, payload)
247
+ worker.deliver_payload(payload)
248
+ end
257
249
 
258
- worker.deliver_transaction(envelope)
250
+ def replay_fanout_payload(destination_name, payload)
251
+ worker.deliver_payload_to(destination_name, payload)
259
252
  end
260
253
 
261
254
  def replay_resolved?(result)
262
255
  result.fetch(:status) == "delivered" || result.fetch(:duplicate, false)
263
256
  end
264
257
 
265
- def transaction_payload?(payload)
266
- payload.fetch("type", nil) == TransactionEnvelopeSerializer::PAYLOAD_TYPE
267
- end
268
-
269
- def transaction_envelope(payload)
270
- PersistedPayloadDeserializer.transaction(payload)
271
- end
272
-
273
258
  def show_payload(row)
274
259
  row.merge("payload" => JSON.parse(row.fetch("payload_json")))
275
260
  end
@@ -20,9 +20,27 @@ module Mammoth
20
20
  # @param error [Exception, nil] delivery failure
21
21
  # @param retry_count [Integer] number of delivery attempts
22
22
  # @return [Integer] inserted dead letter id
23
- def write(event:, destination_name:, error: nil, retry_count: 0, serializer: EventSerializer) # rubocop:disable Metrics/MethodLength
23
+ def write(event:, destination_name:, error: nil, retry_count: 0, serializer: EventSerializer)
24
+ write_payload(
25
+ payload: serializer.call(event),
26
+ destination_name: destination_name,
27
+ error: error,
28
+ retry_count: retry_count
29
+ )
30
+ end
31
+
32
+ # Store the exact payload prepared for one destination.
33
+ #
34
+ # Persisting the prepared payload prevents retries and replay from restoring
35
+ # fields removed by a destination payload policy.
36
+ #
37
+ # @param payload [Hash] prepared destination payload
38
+ # @param destination_name [String] destination name
39
+ # @param error [Exception, nil] delivery failure
40
+ # @param retry_count [Integer] number of delivery attempts
41
+ # @return [Integer] inserted dead letter id
42
+ def write_payload(payload:, destination_name:, error: nil, retry_count: 0) # rubocop:disable Metrics/MethodLength
24
43
  now = Time.now.utc.iso8601
25
- payload = serializer.call(event)
26
44
  database.execute(
27
45
  <<~SQL,
28
46
  INSERT INTO dead_letters(
@@ -12,7 +12,7 @@ module Mammoth
12
12
  DEFAULT_SOURCE = "postgresql"
13
13
 
14
14
  attr_reader :sink, :checkpoint_store, :dead_letter_store, :delivered_envelope_store, :retry_schedule, :max_attempts,
15
- :sleeper, :source_name, :slot_name, :publication_name, :route_filter, :enabled
15
+ :sleeper, :source_name, :slot_name, :publication_name, :route_filter, :payload_policy, :enabled
16
16
 
17
17
  # @param sink [#deliver] destination sink
18
18
  # @param checkpoint_store [Mammoth::CheckpointStore] retained dependency; shared progress owns writes
@@ -25,10 +25,11 @@ module Mammoth
25
25
  # @param retry_schedule [Array<Integer>] retry wait schedule in seconds
26
26
  # @param sleeper [#call] sleep strategy, injectable for tests
27
27
  # @param route_filter [Mammoth::RouteFilter, nil] optional destination route matcher
28
+ # @param payload_policy [Mammoth::PayloadPolicy, nil] destination payload transformation policy
28
29
  # @param enabled [Boolean] whether this destination accepts new deliveries
29
30
  def initialize(sink:, checkpoint_store:, dead_letter_store:, delivered_envelope_store:, source_name:, slot_name:,
30
31
  publication_name:, max_attempts:, retry_schedule:, sleeper: Kernel.method(:sleep),
31
- route_filter: nil, enabled: true)
32
+ route_filter: nil, payload_policy: nil, enabled: true)
32
33
  @sink = sink
33
34
  @checkpoint_store = checkpoint_store
34
35
  @dead_letter_store = dead_letter_store
@@ -40,7 +41,11 @@ module Mammoth
40
41
  @retry_schedule = retry_schedule
41
42
  @sleeper = sleeper
42
43
  @route_filter = route_filter || RouteFilter.new
44
+ @payload_policy = payload_policy || PayloadPolicy.new
43
45
  @enabled = enabled
46
+ return unless @payload_policy.active? && !sink.respond_to?(:deliver_payload)
47
+
48
+ raise ConfigurationError, "destination #{destination_name} does not accept prepared payloads"
44
49
  end
45
50
 
46
51
  # Build a delivery worker from Mammoth configuration and stores.
@@ -66,6 +71,7 @@ module Mammoth
66
71
  retry_schedule: delivery_policy.fetch("schedule_seconds", config.dig("retry", "schedule_seconds")),
67
72
  sleeper: sleeper,
68
73
  route_filter: delivery_policy.fetch("route_filter", RouteFilter.new),
74
+ payload_policy: delivery_policy.fetch("payload_policy", PayloadPolicy.new),
69
75
  enabled: delivery_policy.fetch("enabled", true)
70
76
  )
71
77
  end
@@ -86,16 +92,32 @@ module Mammoth
86
92
  deliver_work(event, serializer: EventSerializer, delivery_method: :deliver)
87
93
  end
88
94
 
95
+ # Replay one exact destination payload without reapplying the current policy.
96
+ #
97
+ # @param payload [Hash] prepared payload persisted in the dead-letter store
98
+ # @return [Hash] delivery summary
99
+ def deliver_payload(payload)
100
+ prepared = PreparedDelivery.from_payload(payload)
101
+ delivery_method = transaction_payload?(payload) ? :deliver_transaction : :deliver
102
+ deliver_prepared(prepared, work: nil, delivery_method: delivery_method)
103
+ end
104
+
89
105
  private
90
106
 
91
- # rubocop:disable Metrics/MethodLength
92
107
  def deliver_work(work, serializer:, delivery_method:)
108
+ prepared = PreparedDelivery.build(work, serializer: serializer, payload_policy: payload_policy)
109
+ deliver_prepared(prepared, work: work, delivery_method: delivery_method)
110
+ end
111
+
112
+ # rubocop:disable Metrics/MethodLength
113
+ def deliver_prepared(prepared, work:, delivery_method:)
93
114
  attempts = 0
94
- payload = serializer.call(work)
115
+ canonical_payload = prepared.canonical_payload
116
+ payload = prepared.payload
95
117
  delivery_unit = delivery_unit_for(delivery_method)
96
- idempotency_key = idempotency_key_for(payload:, delivery_unit:)
118
+ idempotency_key = idempotency_key_for(payload: canonical_payload, delivery_unit: delivery_unit)
97
119
 
98
- skip_result = skip_result_for(payload, idempotency_key:)
120
+ skip_result = skip_result_for(canonical_payload, idempotency_key: idempotency_key)
99
121
  return skip_result if skip_result
100
122
 
101
123
  if delivered_envelope_store.delivered?(idempotency_key)
@@ -110,7 +132,7 @@ module Mammoth
110
132
 
111
133
  begin
112
134
  attempts += 1
113
- result = sink.public_send(delivery_method, work)
135
+ result = deliver_to_sink(prepared, work: work, delivery_method: delivery_method)
114
136
  delivered_envelope_store.record!(
115
137
  idempotency_key: idempotency_key,
116
138
  source_name: source_name,
@@ -122,7 +144,7 @@ module Mammoth
122
144
  )
123
145
  result.merge(attempts: attempts, idempotency_key: idempotency_key)
124
146
  rescue DeliveryError => e
125
- return dead_letter(work, e, attempts, serializer:) if attempts >= max_attempts
147
+ return dead_letter(payload, e, attempts) if attempts >= max_attempts
126
148
 
127
149
  wait_before_retry(attempts)
128
150
  retry
@@ -166,20 +188,30 @@ module Mammoth
166
188
  }
167
189
  end
168
190
 
169
- def dead_letter(event, error, attempts, serializer:)
170
- id = dead_letter_store.write(
171
- event: event,
191
+ def wait_before_retry(attempts)
192
+ wait_seconds = retry_schedule.fetch(attempts - 1, retry_schedule.last)
193
+ sleeper.call(wait_seconds)
194
+ end
195
+
196
+ def deliver_to_sink(prepared, work:, delivery_method:)
197
+ return sink.deliver_payload(prepared.payload) if sink.respond_to?(:deliver_payload)
198
+ return sink.public_send(delivery_method, work) if work && !payload_policy.active?
199
+
200
+ raise ConfigurationError, "destination #{destination_name} does not accept prepared payloads"
201
+ end
202
+
203
+ def dead_letter(payload, error, attempts)
204
+ id = dead_letter_store.write_payload(
205
+ payload: payload,
172
206
  destination_name: sink.name,
173
207
  error: error,
174
- retry_count: attempts,
175
- serializer: serializer
208
+ retry_count: attempts
176
209
  )
177
210
  { status: "dead_lettered", dead_letter_id: id, attempts: attempts }
178
211
  end
179
212
 
180
- def wait_before_retry(attempts)
181
- wait_seconds = retry_schedule.fetch(attempts - 1, retry_schedule.last)
182
- sleeper.call(wait_seconds)
213
+ def transaction_payload?(payload)
214
+ payload["type"] == TransactionEnvelopeSerializer::PAYLOAD_TYPE
183
215
  end
184
216
  end
185
217
  end
@@ -21,7 +21,13 @@ module Mammoth
21
21
 
22
22
  # @return [Hash] JSON-friendly capabilities
23
23
  def capabilities
24
- { type: adapter_type, delivery_units: %w[event transaction], signing: true, header_env: true }
24
+ {
25
+ type: adapter_type,
26
+ delivery_units: %w[event transaction],
27
+ prepared_payloads: true,
28
+ signing: true,
29
+ header_env: true
30
+ }
25
31
  end
26
32
  end
27
33
  end
@@ -46,6 +46,15 @@ module Mammoth
46
46
  worker_for(destination_name).deliver_transaction(envelope)
47
47
  end
48
48
 
49
+ # Replay one exact prepared payload to one configured destination.
50
+ #
51
+ # @param destination_name [String] destination name
52
+ # @param payload [Hash] prepared destination payload
53
+ # @return [Hash] destination delivery summary
54
+ def deliver_payload_to(destination_name, payload)
55
+ worker_for(destination_name).deliver_payload(payload)
56
+ end
57
+
49
58
  private
50
59
 
51
60
  def fanout(delivery_method, work)
@@ -0,0 +1,149 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "digest"
4
+ require "json"
5
+
6
+ module Mammoth
7
+ # Applies deterministic destination-scoped projection and redaction rules.
8
+ #
9
+ # Policies operate on serialized Mammoth payloads, never on CDC-core work
10
+ # items. Column rules scrub every canonical row-value representation: data,
11
+ # identity, and old/new values in changes.
12
+ class PayloadPolicy
13
+ DEFAULT_MASK = "[REDACTED]"
14
+ POLICY_METADATA_KEY = "mammoth_payload_policy"
15
+ SUPPORTED_ACTIONS = %w[remove mask].freeze
16
+
17
+ attr_reader :config, :fingerprint
18
+
19
+ # @param config [Hash, nil] validated payload-policy configuration
20
+ def initialize(config = nil)
21
+ @config = deep_freeze(stringify_keys(config || {}))
22
+ validate!
23
+ @fingerprint = build_fingerprint if active?
24
+ end
25
+
26
+ # @return [Boolean] whether the policy contains transformation rules
27
+ def active?
28
+ rules.any?
29
+ end
30
+
31
+ # Transform one canonical event or transaction payload.
32
+ #
33
+ # @param payload [Hash] canonical serialized Mammoth payload
34
+ # @return [Hash] independent JSON-compatible destination payload
35
+ def apply(payload)
36
+ return payload unless active?
37
+
38
+ transformed = JSON.parse(JSON.generate(payload))
39
+ events_for(transformed).each { |event| apply_rules(event) }
40
+ metadata = transformed["metadata"]
41
+ metadata = {} unless metadata.is_a?(Hash) # steep:ignore
42
+ transformed["metadata"] = metadata
43
+ metadata[POLICY_METADATA_KEY] = { "fingerprint" => fingerprint }
44
+ transformed
45
+ end
46
+
47
+ private
48
+
49
+ def rules
50
+ config.fetch("rules", [])
51
+ end
52
+
53
+ def validate!
54
+ rules.each_with_index do |rule, index|
55
+ action = rule["action"].to_s
56
+ unless SUPPORTED_ACTIONS.include?(action)
57
+ raise ConfigurationError, "payload policy rule #{index} has unsupported action #{action}"
58
+ end
59
+ raise ConfigurationError, "payload policy rule #{index} requires columns" if Array(rule["columns"]).empty?
60
+ end
61
+ end
62
+
63
+ def events_for(payload)
64
+ return payload.fetch("events") if payload["type"] == TransactionEnvelopeSerializer::PAYLOAD_TYPE
65
+
66
+ [payload]
67
+ end
68
+
69
+ def apply_rules(event)
70
+ rules.each do |rule|
71
+ apply_rule(event, rule) if match?(event, rule)
72
+ end
73
+ end
74
+
75
+ def apply_rule(event, rule)
76
+ columns = rule.fetch("columns")
77
+ if rule.fetch("action") == "remove"
78
+ remove_columns(event, columns)
79
+ else
80
+ mask_columns(event, columns, rule.fetch("replacement", DEFAULT_MASK))
81
+ end
82
+ end
83
+
84
+ def remove_columns(event, columns)
85
+ %w[data identity].each do |field|
86
+ columns.each { |column| event[field]&.delete(column) }
87
+ end
88
+ changes = Array(event["changes"]) # : Array[untyped]
89
+ event["changes"] = changes.reject { |change| columns.include?(change["name"]) }
90
+ end
91
+
92
+ def mask_columns(event, columns, replacement)
93
+ %w[data identity].each do |field|
94
+ columns.each { |column| mask_map_value(event[field], column, replacement) }
95
+ end
96
+ changes = Array(event["changes"]) # : Array[untyped]
97
+ changes.each do |change|
98
+ next unless columns.include?(change["name"])
99
+
100
+ %w[old_value new_value].each do |value_key|
101
+ change[value_key] = replacement if change.key?(value_key) && !change[value_key].nil?
102
+ end
103
+ end
104
+ end
105
+
106
+ def mask_map_value(map, column, replacement)
107
+ return unless map.is_a?(Hash) && map.key?(column)
108
+ return if map[column].nil?
109
+
110
+ map[column] = replacement
111
+ end
112
+
113
+ def match?(event, rule)
114
+ matches?(rule["schemas"], event["namespace"]) &&
115
+ matches?(rule["tables"], event["entity"]) &&
116
+ matches?(rule["operations"], event["operation"])
117
+ end
118
+
119
+ def matches?(configured, actual)
120
+ values = Array(configured) # : Array[untyped]
121
+ values.empty? || values.any? { |value| value.to_s.casecmp?(actual.to_s) }
122
+ end
123
+
124
+ def build_fingerprint
125
+ canonical = canonicalize(config)
126
+ "sha256:#{Digest::SHA256.hexdigest(JSON.generate(canonical))}"
127
+ end
128
+
129
+ def canonicalize(value)
130
+ return value.map { |item| canonicalize(item) } if value.is_a?(Array)
131
+ return value unless value.is_a?(Hash)
132
+
133
+ value.keys.sort.to_h { |key| [key, canonicalize(value.fetch(key))] }
134
+ end
135
+
136
+ def stringify_keys(value)
137
+ return value.map { |item| stringify_keys(item) } if value.is_a?(Array)
138
+ return value unless value.is_a?(Hash)
139
+
140
+ value.to_h { |key, item| [key.to_s, stringify_keys(item)] }
141
+ end
142
+
143
+ def deep_freeze(value)
144
+ value.each_value { |item| deep_freeze(item) } if value.is_a?(Hash)
145
+ value.each { |item| deep_freeze(item) } if value.is_a?(Array)
146
+ value.freeze
147
+ end
148
+ end
149
+ end
@@ -0,0 +1,30 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Mammoth
4
+ # Immutable delivery boundary containing canonical identity and outbound data.
5
+ PreparedDelivery = Data.define(:canonical_payload, :payload, :policy_fingerprint) do
6
+ # Serialize and apply one destination payload policy.
7
+ #
8
+ # @param work [CDC::Core::ChangeEvent, CDC::Core::TransactionEnvelope]
9
+ # @param serializer [#call] canonical Mammoth serializer
10
+ # @param payload_policy [Mammoth::PayloadPolicy]
11
+ # @return [Mammoth::PreparedDelivery]
12
+ def self.build(work, serializer:, payload_policy:)
13
+ canonical_payload = serializer.call(work)
14
+ new(
15
+ canonical_payload: canonical_payload,
16
+ payload: payload_policy.apply(canonical_payload),
17
+ policy_fingerprint: payload_policy.fingerprint
18
+ )
19
+ end
20
+
21
+ # Reconstruct a prepared delivery from a persisted destination payload.
22
+ #
23
+ # @param payload [Hash] previously prepared destination payload
24
+ # @return [Mammoth::PreparedDelivery]
25
+ def self.from_payload(payload)
26
+ fingerprint = payload.dig("metadata", PayloadPolicy::POLICY_METADATA_KEY, "fingerprint")
27
+ new(canonical_payload: payload, payload: payload, policy_fingerprint: fingerprint)
28
+ end
29
+ end
30
+ end
@@ -2,5 +2,5 @@
2
2
 
3
3
  module Mammoth
4
4
  # Current Mammoth gem version.
5
- VERSION = "1.2.0"
5
+ VERSION = "1.4.0"
6
6
  end
@@ -110,8 +110,14 @@ module Mammoth
110
110
  deliver_payload(TransactionEnvelopeSerializer.call(envelope))
111
111
  end
112
112
 
113
- private
114
-
113
+ # Deliver an already prepared destination payload.
114
+ #
115
+ # The caller owns canonical serialization and payload-policy application.
116
+ # WebhookSink signs and sends exactly this payload.
117
+ #
118
+ # @param payload [Hash] prepared JSON-compatible destination payload
119
+ # @return [Hash] delivery result
120
+ # @raise [Mammoth::DeliveryError] when delivery fails
115
121
  def deliver_payload(payload)
116
122
  response = perform_request(payload)
117
123
  return delivery_result(payload, response) if SUCCESS_RANGE.cover?(response.code.to_i)
@@ -121,6 +127,8 @@ module Mammoth
121
127
  raise DeliveryError, "webhook #{name} delivery failed: #{e.message}"
122
128
  end
123
129
 
130
+ private
131
+
124
132
  def perform_request(payload)
125
133
  Net::HTTP.start(url.host, url.port, use_ssl: url.scheme == "https", open_timeout: timeout_seconds,
126
134
  read_timeout: timeout_seconds) do |http|
data/lib/mammoth.rb CHANGED
@@ -33,6 +33,8 @@ require_relative "mammoth/event_serializer"
33
33
  require_relative "mammoth/transaction_envelope_serializer"
34
34
  require_relative "mammoth/persisted_payload_deserializer"
35
35
  require_relative "mammoth/route_filter"
36
+ require_relative "mammoth/payload_policy"
37
+ require_relative "mammoth/prepared_delivery"
36
38
  require_relative "mammoth/webhook_sink"
37
39
  require_relative "mammoth/operational_state/adapter"
38
40
  require_relative "mammoth/operational_state/sqlite_adapter"
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: mammoth
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.2.0
4
+ version: 1.4.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Ken C. Demanawa
@@ -219,8 +219,10 @@ files:
219
219
  - lib/mammoth/operational_state/adapter.rb
220
220
  - lib/mammoth/operational_state/registry.rb
221
221
  - lib/mammoth/operational_state/sqlite_adapter.rb
222
+ - lib/mammoth/payload_policy.rb
222
223
  - lib/mammoth/persisted_payload_deserializer.rb
223
224
  - lib/mammoth/postgres_observability_metrics.rb
225
+ - lib/mammoth/prepared_delivery.rb
224
226
  - lib/mammoth/registry.rb
225
227
  - lib/mammoth/replication_consumer.rb
226
228
  - lib/mammoth/route_filter.rb