mammoth 1.2.0 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +36 -1
- data/README.md +52 -10
- data/config/mammoth.example.yml +25 -0
- data/config/mammoth.schema.json +85 -0
- data/lib/mammoth/application.rb +13 -4
- data/lib/mammoth/capabilities.rb +1 -0
- data/lib/mammoth/dead_letter_commands.rb +7 -22
- data/lib/mammoth/dead_letter_store.rb +20 -2
- data/lib/mammoth/delivery_worker.rb +48 -16
- data/lib/mammoth/destinations/webhook_adapter.rb +7 -1
- data/lib/mammoth/fanout_delivery_worker.rb +9 -0
- data/lib/mammoth/payload_policy.rb +149 -0
- data/lib/mammoth/prepared_delivery.rb +30 -0
- data/lib/mammoth/version.rb +1 -1
- data/lib/mammoth/webhook_sink.rb +10 -2
- data/lib/mammoth.rb +2 -0
- metadata +3 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 55bd93a58c383e07fe1930ed71f20ea52c05dbfa77a2354e258caddd726865ce
|
|
4
|
+
data.tar.gz: 2b01106f391f82ed0079d7efdf934447cdeb168d2ee5d6029fb8ac7cd8a617b8
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 4359b50a25fada22578bca1ab80c3afcc187ce6d181110d83157d7231bdc9fb5f32ab407b24d39619cd7e63fa8d4a4fc01c14fd27474f640d53c874c5411eff8
|
|
7
|
+
data.tar.gz: 62eb55bba9825caaf491f8ed38ceb9f3da95580a57736c6998bb2349fa1980eb472e2ce7c775972ce7d7d9f8d6fa1ad43daffb22da7847b03c6b898c02e25762
|
data/CHANGELOG.md
CHANGED
|
@@ -1,6 +1,41 @@
|
|
|
1
|
+
<!--
|
|
2
|
+
# @title Changelog
|
|
3
|
+
-->
|
|
4
|
+
|
|
1
5
|
# Changelog
|
|
2
6
|
|
|
3
|
-
##
|
|
7
|
+
## 1.4.0 - 2026-07-19
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
|
|
11
|
+
- Declared PostgreSQL 14 through PostgreSQL 18, inclusive, as Mammoth's
|
|
12
|
+
supported server range.
|
|
13
|
+
- Replaced the PostgreSQL 17-only CI gate with real logical-replication E2E
|
|
14
|
+
coverage across PostgreSQL 14, 15, 16, 17, and 18.
|
|
15
|
+
- Documented the maintained-version support policy, version-dependent
|
|
16
|
+
`pg_replication_slots` fields, and PostgreSQL 18-only
|
|
17
|
+
`idle_replication_slot_timeout`.
|
|
18
|
+
|
|
19
|
+
### Fixed
|
|
20
|
+
|
|
21
|
+
- Query `pg_replication_slots.invalidation_reason` in the E2E helper only on
|
|
22
|
+
PostgreSQL 17 and newer; PostgreSQL 16 does not expose that field.
|
|
23
|
+
|
|
24
|
+
## 1.3.0 - 2026-07-19
|
|
25
|
+
|
|
26
|
+
### Added
|
|
27
|
+
|
|
28
|
+
- Added opt-in, destination-scoped payload policies that remove or mask
|
|
29
|
+
selected columns across event data, identity, and changes before webhook
|
|
30
|
+
delivery.
|
|
31
|
+
- Added deterministic policy fingerprints and exact prepared-payload handling
|
|
32
|
+
across signing, retries, dead-letter persistence, and replay.
|
|
33
|
+
- Added a payload-policy projection benchmark and updated the dead-letter replay
|
|
34
|
+
benchmark to measure exact prepared-payload replay without CDC reconstruction
|
|
35
|
+
or policy reapplication.
|
|
36
|
+
- Updated `webhooks-quickstart` to mask `orders.customer_email` by default and
|
|
37
|
+
highlight the applied policy and fingerprint in the Event Console and smoke
|
|
38
|
+
test.
|
|
4
39
|
|
|
5
40
|
## 1.2.0 - 2026-07-19
|
|
6
41
|
|
data/README.md
CHANGED
|
@@ -1,13 +1,25 @@
|
|
|
1
|
+
<!--
|
|
2
|
+
# @markup markdown
|
|
3
|
+
# @title Github Repo README
|
|
4
|
+
# @author Ken C. Demanawa
|
|
5
|
+
-->
|
|
6
|
+
|
|
1
7
|
<p align="center">
|
|
2
|
-
<
|
|
8
|
+
<picture>
|
|
9
|
+
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/kanutocd/mammoth/main/docs/branding/logo/exports/png/mammoth-primary-horizontal-reversed-transparent.png">
|
|
10
|
+
<source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/kanutocd/mammoth/main/docs/branding/logo/exports/png/mammoth-primary-horizontal-light.png">
|
|
11
|
+
<img src="https://raw.githubusercontent.com/kanutocd/mammoth/main/docs/branding/logo/exports/png/mammoth-primary-horizontal-light.png" alt="Mammoth" width="620">
|
|
12
|
+
</picture>
|
|
3
13
|
</p>
|
|
4
14
|
|
|
5
15
|
# Mammoth
|
|
6
16
|
|
|
7
|
-
[](https://rubygems.org/gems/mammoth)
|
|
18
|
+
[](https://www.ruby-lang.org/)
|
|
19
|
+
[](https://github.com/kanutocd/mammoth/actions/workflows/ci.yml)
|
|
20
|
+
[](https://github.com/kanutocd/mammoth/blob/main/.github/workflows/ci.yml#L50)
|
|
21
|
+
[](LICENSE.txt)
|
|
22
|
+
|
|
11
23
|
|
|
12
24
|
Mammoth is a self-hosted PostgreSQL event relay focused on reliable delivery
|
|
13
25
|
of database change events.
|
|
@@ -28,6 +40,20 @@ Mammoth is intentionally boring infrastructure. It uses YAML configuration,
|
|
|
28
40
|
JSON Schema validation, local SQLite operational state, and the CDC Ecosystem's
|
|
29
41
|
shared vocabulary so operators can inspect, recover, and reason about delivery.
|
|
30
42
|
|
|
43
|
+
## Supported PostgreSQL versions
|
|
44
|
+
|
|
45
|
+
Mammoth supports PostgreSQL 14 through PostgreSQL 18, inclusive. These are the
|
|
46
|
+
PostgreSQL major versions currently maintained by the PostgreSQL community and
|
|
47
|
+
covered by Mammoth's real logical-replication E2E compatibility matrix.
|
|
48
|
+
|
|
49
|
+
Mammoth supports PostgreSQL major versions that are both maintained by the
|
|
50
|
+
PostgreSQL community and included in Mammoth's compatibility test matrix. New
|
|
51
|
+
PostgreSQL majors are unsupported until explicitly tested and documented. EOL
|
|
52
|
+
versions may be removed from the supported range in a subsequent Mammoth minor
|
|
53
|
+
release with release-note notice.
|
|
54
|
+
|
|
55
|
+
PostgreSQL 19 is a development release and is not supported.
|
|
56
|
+
|
|
31
57
|
## Documentation
|
|
32
58
|
|
|
33
59
|
Documentation site:
|
|
@@ -43,13 +69,18 @@ https://kanutocd.github.io/mammoth/Mammoth.html
|
|
|
43
69
|
The recommended first-run experience is the
|
|
44
70
|
[`webhooks-quickstart`](webhooks-quickstart) stack. One Compose command starts a
|
|
45
71
|
demo application, logical-replication-enabled PostgreSQL, Mammoth, a signed
|
|
46
|
-
webhook receiver with visible retries, and Mammoth
|
|
72
|
+
webhook receiver with visible retries and payload-policy masking, and Mammoth
|
|
73
|
+
health endpoints:
|
|
47
74
|
|
|
48
75
|
```bash
|
|
49
76
|
cd webhooks-quickstart
|
|
50
77
|
docker compose up --build --wait
|
|
51
78
|
```
|
|
52
79
|
|
|
80
|
+
Released tags default to their matching Mammoth image. When running `main`
|
|
81
|
+
with Unreleased configuration features, follow the quickstart's local-image
|
|
82
|
+
instructions.
|
|
83
|
+
|
|
53
84
|
Once the flow is visible, follow its
|
|
54
85
|
[`ADAPTING.md`](webhooks-quickstart/ADAPTING.md) guide to connect Mammoth to an
|
|
55
86
|
existing PostgreSQL application. Its optional `monitoring` profile adds seeded
|
|
@@ -72,6 +103,7 @@ Mammoth 1.x includes:
|
|
|
72
103
|
- webhook delivery sink
|
|
73
104
|
- webhook fanout to multiple destinations
|
|
74
105
|
- fanout route filters by schema, table, and operation
|
|
106
|
+
- per-destination payload removal and masking policies
|
|
75
107
|
- per-destination enable/disable and retry policy controls
|
|
76
108
|
- delivery worker with retry, delivered-ledger, and DLQ handling
|
|
77
109
|
- contiguous delivery watermark for checkpoint and PostgreSQL acknowledgement
|
|
@@ -99,7 +131,7 @@ modes, not isolated API snippets.
|
|
|
99
131
|
| --- | --- |
|
|
100
132
|
| [`live_postgres_webhook`](examples/live_postgres_webhook) | End-to-end PostgreSQL logical replication into webhook delivery. |
|
|
101
133
|
| [`transaction_webhook`](examples/transaction_webhook) | TransactionEnvelope preservation through the concurrent runtime. |
|
|
102
|
-
| [`webhook_fanout`](examples/webhook_fanout) | Routed multi-destination fanout, environment-backed headers, signing, and independent retry policies. |
|
|
134
|
+
| [`webhook_fanout`](examples/webhook_fanout) | Routed multi-destination fanout, destination payload masking, environment-backed headers, signing, and independent retry policies. |
|
|
103
135
|
| [`ordering`](examples/ordering) | Ordered and throughput-oriented transaction scheduling. |
|
|
104
136
|
| [`checkpoint_recovery`](examples/checkpoint_recovery) | Durable restart recovery, replay suppression, checkpointing, and acknowledgement. |
|
|
105
137
|
| [`slot_invalidation_recovery`](examples/slot_invalidation_recovery) | Fail-closed slot invalidation and explicit operator reconciliation. |
|
|
@@ -155,9 +187,12 @@ maps the canonical started, succeeded, failed, and skipped notifications to
|
|
|
155
187
|
Prometheus counters.
|
|
156
188
|
|
|
157
189
|
`Mammoth::ReplicationConsumer` accepts only exact core events and transaction
|
|
158
|
-
envelopes.
|
|
159
|
-
|
|
160
|
-
|
|
190
|
+
envelopes. `deliver-sample` input crosses
|
|
191
|
+
`PersistedPayloadDeserializer` before entering the live delivery path, so its
|
|
192
|
+
stored hash does not masquerade as CDC-core work. Dead-letter replay is a
|
|
193
|
+
separate destination boundary: Mammoth sends the exact prepared JSON persisted
|
|
194
|
+
for that destination and does not reconstruct CDC work or reapply the current
|
|
195
|
+
payload policy.
|
|
161
196
|
|
|
162
197
|
## Extensions
|
|
163
198
|
|
|
@@ -199,11 +234,18 @@ destinations:
|
|
|
199
234
|
schemas: [public]
|
|
200
235
|
tables: [orders]
|
|
201
236
|
operations: [insert, update]
|
|
237
|
+
payload_policy:
|
|
238
|
+
rules:
|
|
239
|
+
- columns: [customer_email]
|
|
240
|
+
action: mask
|
|
202
241
|
retry:
|
|
203
242
|
max_attempts: 3
|
|
204
243
|
schedule_seconds: [1, 10]
|
|
205
244
|
```
|
|
206
245
|
|
|
246
|
+
See [`docs/PAYLOAD-POLICIES.md`](docs/PAYLOAD-POLICIES.md) for deterministic
|
|
247
|
+
data minimization, retry and dead-letter behavior, and policy boundaries.
|
|
248
|
+
|
|
207
249
|
## CLI
|
|
208
250
|
|
|
209
251
|
```bash
|
data/config/mammoth.example.yml
CHANGED
|
@@ -288,6 +288,23 @@ webhook:
|
|
|
288
288
|
# signature_header: X-Mammoth-Signature
|
|
289
289
|
# timestamp_header: X-Mammoth-Timestamp
|
|
290
290
|
|
|
291
|
+
# Optional deterministic payload projection and PII redaction.
|
|
292
|
+
#
|
|
293
|
+
# Rules apply after canonical serialization and route evaluation. Matching
|
|
294
|
+
# columns are scrubbed consistently from data, identity, and changes before
|
|
295
|
+
# signing, HTTP delivery, retry, and dead-letter persistence.
|
|
296
|
+
#
|
|
297
|
+
# payload_policy:
|
|
298
|
+
# rules:
|
|
299
|
+
# - schemas: [public]
|
|
300
|
+
# tables: [customers, orders]
|
|
301
|
+
# columns: [customer_email, phone]
|
|
302
|
+
# action: remove
|
|
303
|
+
# - tables: [customers]
|
|
304
|
+
# columns: [external_customer_id]
|
|
305
|
+
# action: mask
|
|
306
|
+
# replacement: "[PRIVATE]"
|
|
307
|
+
|
|
291
308
|
# Optional fanout configuration.
|
|
292
309
|
#
|
|
293
310
|
# Use destinations instead of webhook when one PostgreSQL replication stream
|
|
@@ -324,6 +341,14 @@ webhook:
|
|
|
324
341
|
# - insert
|
|
325
342
|
# - update
|
|
326
343
|
#
|
|
344
|
+
# # Optional destination-specific payload policy. Other destinations still
|
|
345
|
+
# # receive their own independently prepared payloads.
|
|
346
|
+
# payload_policy:
|
|
347
|
+
# rules:
|
|
348
|
+
# - tables: [orders]
|
|
349
|
+
# columns: [customer_email]
|
|
350
|
+
# action: remove
|
|
351
|
+
#
|
|
327
352
|
# # Optional per-destination retry override. Omit to use top-level retry.
|
|
328
353
|
# retry:
|
|
329
354
|
# max_attempts: 3
|
data/config/mammoth.schema.json
CHANGED
|
@@ -4,6 +4,85 @@
|
|
|
4
4
|
"title": "Mammoth Configuration",
|
|
5
5
|
"type": "object",
|
|
6
6
|
"additionalProperties": false,
|
|
7
|
+
"definitions": {
|
|
8
|
+
"payloadPolicy": {
|
|
9
|
+
"type": "object",
|
|
10
|
+
"additionalProperties": false,
|
|
11
|
+
"required": [
|
|
12
|
+
"rules"
|
|
13
|
+
],
|
|
14
|
+
"properties": {
|
|
15
|
+
"rules": {
|
|
16
|
+
"type": "array",
|
|
17
|
+
"minItems": 1,
|
|
18
|
+
"items": {
|
|
19
|
+
"type": "object",
|
|
20
|
+
"additionalProperties": false,
|
|
21
|
+
"required": [
|
|
22
|
+
"columns",
|
|
23
|
+
"action"
|
|
24
|
+
],
|
|
25
|
+
"properties": {
|
|
26
|
+
"schemas": {
|
|
27
|
+
"type": "array",
|
|
28
|
+
"minItems": 1,
|
|
29
|
+
"items": {
|
|
30
|
+
"type": "string",
|
|
31
|
+
"minLength": 1
|
|
32
|
+
},
|
|
33
|
+
"description": "PostgreSQL schemas matched by this rule. Omit to match every schema."
|
|
34
|
+
},
|
|
35
|
+
"tables": {
|
|
36
|
+
"type": "array",
|
|
37
|
+
"minItems": 1,
|
|
38
|
+
"items": {
|
|
39
|
+
"type": "string",
|
|
40
|
+
"minLength": 1
|
|
41
|
+
},
|
|
42
|
+
"description": "PostgreSQL tables matched by this rule. Omit to match every table."
|
|
43
|
+
},
|
|
44
|
+
"operations": {
|
|
45
|
+
"type": "array",
|
|
46
|
+
"minItems": 1,
|
|
47
|
+
"items": {
|
|
48
|
+
"type": "string",
|
|
49
|
+
"enum": [
|
|
50
|
+
"insert",
|
|
51
|
+
"update",
|
|
52
|
+
"delete"
|
|
53
|
+
]
|
|
54
|
+
},
|
|
55
|
+
"description": "CDC operations matched by this rule. Omit to match every operation."
|
|
56
|
+
},
|
|
57
|
+
"columns": {
|
|
58
|
+
"type": "array",
|
|
59
|
+
"minItems": 1,
|
|
60
|
+
"uniqueItems": true,
|
|
61
|
+
"items": {
|
|
62
|
+
"type": "string",
|
|
63
|
+
"minLength": 1
|
|
64
|
+
},
|
|
65
|
+
"description": "Column names scrubbed from data, identity, and changes."
|
|
66
|
+
},
|
|
67
|
+
"action": {
|
|
68
|
+
"type": "string",
|
|
69
|
+
"enum": [
|
|
70
|
+
"remove",
|
|
71
|
+
"mask"
|
|
72
|
+
],
|
|
73
|
+
"description": "Remove matching columns or replace their non-null values."
|
|
74
|
+
},
|
|
75
|
+
"replacement": {
|
|
76
|
+
"type": "string",
|
|
77
|
+
"description": "Replacement used by mask. Defaults to [REDACTED]."
|
|
78
|
+
}
|
|
79
|
+
}
|
|
80
|
+
}
|
|
81
|
+
}
|
|
82
|
+
},
|
|
83
|
+
"description": "Deterministic destination payload projection and PII-redaction rules."
|
|
84
|
+
}
|
|
85
|
+
},
|
|
7
86
|
"required": [
|
|
8
87
|
"mammoth",
|
|
9
88
|
"postgres",
|
|
@@ -237,6 +316,9 @@
|
|
|
237
316
|
}
|
|
238
317
|
},
|
|
239
318
|
"description": "Optional HMAC-SHA256 request signing for webhook receivers."
|
|
319
|
+
},
|
|
320
|
+
"payload_policy": {
|
|
321
|
+
"$ref": "#/definitions/payloadPolicy"
|
|
240
322
|
}
|
|
241
323
|
},
|
|
242
324
|
"description": "HTTP webhook destination settings. This is shorthand for a single webhook destination; use destinations for fanout."
|
|
@@ -361,6 +443,9 @@
|
|
|
361
443
|
},
|
|
362
444
|
"description": "Optional destination route filter. Event delivery matches one event; transaction delivery matches when any event in the transaction matches and delivers the full envelope."
|
|
363
445
|
},
|
|
446
|
+
"payload_policy": {
|
|
447
|
+
"$ref": "#/definitions/payloadPolicy"
|
|
448
|
+
},
|
|
364
449
|
"retry": {
|
|
365
450
|
"type": "object",
|
|
366
451
|
"additionalProperties": false,
|
data/lib/mammoth/application.rb
CHANGED
|
@@ -30,7 +30,15 @@ module Mammoth
|
|
|
30
30
|
@checkpoint_store = @state_adapter.checkpoint_store
|
|
31
31
|
@consumer = ReplicationConsumer.new(source: source || build_source, delivery_unit: delivery_unit)
|
|
32
32
|
@progress_coordinator = build_progress_coordinator
|
|
33
|
-
@delivery_worker =
|
|
33
|
+
@delivery_worker = if sink
|
|
34
|
+
build_delivery_worker(
|
|
35
|
+
sink: sink,
|
|
36
|
+
sleeper: sleeper,
|
|
37
|
+
delivery_policy: destination_delivery_policy(config.data["webhook"] || {})
|
|
38
|
+
)
|
|
39
|
+
else
|
|
40
|
+
build_configured_delivery_worker(sleeper: sleeper)
|
|
41
|
+
end
|
|
34
42
|
end
|
|
35
43
|
|
|
36
44
|
# @return [Mammoth::SQLiteStore] underlying SQLite store for compatibility
|
|
@@ -155,8 +163,8 @@ module Mammoth
|
|
|
155
163
|
def destination_specs
|
|
156
164
|
destinations = config.data["destinations"]
|
|
157
165
|
unless destinations
|
|
158
|
-
|
|
159
|
-
return [{ sink: WebhookSink.from_config(config), delivery_policy:
|
|
166
|
+
webhook = config.data.fetch("webhook")
|
|
167
|
+
return [{ sink: WebhookSink.from_config(config), delivery_policy: destination_delivery_policy(webhook) }]
|
|
160
168
|
end
|
|
161
169
|
|
|
162
170
|
destinations.map.with_index(1) do |destination, index|
|
|
@@ -173,7 +181,8 @@ module Mammoth
|
|
|
173
181
|
"enabled" => destination.fetch("enabled", true),
|
|
174
182
|
"max_attempts" => retry_config.fetch("max_attempts", config.dig("retry", "max_attempts")),
|
|
175
183
|
"schedule_seconds" => retry_config.fetch("schedule_seconds", config.dig("retry", "schedule_seconds")),
|
|
176
|
-
"route_filter" => RouteFilter.new(destination["route"])
|
|
184
|
+
"route_filter" => RouteFilter.new(destination["route"]),
|
|
185
|
+
"payload_policy" => PayloadPolicy.new(destination["payload_policy"])
|
|
177
186
|
}
|
|
178
187
|
end
|
|
179
188
|
|
data/lib/mammoth/capabilities.rb
CHANGED
|
@@ -238,38 +238,23 @@ module Mammoth
|
|
|
238
238
|
|
|
239
239
|
def replay_row(row)
|
|
240
240
|
payload = JSON.parse(row.fetch("payload_json"))
|
|
241
|
-
if
|
|
242
|
-
replay_transaction(row.fetch("destination_name"), transaction_envelope(payload))
|
|
243
|
-
else
|
|
244
|
-
replay_event(row.fetch("destination_name"), payload)
|
|
245
|
-
end
|
|
246
|
-
end
|
|
241
|
+
return replay_payload(row.fetch("destination_name"), payload) if worker.respond_to?(:deliver_payload)
|
|
247
242
|
|
|
248
|
-
|
|
249
|
-
event = PersistedPayloadDeserializer.event(payload)
|
|
250
|
-
return worker.deliver_to(destination_name, event) if worker.respond_to?(:deliver_to)
|
|
251
|
-
|
|
252
|
-
worker.deliver(event)
|
|
243
|
+
replay_fanout_payload(row.fetch("destination_name"), payload)
|
|
253
244
|
end
|
|
254
245
|
|
|
255
|
-
def
|
|
256
|
-
|
|
246
|
+
def replay_payload(_destination_name, payload)
|
|
247
|
+
worker.deliver_payload(payload)
|
|
248
|
+
end
|
|
257
249
|
|
|
258
|
-
|
|
250
|
+
def replay_fanout_payload(destination_name, payload)
|
|
251
|
+
worker.deliver_payload_to(destination_name, payload)
|
|
259
252
|
end
|
|
260
253
|
|
|
261
254
|
def replay_resolved?(result)
|
|
262
255
|
result.fetch(:status) == "delivered" || result.fetch(:duplicate, false)
|
|
263
256
|
end
|
|
264
257
|
|
|
265
|
-
def transaction_payload?(payload)
|
|
266
|
-
payload.fetch("type", nil) == TransactionEnvelopeSerializer::PAYLOAD_TYPE
|
|
267
|
-
end
|
|
268
|
-
|
|
269
|
-
def transaction_envelope(payload)
|
|
270
|
-
PersistedPayloadDeserializer.transaction(payload)
|
|
271
|
-
end
|
|
272
|
-
|
|
273
258
|
def show_payload(row)
|
|
274
259
|
row.merge("payload" => JSON.parse(row.fetch("payload_json")))
|
|
275
260
|
end
|
|
@@ -20,9 +20,27 @@ module Mammoth
|
|
|
20
20
|
# @param error [Exception, nil] delivery failure
|
|
21
21
|
# @param retry_count [Integer] number of delivery attempts
|
|
22
22
|
# @return [Integer] inserted dead letter id
|
|
23
|
-
def write(event:, destination_name:, error: nil, retry_count: 0, serializer: EventSerializer)
|
|
23
|
+
def write(event:, destination_name:, error: nil, retry_count: 0, serializer: EventSerializer)
|
|
24
|
+
write_payload(
|
|
25
|
+
payload: serializer.call(event),
|
|
26
|
+
destination_name: destination_name,
|
|
27
|
+
error: error,
|
|
28
|
+
retry_count: retry_count
|
|
29
|
+
)
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
# Store the exact payload prepared for one destination.
|
|
33
|
+
#
|
|
34
|
+
# Persisting the prepared payload prevents retries and replay from restoring
|
|
35
|
+
# fields removed by a destination payload policy.
|
|
36
|
+
#
|
|
37
|
+
# @param payload [Hash] prepared destination payload
|
|
38
|
+
# @param destination_name [String] destination name
|
|
39
|
+
# @param error [Exception, nil] delivery failure
|
|
40
|
+
# @param retry_count [Integer] number of delivery attempts
|
|
41
|
+
# @return [Integer] inserted dead letter id
|
|
42
|
+
def write_payload(payload:, destination_name:, error: nil, retry_count: 0) # rubocop:disable Metrics/MethodLength
|
|
24
43
|
now = Time.now.utc.iso8601
|
|
25
|
-
payload = serializer.call(event)
|
|
26
44
|
database.execute(
|
|
27
45
|
<<~SQL,
|
|
28
46
|
INSERT INTO dead_letters(
|
|
@@ -12,7 +12,7 @@ module Mammoth
|
|
|
12
12
|
DEFAULT_SOURCE = "postgresql"
|
|
13
13
|
|
|
14
14
|
attr_reader :sink, :checkpoint_store, :dead_letter_store, :delivered_envelope_store, :retry_schedule, :max_attempts,
|
|
15
|
-
:sleeper, :source_name, :slot_name, :publication_name, :route_filter, :enabled
|
|
15
|
+
:sleeper, :source_name, :slot_name, :publication_name, :route_filter, :payload_policy, :enabled
|
|
16
16
|
|
|
17
17
|
# @param sink [#deliver] destination sink
|
|
18
18
|
# @param checkpoint_store [Mammoth::CheckpointStore] retained dependency; shared progress owns writes
|
|
@@ -25,10 +25,11 @@ module Mammoth
|
|
|
25
25
|
# @param retry_schedule [Array<Integer>] retry wait schedule in seconds
|
|
26
26
|
# @param sleeper [#call] sleep strategy, injectable for tests
|
|
27
27
|
# @param route_filter [Mammoth::RouteFilter, nil] optional destination route matcher
|
|
28
|
+
# @param payload_policy [Mammoth::PayloadPolicy, nil] destination payload transformation policy
|
|
28
29
|
# @param enabled [Boolean] whether this destination accepts new deliveries
|
|
29
30
|
def initialize(sink:, checkpoint_store:, dead_letter_store:, delivered_envelope_store:, source_name:, slot_name:,
|
|
30
31
|
publication_name:, max_attempts:, retry_schedule:, sleeper: Kernel.method(:sleep),
|
|
31
|
-
route_filter: nil, enabled: true)
|
|
32
|
+
route_filter: nil, payload_policy: nil, enabled: true)
|
|
32
33
|
@sink = sink
|
|
33
34
|
@checkpoint_store = checkpoint_store
|
|
34
35
|
@dead_letter_store = dead_letter_store
|
|
@@ -40,7 +41,11 @@ module Mammoth
|
|
|
40
41
|
@retry_schedule = retry_schedule
|
|
41
42
|
@sleeper = sleeper
|
|
42
43
|
@route_filter = route_filter || RouteFilter.new
|
|
44
|
+
@payload_policy = payload_policy || PayloadPolicy.new
|
|
43
45
|
@enabled = enabled
|
|
46
|
+
return unless @payload_policy.active? && !sink.respond_to?(:deliver_payload)
|
|
47
|
+
|
|
48
|
+
raise ConfigurationError, "destination #{destination_name} does not accept prepared payloads"
|
|
44
49
|
end
|
|
45
50
|
|
|
46
51
|
# Build a delivery worker from Mammoth configuration and stores.
|
|
@@ -66,6 +71,7 @@ module Mammoth
|
|
|
66
71
|
retry_schedule: delivery_policy.fetch("schedule_seconds", config.dig("retry", "schedule_seconds")),
|
|
67
72
|
sleeper: sleeper,
|
|
68
73
|
route_filter: delivery_policy.fetch("route_filter", RouteFilter.new),
|
|
74
|
+
payload_policy: delivery_policy.fetch("payload_policy", PayloadPolicy.new),
|
|
69
75
|
enabled: delivery_policy.fetch("enabled", true)
|
|
70
76
|
)
|
|
71
77
|
end
|
|
@@ -86,16 +92,32 @@ module Mammoth
|
|
|
86
92
|
deliver_work(event, serializer: EventSerializer, delivery_method: :deliver)
|
|
87
93
|
end
|
|
88
94
|
|
|
95
|
+
# Replay one exact destination payload without reapplying the current policy.
|
|
96
|
+
#
|
|
97
|
+
# @param payload [Hash] prepared payload persisted in the dead-letter store
|
|
98
|
+
# @return [Hash] delivery summary
|
|
99
|
+
def deliver_payload(payload)
|
|
100
|
+
prepared = PreparedDelivery.from_payload(payload)
|
|
101
|
+
delivery_method = transaction_payload?(payload) ? :deliver_transaction : :deliver
|
|
102
|
+
deliver_prepared(prepared, work: nil, delivery_method: delivery_method)
|
|
103
|
+
end
|
|
104
|
+
|
|
89
105
|
private
|
|
90
106
|
|
|
91
|
-
# rubocop:disable Metrics/MethodLength
|
|
92
107
|
def deliver_work(work, serializer:, delivery_method:)
|
|
108
|
+
prepared = PreparedDelivery.build(work, serializer: serializer, payload_policy: payload_policy)
|
|
109
|
+
deliver_prepared(prepared, work: work, delivery_method: delivery_method)
|
|
110
|
+
end
|
|
111
|
+
|
|
112
|
+
# rubocop:disable Metrics/MethodLength
|
|
113
|
+
def deliver_prepared(prepared, work:, delivery_method:)
|
|
93
114
|
attempts = 0
|
|
94
|
-
|
|
115
|
+
canonical_payload = prepared.canonical_payload
|
|
116
|
+
payload = prepared.payload
|
|
95
117
|
delivery_unit = delivery_unit_for(delivery_method)
|
|
96
|
-
idempotency_key = idempotency_key_for(payload
|
|
118
|
+
idempotency_key = idempotency_key_for(payload: canonical_payload, delivery_unit: delivery_unit)
|
|
97
119
|
|
|
98
|
-
skip_result = skip_result_for(
|
|
120
|
+
skip_result = skip_result_for(canonical_payload, idempotency_key: idempotency_key)
|
|
99
121
|
return skip_result if skip_result
|
|
100
122
|
|
|
101
123
|
if delivered_envelope_store.delivered?(idempotency_key)
|
|
@@ -110,7 +132,7 @@ module Mammoth
|
|
|
110
132
|
|
|
111
133
|
begin
|
|
112
134
|
attempts += 1
|
|
113
|
-
result =
|
|
135
|
+
result = deliver_to_sink(prepared, work: work, delivery_method: delivery_method)
|
|
114
136
|
delivered_envelope_store.record!(
|
|
115
137
|
idempotency_key: idempotency_key,
|
|
116
138
|
source_name: source_name,
|
|
@@ -122,7 +144,7 @@ module Mammoth
|
|
|
122
144
|
)
|
|
123
145
|
result.merge(attempts: attempts, idempotency_key: idempotency_key)
|
|
124
146
|
rescue DeliveryError => e
|
|
125
|
-
return dead_letter(
|
|
147
|
+
return dead_letter(payload, e, attempts) if attempts >= max_attempts
|
|
126
148
|
|
|
127
149
|
wait_before_retry(attempts)
|
|
128
150
|
retry
|
|
@@ -166,20 +188,30 @@ module Mammoth
|
|
|
166
188
|
}
|
|
167
189
|
end
|
|
168
190
|
|
|
169
|
-
def
|
|
170
|
-
|
|
171
|
-
|
|
191
|
+
def wait_before_retry(attempts)
|
|
192
|
+
wait_seconds = retry_schedule.fetch(attempts - 1, retry_schedule.last)
|
|
193
|
+
sleeper.call(wait_seconds)
|
|
194
|
+
end
|
|
195
|
+
|
|
196
|
+
def deliver_to_sink(prepared, work:, delivery_method:)
|
|
197
|
+
return sink.deliver_payload(prepared.payload) if sink.respond_to?(:deliver_payload)
|
|
198
|
+
return sink.public_send(delivery_method, work) if work && !payload_policy.active?
|
|
199
|
+
|
|
200
|
+
raise ConfigurationError, "destination #{destination_name} does not accept prepared payloads"
|
|
201
|
+
end
|
|
202
|
+
|
|
203
|
+
def dead_letter(payload, error, attempts)
|
|
204
|
+
id = dead_letter_store.write_payload(
|
|
205
|
+
payload: payload,
|
|
172
206
|
destination_name: sink.name,
|
|
173
207
|
error: error,
|
|
174
|
-
retry_count: attempts
|
|
175
|
-
serializer: serializer
|
|
208
|
+
retry_count: attempts
|
|
176
209
|
)
|
|
177
210
|
{ status: "dead_lettered", dead_letter_id: id, attempts: attempts }
|
|
178
211
|
end
|
|
179
212
|
|
|
180
|
-
def
|
|
181
|
-
|
|
182
|
-
sleeper.call(wait_seconds)
|
|
213
|
+
def transaction_payload?(payload)
|
|
214
|
+
payload["type"] == TransactionEnvelopeSerializer::PAYLOAD_TYPE
|
|
183
215
|
end
|
|
184
216
|
end
|
|
185
217
|
end
|
|
@@ -21,7 +21,13 @@ module Mammoth
|
|
|
21
21
|
|
|
22
22
|
# @return [Hash] JSON-friendly capabilities
|
|
23
23
|
def capabilities
|
|
24
|
-
{
|
|
24
|
+
{
|
|
25
|
+
type: adapter_type,
|
|
26
|
+
delivery_units: %w[event transaction],
|
|
27
|
+
prepared_payloads: true,
|
|
28
|
+
signing: true,
|
|
29
|
+
header_env: true
|
|
30
|
+
}
|
|
25
31
|
end
|
|
26
32
|
end
|
|
27
33
|
end
|
|
@@ -46,6 +46,15 @@ module Mammoth
|
|
|
46
46
|
worker_for(destination_name).deliver_transaction(envelope)
|
|
47
47
|
end
|
|
48
48
|
|
|
49
|
+
# Replay one exact prepared payload to one configured destination.
|
|
50
|
+
#
|
|
51
|
+
# @param destination_name [String] destination name
|
|
52
|
+
# @param payload [Hash] prepared destination payload
|
|
53
|
+
# @return [Hash] destination delivery summary
|
|
54
|
+
def deliver_payload_to(destination_name, payload)
|
|
55
|
+
worker_for(destination_name).deliver_payload(payload)
|
|
56
|
+
end
|
|
57
|
+
|
|
49
58
|
private
|
|
50
59
|
|
|
51
60
|
def fanout(delivery_method, work)
|
|
@@ -0,0 +1,149 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "digest"
|
|
4
|
+
require "json"
|
|
5
|
+
|
|
6
|
+
module Mammoth
|
|
7
|
+
# Applies deterministic destination-scoped projection and redaction rules.
|
|
8
|
+
#
|
|
9
|
+
# Policies operate on serialized Mammoth payloads, never on CDC-core work
|
|
10
|
+
# items. Column rules scrub every canonical row-value representation: data,
|
|
11
|
+
# identity, and old/new values in changes.
|
|
12
|
+
class PayloadPolicy
|
|
13
|
+
DEFAULT_MASK = "[REDACTED]"
|
|
14
|
+
POLICY_METADATA_KEY = "mammoth_payload_policy"
|
|
15
|
+
SUPPORTED_ACTIONS = %w[remove mask].freeze
|
|
16
|
+
|
|
17
|
+
attr_reader :config, :fingerprint
|
|
18
|
+
|
|
19
|
+
# @param config [Hash, nil] validated payload-policy configuration
|
|
20
|
+
def initialize(config = nil)
|
|
21
|
+
@config = deep_freeze(stringify_keys(config || {}))
|
|
22
|
+
validate!
|
|
23
|
+
@fingerprint = build_fingerprint if active?
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
# @return [Boolean] whether the policy contains transformation rules
|
|
27
|
+
def active?
|
|
28
|
+
rules.any?
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
# Transform one canonical event or transaction payload.
|
|
32
|
+
#
|
|
33
|
+
# @param payload [Hash] canonical serialized Mammoth payload
|
|
34
|
+
# @return [Hash] independent JSON-compatible destination payload
|
|
35
|
+
def apply(payload)
|
|
36
|
+
return payload unless active?
|
|
37
|
+
|
|
38
|
+
transformed = JSON.parse(JSON.generate(payload))
|
|
39
|
+
events_for(transformed).each { |event| apply_rules(event) }
|
|
40
|
+
metadata = transformed["metadata"]
|
|
41
|
+
metadata = {} unless metadata.is_a?(Hash) # steep:ignore
|
|
42
|
+
transformed["metadata"] = metadata
|
|
43
|
+
metadata[POLICY_METADATA_KEY] = { "fingerprint" => fingerprint }
|
|
44
|
+
transformed
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
private
|
|
48
|
+
|
|
49
|
+
def rules
|
|
50
|
+
config.fetch("rules", [])
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
def validate!
|
|
54
|
+
rules.each_with_index do |rule, index|
|
|
55
|
+
action = rule["action"].to_s
|
|
56
|
+
unless SUPPORTED_ACTIONS.include?(action)
|
|
57
|
+
raise ConfigurationError, "payload policy rule #{index} has unsupported action #{action}"
|
|
58
|
+
end
|
|
59
|
+
raise ConfigurationError, "payload policy rule #{index} requires columns" if Array(rule["columns"]).empty?
|
|
60
|
+
end
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
def events_for(payload)
|
|
64
|
+
return payload.fetch("events") if payload["type"] == TransactionEnvelopeSerializer::PAYLOAD_TYPE
|
|
65
|
+
|
|
66
|
+
[payload]
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
def apply_rules(event)
|
|
70
|
+
rules.each do |rule|
|
|
71
|
+
apply_rule(event, rule) if match?(event, rule)
|
|
72
|
+
end
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
def apply_rule(event, rule)
|
|
76
|
+
columns = rule.fetch("columns")
|
|
77
|
+
if rule.fetch("action") == "remove"
|
|
78
|
+
remove_columns(event, columns)
|
|
79
|
+
else
|
|
80
|
+
mask_columns(event, columns, rule.fetch("replacement", DEFAULT_MASK))
|
|
81
|
+
end
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
def remove_columns(event, columns)
|
|
85
|
+
%w[data identity].each do |field|
|
|
86
|
+
columns.each { |column| event[field]&.delete(column) }
|
|
87
|
+
end
|
|
88
|
+
changes = Array(event["changes"]) # : Array[untyped]
|
|
89
|
+
event["changes"] = changes.reject { |change| columns.include?(change["name"]) }
|
|
90
|
+
end
|
|
91
|
+
|
|
92
|
+
def mask_columns(event, columns, replacement)
|
|
93
|
+
%w[data identity].each do |field|
|
|
94
|
+
columns.each { |column| mask_map_value(event[field], column, replacement) }
|
|
95
|
+
end
|
|
96
|
+
changes = Array(event["changes"]) # : Array[untyped]
|
|
97
|
+
changes.each do |change|
|
|
98
|
+
next unless columns.include?(change["name"])
|
|
99
|
+
|
|
100
|
+
%w[old_value new_value].each do |value_key|
|
|
101
|
+
change[value_key] = replacement if change.key?(value_key) && !change[value_key].nil?
|
|
102
|
+
end
|
|
103
|
+
end
|
|
104
|
+
end
|
|
105
|
+
|
|
106
|
+
def mask_map_value(map, column, replacement)
|
|
107
|
+
return unless map.is_a?(Hash) && map.key?(column)
|
|
108
|
+
return if map[column].nil?
|
|
109
|
+
|
|
110
|
+
map[column] = replacement
|
|
111
|
+
end
|
|
112
|
+
|
|
113
|
+
def match?(event, rule)
|
|
114
|
+
matches?(rule["schemas"], event["namespace"]) &&
|
|
115
|
+
matches?(rule["tables"], event["entity"]) &&
|
|
116
|
+
matches?(rule["operations"], event["operation"])
|
|
117
|
+
end
|
|
118
|
+
|
|
119
|
+
def matches?(configured, actual)
|
|
120
|
+
values = Array(configured) # : Array[untyped]
|
|
121
|
+
values.empty? || values.any? { |value| value.to_s.casecmp?(actual.to_s) }
|
|
122
|
+
end
|
|
123
|
+
|
|
124
|
+
def build_fingerprint
|
|
125
|
+
canonical = canonicalize(config)
|
|
126
|
+
"sha256:#{Digest::SHA256.hexdigest(JSON.generate(canonical))}"
|
|
127
|
+
end
|
|
128
|
+
|
|
129
|
+
def canonicalize(value)
|
|
130
|
+
return value.map { |item| canonicalize(item) } if value.is_a?(Array)
|
|
131
|
+
return value unless value.is_a?(Hash)
|
|
132
|
+
|
|
133
|
+
value.keys.sort.to_h { |key| [key, canonicalize(value.fetch(key))] }
|
|
134
|
+
end
|
|
135
|
+
|
|
136
|
+
def stringify_keys(value)
|
|
137
|
+
return value.map { |item| stringify_keys(item) } if value.is_a?(Array)
|
|
138
|
+
return value unless value.is_a?(Hash)
|
|
139
|
+
|
|
140
|
+
value.to_h { |key, item| [key.to_s, stringify_keys(item)] }
|
|
141
|
+
end
|
|
142
|
+
|
|
143
|
+
def deep_freeze(value)
|
|
144
|
+
value.each_value { |item| deep_freeze(item) } if value.is_a?(Hash)
|
|
145
|
+
value.each { |item| deep_freeze(item) } if value.is_a?(Array)
|
|
146
|
+
value.freeze
|
|
147
|
+
end
|
|
148
|
+
end
|
|
149
|
+
end
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Mammoth
|
|
4
|
+
# Immutable delivery boundary containing canonical identity and outbound data.
|
|
5
|
+
PreparedDelivery = Data.define(:canonical_payload, :payload, :policy_fingerprint) do
|
|
6
|
+
# Serialize and apply one destination payload policy.
|
|
7
|
+
#
|
|
8
|
+
# @param work [CDC::Core::ChangeEvent, CDC::Core::TransactionEnvelope]
|
|
9
|
+
# @param serializer [#call] canonical Mammoth serializer
|
|
10
|
+
# @param payload_policy [Mammoth::PayloadPolicy]
|
|
11
|
+
# @return [Mammoth::PreparedDelivery]
|
|
12
|
+
def self.build(work, serializer:, payload_policy:)
|
|
13
|
+
canonical_payload = serializer.call(work)
|
|
14
|
+
new(
|
|
15
|
+
canonical_payload: canonical_payload,
|
|
16
|
+
payload: payload_policy.apply(canonical_payload),
|
|
17
|
+
policy_fingerprint: payload_policy.fingerprint
|
|
18
|
+
)
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
# Reconstruct a prepared delivery from a persisted destination payload.
|
|
22
|
+
#
|
|
23
|
+
# @param payload [Hash] previously prepared destination payload
|
|
24
|
+
# @return [Mammoth::PreparedDelivery]
|
|
25
|
+
def self.from_payload(payload)
|
|
26
|
+
fingerprint = payload.dig("metadata", PayloadPolicy::POLICY_METADATA_KEY, "fingerprint")
|
|
27
|
+
new(canonical_payload: payload, payload: payload, policy_fingerprint: fingerprint)
|
|
28
|
+
end
|
|
29
|
+
end
|
|
30
|
+
end
|
data/lib/mammoth/version.rb
CHANGED
data/lib/mammoth/webhook_sink.rb
CHANGED
|
@@ -110,8 +110,14 @@ module Mammoth
|
|
|
110
110
|
deliver_payload(TransactionEnvelopeSerializer.call(envelope))
|
|
111
111
|
end
|
|
112
112
|
|
|
113
|
-
|
|
114
|
-
|
|
113
|
+
# Deliver an already prepared destination payload.
|
|
114
|
+
#
|
|
115
|
+
# The caller owns canonical serialization and payload-policy application.
|
|
116
|
+
# WebhookSink signs and sends exactly this payload.
|
|
117
|
+
#
|
|
118
|
+
# @param payload [Hash] prepared JSON-compatible destination payload
|
|
119
|
+
# @return [Hash] delivery result
|
|
120
|
+
# @raise [Mammoth::DeliveryError] when delivery fails
|
|
115
121
|
def deliver_payload(payload)
|
|
116
122
|
response = perform_request(payload)
|
|
117
123
|
return delivery_result(payload, response) if SUCCESS_RANGE.cover?(response.code.to_i)
|
|
@@ -121,6 +127,8 @@ module Mammoth
|
|
|
121
127
|
raise DeliveryError, "webhook #{name} delivery failed: #{e.message}"
|
|
122
128
|
end
|
|
123
129
|
|
|
130
|
+
private
|
|
131
|
+
|
|
124
132
|
def perform_request(payload)
|
|
125
133
|
Net::HTTP.start(url.host, url.port, use_ssl: url.scheme == "https", open_timeout: timeout_seconds,
|
|
126
134
|
read_timeout: timeout_seconds) do |http|
|
data/lib/mammoth.rb
CHANGED
|
@@ -33,6 +33,8 @@ require_relative "mammoth/event_serializer"
|
|
|
33
33
|
require_relative "mammoth/transaction_envelope_serializer"
|
|
34
34
|
require_relative "mammoth/persisted_payload_deserializer"
|
|
35
35
|
require_relative "mammoth/route_filter"
|
|
36
|
+
require_relative "mammoth/payload_policy"
|
|
37
|
+
require_relative "mammoth/prepared_delivery"
|
|
36
38
|
require_relative "mammoth/webhook_sink"
|
|
37
39
|
require_relative "mammoth/operational_state/adapter"
|
|
38
40
|
require_relative "mammoth/operational_state/sqlite_adapter"
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: mammoth
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.4.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Ken C. Demanawa
|
|
@@ -219,8 +219,10 @@ files:
|
|
|
219
219
|
- lib/mammoth/operational_state/adapter.rb
|
|
220
220
|
- lib/mammoth/operational_state/registry.rb
|
|
221
221
|
- lib/mammoth/operational_state/sqlite_adapter.rb
|
|
222
|
+
- lib/mammoth/payload_policy.rb
|
|
222
223
|
- lib/mammoth/persisted_payload_deserializer.rb
|
|
223
224
|
- lib/mammoth/postgres_observability_metrics.rb
|
|
225
|
+
- lib/mammoth/prepared_delivery.rb
|
|
224
226
|
- lib/mammoth/registry.rb
|
|
225
227
|
- lib/mammoth/replication_consumer.rb
|
|
226
228
|
- lib/mammoth/route_filter.rb
|