mammoth 1.5.0 → 1.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +9 -1
- data/README.md +41 -19
- data/config/mammoth.example.yml +1 -1
- data/config/mammoth.schema.json +3 -3
- data/lib/mammoth/lifecycle_hooks.rb +3 -0
- data/lib/mammoth/logging.rb +47 -0
- data/lib/mammoth/operational_state/sqlite_adapter.rb +1 -1
- data/lib/mammoth/payload_policy.rb +5 -0
- data/lib/mammoth/sources/postgres_publication_inspector.rb +1 -0
- data/lib/mammoth/version.rb +1 -1
- metadata +6 -6
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 3d7fd1f170a51138c5f3f1b8cc31c388428dd9052020f38653242c774bc6a92e
|
|
4
|
+
data.tar.gz: 134f6734ff5a1c377ba318a924630f6792dd51f66b0510ca37157836caacc993
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 7421871882b9288892ee4c3c2e3e0701839dd60165b238b771a98cd2ec21aa0ed1323b4605ce1f1c9270cd711430aae97f61b0e5f7edd5316a6b77a8686e1f39
|
|
7
|
+
data.tar.gz: 6b9d480c5c71beadbbf2633597e6af88176764b1a8e44e49b41cfa24c4636e0f14a08287f2522ce8bf885b0b7b95930600836697cd66acefb67452e5e2c3f2b5
|
data/CHANGELOG.md
CHANGED
|
@@ -6,7 +6,15 @@
|
|
|
6
6
|
|
|
7
7
|
## Unreleased
|
|
8
8
|
|
|
9
|
-
## 1.5.
|
|
9
|
+
## 1.5.1 - 2026-07-25
|
|
10
|
+
|
|
11
|
+
### Changed
|
|
12
|
+
|
|
13
|
+
- Adopt Mammoth as the ecosystem brand, identify this open-source project as
|
|
14
|
+
Mammoth Data Plane, and align documentation, published gem metadata,
|
|
15
|
+
configuration descriptions, and branding assets with the new hierarchy.
|
|
16
|
+
|
|
17
|
+
## 1.5.0 - 2026-07-24
|
|
10
18
|
|
|
11
19
|
### Changed
|
|
12
20
|
|
data/README.md
CHANGED
|
@@ -25,6 +25,17 @@
|
|
|
25
25
|
Mammoth is a self-hosted PostgreSQL event relay focused on reliable delivery
|
|
26
26
|
of database change events.
|
|
27
27
|
|
|
28
|
+
This repository contains the **Mammoth Data Plane**, the open-source runtime in
|
|
29
|
+
the broader Mammoth ecosystem:
|
|
30
|
+
|
|
31
|
+
```text
|
|
32
|
+
Mammoth
|
|
33
|
+
├── Mammoth Data Plane
|
|
34
|
+
├── Mammoth Control Plane
|
|
35
|
+
├── Mammoth Control Agent
|
|
36
|
+
└── Mammoth Extensions
|
|
37
|
+
```
|
|
38
|
+
|
|
28
39
|
```text
|
|
29
40
|
PostgreSQL
|
|
30
41
|
↓
|
|
@@ -32,7 +43,7 @@ CDC Ecosystem source adapter
|
|
|
32
43
|
↓
|
|
33
44
|
CDC::Core::TransactionEnvelope
|
|
34
45
|
↓
|
|
35
|
-
Mammoth
|
|
46
|
+
Mammoth Data Plane
|
|
36
47
|
↓
|
|
37
48
|
Webhook fanout
|
|
38
49
|
```
|
|
@@ -198,7 +209,7 @@ payload policy.
|
|
|
198
209
|
|
|
199
210
|
## Extensions
|
|
200
211
|
|
|
201
|
-
Mammoth
|
|
212
|
+
Mammoth Data Plane exposes small adapter registries for future extensions:
|
|
202
213
|
|
|
203
214
|
- operational state adapters
|
|
204
215
|
- destination adapters
|
|
@@ -307,24 +318,35 @@ rather than opening SQLite directly.
|
|
|
307
318
|
|
|
308
319
|
## Performance
|
|
309
320
|
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
321
|
+
A reference benchmark run on Ruby 4.0.5 produced:
|
|
322
|
+
|
|
323
|
+
| Benchmark | Result |
|
|
324
|
+
| --------------------------------------------------- | --------------------: |
|
|
325
|
+
| Event payload projection with explicit ID | **56,265 events/sec** |
|
|
326
|
+
| Transaction payload projection with explicit IDs | **55,612 events/sec** |
|
|
327
|
+
| Mask payload-policy projection | **72,403 events/sec** |
|
|
328
|
+
| Concurrent delivery, 25 workers | **3,825 events/sec** |
|
|
329
|
+
| Concurrent delivery, 50 workers | **7,447 events/sec** |
|
|
330
|
+
| Signed webhook delivery with 10 ms receiver latency | **83.5 requests/sec** |
|
|
331
|
+
| SQLite delivered-ledger writes | **1,630 writes/sec** |
|
|
332
|
+
| Dead-letter replay mechanics | **1,030 entries/sec** |
|
|
333
|
+
|
|
334
|
+
At concurrency 50 with a synthetic 25 ms destination, Mammoth processed
|
|
335
|
+
**1,862 transactions/sec** against a theoretical ceiling of 2,000, while
|
|
336
|
+
maintaining **28.4 ms P95 latency**. This indicates low scheduling overhead for
|
|
337
|
+
I/O-bound downstream delivery.
|
|
338
|
+
|
|
339
|
+
These are local, single-trial reference measurements, not universal capacity
|
|
340
|
+
claims or comparisons with other CDC platforms. The benchmarks isolate specific
|
|
341
|
+
Mammoth components and may exclude PostgreSQL transport, decoding, checkpoint
|
|
342
|
+
coordination, network latency, or destination processing.
|
|
320
343
|
|
|
321
344
|
The [reference snapshot](https://github.com/kanutocd/mammoth/blob/main/benchmark/results/20260724T104950Z/snapshot.md)
|
|
322
|
-
records
|
|
323
|
-
interpretation.
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
DLQ replay expectations.
|
|
345
|
+
records the exact commands, environment, raw output, and detailed
|
|
346
|
+
interpretation. Re-run the scripts in
|
|
347
|
+
[`benchmark/`](https://github.com/kanutocd/mammoth/tree/main/benchmark)
|
|
348
|
+
on your own hardware when choosing `runtime.concurrency`, destination count,
|
|
349
|
+
SQLite storage, scrape frequency, and DLQ replay expectations.
|
|
328
350
|
|
|
329
351
|
Create a publishable benchmark snapshot with:
|
|
330
352
|
|
|
@@ -380,4 +402,4 @@ building a writable database copy. See
|
|
|
380
402
|
|
|
381
403
|
## License
|
|
382
404
|
|
|
383
|
-
Mammoth OSS is licensed under the [MIT License](LICENSE.txt).
|
|
405
|
+
Mammoth OSS Data Plane is licensed under the [MIT License](LICENSE.txt).
|
data/config/mammoth.example.yml
CHANGED
data/config/mammoth.schema.json
CHANGED
|
@@ -158,7 +158,7 @@
|
|
|
158
158
|
"description": "Operator-defined node metadata."
|
|
159
159
|
}
|
|
160
160
|
},
|
|
161
|
-
"description": "Optional local node identity. Mammoth
|
|
161
|
+
"description": "Optional local node identity. Mammoth Data Plane uses this for status and future extension surfaces only."
|
|
162
162
|
},
|
|
163
163
|
"postgres": {
|
|
164
164
|
"type": "object",
|
|
@@ -346,7 +346,7 @@
|
|
|
346
346
|
"enum": [
|
|
347
347
|
"webhook"
|
|
348
348
|
],
|
|
349
|
-
"description": "Destination adapter type. Mammoth
|
|
349
|
+
"description": "Destination adapter type. Mammoth Data Plane 0.7.x supports webhook destinations."
|
|
350
350
|
},
|
|
351
351
|
"enabled": {
|
|
352
352
|
"type": "boolean",
|
|
@@ -520,7 +520,7 @@
|
|
|
520
520
|
"enum": [
|
|
521
521
|
"sqlite"
|
|
522
522
|
],
|
|
523
|
-
"description": "Operational state adapter. Mammoth
|
|
523
|
+
"description": "Operational state adapter. Mammoth Data Plane 0.7.x ships the sqlite adapter."
|
|
524
524
|
}
|
|
525
525
|
},
|
|
526
526
|
"description": "Operational state adapter selection. SQLite remains the default OSS adapter."
|
|
@@ -7,6 +7,7 @@ module Mammoth
|
|
|
7
7
|
# future control agents stable observation points without changing the data
|
|
8
8
|
# plane or introducing remote behavior into OSS.
|
|
9
9
|
class LifecycleHooks
|
|
10
|
+
# Supported lifecycle callback names.
|
|
10
11
|
EVENTS = %i[
|
|
11
12
|
before_start
|
|
12
13
|
after_start
|
|
@@ -15,6 +16,8 @@ module Mammoth
|
|
|
15
16
|
before_replay
|
|
16
17
|
after_replay
|
|
17
18
|
].freeze
|
|
19
|
+
|
|
20
|
+
# Frozen empty callback list used for events without registrations.
|
|
18
21
|
EMPTY_CALLBACKS = [nil].compact.freeze
|
|
19
22
|
|
|
20
23
|
attr_reader :callbacks
|
data/lib/mammoth/logging.rb
CHANGED
|
@@ -6,6 +6,7 @@ require "time"
|
|
|
6
6
|
module Mammoth
|
|
7
7
|
# Container-friendly structured application logging.
|
|
8
8
|
module Logging
|
|
9
|
+
# Supported severity names mapped to their filtering priority.
|
|
9
10
|
LEVELS = { "debug" => 0, "info" => 1, "warn" => 2, "error" => 3 }.freeze
|
|
10
11
|
|
|
11
12
|
# Logger that emits one JSON object per line.
|
|
@@ -19,9 +20,32 @@ module Mammoth
|
|
|
19
20
|
raise ConfigurationError, "unsupported logging level: #{@level}" unless LEVELS.key?(@level)
|
|
20
21
|
end
|
|
21
22
|
|
|
23
|
+
# Emit a debug event when debug logging is enabled.
|
|
24
|
+
#
|
|
25
|
+
# @param event [String, Symbol] stable event name
|
|
26
|
+
# @param context [Hash] structured event fields
|
|
27
|
+
# @return [Boolean] whether the event was emitted
|
|
22
28
|
def debug(event, **context) = write("debug", event, context)
|
|
29
|
+
|
|
30
|
+
# Emit an informational event when info logging is enabled.
|
|
31
|
+
#
|
|
32
|
+
# @param event [String, Symbol] stable event name
|
|
33
|
+
# @param context [Hash] structured event fields
|
|
34
|
+
# @return [Boolean] whether the event was emitted
|
|
23
35
|
def info(event, **context) = write("info", event, context)
|
|
36
|
+
|
|
37
|
+
# Emit a warning event when warn logging is enabled.
|
|
38
|
+
#
|
|
39
|
+
# @param event [String, Symbol] stable event name
|
|
40
|
+
# @param context [Hash] structured event fields
|
|
41
|
+
# @return [Boolean] whether the event was emitted
|
|
24
42
|
def warn(event, **context) = write("warn", event, context)
|
|
43
|
+
|
|
44
|
+
# Emit an error event.
|
|
45
|
+
#
|
|
46
|
+
# @param event [String, Symbol] stable event name
|
|
47
|
+
# @param context [Hash] structured event fields
|
|
48
|
+
# @return [Boolean] whether the event was emitted
|
|
25
49
|
def error(event, **context) = write("error", event, context)
|
|
26
50
|
|
|
27
51
|
def enabled?(severity)
|
|
@@ -42,17 +66,40 @@ module Mammoth
|
|
|
42
66
|
|
|
43
67
|
# No-op logger used at injectable library boundaries.
|
|
44
68
|
class NullLogger
|
|
69
|
+
# Shared immutable no-op logger.
|
|
45
70
|
INSTANCE = new
|
|
46
71
|
|
|
72
|
+
# Suppress a debug event.
|
|
73
|
+
#
|
|
74
|
+
# @return [false]
|
|
47
75
|
def debug(_event, **_context) = false
|
|
76
|
+
|
|
77
|
+
# Suppress an informational event.
|
|
78
|
+
#
|
|
79
|
+
# @return [false]
|
|
48
80
|
def info(_event, **_context) = false
|
|
81
|
+
|
|
82
|
+
# Suppress a warning event.
|
|
83
|
+
#
|
|
84
|
+
# @return [false]
|
|
49
85
|
def warn(_event, **_context) = false
|
|
86
|
+
|
|
87
|
+
# Suppress an error event.
|
|
88
|
+
#
|
|
89
|
+
# @return [false]
|
|
50
90
|
def error(_event, **_context) = false
|
|
91
|
+
|
|
51
92
|
def enabled?(_severity) = false
|
|
52
93
|
end
|
|
53
94
|
|
|
54
95
|
module_function
|
|
55
96
|
|
|
97
|
+
# Build the configured structured logger.
|
|
98
|
+
#
|
|
99
|
+
# @param config [Mammoth::Configuration, Hash] loaded Mammoth configuration
|
|
100
|
+
# @param output [IO] destination for newline-delimited JSON
|
|
101
|
+
# @param clock [#call] UTC-compatible time source
|
|
102
|
+
# @return [Mammoth::Logging::Logger]
|
|
56
103
|
def build(config, output: $stdout, clock: -> { Time.now.utc })
|
|
57
104
|
Logger.new(level: config.dig("logging", "level") || "info", output:, clock:)
|
|
58
105
|
end
|
|
@@ -10,8 +10,13 @@ module Mammoth
|
|
|
10
10
|
# items. Column rules scrub every canonical row-value representation: data,
|
|
11
11
|
# identity, and old/new values in changes.
|
|
12
12
|
class PayloadPolicy
|
|
13
|
+
# Default replacement used by mask rules.
|
|
13
14
|
DEFAULT_MASK = "[REDACTED]"
|
|
15
|
+
|
|
16
|
+
# Payload metadata key containing the applied policy fingerprint.
|
|
14
17
|
POLICY_METADATA_KEY = "mammoth_payload_policy"
|
|
18
|
+
|
|
19
|
+
# Transformation actions accepted by payload-policy rules.
|
|
15
20
|
SUPPORTED_ACTIONS = %w[remove mask].freeze
|
|
16
21
|
|
|
17
22
|
attr_reader :config, :fingerprint
|
|
@@ -70,6 +70,7 @@ module Mammoth
|
|
|
70
70
|
|
|
71
71
|
# Read-only PostgreSQL publication and replica-identity catalog inspector.
|
|
72
72
|
class PostgresPublicationInspector
|
|
73
|
+
# Catalog query used to inspect publication actions and replica identity.
|
|
73
74
|
QUERY = <<~SQL
|
|
74
75
|
SELECT
|
|
75
76
|
relation.oid AS relation_id,
|
data/lib/mammoth/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: mammoth
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.5.
|
|
4
|
+
version: 1.5.1
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Ken C. Demanawa
|
|
@@ -182,14 +182,15 @@ dependencies:
|
|
|
182
182
|
- !ruby/object:Gem::Version
|
|
183
183
|
version: '1.9'
|
|
184
184
|
description: |
|
|
185
|
-
Mammoth is
|
|
185
|
+
Mammoth Data Plane is the open-source PostgreSQL change-event delivery
|
|
186
|
+
runtime in the Mammoth ecosystem.
|
|
186
187
|
|
|
187
188
|
It realizes the CDC Ecosystem pgoutput and cdc-core libraries for PostgreSQL,
|
|
188
189
|
then delivers normalized changes to webhook endpoints with durable
|
|
189
190
|
checkpointing, retry state, dead letters, and operational visibility.
|
|
190
191
|
|
|
191
|
-
Mammoth is application-first: it can be installed as a Ruby gem,
|
|
192
|
-
into a container image, or deployed into Kubernetes with Helm.
|
|
192
|
+
Mammoth Data Plane is application-first: it can be installed as a Ruby gem,
|
|
193
|
+
packaged into a container image, or deployed into Kubernetes with Helm.
|
|
193
194
|
email:
|
|
194
195
|
- kenneth.c.demanawa@gmail.com
|
|
195
196
|
executables:
|
|
@@ -286,6 +287,5 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
286
287
|
requirements: []
|
|
287
288
|
rubygems_version: 4.0.10
|
|
288
289
|
specification_version: 4
|
|
289
|
-
summary:
|
|
290
|
-
dead letters, and operational state.
|
|
290
|
+
summary: Mammoth Data Plane for reliable PostgreSQL change-event delivery.
|
|
291
291
|
test_files: []
|