mammoth 1.5.0 → 1.5.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +21 -1
- data/README.md +51 -19
- data/config/mammoth.example.yml +1 -1
- data/config/mammoth.schema.json +3 -3
- data/lib/mammoth/lifecycle_hooks.rb +3 -0
- data/lib/mammoth/logging.rb +47 -0
- data/lib/mammoth/operational_state/sqlite_adapter.rb +1 -1
- data/lib/mammoth/payload_policy.rb +5 -0
- data/lib/mammoth/sources/postgres_publication_inspector.rb +1 -0
- data/lib/mammoth/version.rb +1 -1
- metadata +6 -6
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 9f17ca40374989940289e36db2ee681398d2e511f88fd120344ff0cd26aa7eb2
|
|
4
|
+
data.tar.gz: fdbbd994bf1efc42ea6bd2699c9f0dd46dd790fdf1b930b389b0b3c9f853846b
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 66a28c5f733c9512e1a78d0a3336dadb46a12bd53d89c13d74a0d3449e4fe3f279a7590204052e13b015c14144cabfe26f64c40aa690bd89a761c347af2b5ea3
|
|
7
|
+
data.tar.gz: a87703154c4b33c177be2c9c37a836190f40030c16fc01c3f8ddbf64d14df3d6842e56284bd923d18370c496b008cfac46c0b8e56905ab39fef6cccdc7142874
|
data/CHANGELOG.md
CHANGED
|
@@ -6,7 +6,27 @@
|
|
|
6
6
|
|
|
7
7
|
## Unreleased
|
|
8
8
|
|
|
9
|
-
## 1.5.
|
|
9
|
+
## 1.5.2 - 2026-07-26
|
|
10
|
+
|
|
11
|
+
### Changed
|
|
12
|
+
|
|
13
|
+
- Document the roadmap for publishing Mammoth-owned, versioned JSON Schema
|
|
14
|
+
artifacts for canonical webhook event and transaction payloads.
|
|
15
|
+
|
|
16
|
+
### Fixed
|
|
17
|
+
|
|
18
|
+
- Correct the Mammoth Helm chart's default image tag to `1.5.2`, matching the
|
|
19
|
+
published container image tag convention.
|
|
20
|
+
|
|
21
|
+
## 1.5.1 - 2026-07-25
|
|
22
|
+
|
|
23
|
+
### Changed
|
|
24
|
+
|
|
25
|
+
- Adopt Mammoth as the ecosystem brand, identify this open-source project as
|
|
26
|
+
Mammoth Data Plane, and align documentation, published gem metadata,
|
|
27
|
+
configuration descriptions, and branding assets with the new hierarchy.
|
|
28
|
+
|
|
29
|
+
## 1.5.0 - 2026-07-24
|
|
10
30
|
|
|
11
31
|
### Changed
|
|
12
32
|
|
data/README.md
CHANGED
|
@@ -25,6 +25,17 @@
|
|
|
25
25
|
Mammoth is a self-hosted PostgreSQL event relay focused on reliable delivery
|
|
26
26
|
of database change events.
|
|
27
27
|
|
|
28
|
+
This repository contains the **Mammoth Data Plane**, the open-source runtime in
|
|
29
|
+
the broader Mammoth ecosystem:
|
|
30
|
+
|
|
31
|
+
```text
|
|
32
|
+
Mammoth
|
|
33
|
+
├── Mammoth Data Plane
|
|
34
|
+
├── Mammoth Control Plane
|
|
35
|
+
├── Mammoth Control Agent
|
|
36
|
+
└── Mammoth Extensions
|
|
37
|
+
```
|
|
38
|
+
|
|
28
39
|
```text
|
|
29
40
|
PostgreSQL
|
|
30
41
|
↓
|
|
@@ -32,7 +43,7 @@ CDC Ecosystem source adapter
|
|
|
32
43
|
↓
|
|
33
44
|
CDC::Core::TransactionEnvelope
|
|
34
45
|
↓
|
|
35
|
-
Mammoth
|
|
46
|
+
Mammoth Data Plane
|
|
36
47
|
↓
|
|
37
48
|
Webhook fanout
|
|
38
49
|
```
|
|
@@ -41,6 +52,16 @@ Mammoth is intentionally boring infrastructure. It uses YAML configuration,
|
|
|
41
52
|
JSON Schema validation, local SQLite operational state, and the CDC Ecosystem's
|
|
42
53
|
shared vocabulary so operators can inspect, recover, and reason about delivery.
|
|
43
54
|
|
|
55
|
+
## Mammoth ecosystem
|
|
56
|
+
|
|
57
|
+
The Data Plane is one component of the broader Mammoth ecosystem. The companion
|
|
58
|
+
Control Plane, Control Agent, and Extensions are developed separately and are
|
|
59
|
+
not part of this open-source repository or its MIT distribution.
|
|
60
|
+
|
|
61
|
+
See the [Mammoth ecosystem overview](docs/ECOSYSTEM.md) for public scope,
|
|
62
|
+
integration boundaries, and roadmap status. Planned capabilities are
|
|
63
|
+
directional and do not represent a release commitment.
|
|
64
|
+
|
|
44
65
|
## Supported PostgreSQL versions
|
|
45
66
|
|
|
46
67
|
Mammoth supports PostgreSQL 14 through PostgreSQL 18, inclusive. These are the
|
|
@@ -198,7 +219,7 @@ payload policy.
|
|
|
198
219
|
|
|
199
220
|
## Extensions
|
|
200
221
|
|
|
201
|
-
Mammoth
|
|
222
|
+
Mammoth Data Plane exposes small adapter registries for future extensions:
|
|
202
223
|
|
|
203
224
|
- operational state adapters
|
|
204
225
|
- destination adapters
|
|
@@ -307,24 +328,35 @@ rather than opening SQLite directly.
|
|
|
307
328
|
|
|
308
329
|
## Performance
|
|
309
330
|
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
331
|
+
A reference benchmark run on Ruby 4.0.5 produced:
|
|
332
|
+
|
|
333
|
+
| Benchmark | Result |
|
|
334
|
+
| --------------------------------------------------- | --------------------: |
|
|
335
|
+
| Event payload projection with explicit ID | **56,265 events/sec** |
|
|
336
|
+
| Transaction payload projection with explicit IDs | **55,612 events/sec** |
|
|
337
|
+
| Mask payload-policy projection | **72,403 events/sec** |
|
|
338
|
+
| Concurrent delivery, 25 workers | **3,825 events/sec** |
|
|
339
|
+
| Concurrent delivery, 50 workers | **7,447 events/sec** |
|
|
340
|
+
| Signed webhook delivery with 10 ms receiver latency | **83.5 requests/sec** |
|
|
341
|
+
| SQLite delivered-ledger writes | **1,630 writes/sec** |
|
|
342
|
+
| Dead-letter replay mechanics | **1,030 entries/sec** |
|
|
343
|
+
|
|
344
|
+
At concurrency 50 with a synthetic 25 ms destination, Mammoth processed
|
|
345
|
+
**1,862 transactions/sec** against a theoretical ceiling of 2,000, while
|
|
346
|
+
maintaining **28.4 ms P95 latency**. This indicates low scheduling overhead for
|
|
347
|
+
I/O-bound downstream delivery.
|
|
348
|
+
|
|
349
|
+
These are local, single-trial reference measurements, not universal capacity
|
|
350
|
+
claims or comparisons with other CDC platforms. The benchmarks isolate specific
|
|
351
|
+
Mammoth components and may exclude PostgreSQL transport, decoding, checkpoint
|
|
352
|
+
coordination, network latency, or destination processing.
|
|
320
353
|
|
|
321
354
|
The [reference snapshot](https://github.com/kanutocd/mammoth/blob/main/benchmark/results/20260724T104950Z/snapshot.md)
|
|
322
|
-
records
|
|
323
|
-
interpretation.
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
DLQ replay expectations.
|
|
355
|
+
records the exact commands, environment, raw output, and detailed
|
|
356
|
+
interpretation. Re-run the scripts in
|
|
357
|
+
[`benchmark/`](https://github.com/kanutocd/mammoth/tree/main/benchmark)
|
|
358
|
+
on your own hardware when choosing `runtime.concurrency`, destination count,
|
|
359
|
+
SQLite storage, scrape frequency, and DLQ replay expectations.
|
|
328
360
|
|
|
329
361
|
Create a publishable benchmark snapshot with:
|
|
330
362
|
|
|
@@ -380,4 +412,4 @@ building a writable database copy. See
|
|
|
380
412
|
|
|
381
413
|
## License
|
|
382
414
|
|
|
383
|
-
Mammoth OSS is licensed under the [MIT License](LICENSE.txt).
|
|
415
|
+
Mammoth OSS Data Plane is licensed under the [MIT License](LICENSE.txt).
|
data/config/mammoth.example.yml
CHANGED
data/config/mammoth.schema.json
CHANGED
|
@@ -158,7 +158,7 @@
|
|
|
158
158
|
"description": "Operator-defined node metadata."
|
|
159
159
|
}
|
|
160
160
|
},
|
|
161
|
-
"description": "Optional local node identity. Mammoth
|
|
161
|
+
"description": "Optional local node identity. Mammoth Data Plane uses this for status and future extension surfaces only."
|
|
162
162
|
},
|
|
163
163
|
"postgres": {
|
|
164
164
|
"type": "object",
|
|
@@ -346,7 +346,7 @@
|
|
|
346
346
|
"enum": [
|
|
347
347
|
"webhook"
|
|
348
348
|
],
|
|
349
|
-
"description": "Destination adapter type. Mammoth
|
|
349
|
+
"description": "Destination adapter type. Mammoth Data Plane 0.7.x supports webhook destinations."
|
|
350
350
|
},
|
|
351
351
|
"enabled": {
|
|
352
352
|
"type": "boolean",
|
|
@@ -520,7 +520,7 @@
|
|
|
520
520
|
"enum": [
|
|
521
521
|
"sqlite"
|
|
522
522
|
],
|
|
523
|
-
"description": "Operational state adapter. Mammoth
|
|
523
|
+
"description": "Operational state adapter. Mammoth Data Plane 0.7.x ships the sqlite adapter."
|
|
524
524
|
}
|
|
525
525
|
},
|
|
526
526
|
"description": "Operational state adapter selection. SQLite remains the default OSS adapter."
|
|
@@ -7,6 +7,7 @@ module Mammoth
|
|
|
7
7
|
# future control agents stable observation points without changing the data
|
|
8
8
|
# plane or introducing remote behavior into OSS.
|
|
9
9
|
class LifecycleHooks
|
|
10
|
+
# Supported lifecycle callback names.
|
|
10
11
|
EVENTS = %i[
|
|
11
12
|
before_start
|
|
12
13
|
after_start
|
|
@@ -15,6 +16,8 @@ module Mammoth
|
|
|
15
16
|
before_replay
|
|
16
17
|
after_replay
|
|
17
18
|
].freeze
|
|
19
|
+
|
|
20
|
+
# Frozen empty callback list used for events without registrations.
|
|
18
21
|
EMPTY_CALLBACKS = [nil].compact.freeze
|
|
19
22
|
|
|
20
23
|
attr_reader :callbacks
|
data/lib/mammoth/logging.rb
CHANGED
|
@@ -6,6 +6,7 @@ require "time"
|
|
|
6
6
|
module Mammoth
|
|
7
7
|
# Container-friendly structured application logging.
|
|
8
8
|
module Logging
|
|
9
|
+
# Supported severity names mapped to their filtering priority.
|
|
9
10
|
LEVELS = { "debug" => 0, "info" => 1, "warn" => 2, "error" => 3 }.freeze
|
|
10
11
|
|
|
11
12
|
# Logger that emits one JSON object per line.
|
|
@@ -19,9 +20,32 @@ module Mammoth
|
|
|
19
20
|
raise ConfigurationError, "unsupported logging level: #{@level}" unless LEVELS.key?(@level)
|
|
20
21
|
end
|
|
21
22
|
|
|
23
|
+
# Emit a debug event when debug logging is enabled.
|
|
24
|
+
#
|
|
25
|
+
# @param event [String, Symbol] stable event name
|
|
26
|
+
# @param context [Hash] structured event fields
|
|
27
|
+
# @return [Boolean] whether the event was emitted
|
|
22
28
|
def debug(event, **context) = write("debug", event, context)
|
|
29
|
+
|
|
30
|
+
# Emit an informational event when info logging is enabled.
|
|
31
|
+
#
|
|
32
|
+
# @param event [String, Symbol] stable event name
|
|
33
|
+
# @param context [Hash] structured event fields
|
|
34
|
+
# @return [Boolean] whether the event was emitted
|
|
23
35
|
def info(event, **context) = write("info", event, context)
|
|
36
|
+
|
|
37
|
+
# Emit a warning event when warn logging is enabled.
|
|
38
|
+
#
|
|
39
|
+
# @param event [String, Symbol] stable event name
|
|
40
|
+
# @param context [Hash] structured event fields
|
|
41
|
+
# @return [Boolean] whether the event was emitted
|
|
24
42
|
def warn(event, **context) = write("warn", event, context)
|
|
43
|
+
|
|
44
|
+
# Emit an error event.
|
|
45
|
+
#
|
|
46
|
+
# @param event [String, Symbol] stable event name
|
|
47
|
+
# @param context [Hash] structured event fields
|
|
48
|
+
# @return [Boolean] whether the event was emitted
|
|
25
49
|
def error(event, **context) = write("error", event, context)
|
|
26
50
|
|
|
27
51
|
def enabled?(severity)
|
|
@@ -42,17 +66,40 @@ module Mammoth
|
|
|
42
66
|
|
|
43
67
|
# No-op logger used at injectable library boundaries.
|
|
44
68
|
class NullLogger
|
|
69
|
+
# Shared immutable no-op logger.
|
|
45
70
|
INSTANCE = new
|
|
46
71
|
|
|
72
|
+
# Suppress a debug event.
|
|
73
|
+
#
|
|
74
|
+
# @return [false]
|
|
47
75
|
def debug(_event, **_context) = false
|
|
76
|
+
|
|
77
|
+
# Suppress an informational event.
|
|
78
|
+
#
|
|
79
|
+
# @return [false]
|
|
48
80
|
def info(_event, **_context) = false
|
|
81
|
+
|
|
82
|
+
# Suppress a warning event.
|
|
83
|
+
#
|
|
84
|
+
# @return [false]
|
|
49
85
|
def warn(_event, **_context) = false
|
|
86
|
+
|
|
87
|
+
# Suppress an error event.
|
|
88
|
+
#
|
|
89
|
+
# @return [false]
|
|
50
90
|
def error(_event, **_context) = false
|
|
91
|
+
|
|
51
92
|
def enabled?(_severity) = false
|
|
52
93
|
end
|
|
53
94
|
|
|
54
95
|
module_function
|
|
55
96
|
|
|
97
|
+
# Build the configured structured logger.
|
|
98
|
+
#
|
|
99
|
+
# @param config [Mammoth::Configuration, Hash] loaded Mammoth configuration
|
|
100
|
+
# @param output [IO] destination for newline-delimited JSON
|
|
101
|
+
# @param clock [#call] UTC-compatible time source
|
|
102
|
+
# @return [Mammoth::Logging::Logger]
|
|
56
103
|
def build(config, output: $stdout, clock: -> { Time.now.utc })
|
|
57
104
|
Logger.new(level: config.dig("logging", "level") || "info", output:, clock:)
|
|
58
105
|
end
|
|
@@ -10,8 +10,13 @@ module Mammoth
|
|
|
10
10
|
# items. Column rules scrub every canonical row-value representation: data,
|
|
11
11
|
# identity, and old/new values in changes.
|
|
12
12
|
class PayloadPolicy
|
|
13
|
+
# Default replacement used by mask rules.
|
|
13
14
|
DEFAULT_MASK = "[REDACTED]"
|
|
15
|
+
|
|
16
|
+
# Payload metadata key containing the applied policy fingerprint.
|
|
14
17
|
POLICY_METADATA_KEY = "mammoth_payload_policy"
|
|
18
|
+
|
|
19
|
+
# Transformation actions accepted by payload-policy rules.
|
|
15
20
|
SUPPORTED_ACTIONS = %w[remove mask].freeze
|
|
16
21
|
|
|
17
22
|
attr_reader :config, :fingerprint
|
|
@@ -70,6 +70,7 @@ module Mammoth
|
|
|
70
70
|
|
|
71
71
|
# Read-only PostgreSQL publication and replica-identity catalog inspector.
|
|
72
72
|
class PostgresPublicationInspector
|
|
73
|
+
# Catalog query used to inspect publication actions and replica identity.
|
|
73
74
|
QUERY = <<~SQL
|
|
74
75
|
SELECT
|
|
75
76
|
relation.oid AS relation_id,
|
data/lib/mammoth/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: mammoth
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.5.
|
|
4
|
+
version: 1.5.2
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Ken C. Demanawa
|
|
@@ -182,14 +182,15 @@ dependencies:
|
|
|
182
182
|
- !ruby/object:Gem::Version
|
|
183
183
|
version: '1.9'
|
|
184
184
|
description: |
|
|
185
|
-
Mammoth is
|
|
185
|
+
Mammoth Data Plane is the open-source PostgreSQL change-event delivery
|
|
186
|
+
runtime in the Mammoth ecosystem.
|
|
186
187
|
|
|
187
188
|
It realizes the CDC Ecosystem pgoutput and cdc-core libraries for PostgreSQL,
|
|
188
189
|
then delivers normalized changes to webhook endpoints with durable
|
|
189
190
|
checkpointing, retry state, dead letters, and operational visibility.
|
|
190
191
|
|
|
191
|
-
Mammoth is application-first: it can be installed as a Ruby gem,
|
|
192
|
-
into a container image, or deployed into Kubernetes with Helm.
|
|
192
|
+
Mammoth Data Plane is application-first: it can be installed as a Ruby gem,
|
|
193
|
+
packaged into a container image, or deployed into Kubernetes with Helm.
|
|
193
194
|
email:
|
|
194
195
|
- kenneth.c.demanawa@gmail.com
|
|
195
196
|
executables:
|
|
@@ -286,6 +287,5 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
286
287
|
requirements: []
|
|
287
288
|
rubygems_version: 4.0.10
|
|
288
289
|
specification_version: 4
|
|
289
|
-
summary:
|
|
290
|
-
dead letters, and operational state.
|
|
290
|
+
summary: Mammoth Data Plane for reliable PostgreSQL change-event delivery.
|
|
291
291
|
test_files: []
|