mammoth 0.1.1 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3c304ab7c88b218cf4a09e3e672a914510636656941248f9dfd6bff00f1ac454
-  data.tar.gz: 40107e6a3400c92cc72f0c57a41d9b757b49d9bc30e51f8fb7b3a287502db723
+  metadata.gz: 19aba46c6ef2ae1099287e37281015b747dde1b32c94cbd1c8ec8a23d6b1a19b
+  data.tar.gz: 21430175d42c71e83eaede3f70390a87d760a9ac2d8b9b4dedcfb446feefaaa0
 SHA512:
-  metadata.gz: c8f4f13c376eeb33b4768d06883131be409664411501ee3c1d525e508e4109531a58f4dc937d38576aaa9d84c2af6352b30d90b1744597b4c263322c2323dc56
-  data.tar.gz: 85cab27b802aec51af685afac69e10f0b56682d512b9271ff407b383807dab37d9d0d875624ba3a7d718cf1c684de2fbb106de87ad7635c22aef5f347780e39d
+  metadata.gz: b29ea1b4ace38ffa6eb155251dbcd0056e400849aa6a37093d4ecec09cc314089c36523b22b3bbbf898b0247fb6918e3320ef0f14b24d280fed91df2a2287f9a
+  data.tar.gz: b2e6f58f77884641970e838a5aaa8c2c1ef0a2531b868a53c3ae8ee450387128b04b39c2661e5fac507c82d05b11739a913285c8f9c2052db137810881073a3e

data/CHANGELOG.md

@@ -2,6 +2,60 @@
 
 ## Unreleased
 
+## 0.2.0
+
+### Added
+
+- Added transaction-level delivery mode using `TransactionEnvelope`.
+- Added transaction-aware buffering and aggregation of CDC events.
+- Added transaction webhook example demonstrating end-to-end delivery from PostgreSQL logical replication to HTTP webhook.
+- Added source position propagation for transaction deliveries.
+- Added concurrent delivery runtime integration powered by `cdc-concurrent`.
+- Added `DeliveryProcessor` abstraction for runtime execution.
+- Added `TransactionEnvelopeSerializer` for transaction payload serialization.
+- Added rich self-documenting YAML configuration examples.
+- Added configurable webhook HTTP headers through `webhook.headers`.
+- Added environment-backed webhook headers through `webhook.header_env` for secrets such as bearer tokens.
+- Added optional HMAC-SHA256 webhook request signing through `webhook.signing`.
+- Added E2E coverage for duplicate delivery suppression, transaction webhook payloads, dead-letter persistence, and signed/authenticated webhook requests.
+- Added Helm chart rendering for delivery runtime, webhook auth headers, signing, and secret-backed webhook environment values.
+
+### Changed
+
+- Transaction delivery now emits a single webhook payload per committed PostgreSQL transaction.
+- CDC events belonging to the same database transaction are now grouped into a single `TransactionEnvelope`.
+- Delivery pipeline now preserves transaction boundaries from ingestion through webhook delivery.
+- Replication slot creation now correctly honors configured temporary/permanent slot settings.
+- Concurrent runtime integration now complies with `cdc-concurrent` processor safety requirements.
+- Webhook delivery now applies configured static headers, env-backed headers, and per-request signature headers before sending payloads.
+
+### Fixed
+
+- Fixed transaction delivery incorrectly emitting one transaction payload per CDC event.
+- Fixed missing source position metadata in transaction deliveries.
+- Fixed replication slot option handling for boolean configuration values.
+- Fixed transaction webhook example startup and producer execution flow.
+- Fixed `cdc-concurrent` runtime compatibility and processor validation failures.
+- Fixed quality gate drift by restoring passing coverage, RuboCop, Steep, and YARD validation.
+
+### Examples
+
+- Added `examples/transaction_webhook` demonstrating:
+
+  - PostgreSQL logical replication
+  - TransactionEnvelope aggregation
+  - Transaction-level webhook delivery
+  - Source position propagation
+  - Concurrent delivery runtime integration
+
+### Internal
+
+- Established the foundation for transaction-aware checkpointing.
+- Established the foundation for ordering policies based on transaction boundaries.
+- Established the foundation for future multi-destination fanout delivery.
+- Added real `cdc-concurrent` runtime coverage outside the unit-test fake runtime.
+
+
 ## [0.1.1] - 2026-06-17
 
 Documentation and operational guidance release.

data/README.md

@@ -13,7 +13,7 @@ PostgreSQL
       ↓
 CDC Ecosystem source adapter
       ↓
-CDC::Core::ChangeEvent
+CDC::Core::TransactionEnvelope
       ↓
 Mammoth
       ↓
@@ -24,10 +24,6 @@ Webhook
 JSON Schema validation, local SQLite operational state, and the CDC Ecosystem's
 shared vocabulary so operators can inspect, recover, and reason about delivery.
 
-🦣 Mammoth is intentionally boring infrastructure. It uses YAML configuration,
-JSON Schema validation, local SQLite operational state, and the CDC Ecosystem's
-shared vocabulary so operators can inspect, recover, and reason about delivery.
-
 ## Documentation
 
 Documentation site:
@@ -105,6 +101,50 @@ Mammoth stores operational memory in SQLite:
 - `checkpoints`
 - `dead_letters`
 
+## Performance
+
+Mammoth scales downstream delivery throughput using the `cdc-concurrent` runtime while maintaining a single PostgreSQL logical replication stream.
+
+### Concurrent Delivery Benchmark
+
+Environment:
+
+* 10,000 transactions
+* 4 events per transaction
+* 40,000 total events
+
+#### Fast Downstream (10ms)
+
+| Concurrency | Transactions/sec |
+| ----------- | ---------------: |
+| 1           |            96.50 |
+| 25          |          2419.65 |
+
+#### Realistic Webhook (50ms)
+
+| Concurrency | Transactions/sec |
+| ----------- | ---------------: |
+| 1           |            19.85 |
+| 25          |           495.11 |
+
+Observed throughput improved from:
+
+- 96.50 → 2419.65 transactions/sec (10ms sink latency)
+- 19.85 → 495.11 transactions/sec (50ms sink latency)
+
+when increasing delivery concurrency from 1 to 25.
+
+The benchmark demonstrates near-linear scaling of delivery throughput without increasing PostgreSQL replication connections.
+
+See:
+
+- docs/BENCHMARKS.md
+- examples/transaction_webhook
+
+For full benchmark methodology and results see:
+
+See [Benchmarks](docs/BENCHMARKS.md).
+
 ## E2E
 
 ```bash

data/config/mammoth.example.yml

@@ -1,30 +1,265 @@
 # yaml-language-server: $schema=./mammoth.schema.json
 
+# Mammoth example configuration.
+#
+# This file is intentionally verbose. It is both a runnable starting point and
+# the primary operator-facing documentation for local development, Kubernetes
+# values, and production deployments.
+#
+# Architecture reminder:
+#
+#   PostgreSQL
+#      ↓
+#   one logical replication slot
+#      ↓
+#   one WAL stream
+#      ↓
+#   Mammoth delivery runtime
+#      ↓
+#   one or more downstream deliveries
+#
+# Runtime concurrency affects downstream delivery work only. It does not create
+# additional PostgreSQL replication slots or replication connections.
+
 mammoth:
+  # Human-readable Mammoth instance name.
+  #
+  # Used in logs, status output, metrics, and future operational dashboards.
+  # Keep this stable per environment.
+  #
+  # Examples:
+  #   local_mammoth
+  #   staging_mammoth
+  #   production_mammoth
   name: local_mammoth
 
 postgres:
+  # PostgreSQL host reachable from the Mammoth process or pod.
+  #
+  # For local development this is usually localhost.
+  # In Kubernetes this is usually a service DNS name.
   host: localhost
+
+  # PostgreSQL port.
   port: 5432
+
+  # Source database name.
   database: app_development
+
+  # PostgreSQL user used by Mammoth.
+  #
+  # This user must be able to connect in replication mode and read publication
+  # metadata. In many deployments this means a dedicated replication user.
   username: mammoth
+
+  # Environment variable containing the PostgreSQL password.
+  #
+  # The password is intentionally not stored directly in YAML. In Kubernetes,
+  # set this variable from a Secret.
   password_env: MAMMOTH_POSTGRES_PASSWORD
 
 replication:
+  # PostgreSQL logical replication slot name.
+  #
+  # Permanent slots survive PostgreSQL restarts and preserve WAL until Mammoth
+  # acknowledges progress. Changing this value creates a separate replication
+  # position and should be treated as an operational migration.
+  #
+  # Recommended:
+  #   development: mammoth_dev
+  #   staging:     mammoth_staging
+  #   production:  mammoth_prod
   slot: mammoth_prod
+
+  # PostgreSQL publications to subscribe to.
+  #
+  # Publications define which table changes PostgreSQL sends to Mammoth.
+  # Create them explicitly in production so the database boundary is clear.
+  #
+  # Example SQL:
+  #   CREATE PUBLICATION mammoth_publication FOR TABLE users, orders;
   publications:
     - mammoth_publication
+
+  # Optional start LSN for advanced recovery or controlled replay.
+  #
+  # Leave unset for normal operation. Mammoth will use its checkpoint store and
+  # the replication slot position.
+  #
+  # Example:
+  #   start_lsn: "0/16B6C50"
+  #
+  # start_lsn:
+
+  # Automatically create the replication slot if it does not exist.
+  #
+  # true:
+  #   Convenient for development, demos, and first-time bootstrap.
+  #
+  # false:
+  #   Recommended for production when slots are DBA-managed infrastructure.
   auto_create_slot: false
+
+  # Use a temporary replication slot.
+  #
+  # false:
+  #   Recommended for durable Mammoth deployments. The slot survives reconnects
+  #   and PostgreSQL restarts.
+  #
+  # true:
+  #   Useful for experiments only. A temporary slot disappears when the
+  #   replication session ends and is not suitable for durable delivery.
   temporary_slot: false
+
+  # Standby feedback interval in seconds.
+  #
+  # Mammoth sends periodic feedback while streaming so PostgreSQL knows the
+  # consumer is alive and can advance slot state safely.
+  #
+  # Lower values:
+  #   faster liveness/feedback visibility, slightly more network chatter
+  #
+  # Higher values:
+  #   less chatter, slower visibility into consumer progress
+  #
+  # Recommended:
+  #   development: 2.0
+  #   production:  5.0-10.0
   feedback_interval: 10.0
 
+delivery:
+  # Delivery granularity.
+  #
+  # event:
+  #   Deliver each change event independently. This has lower latency but weaker
+  #   transaction-level semantics.
+  #
+  # transaction:
+  #   Deliver a complete TransactionEnvelope as one payload. This is the safer
+  #   default because checkpointing can advance only after the whole transaction
+  #   has been delivered successfully.
+  #
+  # Recommended:
+  #   transaction
+  unit: transaction
+
+  ordering:
+    # Ordering policy used by the delivery runtime.
+    #
+    # global:
+    #   Preserve complete WAL/stream order. Safest, lowest concurrency.
+    #
+    # transaction:
+    #   Preserve transaction commit order. Recommended for transaction delivery.
+    #
+    # relation:
+    #   Preserve order per table/relation while allowing unrelated tables to run
+    #   concurrently.
+    #
+    # primary_key:
+    #   Preserve order per row identity while allowing unrelated rows to run
+    #   concurrently.
+    #
+    # none:
+    #   Best-effort delivery order. Highest freedom for concurrency, but only use
+    #   when destinations are idempotent and ordering-insensitive.
+    #
+    # Recommended:
+    #   transaction
+    scope: transaction
+
+runtime:
+  # Runtime adapter for downstream delivery work.
+  #
+  # inline:
+  #   Execute delivery work synchronously in the caller. Best for debugging and
+  #   the safest compatibility mode.
+  #
+  # concurrent:
+  #   Use CDC::Concurrent::ProcessorPool for I/O-heavy delivery workloads such as
+  #   webhook HTTP calls, retry sleeps/backoff, and future multi-destination
+  #   fanout.
+  adapter: concurrent
+
+  # Number of downstream delivery workers.
+  #
+  # This affects webhook/destination delivery concurrency only.
+  # It does NOT create additional replication slots.
+  # It does NOT create additional PostgreSQL replication connections.
+  #
+  # Recommended:
+  #   development: 1
+  #   small production: 2-5
+  #   webhook-heavy production: 10-50, after soak testing
+  concurrency: 1
+
+  # Preserve configured ordering while using the runtime.
+  #
+  # true:
+  #   Safer default. Related work is kept in order according to delivery.ordering.
+  #
+  # false:
+  #   Higher throughput potential, but only safe for idempotent destinations that
+  #   do not depend on event order.
+  preserve_order: true
+
+  # Optional runtime timeout in seconds.
+  #
+  # Leave blank to let destination-specific timeouts control delivery behavior.
+  # Useful later for bounding end-to-end processor execution.
+  timeout_seconds:
+
 webhook:
+  # Destination name used in logs, dead-letter records, and future metrics.
   name: primary_webhook
+
+  # Destination URL for delivered CDC payloads.
+  #
+  # For delivery.unit: transaction, the payload represents a TransactionEnvelope.
+  # For delivery.unit: event, the payload represents one change event.
   url: https://example.com/webhooks/postgres
+
+  # HTTP timeout in seconds for webhook delivery attempts.
   timeout_seconds: 5
 
+  # Optional static HTTP headers sent with every webhook request.
+  #
+  # Use these for non-secret routing or receiver metadata. Secret values should
+  # use header_env instead.
+  headers:
+    X-Mammoth-Source: local_mammoth
+
+  # Optional HTTP headers whose values are read from environment variables.
+  #
+  # Example:
+  #   header_env:
+  #     Authorization: MAMMOTH_WEBHOOK_AUTHORIZATION
+  header_env: {}
+
+  # Optional HMAC-SHA256 webhook signing.
+  #
+  # When enabled, Mammoth signs:
+  #
+  #   <timestamp>.<json request body>
+  #
+  # using the secret from secret_env, then sends the timestamp and signature in
+  # configurable headers. The signature value is prefixed with "sha256=".
+  #
+  # signing:
+  #   algorithm: hmac_sha256
+  #   secret_env: MAMMOTH_WEBHOOK_SIGNING_SECRET
+  #   signature_header: X-Mammoth-Signature
+  #   timestamp_header: X-Mammoth-Timestamp
+
 retry:
+  # Maximum delivery attempts before the event/envelope is moved to the dead
+  # letter store.
   max_attempts: 5
+
+  # Retry delay schedule in seconds.
+  #
+  # Mammoth uses this as progressive backoff. If attempts exceed the length of
+  # this list, the final value may be reused by the delivery worker.
   schedule_seconds:
     - 1
     - 5
@@ -33,7 +268,21 @@ retry:
     - 300
 
 sqlite:
+  # Local operational database path.
+  #
+  # Mammoth stores checkpoints, delivery attempts, and dead-letter records here.
+  # In Kubernetes, back this path with a PersistentVolumeClaim.
   path: data/mammoth.db
 
 logging:
+  # Log verbosity.
+  #
+  # debug:
+  #   Very chatty; useful while diagnosing delivery or replication behavior.
+  #
+  # info:
+  #   Recommended normal default.
+  #
+  # warn/error:
+  #   Quieter production modes.
   level: info