mammoth 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 137dbf7bc96183b45b2756dcb736884a66a45e920a884d35829f74518f14cef6
-  data.tar.gz: 0dd31f792456297c5ac2e6f16ef6842211a87de63824e2e6707c7bac00a68e59
+  metadata.gz: 19aba46c6ef2ae1099287e37281015b747dde1b32c94cbd1c8ec8a23d6b1a19b
+  data.tar.gz: 21430175d42c71e83eaede3f70390a87d760a9ac2d8b9b4dedcfb446feefaaa0
 SHA512:
-  metadata.gz: ff73c21bfee44d9b5c9d2f02535a0fdd2dadc9b86b868b8219305889b54c3ef50c8818092d6d60c65f96b8fa39097104e64cc7cc694af646bdd1208be128d134
-  data.tar.gz: 991cb45e7f3cbeb18afde25f41d0a1f1c4c3479f920c0551ca33ab2b93b266ad48a9bf258e46c79c68164c1deb7c493c5b252f616563d67f98b9df4c49057a08
+  metadata.gz: b29ea1b4ace38ffa6eb155251dbcd0056e400849aa6a37093d4ecec09cc314089c36523b22b3bbbf898b0247fb6918e3320ef0f14b24d280fed91df2a2287f9a
+  data.tar.gz: b2e6f58f77884641970e838a5aaa8c2c1ef0a2531b868a53c3ae8ee450387128b04b39c2661e5fac507c82d05b11739a913285c8f9c2052db137810881073a3e

data/CHANGELOG.md

@@ -2,6 +2,108 @@
 
 ## Unreleased
 
+## 0.2.0
+
+### Added
+
+- Added transaction-level delivery mode using `TransactionEnvelope`.
+- Added transaction-aware buffering and aggregation of CDC events.
+- Added transaction webhook example demonstrating end-to-end delivery from PostgreSQL logical replication to HTTP webhook.
+- Added source position propagation for transaction deliveries.
+- Added concurrent delivery runtime integration powered by `cdc-concurrent`.
+- Added `DeliveryProcessor` abstraction for runtime execution.
+- Added `TransactionEnvelopeSerializer` for transaction payload serialization.
+- Added rich self-documenting YAML configuration examples.
+- Added configurable webhook HTTP headers through `webhook.headers`.
+- Added environment-backed webhook headers through `webhook.header_env` for secrets such as bearer tokens.
+- Added optional HMAC-SHA256 webhook request signing through `webhook.signing`.
+- Added E2E coverage for duplicate delivery suppression, transaction webhook payloads, dead-letter persistence, and signed/authenticated webhook requests.
+- Added Helm chart rendering for delivery runtime, webhook auth headers, signing, and secret-backed webhook environment values.
+
+### Changed
+
+- Transaction delivery now emits a single webhook payload per committed PostgreSQL transaction.
+- CDC events belonging to the same database transaction are now grouped into a single `TransactionEnvelope`.
+- Delivery pipeline now preserves transaction boundaries from ingestion through webhook delivery.
+- Replication slot creation now correctly honors configured temporary/permanent slot settings.
+- Concurrent runtime integration now complies with `cdc-concurrent` processor safety requirements.
+- Webhook delivery now applies configured static headers, env-backed headers, and per-request signature headers before sending payloads.
+
+### Fixed
+
+- Fixed transaction delivery incorrectly emitting one transaction payload per CDC event.
+- Fixed missing source position metadata in transaction deliveries.
+- Fixed replication slot option handling for boolean configuration values.
+- Fixed transaction webhook example startup and producer execution flow.
+- Fixed `cdc-concurrent` runtime compatibility and processor validation failures.
+- Fixed quality gate drift by restoring passing coverage, RuboCop, Steep, and YARD validation.
+
+### Examples
+
+- Added `examples/transaction_webhook` demonstrating:
+
+  - PostgreSQL logical replication
+  - TransactionEnvelope aggregation
+  - Transaction-level webhook delivery
+  - Source position propagation
+  - Concurrent delivery runtime integration
+
+### Internal
+
+- Established the foundation for transaction-aware checkpointing.
+- Established the foundation for ordering policies based on transaction boundaries.
+- Established the foundation for future multi-destination fanout delivery.
+- Added real `cdc-concurrent` runtime coverage outside the unit-test fake runtime.
+
+
+## [0.1.1] - 2026-06-17
+
+Documentation and operational guidance release.
+
+### Added
+
+* Documentation site structure under `docs/`.
+* Quick Start guide.
+* PostgreSQL setup guide.
+* Configuration reference.
+* CLI reference.
+* Operational state guide.
+* Examples guide.
+* Helm deployment guide.
+* Troubleshooting guide.
+
+### Improved
+
+* GitHub Pages documentation experience.
+* API documentation discoverability.
+* Documentation navigation and onboarding flow.
+* Cross-linking between guides and API reference.
+
+### Clarified
+
+* PostgreSQL logical replication requirements.
+* Publication and replication slot expectations.
+* SQLite operational state responsibilities.
+* Checkpoint persistence behavior.
+* Dead-letter persistence behavior.
+* Helm deployment assumptions.
+* Kubernetes operational considerations.
+
+### Validation
+
+Documentation was aligned with manually validated deployment scenarios:
+
+* PostgreSQL logical replication.
+* Live webhook delivery.
+* Retry exhaustion and dead-letter persistence.
+* SQLite operational state.
+* Helm chart deployment.
+* Kind-based Kubernetes deployment.
+
+### Notes
+
+This release focuses on documentation, onboarding, and operational clarity. No runtime or configuration changes were introduced.
+
 
 ## [0.1.0] - 2026-06-17
 

data/README.md

@@ -5,7 +5,7 @@
 [![Ruby Version](https://img.shields.io/badge/ruby-%3E%3D%204.0-ruby.svg)](https://www.ruby-lang.org/en/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-Mammoth is a self-hosted PostgreSQL event relay focused on reliable delivery
+🦣 Mammoth is a self-hosted PostgreSQL event relay focused on reliable delivery
 of database change events.
 
 ```text
@@ -13,20 +13,31 @@ PostgreSQL
       ↓
 CDC Ecosystem source adapter
       ↓
-CDC::Core::ChangeEvent
+CDC::Core::TransactionEnvelope
       ↓
 Mammoth
       ↓
 Webhook
 ```
 
-Mammoth is intentionally boring infrastructure. It uses YAML configuration,
+🦣 Mammoth is intentionally boring infrastructure. It uses YAML configuration,
 JSON Schema validation, local SQLite operational state, and the CDC Ecosystem's
 shared vocabulary so operators can inspect, recover, and reason about delivery.
 
+## Documentation
+
+Documentation site:
+
+https://kanutocd.github.io/mammoth/
+
+API documentation:
+
+https://kanutocd.github.io/mammoth/Mammoth.html
+
+
 ## OSS MVP
 
-Mammoth OSS includes:
+🦣 Mammoth OSS includes:
 
 - CLI foundation
 - YAML configuration loading
@@ -90,6 +101,50 @@ Mammoth stores operational memory in SQLite:
 - `checkpoints`
 - `dead_letters`
 
+## Performance
+
+Mammoth scales downstream delivery throughput using the `cdc-concurrent` runtime while maintaining a single PostgreSQL logical replication stream.
+
+### Concurrent Delivery Benchmark
+
+Environment:
+
+* 10,000 transactions
+* 4 events per transaction
+* 40,000 total events
+
+#### Fast Downstream (10ms)
+
+| Concurrency | Transactions/sec |
+| ----------- | ---------------: |
+| 1           |            96.50 |
+| 25          |          2419.65 |
+
+#### Realistic Webhook (50ms)
+
+| Concurrency | Transactions/sec |
+| ----------- | ---------------: |
+| 1           |            19.85 |
+| 25          |           495.11 |
+
+Observed throughput improved from:
+
+- 96.50 → 2419.65 transactions/sec (10ms sink latency)
+- 19.85 → 495.11 transactions/sec (50ms sink latency)
+
+when increasing delivery concurrency from 1 to 25.
+
+The benchmark demonstrates near-linear scaling of delivery throughput without increasing PostgreSQL replication connections.
+
+See:
+
+- docs/BENCHMARKS.md
+- examples/transaction_webhook
+
+For full benchmark methodology and results see:
+
+See [Benchmarks](docs/BENCHMARKS.md).
+
 ## E2E
 
 ```bash

data/config/mammoth.example.yml

@@ -1,30 +1,265 @@
 # yaml-language-server: $schema=./mammoth.schema.json
 
+# Mammoth example configuration.
+#
+# This file is intentionally verbose. It is both a runnable starting point and
+# the primary operator-facing documentation for local development, Kubernetes
+# values, and production deployments.
+#
+# Architecture reminder:
+#
+#   PostgreSQL
+#      ↓
+#   one logical replication slot
+#      ↓
+#   one WAL stream
+#      ↓
+#   Mammoth delivery runtime
+#      ↓
+#   one or more downstream deliveries
+#
+# Runtime concurrency affects downstream delivery work only. It does not create
+# additional PostgreSQL replication slots or replication connections.
+
 mammoth:
+  # Human-readable Mammoth instance name.
+  #
+  # Used in logs, status output, metrics, and future operational dashboards.
+  # Keep this stable per environment.
+  #
+  # Examples:
+  #   local_mammoth
+  #   staging_mammoth
+  #   production_mammoth
   name: local_mammoth
 
 postgres:
+  # PostgreSQL host reachable from the Mammoth process or pod.
+  #
+  # For local development this is usually localhost.
+  # In Kubernetes this is usually a service DNS name.
   host: localhost
+
+  # PostgreSQL port.
   port: 5432
+
+  # Source database name.
   database: app_development
+
+  # PostgreSQL user used by Mammoth.
+  #
+  # This user must be able to connect in replication mode and read publication
+  # metadata. In many deployments this means a dedicated replication user.
   username: mammoth
+
+  # Environment variable containing the PostgreSQL password.
+  #
+  # The password is intentionally not stored directly in YAML. In Kubernetes,
+  # set this variable from a Secret.
   password_env: MAMMOTH_POSTGRES_PASSWORD
 
 replication:
+  # PostgreSQL logical replication slot name.
+  #
+  # Permanent slots survive PostgreSQL restarts and preserve WAL until Mammoth
+  # acknowledges progress. Changing this value creates a separate replication
+  # position and should be treated as an operational migration.
+  #
+  # Recommended:
+  #   development: mammoth_dev
+  #   staging:     mammoth_staging
+  #   production:  mammoth_prod
   slot: mammoth_prod
+
+  # PostgreSQL publications to subscribe to.
+  #
+  # Publications define which table changes PostgreSQL sends to Mammoth.
+  # Create them explicitly in production so the database boundary is clear.
+  #
+  # Example SQL:
+  #   CREATE PUBLICATION mammoth_publication FOR TABLE users, orders;
   publications:
     - mammoth_publication
+
+  # Optional start LSN for advanced recovery or controlled replay.
+  #
+  # Leave unset for normal operation. Mammoth will use its checkpoint store and
+  # the replication slot position.
+  #
+  # Example:
+  #   start_lsn: "0/16B6C50"
+  #
+  # start_lsn:
+
+  # Automatically create the replication slot if it does not exist.
+  #
+  # true:
+  #   Convenient for development, demos, and first-time bootstrap.
+  #
+  # false:
+  #   Recommended for production when slots are DBA-managed infrastructure.
   auto_create_slot: false
+
+  # Use a temporary replication slot.
+  #
+  # false:
+  #   Recommended for durable Mammoth deployments. The slot survives reconnects
+  #   and PostgreSQL restarts.
+  #
+  # true:
+  #   Useful for experiments only. A temporary slot disappears when the
+  #   replication session ends and is not suitable for durable delivery.
   temporary_slot: false
+
+  # Standby feedback interval in seconds.
+  #
+  # Mammoth sends periodic feedback while streaming so PostgreSQL knows the
+  # consumer is alive and can advance slot state safely.
+  #
+  # Lower values:
+  #   faster liveness/feedback visibility, slightly more network chatter
+  #
+  # Higher values:
+  #   less chatter, slower visibility into consumer progress
+  #
+  # Recommended:
+  #   development: 2.0
+  #   production:  5.0-10.0
   feedback_interval: 10.0
 
+delivery:
+  # Delivery granularity.
+  #
+  # event:
+  #   Deliver each change event independently. This has lower latency but weaker
+  #   transaction-level semantics.
+  #
+  # transaction:
+  #   Deliver a complete TransactionEnvelope as one payload. This is the safer
+  #   default because checkpointing can advance only after the whole transaction
+  #   has been delivered successfully.
+  #
+  # Recommended:
+  #   transaction
+  unit: transaction
+
+  ordering:
+    # Ordering policy used by the delivery runtime.
+    #
+    # global:
+    #   Preserve complete WAL/stream order. Safest, lowest concurrency.
+    #
+    # transaction:
+    #   Preserve transaction commit order. Recommended for transaction delivery.
+    #
+    # relation:
+    #   Preserve order per table/relation while allowing unrelated tables to run
+    #   concurrently.
+    #
+    # primary_key:
+    #   Preserve order per row identity while allowing unrelated rows to run
+    #   concurrently.
+    #
+    # none:
+    #   Best-effort delivery order. Highest freedom for concurrency, but only use
+    #   when destinations are idempotent and ordering-insensitive.
+    #
+    # Recommended:
+    #   transaction
+    scope: transaction
+
+runtime:
+  # Runtime adapter for downstream delivery work.
+  #
+  # inline:
+  #   Execute delivery work synchronously in the caller. Best for debugging and
+  #   the safest compatibility mode.
+  #
+  # concurrent:
+  #   Use CDC::Concurrent::ProcessorPool for I/O-heavy delivery workloads such as
+  #   webhook HTTP calls, retry sleeps/backoff, and future multi-destination
+  #   fanout.
+  adapter: concurrent
+
+  # Number of downstream delivery workers.
+  #
+  # This affects webhook/destination delivery concurrency only.
+  # It does NOT create additional replication slots.
+  # It does NOT create additional PostgreSQL replication connections.
+  #
+  # Recommended:
+  #   development: 1
+  #   small production: 2-5
+  #   webhook-heavy production: 10-50, after soak testing
+  concurrency: 1
+
+  # Preserve configured ordering while using the runtime.
+  #
+  # true:
+  #   Safer default. Related work is kept in order according to delivery.ordering.
+  #
+  # false:
+  #   Higher throughput potential, but only safe for idempotent destinations that
+  #   do not depend on event order.
+  preserve_order: true
+
+  # Optional runtime timeout in seconds.
+  #
+  # Leave blank to let destination-specific timeouts control delivery behavior.
+  # Useful later for bounding end-to-end processor execution.
+  timeout_seconds:
+
 webhook:
+  # Destination name used in logs, dead-letter records, and future metrics.
   name: primary_webhook
+
+  # Destination URL for delivered CDC payloads.
+  #
+  # For delivery.unit: transaction, the payload represents a TransactionEnvelope.
+  # For delivery.unit: event, the payload represents one change event.
   url: https://example.com/webhooks/postgres
+
+  # HTTP timeout in seconds for webhook delivery attempts.
   timeout_seconds: 5
 
+  # Optional static HTTP headers sent with every webhook request.
+  #
+  # Use these for non-secret routing or receiver metadata. Secret values should
+  # use header_env instead.
+  headers:
+    X-Mammoth-Source: local_mammoth
+
+  # Optional HTTP headers whose values are read from environment variables.
+  #
+  # Example:
+  #   header_env:
+  #     Authorization: MAMMOTH_WEBHOOK_AUTHORIZATION
+  header_env: {}
+
+  # Optional HMAC-SHA256 webhook signing.
+  #
+  # When enabled, Mammoth signs:
+  #
+  #   <timestamp>.<json request body>
+  #
+  # using the secret from secret_env, then sends the timestamp and signature in
+  # configurable headers. The signature value is prefixed with "sha256=".
+  #
+  # signing:
+  #   algorithm: hmac_sha256
+  #   secret_env: MAMMOTH_WEBHOOK_SIGNING_SECRET
+  #   signature_header: X-Mammoth-Signature
+  #   timestamp_header: X-Mammoth-Timestamp
+
 retry:
+  # Maximum delivery attempts before the event/envelope is moved to the dead
+  # letter store.
   max_attempts: 5
+
+  # Retry delay schedule in seconds.
+  #
+  # Mammoth uses this as progressive backoff. If attempts exceed the length of
+  # this list, the final value may be reused by the delivery worker.
   schedule_seconds:
     - 1
     - 5
@@ -33,7 +268,21 @@ retry:
     - 300
 
 sqlite:
+  # Local operational database path.
+  #
+  # Mammoth stores checkpoints, delivery attempts, and dead-letter records here.
+  # In Kubernetes, back this path with a PersistentVolumeClaim.
   path: data/mammoth.db
 
 logging:
+  # Log verbosity.
+  #
+  # debug:
+  #   Very chatty; useful while diagnosing delivery or replication behavior.
+  #
+  # info:
+  #   Recommended normal default.
+  #
+  # warn/error:
+  #   Quieter production modes.
   level: info