mammoth 0.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 8226f7d0510693efd8d3c88623f346bcd24e88f80935c54581994d23b5a64767
+  data.tar.gz: 69a7da12ff8b589851ea0a51d3a68e7c25a2521c49eaaf222afef0a020a5d524
+SHA512:
+  metadata.gz: 4ecaf03600872e85d979e4c546e2e5f102dea80331ba216d18dd4d8df3c1c09d4c17148634f09b17f755cff86051e2205e290ffe1e4be1a760d9e5851237b179
+  data.tar.gz: 4b8ade4da78b6a4db908fb909c7a11b6dad9572a0d8d4bf3fdb55a55d6b491adb61785b89888deea4c8c9e140cb9c48d78847a5190f4601a03228de4bd6ba577

data/CHANGELOG.md

@@ -0,0 +1,14 @@
+# Changelog
+
+## 0.1.0 - Unreleased
+
+- Rename product and gem from Echo to Mammoth.
+- Position Mammoth OSS as the reliable PostgreSQL change-event delivery appliance.
+- Add pgoutput-client / parser / decoder / source-adapter integration boundary.
+- Serialize CDC-core `ChangeEvent` shaped work into webhook payloads.
+- Flatten CDC-core `TransactionEnvelope` shaped work before sink delivery.
+- Add `mammoth start CONFIG` CLI command for live operation.
+- Add public Helm chart under `charts/mammoth`.
+- Add slim multi-stage Dockerfile for OSS image builds.
+- Add e2e test task and script using real HTTP, SQLite, and filesystem paths.
+- Switch OSS license metadata to MIT.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kenneth C. Demanawa
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,125 @@
+# Mammoth
+
+[![Gem Version](https://badge.fury.io/rb/mammoth.svg)](https://badge.fury.io/rb/mammoth)
+[![CI](https://github.com/kanutocd/mammoth/workflows/CI/badge.svg)](https://github.com/kanutocd/mammoth/actions)
+[![Ruby Version](https://img.shields.io/badge/ruby-%3E%3D%203.4-ruby.svg)](https://www.ruby-lang.org/en/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+Mammoth is a self-hosted PostgreSQL event relay focused on reliable delivery
+of database change events.
+
+```text
+PostgreSQL
+      ↓
+pgoutput-client / parser / decoder
+      ↓
+Pgoutput::SourceAdapter::Cdc
+      ↓
+CDC::Core::ChangeEvent
+      ↓
+Mammoth
+      ↓
+Webhook
+```
+
+Mammoth is intentionally boring infrastructure. It uses YAML configuration,
+JSON Schema validation, local SQLite operational state, and the CDC Ecosystem's
+shared vocabulary so operators can inspect, recover, and reason about delivery.
+
+## OSS MVP
+
+Mammoth OSS includes:
+
+- CLI foundation
+- YAML configuration loading
+- JSON Schema-backed configuration validation
+- SQLite operational memory bootstrap
+- checkpoint persistence
+- dead letter persistence
+- webhook delivery sink
+- delivery worker with retry, checkpoint, and DLQ handling
+- CDC-core event serialization boundary
+- pgoutput-client / parser / decoder / source-adapter integration boundary
+- Docker image support
+- public Helm chart support
+- unit and e2e test tasks
+
+## Boundary
+
+Mammoth begins at CDC-core work items and ends at webhook delivery.
+
+Mammoth does not own pgoutput protocol parsing, value decoding, source
+normalization, ordering policy, or runtime execution. Those belong to the
+upstream CDC Ecosystem components.
+
+## Configuration
+
+Mammoth configuration is YAML-backed and IDE-friendly.
+
+```yaml
+# yaml-language-server: $schema=./mammoth.schema.json
+```
+
+Validate configuration:
+
+```bash
+bundle exec ./exe/mammoth validate config/mammoth.example.yml
+```
+
+## CLI
+
+```bash
+bundle exec ./exe/mammoth version
+bundle exec ./exe/mammoth validate config/mammoth.example.yml
+bundle exec ./exe/mammoth bootstrap config/mammoth.example.yml
+bundle exec ./exe/mammoth status config/mammoth.example.yml
+bundle exec ./exe/mammoth start config/mammoth.example.yml
+```
+
+Deliver a single normalized event JSON file through Mammoth's delivery path:
+
+```bash
+bundle exec ./exe/mammoth deliver-sample \
+  examples/postgres_webhook/config/mammoth.yml \
+  examples/postgres_webhook/events/order_insert.json
+```
+
+## SQLite Operational State
+
+Mammoth stores operational memory in SQLite:
+
+- `schema_migrations`
+- `checkpoints`
+- `dead_letters`
+
+## E2E
+
+```bash
+bundle exec rake test:e2e
+# or
+script/test-e2e
+```
+
+The e2e task uses a real HTTP receiver, real SQLite database, and real
+filesystem paths.
+
+## Kubernetes
+
+The public Helm chart lives under:
+
+```text
+charts/mammoth
+```
+
+Install example:
+
+```bash
+helm install mammoth charts/mammoth
+```
+
+The chart uses one replica and `Recreate` strategy to respect PostgreSQL's
+logical replication slot constraint: one slot, one active subscriber.
+
+## License
+
+Mammoth OSS is licensed under the [MIT License](LICENSE.txt).

data/lib/mammoth/application.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+module Mammoth
+  # Top-level Mammoth application composition root.
+  #
+  # Application wires Mammoth's current v0.1.0 runtime pieces: configuration,
+  # SQLite operational memory, replication boundary, delivery worker, checkpoint
+  # store, dead letter store, and webhook sink.
+  class Application
+    attr_reader :config, :sqlite_store, :consumer, :delivery_worker
+
+    # @param config [Mammoth::Configuration] loaded configuration
+    # @param source [#each, nil] injectable event source for tests and demos
+    # @param adapter [#call, nil] optional source adapter
+    # @param sink [#deliver, nil] optional destination sink
+    # @param sleeper [#call] retry sleep strategy
+    def initialize(config, source: nil, adapter: nil, sink: nil, sleeper: Kernel.method(:sleep))
+      @config = config
+      @sqlite_store = SQLiteStore.connect(config.dig("sqlite", "path")).bootstrap!
+      @consumer = ReplicationConsumer.new(config, source: source, adapter: adapter)
+      @delivery_worker = build_delivery_worker(sink: sink || WebhookSink.from_config(config), sleeper: sleeper)
+    end
+
+    # Start the application runtime and deliver consumed events.
+    #
+    # @return [Integer] number of processed events
+    def start
+      processed = 0
+      consumer.start do |event|
+        delivery_worker.deliver(event)
+        processed += 1
+      end
+      processed
+    end
+
+    private
+
+    def build_delivery_worker(sink:, sleeper:)
+      DeliveryWorker.from_config(
+        config,
+        sink: sink,
+        checkpoint_store: CheckpointStore.new(sqlite_store),
+        dead_letter_store: DeadLetterStore.new(sqlite_store),
+        sleeper: sleeper
+      )
+    end
+  end
+end

data/lib/mammoth/checkpoint_store.rb

@@ -0,0 +1,64 @@
+# frozen_string_literal: true
+
+require "time"
+
+module Mammoth
+  # Persists source checkpoints in Mammoth's SQLite operational store.
+  class CheckpointStore
+    attr_reader :sqlite_store
+
+    # @param sqlite_store [Mammoth::SQLiteStore] bootstrapped SQLite store
+    def initialize(sqlite_store)
+      @sqlite_store = sqlite_store
+    end
+
+    # Insert or update the last successfully delivered source position.
+    #
+    # @param source_name [String] logical source name
+    # @param slot_name [String] PostgreSQL replication slot name
+    # @param publication_name [String] PostgreSQL publication name
+    # @param last_lsn [String, nil] last delivered LSN/source position
+    # @return [Hash] stored checkpoint row
+    def write(source_name:, slot_name:, publication_name:, last_lsn:)
+      now = Time.now.utc.iso8601
+      database.execute(
+        <<~SQL,
+          INSERT INTO checkpoints(source_name, slot_name, publication_name, last_lsn, updated_at)
+          VALUES (?, ?, ?, ?, ?)
+          ON CONFLICT(source_name, slot_name)
+          DO UPDATE SET
+            publication_name = excluded.publication_name,
+            last_lsn = excluded.last_lsn,
+            updated_at = excluded.updated_at
+        SQL
+        [source_name, slot_name, publication_name, last_lsn, now]
+      )
+      fetch(source_name: source_name, slot_name: slot_name)
+    end
+
+    # Fetch a checkpoint row.
+    #
+    # @param source_name [String] logical source name
+    # @param slot_name [String] PostgreSQL replication slot name
+    # @return [Hash, nil] checkpoint row or nil
+    def fetch(source_name:, slot_name:)
+      database.get_first_row(
+        "SELECT * FROM checkpoints WHERE source_name = ? AND slot_name = ? LIMIT 1",
+        [source_name, slot_name]
+      )
+    end
+
+    # Count checkpoint rows.
+    #
+    # @return [Integer] checkpoint count
+    def count
+      database.get_first_value("SELECT COUNT(*) FROM checkpoints")
+    end
+
+    private
+
+    def database
+      sqlite_store.bootstrap!.database
+    end
+  end
+end

data/lib/mammoth/cli.rb

@@ -0,0 +1,116 @@
+# frozen_string_literal: true
+
+require "json"
+
+module Mammoth
+  # Small command dispatcher for Mammoth's operator-facing CLI.
+  class CLI
+    USAGE = [
+      "Usage:",
+      "  mammoth version",
+      "  mammoth validate CONFIG",
+      "  mammoth bootstrap CONFIG",
+      "  mammoth status CONFIG",
+      "  mammoth start CONFIG",
+      "  mammoth deliver-sample CONFIG EVENT_JSON"
+    ].join("\n")
+
+    # Run the CLI.
+    #
+    # @param argv [Array<String>] command line arguments
+    # @return [Integer] process status code
+    def self.call(argv)
+      new(argv).call
+    end
+
+    attr_reader :argv
+
+    # @param argv [Array<String>] command line arguments
+    def initialize(argv)
+      @argv = argv
+    end
+
+    # Dispatch the requested command.
+    #
+    # @return [Integer] process status code
+    def call
+      case command
+      when "version" then version
+      when "validate" then validate
+      when "bootstrap" then bootstrap
+      when "status" then status
+      when "start" then start
+      when "deliver-sample" then deliver_sample
+      else
+        warn USAGE
+        1
+      end
+    rescue Mammoth::Error => e
+      warn e.message
+      1
+    end
+
+    private
+
+    def command
+      argv.fetch(0, nil)
+    end
+
+    def config_path
+      argv.fetch(1, nil)
+    end
+
+    def version
+      puts "Mammoth #{Mammoth::VERSION}"
+      0
+    end
+
+    def validate
+      load_config
+      puts "Configuration OK: #{config_path}"
+      0
+    end
+
+    def bootstrap
+      config = load_config
+      store = SQLiteStore.connect(config.dig("sqlite", "path")).bootstrap!
+      puts "SQLite database initialized"
+      puts "Path: #{store.path}"
+      puts "Tables: #{store.tables.join(", ")}"
+      0
+    end
+
+    def status
+      config = load_config
+      store = SQLiteStore.connect(config.dig("sqlite", "path"))
+      Status.call(config, sqlite_store: store)
+      0
+    end
+
+    def start
+      processed = Application.new(load_config).start
+      puts "Delivered events: #{processed}"
+      0
+    end
+
+    def deliver_sample
+      config = load_config
+      event_path = argv.fetch(2, nil)
+      raise ConfigurationError, "event JSON path required\n#{USAGE}" unless event_path
+      raise ConfigurationError, "event JSON file not found: #{event_path}" unless File.file?(event_path)
+
+      event = JSON.parse(File.read(event_path))
+      processed = Application.new(config, source: [event]).start
+      puts "Delivered sample events: #{processed}"
+      0
+    rescue JSON::ParserError => e
+      raise ConfigurationError, "invalid event JSON in #{event_path}: #{e.message}"
+    end
+
+    def load_config
+      raise ConfigurationError, "configuration path required\n#{USAGE}" unless config_path
+
+      Configuration.load(config_path)
+    end
+  end
+end

data/lib/mammoth/configuration.rb

@@ -0,0 +1,76 @@
+# frozen_string_literal: true
+
+require "json"
+require "json-schema"
+require "yaml"
+
+module Mammoth
+  # Loads and validates Mammoth YAML configuration.
+  #
+  # Configuration is intentionally schema-backed so the same contract can power
+  # editor IntelliSense, preflight validation, and runtime startup checks.
+  class Configuration
+    DEFAULT_SCHEMA_PATH = File.expand_path("../../config/mammoth.schema.json", __dir__)
+
+    attr_reader :path, :data, :schema_path
+
+    # Load and validate a configuration file.
+    #
+    # @param path [String] YAML configuration path
+    # @param schema_path [String] JSON Schema path
+    # @return [Mammoth::Configuration] loaded configuration
+    # @raise [Mammoth::ConfigurationError] when the file is missing or invalid
+    def self.load(path, schema_path: DEFAULT_SCHEMA_PATH)
+      new(path, schema_path: schema_path).load
+    end
+
+    # @param path [String] YAML configuration path
+    # @param schema_path [String] JSON Schema path
+    def initialize(path, schema_path: DEFAULT_SCHEMA_PATH)
+      @path = path
+      @schema_path = schema_path
+      @data = nil
+    end
+
+    # Load and validate the configuration file.
+    #
+    # @return [Mammoth::Configuration] self
+    # @raise [Mammoth::ConfigurationError] when validation fails
+    def load
+      raise ConfigurationError, "configuration file not found: #{path}" unless File.file?(path)
+
+      @data = YAML.safe_load_file(path, permitted_classes: [], aliases: false)
+      raise ConfigurationError, "configuration must be a YAML mapping" unless data.is_a?(Hash)
+
+      validate_schema!
+      self
+    rescue Psych::SyntaxError => e
+      raise ConfigurationError, "invalid YAML in #{path}: #{e.message}"
+    end
+
+    # Fetch a nested value from the loaded configuration.
+    #
+    # @param keys [Array<String>] nested hash keys
+    # @return [Object, nil] value or nil
+    def dig(*keys)
+      data&.dig(*keys)
+    end
+
+    private
+
+    def validate_schema!
+      raise ConfigurationError, "schema file not found: #{schema_path}" unless File.file?(schema_path)
+
+      schema = JSON.parse(File.read(schema_path))
+      JSON::Validator.fully_validate(schema, data, validate_schema: false).tap do |errors|
+        raise ConfigurationError, schema_error_message(errors) unless errors.empty?
+      end
+    rescue JSON::ParserError => e
+      raise ConfigurationError, "invalid JSON schema in #{schema_path}: #{e.message}"
+    end
+
+    def schema_error_message(errors)
+      (["configuration failed schema validation:"] + errors.map { |error| "- #{error}" }).join("\n")
+    end
+  end
+end

data/lib/mammoth/dead_letter_store.rb

@@ -0,0 +1,116 @@
+# frozen_string_literal: true
+
+require "json"
+require "time"
+
+module Mammoth
+  # Persists failed deliveries in Mammoth's SQLite dead letter queue.
+  class DeadLetterStore
+    attr_reader :sqlite_store
+
+    # @param sqlite_store [Mammoth::SQLiteStore] bootstrapped SQLite store
+    def initialize(sqlite_store)
+      @sqlite_store = sqlite_store
+    end
+
+    # Store a failed delivery.
+    #
+    # @param event [Hash] normalized event payload
+    # @param destination_name [String] destination name
+    # @param error [Exception, nil] delivery failure
+    # @param retry_count [Integer] number of delivery attempts
+    # @return [Integer] inserted dead letter id
+    def write(event:, destination_name:, error: nil, retry_count: 0) # rubocop:disable Metrics/MethodLength
+      now = Time.now.utc.iso8601
+      payload = EventSerializer.call(event)
+      database.execute(
+        <<~SQL,
+          INSERT INTO dead_letters(
+            event_id,
+            source_name,
+            destination_name,
+            operation,
+            namespace,
+            entity,
+            source_position,
+            payload_json,
+            error_class,
+            error_message,
+            retry_count,
+            status,
+            failed_at,
+            updated_at
+          ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending', ?, ?)
+        SQL
+        [
+          payload.fetch("event_id"),
+          payload.fetch("source"),
+          destination_name,
+          payload.fetch("operation"),
+          payload["namespace"],
+          payload["entity"],
+          payload["source_position"],
+          JSON.generate(payload),
+          error&.class&.name,
+          error&.message,
+          retry_count,
+          now,
+          now
+        ]
+      )
+      database.last_insert_row_id
+    end
+
+    # Fetch pending dead letters.
+    #
+    # @param limit [Integer] maximum number of rows
+    # @return [Array<Hash>] pending dead letter rows
+    def pending(limit: 100)
+      database.execute(
+        "SELECT * FROM dead_letters WHERE status = ? ORDER BY failed_at ASC LIMIT ?",
+        ["pending", limit]
+      )
+    end
+
+    # Count dead letters by status.
+    #
+    # @param status [String, nil] optional status
+    # @return [Integer] dead letter count
+    def count(status: nil)
+      if status
+        database.get_first_value("SELECT COUNT(*) FROM dead_letters WHERE status = ?", [status])
+      else
+        database.get_first_value("SELECT COUNT(*) FROM dead_letters")
+      end
+    end
+
+    # Mark a dead letter as resolved.
+    #
+    # @param id [Integer] dead letter id
+    # @return [void]
+    def resolve(id)
+      update_status(id, "resolved")
+    end
+
+    # Mark a dead letter as ignored.
+    #
+    # @param id [Integer] dead letter id
+    # @return [void]
+    def ignore(id)
+      update_status(id, "ignored")
+    end
+
+    private
+
+    def database
+      sqlite_store.bootstrap!.database
+    end
+
+    def update_status(id, status)
+      database.execute(
+        "UPDATE dead_letters SET status = ?, updated_at = ? WHERE id = ?",
+        [status, Time.now.utc.iso8601, id]
+      )
+    end
+  end
+end

data/lib/mammoth/delivery_worker.rb

@@ -0,0 +1,107 @@
+# frozen_string_literal: true
+
+module Mammoth
+  # Delivers normalized events with retry, checkpoint, and dead-letter handling.
+  #
+  # DeliveryWorker is Mammoth's first reliable delivery unit. It intentionally keeps
+  # the delivery contract small: attempt webhook delivery, advance the checkpoint
+  # after success, and persist the failed event to the dead letter queue after
+  # retry exhaustion.
+  class DeliveryWorker
+    DEFAULT_SOURCE = "postgresql"
+
+    attr_reader :sink, :checkpoint_store, :dead_letter_store, :retry_schedule, :max_attempts, :sleeper, :source_name,
+                :slot_name, :publication_name
+
+    # @param sink [#deliver] destination sink
+    # @param checkpoint_store [Mammoth::CheckpointStore] checkpoint persistence
+    # @param dead_letter_store [Mammoth::DeadLetterStore] dead letter persistence
+    # @param source_name [String] logical source name
+    # @param slot_name [String] replication slot name
+    # @param publication_name [String] publication name
+    # @param max_attempts [Integer] maximum delivery attempts
+    # @param retry_schedule [Array<Integer>] retry wait schedule in seconds
+    # @param sleeper [#call] sleep strategy, injectable for tests
+    def initialize(sink:, checkpoint_store:, dead_letter_store:, source_name:, slot_name:, publication_name:,
+                   max_attempts:, retry_schedule:, sleeper: Kernel.method(:sleep))
+      @sink = sink
+      @checkpoint_store = checkpoint_store
+      @dead_letter_store = dead_letter_store
+      @source_name = source_name
+      @slot_name = slot_name
+      @publication_name = publication_name
+      @max_attempts = max_attempts
+      @retry_schedule = retry_schedule
+      @sleeper = sleeper
+    end
+
+    # Build a delivery worker from Mammoth configuration and stores.
+    #
+    # @param config [Mammoth::Configuration] loaded configuration
+    # @param sink [#deliver] destination sink
+    # @param checkpoint_store [Mammoth::CheckpointStore] checkpoint persistence
+    # @param dead_letter_store [Mammoth::DeadLetterStore] dead letter persistence
+    # @param sleeper [#call] sleep strategy
+    # @return [Mammoth::DeliveryWorker]
+    def self.from_config(config, sink:, checkpoint_store:, dead_letter_store:, sleeper: Kernel.method(:sleep))
+      new(
+        sink: sink,
+        checkpoint_store: checkpoint_store,
+        dead_letter_store: dead_letter_store,
+        source_name: config.dig("mammoth", "name"),
+        slot_name: config.dig("replication", "slot"),
+        publication_name: config.dig("replication", "publication"),
+        max_attempts: config.dig("retry", "max_attempts"),
+        retry_schedule: config.dig("retry", "schedule_seconds"),
+        sleeper: sleeper
+      )
+    end
+
+    # Deliver an event with retry, checkpoint, and DLQ handling.
+    #
+    # @param event [Hash, #to_h] normalized event
+    # @return [Hash] delivery summary
+    def deliver(event)
+      attempts = 0
+
+      begin
+        attempts += 1
+        result = sink.deliver(event)
+        checkpoint(event)
+        result.merge(attempts: attempts)
+      rescue DeliveryError => e
+        return dead_letter(event, e, attempts) if attempts >= max_attempts
+
+        wait_before_retry(attempts)
+        retry
+      end
+    end
+
+    private
+
+    def checkpoint(event)
+      payload = EventSerializer.call(event)
+      checkpoint_store.write(
+        source_name: source_name,
+        slot_name: slot_name,
+        publication_name: publication_name,
+        last_lsn: payload["source_position"]
+      )
+    end
+
+    def dead_letter(event, error, attempts)
+      id = dead_letter_store.write(
+        event: event,
+        destination_name: sink.name,
+        error: error,
+        retry_count: attempts
+      )
+      { status: "dead_lettered", dead_letter_id: id, attempts: attempts }
+    end
+
+    def wait_before_retry(attempts)
+      wait_seconds = retry_schedule.fetch(attempts - 1, retry_schedule.last)
+      sleeper.call(wait_seconds)
+    end
+  end
+end