pgoutput-source-adapter 0.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 9c0d3598bffe622103beb9a3f48c750476dcb06ec81b6d675d3a9b4628d61d1d
+  data.tar.gz: b4ab4d056f6d9f34620c88d78d28ddc6ccf678463b9975b3e4ad02328d7a61cb
+SHA512:
+  metadata.gz: 80770110f68b77ac080ebd48fe637f8bf7cb3e054e9c9345523535c27f3f8ab35f0c8e4d3ea5387aedbd6bdfc959795556644e2b5a18685c6749dd541ff2b3a7
+  data.tar.gz: '08b336516e7466179039bbc9a1a8874e96d70d3f9511566ae520e2417a373817ba716bbb20226e606ab532da1f6e00805bdb0da9a025f4d02f1a064bbb1bfdcd'

data/.ruby-version

@@ -0,0 +1 @@
+3.4

data/.yardopts

@@ -0,0 +1,9 @@
+--title "pgoutput-source-adapter API Documentation"
+--readme README.md
+--markup markdown
+--output-dir doc
+lib/**/*.rb
+-
+LICENSE.txt
+CHANGELOG.md
+CODE_OF_CONDUCT.md

data/CHANGELOG.md

@@ -0,0 +1,8 @@
+## [Unreleased]
+
+## [0.1.0] - 2026-06-08
+
+- Initial CDC source adapter implementation.
+- Added `Pgoutput::SourceAdapter::Cdc`.
+- Added normalization from `Pgoutput::Decoder::Events` to `CDC::Core::ChangeEvent`.
+- Added transaction batch normalization to `CDC::Core::TransactionEnvelope`.

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,14 @@
+# Code of Conduct
+
+"cdc-core" follows [The Ruby Community Conduct Guideline](https://www.ruby-lang.org/en/conduct) in all "collaborative space", which is defined as community communications channels (such as mailing lists, submitted patches, commit comments, etc.):
+
+- Participants will be tolerant of opposing views.
+- Participants must ensure that their language and actions are free of personal attacks and disparaging personal remarks.
+- When interpreting the words and actions of others, participants should always assume good intentions.
+- Behaviour which can be reasonably considered harassment will not be tolerated.
+
+If you have concerns about behaviour within this project, please open an issue at:
+
+https://github.com/kanutocd/cdc-core/issues/new
+
+Please note that GitHub issues are public. Do not include sensitive personal information in your report.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Keneth C. Demanawa
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,156 @@
+# pgoutput-source-adapter
+
+[![Gem Version](https://badge.fury.io/rb/pgoutput-source-adapter.svg)](https://badge.fury.io/rb/pgoutput-source-adapter)
+[![CI](https://github.com/kanutocd/pgoutput-source-adapter/workflows/CI/badge.svg)](https://github.com/kanutocd/pgoutput-source-adapter/actions)
+[![Ruby Version](https://img.shields.io/badge/ruby-%3E%3D%203.4-ruby.svg)](https://www.ruby-lang.org/en/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+`pgoutput-source-adapter` adapts decoded pgoutput events into downstream change-event platform primitives.
+
+The first supported target is the CDC Ecosystem:
+
+```ruby
+Pgoutput::SourceAdapter::Cdc
+```
+
+It normalizes `Pgoutput::Decoder::Events` into `CDC::Core::ChangeEvent` and `CDC::Core::TransactionEnvelope` objects.
+
+## Boundary
+
+The pgoutput family remains standalone:
+
+```text
+pgoutput-client   -> PostgreSQL logical replication transport
+pgoutput-parser   -> pgoutput protocol messages
+pgoutput-decoder  -> typed Ruby row-change events
+```
+
+This gem is the adapter layer:
+
+```text
+Pgoutput::Decoder::Events
+        |
+        v
+Pgoutput::SourceAdapter::Cdc
+        |
+        v
+CDC::Core::ChangeEvent / TransactionEnvelope
+```
+
+That keeps the lower-level pgoutput gems usable outside the CDC Ecosystem while still providing a clean bridge into `cdc-core` for users building CDC platforms.
+
+## Installation
+
+```ruby
+gem "pgoutput-source-adapter"
+```
+
+```ruby
+require "pgoutput/source_adapter"
+```
+
+The generated `bundle gem` require path also works:
+
+```ruby
+require "pgoutput/source/adapter"
+```
+
+## Usage
+
+Normalize a decoded insert event:
+
+```ruby
+adapter = Pgoutput::SourceAdapter::Cdc.new
+change_event = adapter.normalize(decoded_insert)
+
+change_event.operation
+# => :insert
+
+change_event.schema
+change_event.table
+change_event.new_values
+```
+
+Normalize a transaction-shaped batch:
+
+```ruby
+results = adapter.normalize_many([
+  decoded_begin,
+  decoded_insert,
+  decoded_update,
+  decoded_commit
+])
+
+envelope = results.first
+# => CDC::Core::TransactionEnvelope
+```
+
+## Primary keys
+
+For update and delete events, pgoutput may provide an old-key tuple. When it does, that tuple is used as the `CDC::Core::ChangeEvent#primary_key`.
+
+For insert events, or for sources without old-key tuples, the adapter defaults to `id` / `"id"` when present.
+
+You can provide your own resolver:
+
+```ruby
+adapter = Pgoutput::SourceAdapter::Cdc.new(
+  primary_key_resolver: ->(_event, values) { { "uuid" => values.fetch("uuid") } }
+)
+```
+
+## Metadata
+
+Each normalized event includes pgoutput metadata:
+
+```ruby
+{
+  "source" => "pgoutput",
+  "relation_id" => 123,
+  "pgoutput_event" => "Insert"
+}
+```
+
+Additional metadata can be injected:
+
+```ruby
+adapter = Pgoutput::SourceAdapter::Cdc.new(
+  metadata_builder: ->(_event) { { pipeline: "default" } }
+)
+```
+
+## Public namespace
+
+```ruby
+Pgoutput::SourceAdapter
+Pgoutput::SourceAdapter::Cdc
+```
+
+A compatibility alias is also provided for the generated gem path:
+
+```ruby
+Pgoutput::Source::Adapter
+```
+
+## Non-goals
+
+This gem does not:
+
+- connect to PostgreSQL
+- parse pgoutput protocol messages
+- decode PostgreSQL values
+- run processors
+- manage replication slots
+- persist sink data
+
+Those responsibilities belong to `pgoutput-client`, `pgoutput-parser`, `pgoutput-decoder`, runtime gems, or application code.
+
+## Development
+
+```bash
+bundle exec rake
+```
+
+## License
+
+[MIT](./LICENSE.txt).

data/Rakefile

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+require 'bundler/gem_tasks'
+require 'rake/testtask'
+require 'rubocop/rake_task'
+require 'yard'
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << 'lib'
+  t.libs << 'test'
+  t.test_files = FileList['test/**/*_test.rb']
+end
+
+RuboCop::RakeTask.new(:rubocop) do |task|
+  task.options = ['--parallel']
+end
+
+YARD::Rake::YardocTask.new(:yard)
+
+task :steep do
+  sh 'bundle exec steep check'
+end
+
+desc 'Open coverage report'
+task :coverage do
+  sh 'xdg-open coverage/index.html'
+end
+
+task default: %i[test rubocop steep yard]

data/lib/pgoutput/source/adapter.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+require_relative '../source_adapter'
+
+module Pgoutput
+  module Source
+    # Compatibility alias for {Pgoutput::SourceAdapter}.
+    #
+    # This alias preserves the generated bundle-gem path shape while the public
+    # adapter API lives under {Pgoutput::SourceAdapter}.
+    #
+    # @api public
+    Adapter = Pgoutput::SourceAdapter
+  end
+end

data/lib/pgoutput/source_adapter/cdc.rb

@@ -0,0 +1,232 @@
+# frozen_string_literal: true
+
+require 'cdc/core'
+
+unless CDC::Core.const_defined?(:SourceAdapter)
+  raise LoadError, 'pgoutput-source-adapter requires a cdc-core version that defines CDC::Core::SourceAdapter'
+end
+
+module Pgoutput
+  module SourceAdapter
+    # Normalizes Pgoutput::Decoder::Events into CDC::Core primitives.
+    #
+    # This adapter is intentionally located under the Pgoutput namespace because
+    # it adapts pgoutput decoded events. The target is CDC::Core, so this class
+    # depends on cdc-core while the lower-level pgoutput-client, pgoutput-parser,
+    # and pgoutput-decoder gems remain standalone.
+    #
+    # @example Normalize a decoded insert event
+    #   adapter = Pgoutput::SourceAdapter::Cdc.new
+    #   change_event = adapter.normalize(decoded_insert)
+    #
+    # @example Normalize a decoded transaction event batch
+    #   envelope = adapter.normalize_many([begin_event, insert_event, commit_event]).first
+    #
+    # @api public
+    class Cdc < CDC::Core::SourceAdapter
+      SOURCE_NAME = 'pgoutput'
+
+      # @param primary_key_resolver [#call, nil] optional callable used to infer
+      #   primary keys from decoded row values when pgoutput does not provide an
+      #   old-key tuple. The callable receives the decoded event and value hash.
+      # @param metadata_builder [#call, nil] optional callable that can return
+      #   extra metadata for each decoded event. Returned keys are stringified.
+      # @return [void]
+      def initialize(primary_key_resolver: nil, metadata_builder: nil)
+        @primary_key_resolver = primary_key_resolver || method(:default_primary_key)
+        @metadata_builder = metadata_builder
+        super()
+      end
+
+      # Normalize one decoded pgoutput event.
+      #
+      # Transaction boundary events return nil because they do not represent a
+      # row-level change by themselves. Use #normalize_many when transaction
+      # envelopes are desired.
+      #
+      # @param event [Object] a Pgoutput::Decoder::Events object.
+      # @return [CDC::Core::ChangeEvent, nil] normalized row change event, or
+      #   nil for transaction boundary events.
+      # @raise [Pgoutput::SourceAdapter::Error] when the decoded event type is
+      #   unsupported.
+      def normalize(event)
+        case event_name(event)
+        when 'Insert'
+          change_event(
+            event,
+            operation: :insert,
+            old_values: nil,
+            new_values: event.values,
+            primary_key: primary_key_for(event, event.values)
+          )
+        when 'Update'
+          old_values = event.old_values || event.old_key
+
+          change_event(
+            event,
+            operation: :update,
+            old_values: old_values,
+            new_values: event.new_values,
+            primary_key: primary_key_for(event, event.new_values)
+          )
+        when 'Delete'
+          old_values = event.old_values || event.old_key
+
+          change_event(
+            event,
+            operation: :delete,
+            old_values: old_values,
+            new_values: nil,
+            primary_key: primary_key_for(event, old_values)
+          )
+        when 'Begin', 'Commit'
+          nil
+        else
+          raise Error, "unsupported pgoutput decoded event: #{event.class}"
+        end
+      end
+
+      # Normalize a sequence of decoded pgoutput events.
+      #
+      # If the sequence contains transaction boundaries, row changes between a
+      # Begin and Commit are grouped into CDC::Core::TransactionEnvelope. If no
+      # transaction boundaries are present, row changes are returned individually.
+      #
+      # @param events [Enumerable<Object>] decoded pgoutput events.
+      # @return [Array<CDC::Core::ChangeEvent, CDC::Core::TransactionEnvelope>]
+      #   normalized row changes and transaction envelopes in input order.
+      # @raise [Pgoutput::SourceAdapter::Error] when any decoded event type is
+      #   unsupported.
+      def normalize_many(events)
+        results = [] # : [Array[CDC::Core::ChangeEvent, CDC::Core::TransactionEnvelope]]
+        transaction_id = nil
+        transaction_events = []
+        transaction_metadata = {}
+
+        events.each do |event|
+          case event_name(event)
+          when 'Begin'
+            transaction_id = event.transaction_id
+            transaction_events = []
+            transaction_metadata = metadata_for(event).merge(
+              'begin_final_lsn' => lsn_string(event.final_lsn),
+              'begin_commit_timestamp' => event.commit_timestamp
+            )
+          when 'Commit'
+            if transaction_id || !transaction_events.empty?
+              results << transaction_envelope(
+                event,
+                transaction_id: transaction_id || event.transaction_id,
+                events: transaction_events,
+                metadata: transaction_metadata
+              )
+            end
+
+            transaction_id = nil
+            transaction_events = []
+            transaction_metadata = {}
+          else
+            normalized = normalize(event)
+            next if normalized.nil?
+
+            if transaction_id
+              transaction_events << normalized
+            else
+              results << normalized
+            end
+          end
+        end
+
+        results.concat(transaction_events) if transaction_id && !transaction_events.empty?
+        share(results.freeze)
+      end
+
+      private
+
+      attr_reader :primary_key_resolver, :metadata_builder
+
+      def change_event(event, operation:, old_values:, new_values:, primary_key:)
+        share(
+          CDC::Core::ChangeEvent.new(
+            operation: operation,
+            schema: event.schema,
+            table: event.table,
+            old_values: old_values,
+            new_values: new_values,
+            primary_key: primary_key,
+            transaction_id: event.transaction_id,
+            metadata: metadata_for(event)
+          )
+        )
+      end
+
+      def transaction_envelope(event, transaction_id:, events:, metadata:)
+        share(
+          CDC::Core::TransactionEnvelope.new(
+            transaction_id: transaction_id,
+            events: events.freeze,
+            commit_lsn: lsn_string(event.commit_lsn),
+            committed_at: event.commit_timestamp,
+            metadata: metadata.merge(metadata_for(event)).merge(
+              'commit_flags' => event.flags,
+              'transaction_end_lsn' => lsn_string(event.transaction_end_lsn)
+            )
+          )
+        )
+      end
+
+      def primary_key_for(event, values)
+        return event.old_key if event.respond_to?(:old_key) && event.old_key
+
+        primary_key_resolver.call(event, values)
+      end
+
+      def default_primary_key(_event, values)
+        return nil unless values.respond_to?(:key?)
+
+        if values.key?('id')
+          { 'id' => values['id'] }
+        elsif values.key?(:id)
+          { 'id' => values[:id] }
+        end
+      end
+
+      def metadata_for(event)
+        metadata = {
+          'source' => SOURCE_NAME,
+          'relation_id' => relation_id_for(event),
+          'pgoutput_event' => event_name(event)
+        }
+        extra = metadata_builder&.call(event)
+        metadata.merge!(stringify_keys(extra)) if extra
+        compact(metadata)
+      end
+
+      def relation_id_for(event)
+        event.relation_id if event.respond_to?(:relation_id)
+      end
+
+      def event_name(event)
+        event.class.name.split('::').last
+      end
+
+      def lsn_string(lsn)
+        lsn&.to_s
+      end
+
+      def stringify_keys(hash)
+        hash.to_h.transform_keys(&:to_s)
+      end
+
+      def compact(hash)
+        hash.compact
+      end
+
+      def share(object)
+        Ractor.make_shareable(object)
+      rescue Ractor::Error
+        object
+      end
+    end
+  end
+end

data/lib/pgoutput/source_adapter/version.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+module Pgoutput
+  module SourceAdapter
+    # Current pgoutput-source-adapter gem version.
+    #
+    # @return [String] semantic version published to RubyGems.
+    VERSION = '0.0.0'
+  end
+end

data/lib/pgoutput/source_adapter.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+require_relative 'source_adapter/version'
+
+module Pgoutput
+  # Source adapters normalize decoded pgoutput events into downstream platform
+  # event models.
+  #
+  # The pgoutput family remains independently useful outside the CDC Ecosystem.
+  # This namespace leaves room for adapters targeting other change-event
+  # platforms while providing {Pgoutput::SourceAdapter::Cdc} as the CDC
+  # Ecosystem adapter.
+  #
+  # @api public
+  module SourceAdapter
+    # Raised when a decoded pgoutput event cannot be normalized.
+    #
+    # The adapter raises this error for event objects outside the supported
+    # pgoutput decoder event family. Callers should treat it as a programming or
+    # integration error rather than a transient transport failure.
+    #
+    # @api public
+    class Error < StandardError; end
+  end
+end
+
+require_relative 'source_adapter/cdc'

data/mise.toml

@@ -0,0 +1,2 @@
+[tools]
+ruby = "3.4"

data/sig/pgoutput/source/adapter.rbs

@@ -0,0 +1,6 @@
+module Pgoutput
+  module Source
+    # Compatibility alias for Pgoutput::SourceAdapter.
+    Adapter: untyped
+  end
+end

data/sig/pgoutput/source_adapter.rbs

@@ -0,0 +1,9 @@
+module Pgoutput
+  # Source adapters normalize pgoutput decoded events into downstream platform
+  # event models.
+  module SourceAdapter
+    # Raised when a decoded pgoutput event cannot be normalized.
+    class Error < StandardError
+    end
+  end
+end

metadata

@@ -0,0 +1,88 @@
+--- !ruby/object:Gem::Specification
+name: pgoutput-source-adapter
+version: !ruby/object:Gem::Version
+  version: 0.0.0
+platform: ruby
+authors:
+- Ken C. Demanawa
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: cdc-core
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+- !ruby/object:Gem::Dependency
+  name: pgoutput-decoder
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+description: Normalizes pgoutput decoded events into downstream change-event platform
+  primitives, including CDC::Core.
+email:
+- kenneth.c.demanawa@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".ruby-version"
+- ".yardopts"
+- CHANGELOG.md
+- CODE_OF_CONDUCT.md
+- LICENSE.txt
+- README.md
+- Rakefile
+- lib/pgoutput/source/adapter.rb
+- lib/pgoutput/source_adapter.rb
+- lib/pgoutput/source_adapter/cdc.rb
+- lib/pgoutput/source_adapter/version.rb
+- mise.toml
+- sig/pgoutput/source/adapter.rbs
+- sig/pgoutput/source_adapter.rbs
+homepage: https://github.com/kanutocd/pgoutput-source-adapter
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/kanutocd/pgoutput-source-adapter
+  source_code_uri: https://github.com/kanutocd/pgoutput-source-adapter
+  changelog_uri: https://github.com/kanutocd/pgoutput-source-adapter/blob/main/CHANGELOG.md
+  documentation_uri: https://kanutocd.github.io/pgoutput-source-adapter/
+  rubygems_mfa_required: 'true'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.4.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.6.9
+specification_version: 4
+summary: Source adapters for pgoutput decoded events.
+test_files: []