cdc-core 0.1.0 → 0.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5ace8b27c4df8285c5354599650cb450cddaad1402c913689e638fb9ec57d8b9
-  data.tar.gz: 381f77ee16278ea147b083b677d5e9248a2fb6bf1955c40f9d2eab82055a1e49
+  metadata.gz: a71182bac774b200c13556d0a887754a2788dc4e3581fc84a94288764900f365
+  data.tar.gz: a67bb53791b19347fb0c589b59603abfc6f085504597531d7c9932da3a3d8749
 SHA512:
-  metadata.gz: c0447c01564b04f20170e8a7f9e60cb95c12f77ef87ce54ab8a090083b7054a041224247349a9424e803f22255bbe097b33706a16c267ed1710e67fab7e6fff0
-  data.tar.gz: 9aef7f1a3ffd777c6031e700c49f17a3aa460fa46e898d648d55b81c1a2afaf6df4e19fd700bbadad57be382d0809990909f9a97ac721ea7121fc3e327c868b7
+  metadata.gz: a96b523f2144fd94d0ecd633bf956fdcad791a994f3a2bf9e4559da9e4c713570f52c842e3cce15cb9a1491345190087c7cd1599612f5e4ecd5f933d22a6a9d6
+  data.tar.gz: 44acef005734403fea47a88e8d142cf8f9667ef032b032af24dfc25cf25c491d1c2a6caf3fab3bb06fc7790d63b8c88a8cb9984e61803dac315cd6d8118a2957

data/CHANGELOG.md

@@ -1,15 +1,23 @@
-# Changelog
+## [Unreleased]
 
-All notable changes to this project will be documented in this file.
+## [0.1.1] - 2026-06-09
 
-The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
-and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+### Added
 
-## [Unreleased]
+- Added `CDC::Core::OrderingScope`, `OrderingPolicy`, `OrderingKey`, and `EventPosition`.
+- Added `CDC::Core::SourceAdapter` as the shared source normalization contract.
+- Added `CDC::Core::Router` and `UnsupportedWorkItemError`.
+- Added optional lifecycle hooks to `CDC::Core::Processor`.
+- Added structured failure metadata to `CDC::Core::ProcessorResult`.
+- Added status validation and serializable projection helpers to `CDC::Core::ProcessorResult`.
+- Added backend-agnostic observer hooks for instrumentation.
+- Added canonical metric names and tag helpers to `CDC::Core::Observer`.
 
-### Added
+### Fixed
 
-- Placeholder for future development.
+- Preserve explicit `false` metadata values when reading `EventMetadata` by symbol or string key.
+- Raise `CDC::Core::InvalidOperationError` for nil operation input instead of leaking `NoMethodError`.
+- Align API documentation examples with the current `#process`, `Pipeline.new(processor:)`, and `Filter#match?` contracts.
 
 ---
 

data/README.md

@@ -1,8 +1,8 @@
 # cdc-core
 
-Database-agnostic Change Data Capture domain primitives for Ruby.
+Shared Change Data Capture vocabulary for Ruby.
 
-`cdc-core` provides immutable, Ractor-safe event objects and processor contracts for building CDC systems. It intentionally does not connect to databases, parse wire protocols, decode PostgreSQL OIDs, or integrate with Rails.
+`cdc-core` provides immutable, Ractor-safe event objects and processor contracts for building CDC systems. It intentionally does not connect to databases, parse wire protocols, decode PostgreSQL OIDs, run schedulers, or integrate with Rails.
 
 ## Requirements
 
@@ -10,12 +10,16 @@ Database-agnostic Change Data Capture domain primitives for Ruby.
 
 ## Features
 
+- SourceAdapter normalization contract
 - Immutable `ChangeEvent` objects
 - Transaction grouping via `TransactionEnvelope`
 - Column-level change objects
+- Ordering vocabulary
 - Processor and composite processor contracts
 - Event filters
 - Small pipeline orchestration object
+- Router for supported work item shapes
+- Observer hooks and canonical metric names
 - Ractor-safe event and transaction objects
 - RBS signatures
 - YARD-compatible documentation
@@ -24,22 +28,88 @@ Database-agnostic Change Data Capture domain primitives for Ruby.
 ## Ecosystem Position
 
 ```text
-pgoutput-client
-      │
-      ▼
-pgoutput-parser
-      │
-      ▼
-pgoutput-decoder
-      │
-      ▼
+upstream source
+      |
+      v
+source adapter
+      |
+      v
 cdc-core
-      │
-      ▼
-whodunit-chronicles
+      |
+      +--> cdc-parallel       CPU-bound processing
+      |
+      +--> cdc-concurrent     I/O-bound processing
+      |
+      +--> application sinks / processors
 ```
 
-`cdc-core` is the shared vocabulary layer. It defines what a change event, transaction, processor, and pipeline result means without caring where the event came from.
+`cdc-core` is the shared vocabulary layer. It defines what a change event, transaction, processor, ordering policy, observer notification, and processor result mean without caring where the event came from or how it will be executed.
+
+## Boundary Summary
+
+`cdc-core` is for vocabulary.
+
+Runtime gems are for execution.
+
+Sinks are for persistence or side effects.
+
+```text
+source adapter -> cdc-core vocabulary -> runtime gem -> sink
+```
+
+## Source Adapters
+
+CDC::Core::SourceAdapter defines the normalization contract used to translate source-specific payloads into cdc-core vocabulary objects.
+
+It translates source-specific payloads into:
+
+- `CDC::Core::ChangeEvent`
+- `CDC::Core::TransactionEnvelope`
+- batches of core work items
+
+The current PostgreSQL-oriented path is:
+
+```text
+pgoutput-client -> pgoutput-parser -> pgoutput-decoder -> source adapter -> cdc-core
+```
+
+The `pgoutput*` family handles PostgreSQL transport, protocol parsing, and type decoding. The source-adapter boundary is where those source-specific details become generic `cdc-core` objects.
+
+Other adapters can normalize logs, API payloads, application events, or other database streams into the same vocabulary.
+
+## Downstream Runtime Gems
+
+`cdc-parallel` and `cdc-concurrent` are downstream consumers of `cdc-core` events.
+
+### cdc-parallel
+
+Use `cdc-parallel` for heavy CPU-bound processing.
+
+Examples:
+
+- transformations
+- enrichment
+- encoding
+- compression
+- scoring
+- in-memory calculations
+
+It is the Ractor-oriented runtime path.
+
+### cdc-concurrent
+
+Use `cdc-concurrent` for I/O-heavy processing.
+
+Examples:
+
+- HTTP calls
+- webhook delivery
+- Redis writes
+- search indexing
+- object storage writes
+- database sink writes
+
+It is the fiber-friendly runtime path.
 
 ## Installation
 
@@ -86,7 +156,7 @@ transaction = CDC::Core::TransactionEnvelope.new(
 )
 ```
 
-A transaction envelope is the natural unit for future parallel processing because it preserves database transaction boundaries.
+A transaction envelope preserves database transaction boundaries. Runtime gems may use that boundary when they need ordering, batching, or parallel execution decisions.
 
 ## Processors
 
@@ -114,7 +184,7 @@ AnalyticsProcessor.new.ractor_safe?
 # => true
 ```
 
-This declares intent only. `cdc-core` does not execute processors in Ractors. A future runtime gem can use this signal.
+This declares intent only. `cdc-core` does not execute processors in Ractors. `cdc-parallel` can use this signal before moving processor work across Ractors.
 
 ## Composite Processor
 
@@ -149,10 +219,26 @@ result = pipeline.process(event)
 - Parse `pgoutput`
 - Decode PostgreSQL values
 - Manage replication slots
+- Implement concrete source adapters
 - Run Ractor pools
+- Run fiber schedulers
 - Persist audit records
 - Integrate with ActiveRecord
-- Publish to Kafka, Redis, or HTTP sinks
+- Publish to Kafka, Redis, HTTP, or other sinks
+
+## Documentation
+
+The YARD documentation uses `docs/index.md` as its readme and includes the Markdown files under `docs/`.
+
+```text
+--title "cdc-core API Documentation"
+--readme docs/index.md
+--markup markdown
+--output-dir doc
+lib/**/*.rb
+-
+docs/**/*.md
+```
 
 ## Development
 
@@ -163,4 +249,4 @@ bundle exec steep check
 
 ## License
 
-MIT.
+[MIT](./LICENSE.txt).

data/lib/cdc/core/composite_processor.rb

@@ -10,15 +10,17 @@ module CDC
     class CompositeProcessor < Processor
       # @return [Array<Processor>] processors executed for each event
       # @return [Boolean] whether processing stops on the first failure
-      attr_reader :processors, :fail_fast
+      # @return [Observer] observer notified of dispatch events
+      attr_reader :processors, :fail_fast, :observer
 
       # Build a composite processor.
       #
       # @param processors [Array<#process>] processors to execute
       # @param fail_fast [Boolean] whether to stop after the first failure
-      def initialize(processors, fail_fast: true) # rubocop:disable Lint/MissingSuper
+      def initialize(processors, fail_fast: true, observer: NullObserver::INSTANCE) # rubocop:disable Lint/MissingSuper
         @processors = processors.freeze
         @fail_fast = fail_fast
+        @observer = observer || NullObserver::INSTANCE
       end
 
       # Process an event through each configured processor.
@@ -26,18 +28,12 @@ module CDC
       # @param event [ChangeEvent] event to process
       # @return [Array<ProcessorResult>] result from each attempted processor
       def process(event)
-        results = [] # : Array[ProcessorResult]
-        processors.each do |processor|
-          result = normalize_result(processor.process(event), event)
-          results << result
-          break if fail_fast && result.failure?
-        rescue StandardError => e
-          result = ProcessorResult.failure(e, event:)
-          results << result
-          break if fail_fast
-        end
-        Ractor.make_shareable(results.freeze) if results.none?(&:failure?)
-        results.freeze
+        observer.dispatch_started(event)
+        results = collect_results(event)
+        final_results = results.freeze
+        observe_results(final_results)
+        Ractor.make_shareable(final_results) if results.none?(&:failure?)
+        final_results
       end
 
       # Processors that declared Ractor safety.
@@ -66,6 +62,35 @@ module CDC
 
         result ? ProcessorResult.success(event) : ProcessorResult.skipped(event)
       end
+
+      def collect_results(event)
+        results = [] # : Array[ProcessorResult]
+        processors.each do |processor|
+          result = process_with(processor, event)
+          results << result
+          break if fail_fast && result.failure?
+        end
+        results
+      end
+
+      def process_with(processor, event)
+        normalize_result(processor.process(event), event)
+      rescue StandardError => e
+        ProcessorResult.failure(e, event:, processor: processor.class.name)
+      end
+
+      def observe_results(results)
+        results.each do |result|
+          case result.status
+          when :success
+            observer.dispatch_succeeded(result)
+          when :failure
+            observer.dispatch_failed(result)
+          when :skipped
+            observer.dispatch_skipped(result)
+          end
+        end
+      end
     end
   end
 end

data/lib/cdc/core/errors.rb

@@ -8,6 +8,15 @@ module CDC
     # Raised when an operation cannot be normalized to a supported CDC action.
     class InvalidOperationError < Error; end
 
+    # Raised when an ordering scope cannot be normalized to a supported value.
+    class InvalidOrderingScopeError < Error; end
+
+    # Raised when an ordering position cannot be normalized to a supported value.
+    class InvalidOrderingPositionError < Error; end
+
+    # Raised when a router receives an unsupported CDC work item.
+    class UnsupportedWorkItemError < Error; end
+
     # Raised by processors when a processor-specific failure needs wrapping.
     class ProcessorError < Error; end
   end

data/lib/cdc/core/event_metadata.rb

@@ -24,7 +24,10 @@ module CDC
       # @param key [String, Symbol] metadata key
       # @return [Object, nil]
       def [](key)
-        data[key] || data[key.to_s] || data[key.to_sym]
+        string_key = key.to_s
+        return data[string_key] if data.key?(string_key)
+
+        data[key]
       end
 
       # Return the normalized Ractor-shareable hash.

data/lib/cdc/core/event_position.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+module CDC
+  module Core
+    # Immutable representation of an event's position metadata.
+    #
+    # EventPosition is intentionally small and transport-agnostic. It captures
+    # the position strategy plus the event fields that a runtime may use to
+    # preserve ordering guarantees.
+    class EventPosition
+      # @return [Symbol] position strategy
+      # @return [Object, nil] primary position value for the chosen strategy
+      # @return [Object, nil] transaction identifier associated with the event
+      # @return [Integer, nil] sequence number within a transaction or stream
+      # @return [Time, nil] timestamp associated with the event
+      attr_reader :strategy, :value, :transaction_id, :sequence_number, :occurred_at
+
+      # Build an event position.
+      #
+      # @param strategy [#to_sym] position strategy
+      # @param value [Object, nil] primary position value
+      # @param transaction_id [Object, nil] transaction identifier
+      # @param sequence_number [Integer, nil] sequence number
+      # @param occurred_at [Time, nil] event timestamp
+      def initialize(strategy:, value:, transaction_id: nil, sequence_number: nil, occurred_at: nil)
+        @strategy = strategy.to_sym
+        @value = value
+        @transaction_id = transaction_id
+        @sequence_number = sequence_number
+        @occurred_at = occurred_at
+        Ractor.make_shareable(self)
+      end
+
+      # Convert the position into a Ractor-shareable hash.
+      #
+      # @return [Hash{String=>Object,nil}]
+      def to_h
+        Ractor.make_shareable({
+          'strategy' => strategy,
+          'value' => value,
+          'transaction_id' => transaction_id,
+          'sequence_number' => sequence_number,
+          'occurred_at' => occurred_at
+        }.freeze)
+      end
+    end
+  end
+end

data/lib/cdc/core/null_observer.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+module CDC
+  module Core
+    # No-op observer for callers that do not need instrumentation.
+    class NullObserver < Observer
+      INSTANCE = new
+
+      private_class_method :new
+    end
+  end
+end

data/lib/cdc/core/observer.rb

@@ -0,0 +1,123 @@
+# frozen_string_literal: true
+
+module CDC
+  module Core
+    # Observer interface for CDC runtime instrumentation.
+    #
+    # Observers receive lifecycle and result notifications from core runtime
+    # objects. The default implementation is a no-op so callers can opt in
+    # without taking a dependency on a metrics backend.
+    class Observer
+      # Canonical metric names emitted by core runtime hooks.
+      METRIC_NAMES = Ractor.make_shareable({
+        dispatch_started: 'cdc_core.dispatch.started',
+        dispatch_succeeded: 'cdc_core.dispatch.succeeded',
+        dispatch_failed: 'cdc_core.dispatch.failed',
+        dispatch_skipped: 'cdc_core.dispatch.skipped'
+      }.freeze)
+
+      # Build a canonical metric tag set for a CDC work item or result.
+      #
+      # @param payload [ChangeEvent, TransactionEnvelope, ProcessorResult, Array]
+      # @return [Hash{String=>Object}]
+      def self.metric_tags(payload)
+        tags = {} # : Hash[String, untyped]
+        case payload
+        when ChangeEvent
+          tags.merge!(change_event_metric_tags(payload))
+        when TransactionEnvelope
+          tags.merge!(transaction_envelope_metric_tags(payload))
+        when ProcessorResult
+          tags.merge!(processor_result_metric_tags(payload))
+        when Array
+          tags.merge!(batch_metric_tags(payload))
+        else
+          tags['kind'] = payload.class.name
+        end
+
+        Ractor.make_shareable(tags.freeze)
+      end
+
+      # Canonical metric name for the start hook.
+      #
+      # @return [String]
+      def self.started_metric_name = METRIC_NAMES.fetch(:dispatch_started)
+
+      # Canonical metric name for the success hook.
+      #
+      # @return [String]
+      def self.succeeded_metric_name = METRIC_NAMES.fetch(:dispatch_succeeded)
+
+      # Canonical metric name for the failure hook.
+      #
+      # @return [String]
+      def self.failed_metric_name = METRIC_NAMES.fetch(:dispatch_failed)
+
+      # Canonical metric name for the skip hook.
+      #
+      # @return [String]
+      def self.skipped_metric_name = METRIC_NAMES.fetch(:dispatch_skipped)
+
+      # Called before a work item is dispatched.
+      #
+      # @param _event [ChangeEvent, TransactionEnvelope, Array]
+      # @return [void]
+      def dispatch_started(_event); end
+
+      # Called after a work item is processed successfully.
+      #
+      # @param _result [ProcessorResult, Array<ProcessorResult>]
+      # @return [void]
+      def dispatch_succeeded(_result); end
+
+      # Called after a work item fails.
+      #
+      # @param _result [ProcessorResult]
+      # @return [void]
+      def dispatch_failed(_result); end
+
+      # Called when a work item is filtered or skipped.
+      #
+      # @param _result [ProcessorResult]
+      # @return [void]
+      def dispatch_skipped(_result); end
+
+      private_class_method def self.change_event_metric_tags(event)
+        {
+          'kind' => 'change_event',
+          'operation' => event.operation,
+          'schema' => event.schema,
+          'table' => event.table
+        }.tap do |tags|
+          tags['transaction_id'] = event.transaction_id if event.transaction_id
+        end
+      end
+
+      private_class_method def self.transaction_envelope_metric_tags(transaction)
+        {
+          'kind' => 'transaction_envelope',
+          'transaction_id' => transaction.transaction_id,
+          'size' => transaction.size
+        }
+      end
+
+      private_class_method def self.processor_result_metric_tags(result)
+        {
+          'kind' => 'processor_result',
+          'status' => result.status,
+          'retryable' => result.retryable?
+        }.tap do |tags|
+          tags['processor'] = result.processor_name if result.processor_name
+          tags['failure_reason'] = result.failure_reason if result.failure?
+        end
+      end
+
+      private_class_method def self.batch_metric_tags(batch)
+        {
+          'kind' => 'batch',
+          'size' => batch.size
+        }
+      end
+    end
+  end
+end

data/lib/cdc/core/operation.rb

@@ -34,6 +34,8 @@ module CDC
         value = operation.to_sym
         return value if SUPPORTED.include?(value)
 
+        raise InvalidOperationError, "unsupported CDC operation: #{operation.inspect}"
+      rescue NoMethodError
         raise InvalidOperationError, "unsupported CDC operation: #{operation.inspect}"
       end
 

data/lib/cdc/core/ordering_key.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module CDC
+  module Core
+    # Immutable grouping key for ordering-related dispatch.
+    #
+    # OrderingKey captures the scope plus the components that define a
+    # particular ordered lane. It does not choose an execution strategy.
+    class OrderingKey
+      # @return [Symbol] ordering scope
+      # @return [Hash{String=>Object}] normalized key components
+      attr_reader :scope, :components
+
+      # Build an ordering key.
+      #
+      # @param scope [#to_sym] ordering scope
+      # @param components [Hash] key components
+      def initialize(scope:, components: {})
+        @scope = OrderingScope.normalize(scope)
+        @components = EventMetadata.new(components).to_h
+        Ractor.make_shareable(self)
+      end
+
+      # Whether the key has no components.
+      #
+      # @return [Boolean]
+      def empty? = components.empty?
+
+      # Convert the key into a Ractor-shareable hash.
+      #
+      # @return [Hash{String=>Object}]
+      def to_h
+        Ractor.make_shareable({
+          'scope' => scope,
+          'components' => components
+        }.freeze)
+      end
+    end
+  end
+end

data/lib/cdc/core/ordering_policy.rb

@@ -0,0 +1,106 @@
+# frozen_string_literal: true
+
+module CDC
+  module Core
+    # Immutable description of how ordered CDC work should be grouped.
+    #
+    # OrderingPolicy defines the vocabulary for ordering guarantees without
+    # performing any scheduling itself. Runtime layers can consume the policy to
+    # route events into ordered lanes.
+    class OrderingPolicy
+      # Position strategies supported by the core contract.
+      SUPPORTED_POSITIONS = Ractor.make_shareable(
+        %i[commit_lsn transaction_id sequence_number occurred_at].freeze
+      )
+
+      # @return [Symbol] ordering scope
+      # @return [Symbol] position strategy
+      # @return [Boolean] whether transaction boundaries should be preserved
+      attr_reader :scope, :position, :transaction_aware
+
+      # Build an ordering policy.
+      #
+      # @param scope [#to_sym] ordering scope
+      # @param position [#to_sym] position strategy
+      # @param transaction_aware [Boolean] whether transaction boundaries matter
+      def initialize(scope:, position: :commit_lsn, transaction_aware: true)
+        @scope = OrderingScope.normalize(scope)
+        @position = normalize_position(position)
+        @transaction_aware = transaction_aware == true
+        Ractor.make_shareable(self)
+      end
+
+      # Whether transaction boundaries should be preserved.
+      #
+      # @return [Boolean]
+      def transaction_aware? = transaction_aware
+
+      # Derive an ordering key for an event.
+      #
+      # @param event [ChangeEvent] event to classify
+      # @return [OrderingKey, nil]
+      def key_for(event)
+        return nil if scope == OrderingScope::NONE
+
+        OrderingKey.new(scope: scope, components: key_components(event))
+      end
+
+      # Derive an event position for an event.
+      #
+      # @param event [ChangeEvent] event to classify
+      # @return [EventPosition]
+      def position_for(event)
+        EventPosition.new(
+          strategy: position,
+          value: event.public_send(position),
+          transaction_id: event.transaction_id,
+          sequence_number: event.sequence_number,
+          occurred_at: event.occurred_at
+        )
+      end
+
+      # Convert the policy into a Ractor-shareable hash.
+      #
+      # @return [Hash{String=>Object}]
+      def to_h
+        Ractor.make_shareable({
+          'scope' => scope,
+          'position' => position,
+          'transaction_aware' => transaction_aware
+        }.freeze)
+      end
+
+      private
+
+      # Normalize the position strategy.
+      #
+      # @param position [#to_sym] position strategy
+      # @return [Symbol]
+      def normalize_position(position)
+        value = position.to_sym
+        return value if SUPPORTED_POSITIONS.include?(value)
+
+        raise InvalidOrderingPositionError, "unsupported CDC ordering position: #{position.inspect}"
+      rescue NoMethodError
+        raise InvalidOrderingPositionError, "unsupported CDC ordering position: #{position.inspect}"
+      end
+
+      # Build the components for the current scope.
+      #
+      # @param event [ChangeEvent]
+      # @return [Hash]
+      def key_components(event)
+        case scope
+        when OrderingScope::GLOBAL
+          {}
+        when OrderingScope::TRANSACTION
+          { transaction_id: event.transaction_id }
+        when OrderingScope::RELATION
+          { schema: event.schema, table: event.table }
+        when OrderingScope::PRIMARY_KEY
+          { schema: event.schema, table: event.table, primary_key: event.primary_key }
+        end
+      end
+    end
+  end
+end