cdc-solid-queue 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 3f4a62ca92fb8b26d59aa1afa8fb4790aff8ae1703dca92aeee332cf9fb63e25
4
+ data.tar.gz: 5b15208473f3163aae05a65f17f3e0293344c38a85d4ad47415245ae3053b953
5
+ SHA512:
6
+ metadata.gz: 8b1e6647bae6782d8952626f018e0579907db31bf31f99fe01bf9ec95ca48e649b61985e1af3ac0a5a1b8572e16533b9775435e237a7305792adb31b61260f09
7
+ data.tar.gz: c3759751bb907ee349ec68b92eb3ca9b8f9e7deaac5e26a762a3cbd94b3958180fd5e8ce0a01e3d3ddf17e98be66ae106144df24fcf6eebd552e52f56c88c25e
data/CHANGELOG.md ADDED
@@ -0,0 +1,11 @@
1
+ # Changelog
2
+
3
+ ## 0.1.0
4
+
5
+ - Initial implementation skeleton.
6
+ - Configuration object.
7
+ - Event serializer.
8
+ - Enqueuer.
9
+ - ProcessorJob mixin.
10
+ - Test coverage gate.
11
+ - RBS signatures.
data/LICENSE.txt ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Kenneth C. Demanawa
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,87 @@
1
+ # cdc-solid-queue
2
+
3
+ Rails-native durable CDC job backend for Solid Queue.
4
+
5
+ ```text
6
+ PostgreSQL WAL
7
+ -> pgoutput-client
8
+ -> pgoutput-parser / pgoutput-decoder
9
+ -> pgoutput-source-adapter
10
+ -> CDC::Core::ChangeEvent
11
+ -> cdc-solid-queue
12
+ -> Solid Queue
13
+ -> ApplicationJob
14
+ ```
15
+
16
+ ## Requirements
17
+
18
+ - Ruby 3.4+
19
+ - Rails 7.1+
20
+ - Solid Queue 1.0+
21
+ - PostgreSQL logical replication
22
+
23
+ Only PostgreSQL is supported in the initial implementation.
24
+
25
+ ## Usage
26
+
27
+ ```ruby
28
+ class UserChangedJob < ApplicationJob
29
+ include CDC::SolidQueue::ProcessorJob
30
+
31
+ def process(event)
32
+ # event is a CDC::Core::ChangeEvent
33
+ end
34
+ end
35
+ ```
36
+
37
+ ```ruby
38
+ CDC::SolidQueue.configure do |config|
39
+ config.processor_job = UserChangedJob
40
+ config.queue = "cdc"
41
+ config.preserve_order = true
42
+ config.ordering_key = :identity
43
+ config.checkpoint = CDC::SolidQueue::Checkpoint.new
44
+ config.postgresql = {
45
+ database_url: ENV.fetch("DATABASE_URL"),
46
+ slot: "cdc_solid_queue",
47
+ publication: "cdc_publication"
48
+ }
49
+ end
50
+ ```
51
+
52
+ `config.queue` is applied through Active Job's `set(queue:)` API when the job
53
+ class supports it. When `preserve_order` is enabled, the enqueued payload also
54
+ includes cdc-solid-queue metadata with the configured ordering key and computed
55
+ ordering value.
56
+
57
+ ## Rails Task
58
+
59
+ Rails applications can load the Railtie integration:
60
+
61
+ ```ruby
62
+ require "cdc/solid_queue/railtie"
63
+ ```
64
+
65
+ Then start ingestion with:
66
+
67
+ ```bash
68
+ bin/rails cdc_solid_queue:start
69
+ ```
70
+
71
+ The task wires `Pgoutput::Client::Runner`, `Pgoutput::RelationTracker`,
72
+ `Pgoutput::Decoder`, and `Pgoutput::SourceAdapter::Cdc` into the
73
+ `CDC::SolidQueue::Runner`.
74
+
75
+ ## MVP Checkpoint Rule
76
+
77
+ A checkpoint advances after the Solid Queue job is durably inserted. Job execution success is handled by Solid Queue retry semantics.
78
+
79
+ ## Quality Gates
80
+
81
+ The first implementation is designed around:
82
+
83
+ - 100% line coverage
84
+ - 100% branch coverage
85
+ - RBS validation
86
+ - RuboCop configuration
87
+ - YARD documentation
@@ -0,0 +1,43 @@
1
+ # frozen_string_literal: true
2
+
3
+ module CDC
4
+ module SolidQueue
5
+ # Minimal in-memory checkpoint store.
6
+ #
7
+ # Applications that need durable replay safety should provide a persistent
8
+ # object that responds to #advance(event, result).
9
+ class Checkpoint
10
+ # @return [Object, nil]
11
+ attr_reader :position
12
+
13
+ # Build an empty checkpoint.
14
+ def initialize
15
+ @position = nil
16
+ end
17
+
18
+ # Advance to the best known source position for an enqueued event.
19
+ #
20
+ # @param event [Object]
21
+ # @param result [Object]
22
+ # @return [Object, nil]
23
+ def advance(event, result = nil)
24
+ @position = position_for(event) || result_position(result) || @position
25
+ end
26
+
27
+ private
28
+
29
+ def position_for(event)
30
+ payload = EventSerializer.dump(event)
31
+ payload['source_position'] || payload['commit_lsn'] || payload.dig('metadata', 'wal_end_lsn')
32
+ rescue SerializationError
33
+ nil
34
+ end
35
+
36
+ def result_position(result)
37
+ return result.checkpoint_position if result.respond_to?(:checkpoint_position)
38
+
39
+ nil
40
+ end
41
+ end
42
+ end
43
+ end
@@ -0,0 +1,86 @@
1
+ # frozen_string_literal: true
2
+
3
+ module CDC
4
+ module SolidQueue
5
+ # Runtime configuration for PostgreSQL CDC ingestion into Solid Queue.
6
+ #
7
+ # The configuration object is intentionally small in the first release. It
8
+ # describes the target job class, Solid Queue queue name, ordering behavior,
9
+ # and PostgreSQL replication settings.
10
+ class Configuration
11
+ # The only CDC source supported by the initial implementation.
12
+ SUPPORTED_SOURCE = :postgresql
13
+ # Supported ordering scopes for serialized CDC events.
14
+ ORDERING_KEYS = %i[identity primary_key relation transaction global none].freeze
15
+
16
+ attr_accessor :processor_job, :queue, :preserve_order, :ordering_key, :postgresql, :checkpoint
17
+
18
+ # Build a configuration with safe defaults.
19
+ def initialize
20
+ @processor_job = nil
21
+ @queue = 'cdc'
22
+ @preserve_order = true
23
+ @ordering_key = :identity
24
+ @postgresql = {}
25
+ @checkpoint = Checkpoint.new
26
+ end
27
+
28
+ # Validate this configuration.
29
+ #
30
+ # @return [true]
31
+ # @raise [ConfigurationError] if required values are missing
32
+ # @raise [UnsupportedSourceError] if a non-PostgreSQL source is supplied
33
+ # rubocop:disable Naming/PredicateMethod
34
+ def validate!
35
+ validate_processor_job!
36
+ validate_queue!
37
+ validate_ordering_key!
38
+ validate_postgresql!
39
+ validate_checkpoint!
40
+ true
41
+ end
42
+ # rubocop:enable Naming/PredicateMethod
43
+
44
+ # Return a normalized source name.
45
+ #
46
+ # @return [Symbol]
47
+ def source
48
+ configured = @postgresql.fetch(:source, SUPPORTED_SOURCE)
49
+ configured.to_sym
50
+ end
51
+
52
+ private
53
+
54
+ def validate_processor_job!
55
+ return if @processor_job.respond_to?(:perform_later) || @processor_job.respond_to?(:perform_now)
56
+
57
+ raise ConfigurationError, 'processor_job must respond to perform_later or perform_now'
58
+ end
59
+
60
+ def validate_queue!
61
+ return if @queue.is_a?(String) && !@queue.empty?
62
+
63
+ raise ConfigurationError, 'queue must be a non-empty String'
64
+ end
65
+
66
+ def validate_ordering_key!
67
+ return if ORDERING_KEYS.include?(@ordering_key)
68
+
69
+ raise ConfigurationError, "ordering_key must be one of: #{ORDERING_KEYS.join(', ')}"
70
+ end
71
+
72
+ def validate_postgresql!
73
+ raise UnsupportedSourceError, 'cdc-solid-queue supports only PostgreSQL' unless source == SUPPORTED_SOURCE
74
+ return unless @postgresql.empty?
75
+
76
+ raise ConfigurationError, 'postgresql settings are required'
77
+ end
78
+
79
+ def validate_checkpoint!
80
+ return if @checkpoint.nil? || @checkpoint.respond_to?(:advance)
81
+
82
+ raise ConfigurationError, 'checkpoint must respond to advance'
83
+ end
84
+ end
85
+ end
86
+ end
@@ -0,0 +1,53 @@
1
+ # frozen_string_literal: true
2
+
3
+ module CDC
4
+ module SolidQueue
5
+ # Enqueues normalized CDC events as Solid Queue-backed Active Job jobs.
6
+ class Enqueuer
7
+ # @return [Configuration]
8
+ attr_reader :configuration
9
+
10
+ # @param configuration [Configuration]
11
+ def initialize(configuration)
12
+ @configuration = configuration
13
+ @configuration.validate!
14
+ end
15
+
16
+ # Enqueue one CDC event.
17
+ #
18
+ # @param event [Object, Hash]
19
+ # @return [Object] Active Job return value
20
+ def enqueue(event)
21
+ payload = EventSerializer.dump(event)
22
+ payload = EventSerializer.with_enqueue_metadata(payload, enqueue_metadata(payload))
23
+ job = configuration.processor_job
24
+ return async_job(job).perform_later(payload) if job.respond_to?(:perform_later)
25
+
26
+ job.perform_now(payload)
27
+ end
28
+
29
+ private
30
+
31
+ def async_job(job)
32
+ return job.set(queue: configuration.queue) if job.respond_to?(:set)
33
+
34
+ job
35
+ end
36
+
37
+ def enqueue_metadata(payload)
38
+ {
39
+ 'queue' => configuration.queue,
40
+ 'preserve_order' => configuration.preserve_order,
41
+ 'ordering_key' => configuration.ordering_key,
42
+ 'ordering_value' => ordering_value(payload)
43
+ }
44
+ end
45
+
46
+ def ordering_value(payload)
47
+ return nil unless configuration.preserve_order
48
+
49
+ EventSerializer.ordering_value(payload, configuration.ordering_key)
50
+ end
51
+ end
52
+ end
53
+ end
@@ -0,0 +1,17 @@
1
+ # frozen_string_literal: true
2
+
3
+ module CDC
4
+ module SolidQueue
5
+ # Base error for all cdc-solid-queue failures.
6
+ class Error < StandardError; end
7
+
8
+ # Raised when configuration is missing or invalid.
9
+ class ConfigurationError < Error; end
10
+
11
+ # Raised when an unsupported source is configured.
12
+ class UnsupportedSourceError < ConfigurationError; end
13
+
14
+ # Raised when an event cannot be serialized for job execution.
15
+ class SerializationError < Error; end
16
+ end
17
+ end
@@ -0,0 +1,143 @@
1
+ # frozen_string_literal: true
2
+
3
+ module CDC
4
+ module SolidQueue
5
+ # Converts CDC events into Solid Queue-safe payloads.
6
+ #
7
+ # Payloads are plain hashes so Active Job can serialize them without needing
8
+ # to load the original event object in the queue database.
9
+ class EventSerializer
10
+ # Reserved payload key for cdc-solid-queue enqueue metadata.
11
+ INTERNAL_METADATA_KEY = '_cdc_solid_queue'
12
+
13
+ # Serialize an event-like object.
14
+ #
15
+ # @param event [Object] event object or Hash
16
+ # @return [Hash] serializable event payload
17
+ # @raise [SerializationError] when the event cannot be represented
18
+ def self.dump(event)
19
+ payload = if event.is_a?(Hash)
20
+ event
21
+ elsif event.respond_to?(:to_h)
22
+ event.to_h
23
+ else
24
+ raise SerializationError, 'event must respond to to_h or be a Hash'
25
+ end
26
+
27
+ normalize_hash(payload)
28
+ end
29
+
30
+ # Load a serialized event payload.
31
+ #
32
+ # @param payload [Hash]
33
+ # @return [Hash]
34
+ # @raise [SerializationError] when payload is invalid
35
+ def self.load(payload)
36
+ raise SerializationError, 'payload must be a Hash' unless payload.is_a?(Hash)
37
+
38
+ strip_internal_metadata(normalize_hash(payload))
39
+ end
40
+
41
+ # Load a serialized event payload into a CDC event when possible.
42
+ #
43
+ # @param payload [Hash]
44
+ # @return [CDC::Core::ChangeEvent, Hash]
45
+ def self.load_event(payload)
46
+ normalized = load(payload)
47
+ return normalized unless change_event_payload?(normalized)
48
+
49
+ build_change_event(normalized)
50
+ end
51
+
52
+ # Attach enqueue metadata without changing the event representation.
53
+ #
54
+ # @param payload [Hash]
55
+ # @param metadata [Hash]
56
+ # @return [Hash]
57
+ def self.with_enqueue_metadata(payload, metadata)
58
+ normalized = normalize_hash(payload)
59
+ normalized.merge(INTERNAL_METADATA_KEY => normalize_hash(metadata))
60
+ end
61
+
62
+ # Return cdc-solid-queue metadata from an enqueued payload.
63
+ #
64
+ # @param payload [Hash]
65
+ # @return [Hash]
66
+ def self.enqueue_metadata(payload)
67
+ normalized = normalize_hash(payload)
68
+ metadata = normalized[INTERNAL_METADATA_KEY]
69
+ metadata.is_a?(Hash) ? metadata : {}
70
+ end
71
+
72
+ # Return the ordering value for a serialized event.
73
+ #
74
+ # @param payload [Hash]
75
+ # @param key [Symbol]
76
+ # @return [Object, nil]
77
+ def self.ordering_value(payload, key)
78
+ normalized = load(payload)
79
+ case key
80
+ when :identity, :primary_key
81
+ normalized['identity'] || normalized['primary_key']
82
+ when :relation
83
+ [normalized['namespace'] || normalized['schema'], normalized['entity'] || normalized['table']]
84
+ when :transaction
85
+ normalized['transaction_id']
86
+ when :global
87
+ normalized['source_position'] || normalized['commit_lsn']
88
+ when :none
89
+ nil
90
+ end
91
+ end
92
+
93
+ # Normalize hash keys to strings recursively.
94
+ #
95
+ # @param value [Object]
96
+ # @return [Object]
97
+ def self.normalize_hash(value)
98
+ case value
99
+ when Hash
100
+ # @type var normalized: Hash[String, untyped]
101
+ normalized = {}
102
+ value.each_with_object(normalized) do |(key, child), normalized|
103
+ normalized[key.to_s] = normalize_hash(child)
104
+ end
105
+ when Array
106
+ value.map { |child| normalize_hash(child) }
107
+ when String, Symbol, Numeric, true, false, nil
108
+ value
109
+ else
110
+ value.to_s
111
+ end
112
+ end
113
+ private_class_method :normalize_hash
114
+
115
+ def self.build_change_event(normalized)
116
+ CDC::Core::ChangeEvent.new(
117
+ operation: normalized.fetch('operation'),
118
+ schema: normalized.fetch('schema'),
119
+ table: normalized.fetch('table'),
120
+ old_values: normalized['old_values'],
121
+ new_values: normalized['new_values'],
122
+ primary_key: normalized['primary_key'],
123
+ transaction_id: normalized['transaction_id'],
124
+ commit_lsn: normalized['commit_lsn'],
125
+ sequence_number: normalized['sequence_number'],
126
+ occurred_at: normalized['occurred_at'],
127
+ metadata: normalized['metadata'] || {}
128
+ )
129
+ end
130
+ private_class_method :build_change_event
131
+
132
+ def self.strip_internal_metadata(payload)
133
+ payload.reject { |key, _value| key == INTERNAL_METADATA_KEY }
134
+ end
135
+ private_class_method :strip_internal_metadata
136
+
137
+ def self.change_event_payload?(payload)
138
+ payload.key?('operation') && payload.key?('schema') && payload.key?('table')
139
+ end
140
+ private_class_method :change_event_payload?
141
+ end
142
+ end
143
+ end
@@ -0,0 +1,90 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'pgoutput_client'
4
+ require 'pgoutput'
5
+ require 'pgoutput/decoder'
6
+ require 'pgoutput/source_adapter'
7
+
8
+ module CDC
9
+ module SolidQueue
10
+ # Enumerable stream that normalizes PostgreSQL pgoutput payloads into CDC events.
11
+ class PostgresqlStream
12
+ # @param configuration [Configuration]
13
+ # @param client_runner [Object, nil]
14
+ # @param relation_tracker [Object]
15
+ # @param decoder [Object]
16
+ # @param adapter [Object, nil]
17
+ def initialize(configuration, client_runner: nil, relation_tracker: Pgoutput::RelationTracker.new,
18
+ decoder: Pgoutput::Decoder.new, adapter: nil)
19
+ @configuration = configuration
20
+ @client_runner = client_runner || build_client_runner
21
+ @relation_tracker = relation_tracker
22
+ @decoder = decoder
23
+ @adapter = adapter || build_adapter
24
+ @transport_metadata = nil
25
+ end
26
+
27
+ # Stream normalized CDC::Core::ChangeEvent instances.
28
+ #
29
+ # @yieldparam event [CDC::Core::ChangeEvent]
30
+ # @return [void]
31
+ def each
32
+ return enum_for(:each) unless block_given?
33
+
34
+ @client_runner.start do |payload, metadata|
35
+ @transport_metadata = metadata
36
+ event = normalize(payload)
37
+ yield event unless event.nil?
38
+ ensure
39
+ @transport_metadata = nil
40
+ end
41
+ end
42
+
43
+ private
44
+
45
+ def build_client_runner
46
+ Pgoutput::Client::Runner.new(**postgresql_options)
47
+ end
48
+
49
+ def postgresql_options
50
+ settings = @configuration.postgresql
51
+ {
52
+ database_url: settings.fetch(:database_url) { ENV.fetch('DATABASE_URL') },
53
+ slot_name: settings.fetch(:slot) { settings.fetch(:slot_name) },
54
+ publication_names: settings.fetch(:publication) { settings.fetch(:publication_names) },
55
+ start_lsn: settings[:start_lsn],
56
+ auto_create_slot: settings.fetch(:auto_create_slot, false),
57
+ temporary_slot: settings.fetch(:temporary_slot, false),
58
+ binary: settings.fetch(:binary, false),
59
+ messages: settings.fetch(:messages, false)
60
+ }.compact
61
+ end
62
+
63
+ def build_adapter
64
+ Pgoutput::SourceAdapter::Cdc.new(metadata_builder: method(:metadata_for))
65
+ end
66
+
67
+ def normalize(payload)
68
+ message = @relation_tracker.process(payload)
69
+ decoded = @decoder.decode(message)
70
+ return nil if decoded.nil?
71
+
72
+ @adapter.normalize(decoded)
73
+ end
74
+
75
+ def metadata_for(_event)
76
+ metadata = @transport_metadata
77
+ return {} if metadata.nil?
78
+
79
+ {
80
+ 'wal_end_lsn' => metadata_value(metadata, :wal_end_lsn),
81
+ 'server_time' => metadata_value(metadata, :server_time)
82
+ }.compact
83
+ end
84
+
85
+ def metadata_value(metadata, name)
86
+ metadata.public_send(name) if metadata.respond_to?(name)
87
+ end
88
+ end
89
+ end
90
+ end
@@ -0,0 +1,35 @@
1
+ # frozen_string_literal: true
2
+
3
+ module CDC
4
+ module SolidQueue
5
+ # Mixin for Rails ApplicationJob classes that consume CDC event payloads.
6
+ #
7
+ # Including classes implement #process(event). Active Job calls #perform,
8
+ # this mixin deserializes the payload, then delegates to #process.
9
+ module ProcessorJob
10
+ # Add a default queue name when Active Job provides queue_as.
11
+ #
12
+ # @param base [Class]
13
+ # @return [void]
14
+ def self.included(base)
15
+ base.queue_as(:cdc) if base.respond_to?(:queue_as)
16
+ end
17
+
18
+ # Active Job entrypoint.
19
+ #
20
+ # @param payload [Hash]
21
+ # @return [Object] process return value
22
+ def perform(payload)
23
+ process(EventSerializer.load_event(payload))
24
+ end
25
+
26
+ # Process a normalized CDC event payload.
27
+ #
28
+ # @param event [Hash]
29
+ # @raise [NotImplementedError] when the including job does not override it
30
+ def process(event)
31
+ raise NotImplementedError, "#{self.class} must implement #process"
32
+ end
33
+ end
34
+ end
35
+ end
@@ -0,0 +1,14 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'rails/railtie'
4
+
5
+ module CDC
6
+ module SolidQueue
7
+ # Rails integration for cdc-solid-queue tasks.
8
+ class Railtie < Rails::Railtie
9
+ rake_tasks do
10
+ load File.expand_path('tasks/cdc_solid_queue.rake', File.dirname(__FILE__))
11
+ end
12
+ end
13
+ end
14
+ end
@@ -0,0 +1,41 @@
1
+ # frozen_string_literal: true
2
+
3
+ module CDC
4
+ module SolidQueue
5
+ # Minimal ingestion runner boundary.
6
+ #
7
+ # The runner accepts any stream object that yields PostgreSQL-derived CDC
8
+ # events. This keeps the class testable while production code can supply a
9
+ # pgoutput-client backed stream.
10
+ class Runner
11
+ # @param stream [#each]
12
+ # @param enqueuer [Enqueuer]
13
+ def initialize(stream:, enqueuer:)
14
+ raise ArgumentError, 'stream must respond to #each' unless stream.respond_to?(:each)
15
+
16
+ @stream = stream
17
+ @enqueuer = enqueuer
18
+ end
19
+
20
+ # Start reading events and enqueueing jobs.
21
+ #
22
+ # @return [Integer] number of enqueued events
23
+ def start
24
+ count = 0
25
+ @stream.each do |event|
26
+ result = @enqueuer.enqueue(event)
27
+ checkpoint(event, result)
28
+ count += 1
29
+ end
30
+ count
31
+ end
32
+
33
+ private
34
+
35
+ def checkpoint(event, result)
36
+ store = @enqueuer.configuration.checkpoint
37
+ store&.advance(event, result)
38
+ end
39
+ end
40
+ end
41
+ end
@@ -0,0 +1,12 @@
1
+ # frozen_string_literal: true
2
+
3
+ namespace :cdc_solid_queue do
4
+ desc 'Start PostgreSQL CDC ingestion into Solid Queue'
5
+ task start: :environment do
6
+ configuration = CDC::SolidQueue.configuration
7
+ enqueuer = CDC::SolidQueue::Enqueuer.new(configuration)
8
+ stream = CDC::SolidQueue::PostgresqlStream.new(configuration)
9
+
10
+ CDC::SolidQueue::Runner.new(stream:, enqueuer:).start
11
+ end
12
+ end
@@ -0,0 +1,8 @@
1
+ # frozen_string_literal: true
2
+
3
+ module CDC
4
+ module SolidQueue
5
+ # Current cdc-solid-queue gem version.
6
+ VERSION = '0.1.0'
7
+ end
8
+ end
@@ -0,0 +1,42 @@
1
+ # frozen_string_literal: true
2
+
3
+ require_relative 'solid_queue/version'
4
+ require_relative 'solid_queue/error'
5
+ require_relative 'solid_queue/event_serializer'
6
+ require_relative 'solid_queue/checkpoint'
7
+ require_relative 'solid_queue/configuration'
8
+ require_relative 'solid_queue/enqueuer'
9
+ require_relative 'solid_queue/processor_job'
10
+ require_relative 'solid_queue/postgresql_stream'
11
+ require_relative 'solid_queue/runner'
12
+
13
+ # Namespace for Change Data Capture integrations.
14
+ module CDC
15
+ # Rails-native durable CDC job backend built on Solid Queue.
16
+ module SolidQueue
17
+ class << self
18
+ # Return the global configuration.
19
+ #
20
+ # @return [Configuration]
21
+ def configuration
22
+ @configuration ||= Configuration.new
23
+ end
24
+
25
+ # Configure cdc-solid-queue.
26
+ #
27
+ # @yieldparam config [Configuration]
28
+ # @return [Configuration]
29
+ def configure
30
+ yield configuration
31
+ configuration
32
+ end
33
+
34
+ # Reset configuration. Intended for tests and console experiments.
35
+ #
36
+ # @return [Configuration]
37
+ def reset_configuration!
38
+ @configuration = Configuration.new
39
+ end
40
+ end
41
+ end
42
+ end
@@ -0,0 +1,158 @@
1
+ module Rails
2
+ class Railtie
3
+ def self.rake_tasks: () { () -> untyped } -> untyped
4
+ end
5
+ end
6
+
7
+ module Pgoutput
8
+ class RelationTracker
9
+ def initialize: () -> void
10
+ def process: (untyped payload) -> untyped
11
+ end
12
+
13
+ class Decoder
14
+ def initialize: (?type_registry: untyped) -> void
15
+ def decode: (untyped message) -> untyped
16
+ end
17
+
18
+ module Client
19
+ class Runner
20
+ attr_reader configuration: untyped
21
+ def initialize: (**untyped options) -> void
22
+ def start: () { (String payload, untyped metadata) -> untyped } -> void
23
+ end
24
+ end
25
+
26
+ module SourceAdapter
27
+ class Cdc
28
+ def initialize: (?primary_key_resolver: untyped, ?metadata_builder: untyped) -> void
29
+ def normalize: (untyped event) -> untyped
30
+ end
31
+ end
32
+ end
33
+
34
+ module CDC
35
+ module SolidQueue
36
+ VERSION: String
37
+
38
+ def self.configuration: () -> Configuration
39
+ def self.configure: () { (Configuration) -> untyped } -> Configuration
40
+ def self.reset_configuration!: () -> Configuration
41
+
42
+ class Error < StandardError
43
+ end
44
+
45
+ class ConfigurationError < Error
46
+ end
47
+
48
+ class UnsupportedSourceError < ConfigurationError
49
+ end
50
+
51
+ class SerializationError < Error
52
+ end
53
+
54
+ class Configuration
55
+ SUPPORTED_SOURCE: Symbol
56
+ ORDERING_KEYS: Array[Symbol]
57
+
58
+ attr_accessor processor_job: untyped
59
+ attr_accessor queue: String
60
+ attr_accessor preserve_order: bool
61
+ attr_accessor ordering_key: Symbol
62
+ attr_accessor postgresql: Hash[Symbol, untyped]
63
+ attr_accessor checkpoint: untyped
64
+
65
+ def initialize: () -> void
66
+ def validate!: () -> true
67
+ def source: () -> Symbol
68
+
69
+ private
70
+
71
+ def validate_processor_job!: () -> nil
72
+ def validate_queue!: () -> nil
73
+ def validate_ordering_key!: () -> nil
74
+ def validate_postgresql!: () -> nil
75
+ def validate_checkpoint!: () -> nil
76
+ end
77
+
78
+ class EventSerializer
79
+ INTERNAL_METADATA_KEY: String
80
+
81
+ def self.dump: (untyped event) -> Hash[String, untyped]
82
+ def self.load: (Hash[untyped, untyped] payload) -> Hash[String, untyped]
83
+ def self.load_event: (Hash[untyped, untyped] payload) -> untyped
84
+ def self.with_enqueue_metadata: (Hash[untyped, untyped] payload, Hash[untyped, untyped] metadata) -> Hash[String, untyped]
85
+ def self.enqueue_metadata: (Hash[untyped, untyped] payload) -> Hash[String, untyped]
86
+ def self.ordering_value: (Hash[untyped, untyped] payload, Symbol key) -> untyped
87
+
88
+ private
89
+
90
+ def self.normalize_hash: (untyped value) -> untyped
91
+ def self.build_change_event: (Hash[String, untyped] normalized) -> untyped
92
+ def self.strip_internal_metadata: (Hash[String, untyped] payload) -> Hash[String, untyped]
93
+ def self.change_event_payload?: (Hash[String, untyped] payload) -> bool
94
+ end
95
+
96
+ class Checkpoint
97
+ attr_reader position: untyped
98
+ def initialize: () -> void
99
+ def advance: (untyped event, ?untyped result) -> untyped
100
+
101
+ private
102
+
103
+ def position_for: (untyped event) -> untyped
104
+ def result_position: (untyped result) -> untyped
105
+ end
106
+
107
+ class Enqueuer
108
+ attr_reader configuration: Configuration
109
+ def initialize: (Configuration configuration) -> void
110
+ def enqueue: (untyped event) -> untyped
111
+
112
+ private
113
+
114
+ def async_job: (untyped job) -> untyped
115
+ def enqueue_metadata: (Hash[untyped, untyped] payload) -> Hash[String, untyped]
116
+ def ordering_value: (Hash[untyped, untyped] payload) -> untyped
117
+ end
118
+
119
+ module ProcessorJob
120
+ def self.included: (untyped base) -> void
121
+ def perform: (Hash[untyped, untyped] payload) -> untyped
122
+ def process: (Hash[String, untyped] event) -> untyped
123
+ end
124
+
125
+ class Runner
126
+ def initialize: (stream: untyped, enqueuer: Enqueuer) -> void
127
+ def start: () -> Integer
128
+
129
+ private
130
+
131
+ def checkpoint: (untyped event, untyped result) -> untyped
132
+ end
133
+
134
+ class PostgresqlStream
135
+ def initialize: (
136
+ Configuration configuration,
137
+ ?client_runner: untyped,
138
+ ?relation_tracker: untyped,
139
+ ?decoder: untyped,
140
+ ?adapter: untyped
141
+ ) -> void
142
+ def each: () { (untyped event) -> untyped } -> void
143
+ | () -> Enumerator[untyped, void]
144
+
145
+ private
146
+
147
+ def build_client_runner: () -> untyped
148
+ def postgresql_options: () -> Hash[Symbol, untyped]
149
+ def build_adapter: () -> untyped
150
+ def normalize: (untyped payload) -> untyped
151
+ def metadata_for: (untyped event) -> Hash[String, untyped]
152
+ def metadata_value: (untyped metadata, Symbol name) -> untyped
153
+ end
154
+
155
+ class Railtie < ::Rails::Railtie
156
+ end
157
+ end
158
+ end
metadata ADDED
@@ -0,0 +1,131 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: cdc-solid-queue
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: ruby
6
+ authors:
7
+ - Ken C. Demanawa
8
+ bindir: bin
9
+ cert_chain: []
10
+ date: 1980-01-02 00:00:00.000000000 Z
11
+ dependencies:
12
+ - !ruby/object:Gem::Dependency
13
+ name: cdc-core
14
+ requirement: !ruby/object:Gem::Requirement
15
+ requirements:
16
+ - - "~>"
17
+ - !ruby/object:Gem::Version
18
+ version: 0.1.3
19
+ type: :runtime
20
+ prerelease: false
21
+ version_requirements: !ruby/object:Gem::Requirement
22
+ requirements:
23
+ - - "~>"
24
+ - !ruby/object:Gem::Version
25
+ version: 0.1.3
26
+ - !ruby/object:Gem::Dependency
27
+ name: pgoutput-client
28
+ requirement: !ruby/object:Gem::Requirement
29
+ requirements:
30
+ - - "~>"
31
+ - !ruby/object:Gem::Version
32
+ version: 0.2.4
33
+ type: :runtime
34
+ prerelease: false
35
+ version_requirements: !ruby/object:Gem::Requirement
36
+ requirements:
37
+ - - "~>"
38
+ - !ruby/object:Gem::Version
39
+ version: 0.2.4
40
+ - !ruby/object:Gem::Dependency
41
+ name: pgoutput-decoder
42
+ requirement: !ruby/object:Gem::Requirement
43
+ requirements:
44
+ - - "~>"
45
+ - !ruby/object:Gem::Version
46
+ version: 0.1.1
47
+ type: :runtime
48
+ prerelease: false
49
+ version_requirements: !ruby/object:Gem::Requirement
50
+ requirements:
51
+ - - "~>"
52
+ - !ruby/object:Gem::Version
53
+ version: 0.1.1
54
+ - !ruby/object:Gem::Dependency
55
+ name: pgoutput-parser
56
+ requirement: !ruby/object:Gem::Requirement
57
+ requirements:
58
+ - - "~>"
59
+ - !ruby/object:Gem::Version
60
+ version: 0.1.1
61
+ type: :runtime
62
+ prerelease: false
63
+ version_requirements: !ruby/object:Gem::Requirement
64
+ requirements:
65
+ - - "~>"
66
+ - !ruby/object:Gem::Version
67
+ version: 0.1.1
68
+ - !ruby/object:Gem::Dependency
69
+ name: pgoutput-source-adapter
70
+ requirement: !ruby/object:Gem::Requirement
71
+ requirements:
72
+ - - "~>"
73
+ - !ruby/object:Gem::Version
74
+ version: 0.1.1
75
+ type: :runtime
76
+ prerelease: false
77
+ version_requirements: !ruby/object:Gem::Requirement
78
+ requirements:
79
+ - - "~>"
80
+ - !ruby/object:Gem::Version
81
+ version: 0.1.1
82
+ description: Bridges PostgreSQL CDC events into Solid Queue-backed Active Job processors.
83
+ email:
84
+ - kenneth.c.demanawa@gmail.com
85
+ executables: []
86
+ extensions: []
87
+ extra_rdoc_files: []
88
+ files:
89
+ - CHANGELOG.md
90
+ - LICENSE.txt
91
+ - README.md
92
+ - lib/cdc/solid_queue.rb
93
+ - lib/cdc/solid_queue/checkpoint.rb
94
+ - lib/cdc/solid_queue/configuration.rb
95
+ - lib/cdc/solid_queue/enqueuer.rb
96
+ - lib/cdc/solid_queue/error.rb
97
+ - lib/cdc/solid_queue/event_serializer.rb
98
+ - lib/cdc/solid_queue/postgresql_stream.rb
99
+ - lib/cdc/solid_queue/processor_job.rb
100
+ - lib/cdc/solid_queue/railtie.rb
101
+ - lib/cdc/solid_queue/runner.rb
102
+ - lib/cdc/solid_queue/tasks/cdc_solid_queue.rake
103
+ - lib/cdc/solid_queue/version.rb
104
+ - sig/cdc/solid_queue.rbs
105
+ homepage: https://github.com/kanutocd/cdc-solid-queue
106
+ licenses:
107
+ - MIT
108
+ metadata:
109
+ allowed_push_host: https://rubygems.org
110
+ homepage_uri: https://github.com/kanutocd/cdc-solid-queue
111
+ source_code_uri: https://github.com/kanutocd/cdc-solid-queue/tree/main
112
+ changelog_uri: https://github.com/kanutocd/cdc-solid-queue/blob/main/CHANGELOG.md
113
+ rubygems_mfa_required: 'true'
114
+ rdoc_options: []
115
+ require_paths:
116
+ - lib
117
+ required_ruby_version: !ruby/object:Gem::Requirement
118
+ requirements:
119
+ - - ">="
120
+ - !ruby/object:Gem::Version
121
+ version: '3.4'
122
+ required_rubygems_version: !ruby/object:Gem::Requirement
123
+ requirements:
124
+ - - ">="
125
+ - !ruby/object:Gem::Version
126
+ version: '0'
127
+ requirements: []
128
+ rubygems_version: 3.6.9
129
+ specification_version: 4
130
+ summary: Rails-native durable CDC job backend for Solid Queue.
131
+ test_files: []