pgoutput-source-adapter 0.1.1 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +43 -1
- data/README.md +46 -4
- data/Rakefile +12 -1
- data/lib/pgoutput/source/adapter.rb +8 -0
- data/lib/pgoutput/source_adapter/cdc.rb +137 -30
- data/lib/pgoutput/source_adapter/replica_identity_resolver.rb +99 -0
- data/lib/pgoutput/source_adapter/version.rb +1 -1
- data/lib/pgoutput/source_adapter.rb +1 -0
- data/sig/pgoutput/source_adapter/cdc.rbs +29 -4
- data/sig/pgoutput/source_adapter/replica_identity_resolver.rbs +20 -0
- metadata +3 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 96fa814d19258cf12c3f2d6398f72f21f1a2dfab59e89562b880f26fbb7bf1cd
|
|
4
|
+
data.tar.gz: 300f6a751391ee90ba84dfdbe743839e605fb6655aafc5057f9b87a1e25c3faf
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 314c92d8ec4d3853056f4c69d99eefb6794aaefbc58a1b5f9e0b2de25c372cce1e8c4864210ade04400d23feb60fc47d68d535c1df901d7ef60a373bc4c735b8
|
|
7
|
+
data.tar.gz: 88c5f21bc01667b196a1b347e176cca6aa7caa655a058fa1550bdee3bcb3b88c511fd9371225f5a95fe585dac572a219b46a14a7135e956c97ac47fd8466e986
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,47 @@
|
|
|
1
1
|
## [Unreleased]
|
|
2
2
|
|
|
3
|
+
## [0.3.0] - 2026-07-17
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
|
|
7
|
+
- Added `ReplicaIdentityResolver` for schema-aware non-`id` and composite keys,
|
|
8
|
+
keyed by relation id, schema/table pair, or qualified relation name.
|
|
9
|
+
- Added default extraction from decoded relation key-column metadata when
|
|
10
|
+
events expose replica-identity columns or flagged relation columns.
|
|
11
|
+
|
|
12
|
+
### Fixed
|
|
13
|
+
|
|
14
|
+
- Preserved complete composite and non-`id` old-key tuples for normalized
|
|
15
|
+
UPDATE and DELETE events.
|
|
16
|
+
- Fail closed when configured replica-key columns are absent instead of
|
|
17
|
+
emitting an incomplete `ChangeEvent#primary_key`.
|
|
18
|
+
|
|
19
|
+
### Documentation
|
|
20
|
+
|
|
21
|
+
- Documented relation metadata, schema-aware resolver configuration, and the
|
|
22
|
+
legacy `id` compatibility fallback.
|
|
23
|
+
|
|
24
|
+
## [0.2.0] - 2026-07-17
|
|
25
|
+
|
|
26
|
+
### Added
|
|
27
|
+
|
|
28
|
+
- Added incremental `Cdc#each_normalized` transaction normalization for
|
|
29
|
+
unbounded replication streams.
|
|
30
|
+
- Added `Cdc#stream_event` so source integrations can preserve transport WAL
|
|
31
|
+
positions without owning transaction buffering.
|
|
32
|
+
|
|
33
|
+
### Documentation
|
|
34
|
+
|
|
35
|
+
- Raised and enforced generated YARD API documentation coverage at 100%.
|
|
36
|
+
|
|
37
|
+
### Quality
|
|
38
|
+
|
|
39
|
+
- Raised and enforced SimpleCov line and branch coverage at 100%.
|
|
40
|
+
|
|
41
|
+
## [0.1.1] - 2026-06-16
|
|
42
|
+
|
|
43
|
+
- Improved gemspec summary and description
|
|
44
|
+
|
|
3
45
|
## [0.1.0] - 2026-06-16
|
|
4
46
|
|
|
5
47
|
### Added
|
|
@@ -31,4 +73,4 @@
|
|
|
31
73
|
### Coverage
|
|
32
74
|
|
|
33
75
|
- Line Coverage: 97.62%
|
|
34
|
-
- Branch Coverage: 91.18%
|
|
76
|
+
- Branch Coverage: 91.18%
|
data/README.md
CHANGED
|
@@ -85,20 +85,57 @@ envelope = results.first
|
|
|
85
85
|
# => CDC::Core::TransactionEnvelope
|
|
86
86
|
```
|
|
87
87
|
|
|
88
|
+
Normalize an unbounded replication stream incrementally:
|
|
89
|
+
|
|
90
|
+
```ruby
|
|
91
|
+
inputs = Enumerator.new do |stream|
|
|
92
|
+
runner.start do |payload, metadata|
|
|
93
|
+
decoded = decoder.decode(parser.process(payload))
|
|
94
|
+
stream << adapter.stream_event(decoded, source_position: metadata.wal_end_lsn)
|
|
95
|
+
end
|
|
96
|
+
end
|
|
97
|
+
|
|
98
|
+
adapter.each_normalized(inputs) do |work|
|
|
99
|
+
processor.process(work)
|
|
100
|
+
end
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Transaction buffering belongs to the source adapter. Changes outside a
|
|
104
|
+
transaction are yielded immediately; changes between `Begin` and `Commit` are
|
|
105
|
+
yielded as a `CDC::Core::TransactionEnvelope`. `normalize_many` uses the same
|
|
106
|
+
normalization path for finite batches.
|
|
107
|
+
|
|
88
108
|
## Primary keys
|
|
89
109
|
|
|
90
|
-
For update and delete events, pgoutput may provide an old-key tuple. When it
|
|
110
|
+
For update and delete events, pgoutput may provide an old-key tuple. When it
|
|
111
|
+
does, the complete tuple is used as `CDC::Core::ChangeEvent#primary_key`,
|
|
112
|
+
including composite and non-`id` replica keys.
|
|
91
113
|
|
|
92
|
-
For
|
|
114
|
+
For inserts and events without an old-key tuple, the default resolver uses
|
|
115
|
+
relation key-column metadata when the decoded event exposes
|
|
116
|
+
`replica_identity_columns`, `key_columns`, or a relation whose column flags mark
|
|
117
|
+
replica-key columns. The adapter extracts every declared column and raises
|
|
118
|
+
`Pgoutput::SourceAdapter::Error` rather than returning a partial key.
|
|
93
119
|
|
|
94
|
-
|
|
120
|
+
When decoded events do not carry relation key metadata, configure the reusable
|
|
121
|
+
schema-aware resolver. Relations may be keyed by relation id,
|
|
122
|
+
`[schema, table]`, or a qualified name:
|
|
95
123
|
|
|
96
124
|
```ruby
|
|
125
|
+
resolver = Pgoutput::SourceAdapter::ReplicaIdentityResolver.new(
|
|
126
|
+
["public", "memberships"] => ["tenant_id", "member_uuid"],
|
|
127
|
+
"public.accounts" => ["account_uuid"]
|
|
128
|
+
)
|
|
129
|
+
|
|
97
130
|
adapter = Pgoutput::SourceAdapter::Cdc.new(
|
|
98
|
-
primary_key_resolver:
|
|
131
|
+
primary_key_resolver: resolver
|
|
99
132
|
)
|
|
100
133
|
```
|
|
101
134
|
|
|
135
|
+
The legacy `id` / `"id"` inference remains as a compatibility fallback only
|
|
136
|
+
when neither relation metadata nor an explicit resolver is available. Custom
|
|
137
|
+
callables remain supported through `primary_key_resolver:`.
|
|
138
|
+
|
|
102
139
|
## Metadata
|
|
103
140
|
|
|
104
141
|
Each normalized event includes pgoutput metadata:
|
|
@@ -124,6 +161,7 @@ adapter = Pgoutput::SourceAdapter::Cdc.new(
|
|
|
124
161
|
```ruby
|
|
125
162
|
Pgoutput::SourceAdapter
|
|
126
163
|
Pgoutput::SourceAdapter::Cdc
|
|
164
|
+
Pgoutput::SourceAdapter::ReplicaIdentityResolver
|
|
127
165
|
```
|
|
128
166
|
|
|
129
167
|
A compatibility alias is also provided for the generated gem path:
|
|
@@ -151,6 +189,10 @@ Those responsibilities belong to `pgoutput-client`, `pgoutput-parser`, `pgoutput
|
|
|
151
189
|
bundle exec rake
|
|
152
190
|
```
|
|
153
191
|
|
|
192
|
+
The default quality task enforces 100% line and branch coverage and 100% YARD
|
|
193
|
+
API documentation coverage. Run `bundle exec yard stats --list-undoc` to inspect
|
|
194
|
+
the documented public API objects.
|
|
195
|
+
|
|
154
196
|
## License
|
|
155
197
|
|
|
156
198
|
[MIT](./LICENSE.txt).
|
data/Rakefile
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require 'bundler/gem_tasks'
|
|
4
|
+
require 'open3'
|
|
4
5
|
require 'rake/testtask'
|
|
5
6
|
require 'rubocop/rake_task'
|
|
6
7
|
require 'yard'
|
|
@@ -17,6 +18,16 @@ end
|
|
|
17
18
|
|
|
18
19
|
YARD::Rake::YardocTask.new(:yard)
|
|
19
20
|
|
|
21
|
+
desc 'Require complete public API documentation'
|
|
22
|
+
task yard_coverage: :yard do
|
|
23
|
+
output, status = Open3.capture2e('bundle', 'exec', 'yard', 'stats', '--list-undoc')
|
|
24
|
+
puts output
|
|
25
|
+
|
|
26
|
+
next if status.success? && output.include?('100.00% documented')
|
|
27
|
+
|
|
28
|
+
abort 'YARD documentation coverage must remain at 100%'
|
|
29
|
+
end
|
|
30
|
+
|
|
20
31
|
desc 'Validate rbs sig files'
|
|
21
32
|
task :steep do
|
|
22
33
|
sh 'bundle exec steep check'
|
|
@@ -27,4 +38,4 @@ task :coverage do
|
|
|
27
38
|
sh 'xdg-open coverage/index.html'
|
|
28
39
|
end
|
|
29
40
|
|
|
30
|
-
task default: %i[test rubocop steep
|
|
41
|
+
task default: %i[test rubocop steep yard_coverage]
|
|
@@ -2,7 +2,15 @@
|
|
|
2
2
|
|
|
3
3
|
require_relative '../source_adapter'
|
|
4
4
|
|
|
5
|
+
# Root namespace for pgoutput protocol libraries.
|
|
6
|
+
#
|
|
7
|
+
# @api public
|
|
5
8
|
module Pgoutput
|
|
9
|
+
# Backward-compatible namespace for the original generated gem layout.
|
|
10
|
+
#
|
|
11
|
+
# New integrations should use {Pgoutput::SourceAdapter} directly.
|
|
12
|
+
#
|
|
13
|
+
# @api public
|
|
6
14
|
module Source
|
|
7
15
|
# Compatibility alias for {Pgoutput::SourceAdapter}.
|
|
8
16
|
#
|
|
@@ -2,12 +2,20 @@
|
|
|
2
2
|
|
|
3
3
|
require 'cdc/core'
|
|
4
4
|
|
|
5
|
-
unless CDC::Core.const_defined?(:SourceAdapter)
|
|
6
|
-
raise LoadError, 'pgoutput-source-adapter requires a cdc-core version that defines CDC::Core::SourceAdapter'
|
|
7
|
-
end
|
|
8
|
-
|
|
9
5
|
module Pgoutput
|
|
10
6
|
module SourceAdapter
|
|
7
|
+
# Resolves the required cdc-core adapter contract.
|
|
8
|
+
#
|
|
9
|
+
# @api private
|
|
10
|
+
def self.ensure_core_source_adapter!(core)
|
|
11
|
+
return if core.const_defined?(:SourceAdapter)
|
|
12
|
+
|
|
13
|
+
raise LoadError, 'pgoutput-source-adapter requires a cdc-core version that defines CDC::Core::SourceAdapter'
|
|
14
|
+
end
|
|
15
|
+
private_class_method :ensure_core_source_adapter!
|
|
16
|
+
|
|
17
|
+
ensure_core_source_adapter!(CDC::Core)
|
|
18
|
+
|
|
11
19
|
# Normalizes Pgoutput::Decoder::Events into CDC::Core primitives.
|
|
12
20
|
#
|
|
13
21
|
# This adapter is intentionally located under the Pgoutput namespace because
|
|
@@ -22,10 +30,40 @@ module Pgoutput
|
|
|
22
30
|
# @example Normalize a decoded transaction event batch
|
|
23
31
|
# envelope = adapter.normalize_many([begin_event, insert_event, commit_event]).first
|
|
24
32
|
#
|
|
33
|
+
# @example Normalize an unbounded decoded event stream
|
|
34
|
+
# adapter.each_normalized(decoded_events) { |work| process(work) }
|
|
35
|
+
#
|
|
25
36
|
# @api public
|
|
37
|
+
# rubocop:disable Metrics/ClassLength
|
|
26
38
|
class Cdc < CDC::Core::SourceAdapter
|
|
39
|
+
# Canonical source name included in normalized event metadata.
|
|
40
|
+
#
|
|
41
|
+
# @return [String]
|
|
27
42
|
SOURCE_NAME = 'pgoutput'
|
|
28
43
|
|
|
44
|
+
# A decoded event paired with its transport-level source position.
|
|
45
|
+
#
|
|
46
|
+
# Source integrations use this value object to preserve WAL position
|
|
47
|
+
# metadata without teaching downstream applications about pgoutput
|
|
48
|
+
# transaction boundaries.
|
|
49
|
+
#
|
|
50
|
+
# @api public
|
|
51
|
+
class StreamEvent
|
|
52
|
+
# @return [Object] decoded pgoutput event
|
|
53
|
+
attr_reader :event
|
|
54
|
+
# @return [String, nil] transport WAL position
|
|
55
|
+
attr_reader :source_position
|
|
56
|
+
|
|
57
|
+
# @param event [Object] decoded pgoutput event
|
|
58
|
+
# @param source_position [String, nil] transport WAL position
|
|
59
|
+
# @return [void]
|
|
60
|
+
def initialize(event, source_position)
|
|
61
|
+
@event = event
|
|
62
|
+
@source_position = source_position
|
|
63
|
+
freeze
|
|
64
|
+
end
|
|
65
|
+
end
|
|
66
|
+
|
|
29
67
|
# @param primary_key_resolver [#call, nil] optional callable used to infer
|
|
30
68
|
# primary keys from decoded row values when pgoutput does not provide an
|
|
31
69
|
# old-key tuple. The callable receives the decoded event and value hash.
|
|
@@ -50,7 +88,7 @@ module Pgoutput
|
|
|
50
88
|
# @raise [Pgoutput::SourceAdapter::Error] when the decoded event type is
|
|
51
89
|
# unsupported.
|
|
52
90
|
# rubocop:disable Metrics/MethodLength
|
|
53
|
-
def normalize(event)
|
|
91
|
+
def normalize(event, source_position: nil)
|
|
54
92
|
case event_name(event)
|
|
55
93
|
when 'Insert'
|
|
56
94
|
change_event(
|
|
@@ -58,7 +96,8 @@ module Pgoutput
|
|
|
58
96
|
operation: :insert,
|
|
59
97
|
old_values: nil,
|
|
60
98
|
new_values: event.values,
|
|
61
|
-
primary_key: primary_key_for(event, event.values)
|
|
99
|
+
primary_key: primary_key_for(event, event.values),
|
|
100
|
+
source_position: source_position
|
|
62
101
|
)
|
|
63
102
|
when 'Update'
|
|
64
103
|
old_values = event.old_values || event.old_key
|
|
@@ -68,7 +107,8 @@ module Pgoutput
|
|
|
68
107
|
operation: :update,
|
|
69
108
|
old_values: old_values,
|
|
70
109
|
new_values: event.new_values,
|
|
71
|
-
primary_key: primary_key_for(event, event.new_values)
|
|
110
|
+
primary_key: primary_key_for(event, event.new_values),
|
|
111
|
+
source_position: source_position
|
|
72
112
|
)
|
|
73
113
|
when 'Delete'
|
|
74
114
|
old_values = event.old_values || event.old_key
|
|
@@ -78,7 +118,8 @@ module Pgoutput
|
|
|
78
118
|
operation: :delete,
|
|
79
119
|
old_values: old_values,
|
|
80
120
|
new_values: nil,
|
|
81
|
-
primary_key: primary_key_for(event, old_values)
|
|
121
|
+
primary_key: primary_key_for(event, old_values),
|
|
122
|
+
source_position: source_position
|
|
82
123
|
)
|
|
83
124
|
when 'Begin', 'Commit'
|
|
84
125
|
nil
|
|
@@ -88,6 +129,31 @@ module Pgoutput
|
|
|
88
129
|
end
|
|
89
130
|
# rubocop:enable Metrics/MethodLength
|
|
90
131
|
|
|
132
|
+
# Pair a decoded event with the source position supplied by its transport.
|
|
133
|
+
#
|
|
134
|
+
# @param event [Object] decoded pgoutput event
|
|
135
|
+
# @param source_position [Object, nil] WAL position associated with the event
|
|
136
|
+
# @return [StreamEvent] immutable streaming input
|
|
137
|
+
def stream_event(event, source_position: nil)
|
|
138
|
+
StreamEvent.new(event, lsn_string(source_position))
|
|
139
|
+
end
|
|
140
|
+
|
|
141
|
+
# Incrementally normalize a decoded pgoutput event stream.
|
|
142
|
+
#
|
|
143
|
+
# Transaction buffering belongs to this source adapter: row changes are
|
|
144
|
+
# retained after Begin and emitted as one CDC::Core::TransactionEnvelope
|
|
145
|
+
# at Commit. Changes outside a transaction are yielded immediately.
|
|
146
|
+
#
|
|
147
|
+
# @param events [Enumerable<Object, StreamEvent>] decoded stream
|
|
148
|
+
# @yieldparam work [CDC::Core::ChangeEvent, CDC::Core::TransactionEnvelope]
|
|
149
|
+
# @return [Enumerator, nil] enumerator without a block, otherwise nil
|
|
150
|
+
def each_normalized(events, &block)
|
|
151
|
+
return enum_for(:each_normalized, events) unless block
|
|
152
|
+
|
|
153
|
+
normalize_stream(events, flush_incomplete: false, &block)
|
|
154
|
+
nil
|
|
155
|
+
end
|
|
156
|
+
|
|
91
157
|
# Normalize a sequence of decoded pgoutput events.
|
|
92
158
|
#
|
|
93
159
|
# If the sequence contains transaction boundaries, row changes between a
|
|
@@ -99,17 +165,28 @@ module Pgoutput
|
|
|
99
165
|
# normalized row changes and transaction envelopes in input order.
|
|
100
166
|
# @raise [Pgoutput::SourceAdapter::Error] when any decoded event type is
|
|
101
167
|
# unsupported.
|
|
168
|
+
def normalize_many(events)
|
|
169
|
+
results = [] # : Array[CDC::Core::ChangeEvent | CDC::Core::TransactionEnvelope]
|
|
170
|
+
normalize_stream(events, flush_incomplete: true) { |work| results << work }
|
|
171
|
+
share(results.freeze)
|
|
172
|
+
end
|
|
173
|
+
|
|
174
|
+
private
|
|
175
|
+
|
|
176
|
+
attr_reader :primary_key_resolver, :metadata_builder
|
|
177
|
+
|
|
102
178
|
# rubocop:disable Metrics/AbcSize
|
|
103
179
|
# rubocop:disable Metrics/PerceivedComplexity
|
|
104
180
|
# rubocop:disable Metrics/CyclomaticComplexity
|
|
105
181
|
# rubocop:disable Metrics/MethodLength
|
|
106
|
-
def
|
|
107
|
-
results = [] # : Array[CDC::Core::ChangeEvent | CDC::Core::TransactionEnvelope]
|
|
182
|
+
def normalize_stream(events, flush_incomplete:, &block)
|
|
108
183
|
transaction_id = nil
|
|
109
184
|
transaction_events = [] # : Array[CDC::Core::ChangeEvent]
|
|
110
185
|
transaction_metadata = {} # : Hash[String, untyped]
|
|
111
186
|
|
|
112
|
-
|
|
187
|
+
# rubocop:disable Metrics/BlockLength
|
|
188
|
+
events.each do |input|
|
|
189
|
+
event, source_position = stream_input(input)
|
|
113
190
|
case event_name(event)
|
|
114
191
|
when 'Begin'
|
|
115
192
|
transaction_id = event.transaction_id
|
|
@@ -120,11 +197,14 @@ module Pgoutput
|
|
|
120
197
|
)
|
|
121
198
|
when 'Commit'
|
|
122
199
|
if transaction_id || !transaction_events.empty?
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
200
|
+
block.call(
|
|
201
|
+
transaction_envelope(
|
|
202
|
+
event,
|
|
203
|
+
transaction_id: transaction_id || event.transaction_id,
|
|
204
|
+
events: transaction_events,
|
|
205
|
+
metadata: transaction_metadata,
|
|
206
|
+
source_position: source_position
|
|
207
|
+
)
|
|
128
208
|
)
|
|
129
209
|
end
|
|
130
210
|
|
|
@@ -132,30 +212,26 @@ module Pgoutput
|
|
|
132
212
|
transaction_events = []
|
|
133
213
|
transaction_metadata = {}
|
|
134
214
|
else
|
|
135
|
-
normalized = normalize(event)
|
|
215
|
+
normalized = normalize(event, source_position: source_position)
|
|
136
216
|
next if normalized.nil?
|
|
137
217
|
|
|
138
218
|
if transaction_id
|
|
139
219
|
transaction_events << normalized
|
|
140
220
|
else
|
|
141
|
-
|
|
221
|
+
block.call(normalized)
|
|
142
222
|
end
|
|
143
223
|
end
|
|
144
224
|
end
|
|
225
|
+
# rubocop:enable Metrics/BlockLength
|
|
145
226
|
|
|
146
|
-
|
|
147
|
-
share(results.freeze)
|
|
227
|
+
transaction_events.each(&block) if flush_incomplete && transaction_id && !transaction_events.empty?
|
|
148
228
|
end
|
|
149
229
|
# rubocop:enable Metrics/AbcSize
|
|
150
230
|
# rubocop:enable Metrics/PerceivedComplexity
|
|
151
231
|
# rubocop:enable Metrics/CyclomaticComplexity
|
|
152
232
|
# rubocop:enable Metrics/MethodLength
|
|
153
233
|
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
attr_reader :primary_key_resolver, :metadata_builder
|
|
157
|
-
|
|
158
|
-
def change_event(event, operation:, old_values:, new_values:, primary_key:)
|
|
234
|
+
def change_event(event, operation:, old_values:, new_values:, primary_key:, source_position:)
|
|
159
235
|
share(
|
|
160
236
|
CDC::Core::ChangeEvent.new(
|
|
161
237
|
operation: operation,
|
|
@@ -165,17 +241,18 @@ module Pgoutput
|
|
|
165
241
|
new_values: new_values,
|
|
166
242
|
primary_key: primary_key,
|
|
167
243
|
transaction_id: event.transaction_id,
|
|
244
|
+
commit_lsn: lsn_string(source_position),
|
|
168
245
|
metadata: metadata_for(event)
|
|
169
246
|
)
|
|
170
247
|
)
|
|
171
248
|
end
|
|
172
249
|
|
|
173
|
-
def transaction_envelope(event, transaction_id:, events:, metadata:)
|
|
250
|
+
def transaction_envelope(event, transaction_id:, events:, metadata:, source_position: nil)
|
|
174
251
|
share(
|
|
175
252
|
CDC::Core::TransactionEnvelope.new(
|
|
176
253
|
transaction_id: transaction_id,
|
|
177
254
|
events: events.freeze,
|
|
178
|
-
commit_lsn: lsn_string(event.commit_lsn),
|
|
255
|
+
commit_lsn: lsn_string(event.commit_lsn || source_position),
|
|
179
256
|
committed_at: event.commit_timestamp,
|
|
180
257
|
metadata: metadata.merge(metadata_for(event)).merge(
|
|
181
258
|
'commit_flags' => event.flags,
|
|
@@ -191,7 +268,15 @@ module Pgoutput
|
|
|
191
268
|
primary_key_resolver.call(event, values)
|
|
192
269
|
end
|
|
193
270
|
|
|
194
|
-
def
|
|
271
|
+
def stream_input(input)
|
|
272
|
+
return [input.event, input.source_position] if input.is_a?(StreamEvent)
|
|
273
|
+
|
|
274
|
+
[input, nil]
|
|
275
|
+
end
|
|
276
|
+
|
|
277
|
+
def default_primary_key(event, values)
|
|
278
|
+
key_columns = replica_identity_columns_for(event)
|
|
279
|
+
return key_from_columns(event, values, key_columns) unless key_columns.nil?
|
|
195
280
|
return nil unless values.respond_to?(:key?)
|
|
196
281
|
|
|
197
282
|
if values.key?('id')
|
|
@@ -201,6 +286,27 @@ module Pgoutput
|
|
|
201
286
|
end
|
|
202
287
|
end
|
|
203
288
|
|
|
289
|
+
def replica_identity_columns_for(event)
|
|
290
|
+
return normalize_key_columns(event.replica_identity_columns) if event.respond_to?(:replica_identity_columns)
|
|
291
|
+
return normalize_key_columns(event.key_columns) if event.respond_to?(:key_columns)
|
|
292
|
+
return unless event.respond_to?(:relation) && event.relation.respond_to?(:columns)
|
|
293
|
+
|
|
294
|
+
normalize_key_columns(
|
|
295
|
+
event.relation.columns.select { |column| column.flags.allbits?(1) }.map(&:name)
|
|
296
|
+
)
|
|
297
|
+
end
|
|
298
|
+
|
|
299
|
+
def normalize_key_columns(columns)
|
|
300
|
+
column_list = Array(columns) # : Array[untyped]
|
|
301
|
+
column_list.map(&:to_s)
|
|
302
|
+
end
|
|
303
|
+
|
|
304
|
+
def key_from_columns(event, values, columns)
|
|
305
|
+
return nil if columns.empty?
|
|
306
|
+
|
|
307
|
+
ReplicaIdentityResolver.extract(event, values, columns)
|
|
308
|
+
end
|
|
309
|
+
|
|
204
310
|
def metadata_for(event)
|
|
205
311
|
metadata = {
|
|
206
312
|
'source' => SOURCE_NAME,
|
|
@@ -232,11 +338,12 @@ module Pgoutput
|
|
|
232
338
|
hash.compact
|
|
233
339
|
end
|
|
234
340
|
|
|
235
|
-
def share(object)
|
|
236
|
-
|
|
341
|
+
def share(object, ractor: Ractor)
|
|
342
|
+
ractor.make_shareable(object)
|
|
237
343
|
rescue Ractor::Error
|
|
238
344
|
object
|
|
239
345
|
end
|
|
240
346
|
end
|
|
347
|
+
# rubocop:enable Metrics/ClassLength
|
|
241
348
|
end
|
|
242
349
|
end
|
|
@@ -0,0 +1,99 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Pgoutput
|
|
4
|
+
module SourceAdapter
|
|
5
|
+
# Resolves schema-aware replica keys from decoded row values.
|
|
6
|
+
#
|
|
7
|
+
# Relations may be keyed by pgoutput relation id, `[schema, table]`, or a
|
|
8
|
+
# `"schema.table"` string. Each value is the ordered list of replica-key
|
|
9
|
+
# columns. Relation-id mappings take precedence because they identify the
|
|
10
|
+
# exact relation metadata seen by the replication stream.
|
|
11
|
+
#
|
|
12
|
+
# @example Resolve a composite key
|
|
13
|
+
# resolver = ReplicaIdentityResolver.new(
|
|
14
|
+
# ["public", "memberships"] => ["tenant_id", "member_uuid"]
|
|
15
|
+
# )
|
|
16
|
+
#
|
|
17
|
+
# @api public
|
|
18
|
+
class ReplicaIdentityResolver
|
|
19
|
+
# Normalized relation keys mapped to ordered replica-identity columns.
|
|
20
|
+
#
|
|
21
|
+
# @return [Hash]
|
|
22
|
+
attr_reader :relations
|
|
23
|
+
|
|
24
|
+
# Extract a complete replica key from an ordered column list.
|
|
25
|
+
#
|
|
26
|
+
# @param event [Object] decoded row event with schema/table metadata
|
|
27
|
+
# @param values [Hash, nil] decoded row values
|
|
28
|
+
# @param columns [Array<String>] ordered replica-key column names
|
|
29
|
+
# @return [Hash{String=>Object}] complete replica key
|
|
30
|
+
# @raise [Error] when values or any configured column are unavailable
|
|
31
|
+
def self.extract(event, values, columns)
|
|
32
|
+
unless values.respond_to?(:key?)
|
|
33
|
+
raise Error, "replica identity values are unavailable for #{qualified_name(event)}"
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
columns.to_h do |column|
|
|
37
|
+
[column, fetch_column(event, values, column)]
|
|
38
|
+
end
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
# @param relations [Hash{Integer, Array<String>, Array<String>=>Array<String>}]
|
|
42
|
+
# relation identity to ordered replica-key column names
|
|
43
|
+
# @return [void]
|
|
44
|
+
def initialize(relations)
|
|
45
|
+
@relations = relations.to_h.transform_values { |columns| normalize_columns(columns) }.freeze
|
|
46
|
+
freeze
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
# Extract the configured replica key for a decoded row event.
|
|
50
|
+
#
|
|
51
|
+
# @param event [Object] decoded row event with relation/schema/table metadata
|
|
52
|
+
# @param values [Hash, nil] decoded row values
|
|
53
|
+
# @return [Hash{String=>Object}, nil] complete replica key, or `nil` when
|
|
54
|
+
# the relation is not configured
|
|
55
|
+
# @raise [Error] when a configured key column is absent from row values
|
|
56
|
+
def call(event, values)
|
|
57
|
+
columns = columns_for(event)
|
|
58
|
+
return nil unless columns
|
|
59
|
+
|
|
60
|
+
self.class.extract(event, values, columns)
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
def self.fetch_column(event, values, column)
|
|
64
|
+
return values[column] if values.key?(column)
|
|
65
|
+
|
|
66
|
+
symbol = column.to_sym
|
|
67
|
+
return values[symbol] if values.key?(symbol)
|
|
68
|
+
|
|
69
|
+
raise Error, "replica identity column #{column.inspect} is missing from #{qualified_name(event)} values"
|
|
70
|
+
end
|
|
71
|
+
private_class_method :fetch_column
|
|
72
|
+
|
|
73
|
+
def self.qualified_name(event)
|
|
74
|
+
[event.schema, event.table].compact.join('.')
|
|
75
|
+
end
|
|
76
|
+
private_class_method :qualified_name
|
|
77
|
+
|
|
78
|
+
private
|
|
79
|
+
|
|
80
|
+
def normalize_columns(columns)
|
|
81
|
+
column_list = Array(columns) # : Array[untyped]
|
|
82
|
+
names = column_list.map(&:to_s)
|
|
83
|
+
raise ArgumentError, 'replica identity columns must not be empty' if names.empty?
|
|
84
|
+
raise ArgumentError, 'replica identity columns must be unique' unless names.uniq.length == names.length
|
|
85
|
+
|
|
86
|
+
names.freeze
|
|
87
|
+
end
|
|
88
|
+
|
|
89
|
+
def columns_for(event)
|
|
90
|
+
relation_id = event.relation_id if event.respond_to?(:relation_id)
|
|
91
|
+
return relations[relation_id] if relation_id && relations.key?(relation_id)
|
|
92
|
+
|
|
93
|
+
schema = event.schema
|
|
94
|
+
table = event.table
|
|
95
|
+
relations[[schema.to_s, table.to_s]] || relations["#{schema}.#{table}"]
|
|
96
|
+
end
|
|
97
|
+
end
|
|
98
|
+
end
|
|
99
|
+
end
|
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
module Pgoutput
|
|
2
2
|
module SourceAdapter
|
|
3
|
+
def self.ensure_core_source_adapter!: (untyped core) -> nil
|
|
4
|
+
|
|
3
5
|
# Normalizes Pgoutput::Decoder::Events into CDC::Core primitives.
|
|
4
6
|
#
|
|
5
7
|
# This adapter is intentionally located under the Pgoutput namespace because
|
|
@@ -22,6 +24,13 @@ module Pgoutput
|
|
|
22
24
|
|
|
23
25
|
SOURCE_NAME: "pgoutput"
|
|
24
26
|
|
|
27
|
+
class StreamEvent
|
|
28
|
+
attr_reader event: untyped
|
|
29
|
+
attr_reader source_position: untyped
|
|
30
|
+
|
|
31
|
+
def initialize: (untyped event, untyped source_position) -> void
|
|
32
|
+
end
|
|
33
|
+
|
|
25
34
|
# @param primary_key_resolver [#call, nil] optional callable used to infer
|
|
26
35
|
# primary keys from decoded row values when pgoutput does not provide an
|
|
27
36
|
# old-key tuple.
|
|
@@ -37,7 +46,13 @@ module Pgoutput
|
|
|
37
46
|
#
|
|
38
47
|
# @param event [Object] a Pgoutput::Decoder::Events object.
|
|
39
48
|
# @return [CDC::Core::ChangeEvent, nil]
|
|
40
|
-
def normalize: (untyped event) -> untyped
|
|
49
|
+
def normalize: (untyped event, ?source_position: untyped) -> untyped
|
|
50
|
+
|
|
51
|
+
# Pair a decoded event with its transport-level source position.
|
|
52
|
+
def stream_event: (untyped event, ?source_position: untyped) -> StreamEvent
|
|
53
|
+
|
|
54
|
+
# Incrementally normalize an unbounded decoded event stream.
|
|
55
|
+
def each_normalized: (untyped events) ?{ (untyped work) -> untyped } -> (Enumerator[untyped, nil] | nil)
|
|
41
56
|
|
|
42
57
|
# Normalize a sequence of decoded pgoutput events.
|
|
43
58
|
#
|
|
@@ -55,13 +70,23 @@ module Pgoutput
|
|
|
55
70
|
|
|
56
71
|
attr_reader metadata_builder: untyped
|
|
57
72
|
|
|
58
|
-
def
|
|
73
|
+
def normalize_stream: (untyped events, flush_incomplete: bool) { (untyped work) -> untyped } -> void
|
|
59
74
|
|
|
60
|
-
def
|
|
75
|
+
def change_event: (untyped event, operation: untyped, old_values: untyped, new_values: untyped, primary_key: untyped, source_position: untyped) -> untyped
|
|
76
|
+
|
|
77
|
+
def transaction_envelope: (untyped event, transaction_id: untyped, events: untyped, metadata: untyped, ?source_position: untyped) -> untyped
|
|
61
78
|
|
|
62
79
|
def primary_key_for: (untyped event, untyped values) -> untyped
|
|
63
80
|
|
|
64
|
-
def
|
|
81
|
+
def stream_input: (untyped input) -> [untyped, untyped]
|
|
82
|
+
|
|
83
|
+
def default_primary_key: (untyped event, untyped values) -> (nil | untyped)
|
|
84
|
+
|
|
85
|
+
def replica_identity_columns_for: (untyped event) -> untyped
|
|
86
|
+
|
|
87
|
+
def normalize_key_columns: (untyped columns) -> Array[String]
|
|
88
|
+
|
|
89
|
+
def key_from_columns: (untyped event, untyped values, Array[String] columns) -> untyped
|
|
65
90
|
|
|
66
91
|
def metadata_for: (untyped event) -> untyped
|
|
67
92
|
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
module Pgoutput
|
|
2
|
+
module SourceAdapter
|
|
3
|
+
class ReplicaIdentityResolver
|
|
4
|
+
@relations: untyped
|
|
5
|
+
|
|
6
|
+
attr_reader relations: untyped
|
|
7
|
+
|
|
8
|
+
def self.extract: (untyped event, untyped values, Array[String] columns) -> Hash[String, untyped]
|
|
9
|
+
def initialize: (untyped relations) -> void
|
|
10
|
+
def call: (untyped event, untyped values) -> untyped
|
|
11
|
+
|
|
12
|
+
private
|
|
13
|
+
|
|
14
|
+
def self.fetch_column: (untyped event, untyped values, String column) -> untyped
|
|
15
|
+
def self.qualified_name: (untyped event) -> String
|
|
16
|
+
def normalize_columns: (untyped columns) -> Array[String]
|
|
17
|
+
def columns_for: (untyped event) -> untyped
|
|
18
|
+
end
|
|
19
|
+
end
|
|
20
|
+
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: pgoutput-source-adapter
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.3.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Ken C. Demanawa
|
|
@@ -64,11 +64,13 @@ files:
|
|
|
64
64
|
- lib/pgoutput/source/adapter.rb
|
|
65
65
|
- lib/pgoutput/source_adapter.rb
|
|
66
66
|
- lib/pgoutput/source_adapter/cdc.rb
|
|
67
|
+
- lib/pgoutput/source_adapter/replica_identity_resolver.rb
|
|
67
68
|
- lib/pgoutput/source_adapter/version.rb
|
|
68
69
|
- mise.toml
|
|
69
70
|
- sig/pgoutput/source/adapter.rbs
|
|
70
71
|
- sig/pgoutput/source_adapter.rbs
|
|
71
72
|
- sig/pgoutput/source_adapter/cdc.rbs
|
|
73
|
+
- sig/pgoutput/source_adapter/replica_identity_resolver.rbs
|
|
72
74
|
- sig/pgoutput/source_adapter/version.rbs
|
|
73
75
|
homepage: https://github.com/kanutocd/pgoutput-source-adapter
|
|
74
76
|
licenses:
|