table_sync 2.3.0 → 4.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (48) hide show
  1. checksums.yaml +4 -4
  2. data/.rubocop.yml +21 -0
  3. data/CHANGELOG.md +57 -0
  4. data/Gemfile.lock +85 -80
  5. data/README.md +4 -2
  6. data/docs/message_protocol.md +24 -0
  7. data/docs/notifications.md +45 -0
  8. data/docs/publishing.md +147 -0
  9. data/docs/receiving.md +341 -0
  10. data/lib/table_sync.rb +16 -31
  11. data/lib/table_sync/errors.rb +39 -23
  12. data/lib/table_sync/publishing.rb +11 -0
  13. data/lib/table_sync/{base_publisher.rb → publishing/base_publisher.rb} +1 -1
  14. data/lib/table_sync/{batch_publisher.rb → publishing/batch_publisher.rb} +4 -4
  15. data/lib/table_sync/{orm_adapter → publishing/orm_adapter}/active_record.rb +3 -7
  16. data/lib/table_sync/{orm_adapter → publishing/orm_adapter}/sequel.rb +2 -6
  17. data/lib/table_sync/{publisher.rb → publishing/publisher.rb} +4 -4
  18. data/lib/table_sync/receiving.rb +14 -0
  19. data/lib/table_sync/receiving/config.rb +218 -0
  20. data/lib/table_sync/receiving/config_decorator.rb +27 -0
  21. data/lib/table_sync/receiving/dsl.rb +28 -0
  22. data/lib/table_sync/receiving/handler.rb +136 -0
  23. data/lib/table_sync/{model → receiving/model}/active_record.rb +43 -36
  24. data/lib/table_sync/receiving/model/sequel.rb +83 -0
  25. data/lib/table_sync/utils.rb +9 -0
  26. data/lib/table_sync/utils/interface_checker.rb +103 -0
  27. data/lib/table_sync/utils/proc_array.rb +17 -0
  28. data/lib/table_sync/utils/proc_keywords_resolver.rb +46 -0
  29. data/lib/table_sync/version.rb +1 -1
  30. data/table_sync.gemspec +2 -1
  31. metadata +45 -33
  32. data/docs/development.md +0 -43
  33. data/docs/synopsis.md +0 -336
  34. data/lib/table_sync/config.rb +0 -105
  35. data/lib/table_sync/config/callback_registry.rb +0 -53
  36. data/lib/table_sync/config_decorator.rb +0 -38
  37. data/lib/table_sync/dsl.rb +0 -25
  38. data/lib/table_sync/event_actions.rb +0 -96
  39. data/lib/table_sync/event_actions/data_wrapper.rb +0 -7
  40. data/lib/table_sync/event_actions/data_wrapper/base.rb +0 -23
  41. data/lib/table_sync/event_actions/data_wrapper/destroy.rb +0 -19
  42. data/lib/table_sync/event_actions/data_wrapper/update.rb +0 -21
  43. data/lib/table_sync/model/sequel.rb +0 -88
  44. data/lib/table_sync/plugins.rb +0 -72
  45. data/lib/table_sync/plugins/abstract.rb +0 -55
  46. data/lib/table_sync/plugins/access_mixin.rb +0 -49
  47. data/lib/table_sync/plugins/registry.rb +0 -153
  48. data/lib/table_sync/receiving_handler.rb +0 -76
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: f0b124973e82ee0bf943d61fb64b4ef0eb2798d1bca0b9c6396a597b2bf00ca5
4
- data.tar.gz: d119902d368a1af85d954f586036b4f2e12b727f82bf4901b1f852b86e047627
3
+ metadata.gz: 1a5d57fed3ce9ac59ad811d1917a57b26ef48c79865250b5ffd441681ab94e94
4
+ data.tar.gz: 78790dfd46fd429e2b379ea3535cb61b42f753f793c4c13b5d82a74e41fdf2e1
5
5
  SHA512:
6
- metadata.gz: 653ffcb319c370378c8e861b0dd0c3468970dc10c8736a570c9f6fabbb1af05ade6a0506a744617bb9b9c2b07f218405ab17ecf969c9e7fd584ef96468d8e990
7
- data.tar.gz: 0d080baf2462565addf4cfada754a5ba1ef67772a5e155d3958a354b938e190a99eb2d961dc56c70f26fa903cdc1ee76d0f4761999a13caf05c7e93c351ac22c
6
+ metadata.gz: 7e314b1e227a0754be612963e560fee2823d231d988303f45fdb6a1f52f5113014d0d269688c7da10bad71a4ff020155bdc51c39d180d832687ab97a08581ab3
7
+ data.tar.gz: 0251e5fa4d4f5bf2687fb2f537e50dab97b25c317cac6c61c607e76ba0f81d970fe7498915f7e37b4121b2ba8768a09f98969d356e06b0f368bcee691a4434da
data/.rubocop.yml CHANGED
@@ -14,3 +14,24 @@ AllCops:
14
14
  Style/Alias:
15
15
  Enabled: true
16
16
  EnforcedStyle: prefer_alias_method
17
+
18
+ RSpec/ExampleLength:
19
+ Enabled: false
20
+
21
+ Lint/UnusedBlockArgument:
22
+ AllowUnusedKeywordArguments: true
23
+
24
+ Lint/UnusedMethodArgument:
25
+ AllowUnusedKeywordArguments: true
26
+
27
+ Style/AccessorGrouping:
28
+ Enabled: false
29
+
30
+ Style/NumericPredicate:
31
+ Enabled: false
32
+
33
+ Lint/MissingSuper:
34
+ Enabled: false
35
+
36
+ Style/MissingRespondToMissing:
37
+ Enabled: false
data/CHANGELOG.md CHANGED
@@ -1,6 +1,63 @@
1
1
  # Changelog
2
2
  All notable changes to this project will be documented in this file.
3
3
 
4
+ ## [4.2.0] - 2020-11-19
5
+
6
+ - No changes. Just stabilization release.
7
+
8
+ ## [4.1.2] - 2020-11-19
9
+ ### Fixed
10
+ - bug with sorting data in handler
11
+
12
+ ## [4.1.1] - 2020-11-06
13
+ ### Fixed
14
+ - dead locks in receiving module (see: `spec/receiving/handler_spec.rb#avoid dead locks`)
15
+
16
+ ## [4.1.0] - 2020-11-02
17
+ ### Changed
18
+ - move `TableSync::Instrument.notify` from models to the handler
19
+ - fire `TableSync::Instrument.notify` after commit insted of in transaction
20
+
21
+ ## [4.0.0] - 2020-10-23
22
+ ### Returned
23
+ - config inheritance
24
+
25
+ ### Removed
26
+ - TableSync::Plugins
27
+
28
+ ## [3.0.0] - 2020-09-05
29
+ ### Added
30
+ - option `except`
31
+ - `to_model` in receive method
32
+ - TableSync::Utils::InterfaceChecker
33
+
34
+ ### Changed
35
+ - .rubocop.yml
36
+ - documentation
37
+ - modules hierarchy (split receiving and publishing)
38
+ `TableSync::Publisher` -> `TableSync::Publishing::Publisher`
39
+ `TableSync::BatchPublisher` -> `TableSync::Publishing::BatchPublisher`
40
+ `TableSync::ReceivingHandler` -> `TableSync::Receiving::Handler`
41
+ - made data batches processing as native
42
+ - implemented callbacks as options
43
+ - implemented `wrap_receiving` as option
44
+ - type checking in options
45
+ - `before_commit on: event, &block` -> `before_update(&block)` or `before_destroy(&block)`
46
+ - `after_commit on: event, &block` -> `after_commit_on_update(&block)` or `after_commit_on_destroy(&block)`
47
+ - changed parameters in some options:
48
+ add `raw_data`
49
+ `current_row` -> `row`
50
+ ...
51
+ see documents for details
52
+
53
+ ### Removed
54
+ - TableSync::Config::CallbackRegistry
55
+ - TableSync::EventActions
56
+ - TableSync::EventActions::DataWrapper
57
+ - config option `on_destroy`
58
+ - config option `partitions`
59
+ - config option `first_sync_time_key`
60
+
4
61
  ## [2.3.0] - 2020-07-22
5
62
  ### Added
6
63
  - ruby 2.7 in Travis
data/Gemfile.lock CHANGED
@@ -1,64 +1,65 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- table_sync (2.2.0)
4
+ table_sync (4.2.0)
5
5
  memery
6
6
  rabbit_messaging (~> 0.3)
7
7
  rails
8
+ self_data
8
9
 
9
10
  GEM
10
11
  remote: https://rubygems.org/
11
12
  specs:
12
- actioncable (6.0.3.2)
13
- actionpack (= 6.0.3.2)
13
+ actioncable (6.0.3.4)
14
+ actionpack (= 6.0.3.4)
14
15
  nio4r (~> 2.0)
15
16
  websocket-driver (>= 0.6.1)
16
- actionmailbox (6.0.3.2)
17
- actionpack (= 6.0.3.2)
18
- activejob (= 6.0.3.2)
19
- activerecord (= 6.0.3.2)
20
- activestorage (= 6.0.3.2)
21
- activesupport (= 6.0.3.2)
17
+ actionmailbox (6.0.3.4)
18
+ actionpack (= 6.0.3.4)
19
+ activejob (= 6.0.3.4)
20
+ activerecord (= 6.0.3.4)
21
+ activestorage (= 6.0.3.4)
22
+ activesupport (= 6.0.3.4)
22
23
  mail (>= 2.7.1)
23
- actionmailer (6.0.3.2)
24
- actionpack (= 6.0.3.2)
25
- actionview (= 6.0.3.2)
26
- activejob (= 6.0.3.2)
24
+ actionmailer (6.0.3.4)
25
+ actionpack (= 6.0.3.4)
26
+ actionview (= 6.0.3.4)
27
+ activejob (= 6.0.3.4)
27
28
  mail (~> 2.5, >= 2.5.4)
28
29
  rails-dom-testing (~> 2.0)
29
- actionpack (6.0.3.2)
30
- actionview (= 6.0.3.2)
31
- activesupport (= 6.0.3.2)
30
+ actionpack (6.0.3.4)
31
+ actionview (= 6.0.3.4)
32
+ activesupport (= 6.0.3.4)
32
33
  rack (~> 2.0, >= 2.0.8)
33
34
  rack-test (>= 0.6.3)
34
35
  rails-dom-testing (~> 2.0)
35
36
  rails-html-sanitizer (~> 1.0, >= 1.2.0)
36
- actiontext (6.0.3.2)
37
- actionpack (= 6.0.3.2)
38
- activerecord (= 6.0.3.2)
39
- activestorage (= 6.0.3.2)
40
- activesupport (= 6.0.3.2)
37
+ actiontext (6.0.3.4)
38
+ actionpack (= 6.0.3.4)
39
+ activerecord (= 6.0.3.4)
40
+ activestorage (= 6.0.3.4)
41
+ activesupport (= 6.0.3.4)
41
42
  nokogiri (>= 1.8.5)
42
- actionview (6.0.3.2)
43
- activesupport (= 6.0.3.2)
43
+ actionview (6.0.3.4)
44
+ activesupport (= 6.0.3.4)
44
45
  builder (~> 3.1)
45
46
  erubi (~> 1.4)
46
47
  rails-dom-testing (~> 2.0)
47
48
  rails-html-sanitizer (~> 1.1, >= 1.2.0)
48
- activejob (6.0.3.2)
49
- activesupport (= 6.0.3.2)
49
+ activejob (6.0.3.4)
50
+ activesupport (= 6.0.3.4)
50
51
  globalid (>= 0.3.6)
51
- activemodel (6.0.3.2)
52
- activesupport (= 6.0.3.2)
53
- activerecord (6.0.3.2)
54
- activemodel (= 6.0.3.2)
55
- activesupport (= 6.0.3.2)
56
- activestorage (6.0.3.2)
57
- actionpack (= 6.0.3.2)
58
- activejob (= 6.0.3.2)
59
- activerecord (= 6.0.3.2)
52
+ activemodel (6.0.3.4)
53
+ activesupport (= 6.0.3.4)
54
+ activerecord (6.0.3.4)
55
+ activemodel (= 6.0.3.4)
56
+ activesupport (= 6.0.3.4)
57
+ activestorage (6.0.3.4)
58
+ actionpack (= 6.0.3.4)
59
+ activejob (= 6.0.3.4)
60
+ activerecord (= 6.0.3.4)
60
61
  marcel (~> 0.3.1)
61
- activesupport (6.0.3.2)
62
+ activesupport (6.0.3.4)
62
63
  concurrent-ruby (~> 1.0, >= 1.0.2)
63
64
  i18n (>= 0.7, < 2)
64
65
  minitest (~> 5.1)
@@ -70,10 +71,10 @@ GEM
70
71
  bundler-audit (0.7.0.1)
71
72
  bundler (>= 1.2.0, < 3)
72
73
  thor (>= 0.18, < 2)
73
- bunny (2.15.0)
74
+ bunny (2.17.0)
74
75
  amq-protocol (~> 2.3, >= 2.3.1)
75
76
  coderay (1.1.3)
76
- concurrent-ruby (1.1.6)
77
+ concurrent-ruby (1.1.7)
77
78
  coveralls (0.8.23)
78
79
  json (>= 1.8, < 3)
79
80
  simplecov (~> 0.16.1)
@@ -83,19 +84,18 @@ GEM
83
84
  crass (1.0.6)
84
85
  diff-lcs (1.4.4)
85
86
  docile (1.3.2)
86
- erubi (1.9.0)
87
+ erubi (1.10.0)
87
88
  exception_notification (4.4.3)
88
89
  actionmailer (>= 4.0, < 7)
89
90
  activesupport (>= 4.0, < 7)
90
91
  globalid (0.4.2)
91
92
  activesupport (>= 4.2.0)
92
- i18n (1.8.4)
93
+ i18n (1.8.5)
93
94
  concurrent-ruby (~> 1.0)
94
- jaro_winkler (1.5.4)
95
95
  json (2.3.1)
96
96
  lamian (1.2.0)
97
97
  rails (>= 4.2)
98
- loofah (2.6.0)
98
+ loofah (2.7.0)
99
99
  crass (~> 1.0.2)
100
100
  nokogiri (>= 1.5.9)
101
101
  mail (2.7.1)
@@ -108,55 +108,56 @@ GEM
108
108
  mimemagic (0.3.5)
109
109
  mini_mime (1.0.2)
110
110
  mini_portile2 (2.4.0)
111
- minitest (5.14.1)
112
- nio4r (2.5.2)
111
+ minitest (5.14.2)
112
+ nio4r (2.5.4)
113
113
  nokogiri (1.10.10)
114
114
  mini_portile2 (~> 2.4.0)
115
115
  parallel (1.19.2)
116
- parser (2.7.1.4)
116
+ parser (2.7.1.5)
117
117
  ast (~> 2.4.1)
118
118
  pg (0.21.0)
119
119
  pry (0.13.1)
120
120
  coderay (~> 1.1)
121
121
  method_source (~> 1.0)
122
- rabbit_messaging (0.7.1)
122
+ rabbit_messaging (0.9.0)
123
123
  bunny (~> 2.0)
124
124
  exception_notification
125
125
  lamian
126
- rails
126
+ rails (>= 5.2)
127
127
  sneakers (~> 2.0)
128
128
  tainbox
129
129
  rack (2.2.3)
130
130
  rack-test (1.1.0)
131
131
  rack (>= 1.0, < 3)
132
- rails (6.0.3.2)
133
- actioncable (= 6.0.3.2)
134
- actionmailbox (= 6.0.3.2)
135
- actionmailer (= 6.0.3.2)
136
- actionpack (= 6.0.3.2)
137
- actiontext (= 6.0.3.2)
138
- actionview (= 6.0.3.2)
139
- activejob (= 6.0.3.2)
140
- activemodel (= 6.0.3.2)
141
- activerecord (= 6.0.3.2)
142
- activestorage (= 6.0.3.2)
143
- activesupport (= 6.0.3.2)
132
+ rails (6.0.3.4)
133
+ actioncable (= 6.0.3.4)
134
+ actionmailbox (= 6.0.3.4)
135
+ actionmailer (= 6.0.3.4)
136
+ actionpack (= 6.0.3.4)
137
+ actiontext (= 6.0.3.4)
138
+ actionview (= 6.0.3.4)
139
+ activejob (= 6.0.3.4)
140
+ activemodel (= 6.0.3.4)
141
+ activerecord (= 6.0.3.4)
142
+ activestorage (= 6.0.3.4)
143
+ activesupport (= 6.0.3.4)
144
144
  bundler (>= 1.3.0)
145
- railties (= 6.0.3.2)
145
+ railties (= 6.0.3.4)
146
146
  sprockets-rails (>= 2.0.0)
147
147
  rails-dom-testing (2.0.3)
148
148
  activesupport (>= 4.2.0)
149
149
  nokogiri (>= 1.6)
150
150
  rails-html-sanitizer (1.3.0)
151
151
  loofah (~> 2.3)
152
- railties (6.0.3.2)
153
- actionpack (= 6.0.3.2)
154
- activesupport (= 6.0.3.2)
152
+ railties (6.0.3.4)
153
+ actionpack (= 6.0.3.4)
154
+ activesupport (= 6.0.3.4)
155
155
  method_source
156
156
  rake (>= 0.8.7)
157
157
  thor (>= 0.20.3, < 2.0)
158
158
  rainbow (3.0.0)
159
159
  rake (13.0.1)
160
+ regexp_parser (1.8.1)
160
161
  rexml (3.2.4)
161
162
  rspec (3.9.0)
162
163
  rspec-core (~> 3.9.0)
@@ -171,30 +172,34 @@ GEM
171
172
  diff-lcs (>= 1.2.0, < 2.0)
172
173
  rspec-support (~> 3.9.0)
173
174
  rspec-support (3.9.3)
174
- rubocop (0.81.0)
175
- jaro_winkler (~> 1.5.1)
175
+ rubocop (0.90.0)
176
176
  parallel (~> 1.10)
177
- parser (>= 2.7.0.1)
177
+ parser (>= 2.7.1.1)
178
178
  rainbow (>= 2.2.2, < 4.0)
179
+ regexp_parser (>= 1.7)
179
180
  rexml
181
+ rubocop-ast (>= 0.3.0, < 1.0)
180
182
  ruby-progressbar (~> 1.7)
181
183
  unicode-display_width (>= 1.4.0, < 2.0)
182
- rubocop-config-umbrellio (0.81.0.78)
183
- rubocop (= 0.81.0)
184
- rubocop-performance (= 1.5.2)
185
- rubocop-rails (= 2.5.0)
186
- rubocop-rspec (= 1.38.1)
187
- rubocop-performance (1.5.2)
188
- rubocop (>= 0.71.0)
189
- rubocop-rails (2.5.0)
190
- activesupport
184
+ rubocop-ast (0.7.1)
185
+ parser (>= 2.7.1.5)
186
+ rubocop-config-umbrellio (0.90.0.93)
187
+ rubocop (= 0.90.0)
188
+ rubocop-performance (= 1.7.1)
189
+ rubocop-rails (= 2.7.0)
190
+ rubocop-rspec (= 1.42.0)
191
+ rubocop-performance (1.7.1)
192
+ rubocop (>= 0.82.0)
193
+ rubocop-rails (2.7.0)
194
+ activesupport (>= 4.2.0)
191
195
  rack (>= 1.1)
192
- rubocop (>= 0.72.0)
193
- rubocop-rspec (1.38.1)
194
- rubocop (>= 0.68.1)
196
+ rubocop (>= 0.87.0)
197
+ rubocop-rspec (1.42.0)
198
+ rubocop (>= 0.87.0)
195
199
  ruby-progressbar (1.10.1)
196
200
  ruby2_keywords (0.0.2)
197
- sequel (5.34.0)
201
+ self_data (1.2.1)
202
+ sequel (5.37.0)
198
203
  serverengine (2.0.7)
199
204
  sigdump (~> 0.2.2)
200
205
  sigdump (0.2.4)
@@ -212,7 +217,7 @@ GEM
212
217
  sprockets (4.0.2)
213
218
  concurrent-ruby (~> 1.0)
214
219
  rack (> 1, < 3)
215
- sprockets-rails (3.2.1)
220
+ sprockets-rails (3.2.2)
216
221
  actionpack (>= 4.0)
217
222
  activesupport (>= 4.0)
218
223
  sprockets (>= 3.0.0)
@@ -247,7 +252,7 @@ DEPENDENCIES
247
252
  pry
248
253
  rake
249
254
  rspec (~> 3.8)
250
- rubocop-config-umbrellio (~> 0.81)
255
+ rubocop-config-umbrellio
251
256
  sequel
252
257
  simplecov (~> 0.16)
253
258
  table_sync!
data/README.md CHANGED
@@ -20,8 +20,10 @@ require 'table_sync'
20
20
 
21
21
  ## Usage
22
22
 
23
- - [Documentation](docs/synopsis.md)
24
- - [Development](docs/development.md)
23
+ - [Message protocol](docs/message_protocol.md)
24
+ - [Publising](docs/publishing.md)
25
+ - [Receiving](docs/receiving.md)
26
+ - [Notifications](docs/notifications.md)
25
27
 
26
28
  ## Contributing
27
29
 
data/docs/message_protocol.md ADDED
@@ -0,0 +1,24 @@
1
+ # Messages protocol
2
+
3
+ ```
4
+ {
5
+ project_id: "pid",
6
+ data: {
7
+ "event": "update",
8
+ "model": "User",
9
+ "version": 1.23,
10
+ "attributes": [
11
+ {
12
+ "id": "1",
13
+ "name": "user1",
14
+ "email": "user1@example.com"
15
+ },
16
+ {
17
+ "id": "2",
18
+ "name": "user2",
19
+ "email": "user2@example.com"
20
+ }
21
+ ]
22
+ }
23
+ }
24
+ ```
data/docs/notifications.md ADDED
@@ -0,0 +1,45 @@
1
+ ### Notifications
2
+
3
+ #### ActiveSupport adapter
4
+
5
+ You can use an already existing ActiveSupport adapter:
6
+ ```ruby
7
+ TableSync.notifier = TableSync::InstrumentAdapter::ActiveSupport
8
+ ```
9
+
10
+ This instrumentation API is provided by Active Support. It allows to subscribe to notifications:
11
+
12
+ ```ruby
13
+ ActiveSupport::Notifications.subscribe(/tablesync/) do |name, start, finish, id, payload|
14
+ # do something
15
+ end
16
+ ```
17
+
18
+ Types of events available:
19
+ `"tablesync.receive.update"`, `"tablesync.receive.destroy"`, `"tablesync.publish.update"`
20
+ and `"tablesync.publish.destroy"`.
21
+
22
+ You have access to the payload, which contains `event`, `direction`, `table`, `schema` and `count`.
23
+
24
+ ```
25
+ {
26
+ :event => :update, # one of update / destroy
27
+ :direction => :publish, # one of publish / receive
28
+ :table => "users",
29
+ :schema => "public",
30
+ :count => 1
31
+ }
32
+ ```
33
+
34
+ See more at https://guides.rubyonrails.org/active_support_instrumentation.html
35
+
36
+
37
+ #### Custom adapters
38
+
39
+ You can also create a custom adapter. It is expected to respond to the following method:
40
+
41
+ ```ruby
42
+ def notify(table:, event:, direction:, count:)
43
+ # processes data about table_sync event
44
+ end
45
+ ```