table_sync 2.3.0 → 4.2.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (48) hide show
  1. checksums.yaml +4 -4
  2. data/.rubocop.yml +21 -0
  3. data/CHANGELOG.md +57 -0
  4. data/Gemfile.lock +85 -80
  5. data/README.md +4 -2
  6. data/docs/message_protocol.md +24 -0
  7. data/docs/notifications.md +45 -0
  8. data/docs/publishing.md +147 -0
  9. data/docs/receiving.md +341 -0
  10. data/lib/table_sync.rb +16 -31
  11. data/lib/table_sync/errors.rb +39 -23
  12. data/lib/table_sync/publishing.rb +11 -0
  13. data/lib/table_sync/{base_publisher.rb → publishing/base_publisher.rb} +1 -1
  14. data/lib/table_sync/{batch_publisher.rb → publishing/batch_publisher.rb} +4 -4
  15. data/lib/table_sync/{orm_adapter → publishing/orm_adapter}/active_record.rb +3 -7
  16. data/lib/table_sync/{orm_adapter → publishing/orm_adapter}/sequel.rb +2 -6
  17. data/lib/table_sync/{publisher.rb → publishing/publisher.rb} +4 -4
  18. data/lib/table_sync/receiving.rb +14 -0
  19. data/lib/table_sync/receiving/config.rb +218 -0
  20. data/lib/table_sync/receiving/config_decorator.rb +27 -0
  21. data/lib/table_sync/receiving/dsl.rb +28 -0
  22. data/lib/table_sync/receiving/handler.rb +136 -0
  23. data/lib/table_sync/{model → receiving/model}/active_record.rb +43 -36
  24. data/lib/table_sync/receiving/model/sequel.rb +83 -0
  25. data/lib/table_sync/utils.rb +9 -0
  26. data/lib/table_sync/utils/interface_checker.rb +103 -0
  27. data/lib/table_sync/utils/proc_array.rb +17 -0
  28. data/lib/table_sync/utils/proc_keywords_resolver.rb +46 -0
  29. data/lib/table_sync/version.rb +1 -1
  30. data/table_sync.gemspec +2 -1
  31. metadata +45 -33
  32. data/docs/development.md +0 -43
  33. data/docs/synopsis.md +0 -336
  34. data/lib/table_sync/config.rb +0 -105
  35. data/lib/table_sync/config/callback_registry.rb +0 -53
  36. data/lib/table_sync/config_decorator.rb +0 -38
  37. data/lib/table_sync/dsl.rb +0 -25
  38. data/lib/table_sync/event_actions.rb +0 -96
  39. data/lib/table_sync/event_actions/data_wrapper.rb +0 -7
  40. data/lib/table_sync/event_actions/data_wrapper/base.rb +0 -23
  41. data/lib/table_sync/event_actions/data_wrapper/destroy.rb +0 -19
  42. data/lib/table_sync/event_actions/data_wrapper/update.rb +0 -21
  43. data/lib/table_sync/model/sequel.rb +0 -88
  44. data/lib/table_sync/plugins.rb +0 -72
  45. data/lib/table_sync/plugins/abstract.rb +0 -55
  46. data/lib/table_sync/plugins/access_mixin.rb +0 -49
  47. data/lib/table_sync/plugins/registry.rb +0 -153
  48. data/lib/table_sync/receiving_handler.rb +0 -76
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: f0b124973e82ee0bf943d61fb64b4ef0eb2798d1bca0b9c6396a597b2bf00ca5
4
- data.tar.gz: d119902d368a1af85d954f586036b4f2e12b727f82bf4901b1f852b86e047627
3
+ metadata.gz: 1a5d57fed3ce9ac59ad811d1917a57b26ef48c79865250b5ffd441681ab94e94
4
+ data.tar.gz: 78790dfd46fd429e2b379ea3535cb61b42f753f793c4c13b5d82a74e41fdf2e1
5
5
  SHA512:
6
- metadata.gz: 653ffcb319c370378c8e861b0dd0c3468970dc10c8736a570c9f6fabbb1af05ade6a0506a744617bb9b9c2b07f218405ab17ecf969c9e7fd584ef96468d8e990
7
- data.tar.gz: 0d080baf2462565addf4cfada754a5ba1ef67772a5e155d3958a354b938e190a99eb2d961dc56c70f26fa903cdc1ee76d0f4761999a13caf05c7e93c351ac22c
6
+ metadata.gz: 7e314b1e227a0754be612963e560fee2823d231d988303f45fdb6a1f52f5113014d0d269688c7da10bad71a4ff020155bdc51c39d180d832687ab97a08581ab3
7
+ data.tar.gz: 0251e5fa4d4f5bf2687fb2f537e50dab97b25c317cac6c61c607e76ba0f81d970fe7498915f7e37b4121b2ba8768a09f98969d356e06b0f368bcee691a4434da
data/.rubocop.yml CHANGED
@@ -14,3 +14,24 @@ AllCops:
14
14
  Style/Alias:
15
15
  Enabled: true
16
16
  EnforcedStyle: prefer_alias_method
17
+
18
+ RSpec/ExampleLength:
19
+ Enabled: false
20
+
21
+ Lint/UnusedBlockArgument:
22
+ AllowUnusedKeywordArguments: true
23
+
24
+ Lint/UnusedMethodArgument:
25
+ AllowUnusedKeywordArguments: true
26
+
27
+ Style/AccessorGrouping:
28
+ Enabled: false
29
+
30
+ Style/NumericPredicate:
31
+ Enabled: false
32
+
33
+ Lint/MissingSuper:
34
+ Enabled: false
35
+
36
+ Style/MissingRespondToMissing:
37
+ Enabled: false
data/CHANGELOG.md CHANGED
@@ -1,6 +1,63 @@
1
1
  # Changelog
2
2
  All notable changes to this project will be documented in this file.
3
3
 
4
+ ## [4.2.0] - 2020-11-19
5
+
6
+ - No changes. Just stabilization release.
7
+
8
+ ## [4.1.2] - 2020-11-19
9
+ ### Fixed
10
+ - bug with sorting data in handler
11
+
12
+ ## [4.1.1] - 2020-11-06
13
+ ### Fixed
14
+ - dead locks in receiving module (see: `spec/receiving/handler_spec.rb#avoid dead locks`)
15
+
16
+ ## [4.1.0] - 2020-11-02
17
+ ### Changed
18
+ - move `TableSync::Instrument.notify` from models to the handler
19
+ - fire `TableSync::Instrument.notify` after commit insted of in transaction
20
+
21
+ ## [4.0.0] - 2020-10-23
22
+ ### Returned
23
+ - config inheritance
24
+
25
+ ### Removed
26
+ - TableSync::Plugins
27
+
28
+ ## [3.0.0] - 2020-09-05
29
+ ### Added
30
+ - option `except`
31
+ - `to_model` in receive method
32
+ - TableSync::Utils::InterfaceChecker
33
+
34
+ ### Changed
35
+ - .rubocop.yml
36
+ - documentation
37
+ - modules hierarchy (split receiving and publishing)
38
+ `TableSync::Publisher` -> `TableSync::Publishing::Publisher`
39
+ `TableSync::BatchPublisher` -> `TableSync::Publishing::BatchPublisher`
40
+ `TableSync::ReceivingHandler` -> `TableSync::Receiving::Handler`
41
+ - made data batches processing as native
42
+ - implemented callbacks as options
43
+ - implemented `wrap_receiving` as option
44
+ - type checking in options
45
+ - `before_commit on: event, &block` -> `before_update(&block)` or `before_destroy(&block)`
46
+ - `after_commit on: event, &block` -> `after_commit_on_update(&block)` or `after_commit_on_destroy(&block)`
47
+ - changed parameters in some options:
48
+ add `raw_data`
49
+ `current_row` -> `row`
50
+ ...
51
+ see documents for details
52
+
53
+ ### Removed
54
+ - TableSync::Config::CallbackRegistry
55
+ - TableSync::EventActions
56
+ - TableSync::EventActions::DataWrapper
57
+ - config option `on_destroy`
58
+ - config option `partitions`
59
+ - config option `first_sync_time_key`
60
+
4
61
  ## [2.3.0] - 2020-07-22
5
62
  ### Added
6
63
  - ruby 2.7 in Travis
data/Gemfile.lock CHANGED
@@ -1,64 +1,65 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- table_sync (2.2.0)
4
+ table_sync (4.2.0)
5
5
  memery
6
6
  rabbit_messaging (~> 0.3)
7
7
  rails
8
+ self_data
8
9
 
9
10
  GEM
10
11
  remote: https://rubygems.org/
11
12
  specs:
12
- actioncable (6.0.3.2)
13
- actionpack (= 6.0.3.2)
13
+ actioncable (6.0.3.4)
14
+ actionpack (= 6.0.3.4)
14
15
  nio4r (~> 2.0)
15
16
  websocket-driver (>= 0.6.1)
16
- actionmailbox (6.0.3.2)
17
- actionpack (= 6.0.3.2)
18
- activejob (= 6.0.3.2)
19
- activerecord (= 6.0.3.2)
20
- activestorage (= 6.0.3.2)
21
- activesupport (= 6.0.3.2)
17
+ actionmailbox (6.0.3.4)
18
+ actionpack (= 6.0.3.4)
19
+ activejob (= 6.0.3.4)
20
+ activerecord (= 6.0.3.4)
21
+ activestorage (= 6.0.3.4)
22
+ activesupport (= 6.0.3.4)
22
23
  mail (>= 2.7.1)
23
- actionmailer (6.0.3.2)
24
- actionpack (= 6.0.3.2)
25
- actionview (= 6.0.3.2)
26
- activejob (= 6.0.3.2)
24
+ actionmailer (6.0.3.4)
25
+ actionpack (= 6.0.3.4)
26
+ actionview (= 6.0.3.4)
27
+ activejob (= 6.0.3.4)
27
28
  mail (~> 2.5, >= 2.5.4)
28
29
  rails-dom-testing (~> 2.0)
29
- actionpack (6.0.3.2)
30
- actionview (= 6.0.3.2)
31
- activesupport (= 6.0.3.2)
30
+ actionpack (6.0.3.4)
31
+ actionview (= 6.0.3.4)
32
+ activesupport (= 6.0.3.4)
32
33
  rack (~> 2.0, >= 2.0.8)
33
34
  rack-test (>= 0.6.3)
34
35
  rails-dom-testing (~> 2.0)
35
36
  rails-html-sanitizer (~> 1.0, >= 1.2.0)
36
- actiontext (6.0.3.2)
37
- actionpack (= 6.0.3.2)
38
- activerecord (= 6.0.3.2)
39
- activestorage (= 6.0.3.2)
40
- activesupport (= 6.0.3.2)
37
+ actiontext (6.0.3.4)
38
+ actionpack (= 6.0.3.4)
39
+ activerecord (= 6.0.3.4)
40
+ activestorage (= 6.0.3.4)
41
+ activesupport (= 6.0.3.4)
41
42
  nokogiri (>= 1.8.5)
42
- actionview (6.0.3.2)
43
- activesupport (= 6.0.3.2)
43
+ actionview (6.0.3.4)
44
+ activesupport (= 6.0.3.4)
44
45
  builder (~> 3.1)
45
46
  erubi (~> 1.4)
46
47
  rails-dom-testing (~> 2.0)
47
48
  rails-html-sanitizer (~> 1.1, >= 1.2.0)
48
- activejob (6.0.3.2)
49
- activesupport (= 6.0.3.2)
49
+ activejob (6.0.3.4)
50
+ activesupport (= 6.0.3.4)
50
51
  globalid (>= 0.3.6)
51
- activemodel (6.0.3.2)
52
- activesupport (= 6.0.3.2)
53
- activerecord (6.0.3.2)
54
- activemodel (= 6.0.3.2)
55
- activesupport (= 6.0.3.2)
56
- activestorage (6.0.3.2)
57
- actionpack (= 6.0.3.2)
58
- activejob (= 6.0.3.2)
59
- activerecord (= 6.0.3.2)
52
+ activemodel (6.0.3.4)
53
+ activesupport (= 6.0.3.4)
54
+ activerecord (6.0.3.4)
55
+ activemodel (= 6.0.3.4)
56
+ activesupport (= 6.0.3.4)
57
+ activestorage (6.0.3.4)
58
+ actionpack (= 6.0.3.4)
59
+ activejob (= 6.0.3.4)
60
+ activerecord (= 6.0.3.4)
60
61
  marcel (~> 0.3.1)
61
- activesupport (6.0.3.2)
62
+ activesupport (6.0.3.4)
62
63
  concurrent-ruby (~> 1.0, >= 1.0.2)
63
64
  i18n (>= 0.7, < 2)
64
65
  minitest (~> 5.1)
@@ -70,10 +71,10 @@ GEM
70
71
  bundler-audit (0.7.0.1)
71
72
  bundler (>= 1.2.0, < 3)
72
73
  thor (>= 0.18, < 2)
73
- bunny (2.15.0)
74
+ bunny (2.17.0)
74
75
  amq-protocol (~> 2.3, >= 2.3.1)
75
76
  coderay (1.1.3)
76
- concurrent-ruby (1.1.6)
77
+ concurrent-ruby (1.1.7)
77
78
  coveralls (0.8.23)
78
79
  json (>= 1.8, < 3)
79
80
  simplecov (~> 0.16.1)
@@ -83,19 +84,18 @@ GEM
83
84
  crass (1.0.6)
84
85
  diff-lcs (1.4.4)
85
86
  docile (1.3.2)
86
- erubi (1.9.0)
87
+ erubi (1.10.0)
87
88
  exception_notification (4.4.3)
88
89
  actionmailer (>= 4.0, < 7)
89
90
  activesupport (>= 4.0, < 7)
90
91
  globalid (0.4.2)
91
92
  activesupport (>= 4.2.0)
92
- i18n (1.8.4)
93
+ i18n (1.8.5)
93
94
  concurrent-ruby (~> 1.0)
94
- jaro_winkler (1.5.4)
95
95
  json (2.3.1)
96
96
  lamian (1.2.0)
97
97
  rails (>= 4.2)
98
- loofah (2.6.0)
98
+ loofah (2.7.0)
99
99
  crass (~> 1.0.2)
100
100
  nokogiri (>= 1.5.9)
101
101
  mail (2.7.1)
@@ -108,55 +108,56 @@ GEM
108
108
  mimemagic (0.3.5)
109
109
  mini_mime (1.0.2)
110
110
  mini_portile2 (2.4.0)
111
- minitest (5.14.1)
112
- nio4r (2.5.2)
111
+ minitest (5.14.2)
112
+ nio4r (2.5.4)
113
113
  nokogiri (1.10.10)
114
114
  mini_portile2 (~> 2.4.0)
115
115
  parallel (1.19.2)
116
- parser (2.7.1.4)
116
+ parser (2.7.1.5)
117
117
  ast (~> 2.4.1)
118
118
  pg (0.21.0)
119
119
  pry (0.13.1)
120
120
  coderay (~> 1.1)
121
121
  method_source (~> 1.0)
122
- rabbit_messaging (0.7.1)
122
+ rabbit_messaging (0.9.0)
123
123
  bunny (~> 2.0)
124
124
  exception_notification
125
125
  lamian
126
- rails
126
+ rails (>= 5.2)
127
127
  sneakers (~> 2.0)
128
128
  tainbox
129
129
  rack (2.2.3)
130
130
  rack-test (1.1.0)
131
131
  rack (>= 1.0, < 3)
132
- rails (6.0.3.2)
133
- actioncable (= 6.0.3.2)
134
- actionmailbox (= 6.0.3.2)
135
- actionmailer (= 6.0.3.2)
136
- actionpack (= 6.0.3.2)
137
- actiontext (= 6.0.3.2)
138
- actionview (= 6.0.3.2)
139
- activejob (= 6.0.3.2)
140
- activemodel (= 6.0.3.2)
141
- activerecord (= 6.0.3.2)
142
- activestorage (= 6.0.3.2)
143
- activesupport (= 6.0.3.2)
132
+ rails (6.0.3.4)
133
+ actioncable (= 6.0.3.4)
134
+ actionmailbox (= 6.0.3.4)
135
+ actionmailer (= 6.0.3.4)
136
+ actionpack (= 6.0.3.4)
137
+ actiontext (= 6.0.3.4)
138
+ actionview (= 6.0.3.4)
139
+ activejob (= 6.0.3.4)
140
+ activemodel (= 6.0.3.4)
141
+ activerecord (= 6.0.3.4)
142
+ activestorage (= 6.0.3.4)
143
+ activesupport (= 6.0.3.4)
144
144
  bundler (>= 1.3.0)
145
- railties (= 6.0.3.2)
145
+ railties (= 6.0.3.4)
146
146
  sprockets-rails (>= 2.0.0)
147
147
  rails-dom-testing (2.0.3)
148
148
  activesupport (>= 4.2.0)
149
149
  nokogiri (>= 1.6)
150
150
  rails-html-sanitizer (1.3.0)
151
151
  loofah (~> 2.3)
152
- railties (6.0.3.2)
153
- actionpack (= 6.0.3.2)
154
- activesupport (= 6.0.3.2)
152
+ railties (6.0.3.4)
153
+ actionpack (= 6.0.3.4)
154
+ activesupport (= 6.0.3.4)
155
155
  method_source
156
156
  rake (>= 0.8.7)
157
157
  thor (>= 0.20.3, < 2.0)
158
158
  rainbow (3.0.0)
159
159
  rake (13.0.1)
160
+ regexp_parser (1.8.1)
160
161
  rexml (3.2.4)
161
162
  rspec (3.9.0)
162
163
  rspec-core (~> 3.9.0)
@@ -171,30 +172,34 @@ GEM
171
172
  diff-lcs (>= 1.2.0, < 2.0)
172
173
  rspec-support (~> 3.9.0)
173
174
  rspec-support (3.9.3)
174
- rubocop (0.81.0)
175
- jaro_winkler (~> 1.5.1)
175
+ rubocop (0.90.0)
176
176
  parallel (~> 1.10)
177
- parser (>= 2.7.0.1)
177
+ parser (>= 2.7.1.1)
178
178
  rainbow (>= 2.2.2, < 4.0)
179
+ regexp_parser (>= 1.7)
179
180
  rexml
181
+ rubocop-ast (>= 0.3.0, < 1.0)
180
182
  ruby-progressbar (~> 1.7)
181
183
  unicode-display_width (>= 1.4.0, < 2.0)
182
- rubocop-config-umbrellio (0.81.0.78)
183
- rubocop (= 0.81.0)
184
- rubocop-performance (= 1.5.2)
185
- rubocop-rails (= 2.5.0)
186
- rubocop-rspec (= 1.38.1)
187
- rubocop-performance (1.5.2)
188
- rubocop (>= 0.71.0)
189
- rubocop-rails (2.5.0)
190
- activesupport
184
+ rubocop-ast (0.7.1)
185
+ parser (>= 2.7.1.5)
186
+ rubocop-config-umbrellio (0.90.0.93)
187
+ rubocop (= 0.90.0)
188
+ rubocop-performance (= 1.7.1)
189
+ rubocop-rails (= 2.7.0)
190
+ rubocop-rspec (= 1.42.0)
191
+ rubocop-performance (1.7.1)
192
+ rubocop (>= 0.82.0)
193
+ rubocop-rails (2.7.0)
194
+ activesupport (>= 4.2.0)
191
195
  rack (>= 1.1)
192
- rubocop (>= 0.72.0)
193
- rubocop-rspec (1.38.1)
194
- rubocop (>= 0.68.1)
196
+ rubocop (>= 0.87.0)
197
+ rubocop-rspec (1.42.0)
198
+ rubocop (>= 0.87.0)
195
199
  ruby-progressbar (1.10.1)
196
200
  ruby2_keywords (0.0.2)
197
- sequel (5.34.0)
201
+ self_data (1.2.1)
202
+ sequel (5.37.0)
198
203
  serverengine (2.0.7)
199
204
  sigdump (~> 0.2.2)
200
205
  sigdump (0.2.4)
@@ -212,7 +217,7 @@ GEM
212
217
  sprockets (4.0.2)
213
218
  concurrent-ruby (~> 1.0)
214
219
  rack (> 1, < 3)
215
- sprockets-rails (3.2.1)
220
+ sprockets-rails (3.2.2)
216
221
  actionpack (>= 4.0)
217
222
  activesupport (>= 4.0)
218
223
  sprockets (>= 3.0.0)
@@ -247,7 +252,7 @@ DEPENDENCIES
247
252
  pry
248
253
  rake
249
254
  rspec (~> 3.8)
250
- rubocop-config-umbrellio (~> 0.81)
255
+ rubocop-config-umbrellio
251
256
  sequel
252
257
  simplecov (~> 0.16)
253
258
  table_sync!
data/README.md CHANGED
@@ -20,8 +20,10 @@ require 'table_sync'
20
20
 
21
21
  ## Usage
22
22
 
23
- - [Documentation](docs/synopsis.md)
24
- - [Development](docs/development.md)
23
+ - [Message protocol](docs/message_protocol.md)
24
+ - [Publising](docs/publishing.md)
25
+ - [Receiving](docs/receiving.md)
26
+ - [Notifications](docs/notifications.md)
25
27
 
26
28
  ## Contributing
27
29
 
data/docs/message_protocol.md ADDED
@@ -0,0 +1,24 @@
1
+ # Messages protocol
2
+
3
+ ```
4
+ {
5
+ project_id: "pid",
6
+ data: {
7
+ "event": "update",
8
+ "model": "User",
9
+ "version": 1.23,
10
+ "attributes": [
11
+ {
12
+ "id": "1",
13
+ "name": "user1",
14
+ "email": "user1@example.com"
15
+ },
16
+ {
17
+ "id": "2",
18
+ "name": "user2",
19
+ "email": "user2@example.com"
20
+ }
21
+ ]
22
+ }
23
+ }
24
+ ```
data/docs/notifications.md ADDED
@@ -0,0 +1,45 @@
1
+ ### Notifications
2
+
3
+ #### ActiveSupport adapter
4
+
5
+ You can use an already existing ActiveSupport adapter:
6
+ ```ruby
7
+ TableSync.notifier = TableSync::InstrumentAdapter::ActiveSupport
8
+ ```
9
+
10
+ This instrumentation API is provided by Active Support. It allows to subscribe to notifications:
11
+
12
+ ```ruby
13
+ ActiveSupport::Notifications.subscribe(/tablesync/) do |name, start, finish, id, payload|
14
+ # do something
15
+ end
16
+ ```
17
+
18
+ Types of events available:
19
+ `"tablesync.receive.update"`, `"tablesync.receive.destroy"`, `"tablesync.publish.update"`
20
+ and `"tablesync.publish.destroy"`.
21
+
22
+ You have access to the payload, which contains `event`, `direction`, `table`, `schema` and `count`.
23
+
24
+ ```
25
+ {
26
+ :event => :update, # one of update / destroy
27
+ :direction => :publish, # one of publish / receive
28
+ :table => "users",
29
+ :schema => "public",
30
+ :count => 1
31
+ }
32
+ ```
33
+
34
+ See more at https://guides.rubyonrails.org/active_support_instrumentation.html
35
+
36
+
37
+ #### Custom adapters
38
+
39
+ You can also create a custom adapter. It is expected to respond to the following method:
40
+
41
+ ```ruby
42
+ def notify(table:, event:, direction:, count:)
43
+ # processes data about table_sync event
44
+ end
45
+ ```