pub_sub_model_sync 0.5.10 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.github/workflows/ruby.yml +1 -1
- data/CHANGELOG.md +13 -0
- data/Dockerfile +6 -0
- data/Gemfile.lock +2 -1
- data/README.md +182 -97
- data/docker-compose.yaml +12 -0
- data/docs/notifications-diagram.png +0 -0
- data/lib/pub_sub_model_sync/base.rb +16 -3
- data/lib/pub_sub_model_sync/config.rb +1 -1
- data/lib/pub_sub_model_sync/message_processor.rb +3 -1
- data/lib/pub_sub_model_sync/message_publisher.rb +85 -18
- data/lib/pub_sub_model_sync/mock_google_service.rb +4 -0
- data/lib/pub_sub_model_sync/mock_kafka_service.rb +13 -0
- data/lib/pub_sub_model_sync/payload.rb +16 -4
- data/lib/pub_sub_model_sync/publisher.rb +23 -12
- data/lib/pub_sub_model_sync/publisher_concern.rb +37 -20
- data/lib/pub_sub_model_sync/service_base.rb +13 -4
- data/lib/pub_sub_model_sync/service_google.rb +52 -17
- data/lib/pub_sub_model_sync/service_kafka.rb +35 -12
- data/lib/pub_sub_model_sync/service_rabbit.rb +40 -33
- data/lib/pub_sub_model_sync/subscriber.rb +13 -11
- data/lib/pub_sub_model_sync/subscriber_concern.rb +8 -5
- data/lib/pub_sub_model_sync/tasks/worker.rake +11 -0
- data/lib/pub_sub_model_sync/version.rb +1 -1
- metadata +5 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: c5b72a2c8c7d97a09c17b3985ce466ecf1fba62573dbb860fae9ccab064e84d9
|
4
|
+
data.tar.gz: 4f0f801bc9d51fee36b5f279c4f995d9bf441f608ea26d16314dd2182bf67dc7
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: bdb68283d5feca9b506a11478671e8deea6e425065ececefd2162a227b01a7be837fdd533fd5b51e58e60e3d438ffa8e89009f690e0b68696b06cc6e204f5c24
|
7
|
+
data.tar.gz: 99baadbbd02b889af06b4cd1b17847a61efe074d58c3f7cc6b0e3eb693f277604f5420b8924bca383cfc8cdbba550d8c16755231335e0a1e056663cb0a5f4deb
|
data/.github/workflows/ruby.yml
CHANGED
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,18 @@
|
|
1
1
|
# Change Log
|
2
2
|
|
3
|
+
# 0.6.0 (March 03, 2021)
|
4
|
+
- feat: add support to include custom payload headers
|
5
|
+
- feat: add pubsub transactions to process all payloads inside in the same order they were published
|
6
|
+
- feat: when a model is created/updated/destroyed, process all related payloads in a single transaction
|
7
|
+
- feat: add method to save processed payload (:ps_processed_payload) when saving sync
|
8
|
+
- feat: add "ordering_key" support to process all payloads with the same key in the same order
|
9
|
+
- feat: start multiple workers to process async kafka messages when starting service listeners
|
10
|
+
- feat: make async publisher by reusing exchange connection (rabbit)
|
11
|
+
- feat: add support for forced_ordering_key to always be used as the ordering_key if defined
|
12
|
+
- feat: add feature to publish a message to a custom and/or multiple topics
|
13
|
+
- feat: add model custom action subscriber and publisher
|
14
|
+
- feat: add docker compose settings
|
15
|
+
|
3
16
|
# 0.5.10 (February 13, 2021)
|
4
17
|
- feat: remove duplicated callback :ps_before_save_sync (same result can be achieved with :ps_before_save_sync)
|
5
18
|
- feat: improve message starter to retry when failed or exit system when persists
|
data/Dockerfile
ADDED
data/Gemfile.lock
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
pub_sub_model_sync (0.
|
4
|
+
pub_sub_model_sync (0.6.0)
|
5
5
|
rails
|
6
6
|
|
7
7
|
GEM
|
@@ -243,6 +243,7 @@ DEPENDENCIES
|
|
243
243
|
database_cleaner-active_record
|
244
244
|
google-cloud-pubsub (> 2.0)
|
245
245
|
pub_sub_model_sync!
|
246
|
+
rails (~> 6)
|
246
247
|
rake
|
247
248
|
rspec
|
248
249
|
rubocop (~> 1.6.0)
|
data/README.md
CHANGED
@@ -6,8 +6,9 @@ Note: This gem is based on [MultipleMan](https://github.com/influitive/multiple_
|
|
6
6
|
- [**PubSubModelSync**](#pubsubmodelsync)
|
7
7
|
- [**Features**](#features)
|
8
8
|
- [**Installation**](#installation)
|
9
|
-
- [**
|
10
|
-
- [**
|
9
|
+
- [**Configuration**](#configuration)
|
10
|
+
- [**Notifications Diagram**](#notifications-diagram)
|
11
|
+
- [**Basic Example**](#basic-example)
|
11
12
|
- [**Advanced Example**](#advanced-example)
|
12
13
|
- [**API**](#api)
|
13
14
|
- [**Subscribers**](#subscribers)
|
@@ -33,6 +34,10 @@ Note: This gem is based on [MultipleMan](https://github.com/influitive/multiple_
|
|
33
34
|
- Ability to make class level communication
|
34
35
|
Example: If User from App1 wants to generate_email, this can be listened on App2, App3, ... to make corresponding actions
|
35
36
|
- Change pub/sub service at any time
|
37
|
+
- Support for transactions: Permits to group all payloads with the same ordering_key and be processed in the same order they are published by the subscribers.
|
38
|
+
Grouping by ordering_key allows us to enable multiple workers in our Pub/Sub service(s), and still guarantee that related payloads will be processed in the correct order, despite of the multiple threads.
|
39
|
+
This thanks to the fact that Pub/Sub services will always send messages with the same `ordering_key` into the same worker/thread.
|
40
|
+
- Ability to send notifications to a specific topic or multiple topics
|
36
41
|
|
37
42
|
## **Installation**
|
38
43
|
Add this line to your application's Gemfile:
|
@@ -46,7 +51,7 @@ gem 'ruby-kafka' # to use apache kafka pub/sub service
|
|
46
51
|
And then execute: $ bundle install
|
47
52
|
|
48
53
|
|
49
|
-
## **
|
54
|
+
## **Configuration**
|
50
55
|
|
51
56
|
- Configuration for google pub/sub (You need google pub/sub service account)
|
52
57
|
```ruby
|
@@ -54,7 +59,8 @@ And then execute: $ bundle install
|
|
54
59
|
PubSubModelSync::Config.service_name = :google
|
55
60
|
PubSubModelSync::Config.project = 'google-project-id'
|
56
61
|
PubSubModelSync::Config.credentials = 'path-to-the-config'
|
57
|
-
PubSubModelSync::Config.topic_name = 'sample-topic'
|
62
|
+
PubSubModelSync::Config.topic_name = 'sample-topic'
|
63
|
+
PubSubModelSync::Config.subscription_name = 'my-app1'
|
58
64
|
```
|
59
65
|
See details here:
|
60
66
|
https://github.com/googleapis/google-cloud-ruby/tree/master/google-cloud-pubsub
|
@@ -63,8 +69,8 @@ And then execute: $ bundle install
|
|
63
69
|
```ruby
|
64
70
|
PubSubModelSync::Config.service_name = :rabbitmq
|
65
71
|
PubSubModelSync::Config.bunny_connection = 'amqp://guest:guest@localhost'
|
66
|
-
PubSubModelSync::Config.queue_name = 'model-sync'
|
67
72
|
PubSubModelSync::Config.topic_name = 'sample-topic'
|
73
|
+
PubSubModelSync::Config.subscription_name = 'my-app2'
|
68
74
|
```
|
69
75
|
See details here: https://github.com/ruby-amqp/bunny
|
70
76
|
|
@@ -73,27 +79,25 @@ And then execute: $ bundle install
|
|
73
79
|
PubSubModelSync::Config.service_name = :kafka
|
74
80
|
PubSubModelSync::Config.kafka_connection = [["kafka1:9092", "localhost:2121"], { logger: Rails.logger }]
|
75
81
|
PubSubModelSync::Config.topic_name = 'sample-topic'
|
82
|
+
PubSubModelSync::Config.subscription_name = 'my-app3'
|
76
83
|
```
|
77
84
|
See details here: https://github.com/zendesk/ruby-kafka
|
78
85
|
|
79
86
|
- Add publishers/subscribers to your models (See examples below)
|
80
87
|
|
81
88
|
- Start subscribers to listen for publishers (Only in the app that has subscribers)
|
82
|
-
```
|
83
|
-
rake pub_sub_model_sync:start
|
84
|
-
```
|
85
|
-
Note: Publishers do not need todo this
|
86
|
-
Note2 (Rails 6+): Due to Zeitwerk, you need to load listeners manually when syncing without mentioned task (like rails console)
|
87
|
-
```ruby
|
88
|
-
# PubSubModelSync::Config.subscribers ==> []
|
89
|
-
PubSubModelSync::Runner.preload_listeners
|
90
|
-
# PubSubModelSync::Config.subscribers ==> [#<PubSubModelSync::Subscriber:0x000.. @klass="Article", @action=:create..., ....]
|
89
|
+
```bash
|
90
|
+
DB_POOL=20 bundle exec rake pub_sub_model_sync:start
|
91
91
|
```
|
92
|
+
Note: You need more than 15 DB pools to avoid "could not obtain a connection from the pool within 5.000 seconds". https://devcenter.heroku.com/articles/concurrency-and-database-connections
|
92
93
|
|
93
94
|
- Check the service status with:
|
94
95
|
```PubSubModelSync::MessagePublisher.publish_data('Test message', {sample_value: 10}, :create)```
|
95
96
|
|
96
|
-
## **
|
97
|
+
## **Notifications Diagram**
|
98
|
+

|
99
|
+
|
100
|
+
## **Basic Example**
|
97
101
|
```ruby
|
98
102
|
# App 1 (Publisher)
|
99
103
|
# attributes: name email age
|
@@ -105,20 +109,25 @@ end
|
|
105
109
|
# App 2 (Subscriber)
|
106
110
|
class User < ActiveRecord::Base
|
107
111
|
include PubSubModelSync::SubscriberConcern
|
108
|
-
ps_subscribe(%i[name])
|
109
|
-
|
112
|
+
ps_subscribe(%i[name]) # crud notifications
|
113
|
+
ps_subscribe_custom(:say_welcome) # custom instance notification
|
114
|
+
ps_class_subscribe(:greeting) # class notification
|
110
115
|
|
111
116
|
def self.greeting(data)
|
112
117
|
puts 'Class message called'
|
113
118
|
end
|
119
|
+
|
120
|
+
def say_welcome(data)
|
121
|
+
UserMailer.deliver(id, data)
|
122
|
+
end
|
114
123
|
end
|
115
124
|
|
116
125
|
# Samples
|
117
126
|
User.create(name: 'test user', email: 'sample@gmail.com') # Review your App 2 to see the created user (only name will be saved)
|
118
127
|
User.new(name: 'test user').ps_perform_sync(:create) # similar to above to perform sync on demand
|
119
128
|
|
120
|
-
|
121
|
-
PubSubModelSync::MessagePublisher.publish_data(User, { msg: 'Hello' }, :greeting) #
|
129
|
+
PubSubModelSync::MessagePublisher.publish_model_data(my_user, { id:10, msg: 'Hello' }, :say_welcome, { as_klass: 'RegisteredUser' }) # custom model action notification
|
130
|
+
PubSubModelSync::MessagePublisher.publish_data(User, { msg: 'Hello' }, :greeting) # custom data notification
|
122
131
|
```
|
123
132
|
|
124
133
|
## **Advanced Example**
|
@@ -127,7 +136,7 @@ PubSubModelSync::MessagePublisher.publish_data(User, { msg: 'Hello' }, :greeting
|
|
127
136
|
class User < ActiveRecord::Base
|
128
137
|
self.table_name = 'publisher_users'
|
129
138
|
include PubSubModelSync::PublisherConcern
|
130
|
-
ps_publish(%i[id:client_id name:full_name email], actions: %i[update], as_klass: 'Client')
|
139
|
+
ps_publish(%i[id:client_id name:full_name email], actions: %i[update], as_klass: 'Client', headers: { topic_name: ['topic1', 'topic N'] })
|
131
140
|
|
132
141
|
def ps_skip_callback?(_action)
|
133
142
|
false # here logic with action to skip push message
|
@@ -144,11 +153,16 @@ class User < ActiveRecord::Base
|
|
144
153
|
include PubSubModelSync::SubscriberConcern
|
145
154
|
ps_subscribe(%i[name], actions: %i[update], from_klass: 'Client', id: %i[client_id email])
|
146
155
|
ps_class_subscribe(:greeting, from_action: :custom_greeting, from_klass: 'CustomUser')
|
156
|
+
ps_subscribe_custom(:send_welcome, from_klass: 'CustomUser', id: :id, from_action: :say_welcome)
|
147
157
|
alias_attribute :full_name, :name
|
148
158
|
|
149
159
|
def self.greeting(data)
|
150
160
|
puts 'Class message called through custom_greeting'
|
151
161
|
end
|
162
|
+
|
163
|
+
def send_welcome(data)
|
164
|
+
UserMailer.deliver(id, data)
|
165
|
+
end
|
152
166
|
|
153
167
|
# def self.ps_find_model(data)
|
154
168
|
# where(email: data[:email], ...).first_or_initialize
|
@@ -156,136 +170,207 @@ class User < ActiveRecord::Base
|
|
156
170
|
end
|
157
171
|
```
|
158
172
|
|
159
|
-
Note: Be careful with collision of names
|
160
|
-
```
|
161
|
-
# ps_publish %i[name_data:name name:key] # key will be replaced with name_data
|
162
|
-
ps_publish %i[name_data:name key_data:key] # use alias to avoid collision
|
163
|
-
```
|
164
|
-
|
165
173
|
## **API**
|
166
174
|
### **Subscribers**
|
167
175
|
|
168
|
-
#### **Registering
|
176
|
+
#### **Registering Subscriptions**
|
169
177
|
|
170
|
-
- Configure
|
178
|
+
- Configure class subscriptions
|
171
179
|
```ruby
|
172
180
|
class MyModel < ActiveRecord::Base
|
173
181
|
ps_class_subscribe(action_name, from_action: nil, from_klass: nil)
|
174
182
|
end
|
175
183
|
```
|
176
|
-
|
177
|
-
* `
|
184
|
+
When Class receives the corresponding notification, `action` method will be called on the Class. Like: `User.action(data)`
|
185
|
+
* `action_name`: (String|Sym/Optional) Action name
|
186
|
+
* `from_klass`: (String/Optional) Source class name (Default `model.class.name`)
|
187
|
+
* `from_action`: (Sym/Optional) Source method name. Default `action`
|
178
188
|
|
179
|
-
- Configure
|
189
|
+
- Configure CRUD subscriptions
|
180
190
|
```ruby
|
181
191
|
class MyModel < ActiveRecord::Base
|
182
192
|
ps_subscribe(attrs, from_klass: nil, actions: nil, id: nil)
|
183
193
|
end
|
184
194
|
```
|
195
|
+
When model receives the corresponding notification, `action` method will be called on the model. Like: `model.destroy`
|
185
196
|
* `attrs`: (Array/Required) Array of all attributes to be synced
|
186
|
-
* `from_klass`: (String/Optional) Source class name (
|
197
|
+
* `from_klass`: (String/Optional) Source class name (Default `model.class.name`)
|
187
198
|
* `actions`: (Array/Optional, default: create/update/destroy) permit to customize action names
|
188
199
|
* `id`: (Sym|Array/Optional, default: id) Attr identifier(s) to find the corresponding model
|
189
200
|
|
190
|
-
- Configure
|
201
|
+
- Configure custom model subscriptions
|
191
202
|
```ruby
|
192
203
|
class MyModel < ActiveRecord::Base
|
193
|
-
|
204
|
+
ps_subscribe_custom(action, from_klass: name, id: :id, from_action: nil)
|
194
205
|
end
|
195
206
|
```
|
196
|
-
|
197
|
-
|
207
|
+
When model receives the corresponding notification, `action` method will be called on the model. Like: `model.action(data)`
|
208
|
+
* `action`: (String/Required) Action name
|
209
|
+
* `from_klass`: (String/Optional) Source class name (Default `model.class.name`)
|
210
|
+
* `from_action`: (Sym/Optional) Source method name. Default `action`
|
211
|
+
* `id`: (Sym|Array/Optional, default: id) Attr identifier(s) to find the corresponding model
|
198
212
|
|
199
|
-
|
200
|
-
- Configure CRUD subscription for the class
|
213
|
+
- Perform custom actions before saving sync of the model (`:cancel` can be returned to skip sync)
|
201
214
|
```ruby
|
202
|
-
MyModel
|
215
|
+
class MyModel < ActiveRecord::Base
|
216
|
+
def ps_before_save_sync(action, payload)
|
217
|
+
# puts payload.data[:id]
|
218
|
+
end
|
219
|
+
end
|
203
220
|
```
|
204
|
-
|
221
|
+
|
222
|
+
- Configure a custom model finder (optional)
|
223
|
+
```ruby
|
224
|
+
class MyModel < ActiveRecord::Base
|
225
|
+
def ps_find_model(data)
|
226
|
+
where(custom_finder: data[:custom_value]).first_or_initialize
|
227
|
+
end
|
228
|
+
end
|
229
|
+
```
|
230
|
+
* `data`: (Hash) Data received from sync
|
231
|
+
Must return an existent or a new model object
|
205
232
|
|
206
|
-
|
233
|
+
#### **Subscription helpers**
|
234
|
+
- Inspect all configured subscriptions
|
207
235
|
```ruby
|
208
236
|
PubSubModelSync::Config.subscribers
|
209
237
|
```
|
210
|
-
|
211
|
-
#### **Instance Methods**
|
212
|
-
|
213
|
-
- Perform custom actions before saving sync of the model (On-demand, `:cancel` can be returned to skip sync)
|
238
|
+
- Manually process or reprocess a notification
|
214
239
|
```ruby
|
215
|
-
|
240
|
+
payload = PubSubModelSync::Payload.new(data, attributes, headers)
|
241
|
+
payload.process!
|
216
242
|
```
|
217
243
|
|
244
|
+
|
218
245
|
### **Publishers**
|
219
246
|
|
220
|
-
#### **Registering
|
221
|
-
-
|
247
|
+
#### **Registering Publishers **
|
248
|
+
- Register CRUD publishers that will trigger configured notifications
|
222
249
|
```ruby
|
223
|
-
|
250
|
+
class MyModel < ActiveRecord::Base
|
251
|
+
ps_publish([:id, 'created_at:published_at', :full_name], actions: [:update], as_klass: nil, headers: { ordering_key: 'custom-key', topic_name: 'my-custom-topic' })
|
252
|
+
def full_name
|
253
|
+
[first_name, last_name].join(' ')
|
254
|
+
end
|
255
|
+
end
|
224
256
|
```
|
225
|
-
* `attrs`: (Array/Required) Array of attributes to be published
|
226
|
-
|
257
|
+
* `attrs`: (Array/Required) Array of attributes to be published. Supports for:
|
258
|
+
- aliases: permits to publish with different names, sample: "created_at:published_at" where "created_at" will be published as "published_at"
|
259
|
+
- methods: permits to publish method values as attributes, sample: "full_name"
|
260
|
+
* `actions`: (Array/Optional, default: %i[create update destroy]) permit to define action names
|
227
261
|
* `as_klass`: (String/Optional) Output class name (Instead of the model class name, will use this value)
|
262
|
+
* `headers`: (Hash/Optional) Notification settings which permit to customize the way and the target of the notification (Refer Payload.headers)
|
263
|
+
|
228
264
|
|
265
|
+
#### **Publishing notifications**
|
266
|
+
- CRUD notifications
|
267
|
+
```ruby
|
268
|
+
MyModel.create!(...)
|
269
|
+
```
|
270
|
+
"Create" notification will be delivered with the configured attributes as the payload data
|
229
271
|
|
230
|
-
|
272
|
+
- Manual CRUD notifications
|
273
|
+
```ruby
|
274
|
+
MyModel.ps_perform_sync(action, custom_data: {}, custom_headers: {})
|
275
|
+
```
|
276
|
+
* `action`: (Sym) CRUD action name (create, update or destroy)
|
277
|
+
* `custom_data`: custom_data (nil|Hash) If present custom_data will be used as the payload data. I.E. data generator will be ignored
|
278
|
+
* `custom_headers`: (Hash, optional) override default headers. Refer `payload.headers`
|
279
|
+
|
280
|
+
- Class notifications
|
281
|
+
```ruby
|
282
|
+
PubSubModelSync::MessagePublisher.publish_data((klass, data, action, headers: )
|
283
|
+
```
|
284
|
+
Publishes any data to be listened at a class level.
|
285
|
+
- `klass`: (String) Class name to be used
|
286
|
+
- `data`: (Hash) Data to be delivered
|
287
|
+
- `action`: (Sym) Action name
|
288
|
+
- `headers`: (Hash, optional) Notification settings (Refer Payload.headers)
|
289
|
+
|
290
|
+
- Model custom action notifications
|
291
|
+
```ruby
|
292
|
+
PubSubModelSync::MessagePublisher.publish_model_data(model, data, action, as_klass:, headers:)
|
293
|
+
```
|
294
|
+
Publishes model custom action to be listened at an instance level.
|
295
|
+
- `model`: (ActiveRecord) model owner of the data
|
296
|
+
- `data`: (Hash) Data to be delivered
|
297
|
+
- `action`: (Sym) Action name
|
298
|
+
- `as_klass`: (String, optional) if not provided, `model.class.name` will be used instead
|
299
|
+
- `headers`: (Hash, optional) Notification settings (Refer Payload.headers)
|
300
|
+
|
301
|
+
- Manually publish or republish a notification
|
302
|
+
```ruby
|
303
|
+
payload = PubSubModelSync::Payload.new(data, attributes, headers)
|
304
|
+
payload.publish!
|
305
|
+
```
|
306
|
+
|
307
|
+
#### ** publishing callbacks**
|
231
308
|
|
232
|
-
-
|
309
|
+
- Prevent CRUD sync at model callback level (Called right after :after_create, :after_update, :after_destroy).
|
310
|
+
If returns "true", sync will be cancelled.
|
233
311
|
```ruby
|
234
|
-
|
312
|
+
class MyModel < ActiveRecord::Base
|
313
|
+
def ps_skip_callback?(action)
|
314
|
+
# logic here
|
315
|
+
end
|
316
|
+
end
|
235
317
|
```
|
236
|
-
Default: False
|
237
|
-
Note: Return true to cancel sync
|
238
318
|
|
239
|
-
-
|
319
|
+
- Prevent CRUD sync before processing payload (Affects model.ps_perform_sync(...))).
|
320
|
+
If returns "true", sync will be cancelled
|
240
321
|
```ruby
|
241
|
-
|
322
|
+
class MyModel < ActiveRecord::Base
|
323
|
+
def ps_skip_sync?(action)
|
324
|
+
# logic here
|
325
|
+
end
|
326
|
+
end
|
242
327
|
```
|
243
|
-
Note: return true to cancel sync
|
244
328
|
|
245
|
-
-
|
329
|
+
- Do some actions before publishing a CRUD notification.
|
330
|
+
If returns ":cancel", sync will be cancelled
|
246
331
|
```ruby
|
247
|
-
|
332
|
+
class MyModel < ActiveRecord::Base
|
333
|
+
def ps_before_sync(action, payload)
|
334
|
+
# logic here
|
335
|
+
end
|
336
|
+
end
|
248
337
|
```
|
249
|
-
Note: If the method returns ```:cancel```, the sync will be stopped (message will not be published)
|
250
338
|
|
251
|
-
-
|
339
|
+
- Do some actions after CRUD notification was published.
|
252
340
|
```ruby
|
253
|
-
|
341
|
+
class MyModel < ActiveRecord::Base
|
342
|
+
def ps_after_sync(action, payload)
|
343
|
+
# logic here
|
344
|
+
end
|
345
|
+
end
|
254
346
|
```
|
255
347
|
|
256
|
-
- **Trigger a sync on-demand** (:create, :update, :destroy):
|
257
|
-
The target model will receive a notification to perform the indicated action
|
258
|
-
```ruby
|
259
|
-
model_instance.ps_perform_sync(action_name, custom_settings = {})
|
260
|
-
```
|
261
|
-
* `custom_settings`: override default settings defined for action_name ({ attrs: [], as_klass: nil })
|
262
348
|
|
263
|
-
|
349
|
+
### **Payload**
|
350
|
+
Any notification before delivering is transformed as a Payload for a better portability.
|
264
351
|
|
265
|
-
-
|
352
|
+
- Initialize
|
266
353
|
```ruby
|
267
|
-
|
354
|
+
payload = PubSubModelSync::Payload.new(data, attributes, headers)
|
268
355
|
```
|
269
|
-
|
270
|
-
* `
|
271
|
-
|
272
|
-
|
273
|
-
|
274
|
-
|
356
|
+
* `data`: (Hash) Data to be published or processed
|
357
|
+
* `attributes`: (Hash) Includes class and method info
|
358
|
+
- `action`: (String) action name
|
359
|
+
- `klass`: (String) class name
|
360
|
+
* `headers`: (Hash) Notification settings that defines how the notification will be processed or delivered.
|
361
|
+
- `key`: (String, optional) identifier of the payload, default: `<klass_name>/<action>` when class message, `<model.class.name>/<action>/<model.id>` when model message (Useful for caching techniques).
|
362
|
+
- `ordering_key`: (String, optional): messages with the same key are processed in the same order they were delivered, default: `klass_name` when class message, `<model.class.name>/<model.id>` when model message
|
363
|
+
- `topic_name`: (String|Array<String>, optional): Specific topic name to be used when delivering the message (default first topic from config).
|
364
|
+
- `forced_ordering_key`: (String, optional): Will force to use this value as the `ordering_key`, even withing transactions. Default `nil`.
|
365
|
+
|
366
|
+
- Actions for payloads
|
275
367
|
```ruby
|
276
|
-
payload = PubSubModelSync::Payload.new({ title: 'hello' }, { action: :greeting, klass: 'User' })
|
277
368
|
payload.publish! # publishes notification data. It raises exception if fails and does not call ```:on_error_publishing``` callback
|
278
369
|
payload.publish # publishes notification data. On error does not raise exception but calls ```:on_error_publishing``` callback
|
279
370
|
payload.process! # process a notification data. It raises exception if fails and does not call ```.on_error_processing``` callback
|
280
371
|
payload.publish # process a notification data. It does not raise exception if fails but calls ```.on_error_processing``` callback
|
281
372
|
```
|
282
373
|
|
283
|
-
- Get crud publisher configured for the class
|
284
|
-
```ruby
|
285
|
-
User.ps_publisher(action_name)
|
286
|
-
```
|
287
|
-
* `action_name` (default :create, :sym): can be :create, :update, :destroy
|
288
|
-
|
289
374
|
## **Testing with RSpec**
|
290
375
|
- Config: (spec/rails_helper.rb)
|
291
376
|
```ruby
|
@@ -341,7 +426,7 @@ Note: Be careful with collision of names
|
|
341
426
|
publisher = PubSubModelSync::MessagePublisher
|
342
427
|
data = {msg: 'hello'}
|
343
428
|
action = :greeting
|
344
|
-
|
429
|
+
PubSubModelSync::MessagePublisher.publish_data('User', data, action)
|
345
430
|
expect(publisher).to receive(:publish_data).with('User', data, action)
|
346
431
|
end
|
347
432
|
```
|
@@ -351,9 +436,12 @@ Note: Be careful with collision of names
|
|
351
436
|
config = PubSubModelSync::Config
|
352
437
|
config.debug = true
|
353
438
|
```
|
354
|
-
|
355
|
-
|
356
|
-
|
439
|
+
- `.topic_name = ['topic1', 'topic 2']`: (String|Array<String>)
|
440
|
+
Topic name(s) to be used to listen all notifications from when listening. Additional first topic name is used as the default topic name when publishing a notification.
|
441
|
+
- `.subscription_name = "my-app-1"`: (String, default Rails.application.name)
|
442
|
+
Subscriber's identifier which helps to:
|
443
|
+
* skip self messages
|
444
|
+
* continue the sync from the last synced notification when service was restarted.
|
357
445
|
- ```.debug = true```
|
358
446
|
(true/false*) => show advanced log messages
|
359
447
|
- ```.logger = Rails.logger```
|
@@ -378,18 +466,15 @@ config.debug = true
|
|
378
466
|
- Add flag ```model.ps_process_payload``` to retrieve the payload used to process the pub/sub sync
|
379
467
|
- Auto publish update only if payload has changed
|
380
468
|
- On delete, payload must only be composed by ids
|
381
|
-
-
|
382
|
-
PubSubModelSync::MessagePublisher.
|
383
|
-
- Add DB table to use as a shield to
|
469
|
+
- Improve transactions to exclude similar messages by klass and action. Sample:
|
470
|
+
```PubSubModelSync::MessagePublisher.transaction(key, { same_keys: :use_last_as_first|:use_last|:use_first_as_last|:keep*, same_data: :use_last_as_first*|:use_last|:use_first_as_last|:keep })```
|
471
|
+
- Add DB table to use as a shield to prevent publishing similar notifications and publish partial notifications (similar idea when processing notif)
|
384
472
|
- add callback: on_message_received(payload)
|
385
473
|
|
386
474
|
## **Q&A**
|
387
475
|
- I'm getting error "could not obtain a connection from the pool within 5.000 seconds"... what does this mean?
|
388
476
|
This problem occurs because pub/sub dependencies (kafka, google-pubsub, rabbitmq) use many threads to perform notifications where the qty of threads is greater than qty of DB pools ([Google pubsub info](https://github.com/googleapis/google-cloud-ruby/blob/master/google-cloud-pubsub/lib/google/cloud/pubsub/subscription.rb#L888))
|
389
|
-
To fix the problem, edit config/database.yml and increase the quantity of ```pool:
|
390
|
-
- Google pubsub: How to process notifications parallely and not sequentially (default 1 thread)?
|
391
|
-
```ruby PubSubModelSync::ServiceGoogle::LISTEN_SETTINGS = { threads: { callback: qty_threads } } ```
|
392
|
-
Note: by this way some notifications can be processed before others thus missing relationship errors can appear
|
477
|
+
To fix the problem, edit config/database.yml and increase the quantity of ```pool: 20```
|
393
478
|
- How to retry failed syncs with sidekiq?
|
394
479
|
```ruby
|
395
480
|
# lib/initializers/pub_sub_config.rb
|