pub_sub_model_sync 0.5.10 → 0.6.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.github/workflows/ruby.yml +1 -1
- data/CHANGELOG.md +13 -0
- data/Dockerfile +6 -0
- data/Gemfile.lock +2 -1
- data/README.md +182 -97
- data/docker-compose.yaml +12 -0
- data/docs/notifications-diagram.png +0 -0
- data/lib/pub_sub_model_sync/base.rb +16 -3
- data/lib/pub_sub_model_sync/config.rb +1 -1
- data/lib/pub_sub_model_sync/message_processor.rb +3 -1
- data/lib/pub_sub_model_sync/message_publisher.rb +85 -18
- data/lib/pub_sub_model_sync/mock_google_service.rb +4 -0
- data/lib/pub_sub_model_sync/mock_kafka_service.rb +13 -0
- data/lib/pub_sub_model_sync/payload.rb +16 -4
- data/lib/pub_sub_model_sync/publisher.rb +23 -12
- data/lib/pub_sub_model_sync/publisher_concern.rb +37 -20
- data/lib/pub_sub_model_sync/service_base.rb +13 -4
- data/lib/pub_sub_model_sync/service_google.rb +52 -17
- data/lib/pub_sub_model_sync/service_kafka.rb +35 -12
- data/lib/pub_sub_model_sync/service_rabbit.rb +40 -33
- data/lib/pub_sub_model_sync/subscriber.rb +13 -11
- data/lib/pub_sub_model_sync/subscriber_concern.rb +8 -5
- data/lib/pub_sub_model_sync/tasks/worker.rake +11 -0
- data/lib/pub_sub_model_sync/version.rb +1 -1
- metadata +5 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: c5b72a2c8c7d97a09c17b3985ce466ecf1fba62573dbb860fae9ccab064e84d9
|
4
|
+
data.tar.gz: 4f0f801bc9d51fee36b5f279c4f995d9bf441f608ea26d16314dd2182bf67dc7
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: bdb68283d5feca9b506a11478671e8deea6e425065ececefd2162a227b01a7be837fdd533fd5b51e58e60e3d438ffa8e89009f690e0b68696b06cc6e204f5c24
|
7
|
+
data.tar.gz: 99baadbbd02b889af06b4cd1b17847a61efe074d58c3f7cc6b0e3eb693f277604f5420b8924bca383cfc8cdbba550d8c16755231335e0a1e056663cb0a5f4deb
|
data/.github/workflows/ruby.yml
CHANGED
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,18 @@
|
|
1
1
|
# Change Log
|
2
2
|
|
3
|
+
# 0.6.0 (March 03, 2021)
|
4
|
+
- feat: add support to include custom payload headers
|
5
|
+
- feat: add pubsub transactions to process all payloads inside in the same order they were published
|
6
|
+
- feat: when a model is created/updated/destroyed, process all related payloads in a single transaction
|
7
|
+
- feat: add method to save processed payload (:ps_processed_payload) when saving sync
|
8
|
+
- feat: add "ordering_key" support to process all payloads with the same key in the same order
|
9
|
+
- feat: start multiple workers to process async kafka messages when starting service listeners
|
10
|
+
- feat: make async publisher by reusing exchange connection (rabbit)
|
11
|
+
- feat: add support for forced_ordering_key to always be used as the ordering_key if defined
|
12
|
+
- feat: add feature to publish a message to a custom and/or multiple topics
|
13
|
+
- feat: add model custom action subscriber and publisher
|
14
|
+
- feat: add docker compose settings
|
15
|
+
|
3
16
|
# 0.5.10 (February 13, 2021)
|
4
17
|
- feat: remove duplicated callback :ps_before_save_sync (same result can be achieved with :ps_before_save_sync)
|
5
18
|
- feat: improve message starter to retry when failed or exit system when persists
|
data/Dockerfile
ADDED
data/Gemfile.lock
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
pub_sub_model_sync (0.
|
4
|
+
pub_sub_model_sync (0.6.0)
|
5
5
|
rails
|
6
6
|
|
7
7
|
GEM
|
@@ -243,6 +243,7 @@ DEPENDENCIES
|
|
243
243
|
database_cleaner-active_record
|
244
244
|
google-cloud-pubsub (> 2.0)
|
245
245
|
pub_sub_model_sync!
|
246
|
+
rails (~> 6)
|
246
247
|
rake
|
247
248
|
rspec
|
248
249
|
rubocop (~> 1.6.0)
|
data/README.md
CHANGED
@@ -6,8 +6,9 @@ Note: This gem is based on [MultipleMan](https://github.com/influitive/multiple_
|
|
6
6
|
- [**PubSubModelSync**](#pubsubmodelsync)
|
7
7
|
- [**Features**](#features)
|
8
8
|
- [**Installation**](#installation)
|
9
|
-
- [**
|
10
|
-
- [**
|
9
|
+
- [**Configuration**](#configuration)
|
10
|
+
- [**Notifications Diagram**](#notifications-diagram)
|
11
|
+
- [**Basic Example**](#basic-example)
|
11
12
|
- [**Advanced Example**](#advanced-example)
|
12
13
|
- [**API**](#api)
|
13
14
|
- [**Subscribers**](#subscribers)
|
@@ -33,6 +34,10 @@ Note: This gem is based on [MultipleMan](https://github.com/influitive/multiple_
|
|
33
34
|
- Ability to make class level communication
|
34
35
|
Example: If User from App1 wants to generate_email, this can be listened on App2, App3, ... to make corresponding actions
|
35
36
|
- Change pub/sub service at any time
|
37
|
+
- Support for transactions: Permits to group all payloads with the same ordering_key and be processed in the same order they are published by the subscribers.
|
38
|
+
Grouping by ordering_key allows us to enable multiple workers in our Pub/Sub service(s), and still guarantee that related payloads will be processed in the correct order, despite of the multiple threads.
|
39
|
+
This thanks to the fact that Pub/Sub services will always send messages with the same `ordering_key` into the same worker/thread.
|
40
|
+
- Ability to send notifications to a specific topic or multiple topics
|
36
41
|
|
37
42
|
## **Installation**
|
38
43
|
Add this line to your application's Gemfile:
|
@@ -46,7 +51,7 @@ gem 'ruby-kafka' # to use apache kafka pub/sub service
|
|
46
51
|
And then execute: $ bundle install
|
47
52
|
|
48
53
|
|
49
|
-
## **
|
54
|
+
## **Configuration**
|
50
55
|
|
51
56
|
- Configuration for google pub/sub (You need google pub/sub service account)
|
52
57
|
```ruby
|
@@ -54,7 +59,8 @@ And then execute: $ bundle install
|
|
54
59
|
PubSubModelSync::Config.service_name = :google
|
55
60
|
PubSubModelSync::Config.project = 'google-project-id'
|
56
61
|
PubSubModelSync::Config.credentials = 'path-to-the-config'
|
57
|
-
PubSubModelSync::Config.topic_name = 'sample-topic'
|
62
|
+
PubSubModelSync::Config.topic_name = 'sample-topic'
|
63
|
+
PubSubModelSync::Config.subscription_name = 'my-app1'
|
58
64
|
```
|
59
65
|
See details here:
|
60
66
|
https://github.com/googleapis/google-cloud-ruby/tree/master/google-cloud-pubsub
|
@@ -63,8 +69,8 @@ And then execute: $ bundle install
|
|
63
69
|
```ruby
|
64
70
|
PubSubModelSync::Config.service_name = :rabbitmq
|
65
71
|
PubSubModelSync::Config.bunny_connection = 'amqp://guest:guest@localhost'
|
66
|
-
PubSubModelSync::Config.queue_name = 'model-sync'
|
67
72
|
PubSubModelSync::Config.topic_name = 'sample-topic'
|
73
|
+
PubSubModelSync::Config.subscription_name = 'my-app2'
|
68
74
|
```
|
69
75
|
See details here: https://github.com/ruby-amqp/bunny
|
70
76
|
|
@@ -73,27 +79,25 @@ And then execute: $ bundle install
|
|
73
79
|
PubSubModelSync::Config.service_name = :kafka
|
74
80
|
PubSubModelSync::Config.kafka_connection = [["kafka1:9092", "localhost:2121"], { logger: Rails.logger }]
|
75
81
|
PubSubModelSync::Config.topic_name = 'sample-topic'
|
82
|
+
PubSubModelSync::Config.subscription_name = 'my-app3'
|
76
83
|
```
|
77
84
|
See details here: https://github.com/zendesk/ruby-kafka
|
78
85
|
|
79
86
|
- Add publishers/subscribers to your models (See examples below)
|
80
87
|
|
81
88
|
- Start subscribers to listen for publishers (Only in the app that has subscribers)
|
82
|
-
```
|
83
|
-
rake pub_sub_model_sync:start
|
84
|
-
```
|
85
|
-
Note: Publishers do not need todo this
|
86
|
-
Note2 (Rails 6+): Due to Zeitwerk, you need to load listeners manually when syncing without mentioned task (like rails console)
|
87
|
-
```ruby
|
88
|
-
# PubSubModelSync::Config.subscribers ==> []
|
89
|
-
PubSubModelSync::Runner.preload_listeners
|
90
|
-
# PubSubModelSync::Config.subscribers ==> [#<PubSubModelSync::Subscriber:0x000.. @klass="Article", @action=:create..., ....]
|
89
|
+
```bash
|
90
|
+
DB_POOL=20 bundle exec rake pub_sub_model_sync:start
|
91
91
|
```
|
92
|
+
Note: You need more than 15 DB pools to avoid "could not obtain a connection from the pool within 5.000 seconds". https://devcenter.heroku.com/articles/concurrency-and-database-connections
|
92
93
|
|
93
94
|
- Check the service status with:
|
94
95
|
```PubSubModelSync::MessagePublisher.publish_data('Test message', {sample_value: 10}, :create)```
|
95
96
|
|
96
|
-
## **
|
97
|
+
## **Notifications Diagram**
|
98
|
+
![Diagram](/docs/notifications-diagram.png?raw=true)
|
99
|
+
|
100
|
+
## **Basic Example**
|
97
101
|
```ruby
|
98
102
|
# App 1 (Publisher)
|
99
103
|
# attributes: name email age
|
@@ -105,20 +109,25 @@ end
|
|
105
109
|
# App 2 (Subscriber)
|
106
110
|
class User < ActiveRecord::Base
|
107
111
|
include PubSubModelSync::SubscriberConcern
|
108
|
-
ps_subscribe(%i[name])
|
109
|
-
|
112
|
+
ps_subscribe(%i[name]) # crud notifications
|
113
|
+
ps_subscribe_custom(:say_welcome) # custom instance notification
|
114
|
+
ps_class_subscribe(:greeting) # class notification
|
110
115
|
|
111
116
|
def self.greeting(data)
|
112
117
|
puts 'Class message called'
|
113
118
|
end
|
119
|
+
|
120
|
+
def say_welcome(data)
|
121
|
+
UserMailer.deliver(id, data)
|
122
|
+
end
|
114
123
|
end
|
115
124
|
|
116
125
|
# Samples
|
117
126
|
User.create(name: 'test user', email: 'sample@gmail.com') # Review your App 2 to see the created user (only name will be saved)
|
118
127
|
User.new(name: 'test user').ps_perform_sync(:create) # similar to above to perform sync on demand
|
119
128
|
|
120
|
-
|
121
|
-
PubSubModelSync::MessagePublisher.publish_data(User, { msg: 'Hello' }, :greeting) #
|
129
|
+
PubSubModelSync::MessagePublisher.publish_model_data(my_user, { id:10, msg: 'Hello' }, :say_welcome, { as_klass: 'RegisteredUser' }) # custom model action notification
|
130
|
+
PubSubModelSync::MessagePublisher.publish_data(User, { msg: 'Hello' }, :greeting) # custom data notification
|
122
131
|
```
|
123
132
|
|
124
133
|
## **Advanced Example**
|
@@ -127,7 +136,7 @@ PubSubModelSync::MessagePublisher.publish_data(User, { msg: 'Hello' }, :greeting
|
|
127
136
|
class User < ActiveRecord::Base
|
128
137
|
self.table_name = 'publisher_users'
|
129
138
|
include PubSubModelSync::PublisherConcern
|
130
|
-
ps_publish(%i[id:client_id name:full_name email], actions: %i[update], as_klass: 'Client')
|
139
|
+
ps_publish(%i[id:client_id name:full_name email], actions: %i[update], as_klass: 'Client', headers: { topic_name: ['topic1', 'topic N'] })
|
131
140
|
|
132
141
|
def ps_skip_callback?(_action)
|
133
142
|
false # here logic with action to skip push message
|
@@ -144,11 +153,16 @@ class User < ActiveRecord::Base
|
|
144
153
|
include PubSubModelSync::SubscriberConcern
|
145
154
|
ps_subscribe(%i[name], actions: %i[update], from_klass: 'Client', id: %i[client_id email])
|
146
155
|
ps_class_subscribe(:greeting, from_action: :custom_greeting, from_klass: 'CustomUser')
|
156
|
+
ps_subscribe_custom(:send_welcome, from_klass: 'CustomUser', id: :id, from_action: :say_welcome)
|
147
157
|
alias_attribute :full_name, :name
|
148
158
|
|
149
159
|
def self.greeting(data)
|
150
160
|
puts 'Class message called through custom_greeting'
|
151
161
|
end
|
162
|
+
|
163
|
+
def send_welcome(data)
|
164
|
+
UserMailer.deliver(id, data)
|
165
|
+
end
|
152
166
|
|
153
167
|
# def self.ps_find_model(data)
|
154
168
|
# where(email: data[:email], ...).first_or_initialize
|
@@ -156,136 +170,207 @@ class User < ActiveRecord::Base
|
|
156
170
|
end
|
157
171
|
```
|
158
172
|
|
159
|
-
Note: Be careful with collision of names
|
160
|
-
```
|
161
|
-
# ps_publish %i[name_data:name name:key] # key will be replaced with name_data
|
162
|
-
ps_publish %i[name_data:name key_data:key] # use alias to avoid collision
|
163
|
-
```
|
164
|
-
|
165
173
|
## **API**
|
166
174
|
### **Subscribers**
|
167
175
|
|
168
|
-
#### **Registering
|
176
|
+
#### **Registering Subscriptions**
|
169
177
|
|
170
|
-
- Configure
|
178
|
+
- Configure class subscriptions
|
171
179
|
```ruby
|
172
180
|
class MyModel < ActiveRecord::Base
|
173
181
|
ps_class_subscribe(action_name, from_action: nil, from_klass: nil)
|
174
182
|
end
|
175
183
|
```
|
176
|
-
|
177
|
-
* `
|
184
|
+
When Class receives the corresponding notification, `action` method will be called on the Class. Like: `User.action(data)`
|
185
|
+
* `action_name`: (String|Sym/Optional) Action name
|
186
|
+
* `from_klass`: (String/Optional) Source class name (Default `model.class.name`)
|
187
|
+
* `from_action`: (Sym/Optional) Source method name. Default `action`
|
178
188
|
|
179
|
-
- Configure
|
189
|
+
- Configure CRUD subscriptions
|
180
190
|
```ruby
|
181
191
|
class MyModel < ActiveRecord::Base
|
182
192
|
ps_subscribe(attrs, from_klass: nil, actions: nil, id: nil)
|
183
193
|
end
|
184
194
|
```
|
195
|
+
When model receives the corresponding notification, `action` method will be called on the model. Like: `model.destroy`
|
185
196
|
* `attrs`: (Array/Required) Array of all attributes to be synced
|
186
|
-
* `from_klass`: (String/Optional) Source class name (
|
197
|
+
* `from_klass`: (String/Optional) Source class name (Default `model.class.name`)
|
187
198
|
* `actions`: (Array/Optional, default: create/update/destroy) permit to customize action names
|
188
199
|
* `id`: (Sym|Array/Optional, default: id) Attr identifier(s) to find the corresponding model
|
189
200
|
|
190
|
-
- Configure
|
201
|
+
- Configure custom model subscriptions
|
191
202
|
```ruby
|
192
203
|
class MyModel < ActiveRecord::Base
|
193
|
-
|
204
|
+
ps_subscribe_custom(action, from_klass: name, id: :id, from_action: nil)
|
194
205
|
end
|
195
206
|
```
|
196
|
-
|
197
|
-
|
207
|
+
When model receives the corresponding notification, `action` method will be called on the model. Like: `model.action(data)`
|
208
|
+
* `action`: (String/Required) Action name
|
209
|
+
* `from_klass`: (String/Optional) Source class name (Default `model.class.name`)
|
210
|
+
* `from_action`: (Sym/Optional) Source method name. Default `action`
|
211
|
+
* `id`: (Sym|Array/Optional, default: id) Attr identifier(s) to find the corresponding model
|
198
212
|
|
199
|
-
|
200
|
-
- Configure CRUD subscription for the class
|
213
|
+
- Perform custom actions before saving sync of the model (`:cancel` can be returned to skip sync)
|
201
214
|
```ruby
|
202
|
-
MyModel
|
215
|
+
class MyModel < ActiveRecord::Base
|
216
|
+
def ps_before_save_sync(action, payload)
|
217
|
+
# puts payload.data[:id]
|
218
|
+
end
|
219
|
+
end
|
203
220
|
```
|
204
|
-
|
221
|
+
|
222
|
+
- Configure a custom model finder (optional)
|
223
|
+
```ruby
|
224
|
+
class MyModel < ActiveRecord::Base
|
225
|
+
def ps_find_model(data)
|
226
|
+
where(custom_finder: data[:custom_value]).first_or_initialize
|
227
|
+
end
|
228
|
+
end
|
229
|
+
```
|
230
|
+
* `data`: (Hash) Data received from sync
|
231
|
+
Must return an existent or a new model object
|
205
232
|
|
206
|
-
|
233
|
+
#### **Subscription helpers**
|
234
|
+
- Inspect all configured subscriptions
|
207
235
|
```ruby
|
208
236
|
PubSubModelSync::Config.subscribers
|
209
237
|
```
|
210
|
-
|
211
|
-
#### **Instance Methods**
|
212
|
-
|
213
|
-
- Perform custom actions before saving sync of the model (On-demand, `:cancel` can be returned to skip sync)
|
238
|
+
- Manually process or reprocess a notification
|
214
239
|
```ruby
|
215
|
-
|
240
|
+
payload = PubSubModelSync::Payload.new(data, attributes, headers)
|
241
|
+
payload.process!
|
216
242
|
```
|
217
243
|
|
244
|
+
|
218
245
|
### **Publishers**
|
219
246
|
|
220
|
-
#### **Registering
|
221
|
-
-
|
247
|
+
#### **Registering Publishers **
|
248
|
+
- Register CRUD publishers that will trigger configured notifications
|
222
249
|
```ruby
|
223
|
-
|
250
|
+
class MyModel < ActiveRecord::Base
|
251
|
+
ps_publish([:id, 'created_at:published_at', :full_name], actions: [:update], as_klass: nil, headers: { ordering_key: 'custom-key', topic_name: 'my-custom-topic' })
|
252
|
+
def full_name
|
253
|
+
[first_name, last_name].join(' ')
|
254
|
+
end
|
255
|
+
end
|
224
256
|
```
|
225
|
-
* `attrs`: (Array/Required) Array of attributes to be published
|
226
|
-
|
257
|
+
* `attrs`: (Array/Required) Array of attributes to be published. Supports for:
|
258
|
+
- aliases: permits to publish with different names, sample: "created_at:published_at" where "created_at" will be published as "published_at"
|
259
|
+
- methods: permits to publish method values as attributes, sample: "full_name"
|
260
|
+
* `actions`: (Array/Optional, default: %i[create update destroy]) permit to define action names
|
227
261
|
* `as_klass`: (String/Optional) Output class name (Instead of the model class name, will use this value)
|
262
|
+
* `headers`: (Hash/Optional) Notification settings which permit to customize the way and the target of the notification (Refer Payload.headers)
|
263
|
+
|
228
264
|
|
265
|
+
#### **Publishing notifications**
|
266
|
+
- CRUD notifications
|
267
|
+
```ruby
|
268
|
+
MyModel.create!(...)
|
269
|
+
```
|
270
|
+
"Create" notification will be delivered with the configured attributes as the payload data
|
229
271
|
|
230
|
-
|
272
|
+
- Manual CRUD notifications
|
273
|
+
```ruby
|
274
|
+
MyModel.ps_perform_sync(action, custom_data: {}, custom_headers: {})
|
275
|
+
```
|
276
|
+
* `action`: (Sym) CRUD action name (create, update or destroy)
|
277
|
+
* `custom_data`: custom_data (nil|Hash) If present custom_data will be used as the payload data. I.E. data generator will be ignored
|
278
|
+
* `custom_headers`: (Hash, optional) override default headers. Refer `payload.headers`
|
279
|
+
|
280
|
+
- Class notifications
|
281
|
+
```ruby
|
282
|
+
PubSubModelSync::MessagePublisher.publish_data((klass, data, action, headers: )
|
283
|
+
```
|
284
|
+
Publishes any data to be listened at a class level.
|
285
|
+
- `klass`: (String) Class name to be used
|
286
|
+
- `data`: (Hash) Data to be delivered
|
287
|
+
- `action`: (Sym) Action name
|
288
|
+
- `headers`: (Hash, optional) Notification settings (Refer Payload.headers)
|
289
|
+
|
290
|
+
- Model custom action notifications
|
291
|
+
```ruby
|
292
|
+
PubSubModelSync::MessagePublisher.publish_model_data(model, data, action, as_klass:, headers:)
|
293
|
+
```
|
294
|
+
Publishes model custom action to be listened at an instance level.
|
295
|
+
- `model`: (ActiveRecord) model owner of the data
|
296
|
+
- `data`: (Hash) Data to be delivered
|
297
|
+
- `action`: (Sym) Action name
|
298
|
+
- `as_klass`: (String, optional) if not provided, `model.class.name` will be used instead
|
299
|
+
- `headers`: (Hash, optional) Notification settings (Refer Payload.headers)
|
300
|
+
|
301
|
+
- Manually publish or republish a notification
|
302
|
+
```ruby
|
303
|
+
payload = PubSubModelSync::Payload.new(data, attributes, headers)
|
304
|
+
payload.publish!
|
305
|
+
```
|
306
|
+
|
307
|
+
#### ** publishing callbacks**
|
231
308
|
|
232
|
-
-
|
309
|
+
- Prevent CRUD sync at model callback level (Called right after :after_create, :after_update, :after_destroy).
|
310
|
+
If returns "true", sync will be cancelled.
|
233
311
|
```ruby
|
234
|
-
|
312
|
+
class MyModel < ActiveRecord::Base
|
313
|
+
def ps_skip_callback?(action)
|
314
|
+
# logic here
|
315
|
+
end
|
316
|
+
end
|
235
317
|
```
|
236
|
-
Default: False
|
237
|
-
Note: Return true to cancel sync
|
238
318
|
|
239
|
-
-
|
319
|
+
- Prevent CRUD sync before processing payload (Affects model.ps_perform_sync(...))).
|
320
|
+
If returns "true", sync will be cancelled
|
240
321
|
```ruby
|
241
|
-
|
322
|
+
class MyModel < ActiveRecord::Base
|
323
|
+
def ps_skip_sync?(action)
|
324
|
+
# logic here
|
325
|
+
end
|
326
|
+
end
|
242
327
|
```
|
243
|
-
Note: return true to cancel sync
|
244
328
|
|
245
|
-
-
|
329
|
+
- Do some actions before publishing a CRUD notification.
|
330
|
+
If returns ":cancel", sync will be cancelled
|
246
331
|
```ruby
|
247
|
-
|
332
|
+
class MyModel < ActiveRecord::Base
|
333
|
+
def ps_before_sync(action, payload)
|
334
|
+
# logic here
|
335
|
+
end
|
336
|
+
end
|
248
337
|
```
|
249
|
-
Note: If the method returns ```:cancel```, the sync will be stopped (message will not be published)
|
250
338
|
|
251
|
-
-
|
339
|
+
- Do some actions after CRUD notification was published.
|
252
340
|
```ruby
|
253
|
-
|
341
|
+
class MyModel < ActiveRecord::Base
|
342
|
+
def ps_after_sync(action, payload)
|
343
|
+
# logic here
|
344
|
+
end
|
345
|
+
end
|
254
346
|
```
|
255
347
|
|
256
|
-
- **Trigger a sync on-demand** (:create, :update, :destroy):
|
257
|
-
The target model will receive a notification to perform the indicated action
|
258
|
-
```ruby
|
259
|
-
model_instance.ps_perform_sync(action_name, custom_settings = {})
|
260
|
-
```
|
261
|
-
* `custom_settings`: override default settings defined for action_name ({ attrs: [], as_klass: nil })
|
262
348
|
|
263
|
-
|
349
|
+
### **Payload**
|
350
|
+
Any notification before delivering is transformed as a Payload for a better portability.
|
264
351
|
|
265
|
-
-
|
352
|
+
- Initialize
|
266
353
|
```ruby
|
267
|
-
|
354
|
+
payload = PubSubModelSync::Payload.new(data, attributes, headers)
|
268
355
|
```
|
269
|
-
|
270
|
-
* `
|
271
|
-
|
272
|
-
|
273
|
-
|
274
|
-
|
356
|
+
* `data`: (Hash) Data to be published or processed
|
357
|
+
* `attributes`: (Hash) Includes class and method info
|
358
|
+
- `action`: (String) action name
|
359
|
+
- `klass`: (String) class name
|
360
|
+
* `headers`: (Hash) Notification settings that defines how the notification will be processed or delivered.
|
361
|
+
- `key`: (String, optional) identifier of the payload, default: `<klass_name>/<action>` when class message, `<model.class.name>/<action>/<model.id>` when model message (Useful for caching techniques).
|
362
|
+
- `ordering_key`: (String, optional): messages with the same key are processed in the same order they were delivered, default: `klass_name` when class message, `<model.class.name>/<model.id>` when model message
|
363
|
+
- `topic_name`: (String|Array<String>, optional): Specific topic name to be used when delivering the message (default first topic from config).
|
364
|
+
- `forced_ordering_key`: (String, optional): Will force to use this value as the `ordering_key`, even withing transactions. Default `nil`.
|
365
|
+
|
366
|
+
- Actions for payloads
|
275
367
|
```ruby
|
276
|
-
payload = PubSubModelSync::Payload.new({ title: 'hello' }, { action: :greeting, klass: 'User' })
|
277
368
|
payload.publish! # publishes notification data. It raises exception if fails and does not call ```:on_error_publishing``` callback
|
278
369
|
payload.publish # publishes notification data. On error does not raise exception but calls ```:on_error_publishing``` callback
|
279
370
|
payload.process! # process a notification data. It raises exception if fails and does not call ```.on_error_processing``` callback
|
280
371
|
payload.publish # process a notification data. It does not raise exception if fails but calls ```.on_error_processing``` callback
|
281
372
|
```
|
282
373
|
|
283
|
-
- Get crud publisher configured for the class
|
284
|
-
```ruby
|
285
|
-
User.ps_publisher(action_name)
|
286
|
-
```
|
287
|
-
* `action_name` (default :create, :sym): can be :create, :update, :destroy
|
288
|
-
|
289
374
|
## **Testing with RSpec**
|
290
375
|
- Config: (spec/rails_helper.rb)
|
291
376
|
```ruby
|
@@ -341,7 +426,7 @@ Note: Be careful with collision of names
|
|
341
426
|
publisher = PubSubModelSync::MessagePublisher
|
342
427
|
data = {msg: 'hello'}
|
343
428
|
action = :greeting
|
344
|
-
|
429
|
+
PubSubModelSync::MessagePublisher.publish_data('User', data, action)
|
345
430
|
expect(publisher).to receive(:publish_data).with('User', data, action)
|
346
431
|
end
|
347
432
|
```
|
@@ -351,9 +436,12 @@ Note: Be careful with collision of names
|
|
351
436
|
config = PubSubModelSync::Config
|
352
437
|
config.debug = true
|
353
438
|
```
|
354
|
-
|
355
|
-
|
356
|
-
|
439
|
+
- `.topic_name = ['topic1', 'topic 2']`: (String|Array<String>)
|
440
|
+
Topic name(s) to be used to listen all notifications from when listening. Additional first topic name is used as the default topic name when publishing a notification.
|
441
|
+
- `.subscription_name = "my-app-1"`: (String, default Rails.application.name)
|
442
|
+
Subscriber's identifier which helps to:
|
443
|
+
* skip self messages
|
444
|
+
* continue the sync from the last synced notification when service was restarted.
|
357
445
|
- ```.debug = true```
|
358
446
|
(true/false*) => show advanced log messages
|
359
447
|
- ```.logger = Rails.logger```
|
@@ -378,18 +466,15 @@ config.debug = true
|
|
378
466
|
- Add flag ```model.ps_process_payload``` to retrieve the payload used to process the pub/sub sync
|
379
467
|
- Auto publish update only if payload has changed
|
380
468
|
- On delete, payload must only be composed by ids
|
381
|
-
-
|
382
|
-
PubSubModelSync::MessagePublisher.
|
383
|
-
- Add DB table to use as a shield to
|
469
|
+
- Improve transactions to exclude similar messages by klass and action. Sample:
|
470
|
+
```PubSubModelSync::MessagePublisher.transaction(key, { same_keys: :use_last_as_first|:use_last|:use_first_as_last|:keep*, same_data: :use_last_as_first*|:use_last|:use_first_as_last|:keep })```
|
471
|
+
- Add DB table to use as a shield to prevent publishing similar notifications and publish partial notifications (similar idea when processing notif)
|
384
472
|
- add callback: on_message_received(payload)
|
385
473
|
|
386
474
|
## **Q&A**
|
387
475
|
- I'm getting error "could not obtain a connection from the pool within 5.000 seconds"... what does this mean?
|
388
476
|
This problem occurs because pub/sub dependencies (kafka, google-pubsub, rabbitmq) use many threads to perform notifications where the qty of threads is greater than qty of DB pools ([Google pubsub info](https://github.com/googleapis/google-cloud-ruby/blob/master/google-cloud-pubsub/lib/google/cloud/pubsub/subscription.rb#L888))
|
389
|
-
To fix the problem, edit config/database.yml and increase the quantity of ```pool:
|
390
|
-
- Google pubsub: How to process notifications parallely and not sequentially (default 1 thread)?
|
391
|
-
```ruby PubSubModelSync::ServiceGoogle::LISTEN_SETTINGS = { threads: { callback: qty_threads } } ```
|
392
|
-
Note: by this way some notifications can be processed before others thus missing relationship errors can appear
|
477
|
+
To fix the problem, edit config/database.yml and increase the quantity of ```pool: 20```
|
393
478
|
- How to retry failed syncs with sidekiq?
|
394
479
|
```ruby
|
395
480
|
# lib/initializers/pub_sub_config.rb
|