pub_sub_model_sync 0.5.10 → 0.6.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: c3cb9b467f173cf208d051aead1d069868b9dfd6cc319311aac4fdaa3431d05f
4
- data.tar.gz: b5f6dad938ee545ab4f99c6cd51b84e574856c63f044f5a46e784ca17a01076c
3
+ metadata.gz: c5b72a2c8c7d97a09c17b3985ce466ecf1fba62573dbb860fae9ccab064e84d9
4
+ data.tar.gz: 4f0f801bc9d51fee36b5f279c4f995d9bf441f608ea26d16314dd2182bf67dc7
5
5
  SHA512:
6
- metadata.gz: c548cd499bfde36a4d0db4a8dc75e8ad3cadc389cfffb55c95db9ab306ecc7ce338274603e66ebb1f45932fd1de61cc5f6d4ad123c6ed0931ad3284cbdc957f3
7
- data.tar.gz: 62ced05bba4175c69d4453e35f5c9fa3414b1d2f123d40ada555e75bc876488d78f388e87a20a01c1bf895ab2991eeb1a0f2287cbd593aaec8d0429aec7519ee
6
+ metadata.gz: bdb68283d5feca9b506a11478671e8deea6e425065ececefd2162a227b01a7be837fdd533fd5b51e58e60e3d438ffa8e89009f690e0b68696b06cc6e204f5c24
7
+ data.tar.gz: 99baadbbd02b889af06b4cd1b17847a61efe074d58c3f7cc6b0e3eb693f277604f5420b8924bca383cfc8cdbba550d8c16755231335e0a1e056663cb0a5f4deb
@@ -24,7 +24,7 @@ jobs:
24
24
  steps:
25
25
  - uses: actions/checkout@v2
26
26
  - name: Set up Ruby
27
- uses: actions/setup-ruby@v1
27
+ uses: ruby/setup-ruby@v1
28
28
  with:
29
29
  ruby-version: ${{ matrix.ruby }}
30
30
  - name: Install sqlite3
data/CHANGELOG.md CHANGED
@@ -1,5 +1,18 @@
1
1
  # Change Log
2
2
 
3
+ # 0.6.0 (March 03, 2021)
4
+ - feat: add support to include custom payload headers
5
+ - feat: add pubsub transactions to process all payloads inside in the same order they were published
6
+ - feat: when a model is created/updated/destroyed, process all related payloads in a single transaction
7
+ - feat: add method to save processed payload (:ps_processed_payload) when saving sync
8
+ - feat: add "ordering_key" support to process all payloads with the same key in the same order
9
+ - feat: start multiple workers to process async kafka messages when starting service listeners
10
+ - feat: make async publisher by reusing exchange connection (rabbit)
11
+ - feat: add support for forced_ordering_key to always be used as the ordering_key if defined
12
+ - feat: add feature to publish a message to a custom and/or multiple topics
13
+ - feat: add model custom action subscriber and publisher
14
+ - feat: add docker compose settings
15
+
3
16
  # 0.5.10 (February 13, 2021)
4
17
  - feat: remove duplicated callback :ps_before_save_sync (same result can be achieved with :ps_before_save_sync)
5
18
  - feat: improve message starter to retry when failed or exit system when persists
data/Dockerfile ADDED
@@ -0,0 +1,6 @@
1
+ FROM ruby:2.7-buster
2
+ RUN apt-get update -qq
3
+ WORKDIR /myapp
4
+ COPY . /myapp
5
+ RUN gem update bundler
6
+ RUN bundle install
data/Gemfile.lock CHANGED
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- pub_sub_model_sync (0.5.10)
4
+ pub_sub_model_sync (0.6.0)
5
5
  rails
6
6
 
7
7
  GEM
@@ -243,6 +243,7 @@ DEPENDENCIES
243
243
  database_cleaner-active_record
244
244
  google-cloud-pubsub (> 2.0)
245
245
  pub_sub_model_sync!
246
+ rails (~> 6)
246
247
  rake
247
248
  rspec
248
249
  rubocop (~> 1.6.0)
data/README.md CHANGED
@@ -6,8 +6,9 @@ Note: This gem is based on [MultipleMan](https://github.com/influitive/multiple_
6
6
  - [**PubSubModelSync**](#pubsubmodelsync)
7
7
  - [**Features**](#features)
8
8
  - [**Installation**](#installation)
9
- - [**Usage**](#usage)
10
- - [**Examples**](#examples)
9
+ - [**Configuration**](#configuration)
10
+ - [**Notifications Diagram**](#notifications-diagram)
11
+ - [**Basic Example**](#basic-example)
11
12
  - [**Advanced Example**](#advanced-example)
12
13
  - [**API**](#api)
13
14
  - [**Subscribers**](#subscribers)
@@ -33,6 +34,10 @@ Note: This gem is based on [MultipleMan](https://github.com/influitive/multiple_
33
34
  - Ability to make class level communication
34
35
  Example: If User from App1 wants to generate_email, this can be listened on App2, App3, ... to make corresponding actions
35
36
  - Change pub/sub service at any time
37
+ - Support for transactions: Permits to group all payloads with the same ordering_key and be processed in the same order they are published by the subscribers.
38
+ Grouping by ordering_key allows us to enable multiple workers in our Pub/Sub service(s), and still guarantee that related payloads will be processed in the correct order, despite of the multiple threads.
39
+ This thanks to the fact that Pub/Sub services will always send messages with the same `ordering_key` into the same worker/thread.
40
+ - Ability to send notifications to a specific topic or multiple topics
36
41
 
37
42
  ## **Installation**
38
43
  Add this line to your application's Gemfile:
@@ -46,7 +51,7 @@ gem 'ruby-kafka' # to use apache kafka pub/sub service
46
51
  And then execute: $ bundle install
47
52
 
48
53
 
49
- ## **Usage**
54
+ ## **Configuration**
50
55
 
51
56
  - Configuration for google pub/sub (You need google pub/sub service account)
52
57
  ```ruby
@@ -54,7 +59,8 @@ And then execute: $ bundle install
54
59
  PubSubModelSync::Config.service_name = :google
55
60
  PubSubModelSync::Config.project = 'google-project-id'
56
61
  PubSubModelSync::Config.credentials = 'path-to-the-config'
57
- PubSubModelSync::Config.topic_name = 'sample-topic'
62
+ PubSubModelSync::Config.topic_name = 'sample-topic'
63
+ PubSubModelSync::Config.subscription_name = 'my-app1'
58
64
  ```
59
65
  See details here:
60
66
  https://github.com/googleapis/google-cloud-ruby/tree/master/google-cloud-pubsub
@@ -63,8 +69,8 @@ And then execute: $ bundle install
63
69
  ```ruby
64
70
  PubSubModelSync::Config.service_name = :rabbitmq
65
71
  PubSubModelSync::Config.bunny_connection = 'amqp://guest:guest@localhost'
66
- PubSubModelSync::Config.queue_name = 'model-sync'
67
72
  PubSubModelSync::Config.topic_name = 'sample-topic'
73
+ PubSubModelSync::Config.subscription_name = 'my-app2'
68
74
  ```
69
75
  See details here: https://github.com/ruby-amqp/bunny
70
76
 
@@ -73,27 +79,25 @@ And then execute: $ bundle install
73
79
  PubSubModelSync::Config.service_name = :kafka
74
80
  PubSubModelSync::Config.kafka_connection = [["kafka1:9092", "localhost:2121"], { logger: Rails.logger }]
75
81
  PubSubModelSync::Config.topic_name = 'sample-topic'
82
+ PubSubModelSync::Config.subscription_name = 'my-app3'
76
83
  ```
77
84
  See details here: https://github.com/zendesk/ruby-kafka
78
85
 
79
86
  - Add publishers/subscribers to your models (See examples below)
80
87
 
81
88
  - Start subscribers to listen for publishers (Only in the app that has subscribers)
82
- ```ruby
83
- rake pub_sub_model_sync:start
84
- ```
85
- Note: Publishers do not need todo this
86
- Note2 (Rails 6+): Due to Zeitwerk, you need to load listeners manually when syncing without mentioned task (like rails console)
87
- ```ruby
88
- # PubSubModelSync::Config.subscribers ==> []
89
- PubSubModelSync::Runner.preload_listeners
90
- # PubSubModelSync::Config.subscribers ==> [#<PubSubModelSync::Subscriber:0x000.. @klass="Article", @action=:create..., ....]
89
+ ```bash
90
+ DB_POOL=20 bundle exec rake pub_sub_model_sync:start
91
91
  ```
92
+ Note: You need more than 15 DB pools to avoid "could not obtain a connection from the pool within 5.000 seconds". https://devcenter.heroku.com/articles/concurrency-and-database-connections
92
93
 
93
94
  - Check the service status with:
94
95
  ```PubSubModelSync::MessagePublisher.publish_data('Test message', {sample_value: 10}, :create)```
95
96
 
96
- ## **Examples**
97
+ ## **Notifications Diagram**
98
+ ![Diagram](/docs/notifications-diagram.png?raw=true)
99
+
100
+ ## **Basic Example**
97
101
  ```ruby
98
102
  # App 1 (Publisher)
99
103
  # attributes: name email age
@@ -105,20 +109,25 @@ end
105
109
  # App 2 (Subscriber)
106
110
  class User < ActiveRecord::Base
107
111
  include PubSubModelSync::SubscriberConcern
108
- ps_subscribe(%i[name])
109
- ps_class_subscribe(:greeting)
112
+ ps_subscribe(%i[name]) # crud notifications
113
+ ps_subscribe_custom(:say_welcome) # custom instance notification
114
+ ps_class_subscribe(:greeting) # class notification
110
115
 
111
116
  def self.greeting(data)
112
117
  puts 'Class message called'
113
118
  end
119
+
120
+ def say_welcome(data)
121
+ UserMailer.deliver(id, data)
122
+ end
114
123
  end
115
124
 
116
125
  # Samples
117
126
  User.create(name: 'test user', email: 'sample@gmail.com') # Review your App 2 to see the created user (only name will be saved)
118
127
  User.new(name: 'test user').ps_perform_sync(:create) # similar to above to perform sync on demand
119
128
 
120
- User.ps_class_publish({ msg: 'Hello' }, action: :greeting) # User.greeting method (Class method) will be called in App2
121
- PubSubModelSync::MessagePublisher.publish_data(User, { msg: 'Hello' }, :greeting) # similar to above when not included publisher concern
129
+ PubSubModelSync::MessagePublisher.publish_model_data(my_user, { id:10, msg: 'Hello' }, :say_welcome, { as_klass: 'RegisteredUser' }) # custom model action notification
130
+ PubSubModelSync::MessagePublisher.publish_data(User, { msg: 'Hello' }, :greeting) # custom data notification
122
131
  ```
123
132
 
124
133
  ## **Advanced Example**
@@ -127,7 +136,7 @@ PubSubModelSync::MessagePublisher.publish_data(User, { msg: 'Hello' }, :greeting
127
136
  class User < ActiveRecord::Base
128
137
  self.table_name = 'publisher_users'
129
138
  include PubSubModelSync::PublisherConcern
130
- ps_publish(%i[id:client_id name:full_name email], actions: %i[update], as_klass: 'Client')
139
+ ps_publish(%i[id:client_id name:full_name email], actions: %i[update], as_klass: 'Client', headers: { topic_name: ['topic1', 'topic N'] })
131
140
 
132
141
  def ps_skip_callback?(_action)
133
142
  false # here logic with action to skip push message
@@ -144,11 +153,16 @@ class User < ActiveRecord::Base
144
153
  include PubSubModelSync::SubscriberConcern
145
154
  ps_subscribe(%i[name], actions: %i[update], from_klass: 'Client', id: %i[client_id email])
146
155
  ps_class_subscribe(:greeting, from_action: :custom_greeting, from_klass: 'CustomUser')
156
+ ps_subscribe_custom(:send_welcome, from_klass: 'CustomUser', id: :id, from_action: :say_welcome)
147
157
  alias_attribute :full_name, :name
148
158
 
149
159
  def self.greeting(data)
150
160
  puts 'Class message called through custom_greeting'
151
161
  end
162
+
163
+ def send_welcome(data)
164
+ UserMailer.deliver(id, data)
165
+ end
152
166
 
153
167
  # def self.ps_find_model(data)
154
168
  # where(email: data[:email], ...).first_or_initialize
@@ -156,136 +170,207 @@ class User < ActiveRecord::Base
156
170
  end
157
171
  ```
158
172
 
159
- Note: Be careful with collision of names
160
- ```
161
- # ps_publish %i[name_data:name name:key] # key will be replaced with name_data
162
- ps_publish %i[name_data:name key_data:key] # use alias to avoid collision
163
- ```
164
-
165
173
  ## **API**
166
174
  ### **Subscribers**
167
175
 
168
- #### **Registering Subscription Callbacks**
176
+ #### **Registering Subscriptions**
169
177
 
170
- - Configure model-level subscriptions
178
+ - Configure class subscriptions
171
179
  ```ruby
172
180
  class MyModel < ActiveRecord::Base
173
181
  ps_class_subscribe(action_name, from_action: nil, from_klass: nil)
174
182
  end
175
183
  ```
176
- * `from_action`: (Optional) Source method name
177
- * `from_klass`: (Optional) Source class name
184
+ When Class receives the corresponding notification, `action` method will be called on the Class. Like: `User.action(data)`
185
+ * `action_name`: (String|Sym/Optional) Action name
186
+ * `from_klass`: (String/Optional) Source class name (Default `model.class.name`)
187
+ * `from_action`: (Sym/Optional) Source method name. Default `action`
178
188
 
179
- - Configure instance-level subscriptions (CRUD)
189
+ - Configure CRUD subscriptions
180
190
  ```ruby
181
191
  class MyModel < ActiveRecord::Base
182
192
  ps_subscribe(attrs, from_klass: nil, actions: nil, id: nil)
183
193
  end
184
194
  ```
195
+ When model receives the corresponding notification, `action` method will be called on the model. Like: `model.destroy`
185
196
  * `attrs`: (Array/Required) Array of all attributes to be synced
186
- * `from_klass`: (String/Optional) Source class name (Instead of the model class name, will use this value)
197
+ * `from_klass`: (String/Optional) Source class name (Default `model.class.name`)
187
198
  * `actions`: (Array/Optional, default: create/update/destroy) permit to customize action names
188
199
  * `id`: (Sym|Array/Optional, default: id) Attr identifier(s) to find the corresponding model
189
200
 
190
- - Configure a custom model finder
201
+ - Configure custom model subscriptions
191
202
  ```ruby
192
203
  class MyModel < ActiveRecord::Base
193
- ps_find_model(data)
204
+ ps_subscribe_custom(action, from_klass: name, id: :id, from_action: nil)
194
205
  end
195
206
  ```
196
- * `data`: (Hash) Data received from sync
197
- Must return an existent or a new model object
207
+ When model receives the corresponding notification, `action` method will be called on the model. Like: `model.action(data)`
208
+ * `action`: (String/Required) Action name
209
+ * `from_klass`: (String/Optional) Source class name (Default `model.class.name`)
210
+ * `from_action`: (Sym/Optional) Source method name. Default `action`
211
+ * `id`: (Sym|Array/Optional, default: id) Attr identifier(s) to find the corresponding model
198
212
 
199
- #### **Class Methods**
200
- - Configure CRUD subscription for the class
213
+ - Perform custom actions before saving sync of the model (`:cancel` can be returned to skip sync)
201
214
  ```ruby
202
- MyModel.ps_subscriber(action_name)
215
+ class MyModel < ActiveRecord::Base
216
+ def ps_before_save_sync(action, payload)
217
+ # puts payload.data[:id]
218
+ end
219
+ end
203
220
  ```
204
- * `action_name` (default :create, :sym): can be :create, :update, :destroy
221
+
222
+ - Configure a custom model finder (optional)
223
+ ```ruby
224
+ class MyModel < ActiveRecord::Base
225
+ def ps_find_model(data)
226
+ where(custom_finder: data[:custom_value]).first_or_initialize
227
+ end
228
+ end
229
+ ```
230
+ * `data`: (Hash) Data received from sync
231
+ Must return an existent or a new model object
205
232
 
206
- - Inspect all configured subscribers
233
+ #### **Subscription helpers**
234
+ - Inspect all configured subscriptions
207
235
  ```ruby
208
236
  PubSubModelSync::Config.subscribers
209
237
  ```
210
-
211
- #### **Instance Methods**
212
-
213
- - Perform custom actions before saving sync of the model (On-demand, `:cancel` can be returned to skip sync)
238
+ - Manually process or reprocess a notification
214
239
  ```ruby
215
- my_instance.ps_before_save_sync(payload)
240
+ payload = PubSubModelSync::Payload.new(data, attributes, headers)
241
+ payload.process!
216
242
  ```
217
243
 
244
+
218
245
  ### **Publishers**
219
246
 
220
- #### **Registering Publishing Callbacks**
221
- - You can register Model-level lifecycle callbacks (CRUD) that will trigger publishing events like this:
247
+ #### **Registering Publishers **
248
+ - Register CRUD publishers that will trigger configured notifications
222
249
  ```ruby
223
- ps_publish(attrs, actions: nil, as_klass: nil)
250
+ class MyModel < ActiveRecord::Base
251
+ ps_publish([:id, 'created_at:published_at', :full_name], actions: [:update], as_klass: nil, headers: { ordering_key: 'custom-key', topic_name: 'my-custom-topic' })
252
+ def full_name
253
+ [first_name, last_name].join(' ')
254
+ end
255
+ end
224
256
  ```
225
- * `attrs`: (Array/Required) Array of attributes to be published
226
- * `actions`: (Array/Optional, default: create/update/destroy) permit to customize action names
257
+ * `attrs`: (Array/Required) Array of attributes to be published. Supports for:
258
+ - aliases: permits to publish with different names, sample: "created_at:published_at" where "created_at" will be published as "published_at"
259
+ - methods: permits to publish method values as attributes, sample: "full_name"
260
+ * `actions`: (Array/Optional, default: %i[create update destroy]) permit to define action names
227
261
  * `as_klass`: (String/Optional) Output class name (Instead of the model class name, will use this value)
262
+ * `headers`: (Hash/Optional) Notification settings which permit to customize the way and the target of the notification (Refer Payload.headers)
263
+
228
264
 
265
+ #### **Publishing notifications**
266
+ - CRUD notifications
267
+ ```ruby
268
+ MyModel.create!(...)
269
+ ```
270
+ "Create" notification will be delivered with the configured attributes as the payload data
229
271
 
230
- #### **Instance Methods**
272
+ - Manual CRUD notifications
273
+ ```ruby
274
+ MyModel.ps_perform_sync(action, custom_data: {}, custom_headers: {})
275
+ ```
276
+ * `action`: (Sym) CRUD action name (create, update or destroy)
277
+ * `custom_data`: custom_data (nil|Hash) If present custom_data will be used as the payload data. I.E. data generator will be ignored
278
+ * `custom_headers`: (Hash, optional) override default headers. Refer `payload.headers`
279
+
280
+ - Class notifications
281
+ ```ruby
282
+ PubSubModelSync::MessagePublisher.publish_data((klass, data, action, headers: )
283
+ ```
284
+ Publishes any data to be listened at a class level.
285
+ - `klass`: (String) Class name to be used
286
+ - `data`: (Hash) Data to be delivered
287
+ - `action`: (Sym) Action name
288
+ - `headers`: (Hash, optional) Notification settings (Refer Payload.headers)
289
+
290
+ - Model custom action notifications
291
+ ```ruby
292
+ PubSubModelSync::MessagePublisher.publish_model_data(model, data, action, as_klass:, headers:)
293
+ ```
294
+ Publishes model custom action to be listened at an instance level.
295
+ - `model`: (ActiveRecord) model owner of the data
296
+ - `data`: (Hash) Data to be delivered
297
+ - `action`: (Sym) Action name
298
+ - `as_klass`: (String, optional) if not provided, `model.class.name` will be used instead
299
+ - `headers`: (Hash, optional) Notification settings (Refer Payload.headers)
300
+
301
+ - Manually publish or republish a notification
302
+ ```ruby
303
+ payload = PubSubModelSync::Payload.new(data, attributes, headers)
304
+ payload.publish!
305
+ ```
306
+
307
+ #### ** publishing callbacks**
231
308
 
232
- - **Prevent PS-related callback** (On-demand, before the callback gets triggered)
309
+ - Prevent CRUD sync at model callback level (Called right after :after_create, :after_update, :after_destroy).
310
+ If returns "true", sync will be cancelled.
233
311
  ```ruby
234
- model_instance.ps_skip_callback?(action)
312
+ class MyModel < ActiveRecord::Base
313
+ def ps_skip_callback?(action)
314
+ # logic here
315
+ end
316
+ end
235
317
  ```
236
- Default: False
237
- Note: Return true to cancel sync
238
318
 
239
- - **Prevent sync after create/update/destroy action** (On-demand, before the sync gets triggered)
319
+ - Prevent CRUD sync before processing payload (Affects model.ps_perform_sync(...))).
320
+ If returns "true", sync will be cancelled
240
321
  ```ruby
241
- model_instance.ps_skip_sync?(action)
322
+ class MyModel < ActiveRecord::Base
323
+ def ps_skip_sync?(action)
324
+ # logic here
325
+ end
326
+ end
242
327
  ```
243
- Note: return true to cancel sync
244
328
 
245
- - **Execute a callback before sync** (On-demand, before sync is executed, but after payload is received )
329
+ - Do some actions before publishing a CRUD notification.
330
+ If returns ":cancel", sync will be cancelled
246
331
  ```ruby
247
- model_instance.ps_before_sync(action, data_to_deliver)
332
+ class MyModel < ActiveRecord::Base
333
+ def ps_before_sync(action, payload)
334
+ # logic here
335
+ end
336
+ end
248
337
  ```
249
- Note: If the method returns ```:cancel```, the sync will be stopped (message will not be published)
250
338
 
251
- - **Execute a callback after sync**
339
+ - Do some actions after CRUD notification was published.
252
340
  ```ruby
253
- model_instance.ps_after_sync(action, data_delivered)
341
+ class MyModel < ActiveRecord::Base
342
+ def ps_after_sync(action, payload)
343
+ # logic here
344
+ end
345
+ end
254
346
  ```
255
347
 
256
- - **Trigger a sync on-demand** (:create, :update, :destroy):
257
- The target model will receive a notification to perform the indicated action
258
- ```ruby
259
- model_instance.ps_perform_sync(action_name, custom_settings = {})
260
- ```
261
- * `custom_settings`: override default settings defined for action_name ({ attrs: [], as_klass: nil })
262
348
 
263
- #### **Class Methods**
349
+ ### **Payload**
350
+ Any notification before delivering is transformed as a Payload for a better portability.
264
351
 
265
- - **Publish a class level notification**:
352
+ - Initialize
266
353
  ```ruby
267
- User.ps_class_publish(data, action: action_name, as_klass: custom_klass_name)
354
+ payload = PubSubModelSync::Payload.new(data, attributes, headers)
268
355
  ```
269
- Target class ```User.action_name``` will be called when message is received
270
- * `data`: (required, :hash) message value to deliver
271
- * `action_name`: (required, :sim) Action name
272
- * `as_klass`: (optional, :string) Custom class name (Default current model name)
273
-
274
- #### **Payload actions**
356
+ * `data`: (Hash) Data to be published or processed
357
+ * `attributes`: (Hash) Includes class and method info
358
+ - `action`: (String) action name
359
+ - `klass`: (String) class name
360
+ * `headers`: (Hash) Notification settings that defines how the notification will be processed or delivered.
361
+ - `key`: (String, optional) identifier of the payload, default: `<klass_name>/<action>` when class message, `<model.class.name>/<action>/<model.id>` when model message (Useful for caching techniques).
362
+ - `ordering_key`: (String, optional): messages with the same key are processed in the same order they were delivered, default: `klass_name` when class message, `<model.class.name>/<model.id>` when model message
363
+ - `topic_name`: (String|Array<String>, optional): Specific topic name to be used when delivering the message (default first topic from config).
364
+ - `forced_ordering_key`: (String, optional): Will force to use this value as the `ordering_key`, even withing transactions. Default `nil`.
365
+
366
+ - Actions for payloads
275
367
  ```ruby
276
- payload = PubSubModelSync::Payload.new({ title: 'hello' }, { action: :greeting, klass: 'User' })
277
368
  payload.publish! # publishes notification data. It raises exception if fails and does not call ```:on_error_publishing``` callback
278
369
  payload.publish # publishes notification data. On error does not raise exception but calls ```:on_error_publishing``` callback
279
370
  payload.process! # process a notification data. It raises exception if fails and does not call ```.on_error_processing``` callback
280
371
  payload.publish # process a notification data. It does not raise exception if fails but calls ```.on_error_processing``` callback
281
372
  ```
282
373
 
283
- - Get crud publisher configured for the class
284
- ```ruby
285
- User.ps_publisher(action_name)
286
- ```
287
- * `action_name` (default :create, :sym): can be :create, :update, :destroy
288
-
289
374
  ## **Testing with RSpec**
290
375
  - Config: (spec/rails_helper.rb)
291
376
  ```ruby
@@ -341,7 +426,7 @@ Note: Be careful with collision of names
341
426
  publisher = PubSubModelSync::MessagePublisher
342
427
  data = {msg: 'hello'}
343
428
  action = :greeting
344
- User.ps_class_publish(data, action: action)
429
+ PubSubModelSync::MessagePublisher.publish_data('User', data, action)
345
430
  expect(publisher).to receive(:publish_data).with('User', data, action)
346
431
  end
347
432
  ```
@@ -351,9 +436,12 @@ Note: Be careful with collision of names
351
436
  config = PubSubModelSync::Config
352
437
  config.debug = true
353
438
  ```
354
-
355
- - ```.subscription_name = 'app-2'```
356
- Permit to define a custom consumer identifier (Default: Rails application name)
439
+ - `.topic_name = ['topic1', 'topic 2']`: (String|Array<String>)
440
+ Topic name(s) to be used to listen all notifications from when listening. Additional first topic name is used as the default topic name when publishing a notification.
441
+ - `.subscription_name = "my-app-1"`: (String, default Rails.application.name)
442
+ Subscriber's identifier which helps to:
443
+ * skip self messages
444
+ * continue the sync from the last synced notification when service was restarted.
357
445
  - ```.debug = true```
358
446
  (true/false*) => show advanced log messages
359
447
  - ```.logger = Rails.logger```
@@ -378,18 +466,15 @@ config.debug = true
378
466
  - Add flag ```model.ps_process_payload``` to retrieve the payload used to process the pub/sub sync
379
467
  - Auto publish update only if payload has changed
380
468
  - On delete, payload must only be composed by ids
381
- - Feature to publish multiple message at a time with the ability to exclude similar messages by klass and action (use the last one)
382
- PubSubModelSync::MessagePublisher.batch_publish({ same_keys: :use_last_as_first|:use_last|:use_first_as_last|:keep*, same_data: :use_last_as_first*|:use_last|:use_first_as_last|:keep })
383
- - Add DB table to use as a shield to skip publishing similar notifications or publish partial notifications (similar idea when processing notif)
469
+ - Improve transactions to exclude similar messages by klass and action. Sample:
470
+ ```PubSubModelSync::MessagePublisher.transaction(key, { same_keys: :use_last_as_first|:use_last|:use_first_as_last|:keep*, same_data: :use_last_as_first*|:use_last|:use_first_as_last|:keep })```
471
+ - Add DB table to use as a shield to prevent publishing similar notifications and publish partial notifications (similar idea when processing notif)
384
472
  - add callback: on_message_received(payload)
385
473
 
386
474
  ## **Q&A**
387
475
  - I'm getting error "could not obtain a connection from the pool within 5.000 seconds"... what does this mean?
388
476
  This problem occurs because pub/sub dependencies (kafka, google-pubsub, rabbitmq) use many threads to perform notifications where the qty of threads is greater than qty of DB pools ([Google pubsub info](https://github.com/googleapis/google-cloud-ruby/blob/master/google-cloud-pubsub/lib/google/cloud/pubsub/subscription.rb#L888))
389
- To fix the problem, edit config/database.yml and increase the quantity of ```pool: 10```
390
- - Google pubsub: How to process notifications parallely and not sequentially (default 1 thread)?
391
- ```ruby PubSubModelSync::ServiceGoogle::LISTEN_SETTINGS = { threads: { callback: qty_threads } } ```
392
- Note: by this way some notifications can be processed before others thus missing relationship errors can appear
477
+ To fix the problem, edit config/database.yml and increase the quantity of ```pool: 20```
393
478
  - How to retry failed syncs with sidekiq?
394
479
  ```ruby
395
480
  # lib/initializers/pub_sub_config.rb