pub_sub_model_sync 0.5.9.1 → 0.5.10

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 585431eec8ef5f6f9d4b0ccea9069d03ea2849d94f7a8f43464f588da8cfae6c
4
- data.tar.gz: 4ebf42b8ca72c0bf150f3c27a9ca9b791532954deb1663c571927324be952589
3
+ metadata.gz: c3cb9b467f173cf208d051aead1d069868b9dfd6cc319311aac4fdaa3431d05f
4
+ data.tar.gz: b5f6dad938ee545ab4f99c6cd51b84e574856c63f044f5a46e784ca17a01076c
5
5
  SHA512:
6
- metadata.gz: 1f0c786ff6de119c0add336c0182158f4c5ab3b9b74d2f9e3f180b3753ac288ecc8e2c6c88149ad82a627691c924f049eeb5e02c39280ec7ce3d04831b99dd18
7
- data.tar.gz: f7ae9286d95af747dd8bd5e21c5f4f87e8aec469077663782103738f7ae9d07fa39b4884f5741e572f17b57a1ce6c358fe8ded7334451aa703559b7367218653
6
+ metadata.gz: c548cd499bfde36a4d0db4a8dc75e8ad3cadc389cfffb55c95db9ab306ecc7ce338274603e66ebb1f45932fd1de61cc5f6d4ad123c6ed0931ad3284cbdc957f3
7
+ data.tar.gz: 62ced05bba4175c69d4453e35f5c9fa3414b1d2f123d40ada555e75bc876488d78f388e87a20a01c1bf895ab2991eeb1a0f2287cbd593aaec8d0429aec7519ee
data/CHANGELOG.md CHANGED
@@ -1,5 +1,12 @@
1
1
  # Change Log
2
2
 
3
+ # 0.5.10 (February 13, 2021)
4
+ - feat: remove duplicated callback :ps_before_save_sync (same result can be achieved with :ps_before_save_sync)
5
+ - feat: improve message starter to retry when failed or exit system when persists
6
+ - feat: fix and retry when database connection error (PG::UnableToSend)
7
+ - feat: add method to save processed payload (:ps_processed_payload) when saving sync
8
+ - chore: improved readme (Thanks @CharlieIGG)
9
+
3
10
  # 0.5.9.1 (February 10, 2021)
4
11
  - feat: move :key into headers
5
12
 
data/Gemfile.lock CHANGED
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- pub_sub_model_sync (0.5.9.1)
4
+ pub_sub_model_sync (0.5.10)
5
5
  rails
6
6
 
7
7
  GEM
data/README.md CHANGED
@@ -1,16 +1,40 @@
1
- # PubSubModelSync
2
- Permit to sync models data and make calls between rails apps using google or rabbitmq or apache kafka pub/sub service.
3
-
4
- Note: This gem is based on [MultipleMan](https://github.com/influitive/multiple_man) which for now looks unmaintained.
5
-
6
- ## Features
1
+ # **PubSubModelSync**
2
+ Automatically sync Model data and make calls between Rails applications using Google PubSub, RabbitMQ, or Apache Kafka Pub/Sub services.
3
+
4
+ Note: This gem is based on [MultipleMan](https://github.com/influitive/multiple_man) is now unmaintained.
5
+
6
+ - [**PubSubModelSync**](#pubsubmodelsync)
7
+ - [**Features**](#features)
8
+ - [**Installation**](#installation)
9
+ - [**Usage**](#usage)
10
+ - [**Examples**](#examples)
11
+ - [**Advanced Example**](#advanced-example)
12
+ - [**API**](#api)
13
+ - [**Subscribers**](#subscribers)
14
+ - [**Registering Subscription Callbacks**](#registering-subscription-callbacks)
15
+ - [**Class Methods**](#class-methods)
16
+ - [**Instance Methods**](#instance-methods)
17
+ - [**Publishers**](#publishers)
18
+ - [**Registering Publishing Callbacks**](#registering-publishing-callbacks)
19
+ - [**Instance Methods**](#instance-methods-1)
20
+ - [**Class Methods**](#class-methods-1)
21
+ - [**Payload actions**](#payload-actions)
22
+ - [**Testing with RSpec**](#testing-with-rspec)
23
+ - [**Extra configurations**](#extra-configurations)
24
+ - [**TODO**](#todo)
25
+ - [**Q&A**](#qa)
26
+ - [**Contributing**](#contributing)
27
+ - [**License**](#license)
28
+ - [**Code of Conduct**](#code-of-conduct)
29
+
30
+ ## **Features**
7
31
  - Sync CRUD operations between Rails apps. So, all changes made on App1, will be reflected on App2, App3.
8
32
  Example: If User is created on App1, this user will be created on App2 too with the accepted attributes.
9
33
  - Ability to make class level communication
10
34
  Example: If User from App1 wants to generate_email, this can be listened on App2, App3, ... to make corresponding actions
11
35
  - Change pub/sub service at any time
12
36
 
13
- ## Installation
37
+ ## **Installation**
14
38
  Add this line to your application's Gemfile:
15
39
  ```ruby
16
40
  gem 'pub_sub_model_sync'
@@ -22,12 +46,12 @@ gem 'ruby-kafka' # to use apache kafka pub/sub service
22
46
  And then execute: $ bundle install
23
47
 
24
48
 
25
- ## Usage
49
+ ## **Usage**
26
50
 
27
51
  - Configuration for google pub/sub (You need google pub/sub service account)
28
52
  ```ruby
29
53
  # initializers/pub_sub_config.rb
30
- PubSubModelSync::Config.service_name = :google
54
+ PubSubModelSync::Config.service_name = :google
31
55
  PubSubModelSync::Config.project = 'google-project-id'
32
56
  PubSubModelSync::Config.credentials = 'path-to-the-config'
33
57
  PubSubModelSync::Config.topic_name = 'sample-topic'
@@ -51,18 +75,6 @@ And then execute: $ bundle install
51
75
  PubSubModelSync::Config.topic_name = 'sample-topic'
52
76
  ```
53
77
  See details here: https://github.com/zendesk/ruby-kafka
54
-
55
- Kafka Confluence example:
56
- ```ruby
57
- kafka_settings = {
58
- sasl_plain_username: '...',
59
- sasl_plain_password: '...',
60
- ssl_ca_certs_from_system: true,
61
- logger: Rails.logger, client_id: 'my-app-name'
62
- }
63
- PubSubModelSync::Config.kafka_connection = [['...confluent.cloud:9092'], kafka_settings]
64
- ```
65
- Note: You need to create the topic manually on Kafka Confluence
66
78
 
67
79
  - Add publishers/subscribers to your models (See examples below)
68
80
 
@@ -70,21 +82,21 @@ And then execute: $ bundle install
70
82
  ```ruby
71
83
  rake pub_sub_model_sync:start
72
84
  ```
73
- Note: Publishers do not need todo this
85
+ Note: Publishers do not need todo this
74
86
  Note2 (Rails 6+): Due to Zeitwerk, you need to load listeners manually when syncing without mentioned task (like rails console)
75
- ```ruby
87
+ ```ruby
76
88
  # PubSubModelSync::Config.subscribers ==> []
77
89
  PubSubModelSync::Runner.preload_listeners
78
90
  # PubSubModelSync::Config.subscribers ==> [#<PubSubModelSync::Subscriber:0x000.. @klass="Article", @action=:create..., ....]
79
- ```
91
+ ```
80
92
 
81
- - Check the service status with:
93
+ - Check the service status with:
82
94
  ```PubSubModelSync::MessagePublisher.publish_data('Test message', {sample_value: 10}, :create)```
83
95
 
84
- ## Examples
96
+ ## **Examples**
85
97
  ```ruby
86
98
  # App 1 (Publisher)
87
- # attributes: name email age
99
+ # attributes: name email age
88
100
  class User < ActiveRecord::Base
89
101
  include PubSubModelSync::PublisherConcern
90
102
  ps_publish(%i[id name email])
@@ -109,18 +121,18 @@ User.ps_class_publish({ msg: 'Hello' }, action: :greeting) # User.greeting metho
109
121
  PubSubModelSync::MessagePublisher.publish_data(User, { msg: 'Hello' }, :greeting) # similar to above when not included publisher concern
110
122
  ```
111
123
 
112
- ## Advanced Example
124
+ ## **Advanced Example**
113
125
  ```ruby
114
126
  # App 1 (Publisher)
115
127
  class User < ActiveRecord::Base
116
128
  self.table_name = 'publisher_users'
117
129
  include PubSubModelSync::PublisherConcern
118
130
  ps_publish(%i[id:client_id name:full_name email], actions: %i[update], as_klass: 'Client')
119
-
131
+
120
132
  def ps_skip_callback?(_action)
121
133
  false # here logic with action to skip push message
122
134
  end
123
-
135
+
124
136
  def ps_skip_sync?(_action)
125
137
  false # here logic with action to skip push message
126
138
  end
@@ -133,92 +145,133 @@ class User < ActiveRecord::Base
133
145
  ps_subscribe(%i[name], actions: %i[update], from_klass: 'Client', id: %i[client_id email])
134
146
  ps_class_subscribe(:greeting, from_action: :custom_greeting, from_klass: 'CustomUser')
135
147
  alias_attribute :full_name, :name
136
-
148
+
137
149
  def self.greeting(data)
138
150
  puts 'Class message called through custom_greeting'
139
151
  end
140
-
152
+
141
153
  # def self.ps_find_model(data)
142
- # where(email: data[:email], ...).first_or_initialize
154
+ # where(email: data[:email], ...).first_or_initialize
143
155
  # end
144
156
  end
145
157
  ```
146
158
 
147
159
  Note: Be careful with collision of names
148
160
  ```
149
- # ps_publish %i[name_data:name name:key] # key will be replaced with name_data
161
+ # ps_publish %i[name_data:name name:key] # key will be replaced with name_data
150
162
  ps_publish %i[name_data:name key_data:key] # use alias to avoid collision
151
- ```
152
-
153
- ## API
154
- ### Subscribers
155
- - Permit to configure class level subscriptions
156
- ```ps_class_subscribe(action_name, from_action: nil, from_klass: nil)```
157
- * from_action: (Optional) Source method name
158
- * from_klass: (Optional) Source class name
159
-
160
- - Permit to configure instance level subscriptions (CRUD)
161
- ```ps_subscribe(attrs, from_klass: nil, actions: nil, id: nil)```
162
- * attrs: (Array/Required) Array of all attributes to be synced
163
- * from_klass: (String/Optional) Source class name (Instead of the model class name, will use this value)
164
- * actions: (Array/Optional, default: create/update/destroy) permit to customize action names
165
- * id: (Sym|Array/Optional, default: id) Attr identifier(s) to find the corresponding model
166
-
167
- - Permit to configure a custom model finder
168
- ```ps_find_model(data)```
169
- * data: (Hash) Data received from sync
163
+ ```
164
+
165
+ ## **API**
166
+ ### **Subscribers**
167
+
168
+ #### **Registering Subscription Callbacks**
169
+
170
+ - Configure model-level subscriptions
171
+ ```ruby
172
+ class MyModel < ActiveRecord::Base
173
+ ps_class_subscribe(action_name, from_action: nil, from_klass: nil)
174
+ end
175
+ ```
176
+ * `from_action`: (Optional) Source method name
177
+ * `from_klass`: (Optional) Source class name
178
+
179
+ - Configure instance-level subscriptions (CRUD)
180
+ ```ruby
181
+ class MyModel < ActiveRecord::Base
182
+ ps_subscribe(attrs, from_klass: nil, actions: nil, id: nil)
183
+ end
184
+ ```
185
+ * `attrs`: (Array/Required) Array of all attributes to be synced
186
+ * `from_klass`: (String/Optional) Source class name (Instead of the model class name, will use this value)
187
+ * `actions`: (Array/Optional, default: create/update/destroy) permit to customize action names
188
+ * `id`: (Sym|Array/Optional, default: id) Attr identifier(s) to find the corresponding model
189
+
190
+ - Configure a custom model finder
191
+ ```ruby
192
+ class MyModel < ActiveRecord::Base
193
+ ps_find_model(data)
194
+ end
195
+ ```
196
+ * `data`: (Hash) Data received from sync
170
197
  Must return an existent or a new model object
171
198
 
172
- - Get crud subscription configured for the class
173
- ```User.ps_subscriber(action_name)```
174
- * action_name (default :create, :sym): can be :create, :update, :destroy
199
+ #### **Class Methods**
200
+ - Configure CRUD subscription for the class
201
+ ```ruby
202
+ MyModel.ps_subscriber(action_name)
203
+ ```
204
+ * `action_name` (default :create, :sym): can be :create, :update, :destroy
175
205
 
176
- - Inspect all configured subscribers
177
- ```PubSubModelSync::Config.subscribers```
206
+ - Inspect all configured subscribers
207
+ ```ruby
208
+ PubSubModelSync::Config.subscribers
209
+ ```
210
+
211
+ #### **Instance Methods**
212
+
213
+ - Perform custom actions before saving sync of the model (On-demand, `:cancel` can be returned to skip sync)
214
+ ```ruby
215
+ my_instance.ps_before_save_sync(payload)
216
+ ```
217
+
218
+ ### **Publishers**
178
219
 
179
- - Permit to customize the way to detect if the subscribed model was changed (Only for update action).
180
- ```.ps_subscriber_changed?(data)```
181
- By default: ```model.changed?```
220
+ #### **Registering Publishing Callbacks**
221
+ - You can register Model-level lifecycle callbacks (CRUD) that will trigger publishing events like this:
222
+ ```ruby
223
+ ps_publish(attrs, actions: nil, as_klass: nil)
224
+ ```
225
+ * `attrs`: (Array/Required) Array of attributes to be published
226
+ * `actions`: (Array/Optional, default: create/update/destroy) permit to customize action names
227
+ * `as_klass`: (String/Optional) Output class name (Instead of the model class name, will use this value)
182
228
 
183
- - Permit to perform custom actions before saving sync of the model (:cancel can be returned to skip sync)
184
- ```.ps_before_save_sync(payload)```
185
229
 
186
- ### Publishers
187
- - Permit to configure crud publishers
188
- ```ps_publish(attrs, actions: nil, as_klass: nil)```
189
- * attrs: (Array/Required) Array of attributes to be published
190
- * actions: (Array/Optional, default: create/update/destroy) permit to customize action names
191
- * as_klass: (String/Optional) Output class name (Instead of the model class name, will use this value)
230
+ #### **Instance Methods**
192
231
 
193
- - Permit to cancel sync called after create/update/destroy (Before initializing sync service)
194
- ```model.ps_skip_callback?(action)```
195
- Default: False
232
+ - **Prevent PS-related callback** (On-demand, before the callback gets triggered)
233
+ ```ruby
234
+ model_instance.ps_skip_callback?(action)
235
+ ```
236
+ Default: False
196
237
  Note: Return true to cancel sync
197
-
198
- - Callback called before preparing data for sync (Permit to stop sync)
199
- ```model.ps_skip_sync?(action)```
200
- Note: return true to cancel sync
201
-
202
- - Callback called before sync (After preparing data)
203
- ```model.ps_before_sync(action, data_to_deliver)```
238
+
239
+ - **Prevent sync after create/update/destroy action** (On-demand, before the sync gets triggered)
240
+ ```ruby
241
+ model_instance.ps_skip_sync?(action)
242
+ ```
243
+ Note: return true to cancel sync
244
+
245
+ - **Execute a callback before sync** (On-demand, before sync is executed, but after payload is received )
246
+ ```ruby
247
+ model_instance.ps_before_sync(action, data_to_deliver)
248
+ ```
204
249
  Note: If the method returns ```:cancel```, the sync will be stopped (message will not be published)
205
250
 
206
- - Callback called after sync
207
- ```model.ps_after_sync(action, data_delivered)```
208
-
209
- - Perform sync on demand (:create, :update, :destroy):
210
- The target model will receive a notification to perform the indicated action
211
- ```my_model.ps_perform_sync(action_name, custom_settings = {})```
212
- * custom_settings: override default settings defined for action_name ({ attrs: [], as_klass: nil })
213
-
214
- - Publish a class level notification:
215
- ```User.ps_class_publish(data, action: action_name, as_klass: custom_klass_name)```
216
- Target class ```User.action_name``` will be called when message is received
217
- * data: (required, :hash) message value to deliver
218
- * action_name: (required, :sim) Action name
219
- * as_klass: (optional, :string) Custom class name (Default current model name)
220
-
221
- - Payload actions
251
+ - **Execute a callback after sync**
252
+ ```ruby
253
+ model_instance.ps_after_sync(action, data_delivered)
254
+ ```
255
+
256
+ - **Trigger a sync on-demand** (:create, :update, :destroy):
257
+ The target model will receive a notification to perform the indicated action
258
+ ```ruby
259
+ model_instance.ps_perform_sync(action_name, custom_settings = {})
260
+ ```
261
+ * `custom_settings`: override default settings defined for action_name ({ attrs: [], as_klass: nil })
262
+
263
+ #### **Class Methods**
264
+
265
+ - **Publish a class level notification**:
266
+ ```ruby
267
+ User.ps_class_publish(data, action: action_name, as_klass: custom_klass_name)
268
+ ```
269
+ Target class ```User.action_name``` will be called when message is received
270
+ * `data`: (required, :hash) message value to deliver
271
+ * `action_name`: (required, :sim) Action name
272
+ * `as_klass`: (optional, :string) Custom class name (Default current model name)
273
+
274
+ #### **Payload actions**
222
275
  ```ruby
223
276
  payload = PubSubModelSync::Payload.new({ title: 'hello' }, { action: :greeting, klass: 'User' })
224
277
  payload.publish! # publishes notification data. It raises exception if fails and does not call ```:on_error_publishing``` callback
@@ -226,36 +279,38 @@ Note: Be careful with collision of names
226
279
  payload.process! # process a notification data. It raises exception if fails and does not call ```.on_error_processing``` callback
227
280
  payload.publish # process a notification data. It does not raise exception if fails but calls ```.on_error_processing``` callback
228
281
  ```
229
-
230
- - Get crud publisher configured for the class
231
- ```User.ps_publisher(action_name)```
232
- * action_name (default :create, :sym): can be :create, :update, :destroy
233
-
234
- ## Testing with RSpec
282
+
283
+ - Get crud publisher configured for the class
284
+ ```ruby
285
+ User.ps_publisher(action_name)
286
+ ```
287
+ * `action_name` (default :create, :sym): can be :create, :update, :destroy
288
+
289
+ ## **Testing with RSpec**
235
290
  - Config: (spec/rails_helper.rb)
236
291
  ```ruby
237
-
292
+
238
293
  # when using google service
239
294
  require 'pub_sub_model_sync/mock_google_service'
240
295
  config.before(:each) do
241
296
  google_mock = PubSubModelSync::MockGoogleService.new
242
297
  allow(Google::Cloud::Pubsub).to receive(:new).and_return(google_mock)
243
298
  end
244
-
299
+
245
300
  # when using rabbitmq service
246
- require 'pub_sub_model_sync/mock_rabbit_service'
301
+ require 'pub_sub_model_sync/mock_rabbit_service'
247
302
  config.before(:each) do
248
303
  rabbit_mock = PubSubModelSync::MockRabbitService.new
249
304
  allow(Bunny).to receive(:new).and_return(rabbit_mock)
250
305
  end
251
-
306
+
252
307
  # when using apache kafka service
253
- require 'pub_sub_model_sync/mock_kafka_service'
308
+ require 'pub_sub_model_sync/mock_kafka_service'
254
309
  config.before(:each) do
255
310
  kafka_mock = PubSubModelSync::MockKafkaService.new
256
311
  allow(Kafka).to receive(:new).and_return(kafka_mock)
257
312
  end
258
-
313
+
259
314
  ```
260
315
  - Examples:
261
316
  ```ruby
@@ -266,7 +321,7 @@ Note: Be careful with collision of names
266
321
  payload.process!
267
322
  expect(User.where(id: data[:id]).any?).to be_truth
268
323
  end
269
-
324
+
270
325
  it 'receive class message' do
271
326
  data = { msg: 'hello' }
272
327
  action = :greeting
@@ -274,16 +329,16 @@ Note: Be careful with collision of names
274
329
  payload.process!
275
330
  expect(User).to receive(action)
276
331
  end
277
-
332
+
278
333
  # Publisher
279
334
  it 'publish model action' do
280
- publisher = PubSubModelSync::MessagePublisher
335
+ publisher = PubSubModelSync::MessagePublisher
281
336
  user = User.create(name: 'name', email: 'email')
282
337
  expect(publisher).to receive(:publish_model).with(user, :create, anything)
283
338
  end
284
-
339
+
285
340
  it 'publish class message' do
286
- publisher = PubSubModelSync::MessagePublisher
341
+ publisher = PubSubModelSync::MessagePublisher
287
342
  data = {msg: 'hello'}
288
343
  action = :greeting
289
344
  User.ps_class_publish(data, action: action)
@@ -291,34 +346,34 @@ Note: Be careful with collision of names
291
346
  end
292
347
  ```
293
348
 
294
- ## Extra configurations
349
+ ## **Extra configurations**
295
350
  ```ruby
296
351
  config = PubSubModelSync::Config
297
352
  config.debug = true
298
353
  ```
299
354
 
300
- - ```.subscription_name = 'app-2'```
355
+ - ```.subscription_name = 'app-2'```
301
356
  Permit to define a custom consumer identifier (Default: Rails application name)
302
- - ```.debug = true```
357
+ - ```.debug = true```
303
358
  (true/false*) => show advanced log messages
304
- - ```.logger = Rails.logger```
359
+ - ```.logger = Rails.logger```
305
360
  (Logger) => define custom logger
306
- - ```.disabled_callback_publisher = ->(_model, _action) { false }```
307
- (true/false*) => if true, does not listen model callbacks for auto sync (Create/Update/Destroy)
308
- - ```.on_before_processing = ->(payload, {subscriber:}) { puts payload }```
309
- (Proc) => called before processing received message (:cancel can be returned to skip processing)
310
- - ```.on_success_processing = ->(payload, {subscriber:}) { puts payload }```
361
+ - ```.disabled_callback_publisher = ->(_model, _action) { false }```
362
+ (true/false*) => if true, does not listen model callbacks for auto sync (Create/Update/Destroy)
363
+ - ```.on_before_processing = ->(payload, {subscriber:}) { puts payload }```
364
+ (Proc) => called before processing received message (:cancel can be returned to skip processing)
365
+ - ```.on_success_processing = ->(payload, {subscriber:}) { puts payload }```
311
366
  (Proc) => called when a message was successfully processed
312
- - ```.on_error_processing = ->(exception, {payload:, subscriber:}) { payload.delay(...).process! }```
367
+ - ```.on_error_processing = ->(exception, {payload:, subscriber:}) { payload.delay(...).process! }```
313
368
  (Proc) => called when a message failed when processing (delayed_job or similar can be used for retrying)
314
- - ```.on_before_publish = ->(payload) { puts payload }```
315
- (Proc) => called before publishing a message (:cancel can be returned to skip publishing)
316
- - ```.on_after_publish = ->(payload) { puts payload }```
369
+ - ```.on_before_publish = ->(payload) { puts payload }```
370
+ (Proc) => called before publishing a message (:cancel can be returned to skip publishing)
371
+ - ```.on_after_publish = ->(payload) { puts payload }```
317
372
  (Proc) => called after publishing a message
318
- - ```.on_error_publish = ->(exception, {payload:}) { payload.delay(...).publish! }```
373
+ - ```.on_error_publish = ->(exception, {payload:}) { payload.delay(...).publish! }```
319
374
  (Proc) => called when failed publishing a message (delayed_job or similar can be used for retrying)
320
-
321
- ## TODO
375
+
376
+ ## **TODO**
322
377
  - Add alias attributes when subscribing (similar to publisher)
323
378
  - Add flag ```model.ps_process_payload``` to retrieve the payload used to process the pub/sub sync
324
379
  - Auto publish update only if payload has changed
@@ -327,50 +382,44 @@ config.debug = true
327
382
  PubSubModelSync::MessagePublisher.batch_publish({ same_keys: :use_last_as_first|:use_last|:use_first_as_last|:keep*, same_data: :use_last_as_first*|:use_last|:use_first_as_last|:keep })
328
383
  - Add DB table to use as a shield to skip publishing similar notifications or publish partial notifications (similar idea when processing notif)
329
384
  - add callback: on_message_received(payload)
330
- - Sometimes the listener service stops listening, debug idea:
331
- ```subscriber.on_error do
332
- log('error gogooogle')
333
- end
334
- loop{ log(['loooooop:', {l: subscriber.last_error, stopped: subscriber.stopped?, started: subscriber.started?}]); sleep 2 }
335
- ```
336
385
 
337
- ## Q&A
338
- - Error "could not obtain a connection from the pool within 5.000 seconds"
339
- This problem occurs because pub/sub dependencies (kafka, google-pubsub, rabbitmq) use many threads to perform notifications where the qty of threads is greater than qty of DB pools ([Google pubsub info](https://github.com/googleapis/google-cloud-ruby/blob/master/google-cloud-pubsub/lib/google/cloud/pubsub/subscription.rb#L888))
386
+ ## **Q&A**
387
+ - I'm getting error "could not obtain a connection from the pool within 5.000 seconds"... what does this mean?
388
+ This problem occurs because pub/sub dependencies (kafka, google-pubsub, rabbitmq) use many threads to perform notifications where the qty of threads is greater than qty of DB pools ([Google pubsub info](https://github.com/googleapis/google-cloud-ruby/blob/master/google-cloud-pubsub/lib/google/cloud/pubsub/subscription.rb#L888))
340
389
  To fix the problem, edit config/database.yml and increase the quantity of ```pool: 10```
341
- - Google pubsub: How to process notifications parallely and not sequentially (default 1 thread)?
342
- ```ruby PubSubModelSync::ServiceGoogle::LISTEN_SETTINGS = { threads: { callback: qty_threads } } ```
390
+ - Google pubsub: How to process notifications parallely and not sequentially (default 1 thread)?
391
+ ```ruby PubSubModelSync::ServiceGoogle::LISTEN_SETTINGS = { threads: { callback: qty_threads } } ```
343
392
  Note: by this way some notifications can be processed before others thus missing relationship errors can appear
344
393
  - How to retry failed syncs with sidekiq?
345
394
  ```ruby
346
395
  # lib/initializers/pub_sub_config.rb
347
-
396
+
348
397
  class PubSubRecovery
349
398
  include Sidekiq::Worker
350
399
  sidekiq_options queue: :pubsub, retry: 2, backtrace: true
351
-
400
+
352
401
  def perform(payload_data, action)
353
402
  payload = PubSubModelSync::Payload.from_payload_data(payload_data)
354
403
  payload.send(action)
355
404
  end
356
405
  end
357
-
406
+
358
407
  PubSubModelSync::Config.on_error_publish = lambda do |_e, data|
359
408
  PubSubRecovery.perform_async(data[:payload].to_h, :publish!)
360
409
  end
361
410
  PubSubModelSync::Config.on_error_processing = lambda do |_e, data|
362
411
  PubSubRecovery.perform_async(data[:payload].to_h, :process!)
363
412
  end
364
- ```
413
+ ```
365
414
 
366
- ## Contributing
415
+ ## **Contributing**
367
416
 
368
417
  Bug reports and pull requests are welcome on GitHub at https://github.com/owen2345/pub_sub_model_sync. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
369
418
 
370
- ## License
419
+ ## **License**
371
420
 
372
421
  The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
373
422
 
374
- ## Code of Conduct
423
+ ## **Code of Conduct**
375
424
 
376
425
  Everyone interacting in the PubSubModelSync project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/pub_sub_model_sync/blob/master/CODE_OF_CONDUCT.md).
@@ -15,10 +15,10 @@ module PubSubModelSync
15
15
  end
16
16
 
17
17
  def retry_error(error_klass, qty: 2, &block)
18
- @retries ||= 0
18
+ retries ||= 0
19
19
  block.call
20
20
  rescue error_klass => _e
21
- (@retries += 1) <= qty ? retry : raise
21
+ (retries += 1) <= qty ? retry : raise
22
22
  end
23
23
  end
24
24
  end
@@ -24,21 +24,32 @@ module PubSubModelSync
24
24
 
25
25
  # @param (String: Payload in json format)
26
26
  def process_message(payload_info)
27
+ retries ||= 0
27
28
  payload = parse_payload(payload_info)
28
- log("Received message: #{[payload]}") if config.debug
29
- if same_app_message?(payload)
30
- log("Skip message from same origin: #{[payload]}") if config.debug
29
+ return payload.process unless same_app_message?(payload)
30
+
31
+ log("Skipping message from same origin: #{[payload]}") if config.debug
32
+ rescue => e
33
+ retry if can_retry_process_message?(e, payload, retries += 1)
34
+ end
35
+
36
+ def can_retry_process_message?(error, payload, retries)
37
+ error_payload = [payload, error.message, error.backtrace]
38
+ if retries == 1
39
+ log("Error while starting to process message (retrying...): #{error_payload}", :error)
40
+ rescue_database_connection if lost_db_connection_err?(error)
31
41
  else
32
- payload.process
42
+ log("Retried 1 time and error persists, exiting...: #{error_payload}", :error)
43
+ Process.exit!(true)
33
44
  end
34
- rescue => e
35
- error = [payload, e.message, e.backtrace]
36
- log("Error parsing received message: #{error}", :error)
45
+ retries == 1
37
46
  end
38
47
 
39
48
  def parse_payload(payload_info)
40
49
  info = JSON.parse(payload_info).deep_symbolize_keys
41
- ::PubSubModelSync::Payload.new(info[:data], info[:attributes], info[:headers])
50
+ payload = ::PubSubModelSync::Payload.new(info[:data], info[:attributes], info[:headers])
51
+ log("Received message: #{[payload]}") if config.debug
52
+ payload
42
53
  end
43
54
 
44
55
  # @param payload (Payload)
@@ -46,5 +57,19 @@ module PubSubModelSync
46
57
  key = payload.headers[:app_key]
47
58
  key && key == config.subscription_key
48
59
  end
60
+
61
+ def lost_db_connection_err?(error)
62
+ return true if error.class.name == 'PG::UnableToSend' # rubocop:disable Style/ClassEqualityComparison
63
+
64
+ error.message.match?(/lost connection/i)
65
+ end
66
+
67
+ def rescue_database_connection
68
+ log('Lost DB connection. Attempting to reconnect...', :warn)
69
+ ActiveRecord::Base.connection.reconnect!
70
+ rescue
71
+ log('Cannot reconnect to database, exiting...', :error)
72
+ Process.exit!(true)
73
+ end
49
74
  end
50
75
  end
@@ -36,18 +36,23 @@ module PubSubModelSync
36
36
  # support for: create, update, destroy
37
37
  def run_model_message
38
38
  model = find_model
39
- return if model.ps_before_save_sync(payload) == :cancel
39
+ model.ps_processed_payload = payload
40
40
 
41
41
  if action == :destroy
42
- model.destroy!
42
+ model.destroy! if ensure_sync(model)
43
43
  else
44
44
  populate_model(model)
45
- return if action == :update && !model.ps_subscriber_changed?(payload.data)
46
-
47
- model.save!
45
+ model.save! if ensure_sync(model)
48
46
  end
49
47
  end
50
48
 
49
+ def ensure_sync(model)
50
+ config = PubSubModelSync::Config
51
+ cancelled = model.ps_before_save_sync(payload) == :cancel
52
+ config.log("Cancelled sync with ps_before_save_sync: #{[payload]}") if cancelled && config.debug
53
+ !cancelled
54
+ end
55
+
51
56
  def find_model
52
57
  model_class = klass.constantize
53
58
  return model_class.ps_find_model(payload.data) if model_class.respond_to?(:ps_find_model)
@@ -4,12 +4,7 @@ module PubSubModelSync
4
4
  module SubscriberConcern
5
5
  def self.included(base)
6
6
  base.extend(ClassMethods)
7
- end
8
-
9
- # check if model was changed to skip nonsense .update!()
10
- def ps_subscriber_changed?(_data)
11
- validate
12
- changed?
7
+ base.send(:attr_accessor, :ps_processed_payload)
13
8
  end
14
9
 
15
10
  # permit to apply custom actions before applying sync
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module PubSubModelSync
4
- VERSION = '0.5.9.1'
4
+ VERSION = '0.5.10'
5
5
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: pub_sub_model_sync
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.5.9.1
4
+ version: 0.5.10
5
5
  platform: ruby
6
6
  authors:
7
7
  - Owen
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2021-02-10 00:00:00.000000000 Z
11
+ date: 2021-02-13 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: rails