pub_sub_model_sync 0.5.8.1 → 0.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: f0a2485c0e567e6c012020de909833a5597f5280680b633ea8e9427c51776ddd
4
- data.tar.gz: ba82cc15b42bd39928fc8eb186c08b811bd1f2de8ca9cae1afc8df28653e284b
3
+ metadata.gz: c5b72a2c8c7d97a09c17b3985ce466ecf1fba62573dbb860fae9ccab064e84d9
4
+ data.tar.gz: 4f0f801bc9d51fee36b5f279c4f995d9bf441f608ea26d16314dd2182bf67dc7
5
5
  SHA512:
6
- metadata.gz: 7751c9ffd368b27cd0e6dcbfd5a79900a2e2bb003eb3be7a210be4182fef4bef950a2d793f9edb4280e82fa3a4c602f67842289df286d9c32a69895f6f488613
7
- data.tar.gz: 0f4cfca2f370bd4082537619c9654cf21739e07efaa15ee4b7cedbf3892c3141ce722076edcaae9bda6e65f08028a6819aca8dccc7f8a9ed99d3d0703ee9242b
6
+ metadata.gz: bdb68283d5feca9b506a11478671e8deea6e425065ececefd2162a227b01a7be837fdd533fd5b51e58e60e3d438ffa8e89009f690e0b68696b06cc6e204f5c24
7
+ data.tar.gz: 99baadbbd02b889af06b4cd1b17847a61efe074d58c3f7cc6b0e3eb693f277604f5420b8924bca383cfc8cdbba550d8c16755231335e0a1e056663cb0a5f4deb
@@ -24,7 +24,7 @@ jobs:
24
24
  steps:
25
25
  - uses: actions/checkout@v2
26
26
  - name: Set up Ruby
27
- uses: actions/setup-ruby@v1
27
+ uses: ruby/setup-ruby@v1
28
28
  with:
29
29
  ruby-version: ${{ matrix.ruby }}
30
30
  - name: Install sqlite3
data/CHANGELOG.md CHANGED
@@ -1,5 +1,35 @@
1
1
  # Change Log
2
2
 
3
+ # 0.6.0 (March 03, 2021)
4
+ - feat: add support to include custom payload headers
5
+ - feat: add pubsub transactions to process all payloads inside in the same order they were published
6
+ - feat: when a model is created/updated/destroyed, process all related payloads in a single transaction
7
+ - feat: add method to save processed payload (:ps_processed_payload) when saving sync
8
+ - feat: add "ordering_key" support to process all payloads with the same key in the same order
9
+ - feat: start multiple workers to process async kafka messages when starting service listeners
10
+ - feat: make async publisher by reusing exchange connection (rabbit)
11
+ - feat: add support for forced_ordering_key to always be used as the ordering_key if defined
12
+ - feat: add feature to publish a message to a custom and/or multiple topics
13
+ - feat: add model custom action subscriber and publisher
14
+ - feat: add docker compose settings
15
+
16
+ # 0.5.10 (February 13, 2021)
17
+ - feat: remove duplicated callback :ps_before_save_sync (same result can be achieved with :ps_before_save_sync)
18
+ - feat: improve message starter to retry when failed or exit system when persists
19
+ - feat: fix and retry when database connection error (PG::UnableToSend)
20
+ - feat: add method to save processed payload (:ps_processed_payload) when saving sync
21
+ - chore: improved readme (Thanks @CharlieIGG)
22
+
23
+ # 0.5.9.1 (February 10, 2021)
24
+ - feat: move :key into headers
25
+
26
+ # 0.5.9 (February 10, 2021)
27
+ - feat: reformat :publish and :process methods to include non silence methods
28
+ - feat: add notification key to payloads (can be used for caching strategies)
29
+
30
+ # 0.5.8.2 (February 05, 2021)
31
+ - fix: restore google pubsub topic settings
32
+
3
33
  # 0.5.8.1 (February 05, 2021)
4
34
  - fix: keep message ordering with google pubsub
5
35
 
data/Dockerfile ADDED
@@ -0,0 +1,6 @@
1
+ FROM ruby:2.7-buster
2
+ RUN apt-get update -qq
3
+ WORKDIR /myapp
4
+ COPY . /myapp
5
+ RUN gem update bundler
6
+ RUN bundle install
data/Gemfile.lock CHANGED
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- pub_sub_model_sync (0.5.8.1)
4
+ pub_sub_model_sync (0.6.0)
5
5
  rails
6
6
 
7
7
  GEM
@@ -243,6 +243,7 @@ DEPENDENCIES
243
243
  database_cleaner-active_record
244
244
  google-cloud-pubsub (> 2.0)
245
245
  pub_sub_model_sync!
246
+ rails (~> 6)
246
247
  rake
247
248
  rspec
248
249
  rubocop (~> 1.6.0)
data/README.md CHANGED
@@ -1,16 +1,45 @@
1
- # PubSubModelSync
2
- Permit to sync models data and make calls between rails apps using google or rabbitmq or apache kafka pub/sub service.
3
-
4
- Note: This gem is based on [MultipleMan](https://github.com/influitive/multiple_man) which for now looks unmaintained.
5
-
6
- ## Features
1
+ # **PubSubModelSync**
2
+ Automatically sync Model data and make calls between Rails applications using Google PubSub, RabbitMQ, or Apache Kafka Pub/Sub services.
3
+
4
+ Note: This gem is based on [MultipleMan](https://github.com/influitive/multiple_man) is now unmaintained.
5
+
6
+ - [**PubSubModelSync**](#pubsubmodelsync)
7
+ - [**Features**](#features)
8
+ - [**Installation**](#installation)
9
+ - [**Configuration**](#configuration)
10
+ - [**Notifications Diagram**](#notifications-diagram)
11
+ - [**Basic Example**](#basic-example)
12
+ - [**Advanced Example**](#advanced-example)
13
+ - [**API**](#api)
14
+ - [**Subscribers**](#subscribers)
15
+ - [**Registering Subscription Callbacks**](#registering-subscription-callbacks)
16
+ - [**Class Methods**](#class-methods)
17
+ - [**Instance Methods**](#instance-methods)
18
+ - [**Publishers**](#publishers)
19
+ - [**Registering Publishing Callbacks**](#registering-publishing-callbacks)
20
+ - [**Instance Methods**](#instance-methods-1)
21
+ - [**Class Methods**](#class-methods-1)
22
+ - [**Payload actions**](#payload-actions)
23
+ - [**Testing with RSpec**](#testing-with-rspec)
24
+ - [**Extra configurations**](#extra-configurations)
25
+ - [**TODO**](#todo)
26
+ - [**Q&A**](#qa)
27
+ - [**Contributing**](#contributing)
28
+ - [**License**](#license)
29
+ - [**Code of Conduct**](#code-of-conduct)
30
+
31
+ ## **Features**
7
32
  - Sync CRUD operations between Rails apps. So, all changes made on App1, will be reflected on App2, App3.
8
33
  Example: If User is created on App1, this user will be created on App2 too with the accepted attributes.
9
34
  - Ability to make class level communication
10
35
  Example: If User from App1 wants to generate_email, this can be listened on App2, App3, ... to make corresponding actions
11
36
  - Change pub/sub service at any time
37
+ - Support for transactions: Permits to group all payloads with the same ordering_key and be processed in the same order they are published by the subscribers.
38
+ Grouping by ordering_key allows us to enable multiple workers in our Pub/Sub service(s), and still guarantee that related payloads will be processed in the correct order, despite of the multiple threads.
39
+ This thanks to the fact that Pub/Sub services will always send messages with the same `ordering_key` into the same worker/thread.
40
+ - Ability to send notifications to a specific topic or multiple topics
12
41
 
13
- ## Installation
42
+ ## **Installation**
14
43
  Add this line to your application's Gemfile:
15
44
  ```ruby
16
45
  gem 'pub_sub_model_sync'
@@ -22,15 +51,16 @@ gem 'ruby-kafka' # to use apache kafka pub/sub service
22
51
  And then execute: $ bundle install
23
52
 
24
53
 
25
- ## Usage
54
+ ## **Configuration**
26
55
 
27
56
  - Configuration for google pub/sub (You need google pub/sub service account)
28
57
  ```ruby
29
58
  # initializers/pub_sub_config.rb
30
- PubSubModelSync::Config.service_name = :google
59
+ PubSubModelSync::Config.service_name = :google
31
60
  PubSubModelSync::Config.project = 'google-project-id'
32
61
  PubSubModelSync::Config.credentials = 'path-to-the-config'
33
- PubSubModelSync::Config.topic_name = 'sample-topic'
62
+ PubSubModelSync::Config.topic_name = 'sample-topic'
63
+ PubSubModelSync::Config.subscription_name = 'my-app1'
34
64
  ```
35
65
  See details here:
36
66
  https://github.com/googleapis/google-cloud-ruby/tree/master/google-cloud-pubsub
@@ -39,8 +69,8 @@ And then execute: $ bundle install
39
69
  ```ruby
40
70
  PubSubModelSync::Config.service_name = :rabbitmq
41
71
  PubSubModelSync::Config.bunny_connection = 'amqp://guest:guest@localhost'
42
- PubSubModelSync::Config.queue_name = 'model-sync'
43
72
  PubSubModelSync::Config.topic_name = 'sample-topic'
73
+ PubSubModelSync::Config.subscription_name = 'my-app2'
44
74
  ```
45
75
  See details here: https://github.com/ruby-amqp/bunny
46
76
 
@@ -49,30 +79,28 @@ And then execute: $ bundle install
49
79
  PubSubModelSync::Config.service_name = :kafka
50
80
  PubSubModelSync::Config.kafka_connection = [["kafka1:9092", "localhost:2121"], { logger: Rails.logger }]
51
81
  PubSubModelSync::Config.topic_name = 'sample-topic'
82
+ PubSubModelSync::Config.subscription_name = 'my-app3'
52
83
  ```
53
- See details here: https://github.com/zendesk/ruby-kafka
84
+ See details here: https://github.com/zendesk/ruby-kafka
54
85
 
55
86
  - Add publishers/subscribers to your models (See examples below)
56
87
 
57
88
  - Start subscribers to listen for publishers (Only in the app that has subscribers)
58
- ```ruby
59
- rake pub_sub_model_sync:start
89
+ ```bash
90
+ DB_POOL=20 bundle exec rake pub_sub_model_sync:start
60
91
  ```
61
- Note: Publishers do not need todo this
62
- Note2 (Rails 6+): Due to Zeitwerk, you need to load listeners manually when syncing without mentioned task (like rails console)
63
- ```ruby
64
- # PubSubModelSync::Config.subscribers ==> []
65
- PubSubModelSync::Runner.preload_listeners
66
- # PubSubModelSync::Config.subscribers ==> [#<PubSubModelSync::Subscriber:0x000.. @klass="Article", @action=:create..., ....]
67
- ```
68
-
69
- - Check the service status with:
92
+ Note: You need more than 15 DB pools to avoid "could not obtain a connection from the pool within 5.000 seconds". https://devcenter.heroku.com/articles/concurrency-and-database-connections
93
+
94
+ - Check the service status with:
70
95
  ```PubSubModelSync::MessagePublisher.publish_data('Test message', {sample_value: 10}, :create)```
71
96
 
72
- ## Examples
97
+ ## **Notifications Diagram**
98
+ ![Diagram](/docs/notifications-diagram.png?raw=true)
99
+
100
+ ## **Basic Example**
73
101
  ```ruby
74
102
  # App 1 (Publisher)
75
- # attributes: name email age
103
+ # attributes: name email age
76
104
  class User < ActiveRecord::Base
77
105
  include PubSubModelSync::PublisherConcern
78
106
  ps_publish(%i[id name email])
@@ -81,34 +109,39 @@ end
81
109
  # App 2 (Subscriber)
82
110
  class User < ActiveRecord::Base
83
111
  include PubSubModelSync::SubscriberConcern
84
- ps_subscribe(%i[name])
85
- ps_class_subscribe(:greeting)
112
+ ps_subscribe(%i[name]) # crud notifications
113
+ ps_subscribe_custom(:say_welcome) # custom instance notification
114
+ ps_class_subscribe(:greeting) # class notification
86
115
 
87
116
  def self.greeting(data)
88
117
  puts 'Class message called'
89
118
  end
119
+
120
+ def say_welcome(data)
121
+ UserMailer.deliver(id, data)
122
+ end
90
123
  end
91
124
 
92
125
  # Samples
93
126
  User.create(name: 'test user', email: 'sample@gmail.com') # Review your App 2 to see the created user (only name will be saved)
94
127
  User.new(name: 'test user').ps_perform_sync(:create) # similar to above to perform sync on demand
95
128
 
96
- User.ps_class_publish({ msg: 'Hello' }, action: :greeting) # User.greeting method (Class method) will be called in App2
97
- PubSubModelSync::MessagePublisher.publish_data(User, { msg: 'Hello' }, :greeting) # similar to above when not included publisher concern
129
+ PubSubModelSync::MessagePublisher.publish_model_data(my_user, { id:10, msg: 'Hello' }, :say_welcome, { as_klass: 'RegisteredUser' }) # custom model action notification
130
+ PubSubModelSync::MessagePublisher.publish_data(User, { msg: 'Hello' }, :greeting) # custom data notification
98
131
  ```
99
132
 
100
- ## Advanced Example
133
+ ## **Advanced Example**
101
134
  ```ruby
102
135
  # App 1 (Publisher)
103
136
  class User < ActiveRecord::Base
104
137
  self.table_name = 'publisher_users'
105
138
  include PubSubModelSync::PublisherConcern
106
- ps_publish(%i[id:client_id name:full_name email], actions: %i[update], as_klass: 'Client')
107
-
139
+ ps_publish(%i[id:client_id name:full_name email], actions: %i[update], as_klass: 'Client', headers: { topic_name: ['topic1', 'topic N'] })
140
+
108
141
  def ps_skip_callback?(_action)
109
142
  false # here logic with action to skip push message
110
143
  end
111
-
144
+
112
145
  def ps_skip_sync?(_action)
113
146
  false # here logic with action to skip push message
114
147
  end
@@ -120,130 +153,249 @@ class User < ActiveRecord::Base
120
153
  include PubSubModelSync::SubscriberConcern
121
154
  ps_subscribe(%i[name], actions: %i[update], from_klass: 'Client', id: %i[client_id email])
122
155
  ps_class_subscribe(:greeting, from_action: :custom_greeting, from_klass: 'CustomUser')
156
+ ps_subscribe_custom(:send_welcome, from_klass: 'CustomUser', id: :id, from_action: :say_welcome)
123
157
  alias_attribute :full_name, :name
124
-
158
+
125
159
  def self.greeting(data)
126
160
  puts 'Class message called through custom_greeting'
127
161
  end
128
162
 
163
+ def send_welcome(data)
164
+ UserMailer.deliver(id, data)
165
+ end
166
+
129
167
  # def self.ps_find_model(data)
130
- # where(email: data[:email], ...).first_or_initialize
168
+ # where(email: data[:email], ...).first_or_initialize
131
169
  # end
132
170
  end
133
171
  ```
134
172
 
135
- Note: Be careful with collision of names
136
- ```
137
- # ps_publish %i[name_data:name name:key] # key will be replaced with name_data
138
- ps_publish %i[name_data:name key_data:key] # use alias to avoid collision
139
- ```
140
-
141
- ## API
142
- ### Subscribers
143
- - Permit to configure class level subscriptions
144
- ```ps_class_subscribe(action_name, from_action: nil, from_klass: nil)```
145
- * from_action: (Optional) Source method name
146
- * from_klass: (Optional) Source class name
173
+ ## **API**
174
+ ### **Subscribers**
175
+
176
+ #### **Registering Subscriptions**
177
+
178
+ - Configure class subscriptions
179
+ ```ruby
180
+ class MyModel < ActiveRecord::Base
181
+ ps_class_subscribe(action_name, from_action: nil, from_klass: nil)
182
+ end
183
+ ```
184
+ When Class receives the corresponding notification, `action` method will be called on the Class. Like: `User.action(data)`
185
+ * `action_name`: (String|Sym/Optional) Action name
186
+ * `from_klass`: (String/Optional) Source class name (Default `model.class.name`)
187
+ * `from_action`: (Sym/Optional) Source method name. Default `action`
188
+
189
+ - Configure CRUD subscriptions
190
+ ```ruby
191
+ class MyModel < ActiveRecord::Base
192
+ ps_subscribe(attrs, from_klass: nil, actions: nil, id: nil)
193
+ end
194
+ ```
195
+ When model receives the corresponding notification, `action` method will be called on the model. Like: `model.destroy`
196
+ * `attrs`: (Array/Required) Array of all attributes to be synced
197
+ * `from_klass`: (String/Optional) Source class name (Default `model.class.name`)
198
+ * `actions`: (Array/Optional, default: create/update/destroy) permit to customize action names
199
+ * `id`: (Sym|Array/Optional, default: id) Attr identifier(s) to find the corresponding model
200
+
201
+ - Configure custom model subscriptions
202
+ ```ruby
203
+ class MyModel < ActiveRecord::Base
204
+ ps_subscribe_custom(action, from_klass: name, id: :id, from_action: nil)
205
+ end
206
+ ```
207
+ When model receives the corresponding notification, `action` method will be called on the model. Like: `model.action(data)`
208
+ * `action`: (String/Required) Action name
209
+ * `from_klass`: (String/Optional) Source class name (Default `model.class.name`)
210
+ * `from_action`: (Sym/Optional) Source method name. Default `action`
211
+ * `id`: (Sym|Array/Optional, default: id) Attr identifier(s) to find the corresponding model
212
+
213
+ - Perform custom actions before saving sync of the model (`:cancel` can be returned to skip sync)
214
+ ```ruby
215
+ class MyModel < ActiveRecord::Base
216
+ def ps_before_save_sync(action, payload)
217
+ # puts payload.data[:id]
218
+ end
219
+ end
220
+ ```
147
221
 
148
- - Permit to configure instance level subscriptions (CRUD)
149
- ```ps_subscribe(attrs, from_klass: nil, actions: nil, id: nil)```
150
- * attrs: (Array/Required) Array of all attributes to be synced
151
- * from_klass: (String/Optional) Source class name (Instead of the model class name, will use this value)
152
- * actions: (Array/Optional, default: create/update/destroy) permit to customize action names
153
- * id: (Sym|Array/Optional, default: id) Attr identifier(s) to find the corresponding model
154
-
155
- - Permit to configure a custom model finder
156
- ```ps_find_model(data)```
157
- * data: (Hash) Data received from sync
222
+ - Configure a custom model finder (optional)
223
+ ```ruby
224
+ class MyModel < ActiveRecord::Base
225
+ def ps_find_model(data)
226
+ where(custom_finder: data[:custom_value]).first_or_initialize
227
+ end
228
+ end
229
+ ```
230
+ * `data`: (Hash) Data received from sync
158
231
  Must return an existent or a new model object
159
232
 
160
- - Get crud subscription configured for the class
161
- ```User.ps_subscriber(action_name)```
162
- * action_name (default :create, :sym): can be :create, :update, :destroy
233
+ #### **Subscription helpers**
234
+ - Inspect all configured subscriptions
235
+ ```ruby
236
+ PubSubModelSync::Config.subscribers
237
+ ```
238
+ - Manually process or reprocess a notification
239
+ ```ruby
240
+ payload = PubSubModelSync::Payload.new(data, attributes, headers)
241
+ payload.process!
242
+ ```
163
243
 
164
- - Inspect all configured subscribers
165
- ```PubSubModelSync::Config.subscribers```
166
244
 
167
- - Permit to customize the way to detect if the subscribed model was changed (Only for update action).
168
- ```.ps_subscriber_changed?(data)```
169
- By default: ```model.changed?```
245
+ ### **Publishers**
170
246
 
171
- - Permit to perform custom actions before saving sync of the model (:cancel can be returned to skip sync)
172
- ```.ps_before_save_sync(payload)```
247
+ #### **Registering Publishers **
248
+ - Register CRUD publishers that will trigger configured notifications
249
+ ```ruby
250
+ class MyModel < ActiveRecord::Base
251
+ ps_publish([:id, 'created_at:published_at', :full_name], actions: [:update], as_klass: nil, headers: { ordering_key: 'custom-key', topic_name: 'my-custom-topic' })
252
+ def full_name
253
+ [first_name, last_name].join(' ')
254
+ end
255
+ end
256
+ ```
257
+ * `attrs`: (Array/Required) Array of attributes to be published. Supports for:
258
+ - aliases: permits to publish with different names, sample: "created_at:published_at" where "created_at" will be published as "published_at"
259
+ - methods: permits to publish method values as attributes, sample: "full_name"
260
+ * `actions`: (Array/Optional, default: %i[create update destroy]) permit to define action names
261
+ * `as_klass`: (String/Optional) Output class name (Instead of the model class name, will use this value)
262
+ * `headers`: (Hash/Optional) Notification settings which permit to customize the way and the target of the notification (Refer Payload.headers)
263
+
173
264
 
174
- ### Publishers
175
- - Permit to configure crud publishers
176
- ```ps_publish(attrs, actions: nil, as_klass: nil)```
177
- * attrs: (Array/Required) Array of attributes to be published
178
- * actions: (Array/Optional, default: create/update/destroy) permit to customize action names
179
- * as_klass: (String/Optional) Output class name (Instead of the model class name, will use this value)
265
+ #### **Publishing notifications**
266
+ - CRUD notifications
267
+ ```ruby
268
+ MyModel.create!(...)
269
+ ```
270
+ "Create" notification will be delivered with the configured attributes as the payload data
180
271
 
181
- - Permit to cancel sync called after create/update/destroy (Before initializing sync service)
182
- ```model.ps_skip_callback?(action)```
183
- Default: False
184
- Note: Return true to cancel sync
272
+ - Manual CRUD notifications
273
+ ```ruby
274
+ MyModel.ps_perform_sync(action, custom_data: {}, custom_headers: {})
275
+ ```
276
+ * `action`: (Sym) CRUD action name (create, update or destroy)
277
+ * `custom_data`: custom_data (nil|Hash) If present custom_data will be used as the payload data. I.E. data generator will be ignored
278
+ * `custom_headers`: (Hash, optional) override default headers. Refer `payload.headers`
185
279
 
186
- - Callback called before preparing data for sync (Permit to stop sync)
187
- ```model.ps_skip_sync?(action)```
188
- Note: return true to cancel sync
280
+ - Class notifications
281
+ ```ruby
282
+ PubSubModelSync::MessagePublisher.publish_data((klass, data, action, headers: )
283
+ ```
284
+ Publishes any data to be listened at a class level.
285
+ - `klass`: (String) Class name to be used
286
+ - `data`: (Hash) Data to be delivered
287
+ - `action`: (Sym) Action name
288
+ - `headers`: (Hash, optional) Notification settings (Refer Payload.headers)
289
+
290
+ - Model custom action notifications
291
+ ```ruby
292
+ PubSubModelSync::MessagePublisher.publish_model_data(model, data, action, as_klass:, headers:)
293
+ ```
294
+ Publishes model custom action to be listened at an instance level.
295
+ - `model`: (ActiveRecord) model owner of the data
296
+ - `data`: (Hash) Data to be delivered
297
+ - `action`: (Sym) Action name
298
+ - `as_klass`: (String, optional) if not provided, `model.class.name` will be used instead
299
+ - `headers`: (Hash, optional) Notification settings (Refer Payload.headers)
300
+
301
+ - Manually publish or republish a notification
302
+ ```ruby
303
+ payload = PubSubModelSync::Payload.new(data, attributes, headers)
304
+ payload.publish!
305
+ ```
306
+
307
+ #### ** publishing callbacks**
308
+
309
+ - Prevent CRUD sync at model callback level (Called right after :after_create, :after_update, :after_destroy).
310
+ If returns "true", sync will be cancelled.
311
+ ```ruby
312
+ class MyModel < ActiveRecord::Base
313
+ def ps_skip_callback?(action)
314
+ # logic here
315
+ end
316
+ end
317
+ ```
318
+
319
+ - Prevent CRUD sync before processing payload (Affects model.ps_perform_sync(...))).
320
+ If returns "true", sync will be cancelled
321
+ ```ruby
322
+ class MyModel < ActiveRecord::Base
323
+ def ps_skip_sync?(action)
324
+ # logic here
325
+ end
326
+ end
327
+ ```
328
+
329
+ - Do some actions before publishing a CRUD notification.
330
+ If returns ":cancel", sync will be cancelled
331
+ ```ruby
332
+ class MyModel < ActiveRecord::Base
333
+ def ps_before_sync(action, payload)
334
+ # logic here
335
+ end
336
+ end
337
+ ```
338
+
339
+ - Do some actions after CRUD notification was published.
340
+ ```ruby
341
+ class MyModel < ActiveRecord::Base
342
+ def ps_after_sync(action, payload)
343
+ # logic here
344
+ end
345
+ end
346
+ ```
347
+
348
+
349
+ ### **Payload**
350
+ Any notification before delivering is transformed as a Payload for a better portability.
351
+
352
+ - Initialize
353
+ ```ruby
354
+ payload = PubSubModelSync::Payload.new(data, attributes, headers)
355
+ ```
356
+ * `data`: (Hash) Data to be published or processed
357
+ * `attributes`: (Hash) Includes class and method info
358
+ - `action`: (String) action name
359
+ - `klass`: (String) class name
360
+ * `headers`: (Hash) Notification settings that defines how the notification will be processed or delivered.
361
+ - `key`: (String, optional) identifier of the payload, default: `<klass_name>/<action>` when class message, `<model.class.name>/<action>/<model.id>` when model message (Useful for caching techniques).
362
+ - `ordering_key`: (String, optional): messages with the same key are processed in the same order they were delivered, default: `klass_name` when class message, `<model.class.name>/<model.id>` when model message
363
+ - `topic_name`: (String|Array<String>, optional): Specific topic name to be used when delivering the message (default first topic from config).
364
+ - `forced_ordering_key`: (String, optional): Will force to use this value as the `ordering_key`, even withing transactions. Default `nil`.
189
365
 
190
- - Callback called before sync (After preparing data)
191
- ```model.ps_before_sync(action, data_to_deliver)```
192
- Note: If the method returns ```:cancel```, the sync will be stopped (message will not be published)
193
-
194
- - Callback called after sync
195
- ```model.ps_after_sync(action, data_delivered)```
196
-
197
- - Perform sync on demand (:create, :update, :destroy):
198
- The target model will receive a notification to perform the indicated action
199
- ```my_model.ps_perform_sync(action_name, custom_settings = {})```
200
- * custom_settings: override default settings defined for action_name ({ attrs: [], as_klass: nil })
201
-
202
- - Publish a class level notification:
203
- ```User.ps_class_publish(data, action: action_name, as_klass: custom_klass_name)```
204
- Target class ```User.action_name``` will be called when message is received
205
- * data: (required, :hash) message value to deliver
206
- * action_name: (required, :sim) Action name
207
- * as_klass: (optional, :string) Custom class name (Default current model name)
208
-
209
- - Payload actions
366
+ - Actions for payloads
210
367
  ```ruby
211
- payload = PubSubModelSync::Payload.new({ title: 'hello' }, { action: :greeting, klass: 'User' })
212
368
  payload.publish! # publishes notification data. It raises exception if fails and does not call ```:on_error_publishing``` callback
213
369
  payload.publish # publishes notification data. On error does not raise exception but calls ```:on_error_publishing``` callback
214
370
  payload.process! # process a notification data. It raises exception if fails and does not call ```.on_error_processing``` callback
215
371
  payload.publish # process a notification data. It does not raise exception if fails but calls ```.on_error_processing``` callback
216
372
  ```
217
-
218
- - Get crud publisher configured for the class
219
- ```User.ps_publisher(action_name)```
220
- * action_name (default :create, :sym): can be :create, :update, :destroy
221
-
222
- ## Testing with RSpec
373
+
374
+ ## **Testing with RSpec**
223
375
  - Config: (spec/rails_helper.rb)
224
376
  ```ruby
225
-
377
+
226
378
  # when using google service
227
379
  require 'pub_sub_model_sync/mock_google_service'
228
380
  config.before(:each) do
229
381
  google_mock = PubSubModelSync::MockGoogleService.new
230
382
  allow(Google::Cloud::Pubsub).to receive(:new).and_return(google_mock)
231
383
  end
232
-
384
+
233
385
  # when using rabbitmq service
234
- require 'pub_sub_model_sync/mock_rabbit_service'
386
+ require 'pub_sub_model_sync/mock_rabbit_service'
235
387
  config.before(:each) do
236
388
  rabbit_mock = PubSubModelSync::MockRabbitService.new
237
389
  allow(Bunny).to receive(:new).and_return(rabbit_mock)
238
390
  end
239
-
391
+
240
392
  # when using apache kafka service
241
- require 'pub_sub_model_sync/mock_kafka_service'
393
+ require 'pub_sub_model_sync/mock_kafka_service'
242
394
  config.before(:each) do
243
395
  kafka_mock = PubSubModelSync::MockKafkaService.new
244
396
  allow(Kafka).to receive(:new).and_return(kafka_mock)
245
397
  end
246
-
398
+
247
399
  ```
248
400
  - Examples:
249
401
  ```ruby
@@ -254,7 +406,7 @@ Note: Be careful with collision of names
254
406
  payload.process!
255
407
  expect(User.where(id: data[:id]).any?).to be_truth
256
408
  end
257
-
409
+
258
410
  it 'receive class message' do
259
411
  data = { msg: 'hello' }
260
412
  action = :greeting
@@ -262,97 +414,97 @@ Note: Be careful with collision of names
262
414
  payload.process!
263
415
  expect(User).to receive(action)
264
416
  end
265
-
417
+
266
418
  # Publisher
267
419
  it 'publish model action' do
268
- publisher = PubSubModelSync::MessagePublisher
420
+ publisher = PubSubModelSync::MessagePublisher
269
421
  user = User.create(name: 'name', email: 'email')
270
422
  expect(publisher).to receive(:publish_model).with(user, :create, anything)
271
423
  end
272
-
424
+
273
425
  it 'publish class message' do
274
- publisher = PubSubModelSync::MessagePublisher
426
+ publisher = PubSubModelSync::MessagePublisher
275
427
  data = {msg: 'hello'}
276
428
  action = :greeting
277
- User.ps_class_publish(data, action: action)
429
+ PubSubModelSync::MessagePublisher.publish_data('User', data, action)
278
430
  expect(publisher).to receive(:publish_data).with('User', data, action)
279
431
  end
280
432
  ```
281
433
 
282
- ## Extra configurations
434
+ ## **Extra configurations**
283
435
  ```ruby
284
436
  config = PubSubModelSync::Config
285
437
  config.debug = true
286
438
  ```
287
-
288
- - ```.subscription_name = 'app-2'```
289
- Permit to define a custom consumer identifier (Default: Rails application name)
290
- - ```.debug = true```
439
+ - `.topic_name = ['topic1', 'topic 2']`: (String|Array<String>)
440
+ Topic name(s) to be used to listen all notifications from when listening. Additional first topic name is used as the default topic name when publishing a notification.
441
+ - `.subscription_name = "my-app-1"`: (String, default Rails.application.name)
442
+ Subscriber's identifier which helps to:
443
+ * skip self messages
444
+ * continue the sync from the last synced notification when service was restarted.
445
+ - ```.debug = true```
291
446
  (true/false*) => show advanced log messages
292
- - ```.logger = Rails.logger```
447
+ - ```.logger = Rails.logger```
293
448
  (Logger) => define custom logger
294
- - ```.disabled_callback_publisher = ->(_model, _action) { false }```
295
- (true/false*) => if true, does not listen model callbacks for auto sync (Create/Update/Destroy)
296
- - ```.on_before_processing = ->(payload, {subscriber:}) { puts payload }```
297
- (Proc) => called before processing received message (:cancel can be returned to skip processing)
298
- - ```.on_success_processing = ->(payload, {subscriber:}) { puts payload }```
449
+ - ```.disabled_callback_publisher = ->(_model, _action) { false }```
450
+ (true/false*) => if true, does not listen model callbacks for auto sync (Create/Update/Destroy)
451
+ - ```.on_before_processing = ->(payload, {subscriber:}) { puts payload }```
452
+ (Proc) => called before processing received message (:cancel can be returned to skip processing)
453
+ - ```.on_success_processing = ->(payload, {subscriber:}) { puts payload }```
299
454
  (Proc) => called when a message was successfully processed
300
- - ```.on_error_processing = ->(exception, {payload:, subscriber:}) { payload.delay(...).process! }```
455
+ - ```.on_error_processing = ->(exception, {payload:, subscriber:}) { payload.delay(...).process! }```
301
456
  (Proc) => called when a message failed when processing (delayed_job or similar can be used for retrying)
302
- - ```.on_before_publish = ->(payload) { puts payload }```
303
- (Proc) => called before publishing a message (:cancel can be returned to skip publishing)
304
- - ```.on_after_publish = ->(payload) { puts payload }```
457
+ - ```.on_before_publish = ->(payload) { puts payload }```
458
+ (Proc) => called before publishing a message (:cancel can be returned to skip publishing)
459
+ - ```.on_after_publish = ->(payload) { puts payload }```
305
460
  (Proc) => called after publishing a message
306
- - ```.on_error_publish = ->(exception, {payload:}) { payload.delay(...).publish! }```
461
+ - ```.on_error_publish = ->(exception, {payload:}) { payload.delay(...).publish! }```
307
462
  (Proc) => called when failed publishing a message (delayed_job or similar can be used for retrying)
308
-
309
- ## TODO
463
+
464
+ ## **TODO**
310
465
  - Add alias attributes when subscribing (similar to publisher)
311
466
  - Add flag ```model.ps_process_payload``` to retrieve the payload used to process the pub/sub sync
312
467
  - Auto publish update only if payload has changed
313
468
  - On delete, payload must only be composed by ids
314
- - Feature to publish multiple message at a time with the ability to exclude similar messages by klass and action (use the last one)
315
- PubSubModelSync::MessagePublisher.batch_publish({ same_keys: :use_last_as_first|:use_last|:use_first_as_last|:keep*, same_data: :use_last_as_first*|:use_last|:use_first_as_last|:keep })
316
- - Add DB table to use as a shield to skip publishing similar notifications or publish partial notifications (similar idea when processing notif)
469
+ - Improve transactions to exclude similar messages by klass and action. Sample:
470
+ ```PubSubModelSync::MessagePublisher.transaction(key, { same_keys: :use_last_as_first|:use_last|:use_first_as_last|:keep*, same_data: :use_last_as_first*|:use_last|:use_first_as_last|:keep })```
471
+ - Add DB table to use as a shield to prevent publishing similar notifications and publish partial notifications (similar idea when processing notif)
317
472
  - add callback: on_message_received(payload)
318
473
 
319
- ## Q&A
320
- - Error "could not obtain a connection from the pool within 5.000 seconds"
321
- This problem occurs because pub/sub dependencies (kafka, google-pubsub, rabbitmq) use many threads to perform notifications where the qty of threads is greater than qty of DB pools ([Google pubsub info](https://github.com/googleapis/google-cloud-ruby/blob/master/google-cloud-pubsub/lib/google/cloud/pubsub/subscription.rb#L888))
322
- To fix the problem, edit config/database.yml and increase the quantity of ```pool: 10```
323
- - Google pubsub: How to process notifications parallely and not sequentially (default 1 thread)?
324
- ```ruby PubSubModelSync::ServiceGoogle::LISTEN_SETTINGS = { threads: { callback: qty_threads } } ```
325
- Note: by this way some notifications can be processed before others thus missing relationship errors can appear
474
+ ## **Q&A**
475
+ - I'm getting error "could not obtain a connection from the pool within 5.000 seconds"... what does this mean?
476
+ This problem occurs because pub/sub dependencies (kafka, google-pubsub, rabbitmq) use many threads to perform notifications where the qty of threads is greater than qty of DB pools ([Google pubsub info](https://github.com/googleapis/google-cloud-ruby/blob/master/google-cloud-pubsub/lib/google/cloud/pubsub/subscription.rb#L888))
477
+ To fix the problem, edit config/database.yml and increase the quantity of ```pool: 20```
326
478
  - How to retry failed syncs with sidekiq?
327
479
  ```ruby
328
480
  # lib/initializers/pub_sub_config.rb
329
-
481
+
330
482
  class PubSubRecovery
331
483
  include Sidekiq::Worker
332
484
  sidekiq_options queue: :pubsub, retry: 2, backtrace: true
333
-
485
+
334
486
  def perform(payload_data, action)
335
487
  payload = PubSubModelSync::Payload.from_payload_data(payload_data)
336
488
  payload.send(action)
337
489
  end
338
490
  end
339
-
491
+
340
492
  PubSubModelSync::Config.on_error_publish = lambda do |_e, data|
341
493
  PubSubRecovery.perform_async(data[:payload].to_h, :publish!)
342
494
  end
343
495
  PubSubModelSync::Config.on_error_processing = lambda do |_e, data|
344
496
  PubSubRecovery.perform_async(data[:payload].to_h, :process!)
345
497
  end
346
- ```
498
+ ```
347
499
 
348
- ## Contributing
500
+ ## **Contributing**
349
501
 
350
502
  Bug reports and pull requests are welcome on GitHub at https://github.com/owen2345/pub_sub_model_sync. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
351
503
 
352
- ## License
504
+ ## **License**
353
505
 
354
506
  The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
355
507
 
356
- ## Code of Conduct
508
+ ## **Code of Conduct**
357
509
 
358
510
  Everyone interacting in the PubSubModelSync project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/pub_sub_model_sync/blob/master/CODE_OF_CONDUCT.md).