pub_sub_model_sync 0.5.8.1 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.github/workflows/ruby.yml +1 -1
- data/CHANGELOG.md +30 -0
- data/Dockerfile +6 -0
- data/Gemfile.lock +2 -1
- data/README.md +306 -154
- data/docker-compose.yaml +12 -0
- data/docs/notifications-diagram.png +0 -0
- data/lib/pub_sub_model_sync/base.rb +17 -4
- data/lib/pub_sub_model_sync/config.rb +1 -1
- data/lib/pub_sub_model_sync/message_processor.rb +13 -7
- data/lib/pub_sub_model_sync/message_publisher.rb +100 -18
- data/lib/pub_sub_model_sync/mock_google_service.rb +4 -0
- data/lib/pub_sub_model_sync/mock_kafka_service.rb +13 -0
- data/lib/pub_sub_model_sync/payload.rb +20 -8
- data/lib/pub_sub_model_sync/publisher.rb +28 -12
- data/lib/pub_sub_model_sync/publisher_concern.rb +37 -20
- data/lib/pub_sub_model_sync/service_base.rb +46 -12
- data/lib/pub_sub_model_sync/service_google.rb +53 -18
- data/lib/pub_sub_model_sync/service_kafka.rb +35 -12
- data/lib/pub_sub_model_sync/service_rabbit.rb +40 -33
- data/lib/pub_sub_model_sync/subscriber.rb +22 -15
- data/lib/pub_sub_model_sync/subscriber_concern.rb +9 -11
- data/lib/pub_sub_model_sync/tasks/worker.rake +11 -0
- data/lib/pub_sub_model_sync/version.rb +1 -1
- metadata +5 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: c5b72a2c8c7d97a09c17b3985ce466ecf1fba62573dbb860fae9ccab064e84d9
|
4
|
+
data.tar.gz: 4f0f801bc9d51fee36b5f279c4f995d9bf441f608ea26d16314dd2182bf67dc7
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: bdb68283d5feca9b506a11478671e8deea6e425065ececefd2162a227b01a7be837fdd533fd5b51e58e60e3d438ffa8e89009f690e0b68696b06cc6e204f5c24
|
7
|
+
data.tar.gz: 99baadbbd02b889af06b4cd1b17847a61efe074d58c3f7cc6b0e3eb693f277604f5420b8924bca383cfc8cdbba550d8c16755231335e0a1e056663cb0a5f4deb
|
data/.github/workflows/ruby.yml
CHANGED
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,35 @@
|
|
1
1
|
# Change Log
|
2
2
|
|
3
|
+
# 0.6.0 (March 03, 2021)
|
4
|
+
- feat: add support to include custom payload headers
|
5
|
+
- feat: add pubsub transactions to process all payloads inside in the same order they were published
|
6
|
+
- feat: when a model is created/updated/destroyed, process all related payloads in a single transaction
|
7
|
+
- feat: add method to save processed payload (:ps_processed_payload) when saving sync
|
8
|
+
- feat: add "ordering_key" support to process all payloads with the same key in the same order
|
9
|
+
- feat: start multiple workers to process async kafka messages when starting service listeners
|
10
|
+
- feat: make async publisher by reusing exchange connection (rabbit)
|
11
|
+
- feat: add support for forced_ordering_key to always be used as the ordering_key if defined
|
12
|
+
- feat: add feature to publish a message to a custom and/or multiple topics
|
13
|
+
- feat: add model custom action subscriber and publisher
|
14
|
+
- feat: add docker compose settings
|
15
|
+
|
16
|
+
# 0.5.10 (February 13, 2021)
|
17
|
+
- feat: remove duplicated callback :ps_before_save_sync (same result can be achieved with :ps_before_save_sync)
|
18
|
+
- feat: improve message starter to retry when failed or exit system when persists
|
19
|
+
- feat: fix and retry when database connection error (PG::UnableToSend)
|
20
|
+
- feat: add method to save processed payload (:ps_processed_payload) when saving sync
|
21
|
+
- chore: improved readme (Thanks @CharlieIGG)
|
22
|
+
|
23
|
+
# 0.5.9.1 (February 10, 2021)
|
24
|
+
- feat: move :key into headers
|
25
|
+
|
26
|
+
# 0.5.9 (February 10, 2021)
|
27
|
+
- feat: reformat :publish and :process methods to include non silence methods
|
28
|
+
- feat: add notification key to payloads (can be used for caching strategies)
|
29
|
+
|
30
|
+
# 0.5.8.2 (February 05, 2021)
|
31
|
+
- fix: restore google pubsub topic settings
|
32
|
+
|
3
33
|
# 0.5.8.1 (February 05, 2021)
|
4
34
|
- fix: keep message ordering with google pubsub
|
5
35
|
|
data/Dockerfile
ADDED
data/Gemfile.lock
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
pub_sub_model_sync (0.
|
4
|
+
pub_sub_model_sync (0.6.0)
|
5
5
|
rails
|
6
6
|
|
7
7
|
GEM
|
@@ -243,6 +243,7 @@ DEPENDENCIES
|
|
243
243
|
database_cleaner-active_record
|
244
244
|
google-cloud-pubsub (> 2.0)
|
245
245
|
pub_sub_model_sync!
|
246
|
+
rails (~> 6)
|
246
247
|
rake
|
247
248
|
rspec
|
248
249
|
rubocop (~> 1.6.0)
|
data/README.md
CHANGED
@@ -1,16 +1,45 @@
|
|
1
|
-
# PubSubModelSync
|
2
|
-
|
3
|
-
|
4
|
-
Note: This gem is based on [MultipleMan](https://github.com/influitive/multiple_man)
|
5
|
-
|
6
|
-
|
1
|
+
# **PubSubModelSync**
|
2
|
+
Automatically sync Model data and make calls between Rails applications using Google PubSub, RabbitMQ, or Apache Kafka Pub/Sub services.
|
3
|
+
|
4
|
+
Note: This gem is based on [MultipleMan](https://github.com/influitive/multiple_man) is now unmaintained.
|
5
|
+
|
6
|
+
- [**PubSubModelSync**](#pubsubmodelsync)
|
7
|
+
- [**Features**](#features)
|
8
|
+
- [**Installation**](#installation)
|
9
|
+
- [**Configuration**](#configuration)
|
10
|
+
- [**Notifications Diagram**](#notifications-diagram)
|
11
|
+
- [**Basic Example**](#basic-example)
|
12
|
+
- [**Advanced Example**](#advanced-example)
|
13
|
+
- [**API**](#api)
|
14
|
+
- [**Subscribers**](#subscribers)
|
15
|
+
- [**Registering Subscription Callbacks**](#registering-subscription-callbacks)
|
16
|
+
- [**Class Methods**](#class-methods)
|
17
|
+
- [**Instance Methods**](#instance-methods)
|
18
|
+
- [**Publishers**](#publishers)
|
19
|
+
- [**Registering Publishing Callbacks**](#registering-publishing-callbacks)
|
20
|
+
- [**Instance Methods**](#instance-methods-1)
|
21
|
+
- [**Class Methods**](#class-methods-1)
|
22
|
+
- [**Payload actions**](#payload-actions)
|
23
|
+
- [**Testing with RSpec**](#testing-with-rspec)
|
24
|
+
- [**Extra configurations**](#extra-configurations)
|
25
|
+
- [**TODO**](#todo)
|
26
|
+
- [**Q&A**](#qa)
|
27
|
+
- [**Contributing**](#contributing)
|
28
|
+
- [**License**](#license)
|
29
|
+
- [**Code of Conduct**](#code-of-conduct)
|
30
|
+
|
31
|
+
## **Features**
|
7
32
|
- Sync CRUD operations between Rails apps. So, all changes made on App1, will be reflected on App2, App3.
|
8
33
|
Example: If User is created on App1, this user will be created on App2 too with the accepted attributes.
|
9
34
|
- Ability to make class level communication
|
10
35
|
Example: If User from App1 wants to generate_email, this can be listened on App2, App3, ... to make corresponding actions
|
11
36
|
- Change pub/sub service at any time
|
37
|
+
- Support for transactions: Permits to group all payloads with the same ordering_key and be processed in the same order they are published by the subscribers.
|
38
|
+
Grouping by ordering_key allows us to enable multiple workers in our Pub/Sub service(s), and still guarantee that related payloads will be processed in the correct order, despite of the multiple threads.
|
39
|
+
This thanks to the fact that Pub/Sub services will always send messages with the same `ordering_key` into the same worker/thread.
|
40
|
+
- Ability to send notifications to a specific topic or multiple topics
|
12
41
|
|
13
|
-
## Installation
|
42
|
+
## **Installation**
|
14
43
|
Add this line to your application's Gemfile:
|
15
44
|
```ruby
|
16
45
|
gem 'pub_sub_model_sync'
|
@@ -22,15 +51,16 @@ gem 'ruby-kafka' # to use apache kafka pub/sub service
|
|
22
51
|
And then execute: $ bundle install
|
23
52
|
|
24
53
|
|
25
|
-
##
|
54
|
+
## **Configuration**
|
26
55
|
|
27
56
|
- Configuration for google pub/sub (You need google pub/sub service account)
|
28
57
|
```ruby
|
29
58
|
# initializers/pub_sub_config.rb
|
30
|
-
PubSubModelSync::Config.service_name = :google
|
59
|
+
PubSubModelSync::Config.service_name = :google
|
31
60
|
PubSubModelSync::Config.project = 'google-project-id'
|
32
61
|
PubSubModelSync::Config.credentials = 'path-to-the-config'
|
33
|
-
PubSubModelSync::Config.topic_name = 'sample-topic'
|
62
|
+
PubSubModelSync::Config.topic_name = 'sample-topic'
|
63
|
+
PubSubModelSync::Config.subscription_name = 'my-app1'
|
34
64
|
```
|
35
65
|
See details here:
|
36
66
|
https://github.com/googleapis/google-cloud-ruby/tree/master/google-cloud-pubsub
|
@@ -39,8 +69,8 @@ And then execute: $ bundle install
|
|
39
69
|
```ruby
|
40
70
|
PubSubModelSync::Config.service_name = :rabbitmq
|
41
71
|
PubSubModelSync::Config.bunny_connection = 'amqp://guest:guest@localhost'
|
42
|
-
PubSubModelSync::Config.queue_name = 'model-sync'
|
43
72
|
PubSubModelSync::Config.topic_name = 'sample-topic'
|
73
|
+
PubSubModelSync::Config.subscription_name = 'my-app2'
|
44
74
|
```
|
45
75
|
See details here: https://github.com/ruby-amqp/bunny
|
46
76
|
|
@@ -49,30 +79,28 @@ And then execute: $ bundle install
|
|
49
79
|
PubSubModelSync::Config.service_name = :kafka
|
50
80
|
PubSubModelSync::Config.kafka_connection = [["kafka1:9092", "localhost:2121"], { logger: Rails.logger }]
|
51
81
|
PubSubModelSync::Config.topic_name = 'sample-topic'
|
82
|
+
PubSubModelSync::Config.subscription_name = 'my-app3'
|
52
83
|
```
|
53
|
-
See details here: https://github.com/zendesk/ruby-kafka
|
84
|
+
See details here: https://github.com/zendesk/ruby-kafka
|
54
85
|
|
55
86
|
- Add publishers/subscribers to your models (See examples below)
|
56
87
|
|
57
88
|
- Start subscribers to listen for publishers (Only in the app that has subscribers)
|
58
|
-
```
|
59
|
-
rake pub_sub_model_sync:start
|
89
|
+
```bash
|
90
|
+
DB_POOL=20 bundle exec rake pub_sub_model_sync:start
|
60
91
|
```
|
61
|
-
Note:
|
62
|
-
|
63
|
-
|
64
|
-
# PubSubModelSync::Config.subscribers ==> []
|
65
|
-
PubSubModelSync::Runner.preload_listeners
|
66
|
-
# PubSubModelSync::Config.subscribers ==> [#<PubSubModelSync::Subscriber:0x000.. @klass="Article", @action=:create..., ....]
|
67
|
-
```
|
68
|
-
|
69
|
-
- Check the service status with:
|
92
|
+
Note: You need more than 15 DB pools to avoid "could not obtain a connection from the pool within 5.000 seconds". https://devcenter.heroku.com/articles/concurrency-and-database-connections
|
93
|
+
|
94
|
+
- Check the service status with:
|
70
95
|
```PubSubModelSync::MessagePublisher.publish_data('Test message', {sample_value: 10}, :create)```
|
71
96
|
|
72
|
-
##
|
97
|
+
## **Notifications Diagram**
|
98
|
+

|
99
|
+
|
100
|
+
## **Basic Example**
|
73
101
|
```ruby
|
74
102
|
# App 1 (Publisher)
|
75
|
-
# attributes: name email age
|
103
|
+
# attributes: name email age
|
76
104
|
class User < ActiveRecord::Base
|
77
105
|
include PubSubModelSync::PublisherConcern
|
78
106
|
ps_publish(%i[id name email])
|
@@ -81,34 +109,39 @@ end
|
|
81
109
|
# App 2 (Subscriber)
|
82
110
|
class User < ActiveRecord::Base
|
83
111
|
include PubSubModelSync::SubscriberConcern
|
84
|
-
ps_subscribe(%i[name])
|
85
|
-
|
112
|
+
ps_subscribe(%i[name]) # crud notifications
|
113
|
+
ps_subscribe_custom(:say_welcome) # custom instance notification
|
114
|
+
ps_class_subscribe(:greeting) # class notification
|
86
115
|
|
87
116
|
def self.greeting(data)
|
88
117
|
puts 'Class message called'
|
89
118
|
end
|
119
|
+
|
120
|
+
def say_welcome(data)
|
121
|
+
UserMailer.deliver(id, data)
|
122
|
+
end
|
90
123
|
end
|
91
124
|
|
92
125
|
# Samples
|
93
126
|
User.create(name: 'test user', email: 'sample@gmail.com') # Review your App 2 to see the created user (only name will be saved)
|
94
127
|
User.new(name: 'test user').ps_perform_sync(:create) # similar to above to perform sync on demand
|
95
128
|
|
96
|
-
|
97
|
-
PubSubModelSync::MessagePublisher.publish_data(User, { msg: 'Hello' }, :greeting) #
|
129
|
+
PubSubModelSync::MessagePublisher.publish_model_data(my_user, { id:10, msg: 'Hello' }, :say_welcome, { as_klass: 'RegisteredUser' }) # custom model action notification
|
130
|
+
PubSubModelSync::MessagePublisher.publish_data(User, { msg: 'Hello' }, :greeting) # custom data notification
|
98
131
|
```
|
99
132
|
|
100
|
-
## Advanced Example
|
133
|
+
## **Advanced Example**
|
101
134
|
```ruby
|
102
135
|
# App 1 (Publisher)
|
103
136
|
class User < ActiveRecord::Base
|
104
137
|
self.table_name = 'publisher_users'
|
105
138
|
include PubSubModelSync::PublisherConcern
|
106
|
-
ps_publish(%i[id:client_id name:full_name email], actions: %i[update], as_klass: 'Client')
|
107
|
-
|
139
|
+
ps_publish(%i[id:client_id name:full_name email], actions: %i[update], as_klass: 'Client', headers: { topic_name: ['topic1', 'topic N'] })
|
140
|
+
|
108
141
|
def ps_skip_callback?(_action)
|
109
142
|
false # here logic with action to skip push message
|
110
143
|
end
|
111
|
-
|
144
|
+
|
112
145
|
def ps_skip_sync?(_action)
|
113
146
|
false # here logic with action to skip push message
|
114
147
|
end
|
@@ -120,130 +153,249 @@ class User < ActiveRecord::Base
|
|
120
153
|
include PubSubModelSync::SubscriberConcern
|
121
154
|
ps_subscribe(%i[name], actions: %i[update], from_klass: 'Client', id: %i[client_id email])
|
122
155
|
ps_class_subscribe(:greeting, from_action: :custom_greeting, from_klass: 'CustomUser')
|
156
|
+
ps_subscribe_custom(:send_welcome, from_klass: 'CustomUser', id: :id, from_action: :say_welcome)
|
123
157
|
alias_attribute :full_name, :name
|
124
|
-
|
158
|
+
|
125
159
|
def self.greeting(data)
|
126
160
|
puts 'Class message called through custom_greeting'
|
127
161
|
end
|
128
162
|
|
163
|
+
def send_welcome(data)
|
164
|
+
UserMailer.deliver(id, data)
|
165
|
+
end
|
166
|
+
|
129
167
|
# def self.ps_find_model(data)
|
130
|
-
# where(email: data[:email], ...).first_or_initialize
|
168
|
+
# where(email: data[:email], ...).first_or_initialize
|
131
169
|
# end
|
132
170
|
end
|
133
171
|
```
|
134
172
|
|
135
|
-
|
136
|
-
|
137
|
-
|
138
|
-
|
139
|
-
|
140
|
-
|
141
|
-
|
142
|
-
|
143
|
-
|
144
|
-
|
145
|
-
|
146
|
-
|
173
|
+
## **API**
|
174
|
+
### **Subscribers**
|
175
|
+
|
176
|
+
#### **Registering Subscriptions**
|
177
|
+
|
178
|
+
- Configure class subscriptions
|
179
|
+
```ruby
|
180
|
+
class MyModel < ActiveRecord::Base
|
181
|
+
ps_class_subscribe(action_name, from_action: nil, from_klass: nil)
|
182
|
+
end
|
183
|
+
```
|
184
|
+
When Class receives the corresponding notification, `action` method will be called on the Class. Like: `User.action(data)`
|
185
|
+
* `action_name`: (String|Sym/Optional) Action name
|
186
|
+
* `from_klass`: (String/Optional) Source class name (Default `model.class.name`)
|
187
|
+
* `from_action`: (Sym/Optional) Source method name. Default `action`
|
188
|
+
|
189
|
+
- Configure CRUD subscriptions
|
190
|
+
```ruby
|
191
|
+
class MyModel < ActiveRecord::Base
|
192
|
+
ps_subscribe(attrs, from_klass: nil, actions: nil, id: nil)
|
193
|
+
end
|
194
|
+
```
|
195
|
+
When model receives the corresponding notification, `action` method will be called on the model. Like: `model.destroy`
|
196
|
+
* `attrs`: (Array/Required) Array of all attributes to be synced
|
197
|
+
* `from_klass`: (String/Optional) Source class name (Default `model.class.name`)
|
198
|
+
* `actions`: (Array/Optional, default: create/update/destroy) permit to customize action names
|
199
|
+
* `id`: (Sym|Array/Optional, default: id) Attr identifier(s) to find the corresponding model
|
200
|
+
|
201
|
+
- Configure custom model subscriptions
|
202
|
+
```ruby
|
203
|
+
class MyModel < ActiveRecord::Base
|
204
|
+
ps_subscribe_custom(action, from_klass: name, id: :id, from_action: nil)
|
205
|
+
end
|
206
|
+
```
|
207
|
+
When model receives the corresponding notification, `action` method will be called on the model. Like: `model.action(data)`
|
208
|
+
* `action`: (String/Required) Action name
|
209
|
+
* `from_klass`: (String/Optional) Source class name (Default `model.class.name`)
|
210
|
+
* `from_action`: (Sym/Optional) Source method name. Default `action`
|
211
|
+
* `id`: (Sym|Array/Optional, default: id) Attr identifier(s) to find the corresponding model
|
212
|
+
|
213
|
+
- Perform custom actions before saving sync of the model (`:cancel` can be returned to skip sync)
|
214
|
+
```ruby
|
215
|
+
class MyModel < ActiveRecord::Base
|
216
|
+
def ps_before_save_sync(action, payload)
|
217
|
+
# puts payload.data[:id]
|
218
|
+
end
|
219
|
+
end
|
220
|
+
```
|
147
221
|
|
148
|
-
-
|
149
|
-
```
|
150
|
-
|
151
|
-
|
152
|
-
|
153
|
-
|
154
|
-
|
155
|
-
|
156
|
-
|
157
|
-
* data: (Hash) Data received from sync
|
222
|
+
- Configure a custom model finder (optional)
|
223
|
+
```ruby
|
224
|
+
class MyModel < ActiveRecord::Base
|
225
|
+
def ps_find_model(data)
|
226
|
+
where(custom_finder: data[:custom_value]).first_or_initialize
|
227
|
+
end
|
228
|
+
end
|
229
|
+
```
|
230
|
+
* `data`: (Hash) Data received from sync
|
158
231
|
Must return an existent or a new model object
|
159
232
|
|
160
|
-
|
161
|
-
|
162
|
-
|
233
|
+
#### **Subscription helpers**
|
234
|
+
- Inspect all configured subscriptions
|
235
|
+
```ruby
|
236
|
+
PubSubModelSync::Config.subscribers
|
237
|
+
```
|
238
|
+
- Manually process or reprocess a notification
|
239
|
+
```ruby
|
240
|
+
payload = PubSubModelSync::Payload.new(data, attributes, headers)
|
241
|
+
payload.process!
|
242
|
+
```
|
163
243
|
|
164
|
-
- Inspect all configured subscribers
|
165
|
-
```PubSubModelSync::Config.subscribers```
|
166
244
|
|
167
|
-
|
168
|
-
```.ps_subscriber_changed?(data)```
|
169
|
-
By default: ```model.changed?```
|
245
|
+
### **Publishers**
|
170
246
|
|
171
|
-
|
172
|
-
|
247
|
+
#### **Registering Publishers **
|
248
|
+
- Register CRUD publishers that will trigger configured notifications
|
249
|
+
```ruby
|
250
|
+
class MyModel < ActiveRecord::Base
|
251
|
+
ps_publish([:id, 'created_at:published_at', :full_name], actions: [:update], as_klass: nil, headers: { ordering_key: 'custom-key', topic_name: 'my-custom-topic' })
|
252
|
+
def full_name
|
253
|
+
[first_name, last_name].join(' ')
|
254
|
+
end
|
255
|
+
end
|
256
|
+
```
|
257
|
+
* `attrs`: (Array/Required) Array of attributes to be published. Supports for:
|
258
|
+
- aliases: permits to publish with different names, sample: "created_at:published_at" where "created_at" will be published as "published_at"
|
259
|
+
- methods: permits to publish method values as attributes, sample: "full_name"
|
260
|
+
* `actions`: (Array/Optional, default: %i[create update destroy]) permit to define action names
|
261
|
+
* `as_klass`: (String/Optional) Output class name (Instead of the model class name, will use this value)
|
262
|
+
* `headers`: (Hash/Optional) Notification settings which permit to customize the way and the target of the notification (Refer Payload.headers)
|
263
|
+
|
173
264
|
|
174
|
-
|
175
|
-
-
|
176
|
-
```
|
177
|
-
|
178
|
-
|
179
|
-
|
265
|
+
#### **Publishing notifications**
|
266
|
+
- CRUD notifications
|
267
|
+
```ruby
|
268
|
+
MyModel.create!(...)
|
269
|
+
```
|
270
|
+
"Create" notification will be delivered with the configured attributes as the payload data
|
180
271
|
|
181
|
-
-
|
182
|
-
```
|
183
|
-
|
184
|
-
|
272
|
+
- Manual CRUD notifications
|
273
|
+
```ruby
|
274
|
+
MyModel.ps_perform_sync(action, custom_data: {}, custom_headers: {})
|
275
|
+
```
|
276
|
+
* `action`: (Sym) CRUD action name (create, update or destroy)
|
277
|
+
* `custom_data`: custom_data (nil|Hash) If present custom_data will be used as the payload data. I.E. data generator will be ignored
|
278
|
+
* `custom_headers`: (Hash, optional) override default headers. Refer `payload.headers`
|
185
279
|
|
186
|
-
-
|
187
|
-
```
|
188
|
-
|
280
|
+
- Class notifications
|
281
|
+
```ruby
|
282
|
+
PubSubModelSync::MessagePublisher.publish_data((klass, data, action, headers: )
|
283
|
+
```
|
284
|
+
Publishes any data to be listened at a class level.
|
285
|
+
- `klass`: (String) Class name to be used
|
286
|
+
- `data`: (Hash) Data to be delivered
|
287
|
+
- `action`: (Sym) Action name
|
288
|
+
- `headers`: (Hash, optional) Notification settings (Refer Payload.headers)
|
289
|
+
|
290
|
+
- Model custom action notifications
|
291
|
+
```ruby
|
292
|
+
PubSubModelSync::MessagePublisher.publish_model_data(model, data, action, as_klass:, headers:)
|
293
|
+
```
|
294
|
+
Publishes model custom action to be listened at an instance level.
|
295
|
+
- `model`: (ActiveRecord) model owner of the data
|
296
|
+
- `data`: (Hash) Data to be delivered
|
297
|
+
- `action`: (Sym) Action name
|
298
|
+
- `as_klass`: (String, optional) if not provided, `model.class.name` will be used instead
|
299
|
+
- `headers`: (Hash, optional) Notification settings (Refer Payload.headers)
|
300
|
+
|
301
|
+
- Manually publish or republish a notification
|
302
|
+
```ruby
|
303
|
+
payload = PubSubModelSync::Payload.new(data, attributes, headers)
|
304
|
+
payload.publish!
|
305
|
+
```
|
306
|
+
|
307
|
+
#### ** publishing callbacks**
|
308
|
+
|
309
|
+
- Prevent CRUD sync at model callback level (Called right after :after_create, :after_update, :after_destroy).
|
310
|
+
If returns "true", sync will be cancelled.
|
311
|
+
```ruby
|
312
|
+
class MyModel < ActiveRecord::Base
|
313
|
+
def ps_skip_callback?(action)
|
314
|
+
# logic here
|
315
|
+
end
|
316
|
+
end
|
317
|
+
```
|
318
|
+
|
319
|
+
- Prevent CRUD sync before processing payload (Affects model.ps_perform_sync(...))).
|
320
|
+
If returns "true", sync will be cancelled
|
321
|
+
```ruby
|
322
|
+
class MyModel < ActiveRecord::Base
|
323
|
+
def ps_skip_sync?(action)
|
324
|
+
# logic here
|
325
|
+
end
|
326
|
+
end
|
327
|
+
```
|
328
|
+
|
329
|
+
- Do some actions before publishing a CRUD notification.
|
330
|
+
If returns ":cancel", sync will be cancelled
|
331
|
+
```ruby
|
332
|
+
class MyModel < ActiveRecord::Base
|
333
|
+
def ps_before_sync(action, payload)
|
334
|
+
# logic here
|
335
|
+
end
|
336
|
+
end
|
337
|
+
```
|
338
|
+
|
339
|
+
- Do some actions after CRUD notification was published.
|
340
|
+
```ruby
|
341
|
+
class MyModel < ActiveRecord::Base
|
342
|
+
def ps_after_sync(action, payload)
|
343
|
+
# logic here
|
344
|
+
end
|
345
|
+
end
|
346
|
+
```
|
347
|
+
|
348
|
+
|
349
|
+
### **Payload**
|
350
|
+
Any notification before delivering is transformed as a Payload for a better portability.
|
351
|
+
|
352
|
+
- Initialize
|
353
|
+
```ruby
|
354
|
+
payload = PubSubModelSync::Payload.new(data, attributes, headers)
|
355
|
+
```
|
356
|
+
* `data`: (Hash) Data to be published or processed
|
357
|
+
* `attributes`: (Hash) Includes class and method info
|
358
|
+
- `action`: (String) action name
|
359
|
+
- `klass`: (String) class name
|
360
|
+
* `headers`: (Hash) Notification settings that defines how the notification will be processed or delivered.
|
361
|
+
- `key`: (String, optional) identifier of the payload, default: `<klass_name>/<action>` when class message, `<model.class.name>/<action>/<model.id>` when model message (Useful for caching techniques).
|
362
|
+
- `ordering_key`: (String, optional): messages with the same key are processed in the same order they were delivered, default: `klass_name` when class message, `<model.class.name>/<model.id>` when model message
|
363
|
+
- `topic_name`: (String|Array<String>, optional): Specific topic name to be used when delivering the message (default first topic from config).
|
364
|
+
- `forced_ordering_key`: (String, optional): Will force to use this value as the `ordering_key`, even withing transactions. Default `nil`.
|
189
365
|
|
190
|
-
-
|
191
|
-
```model.ps_before_sync(action, data_to_deliver)```
|
192
|
-
Note: If the method returns ```:cancel```, the sync will be stopped (message will not be published)
|
193
|
-
|
194
|
-
- Callback called after sync
|
195
|
-
```model.ps_after_sync(action, data_delivered)```
|
196
|
-
|
197
|
-
- Perform sync on demand (:create, :update, :destroy):
|
198
|
-
The target model will receive a notification to perform the indicated action
|
199
|
-
```my_model.ps_perform_sync(action_name, custom_settings = {})```
|
200
|
-
* custom_settings: override default settings defined for action_name ({ attrs: [], as_klass: nil })
|
201
|
-
|
202
|
-
- Publish a class level notification:
|
203
|
-
```User.ps_class_publish(data, action: action_name, as_klass: custom_klass_name)```
|
204
|
-
Target class ```User.action_name``` will be called when message is received
|
205
|
-
* data: (required, :hash) message value to deliver
|
206
|
-
* action_name: (required, :sim) Action name
|
207
|
-
* as_klass: (optional, :string) Custom class name (Default current model name)
|
208
|
-
|
209
|
-
- Payload actions
|
366
|
+
- Actions for payloads
|
210
367
|
```ruby
|
211
|
-
payload = PubSubModelSync::Payload.new({ title: 'hello' }, { action: :greeting, klass: 'User' })
|
212
368
|
payload.publish! # publishes notification data. It raises exception if fails and does not call ```:on_error_publishing``` callback
|
213
369
|
payload.publish # publishes notification data. On error does not raise exception but calls ```:on_error_publishing``` callback
|
214
370
|
payload.process! # process a notification data. It raises exception if fails and does not call ```.on_error_processing``` callback
|
215
371
|
payload.publish # process a notification data. It does not raise exception if fails but calls ```.on_error_processing``` callback
|
216
372
|
```
|
217
|
-
|
218
|
-
|
219
|
-
```User.ps_publisher(action_name)```
|
220
|
-
* action_name (default :create, :sym): can be :create, :update, :destroy
|
221
|
-
|
222
|
-
## Testing with RSpec
|
373
|
+
|
374
|
+
## **Testing with RSpec**
|
223
375
|
- Config: (spec/rails_helper.rb)
|
224
376
|
```ruby
|
225
|
-
|
377
|
+
|
226
378
|
# when using google service
|
227
379
|
require 'pub_sub_model_sync/mock_google_service'
|
228
380
|
config.before(:each) do
|
229
381
|
google_mock = PubSubModelSync::MockGoogleService.new
|
230
382
|
allow(Google::Cloud::Pubsub).to receive(:new).and_return(google_mock)
|
231
383
|
end
|
232
|
-
|
384
|
+
|
233
385
|
# when using rabbitmq service
|
234
|
-
require 'pub_sub_model_sync/mock_rabbit_service'
|
386
|
+
require 'pub_sub_model_sync/mock_rabbit_service'
|
235
387
|
config.before(:each) do
|
236
388
|
rabbit_mock = PubSubModelSync::MockRabbitService.new
|
237
389
|
allow(Bunny).to receive(:new).and_return(rabbit_mock)
|
238
390
|
end
|
239
|
-
|
391
|
+
|
240
392
|
# when using apache kafka service
|
241
|
-
require 'pub_sub_model_sync/mock_kafka_service'
|
393
|
+
require 'pub_sub_model_sync/mock_kafka_service'
|
242
394
|
config.before(:each) do
|
243
395
|
kafka_mock = PubSubModelSync::MockKafkaService.new
|
244
396
|
allow(Kafka).to receive(:new).and_return(kafka_mock)
|
245
397
|
end
|
246
|
-
|
398
|
+
|
247
399
|
```
|
248
400
|
- Examples:
|
249
401
|
```ruby
|
@@ -254,7 +406,7 @@ Note: Be careful with collision of names
|
|
254
406
|
payload.process!
|
255
407
|
expect(User.where(id: data[:id]).any?).to be_truth
|
256
408
|
end
|
257
|
-
|
409
|
+
|
258
410
|
it 'receive class message' do
|
259
411
|
data = { msg: 'hello' }
|
260
412
|
action = :greeting
|
@@ -262,97 +414,97 @@ Note: Be careful with collision of names
|
|
262
414
|
payload.process!
|
263
415
|
expect(User).to receive(action)
|
264
416
|
end
|
265
|
-
|
417
|
+
|
266
418
|
# Publisher
|
267
419
|
it 'publish model action' do
|
268
|
-
publisher = PubSubModelSync::MessagePublisher
|
420
|
+
publisher = PubSubModelSync::MessagePublisher
|
269
421
|
user = User.create(name: 'name', email: 'email')
|
270
422
|
expect(publisher).to receive(:publish_model).with(user, :create, anything)
|
271
423
|
end
|
272
|
-
|
424
|
+
|
273
425
|
it 'publish class message' do
|
274
|
-
publisher = PubSubModelSync::MessagePublisher
|
426
|
+
publisher = PubSubModelSync::MessagePublisher
|
275
427
|
data = {msg: 'hello'}
|
276
428
|
action = :greeting
|
277
|
-
|
429
|
+
PubSubModelSync::MessagePublisher.publish_data('User', data, action)
|
278
430
|
expect(publisher).to receive(:publish_data).with('User', data, action)
|
279
431
|
end
|
280
432
|
```
|
281
433
|
|
282
|
-
## Extra configurations
|
434
|
+
## **Extra configurations**
|
283
435
|
```ruby
|
284
436
|
config = PubSubModelSync::Config
|
285
437
|
config.debug = true
|
286
438
|
```
|
287
|
-
|
288
|
-
|
289
|
-
|
290
|
-
|
439
|
+
- `.topic_name = ['topic1', 'topic 2']`: (String|Array<String>)
|
440
|
+
Topic name(s) to be used to listen all notifications from when listening. Additional first topic name is used as the default topic name when publishing a notification.
|
441
|
+
- `.subscription_name = "my-app-1"`: (String, default Rails.application.name)
|
442
|
+
Subscriber's identifier which helps to:
|
443
|
+
* skip self messages
|
444
|
+
* continue the sync from the last synced notification when service was restarted.
|
445
|
+
- ```.debug = true```
|
291
446
|
(true/false*) => show advanced log messages
|
292
|
-
- ```.logger = Rails.logger```
|
447
|
+
- ```.logger = Rails.logger```
|
293
448
|
(Logger) => define custom logger
|
294
|
-
- ```.disabled_callback_publisher = ->(_model, _action) { false }```
|
295
|
-
(true/false*) => if true, does not listen model callbacks for auto sync (Create/Update/Destroy)
|
296
|
-
- ```.on_before_processing = ->(payload, {subscriber:}) { puts payload }```
|
297
|
-
(Proc) => called before processing received message (:cancel can be returned to skip processing)
|
298
|
-
- ```.on_success_processing = ->(payload, {subscriber:}) { puts payload }```
|
449
|
+
- ```.disabled_callback_publisher = ->(_model, _action) { false }```
|
450
|
+
(true/false*) => if true, does not listen model callbacks for auto sync (Create/Update/Destroy)
|
451
|
+
- ```.on_before_processing = ->(payload, {subscriber:}) { puts payload }```
|
452
|
+
(Proc) => called before processing received message (:cancel can be returned to skip processing)
|
453
|
+
- ```.on_success_processing = ->(payload, {subscriber:}) { puts payload }```
|
299
454
|
(Proc) => called when a message was successfully processed
|
300
|
-
- ```.on_error_processing = ->(exception, {payload:, subscriber:}) { payload.delay(...).process! }```
|
455
|
+
- ```.on_error_processing = ->(exception, {payload:, subscriber:}) { payload.delay(...).process! }```
|
301
456
|
(Proc) => called when a message failed when processing (delayed_job or similar can be used for retrying)
|
302
|
-
- ```.on_before_publish = ->(payload) { puts payload }```
|
303
|
-
(Proc) => called before publishing a message (:cancel can be returned to skip publishing)
|
304
|
-
- ```.on_after_publish = ->(payload) { puts payload }```
|
457
|
+
- ```.on_before_publish = ->(payload) { puts payload }```
|
458
|
+
(Proc) => called before publishing a message (:cancel can be returned to skip publishing)
|
459
|
+
- ```.on_after_publish = ->(payload) { puts payload }```
|
305
460
|
(Proc) => called after publishing a message
|
306
|
-
- ```.on_error_publish = ->(exception, {payload:}) { payload.delay(...).publish! }```
|
461
|
+
- ```.on_error_publish = ->(exception, {payload:}) { payload.delay(...).publish! }```
|
307
462
|
(Proc) => called when failed publishing a message (delayed_job or similar can be used for retrying)
|
308
|
-
|
309
|
-
## TODO
|
463
|
+
|
464
|
+
## **TODO**
|
310
465
|
- Add alias attributes when subscribing (similar to publisher)
|
311
466
|
- Add flag ```model.ps_process_payload``` to retrieve the payload used to process the pub/sub sync
|
312
467
|
- Auto publish update only if payload has changed
|
313
468
|
- On delete, payload must only be composed by ids
|
314
|
-
-
|
315
|
-
PubSubModelSync::MessagePublisher.
|
316
|
-
- Add DB table to use as a shield to
|
469
|
+
- Improve transactions to exclude similar messages by klass and action. Sample:
|
470
|
+
```PubSubModelSync::MessagePublisher.transaction(key, { same_keys: :use_last_as_first|:use_last|:use_first_as_last|:keep*, same_data: :use_last_as_first*|:use_last|:use_first_as_last|:keep })```
|
471
|
+
- Add DB table to use as a shield to prevent publishing similar notifications and publish partial notifications (similar idea when processing notif)
|
317
472
|
- add callback: on_message_received(payload)
|
318
473
|
|
319
|
-
## Q&A
|
320
|
-
-
|
321
|
-
This problem occurs because pub/sub dependencies (kafka, google-pubsub, rabbitmq) use many threads to perform notifications where the qty of threads is greater than qty of DB pools ([Google pubsub info](https://github.com/googleapis/google-cloud-ruby/blob/master/google-cloud-pubsub/lib/google/cloud/pubsub/subscription.rb#L888))
|
322
|
-
To fix the problem, edit config/database.yml and increase the quantity of ```pool:
|
323
|
-
- Google pubsub: How to process notifications parallely and not sequentially (default 1 thread)?
|
324
|
-
```ruby PubSubModelSync::ServiceGoogle::LISTEN_SETTINGS = { threads: { callback: qty_threads } } ```
|
325
|
-
Note: by this way some notifications can be processed before others thus missing relationship errors can appear
|
474
|
+
## **Q&A**
|
475
|
+
- I'm getting error "could not obtain a connection from the pool within 5.000 seconds"... what does this mean?
|
476
|
+
This problem occurs because pub/sub dependencies (kafka, google-pubsub, rabbitmq) use many threads to perform notifications where the qty of threads is greater than qty of DB pools ([Google pubsub info](https://github.com/googleapis/google-cloud-ruby/blob/master/google-cloud-pubsub/lib/google/cloud/pubsub/subscription.rb#L888))
|
477
|
+
To fix the problem, edit config/database.yml and increase the quantity of ```pool: 20```
|
326
478
|
- How to retry failed syncs with sidekiq?
|
327
479
|
```ruby
|
328
480
|
# lib/initializers/pub_sub_config.rb
|
329
|
-
|
481
|
+
|
330
482
|
class PubSubRecovery
|
331
483
|
include Sidekiq::Worker
|
332
484
|
sidekiq_options queue: :pubsub, retry: 2, backtrace: true
|
333
|
-
|
485
|
+
|
334
486
|
def perform(payload_data, action)
|
335
487
|
payload = PubSubModelSync::Payload.from_payload_data(payload_data)
|
336
488
|
payload.send(action)
|
337
489
|
end
|
338
490
|
end
|
339
|
-
|
491
|
+
|
340
492
|
PubSubModelSync::Config.on_error_publish = lambda do |_e, data|
|
341
493
|
PubSubRecovery.perform_async(data[:payload].to_h, :publish!)
|
342
494
|
end
|
343
495
|
PubSubModelSync::Config.on_error_processing = lambda do |_e, data|
|
344
496
|
PubSubRecovery.perform_async(data[:payload].to_h, :process!)
|
345
497
|
end
|
346
|
-
```
|
498
|
+
```
|
347
499
|
|
348
|
-
## Contributing
|
500
|
+
## **Contributing**
|
349
501
|
|
350
502
|
Bug reports and pull requests are welcome on GitHub at https://github.com/owen2345/pub_sub_model_sync. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
|
351
503
|
|
352
|
-
## License
|
504
|
+
## **License**
|
353
505
|
|
354
506
|
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|
355
507
|
|
356
|
-
## Code of Conduct
|
508
|
+
## **Code of Conduct**
|
357
509
|
|
358
510
|
Everyone interacting in the PubSubModelSync project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/pub_sub_model_sync/blob/master/CODE_OF_CONDUCT.md).
|