pub_sub_model_sync 1.0.beta2 → 1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.github/workflows/release.yml +43 -0
- data/CHANGELOG.md +7 -4
- data/Gemfile.lock +3 -1
- data/README.md +174 -105
- data/docs/notifications-diagram.png +0 -0
- data/lib/pub_sub_model_sync.rb +1 -1
- data/lib/pub_sub_model_sync/base.rb +0 -20
- data/lib/pub_sub_model_sync/config.rb +1 -1
- data/lib/pub_sub_model_sync/initializers/before_commit.rb +3 -3
- data/lib/pub_sub_model_sync/message_processor.rb +32 -9
- data/lib/pub_sub_model_sync/message_publisher.rb +14 -10
- data/lib/pub_sub_model_sync/payload.rb +15 -12
- data/lib/pub_sub_model_sync/{publisher.rb → payload_builder.rb} +15 -10
- data/lib/pub_sub_model_sync/publisher_concern.rb +27 -16
- data/lib/pub_sub_model_sync/run_subscriber.rb +17 -13
- data/lib/pub_sub_model_sync/service_base.rb +5 -32
- data/lib/pub_sub_model_sync/service_google.rb +1 -1
- data/lib/pub_sub_model_sync/subscriber_concern.rb +6 -4
- data/lib/pub_sub_model_sync/transaction.rb +6 -2
- data/lib/pub_sub_model_sync/version.rb +1 -1
- data/samples/README.md +50 -0
- data/samples/app1/.gitattributes +8 -0
- data/samples/app1/.gitignore +28 -0
- data/samples/app1/Dockerfile +13 -0
- data/samples/app1/Gemfile +37 -0
- data/samples/app1/Gemfile.lock +171 -0
- data/samples/app1/README.md +24 -0
- data/samples/app1/Rakefile +6 -0
- data/samples/app1/app/models/application_record.rb +3 -0
- data/samples/app1/app/models/concerns/.keep +0 -0
- data/samples/app1/app/models/post.rb +19 -0
- data/samples/app1/app/models/user.rb +29 -0
- data/samples/app1/bin/bundle +114 -0
- data/samples/app1/bin/rails +5 -0
- data/samples/app1/bin/rake +5 -0
- data/samples/app1/bin/setup +33 -0
- data/samples/app1/bin/spring +14 -0
- data/samples/app1/config.ru +6 -0
- data/samples/app1/config/application.rb +40 -0
- data/samples/app1/config/boot.rb +4 -0
- data/samples/app1/config/credentials.yml.enc +1 -0
- data/samples/app1/config/database.yml +25 -0
- data/samples/app1/config/environment.rb +5 -0
- data/samples/app1/config/environments/development.rb +63 -0
- data/samples/app1/config/environments/production.rb +105 -0
- data/samples/app1/config/environments/test.rb +57 -0
- data/samples/app1/config/initializers/application_controller_renderer.rb +8 -0
- data/samples/app1/config/initializers/backtrace_silencers.rb +8 -0
- data/samples/app1/config/initializers/cors.rb +16 -0
- data/samples/app1/config/initializers/filter_parameter_logging.rb +6 -0
- data/samples/app1/config/initializers/inflections.rb +16 -0
- data/samples/app1/config/initializers/mime_types.rb +4 -0
- data/samples/app1/config/initializers/pubsub.rb +4 -0
- data/samples/app1/config/initializers/wrap_parameters.rb +14 -0
- data/samples/app1/config/locales/en.yml +33 -0
- data/samples/app1/config/puma.rb +43 -0
- data/samples/app1/config/routes.rb +3 -0
- data/samples/app1/config/spring.rb +6 -0
- data/samples/app1/db/migrate/20210513080700_create_users.rb +12 -0
- data/samples/app1/db/migrate/20210513134332_create_posts.rb +11 -0
- data/samples/app1/db/schema.rb +34 -0
- data/samples/app1/db/seeds.rb +7 -0
- data/samples/app1/docker-compose.yml +32 -0
- data/samples/app1/log/.keep +0 -0
- data/samples/app2/.gitattributes +8 -0
- data/samples/app2/.gitignore +28 -0
- data/samples/app2/Dockerfile +13 -0
- data/samples/app2/Gemfile +37 -0
- data/samples/app2/Gemfile.lock +171 -0
- data/samples/app2/README.md +24 -0
- data/samples/app2/Rakefile +6 -0
- data/samples/app2/app/models/application_record.rb +9 -0
- data/samples/app2/app/models/concerns/.keep +0 -0
- data/samples/app2/app/models/customer.rb +28 -0
- data/samples/app2/app/models/post.rb +10 -0
- data/samples/app2/bin/bundle +114 -0
- data/samples/app2/bin/rails +5 -0
- data/samples/app2/bin/rake +5 -0
- data/samples/app2/bin/setup +33 -0
- data/samples/app2/bin/spring +14 -0
- data/samples/app2/config.ru +6 -0
- data/samples/app2/config/application.rb +40 -0
- data/samples/app2/config/boot.rb +4 -0
- data/samples/app2/config/credentials.yml.enc +1 -0
- data/samples/app2/config/database.yml +25 -0
- data/samples/app2/config/environment.rb +5 -0
- data/samples/app2/config/environments/development.rb +63 -0
- data/samples/app2/config/environments/production.rb +105 -0
- data/samples/app2/config/environments/test.rb +57 -0
- data/samples/app2/config/initializers/application_controller_renderer.rb +8 -0
- data/samples/app2/config/initializers/backtrace_silencers.rb +8 -0
- data/samples/app2/config/initializers/cors.rb +16 -0
- data/samples/app2/config/initializers/filter_parameter_logging.rb +6 -0
- data/samples/app2/config/initializers/inflections.rb +16 -0
- data/samples/app2/config/initializers/mime_types.rb +4 -0
- data/samples/app2/config/initializers/pubsub.rb +4 -0
- data/samples/app2/config/initializers/wrap_parameters.rb +14 -0
- data/samples/app2/config/locales/en.yml +33 -0
- data/samples/app2/config/puma.rb +43 -0
- data/samples/app2/config/routes.rb +3 -0
- data/samples/app2/config/spring.rb +6 -0
- data/samples/app2/db/migrate/20210513080956_create_customers.rb +10 -0
- data/samples/app2/db/migrate/20210513135203_create_posts.rb +10 -0
- data/samples/app2/db/schema.rb +31 -0
- data/samples/app2/db/seeds.rb +7 -0
- data/samples/app2/docker-compose.yml +20 -0
- data/samples/app2/log/.keep +0 -0
- metadata +93 -5
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: caaffd468399e03dbf5582bf6973339ec61348a6dcc67a87bda1d930f9271aae
|
4
|
+
data.tar.gz: 9db2cb5f4a70c218d720130761a24ad7f6450531d845f9517e74968f409c9743
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: ac6c2b0087e411969e4a0fbe6fa4aa57e3fe46b6ff93f4d675e2a213cd0fd75786b5e16e1a4c1df6aee92296eac0c94a054a6b62543a0f7f643ba2ea0bc8ff7b
|
7
|
+
data.tar.gz: 9816c7cedd0d6289d0d73a55f35ede63a46bca7f38805c4c566012fed04a58595498a80fd272ce89b617e34b1d1487ff2ee7f6e050c9712bfd6d75b364ad910a
|
@@ -0,0 +1,43 @@
|
|
1
|
+
on:
|
2
|
+
push:
|
3
|
+
tags: # triggered once a git tag is published
|
4
|
+
- '*'
|
5
|
+
|
6
|
+
name: Create Release
|
7
|
+
|
8
|
+
jobs:
|
9
|
+
build:
|
10
|
+
name: Create Release
|
11
|
+
runs-on: ubuntu-latest
|
12
|
+
steps:
|
13
|
+
- name: Checkout code
|
14
|
+
uses: actions/checkout@v2
|
15
|
+
with:
|
16
|
+
fetch-depth: 0
|
17
|
+
|
18
|
+
# Changelog action adaptations
|
19
|
+
- name: Create required package.json
|
20
|
+
run: test -f package.json || echo '{}' >package.json
|
21
|
+
- name: Detect Previous Tag (action not detecting very well)
|
22
|
+
run: echo "::set-output name=previous_tag::$(git describe --abbrev=0 --tags `git rev-list --tags --skip=1 --max-count=1`)"
|
23
|
+
id: tag_checker
|
24
|
+
|
25
|
+
- name: Generate Changelog
|
26
|
+
uses: scottbrenner/generate-changelog-action@master
|
27
|
+
id: Changelog
|
28
|
+
with:
|
29
|
+
from-tag: ${{steps.tag_checker.outputs.previous_tag}}
|
30
|
+
to-tag: HEAD
|
31
|
+
|
32
|
+
- name: Create Release
|
33
|
+
id: create_release
|
34
|
+
uses: actions/create-release@latest
|
35
|
+
env:
|
36
|
+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # This token is provided by Actions, you do not need to create your own token
|
37
|
+
with:
|
38
|
+
tag_name: ${{ github.ref }}
|
39
|
+
release_name: Release ${{ github.ref }}
|
40
|
+
body: |
|
41
|
+
${{ steps.Changelog.outputs.changelog }}
|
42
|
+
draft: false
|
43
|
+
prerelease: false
|
data/CHANGELOG.md
CHANGED
@@ -1,21 +1,24 @@
|
|
1
1
|
# Change Log
|
2
2
|
|
3
|
-
# 1.0
|
3
|
+
# 1.0 (June 13, 2021)
|
4
|
+
This version includes many changes that was refactored from previous version, and thus it needs manual changes to migrate into this version.
|
4
5
|
- Refactor: Subscribers param renamed `from_action` into `to_action` and added support for block or lambda
|
5
6
|
- Feat: Improved `ps_subscribe` to accept new arguments and support for property mappings
|
6
7
|
- Refactor: Refactored `ps_publish` to be called manually (removes notification assumptions) and accept for new arguments
|
7
|
-
- Feat: Added `
|
8
|
+
- Feat: Added `ps_after_action` to listen CRUD events to send notifications in the expected order
|
8
9
|
- Feat: Added `config.default_topic_name` to define default topic name whe publishing (by default `config.topic_name`)
|
9
10
|
- Refactor: Refactored PubSub Transactions to support rollbacks (any exception inside transactions can automatically cancel all pending notifications: configurable through `config.transactions_use_buffer`)
|
10
11
|
- Feat: Improved CRUD transactions to deliver inner notifications in the expected order to keep data consistency
|
11
12
|
- System refactor: Added subscriber runner
|
12
13
|
- Fix: Class notifications can only be listened by class subscriptions
|
13
14
|
- Refactor: Removed `publish_model_data` to have a unique model publisher `ps_publish`
|
14
|
-
- Refactor: Renamed `ps_before_sync` into `ps_before_publish`, `
|
15
|
+
- Refactor: Renamed `ps_before_sync` into `ps_before_publish`, `ps_after_sync` into `ps_after_publish`
|
15
16
|
- Refactor: Renamed `payload.attributes` into `payload.info`
|
16
17
|
- Feat: Support for plain Ruby Objects (Non ActiveRecord models)
|
17
18
|
- Fix: Retry errors for 5 times before exiting notifications listener
|
18
|
-
- Feat: Added transactions max_buffer
|
19
|
+
- Feat: Added transactions max_buffer
|
20
|
+
- Feat: add `ps_perform_publish` to perform the callback for a specific action
|
21
|
+
- Feat: Removed `ps_skip_sync` callback
|
19
22
|
|
20
23
|
# 0.6.0 (March 03, 2021)
|
21
24
|
- feat: add support to include custom payload headers
|
data/Gemfile.lock
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
pub_sub_model_sync (1.0
|
4
|
+
pub_sub_model_sync (1.0)
|
5
5
|
rails
|
6
6
|
|
7
7
|
GEM
|
@@ -158,6 +158,8 @@ GEM
|
|
158
158
|
nio4r (2.5.7)
|
159
159
|
nokogiri (1.11.3-x86_64-darwin)
|
160
160
|
racc (~> 1.4)
|
161
|
+
nokogiri (1.11.3-x86_64-linux)
|
162
|
+
racc (~> 1.4)
|
161
163
|
os (1.1.1)
|
162
164
|
parallel (1.20.1)
|
163
165
|
parser (3.0.1.1)
|
data/README.md
CHANGED
@@ -1,5 +1,9 @@
|
|
1
1
|
# **PubSubModelSync**
|
2
|
-
|
2
|
+
![Rails badge](https://img.shields.io/badge/Rails-4+-success.png)
|
3
|
+
![Ruby badge](https://img.shields.io/badge/Ruby-2.4+-success.png)
|
4
|
+
![Production badge](https://img.shields.io/badge/Production-ready-success.png)
|
5
|
+
|
6
|
+
This gem permits to sync automatically models and custom data between multiple Rails applications by publishing notifications via pubsub (Google PubSub, RabbitMQ, or Apache Kafka) and automatically processed by all connected applications. Out of the scope, this gem includes transactions to keep Data consistency by processing notifications in the order they were delivered.
|
3
7
|
These notifications use JSON format to easily be decoded by subscribers (Rails applications and even other languages)
|
4
8
|
|
5
9
|
- [**PubSubModelSync**](#pubsubmodelsync)
|
@@ -29,12 +33,12 @@ These notifications use JSON format to easily be decoded by subscribers (Rails a
|
|
29
33
|
- [**Code of Conduct**](#code-of-conduct)
|
30
34
|
|
31
35
|
## **Features**
|
32
|
-
- Sync model data between Rails apps: All changes made on App1, will be immediately reflected on App2, App3, etc.
|
36
|
+
- Sync model data between Rails apps: All changes made on App1, will be immediately reflected on App2, App3, etc.
|
33
37
|
Example: If User is created on App1, this user will be created on App2, App3 too with the accepted attributes.
|
34
|
-
- Ability to send class level
|
38
|
+
- Ability to send instance and class level notifications
|
35
39
|
Example: If App1 wants to send emails to multiple users, this can be listened on App2, to deliver corresponding emails
|
36
40
|
- Change pub/sub service at any time: Switch between rabbitmq, kafka, google pubsub
|
37
|
-
- Support for transactions: Permits to keep data consistency between applications by processing notifications in the same order they were delivered.
|
41
|
+
- Support for transactions: Permits to keep data consistency between applications by processing notifications in the same order they were delivered (auto included in models transactions).
|
38
42
|
- Ability to send notifications to a specific topic (single application) or multiple topics (multiple applications)
|
39
43
|
|
40
44
|
## **Installation**
|
@@ -56,7 +60,7 @@ And then execute: $ bundle install
|
|
56
60
|
# initializers/pub_sub_config.rb
|
57
61
|
PubSubModelSync::Config.service_name = :google
|
58
62
|
PubSubModelSync::Config.project = 'google-project-id'
|
59
|
-
PubSubModelSync::Config.credentials = 'path-to-
|
63
|
+
PubSubModelSync::Config.credentials = 'path-to-google-config.json'
|
60
64
|
PubSubModelSync::Config.topic_name = 'sample-topic'
|
61
65
|
PubSubModelSync::Config.subscription_name = 'my-app1'
|
62
66
|
```
|
@@ -85,12 +89,14 @@ And then execute: $ bundle install
|
|
85
89
|
|
86
90
|
- Start subscribers to listen for publishers (Only in the app that has subscribers)
|
87
91
|
```bash
|
88
|
-
|
92
|
+
DB_POOL=20 bundle exec rake pub_sub_model_sync:start
|
89
93
|
```
|
90
94
|
Note: You need more than 15 DB pools to avoid "could not obtain a connection from the pool within 5.000 seconds". https://devcenter.heroku.com/articles/concurrency-and-database-connections
|
91
95
|
|
92
96
|
- Check the service status with:
|
93
|
-
```
|
97
|
+
```ruby
|
98
|
+
PubSubModelSync::Payload.new({ my_data: 'here' }, { klass: 'MyClass', action: :sample_action }).publish!
|
99
|
+
```
|
94
100
|
|
95
101
|
- More configurations: [here](#extra-configurations)
|
96
102
|
|
@@ -103,9 +109,9 @@ And then execute: $ bundle install
|
|
103
109
|
# App 1 (Publisher)
|
104
110
|
class User < ActiveRecord::Base
|
105
111
|
include PubSubModelSync::PublisherConcern
|
106
|
-
|
107
|
-
|
108
|
-
|
112
|
+
ps_after_action(:create) { ps_publish(:create, mapping: %i[id name email]) }
|
113
|
+
ps_after_action(:update) { ps_publish(:update, mapping: %i[id name email]) }
|
114
|
+
ps_after_action(:destroy) { ps_publish(:destroy, mapping: %i[id]) }
|
109
115
|
end
|
110
116
|
|
111
117
|
# App 2 (Subscriber)
|
@@ -115,9 +121,9 @@ class User < ActiveRecord::Base
|
|
115
121
|
end
|
116
122
|
|
117
123
|
# CRUD syncs
|
118
|
-
my_user = User.create(name: 'test user', email: 'sample@gmail.com') # Publishes `:create` notification (App 2 syncs the new user)
|
119
|
-
my_user.update(name: 'changed user') # Publishes `:update` notification (App2 updates changes)
|
120
|
-
my_user.destroy # Publishes `:destroy` notification (App2 destroys the corresponding user)
|
124
|
+
my_user = User.create!(name: 'test user', email: 'sample@gmail.com') # Publishes `:create` notification (App 2 syncs the new user)
|
125
|
+
my_user.update!(name: 'changed user') # Publishes `:update` notification (App2 updates changes on user with the same id)
|
126
|
+
my_user.destroy! # Publishes `:destroy` notification (App2 destroys the corresponding user)
|
121
127
|
```
|
122
128
|
|
123
129
|
### **Advanced Example**
|
@@ -125,14 +131,16 @@ my_user.destroy # Publishes `:destroy` notification (App2 destroys the correspon
|
|
125
131
|
# App 1 (Publisher)
|
126
132
|
class User < ActiveRecord::Base
|
127
133
|
include PubSubModelSync::PublisherConcern
|
128
|
-
|
134
|
+
ps_after_action([:create, :update]) do |action|
|
135
|
+
ps_publish(action, mapping: %i[name:full_name email], as_klass: 'App1User', headers: { topic_name: %i[topic1 topic2] })
|
136
|
+
end
|
129
137
|
end
|
130
138
|
|
131
139
|
# App 2 (Subscriber)
|
132
140
|
class User < ActiveRecord::Base
|
133
141
|
include PubSubModelSync::SubscriberConcern
|
134
|
-
ps_subscribe(:
|
135
|
-
ps_subscribe(:send_welcome, %i[email], to_action: :send_email, if: ->(model) { model.email.present? })
|
142
|
+
ps_subscribe([:create, :update], %i[full_name:customer_name], id: :email, from_klass: 'App1User')
|
143
|
+
ps_subscribe(:send_welcome, %i[email], id: :email, to_action: :send_email, if: ->(model) { model.email.present? })
|
136
144
|
ps_class_subscribe(:batch_disable) # class subscription
|
137
145
|
|
138
146
|
def send_email
|
@@ -143,9 +151,9 @@ class User < ActiveRecord::Base
|
|
143
151
|
puts "disabling users: #{data[:ids]}"
|
144
152
|
end
|
145
153
|
end
|
146
|
-
my_user = User.create(name: 'test user', email: 's@gmail.com') # Publishes `:
|
154
|
+
my_user = User.create!(name: 'test user', email: 's@gmail.com') # Publishes `:create` notification with classname `App1User` (App2 syncs the new user)
|
147
155
|
my_user.ps_publish(:send_welcome, mapping: %i[id email]) # Publishes `:send_welcome` notification (App2 prints "sending email to...")
|
148
|
-
PubSubModelSync::
|
156
|
+
PubSubModelSync::Payload.new({ ids: [my_user.id] }, { klass: 'User', action: :batch_disable, mode: :klass }).publish! # Publishes class notification (App2 prints "disabling users..")
|
149
157
|
```
|
150
158
|
|
151
159
|
## **API**
|
@@ -167,8 +175,8 @@ PubSubModelSync::MessagePublisher.publish_data(User, { ids: [my_user.id] }, :bat
|
|
167
175
|
- `settings` (Hash<:from_klass, :to_action, :id, :if, :unless>)
|
168
176
|
- `from_klass:` (String, default current class): Only notifications with this class name will be processed by this subscription
|
169
177
|
- `to_action:` (Symbol|Proc, default `action`):
|
170
|
-
When Symbol: Model method to process the notification
|
171
|
-
When Proc: Block to process the notification
|
178
|
+
When Symbol: Model method to process the notification, sample: `def my_method(data)...end`
|
179
|
+
When Proc: Block to process the notification, sample: `{|data| ... }`
|
172
180
|
- `id:` (Symbol|Array<Symbol|String>, default: `:id`) identifier attribute(s) to find the corresponding model instance (Supports for mapping format)
|
173
181
|
Sample: `id: :id` will search for a model like: `model_class.where(id: payload.data[:id])`
|
174
182
|
Sample: `id: [:id, :email:user_email]` will search for a model like: `model_class.where(id: payload.data[:id], user_email: payload.data[:email])`
|
@@ -205,20 +213,20 @@ PubSubModelSync::MessagePublisher.publish_data(User, { ids: [my_user.id] }, :bat
|
|
205
213
|
#### **Subscription helpers**
|
206
214
|
- List all configured subscriptions
|
207
215
|
```ruby
|
208
|
-
|
216
|
+
PubSubModelSync::Config.subscribers
|
209
217
|
```
|
210
|
-
-
|
218
|
+
- Process or reprocess a notification
|
211
219
|
```ruby
|
212
|
-
|
213
|
-
|
220
|
+
payload = PubSubModelSync::Payload.new(data, attributes, headers)
|
221
|
+
payload.process!
|
214
222
|
```
|
215
223
|
|
216
224
|
|
217
225
|
### **Publishers**
|
218
226
|
```ruby
|
219
227
|
class MyModel < ActiveRecord::Base
|
220
|
-
|
221
|
-
|
228
|
+
ps_after_action([:create, :update, :destroy], :method_publisher_name) # using method callback
|
229
|
+
ps_after_action([:create, :update, :destroy]) do |action| # using block callback
|
222
230
|
ps_publish(action, data: {}, mapping: [], headers: {}, as_klass: nil)
|
223
231
|
ps_class_publish({}, action: :my_action, as_klass: nil, headers: {})
|
224
232
|
end
|
@@ -231,25 +239,13 @@ PubSubModelSync::MessagePublisher.publish_data(User, { ids: [my_user.id] }, :bat
|
|
231
239
|
|
232
240
|
#### **Publishing notifications**
|
233
241
|
|
234
|
-
- `
|
242
|
+
- `ps_after_action(crud_actions, method_name = nil, &block)` Listens for CRUD events and calls provided `block` or `method` to process event callback
|
235
243
|
- `crud_actions` (Symbol|Array<Symbol>) Crud event(s) to be observed (Allowed: `:create, :update, :destroy`)
|
236
|
-
- `method_name` (Symbol, optional) method to be called to process action callback
|
237
|
-
- `block` (Proc, optional) Block to be called to process action callback
|
238
|
-
|
239
|
-
|
240
|
-
|
241
|
-
```
|
242
|
-
1: User notification
|
243
|
-
2: First post notification
|
244
|
-
3: Second post notification
|
245
|
-
|
246
|
-
**Note2**: Due to rails callback ordering, this method uses `after_destroy` callback when destroying models to ensure the expected notifications ordering.
|
247
|
-
```ruby
|
248
|
-
user.destroy
|
249
|
-
```
|
250
|
-
1: Second post notification
|
251
|
-
2: First post notification
|
252
|
-
3: User notification
|
244
|
+
- `method_name` (Symbol, optional) method to be called to process action callback, sample: `def my_method(action) ... end`
|
245
|
+
- `block` (Proc, optional) Block to be called to process action callback, sample: `{ |action| ... }`
|
246
|
+
|
247
|
+
**Note1**: Due to rails callback ordering, this method uses `before_commit` callback when creating or updating models to ensure expected notifications order (More details [**here**](#transactions)).
|
248
|
+
**Note2**: Due to rails callback ordering, this method uses `after_destroy` callback when destroying models to ensure the expected notifications order.
|
253
249
|
|
254
250
|
- `ps_publish(action, data: {}, mapping: [], headers: {}, as_klass: nil)` Delivers an instance notification via pubsub
|
255
251
|
- `action` (Sym|String) Action name of the instance notification. Sample: create|update|destroy|<any_other_key>
|
@@ -266,36 +262,23 @@ PubSubModelSync::MessagePublisher.publish_data(User, { ids: [my_user.id] }, :bat
|
|
266
262
|
- When Proc: Block to be called to retrieve header values (must return a `hash`, receives `:model, :action` as args)
|
267
263
|
- `as_klass:` (String, default current class name): Output class name used instead of current class name
|
268
264
|
|
269
|
-
- `ps_class_publish` Delivers a Class notification via pubsub
|
265
|
+
- `ps_class_publish(data, action:, as_klass: nil, headers: {})` Delivers a Class notification via pubsub
|
270
266
|
- `data` (Hash): Data of the notification
|
271
267
|
- `action` (Symbol): action name of the notification
|
272
268
|
- `as_klass:` (String, default current class name): Class name of the notification
|
273
269
|
- `headers:` (Hash, optional): header settings (More in Payload.headers)
|
270
|
+
|
271
|
+
- `ps_perform_publish(action = :create)` Permits to perform manually the callback of a specific `ps_after_action`
|
272
|
+
- `action` (Symbol, default: :create) Only :create|:update|:destroy
|
274
273
|
|
275
274
|
#### **Publisher helpers**
|
276
|
-
- Publish a
|
277
|
-
```ruby
|
278
|
-
PubSubModelSync::MessagePublisher.publish_data(klass, data, action, headers: )
|
279
|
-
```
|
280
|
-
- `klass`: (String) Class name to be used
|
281
|
-
- Refer to `ps_class_publish` except `as_klass:`
|
282
|
-
|
283
|
-
- Manually publish or republish a notification (useful when failed)
|
275
|
+
- Publish or republish a notification
|
284
276
|
```ruby
|
285
|
-
|
286
|
-
|
277
|
+
payload = PubSubModelSync::Payload.new(data, attributes, headers)
|
278
|
+
payload.publish!
|
287
279
|
```
|
288
280
|
|
289
281
|
#### **Publisher callbacks**
|
290
|
-
- Prevent delivering a notification (called before building payload)
|
291
|
-
If returns "true", will not publish notification
|
292
|
-
```ruby
|
293
|
-
class MyModel < ActiveRecord::Base
|
294
|
-
def ps_skip_publish?(action)
|
295
|
-
# logic here
|
296
|
-
end
|
297
|
-
end
|
298
|
-
```
|
299
282
|
|
300
283
|
- Do some actions before publishing notification.
|
301
284
|
If returns ":cancel", notification will not be delivered
|
@@ -328,7 +311,8 @@ Any notification before delivering is transformed as a Payload for a better port
|
|
328
311
|
- `mode`: (Symbol: `:model`|`:class`) Kind of notification
|
329
312
|
* `headers`: (Hash) Notification settings that defines how the notification will be processed or delivered.
|
330
313
|
- `key`: (String, optional) identifier of the payload, default: `<klass_name>/<action>` when class message, `<model.class.name>/<action>/<model.id>` when model message (Useful for caching techniques).
|
331
|
-
- `ordering_key`: (String, optional): messages with the same value are processed in the same order they were delivered, default: `klass_name` when class message, `<model.class.name>/<model.id>` when instance message
|
314
|
+
- `ordering_key`: (String, optional): messages with the same value are processed in the same order they were delivered, default: `klass_name` when class message, `<model.class.name>/<model.id>` when instance message.
|
315
|
+
Note: Final `ordering_key` is calculated by this way: `payload.headers[:forced_ordering_key] || current_transaction&.key || payload.headers[:ordering_key]`
|
332
316
|
- `topic_name`: (String|Array<String>, optional): Specific topic name (can be seen as a channel) to be used when delivering the message (default first topic from config).
|
333
317
|
- `forced_ordering_key`: (String, optional): Will force to use this value as the `ordering_key`, even withing transactions. Default `nil`.
|
334
318
|
|
@@ -345,20 +329,38 @@ Any notification before delivering is transformed as a Payload for a better port
|
|
345
329
|
* Crud syncs auto includes transactions which works as the following:
|
346
330
|
```ruby
|
347
331
|
class User
|
348
|
-
|
349
|
-
has_many :posts
|
332
|
+
ps_after_action([:create, :update, :destroy]) { |action| ps_publish(action, mapping: %i[id name]) }
|
333
|
+
has_many :posts, dependent: :destroy
|
350
334
|
accepts_nested_attributes_for :posts
|
351
335
|
end
|
352
336
|
|
353
337
|
class Post
|
354
338
|
belongs_to :user
|
355
|
-
|
339
|
+
ps_after_action([:create, :update, :destroy]) { |action| ps_publish(action, mapping: %i[id user_id title]) }
|
356
340
|
end
|
357
|
-
|
358
|
-
User.create!(name: 'test', posts_attributes: [{ title: 'Post 1' }, { title: 'Post 2' }])
|
359
341
|
```
|
360
|
-
When
|
361
|
-
|
342
|
+
- When created (all notifications use the same ordering key to be processed in the same order)
|
343
|
+
```ruby
|
344
|
+
user = User.create!(name: 'test', posts_attributes: [{ title: 'Post 1' }, { title: 'Post 2' }])
|
345
|
+
# notification #1 => <Payload data: {id: 1, name: 'sample'}, info: { klass: 'User', action: :create, mode: :model }, headers: { ordering_key = `User/1` }>
|
346
|
+
# notification #2 => <Payload data: {id: 1, title: 'Post 1', user_id: 1}, info: { klass: 'Post', action: :create, mode: :model }, headers: { ordering_key = `User/1` }>
|
347
|
+
# notification #3 => <Payload data: {id: 2, title: 'Post 2', user_id: 1}, info: { klass: 'Post', action: :create, mode: :model }, headers: { ordering_key = `User/1` }>
|
348
|
+
```
|
349
|
+
- When updated (all notifications use the same ordering key to be processed in the same order)
|
350
|
+
```ruby
|
351
|
+
user.update!(name: 'changed', posts_attributes: [{ id: 1, title: 'Post 1C' }, { id: 2, title: 'Post 2C' }])
|
352
|
+
# notification #1 => <Payload data: {id: 1, name: 'changed'}, info: { klass: 'User', action: :update, mode: :model }, headers: { ordering_key = `User/1` }>
|
353
|
+
# notification #2 => <Payload data: {id: 1, title: 'Post 1C', user_id: 1}, info: { klass: 'Post', action: :update, mode: :model }, headers: { ordering_key = `User/1` }>
|
354
|
+
# notification #3 => <Payload data: {id: 2, title: 'Post 2C', user_id: 1}, info: { klass: 'Post', action: :update, mode: :model }, headers: { ordering_key = `User/1` }>
|
355
|
+
```
|
356
|
+
- When destroyed (all notifications use the same ordering key to be processed in the same order)
|
357
|
+
**Note**: The notifications order were reordered in order to avoid inconsistency in other apps
|
358
|
+
```ruby
|
359
|
+
user.destroy!
|
360
|
+
# notification #1 => <Payload data: {id: 1, title: 'Post 1C', user_id: 1}, info: { klass: 'Post', action: :destroy, mode: :model }>
|
361
|
+
# notification #2 => <Payload data: {id: 2, title: 'Post 2C', user_id: 1}, info: { klass: 'Post', action: :destroy, mode: :model }>
|
362
|
+
# notification #3 => <Payload data: {id: 1, name: 'changed'}, info: { klass: 'User', action: :destroy, mode: :model }>
|
363
|
+
```
|
362
364
|
By this way parent notification and all inner notifications are processed in the same order they were published (includes notifications from callbacks like `ps_before_publish`).
|
363
365
|
|
364
366
|
**Note**: When any error is raised when saving user or posts, the transaction is cancelled and thus all notifications wont be delivered (customizable by `PubSubModelSync::Config.transactions_use_buffer`).
|
@@ -374,14 +376,14 @@ Any notification before delivering is transformed as a Payload for a better port
|
|
374
376
|
PubSubModelSync::MessagePublisher::transaction('my-custom-key') do
|
375
377
|
user = User.create(name: 'test') # `User`:`:create` notification
|
376
378
|
post = Post.create(title: 'sample') # `Post`:`:create` notification
|
377
|
-
PubSubModelSync::
|
379
|
+
PubSubModelSync::Payload.new({ ids: [user.id] }, { klass: 'User', action: :send_welcome, mode: :klass }).publish! # `User`:`:send_welcome` notification
|
378
380
|
end
|
379
381
|
```
|
380
382
|
All notifications uses `ordering_key: 'my-custom-key'` and will be processed in the same order they were published.
|
381
383
|
|
382
384
|
## **Testing with RSpec**
|
383
385
|
- Config: (spec/rails_helper.rb)
|
384
|
-
|
386
|
+
```ruby
|
385
387
|
|
386
388
|
# when using google service
|
387
389
|
require 'pub_sub_model_sync/mock_google_service'
|
@@ -404,51 +406,119 @@ Any notification before delivering is transformed as a Payload for a better port
|
|
404
406
|
allow(Kafka).to receive(:new).and_return(kafka_mock)
|
405
407
|
end
|
406
408
|
|
407
|
-
#
|
409
|
+
# disable all models sync by default (reduces testing time)
|
408
410
|
config.before(:each) do
|
409
|
-
# **** disable payloads generation, sync callbacks to improve tests speed
|
410
411
|
allow(PubSubModelSync::MessagePublisher).to receive(:publish_data) # disable class level notif
|
411
412
|
allow(PubSubModelSync::MessagePublisher).to receive(:publish_model) # disable instance level notif
|
412
|
-
|
413
|
-
# **** when testing model syncs, it can be re enabled by:
|
414
|
-
# before do
|
415
|
-
# allow(PubSubModelSync::MessagePublisher).to receive(:publish_data).and_call_original
|
416
|
-
# allow(PubSubModelSync::MessagePublisher).to receive(:publish_model).and_call_original
|
417
|
-
# end
|
418
413
|
end
|
419
|
-
|
414
|
+
|
415
|
+
# enable all models sync only for tests that includes 'sync: true'
|
416
|
+
config.before(:each, sync: true) do
|
417
|
+
allow(PubSubModelSync::MessagePublisher).to receive(:publish_data).and_call_original
|
418
|
+
allow(PubSubModelSync::MessagePublisher).to receive(:publish_model).and_call_original
|
419
|
+
end
|
420
|
+
|
421
|
+
# Only when using database cleaner in old versions of rspec (enables after_commit callback)
|
422
|
+
# config.before(:each, truncate: true) do
|
423
|
+
# DatabaseCleaner.strategy = :truncation
|
424
|
+
# end
|
425
|
+
```
|
420
426
|
- Examples:
|
427
|
+
- **Publisher**
|
421
428
|
```ruby
|
422
|
-
|
423
|
-
|
424
|
-
|
429
|
+
# Do not forget to include 'sync: true' to enable publishing pubsub notifications
|
430
|
+
describe 'When publishing sync', truncate: true, sync: true do
|
431
|
+
it 'publishes user notification when created' do
|
432
|
+
expect_publish_notification(:create, klass: 'User')
|
433
|
+
create(:user)
|
434
|
+
end
|
435
|
+
|
436
|
+
it 'publishes user notification with all defined data' do
|
437
|
+
user = build(:user)
|
438
|
+
data = PubSubModelSync::PayloadBuilder.parse_mapping_for(user, %i[id name:full_name email])
|
439
|
+
data[:id] = be_a(Integer)
|
440
|
+
expect_publish_notification(:create, klass: 'User', data: data)
|
441
|
+
user.save!
|
442
|
+
end
|
443
|
+
|
444
|
+
it 'publishes user notification when created' do
|
445
|
+
email = 'Newemail@gmail.com'
|
446
|
+
user = create(:user)
|
447
|
+
expect_publish_notification(:update, klass: 'User', data: { id: user.id, email: email })
|
448
|
+
user.update!(email: email)
|
449
|
+
end
|
450
|
+
|
451
|
+
it 'publishes user notification when created' do
|
452
|
+
user = create(:user)
|
453
|
+
expect_publish_notification(:destroy, klass: 'User', data: { id: user.id })
|
454
|
+
user.destroy!
|
455
|
+
end
|
456
|
+
|
457
|
+
private
|
458
|
+
|
459
|
+
# @param action (Symbol)
|
460
|
+
# @param klass (String, default described_class name)
|
461
|
+
# @param data (Hash, optional) notification data
|
462
|
+
# @param info (Hash, optional) notification info
|
463
|
+
# @param headers (Hash, optional) notification headers
|
464
|
+
def expect_publish_notification(action, klass: described_class.to_s, data: {}, info: {}, headers: {})
|
465
|
+
publisher = PubSubModelSync::MessagePublisher
|
466
|
+
exp_data = have_attributes(data: hash_including(data),
|
467
|
+
info: hash_including(info.merge(klass: klass, action: action)),
|
468
|
+
headers: hash_including(headers))
|
469
|
+
allow(publisher).to receive(:publish!).and_call_original
|
470
|
+
expect(publisher).to receive(:publish!).with(exp_data)
|
471
|
+
end
|
472
|
+
end
|
473
|
+
```
|
474
|
+
- **Subscriber**
|
475
|
+
```ruby
|
476
|
+
|
477
|
+
describe 'when syncing data from other apps' do
|
478
|
+
it 'creates user when received :create notification' do
|
479
|
+
user = build(:user)
|
480
|
+
data = user.as_json(only: %i[name email]).merge(id: 999)
|
425
481
|
payload = PubSubModelSync::Payload.new(data, { klass: 'User', action: :create })
|
482
|
+
expect { payload.process! }.to change(described_class, :count)
|
483
|
+
end
|
484
|
+
|
485
|
+
it 'updates user when received :update notification' do
|
486
|
+
user = create(:user)
|
487
|
+
name = 'new name'
|
488
|
+
data = user.as_json(only: %i[id email]).merge(name: name)
|
489
|
+
payload = PubSubModelSync::Payload.new(data, { klass: 'User', action: :update })
|
490
|
+
payload.process!
|
491
|
+
expect(user.reload.name).to eq(name)
|
492
|
+
end
|
493
|
+
|
494
|
+
it 'destroys user when received :destroy notification' do
|
495
|
+
user = create(:user)
|
496
|
+
data = user.as_json(only: %i[id])
|
497
|
+
payload = PubSubModelSync::Payload.new(data, { klass: 'User', action: :destroy })
|
498
|
+
payload.process!
|
499
|
+
expect { user.reload }.to raise_error(ActiveRecord::RecordNotFound)
|
500
|
+
end
|
501
|
+
|
502
|
+
|
503
|
+
it 'receive custom model notification' do
|
504
|
+
user = create(:user)
|
505
|
+
data = { id: user.id, custom_data: {} }
|
506
|
+
custom_action = :say_hello
|
507
|
+
expect_any_instance_of(User).to receive(custom_action).with(data)
|
508
|
+
payload = PubSubModelSync::Payload.new(data, { klass: 'User', action: custom_action })
|
426
509
|
payload.process!
|
427
|
-
expect(User.where(id: data[:id])).to be_any
|
428
510
|
end
|
429
511
|
|
430
512
|
it 'receive class notification' do
|
431
513
|
data = { msg: 'hello' }
|
432
514
|
action = :greeting
|
515
|
+
expect(User).to receive(action).with(data)
|
516
|
+
# Do not forget to include `mode: :klass` for class notifications
|
433
517
|
payload = PubSubModelSync::Payload.new(data, { klass: 'User', action: action, mode: :klass })
|
434
518
|
payload.process!
|
435
|
-
expect(User).to receive(action)
|
436
519
|
end
|
437
|
-
|
438
|
-
|
439
|
-
it 'publish model notification' do
|
440
|
-
publisher = PubSubModelSync::MessagePublisher
|
441
|
-
user = User.create(name: 'name', email: 'email')
|
442
|
-
expect(publisher).to receive(:publish_model).with(user, :create, anything)
|
443
|
-
end
|
444
|
-
|
445
|
-
it 'publish class notification' do
|
446
|
-
publisher = PubSubModelSync::MessagePublisher
|
447
|
-
user = User.create(name: 'name', email: 'email')
|
448
|
-
user.ps_class_publish({msg: 'hello'}, action: :greeting)
|
449
|
-
expect(publisher).to receive(:publish_data).with('User', data, :greeting)
|
450
|
-
end
|
451
|
-
```
|
520
|
+
end
|
521
|
+
```
|
452
522
|
|
453
523
|
## **Extra configurations**
|
454
524
|
```ruby
|
@@ -481,7 +551,7 @@ config.debug = true
|
|
481
551
|
(Proc) => called when failed publishing a message (delayed_job or similar can be used for retrying)
|
482
552
|
- ```.transactions_max_buffer = 100``` (Integer) Once this quantity of notifications is reached, then all notifications will immediately be delivered.
|
483
553
|
Note: There is no way to rollback delivered notifications if current transaction fails
|
484
|
-
- ```.enable_rails4_before_commit = true``` (true*|false) When false will disable rails 4 hack compatibility and then CRUD notifications will be prepared using `after_commit` callback instead of `before_commit` which will not rollback sql transactions if
|
554
|
+
- ```.enable_rails4_before_commit = true``` (true*|false) When false will disable rails 4 hack compatibility and then CRUD notifications will be prepared using `after_commit` callback instead of `before_commit` (used in `ps_after_action(...)`) which will not rollback sql transactions if failed when publishing pubsub notification.
|
485
555
|
|
486
556
|
## **TODO**
|
487
557
|
- Auto publish update only if payload has changed (see ways to compare previous payload vs new payload)
|
@@ -490,7 +560,6 @@ config.debug = true
|
|
490
560
|
- Add DB table to use as a shield to prevent publishing similar notifications and publish partial notifications (similar idea when processing notif)
|
491
561
|
- Last notification is not being delivered immediately in google pubsub (maybe force with timeout 10secs and service.deliver_messages)
|
492
562
|
- Update folder structure
|
493
|
-
- Support for blocks in ps_publish and ps_subscribe
|
494
563
|
- Services support to deliver multiple payloads from transactions
|
495
564
|
|
496
565
|
## **Q&A**
|