pub_sub_model_sync 1.0.beta2 → 1.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (109) hide show
  1. checksums.yaml +4 -4
  2. data/.github/workflows/release.yml +43 -0
  3. data/CHANGELOG.md +7 -4
  4. data/Gemfile.lock +3 -1
  5. data/README.md +174 -105
  6. data/docs/notifications-diagram.png +0 -0
  7. data/lib/pub_sub_model_sync.rb +1 -1
  8. data/lib/pub_sub_model_sync/base.rb +0 -20
  9. data/lib/pub_sub_model_sync/config.rb +1 -1
  10. data/lib/pub_sub_model_sync/initializers/before_commit.rb +3 -3
  11. data/lib/pub_sub_model_sync/message_processor.rb +32 -9
  12. data/lib/pub_sub_model_sync/message_publisher.rb +14 -10
  13. data/lib/pub_sub_model_sync/payload.rb +15 -12
  14. data/lib/pub_sub_model_sync/{publisher.rb → payload_builder.rb} +15 -10
  15. data/lib/pub_sub_model_sync/publisher_concern.rb +27 -16
  16. data/lib/pub_sub_model_sync/run_subscriber.rb +17 -13
  17. data/lib/pub_sub_model_sync/service_base.rb +5 -32
  18. data/lib/pub_sub_model_sync/service_google.rb +1 -1
  19. data/lib/pub_sub_model_sync/subscriber_concern.rb +6 -4
  20. data/lib/pub_sub_model_sync/transaction.rb +6 -2
  21. data/lib/pub_sub_model_sync/version.rb +1 -1
  22. data/samples/README.md +50 -0
  23. data/samples/app1/.gitattributes +8 -0
  24. data/samples/app1/.gitignore +28 -0
  25. data/samples/app1/Dockerfile +13 -0
  26. data/samples/app1/Gemfile +37 -0
  27. data/samples/app1/Gemfile.lock +171 -0
  28. data/samples/app1/README.md +24 -0
  29. data/samples/app1/Rakefile +6 -0
  30. data/samples/app1/app/models/application_record.rb +3 -0
  31. data/samples/app1/app/models/concerns/.keep +0 -0
  32. data/samples/app1/app/models/post.rb +19 -0
  33. data/samples/app1/app/models/user.rb +29 -0
  34. data/samples/app1/bin/bundle +114 -0
  35. data/samples/app1/bin/rails +5 -0
  36. data/samples/app1/bin/rake +5 -0
  37. data/samples/app1/bin/setup +33 -0
  38. data/samples/app1/bin/spring +14 -0
  39. data/samples/app1/config.ru +6 -0
  40. data/samples/app1/config/application.rb +40 -0
  41. data/samples/app1/config/boot.rb +4 -0
  42. data/samples/app1/config/credentials.yml.enc +1 -0
  43. data/samples/app1/config/database.yml +25 -0
  44. data/samples/app1/config/environment.rb +5 -0
  45. data/samples/app1/config/environments/development.rb +63 -0
  46. data/samples/app1/config/environments/production.rb +105 -0
  47. data/samples/app1/config/environments/test.rb +57 -0
  48. data/samples/app1/config/initializers/application_controller_renderer.rb +8 -0
  49. data/samples/app1/config/initializers/backtrace_silencers.rb +8 -0
  50. data/samples/app1/config/initializers/cors.rb +16 -0
  51. data/samples/app1/config/initializers/filter_parameter_logging.rb +6 -0
  52. data/samples/app1/config/initializers/inflections.rb +16 -0
  53. data/samples/app1/config/initializers/mime_types.rb +4 -0
  54. data/samples/app1/config/initializers/pubsub.rb +4 -0
  55. data/samples/app1/config/initializers/wrap_parameters.rb +14 -0
  56. data/samples/app1/config/locales/en.yml +33 -0
  57. data/samples/app1/config/puma.rb +43 -0
  58. data/samples/app1/config/routes.rb +3 -0
  59. data/samples/app1/config/spring.rb +6 -0
  60. data/samples/app1/db/migrate/20210513080700_create_users.rb +12 -0
  61. data/samples/app1/db/migrate/20210513134332_create_posts.rb +11 -0
  62. data/samples/app1/db/schema.rb +34 -0
  63. data/samples/app1/db/seeds.rb +7 -0
  64. data/samples/app1/docker-compose.yml +32 -0
  65. data/samples/app1/log/.keep +0 -0
  66. data/samples/app2/.gitattributes +8 -0
  67. data/samples/app2/.gitignore +28 -0
  68. data/samples/app2/Dockerfile +13 -0
  69. data/samples/app2/Gemfile +37 -0
  70. data/samples/app2/Gemfile.lock +171 -0
  71. data/samples/app2/README.md +24 -0
  72. data/samples/app2/Rakefile +6 -0
  73. data/samples/app2/app/models/application_record.rb +9 -0
  74. data/samples/app2/app/models/concerns/.keep +0 -0
  75. data/samples/app2/app/models/customer.rb +28 -0
  76. data/samples/app2/app/models/post.rb +10 -0
  77. data/samples/app2/bin/bundle +114 -0
  78. data/samples/app2/bin/rails +5 -0
  79. data/samples/app2/bin/rake +5 -0
  80. data/samples/app2/bin/setup +33 -0
  81. data/samples/app2/bin/spring +14 -0
  82. data/samples/app2/config.ru +6 -0
  83. data/samples/app2/config/application.rb +40 -0
  84. data/samples/app2/config/boot.rb +4 -0
  85. data/samples/app2/config/credentials.yml.enc +1 -0
  86. data/samples/app2/config/database.yml +25 -0
  87. data/samples/app2/config/environment.rb +5 -0
  88. data/samples/app2/config/environments/development.rb +63 -0
  89. data/samples/app2/config/environments/production.rb +105 -0
  90. data/samples/app2/config/environments/test.rb +57 -0
  91. data/samples/app2/config/initializers/application_controller_renderer.rb +8 -0
  92. data/samples/app2/config/initializers/backtrace_silencers.rb +8 -0
  93. data/samples/app2/config/initializers/cors.rb +16 -0
  94. data/samples/app2/config/initializers/filter_parameter_logging.rb +6 -0
  95. data/samples/app2/config/initializers/inflections.rb +16 -0
  96. data/samples/app2/config/initializers/mime_types.rb +4 -0
  97. data/samples/app2/config/initializers/pubsub.rb +4 -0
  98. data/samples/app2/config/initializers/wrap_parameters.rb +14 -0
  99. data/samples/app2/config/locales/en.yml +33 -0
  100. data/samples/app2/config/puma.rb +43 -0
  101. data/samples/app2/config/routes.rb +3 -0
  102. data/samples/app2/config/spring.rb +6 -0
  103. data/samples/app2/db/migrate/20210513080956_create_customers.rb +10 -0
  104. data/samples/app2/db/migrate/20210513135203_create_posts.rb +10 -0
  105. data/samples/app2/db/schema.rb +31 -0
  106. data/samples/app2/db/seeds.rb +7 -0
  107. data/samples/app2/docker-compose.yml +20 -0
  108. data/samples/app2/log/.keep +0 -0
  109. metadata +93 -5
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 50a27cf5815a6b9be1f38f399e47569d403831d61dec25b7b4ffdc126f6f450e
4
- data.tar.gz: 3a97f970d4f0ca08387b59b6da53e0be9f2ce3429db3baf41b5f60b004300b6f
3
+ metadata.gz: caaffd468399e03dbf5582bf6973339ec61348a6dcc67a87bda1d930f9271aae
4
+ data.tar.gz: 9db2cb5f4a70c218d720130761a24ad7f6450531d845f9517e74968f409c9743
5
5
  SHA512:
6
- metadata.gz: ee52cce7dba866949b64882e385e5b88aabe4725a18fe8b8f4abbf156d50bf2dc840421fd94e2a65a4cc1580c12ff2e086378101cdfbac812745949af1e85c6a
7
- data.tar.gz: ed67b56451d963e63163d1afb03ffd248ad9033bbad9e08b214afc0b15c933608eefb134ca7c67da8c32e55324fe0f1da8dbcc36da440d9f8d5b3c194c41fdf5
6
+ metadata.gz: ac6c2b0087e411969e4a0fbe6fa4aa57e3fe46b6ff93f4d675e2a213cd0fd75786b5e16e1a4c1df6aee92296eac0c94a054a6b62543a0f7f643ba2ea0bc8ff7b
7
+ data.tar.gz: 9816c7cedd0d6289d0d73a55f35ede63a46bca7f38805c4c566012fed04a58595498a80fd272ce89b617e34b1d1487ff2ee7f6e050c9712bfd6d75b364ad910a
@@ -0,0 +1,43 @@
1
+ on:
2
+ push:
3
+ tags: # triggered once a git tag is published
4
+ - '*'
5
+
6
+ name: Create Release
7
+
8
+ jobs:
9
+ build:
10
+ name: Create Release
11
+ runs-on: ubuntu-latest
12
+ steps:
13
+ - name: Checkout code
14
+ uses: actions/checkout@v2
15
+ with:
16
+ fetch-depth: 0
17
+
18
+ # Changelog action adaptations
19
+ - name: Create required package.json
20
+ run: test -f package.json || echo '{}' >package.json
21
+ - name: Detect Previous Tag (action not detecting very well)
22
+ run: echo "::set-output name=previous_tag::$(git describe --abbrev=0 --tags `git rev-list --tags --skip=1 --max-count=1`)"
23
+ id: tag_checker
24
+
25
+ - name: Generate Changelog
26
+ uses: scottbrenner/generate-changelog-action@master
27
+ id: Changelog
28
+ with:
29
+ from-tag: ${{steps.tag_checker.outputs.previous_tag}}
30
+ to-tag: HEAD
31
+
32
+ - name: Create Release
33
+ id: create_release
34
+ uses: actions/create-release@latest
35
+ env:
36
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # This token is provided by Actions, you do not need to create your own token
37
+ with:
38
+ tag_name: ${{ github.ref }}
39
+ release_name: Release ${{ github.ref }}
40
+ body: |
41
+ ${{ steps.Changelog.outputs.changelog }}
42
+ draft: false
43
+ prerelease: false
data/CHANGELOG.md CHANGED
@@ -1,21 +1,24 @@
1
1
  # Change Log
2
2
 
3
- # 1.0.beta (May 13, 2021)
3
+ # 1.0 (June 13, 2021)
4
+ This version includes many changes that was refactored from previous version, and thus it needs manual changes to migrate into this version.
4
5
  - Refactor: Subscribers param renamed `from_action` into `to_action` and added support for block or lambda
5
6
  - Feat: Improved `ps_subscribe` to accept new arguments and support for property mappings
6
7
  - Refactor: Refactored `ps_publish` to be called manually (removes notification assumptions) and accept for new arguments
7
- - Feat: Added `ps_on_crud_event` to listen CRUD events to send notifications in the expected order
8
+ - Feat: Added `ps_after_action` to listen CRUD events to send notifications in the expected order
8
9
  - Feat: Added `config.default_topic_name` to define default topic name whe publishing (by default `config.topic_name`)
9
10
  - Refactor: Refactored PubSub Transactions to support rollbacks (any exception inside transactions can automatically cancel all pending notifications: configurable through `config.transactions_use_buffer`)
10
11
  - Feat: Improved CRUD transactions to deliver inner notifications in the expected order to keep data consistency
11
12
  - System refactor: Added subscriber runner
12
13
  - Fix: Class notifications can only be listened by class subscriptions
13
14
  - Refactor: Removed `publish_model_data` to have a unique model publisher `ps_publish`
14
- - Refactor: Renamed `ps_before_sync` into `ps_before_publish`, `ps_skip_sync` into `ps_skip_publish`, `ps_after_sync` into `ps_after_publish`
15
+ - Refactor: Renamed `ps_before_sync` into `ps_before_publish`, `ps_after_sync` into `ps_after_publish`
15
16
  - Refactor: Renamed `payload.attributes` into `payload.info`
16
17
  - Feat: Support for plain Ruby Objects (Non ActiveRecord models)
17
18
  - Fix: Retry errors for 5 times before exiting notifications listener
18
- - Feat: Added transactions max_buffer
19
+ - Feat: Added transactions max_buffer
20
+ - Feat: add `ps_perform_publish` to perform the callback for a specific action
21
+ - Feat: Removed `ps_skip_sync` callback
19
22
 
20
23
  # 0.6.0 (March 03, 2021)
21
24
  - feat: add support to include custom payload headers
data/Gemfile.lock CHANGED
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- pub_sub_model_sync (1.0.beta2)
4
+ pub_sub_model_sync (1.0)
5
5
  rails
6
6
 
7
7
  GEM
@@ -158,6 +158,8 @@ GEM
158
158
  nio4r (2.5.7)
159
159
  nokogiri (1.11.3-x86_64-darwin)
160
160
  racc (~> 1.4)
161
+ nokogiri (1.11.3-x86_64-linux)
162
+ racc (~> 1.4)
161
163
  os (1.1.1)
162
164
  parallel (1.20.1)
163
165
  parser (3.0.1.1)
data/README.md CHANGED
@@ -1,5 +1,9 @@
1
1
  # **PubSubModelSync**
2
- This gem permits to sync automatically model data, send custom notifications between multiple Rails applications by publishing notifications via pubsub (Google PubSub, RabbitMQ, or Apache Kafka). Out of the scope this gem includes transactions to keep Data consistency by processing notifications in the order they were delivered.
2
+ ![Rails badge](https://img.shields.io/badge/Rails-4+-success.png)
3
+ ![Ruby badge](https://img.shields.io/badge/Ruby-2.4+-success.png)
4
+ ![Production badge](https://img.shields.io/badge/Production-ready-success.png)
5
+
6
+ This gem permits to sync automatically models and custom data between multiple Rails applications by publishing notifications via pubsub (Google PubSub, RabbitMQ, or Apache Kafka) and automatically processed by all connected applications. Out of the scope, this gem includes transactions to keep Data consistency by processing notifications in the order they were delivered.
3
7
  These notifications use JSON format to easily be decoded by subscribers (Rails applications and even other languages)
4
8
 
5
9
  - [**PubSubModelSync**](#pubsubmodelsync)
@@ -29,12 +33,12 @@ These notifications use JSON format to easily be decoded by subscribers (Rails a
29
33
  - [**Code of Conduct**](#code-of-conduct)
30
34
 
31
35
  ## **Features**
32
- - Sync model data between Rails apps: All changes made on App1, will be immediately reflected on App2, App3, etc.
36
+ - Sync model data between Rails apps: All changes made on App1, will be immediately reflected on App2, App3, etc.
33
37
  Example: If User is created on App1, this user will be created on App2, App3 too with the accepted attributes.
34
- - Ability to send class level communications
38
+ - Ability to send instance and class level notifications
35
39
  Example: If App1 wants to send emails to multiple users, this can be listened on App2, to deliver corresponding emails
36
40
  - Change pub/sub service at any time: Switch between rabbitmq, kafka, google pubsub
37
- - Support for transactions: Permits to keep data consistency between applications by processing notifications in the same order they were delivered.
41
+ - Support for transactions: Permits to keep data consistency between applications by processing notifications in the same order they were delivered (auto included in models transactions).
38
42
  - Ability to send notifications to a specific topic (single application) or multiple topics (multiple applications)
39
43
 
40
44
  ## **Installation**
@@ -56,7 +60,7 @@ And then execute: $ bundle install
56
60
  # initializers/pub_sub_config.rb
57
61
  PubSubModelSync::Config.service_name = :google
58
62
  PubSubModelSync::Config.project = 'google-project-id'
59
- PubSubModelSync::Config.credentials = 'path-to-the-config'
63
+ PubSubModelSync::Config.credentials = 'path-to-google-config.json'
60
64
  PubSubModelSync::Config.topic_name = 'sample-topic'
61
65
  PubSubModelSync::Config.subscription_name = 'my-app1'
62
66
  ```
@@ -85,12 +89,14 @@ And then execute: $ bundle install
85
89
 
86
90
  - Start subscribers to listen for publishers (Only in the app that has subscribers)
87
91
  ```bash
88
- DB_POOL=20 bundle exec rake pub_sub_model_sync:start
92
+ DB_POOL=20 bundle exec rake pub_sub_model_sync:start
89
93
  ```
90
94
  Note: You need more than 15 DB pools to avoid "could not obtain a connection from the pool within 5.000 seconds". https://devcenter.heroku.com/articles/concurrency-and-database-connections
91
95
 
92
96
  - Check the service status with:
93
- ```PubSubModelSync::MessagePublisher.publish_data('Test message', {sample_value: 10}, :create)```
97
+ ```ruby
98
+ PubSubModelSync::Payload.new({ my_data: 'here' }, { klass: 'MyClass', action: :sample_action }).publish!
99
+ ```
94
100
 
95
101
  - More configurations: [here](#extra-configurations)
96
102
 
@@ -103,9 +109,9 @@ And then execute: $ bundle install
103
109
  # App 1 (Publisher)
104
110
  class User < ActiveRecord::Base
105
111
  include PubSubModelSync::PublisherConcern
106
- ps_on_crud_event(:create) { ps_publish(:create, mapping: %i[id name email]) }
107
- ps_on_crud_event(:update) { ps_publish(:update, mapping: %i[id name email]) }
108
- ps_on_crud_event(:destroy) { ps_publish(:destroy, mapping: %i[id]) }
112
+ ps_after_action(:create) { ps_publish(:create, mapping: %i[id name email]) }
113
+ ps_after_action(:update) { ps_publish(:update, mapping: %i[id name email]) }
114
+ ps_after_action(:destroy) { ps_publish(:destroy, mapping: %i[id]) }
109
115
  end
110
116
 
111
117
  # App 2 (Subscriber)
@@ -115,9 +121,9 @@ class User < ActiveRecord::Base
115
121
  end
116
122
 
117
123
  # CRUD syncs
118
- my_user = User.create(name: 'test user', email: 'sample@gmail.com') # Publishes `:create` notification (App 2 syncs the new user)
119
- my_user.update(name: 'changed user') # Publishes `:update` notification (App2 updates changes)
120
- my_user.destroy # Publishes `:destroy` notification (App2 destroys the corresponding user)
124
+ my_user = User.create!(name: 'test user', email: 'sample@gmail.com') # Publishes `:create` notification (App 2 syncs the new user)
125
+ my_user.update!(name: 'changed user') # Publishes `:update` notification (App2 updates changes on user with the same id)
126
+ my_user.destroy! # Publishes `:destroy` notification (App2 destroys the corresponding user)
121
127
  ```
122
128
 
123
129
  ### **Advanced Example**
@@ -125,14 +131,16 @@ my_user.destroy # Publishes `:destroy` notification (App2 destroys the correspon
125
131
  # App 1 (Publisher)
126
132
  class User < ActiveRecord::Base
127
133
  include PubSubModelSync::PublisherConcern
128
- ps_on_crud_event([:create, :update]) { ps_publish(:save, mapping: %i[id name:full_name email], as_klass: 'App1User', headers: { topic_name: %i[topic1 topic2] }) }
134
+ ps_after_action([:create, :update]) do |action|
135
+ ps_publish(action, mapping: %i[name:full_name email], as_klass: 'App1User', headers: { topic_name: %i[topic1 topic2] })
136
+ end
129
137
  end
130
138
 
131
139
  # App 2 (Subscriber)
132
140
  class User < ActiveRecord::Base
133
141
  include PubSubModelSync::SubscriberConcern
134
- ps_subscribe(:save, %i[full_name:customer_name], id: [:id, :email], from_klass: 'App1User')
135
- ps_subscribe(:send_welcome, %i[email], to_action: :send_email, if: ->(model) { model.email.present? })
142
+ ps_subscribe([:create, :update], %i[full_name:customer_name], id: :email, from_klass: 'App1User')
143
+ ps_subscribe(:send_welcome, %i[email], id: :email, to_action: :send_email, if: ->(model) { model.email.present? })
136
144
  ps_class_subscribe(:batch_disable) # class subscription
137
145
 
138
146
  def send_email
@@ -143,9 +151,9 @@ class User < ActiveRecord::Base
143
151
  puts "disabling users: #{data[:ids]}"
144
152
  end
145
153
  end
146
- my_user = User.create(name: 'test user', email: 's@gmail.com') # Publishes `:save` notification as class name `App1User` (App2 syncs the new user)
154
+ my_user = User.create!(name: 'test user', email: 's@gmail.com') # Publishes `:create` notification with classname `App1User` (App2 syncs the new user)
147
155
  my_user.ps_publish(:send_welcome, mapping: %i[id email]) # Publishes `:send_welcome` notification (App2 prints "sending email to...")
148
- PubSubModelSync::MessagePublisher.publish_data(User, { ids: [my_user.id] }, :batch_disable) # Publishes class notification (App2 prints "disabling users..")
156
+ PubSubModelSync::Payload.new({ ids: [my_user.id] }, { klass: 'User', action: :batch_disable, mode: :klass }).publish! # Publishes class notification (App2 prints "disabling users..")
149
157
  ```
150
158
 
151
159
  ## **API**
@@ -167,8 +175,8 @@ PubSubModelSync::MessagePublisher.publish_data(User, { ids: [my_user.id] }, :bat
167
175
  - `settings` (Hash<:from_klass, :to_action, :id, :if, :unless>)
168
176
  - `from_klass:` (String, default current class): Only notifications with this class name will be processed by this subscription
169
177
  - `to_action:` (Symbol|Proc, default `action`):
170
- When Symbol: Model method to process the notification
171
- When Proc: Block to process the notification
178
+ When Symbol: Model method to process the notification, sample: `def my_method(data)...end`
179
+ When Proc: Block to process the notification, sample: `{|data| ... }`
172
180
  - `id:` (Symbol|Array<Symbol|String>, default: `:id`) identifier attribute(s) to find the corresponding model instance (Supports for mapping format)
173
181
  Sample: `id: :id` will search for a model like: `model_class.where(id: payload.data[:id])`
174
182
  Sample: `id: [:id, :email:user_email]` will search for a model like: `model_class.where(id: payload.data[:id], user_email: payload.data[:email])`
@@ -205,20 +213,20 @@ PubSubModelSync::MessagePublisher.publish_data(User, { ids: [my_user.id] }, :bat
205
213
  #### **Subscription helpers**
206
214
  - List all configured subscriptions
207
215
  ```ruby
208
- PubSubModelSync::Config.subscribers
216
+ PubSubModelSync::Config.subscribers
209
217
  ```
210
- - Manually process or reprocess a notification (useful when failed)
218
+ - Process or reprocess a notification
211
219
  ```ruby
212
- payload = PubSubModelSync::Payload.new(data, attributes, headers)
213
- payload.process!
220
+ payload = PubSubModelSync::Payload.new(data, attributes, headers)
221
+ payload.process!
214
222
  ```
215
223
 
216
224
 
217
225
  ### **Publishers**
218
226
  ```ruby
219
227
  class MyModel < ActiveRecord::Base
220
- ps_on_crud_event([:create, :update, :destroy], :method_publisher_name) # using method callback
221
- ps_on_crud_event([:create, :update, :destroy]) do |action| # using block callback
228
+ ps_after_action([:create, :update, :destroy], :method_publisher_name) # using method callback
229
+ ps_after_action([:create, :update, :destroy]) do |action| # using block callback
222
230
  ps_publish(action, data: {}, mapping: [], headers: {}, as_klass: nil)
223
231
  ps_class_publish({}, action: :my_action, as_klass: nil, headers: {})
224
232
  end
@@ -231,25 +239,13 @@ PubSubModelSync::MessagePublisher.publish_data(User, { ids: [my_user.id] }, :bat
231
239
 
232
240
  #### **Publishing notifications**
233
241
 
234
- - `ps_on_crud_event(crud_actions, method_name = nil, &block)` Listens for CRUD events and calls provided `block` or `method` to process event callback
242
+ - `ps_after_action(crud_actions, method_name = nil, &block)` Listens for CRUD events and calls provided `block` or `method` to process event callback
235
243
  - `crud_actions` (Symbol|Array<Symbol>) Crud event(s) to be observed (Allowed: `:create, :update, :destroy`)
236
- - `method_name` (Symbol, optional) method to be called to process action callback
237
- - `block` (Proc, optional) Block to be called to process action callback
238
- **Note1**: Due to rails callback ordering, this method uses `before_commit` callback when creating or updating models to ensure expected notifications order, sample:
239
- ```ruby
240
- user = User.create(name: 'asasas', posts_attributes: [{ title: 't1' }, { title: 't2' }])
241
- ```
242
- 1: User notification
243
- 2: First post notification
244
- 3: Second post notification
245
-
246
- **Note2**: Due to rails callback ordering, this method uses `after_destroy` callback when destroying models to ensure the expected notifications ordering.
247
- ```ruby
248
- user.destroy
249
- ```
250
- 1: Second post notification
251
- 2: First post notification
252
- 3: User notification
244
+ - `method_name` (Symbol, optional) method to be called to process action callback, sample: `def my_method(action) ... end`
245
+ - `block` (Proc, optional) Block to be called to process action callback, sample: `{ |action| ... }`
246
+
247
+ **Note1**: Due to rails callback ordering, this method uses `before_commit` callback when creating or updating models to ensure expected notifications order (More details [**here**](#transactions)).
248
+ **Note2**: Due to rails callback ordering, this method uses `after_destroy` callback when destroying models to ensure the expected notifications order.
253
249
 
254
250
  - `ps_publish(action, data: {}, mapping: [], headers: {}, as_klass: nil)` Delivers an instance notification via pubsub
255
251
  - `action` (Sym|String) Action name of the instance notification. Sample: create|update|destroy|<any_other_key>
@@ -266,36 +262,23 @@ PubSubModelSync::MessagePublisher.publish_data(User, { ids: [my_user.id] }, :bat
266
262
  - When Proc: Block to be called to retrieve header values (must return a `hash`, receives `:model, :action` as args)
267
263
  - `as_klass:` (String, default current class name): Output class name used instead of current class name
268
264
 
269
- - `ps_class_publish` Delivers a Class notification via pubsub
265
+ - `ps_class_publish(data, action:, as_klass: nil, headers: {})` Delivers a Class notification via pubsub
270
266
  - `data` (Hash): Data of the notification
271
267
  - `action` (Symbol): action name of the notification
272
268
  - `as_klass:` (String, default current class name): Class name of the notification
273
269
  - `headers:` (Hash, optional): header settings (More in Payload.headers)
270
+
271
+ - `ps_perform_publish(action = :create)` Permits to perform manually the callback of a specific `ps_after_action`
272
+ - `action` (Symbol, default: :create) Only :create|:update|:destroy
274
273
 
275
274
  #### **Publisher helpers**
276
- - Publish a class notification from anywhere
277
- ```ruby
278
- PubSubModelSync::MessagePublisher.publish_data(klass, data, action, headers: )
279
- ```
280
- - `klass`: (String) Class name to be used
281
- - Refer to `ps_class_publish` except `as_klass:`
282
-
283
- - Manually publish or republish a notification (useful when failed)
275
+ - Publish or republish a notification
284
276
  ```ruby
285
- payload = PubSubModelSync::Payload.new(data, attributes, headers)
286
- payload.publish!
277
+ payload = PubSubModelSync::Payload.new(data, attributes, headers)
278
+ payload.publish!
287
279
  ```
288
280
 
289
281
  #### **Publisher callbacks**
290
- - Prevent delivering a notification (called before building payload)
291
- If returns "true", will not publish notification
292
- ```ruby
293
- class MyModel < ActiveRecord::Base
294
- def ps_skip_publish?(action)
295
- # logic here
296
- end
297
- end
298
- ```
299
282
 
300
283
  - Do some actions before publishing notification.
301
284
  If returns ":cancel", notification will not be delivered
@@ -328,7 +311,8 @@ Any notification before delivering is transformed as a Payload for a better port
328
311
  - `mode`: (Symbol: `:model`|`:class`) Kind of notification
329
312
  * `headers`: (Hash) Notification settings that defines how the notification will be processed or delivered.
330
313
  - `key`: (String, optional) identifier of the payload, default: `<klass_name>/<action>` when class message, `<model.class.name>/<action>/<model.id>` when model message (Useful for caching techniques).
331
- - `ordering_key`: (String, optional): messages with the same value are processed in the same order they were delivered, default: `klass_name` when class message, `<model.class.name>/<model.id>` when instance message
314
+ - `ordering_key`: (String, optional): messages with the same value are processed in the same order they were delivered, default: `klass_name` when class message, `<model.class.name>/<model.id>` when instance message.
315
+ Note: Final `ordering_key` is calculated by this way: `payload.headers[:forced_ordering_key] || current_transaction&.key || payload.headers[:ordering_key]`
332
316
  - `topic_name`: (String|Array<String>, optional): Specific topic name (can be seen as a channel) to be used when delivering the message (default first topic from config).
333
317
  - `forced_ordering_key`: (String, optional): Will force to use this value as the `ordering_key`, even withing transactions. Default `nil`.
334
318
 
@@ -345,20 +329,38 @@ Any notification before delivering is transformed as a Payload for a better port
345
329
  * Crud syncs auto includes transactions which works as the following:
346
330
  ```ruby
347
331
  class User
348
- ps_on_crud_event(:create) { ps_publish(:create, mapping: %i[id name]) }
349
- has_many :posts
332
+ ps_after_action([:create, :update, :destroy]) { |action| ps_publish(action, mapping: %i[id name]) }
333
+ has_many :posts, dependent: :destroy
350
334
  accepts_nested_attributes_for :posts
351
335
  end
352
336
 
353
337
  class Post
354
338
  belongs_to :user
355
- ps_on_crud_event(:create) { ps_publish(:create, mapping: %i[id title]) }
339
+ ps_after_action([:create, :update, :destroy]) { |action| ps_publish(action, mapping: %i[id user_id title]) }
356
340
  end
357
-
358
- User.create!(name: 'test', posts_attributes: [{ title: 'Post 1' }, { title: 'Post 2' }])
359
341
  ```
360
- When user is created, `User`:`:save` notification is published with the ordering_key = `User/<user_id>`.
361
- Posts created together with the user model publishes `Post`:`:save` notification each one using its parents (user model) `ordering_key`.
342
+ - When created (all notifications use the same ordering key to be processed in the same order)
343
+ ```ruby
344
+ user = User.create!(name: 'test', posts_attributes: [{ title: 'Post 1' }, { title: 'Post 2' }])
345
+ # notification #1 => <Payload data: {id: 1, name: 'sample'}, info: { klass: 'User', action: :create, mode: :model }, headers: { ordering_key = `User/1` }>
346
+ # notification #2 => <Payload data: {id: 1, title: 'Post 1', user_id: 1}, info: { klass: 'Post', action: :create, mode: :model }, headers: { ordering_key = `User/1` }>
347
+ # notification #3 => <Payload data: {id: 2, title: 'Post 2', user_id: 1}, info: { klass: 'Post', action: :create, mode: :model }, headers: { ordering_key = `User/1` }>
348
+ ```
349
+ - When updated (all notifications use the same ordering key to be processed in the same order)
350
+ ```ruby
351
+ user.update!(name: 'changed', posts_attributes: [{ id: 1, title: 'Post 1C' }, { id: 2, title: 'Post 2C' }])
352
+ # notification #1 => <Payload data: {id: 1, name: 'changed'}, info: { klass: 'User', action: :update, mode: :model }, headers: { ordering_key = `User/1` }>
353
+ # notification #2 => <Payload data: {id: 1, title: 'Post 1C', user_id: 1}, info: { klass: 'Post', action: :update, mode: :model }, headers: { ordering_key = `User/1` }>
354
+ # notification #3 => <Payload data: {id: 2, title: 'Post 2C', user_id: 1}, info: { klass: 'Post', action: :update, mode: :model }, headers: { ordering_key = `User/1` }>
355
+ ```
356
+ - When destroyed (all notifications use the same ordering key to be processed in the same order)
357
+ **Note**: The notifications order were reordered in order to avoid inconsistency in other apps
358
+ ```ruby
359
+ user.destroy!
360
+ # notification #1 => <Payload data: {id: 1, title: 'Post 1C', user_id: 1}, info: { klass: 'Post', action: :destroy, mode: :model }>
361
+ # notification #2 => <Payload data: {id: 2, title: 'Post 2C', user_id: 1}, info: { klass: 'Post', action: :destroy, mode: :model }>
362
+ # notification #3 => <Payload data: {id: 1, name: 'changed'}, info: { klass: 'User', action: :destroy, mode: :model }>
363
+ ```
362
364
  By this way parent notification and all inner notifications are processed in the same order they were published (includes notifications from callbacks like `ps_before_publish`).
363
365
 
364
366
  **Note**: When any error is raised when saving user or posts, the transaction is cancelled and thus all notifications wont be delivered (customizable by `PubSubModelSync::Config.transactions_use_buffer`).
@@ -374,14 +376,14 @@ Any notification before delivering is transformed as a Payload for a better port
374
376
  PubSubModelSync::MessagePublisher::transaction('my-custom-key') do
375
377
  user = User.create(name: 'test') # `User`:`:create` notification
376
378
  post = Post.create(title: 'sample') # `Post`:`:create` notification
377
- PubSubModelSync::MessagePublisher.publish_data(User, { ids: [user.id] }, :send_welcome) # `User`:`:send_welcome` notification
379
+ PubSubModelSync::Payload.new({ ids: [user.id] }, { klass: 'User', action: :send_welcome, mode: :klass }).publish! # `User`:`:send_welcome` notification
378
380
  end
379
381
  ```
380
382
  All notifications uses `ordering_key: 'my-custom-key'` and will be processed in the same order they were published.
381
383
 
382
384
  ## **Testing with RSpec**
383
385
  - Config: (spec/rails_helper.rb)
384
- ```ruby
386
+ ```ruby
385
387
 
386
388
  # when using google service
387
389
  require 'pub_sub_model_sync/mock_google_service'
@@ -404,51 +406,119 @@ Any notification before delivering is transformed as a Payload for a better port
404
406
  allow(Kafka).to receive(:new).and_return(kafka_mock)
405
407
  end
406
408
 
407
- #
409
+ # disable all models sync by default (reduces testing time)
408
410
  config.before(:each) do
409
- # **** disable payloads generation, sync callbacks to improve tests speed
410
411
  allow(PubSubModelSync::MessagePublisher).to receive(:publish_data) # disable class level notif
411
412
  allow(PubSubModelSync::MessagePublisher).to receive(:publish_model) # disable instance level notif
412
-
413
- # **** when testing model syncs, it can be re enabled by:
414
- # before do
415
- # allow(PubSubModelSync::MessagePublisher).to receive(:publish_data).and_call_original
416
- # allow(PubSubModelSync::MessagePublisher).to receive(:publish_model).and_call_original
417
- # end
418
413
  end
419
- ```
414
+
415
+ # enable all models sync only for tests that includes 'sync: true'
416
+ config.before(:each, sync: true) do
417
+ allow(PubSubModelSync::MessagePublisher).to receive(:publish_data).and_call_original
418
+ allow(PubSubModelSync::MessagePublisher).to receive(:publish_model).and_call_original
419
+ end
420
+
421
+ # Only when using database cleaner in old versions of rspec (enables after_commit callback)
422
+ # config.before(:each, truncate: true) do
423
+ # DatabaseCleaner.strategy = :truncation
424
+ # end
425
+ ```
420
426
  - Examples:
427
+ - **Publisher**
421
428
  ```ruby
422
- # Subscriber
423
- it 'receive model notification' do
424
- data = { name: 'name', id: 999 }
429
+ # Do not forget to include 'sync: true' to enable publishing pubsub notifications
430
+ describe 'When publishing sync', truncate: true, sync: true do
431
+ it 'publishes user notification when created' do
432
+ expect_publish_notification(:create, klass: 'User')
433
+ create(:user)
434
+ end
435
+
436
+ it 'publishes user notification with all defined data' do
437
+ user = build(:user)
438
+ data = PubSubModelSync::PayloadBuilder.parse_mapping_for(user, %i[id name:full_name email])
439
+ data[:id] = be_a(Integer)
440
+ expect_publish_notification(:create, klass: 'User', data: data)
441
+ user.save!
442
+ end
443
+
444
+ it 'publishes user notification when created' do
445
+ email = 'Newemail@gmail.com'
446
+ user = create(:user)
447
+ expect_publish_notification(:update, klass: 'User', data: { id: user.id, email: email })
448
+ user.update!(email: email)
449
+ end
450
+
451
+ it 'publishes user notification when created' do
452
+ user = create(:user)
453
+ expect_publish_notification(:destroy, klass: 'User', data: { id: user.id })
454
+ user.destroy!
455
+ end
456
+
457
+ private
458
+
459
+ # @param action (Symbol)
460
+ # @param klass (String, default described_class name)
461
+ # @param data (Hash, optional) notification data
462
+ # @param info (Hash, optional) notification info
463
+ # @param headers (Hash, optional) notification headers
464
+ def expect_publish_notification(action, klass: described_class.to_s, data: {}, info: {}, headers: {})
465
+ publisher = PubSubModelSync::MessagePublisher
466
+ exp_data = have_attributes(data: hash_including(data),
467
+ info: hash_including(info.merge(klass: klass, action: action)),
468
+ headers: hash_including(headers))
469
+ allow(publisher).to receive(:publish!).and_call_original
470
+ expect(publisher).to receive(:publish!).with(exp_data)
471
+ end
472
+ end
473
+ ```
474
+ - **Subscriber**
475
+ ```ruby
476
+
477
+ describe 'when syncing data from other apps' do
478
+ it 'creates user when received :create notification' do
479
+ user = build(:user)
480
+ data = user.as_json(only: %i[name email]).merge(id: 999)
425
481
  payload = PubSubModelSync::Payload.new(data, { klass: 'User', action: :create })
482
+ expect { payload.process! }.to change(described_class, :count)
483
+ end
484
+
485
+ it 'updates user when received :update notification' do
486
+ user = create(:user)
487
+ name = 'new name'
488
+ data = user.as_json(only: %i[id email]).merge(name: name)
489
+ payload = PubSubModelSync::Payload.new(data, { klass: 'User', action: :update })
490
+ payload.process!
491
+ expect(user.reload.name).to eq(name)
492
+ end
493
+
494
+ it 'destroys user when received :destroy notification' do
495
+ user = create(:user)
496
+ data = user.as_json(only: %i[id])
497
+ payload = PubSubModelSync::Payload.new(data, { klass: 'User', action: :destroy })
498
+ payload.process!
499
+ expect { user.reload }.to raise_error(ActiveRecord::RecordNotFound)
500
+ end
501
+
502
+
503
+ it 'receive custom model notification' do
504
+ user = create(:user)
505
+ data = { id: user.id, custom_data: {} }
506
+ custom_action = :say_hello
507
+ expect_any_instance_of(User).to receive(custom_action).with(data)
508
+ payload = PubSubModelSync::Payload.new(data, { klass: 'User', action: custom_action })
426
509
  payload.process!
427
- expect(User.where(id: data[:id])).to be_any
428
510
  end
429
511
 
430
512
  it 'receive class notification' do
431
513
  data = { msg: 'hello' }
432
514
  action = :greeting
515
+ expect(User).to receive(action).with(data)
516
+ # Do not forget to include `mode: :klass` for class notifications
433
517
  payload = PubSubModelSync::Payload.new(data, { klass: 'User', action: action, mode: :klass })
434
518
  payload.process!
435
- expect(User).to receive(action)
436
519
  end
437
-
438
- # Publisher
439
- it 'publish model notification' do
440
- publisher = PubSubModelSync::MessagePublisher
441
- user = User.create(name: 'name', email: 'email')
442
- expect(publisher).to receive(:publish_model).with(user, :create, anything)
443
- end
444
-
445
- it 'publish class notification' do
446
- publisher = PubSubModelSync::MessagePublisher
447
- user = User.create(name: 'name', email: 'email')
448
- user.ps_class_publish({msg: 'hello'}, action: :greeting)
449
- expect(publisher).to receive(:publish_data).with('User', data, :greeting)
450
- end
451
- ```
520
+ end
521
+ ```
452
522
 
453
523
  ## **Extra configurations**
454
524
  ```ruby
@@ -481,7 +551,7 @@ config.debug = true
481
551
  (Proc) => called when failed publishing a message (delayed_job or similar can be used for retrying)
482
552
  - ```.transactions_max_buffer = 100``` (Integer) Once this quantity of notifications is reached, then all notifications will immediately be delivered.
483
553
  Note: There is no way to rollback delivered notifications if current transaction fails
484
- - ```.enable_rails4_before_commit = true``` (true*|false) When false will disable rails 4 hack compatibility and then CRUD notifications will be prepared using `after_commit` callback instead of `before_commit` which will not rollback sql transactions if fails.
554
+ - ```.enable_rails4_before_commit = true``` (true*|false) When false will disable rails 4 hack compatibility and then CRUD notifications will be prepared using `after_commit` callback instead of `before_commit` (used in `ps_after_action(...)`) which will not rollback sql transactions if failed when publishing pubsub notification.
485
555
 
486
556
  ## **TODO**
487
557
  - Auto publish update only if payload has changed (see ways to compare previous payload vs new payload)
@@ -490,7 +560,6 @@ config.debug = true
490
560
  - Add DB table to use as a shield to prevent publishing similar notifications and publish partial notifications (similar idea when processing notif)
491
561
  - Last notification is not being delivered immediately in google pubsub (maybe force with timeout 10secs and service.deliver_messages)
492
562
  - Update folder structure
493
- - Support for blocks in ps_publish and ps_subscribe
494
563
  - Services support to deliver multiple payloads from transactions
495
564
 
496
565
  ## **Q&A**