pheromone 0.1.2 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 72824e89ef3425da33b45c276319f332728461b7
4
- data.tar.gz: 5b5bba2271b1fbc069d06615fec68d66b6279cf2
3
+ metadata.gz: 2b05a50cd3b7e0e2ae796255cd635d6bca7f8d15
4
+ data.tar.gz: da4ec3b6dafb9146438c6886c969731dddf0c61e
5
5
  SHA512:
6
- metadata.gz: bf9d4a27bb39fff06ee9fb98c0b501f3c0815330a73abfaf60d1f267588e523d9d550a5d645dd9d9f20a764f63375aba8a3a196aab1b39dc4be6907216f09061
7
- data.tar.gz: 0c39091b06b7533f7c9989d9116f314c1badb813096e1e5ece4d0a675536b0eaba324e2f38584736bab54cc04bf8bd1f8543a8535364433889c71aae65d4af88
6
+ metadata.gz: f448a3bbcabb7f55143c77cc34dd2b6e6f3cc00ad5f1357ed621385ecd469b4d95ce7258b1ef1d771ca452cd74d2897cca6440a8f947b636ae03d5bb188a6858
7
+ data.tar.gz: 85ddb8a2a665d8c0d8c0011bf3ffa68ae06b6715b816cecead71bb7a90d97a2fa26cdc2d4239ca1f54c1f90515ad615718a617a1307a8b9e402f8ab50b749025
data/.gitignore ADDED
@@ -0,0 +1,10 @@
1
+ /.bundle/
2
+ /.yardoc
3
+ /Gemfile.lock
4
+ /_yardoc/
5
+ /coverage/
6
+ /doc/
7
+ /pkg/
8
+ /spec/reports/
9
+ /tmp/
10
+ .idea/
data/.rspec ADDED
@@ -0,0 +1,2 @@
1
+ --color
2
+ --format documentation
data/CONTRIBUTIONS.rst ADDED
@@ -0,0 +1,5 @@
1
+ Contributions
2
+ =============
3
+
4
+ * `Ali Saifee <https://github.com/alisaifee>`_
5
+ * `Junkai <https://github.com/junkaiii>`_
data/Gemfile ADDED
@@ -0,0 +1,6 @@
1
+ # frozen_string_literal: true
2
+ source 'https://rubygems.org'
3
+
4
+ # Specify your gem's dependencies in pheromone.gemspec
5
+ gemspec
6
+
data/LICENSE.txt ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2017 Ankita Gupta
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
data/README.md CHANGED
@@ -20,52 +20,106 @@ Or install it yourself as:
20
20
 
21
21
  $ gem install pheromone
22
22
 
23
- ## Waterdrop Setup
23
+ ## Pheromone Setup
24
24
 
25
25
  Pheromone depends on `waterdrop` to send messages to Kafka. `waterdrop` settings can be added by following the Setup step on [waterdrop](https://github.com/karafka/waterdrop/blob/master/README.md)
26
26
 
27
- WaterDrop has following configuration options:
27
+ In order to setup `pheromone`, both `waterdrop` and `pheromone` need to be setup. Run this to generate `pheromone` configuration:
28
28
 
29
- | Option | Value type | Description |
30
- |-------------------------|---------------|----------------------------------|
31
- | send_messages | Boolean | Should we send messages to Kafka |
32
- | kafka.hosts | Array<String> | Kafka servers hosts with ports |
33
- | connection_pool_size | Integer | Kafka connection pool size |
34
- | connection_pool_timeout | Integer | Kafka connection pool timeout |
35
- | raise_on_failure | Boolean | Should we raise an exception when we cannot send message to Kafka - if false will silently ignore failures (will just ignore them) |
29
+ $ bundle exec rails generate pheromone:initializer
36
30
 
37
- To apply this configuration, you need to use a *setup* method:
31
+ This will generate the following file in `config/initializers/pheromone.rb`
38
32
 
39
- ```ruby
40
- WaterDrop.setup do |config|
41
- config.send_messages = true
42
- config.connection_pool_size = 20
43
- config.connection_pool_timeout = 1
44
- config.kafka.hosts = ['localhost:9092']
45
- config.raise_on_failure = true
33
+ ```
34
+ Pheromone.setup do |config|
35
+ #config.background_processor.name = ':resque / :sidekiq'
36
+ #config.background_processor.klass = 'BackgroundWorker'
37
+ config.timezone_format = 'UTC'
38
+ config.message_format = :json
39
+ WaterDrop.setup do |config|
40
+ config.send_messages = Rails.env.production?
41
+ config.connection_pool_size = 20
42
+ config.connection_pool_timeout = 1
43
+ config.kafka.hosts = [Rails.env.production? ? ENV['KAFKA_HOST'] : 'localhost:9092']
44
+ config.raise_on_failure = Rails.env.production?
45
+ end
46
46
  end
47
47
  ```
48
48
 
49
- This configuration can be placed in *config/initializers* and can vary based on the environment:
49
+ Edit this file to modify the default config. The following configuration options are available:
50
50
 
51
- ```ruby
52
- WaterDrop.setup do |config|
53
- config.send_messages = Rails.env.production?
54
- config.connection_pool_size = 20
55
- config.connection_pool_timeout = 1
56
- config.kafka.hosts = [Rails.env.production? ? 'prod-host:9091' : 'localhost:9092']
57
- config.raise_on_failure = Rails.env.production?
51
+
52
+ | Option | Value type | Description |
53
+ |-------------------------------|---------------|----------------------------------|
54
+ | background_processor.name | Symbol | Choose :sidekiq or :resque as the background processor only if messages need to be sent to kafka asynchronously |
55
+ | background_processor.klass | String | Background processor class name that sends messages to kafka |
56
+ | timezone_format | String | Valid timezone name for timestamps sent to kafka |
57
+ | message_format | Symbol | Only supports :json format currently |
58
+ | send_messages | Boolean | Should we send messages to Kafka |
59
+ | kafka.hosts | Array<String> | Kafka servers hosts with ports |
60
+ | connection_pool_size | Integer | Kafka connection pool size |
61
+ | connection_pool_timeout | Integer | Kafka connection pool timeout |
62
+ | raise_on_failure | Boolean | Should we raise an exception when we cannot send message to Kafka - if false will silently ignore failures (will just ignore them) |
63
+
64
+ The timezone setting will transform any timestamp attributes in the message to the specified format.
65
+
66
+ ## Usage
67
+
68
+ ### 1. Sending messages to kafka asynchronously
69
+
70
+ The underlying Kafka client used by `pheromone` is `ruby-kafka`. This client provides a normal producer that sends messages to Kafka synchronously, and an `async_producer` to send messages to Kafka asynchronously.
71
+
72
+ It is advisable to use the normal producer in production systems because async producer provides no guarantees that the messages will be delivered. To read more on this, refer the `ruby-kafka` [documentation](https://github.com/zendesk/ruby-kafka#asynchronously-producing-messages)
73
+
74
+ Even while using a synchronous producer, sometimes there might be a need to run send messages to Kafka in a background task. This is especially true for batch processing tasks that send a high message volume to Kafka. To allow for this, `pheromone` provides an `async` mode that can be specified as an option to `publish` by specifying `dispatch_method` as `:async`. By default, `dispatch_method` will be `:sync`. Specifying `:async` will still use the normal producer and NOT the async_producer.
75
+
76
+ ```
77
+ class PublishableModel < ActiveRecord::Base
78
+ include Pheromone::Publishable
79
+ publish [
80
+ {
81
+ event_types: [:create],
82
+ topic: :topic_test,
83
+ message: ->(obj) { { name: obj.name } },
84
+ dispatch_method: :async
85
+ }
86
+ ]
58
87
  end
59
88
  ```
89
+ The background_processor can be set inside `Pheromone.config.background_processor.name` as either `:resque` or `sidekiq`.
60
90
 
61
- ## Usage
91
+ #### 1.a. Using `:resque`
92
+
93
+ Create a new class and add the name under `Pheromone.config.background_processor.klass`. Implement a class method `perform(message)`, and invoke `message.send!` inside the method as shown below:
94
+
95
+ ```
96
+ class ResqueJob
97
+ @queue = :low
62
98
 
63
- ### 1. Supported events
64
- #### 1.a. To send messages for model `create` event, add the following lines to your ActiveRecord model
99
+ def self.perform(message)
100
+ message.send!
101
+ end
102
+ end
103
+ ```
104
+ #### 1.b. Using `:sidekiq`
105
+ Create a new class and add the name under `Pheromone.config.background_processor.klass`. Implement an instance method `perform_async(message)`, and invoke `message.send!` inside the method as shown below:
106
+
107
+ ```
108
+ class SidekiqJob
109
+ include Sidekiq::Worker
110
+ def perform(message)
111
+ message.send!
112
+ end
113
+ end
114
+ ```
115
+ `pheromone` will invoke the class name specified in the config with the message object. This mode can be used if you don't want to block a request that ends up sending messages to Kafka.
116
+
117
+ ### 2. Supported events
118
+ #### 2.a. To send messages for model `create` event, add the following lines to your ActiveRecord model
65
119
 
66
120
  ```
67
121
  class PublishableModel < ActiveRecord::Base
68
- include Pheromone
122
+ include Pheromone::Publishable
69
123
  publish [
70
124
  {
71
125
  event_types: [:create],
@@ -76,11 +130,11 @@ class PublishableModel < ActiveRecord::Base
76
130
  end
77
131
  ```
78
132
 
79
- #### 1.b. To send messages for model `update` event, specify `update` in the `event_types` array:
133
+ #### 2.b. To send messages for model `update` event, specify `update` in the `event_types` array:
80
134
 
81
135
  ```
82
136
  class PublishableModel < ActiveRecord::Base
83
- include Pheromone
137
+ include Pheromone::Publishable
84
138
  publish [
85
139
  {
86
140
  event_types: [:update],
@@ -93,13 +147,13 @@ end
93
147
 
94
148
  Messages can be published for multiple event types by defining `events_types: [:create, :update]`.
95
149
 
96
- ### 2. Supported message formats
150
+ ### 3. Supported message formats
97
151
 
98
- #### 2.a. Using a proc in `message`
152
+ #### 3.a. Using a proc in `message`
99
153
 
100
154
  ```
101
155
  class PublishableModel < ActiveRecord::Base
102
- include Pheromone
156
+ include Pheromone::Publishable
103
157
  publish [
104
158
  {
105
159
  event_types: [:create],
@@ -110,11 +164,11 @@ class PublishableModel < ActiveRecord::Base
110
164
  end
111
165
  ```
112
166
 
113
- #### 2.b. Using a defined function in `message`
167
+ #### 3.b. Using a defined function in `message`
114
168
 
115
169
  ```
116
170
  class PublishableModel < ActiveRecord::Base
117
- include Pheromone
171
+ include Pheromone::Publishable
118
172
  publish [
119
173
  {
120
174
  event_types: [:update],
@@ -129,11 +183,11 @@ class PublishableModel < ActiveRecord::Base
129
183
  end
130
184
  ```
131
185
 
132
- #### 2.c. Using a serializer in `message`
186
+ #### 3.c. Using a serializer in `message`
133
187
 
134
188
  ```
135
189
  class PublishableModel < ActiveRecord::Base
136
- include Pheromone
190
+ include Pheromone::Publishable
137
191
  publish [
138
192
  {
139
193
  event_types: [:create],
@@ -145,13 +199,13 @@ end
145
199
  ```
146
200
 
147
201
 
148
- ### 3. Sending messages conditionally
202
+ ### 4. Sending messages conditionally
149
203
 
150
- #### 3.a. Using a proc in `if`
204
+ #### 4.a. Using a proc in `if`
151
205
 
152
206
  ```
153
207
  class PublishableModel < ActiveRecord::Base
154
- include Pheromone
208
+ include Pheromone::Publishable
155
209
  publish [
156
210
  {
157
211
  event_types: [:update],
@@ -166,11 +220,11 @@ class PublishableModel < ActiveRecord::Base
166
220
  end
167
221
  end
168
222
  ```
169
- #### 3.b. Using a defined function in `if`
223
+ #### 4.b. Using a defined function in `if`
170
224
 
171
225
  ```
172
226
  class PublishableModel < ActiveRecord::Base
173
- include Pheromone
227
+ include Pheromone::Publishable
174
228
  publish [
175
229
  {
176
230
  event_types: [:update],
@@ -190,14 +244,14 @@ class PublishableModel < ActiveRecord::Base
190
244
  end
191
245
  ```
192
246
 
193
- ### 4. Specifying the topic
247
+ ### 5. Specifying the topic
194
248
 
195
249
  The kafka topic can be specified in the `topic` option to `publish`. To publish to `topic_test`, use the following:
196
250
 
197
251
 
198
252
  ```
199
253
  class PublishableModel < ActiveRecord::Base
200
- include Pheromone
254
+ include Pheromone::Publishable
201
255
  publish [
202
256
  {
203
257
  event_types: [:create],
@@ -208,7 +262,7 @@ class PublishableModel < ActiveRecord::Base
208
262
  end
209
263
  ```
210
264
 
211
- ### 5. Specifying producer options
265
+ ### 6. Specifying producer options
212
266
 
213
267
  [Ruby-Kafka](https://github.com/zendesk/ruby-kafka) allows sending options to change the behaviour of Kafka Producer.
214
268
 
data/Rakefile ADDED
@@ -0,0 +1,5 @@
1
+ require 'bundler/gem_tasks'
2
+ require 'rspec/core/rake_task'
3
+
4
+ RSpec::Core::RakeTask.new(:spec)
5
+ task default: :spec
@@ -0,0 +1,24 @@
1
+ require 'rails/generators'
2
+ module Pheromone
3
+ # creates job depedency initializer
4
+ class InitializerGenerator < Rails::Generators::Base
5
+ def create_initializer
6
+ create_file(
7
+ 'config/initializers/pheromone.rb',
8
+ "Pheromone.setup do |config|\n"\
9
+ " # config.background_processor.name = ':resque / :sidekiq'\n"\
10
+ " # config.background_processor.klass = 'BackgroundWorker'\n"\
11
+ " # config.timezone = 'UTC'\n"\
12
+ " config.message_format = :json\n"\
13
+ " WaterDrop.setup do |config|\n"\
14
+ " config.send_messages = Rails.env.production?\n"\
15
+ " config.connection_pool_size = 20\n"\
16
+ " config.connection_pool_timeout = 1\n"\
17
+ " config.kafka.hosts = [Rails.env.production? ? ENV['KAFKA_HOST'] : 'localhost:9092']\n"\
18
+ " config.raise_on_failure = Rails.env.production?\n"\
19
+ " end\n"\
20
+ "end"\
21
+ )
22
+ end
23
+ end
24
+ end
@@ -0,0 +1,27 @@
1
+ # frozen_string_literal: true
2
+ require 'dry-configurable'
3
+ module Pheromone
4
+ # configurator for setting up all the configurable settings for pheromone
5
+ class Config
6
+ extend Dry::Configurable
7
+ # accepts message format. Currently only accepts :json as the permitted value
8
+ setting :message_format, :json
9
+ setting :background_processor do
10
+ # accepts :sidekiq or :resque as a value
11
+ setting :name
12
+ # specify the background job handling message send to kafka
13
+ setting :klass
14
+ end
15
+ # timezone names should match a valid timezone defined here:
16
+ # http://api.rubyonrails.org/classes/ActiveSupport/TimeZone.html
17
+ # accepts a valid timezone name
18
+ setting :timezone, 'UTC'
19
+ class << self
20
+ def setup
21
+ configure do |config|
22
+ yield(config)
23
+ end
24
+ end
25
+ end
26
+ end
27
+ end
@@ -0,0 +1,9 @@
1
+ module Pheromone
2
+ module Exceptions
3
+ class InvalidPublishOptions < StandardError
4
+ def initialize(msg = 'Message format not supported')
5
+ super
6
+ end
7
+ end
8
+ end
9
+ end
@@ -0,0 +1,9 @@
1
+ module Pheromone
2
+ module Exceptions
3
+ class UnsupportedMessageFormat < StandardError
4
+ def initialize(msg = 'Message format not supported')
5
+ super
6
+ end
7
+ end
8
+ end
9
+ end
@@ -0,0 +1,56 @@
1
+ require 'pheromone'
2
+ require 'pheromone/messaging/message_formatter'
3
+ require 'waterdrop'
4
+ # This module is used for sending messages to Kafka
5
+ # Dispatch method can be :sync or :async
6
+ # When dispatch_method is async, the message object is passed to a job
7
+ # the job needs to call `send!` on the WaterDrop::Message object
8
+ module Pheromone
9
+ module Messaging
10
+ class MessageDispatcher
11
+ def initialize(message_parameters:, dispatch_method:)
12
+ @message_parameters = message_parameters
13
+ @dispatch_method = dispatch_method
14
+ end
15
+
16
+ def dispatch
17
+ if @dispatch_method == :sync
18
+ message.send!
19
+ elsif @dispatch_method == :async
20
+ send_message_asynchronously
21
+ end
22
+ end
23
+
24
+ private
25
+
26
+ # Allows sending messages via resque or sidekiq. WaterDrop::Message object
27
+ # is passed and calling `send!` on the object will trigger producing
28
+ # messages to Kafka
29
+ def send_message_asynchronously
30
+ if background_processor.name == :resque
31
+ Resque.enqueue(background_processor_klass, message)
32
+ elsif background_processor.name == :sidekiq
33
+ background_processor_klass.perform_async(message)
34
+ end
35
+ end
36
+
37
+ def message
38
+ ::WaterDrop::Message.new(
39
+ @message_parameters[:topic],
40
+ MessageFormatter.new(
41
+ @message_parameters[:message]
42
+ ).format,
43
+ @message_parameters[:producer_options] || {}
44
+ )
45
+ end
46
+
47
+ def background_processor
48
+ Pheromone.config.background_processor
49
+ end
50
+
51
+ def background_processor_klass
52
+ @klass ||= background_processor.klass.constantize
53
+ end
54
+ end
55
+ end
56
+ end
@@ -0,0 +1,47 @@
1
+ require 'pheromone/exceptions/unsupported_message_format'
2
+ module Pheromone
3
+ module Messaging
4
+ class MessageFormatter
5
+ SUPPORTED_MESSAGE_FORMATS = [:json].freeze
6
+
7
+ def initialize(message)
8
+ @message = message
9
+ end
10
+
11
+ def format
12
+ if Pheromone.config.message_format == :json
13
+ convert_to_time_format.to_json
14
+ elsif !SUPPORTED_MESSAGE_FORMATS.include?(Pheromone.config.message_format)
15
+ raise Pheromone::Exceptions::UnsupportedMessageFormat.new
16
+ end
17
+ end
18
+
19
+ private
20
+
21
+ # recursively converts time to the timezone set in configuration
22
+ def convert_to_time_format
23
+ deep_transform_values!(@message) do |value|
24
+ if value.is_a? Time
25
+ value.in_time_zone(Pheromone.config.timezone)
26
+ else
27
+ value
28
+ end
29
+ end
30
+ end
31
+
32
+ # recursively applies a block to a hash
33
+ def deep_transform_values!(object, &block)
34
+ case object
35
+ when Array
36
+ object.map! { |element| deep_transform_values!(element, &block) }
37
+ when Hash
38
+ object.each do |key, value|
39
+ object[key] = deep_transform_values!(value, &block)
40
+ end
41
+ else
42
+ yield object
43
+ end
44
+ end
45
+ end
46
+ end
47
+ end
@@ -0,0 +1,22 @@
1
+ # calls a proc or method
2
+ module Pheromone
3
+ module MethodInvoker
4
+ module InstanceMethods
5
+ # This method has the :reek:ManualDispatch smell,
6
+ # which is difficult to avoid since it handles
7
+ # either a lambda/Proc or a named method from the including
8
+ # class.
9
+ def call_proc_or_instance_method(proc_or_symbol)
10
+ return proc_or_symbol.call(self) if proc_or_symbol.respond_to?(:call)
11
+ unless respond_to? proc_or_symbol
12
+ raise "Method #{proc_or_symbol} not found for #{self.class.name}"
13
+ end
14
+ __send__(proc_or_symbol)
15
+ end
16
+ end
17
+
18
+ def self.included(base)
19
+ base.send :include, InstanceMethods
20
+ end
21
+ end
22
+ end
@@ -0,0 +1,104 @@
1
+ # Usage: For publishing messages to kafka, include this concern
2
+ # in the model and then add
3
+ #
4
+ # publish message_options: [
5
+ # {
6
+ # topic: :topic1,
7
+ # producer_options: {
8
+ # max_retries: 5,
9
+ # retry_backoff: 5,
10
+ # compression_codec: :snappy,
11
+ # compression_threshold: 10,
12
+ # required_acks: 1
13
+ # }
14
+ # event_types: [:create, :update],
15
+ # message: { a: 1, b: 2 }
16
+ # dispatch_method: :async
17
+ # },....
18
+ # ]
19
+ #
20
+ # Each entry in message_options will be registered as a potential
21
+ # message to be published to kafka on the after_commit hook of the
22
+ # including model. message_options can take an optional if: key which
23
+ # accepts either a callback or an instance method name - which can be
24
+ # used to decide if the message should be published or not.
25
+ #
26
+ # To control how the model is serialized before being published to kafka
27
+ # either provide a Serializer via the `serializer` key or a callback or instance
28
+ # method name via the `message` key
29
+ require 'pheromone/validators/options_validator'
30
+ require 'pheromone/exceptions/invalid_publish_options'
31
+ require 'pheromone/method_invoker'
32
+ require 'pheromone/messaging/message_dispatcher'
33
+ module Pheromone
34
+ module Publishable
35
+ # class methods for the model including Publishable
36
+ module ClassMethods
37
+ def publish(message_options)
38
+ errors = Pheromone::Validators::OptionsValidator.new(
39
+ message_options
40
+ ).validate
41
+ raise Pheromone::Exceptions::InvalidPublishOptions.new(errors) unless errors.empty?
42
+ __send__(:after_commit, proc { dispatch_messages(message_options: message_options) })
43
+ end
44
+ end
45
+
46
+ module InstanceMethods
47
+ include Pheromone::MethodInvoker
48
+
49
+ def dispatch_messages(message_options:)
50
+ message_options.each do |options|
51
+ next unless check_conditions(options)
52
+ send_message(options)
53
+ end
54
+ end
55
+
56
+ private
57
+
58
+ def check_conditions(options)
59
+ condition_callback = options[:if]
60
+ result = check_event(options)
61
+ return result unless condition_callback
62
+ result && call_proc_or_instance_method(condition_callback)
63
+ end
64
+
65
+ def check_event(options)
66
+ options[:event_types].any? { |event| event == current_event }
67
+ end
68
+
69
+ def send_message(options)
70
+ Pheromone::Messaging::MessageDispatcher.new(
71
+ message_parameters: {
72
+ topic: options[:topic],
73
+ message: message_meta_data.merge!(blob: message_blob(options)),
74
+ producer_options: options[:producer_options]
75
+ },
76
+ dispatch_method: options[:dispatch_method] || :sync
77
+ ).dispatch
78
+ end
79
+
80
+ def message_meta_data
81
+ {
82
+ event: current_event,
83
+ entity: self.class.name,
84
+ timestamp: Time.now
85
+ }
86
+ end
87
+
88
+ def current_event
89
+ id_previously_changed? ? :create : :update
90
+ end
91
+
92
+ def message_blob(options)
93
+ message = options[:message]
94
+ return call_proc_or_instance_method(message) if message
95
+ options[:serializer].new(self, options[:serializer_options] || {}).serializable_hash
96
+ end
97
+ end
98
+
99
+ def self.included(base)
100
+ base.send :include, InstanceMethods
101
+ base.send :extend, ClassMethods
102
+ end
103
+ end
104
+ end
@@ -0,0 +1,7 @@
1
+ # class ResqueJob
2
+ # @queue = :low
3
+ #
4
+ # def self.perform(message)
5
+ # message.send!
6
+ # end
7
+ # end
@@ -0,0 +1,6 @@
1
+ # class SidekiqJob
2
+ # include Sidekiq::Worker
3
+ # def perform(message)
4
+ # message.send!
5
+ # end
6
+ # end
@@ -0,0 +1,73 @@
1
+ # frozen_string_literal: true
2
+ # validate message options provided to publish method in Publishable concern
3
+ module Pheromone
4
+ module Validators
5
+ class OptionsValidator
6
+ ACCEPTED_EVENT_TYPES = %i(create update).freeze
7
+ ALLOWED_DISPATCH_METHODS = %i(sync async)
8
+
9
+ def initialize(message_options)
10
+ @errors = {}
11
+ @message_options = message_options
12
+ end
13
+
14
+ def validate
15
+ validate_message_options
16
+ return @errors if @errors.present?
17
+ validate_topic
18
+ validate_event_types
19
+ validate_message_attributes
20
+ validate_dispatch_method
21
+ @errors
22
+ end
23
+
24
+ private
25
+
26
+ def validate_message_options
27
+ return if @message_options.is_a?(Array)
28
+ add_error_message(:message_options, 'Message options should be an array')
29
+ end
30
+
31
+ def validate_topic
32
+ return if @message_options.all? { |options| options[:topic].present? }
33
+ add_error_message(:topic, 'Topic name missing')
34
+ end
35
+
36
+ # :reek:FeatureEnvy
37
+ def validate_event_types
38
+ return if @message_options.all? do |options|
39
+ event_types = options[:event_types]
40
+ next true unless event_types
41
+ event_types.present? &&
42
+ event_types.is_a?(Array) &&
43
+ (event_types - ACCEPTED_EVENT_TYPES).empty?
44
+ end
45
+
46
+ add_error_message(
47
+ :event_types,
48
+ "Event types must be a non-empty array with types #{ACCEPTED_EVENT_TYPES.join(',')}"
49
+ )
50
+ end
51
+
52
+ def validate_message_attributes
53
+ return if @message_options.all? do |options|
54
+ options[:serializer].present? || options[:message].present?
55
+ end
56
+
57
+ add_error_message(:message_attributes, 'Either serializer or message should be specified')
58
+ end
59
+
60
+ def validate_dispatch_method
61
+ dispatch_methods = @message_options.map{ |options| options[:dispatch_method] }
62
+ return if dispatch_methods.all? do |method|
63
+ method.nil? || ALLOWED_DISPATCH_METHODS.include?(method)
64
+ end
65
+ add_error_message(:dispatch_method, 'Invalid dispatch method')
66
+ end
67
+
68
+ def add_error_message(key, value)
69
+ @errors.merge!(key => value)
70
+ end
71
+ end
72
+ end
73
+ end
@@ -1,4 +1,4 @@
1
1
  # frozen_string_literal: true
2
2
  module Pheromone
3
- VERSION = '0.1.2'.freeze
3
+ VERSION = '0.2.0'.freeze
4
4
  end
data/lib/pheromone.rb CHANGED
@@ -1,105 +1,17 @@
1
- require 'pheromone/version'
2
- require 'pheromone/options_validator'
3
- require 'active_support'
4
- require 'waterdrop'
5
- # Usage: For publishing messages to kafka, include this concern
6
- # in the model and then add
7
- #
8
- # include Publishable
9
- # publish message_options: [
10
- # {
11
- # topic: :topic1,
12
- # producer_options: {
13
- # max_retries: 5,
14
- # retry_backoff: 5,
15
- # compression_codec: :snappy,
16
- # compression_threshold: 10,
17
- # required_acks: 1
18
- # }
19
- # event_types: [:create, :update],
20
- # message: { a: 1, b: 2 }
21
- # },....
22
- # ]
23
- #
24
- # Each entry in message_options will be registered as a potential
25
- # message to be published to kafka on the after_commit hook of the
26
- # including model. message_options can take an optional if: key which
27
- # accepts either a callback or an instance method name - which can be
28
- # used to decide if the message should be published or not.
29
- #
30
- # To control how the model is serialized before being published to kafka
31
- # either provide a Serializer via the `serializer` key or a callback or instance
32
- # method name via the `message` key
33
- module Pheromone
34
- extend ActiveSupport::Concern
1
+ require 'pheromone/publishable'
2
+ require 'pheromone/config'
35
3
 
36
- # class methods for the model including Publishable
37
- module ClassMethods
38
- def publish(message_options)
39
- errors = OptionsValidator.new(message_options).validate
40
- raise "Errors: #{errors}" unless errors.empty?
41
- __send__(:after_commit, proc { dispatch_messages(message_options: message_options) })
42
- end
43
- end
44
-
45
- def dispatch_messages(message_options:)
46
- message_options.each do |options|
47
- next unless check_conditions(options)
48
- begin
49
- send_message(options)
50
- rescue => error
51
- puts error
52
- end
4
+ module Pheromone
5
+ class << self
6
+ # return config
7
+ def config
8
+ Config.config
53
9
  end
54
- end
55
-
56
- private
57
-
58
- def check_conditions(options)
59
- condition_callback = options[:if]
60
- return check_event(options) unless condition_callback
61
- call_proc_or_instance_method(condition_callback)
62
- end
63
-
64
- def check_event(options)
65
- options[:event_types].any? { |event| event == current_event }
66
- end
67
-
68
- def send_message(options)
69
- WaterDrop::Message.new(
70
- options[:topic],
71
- message_meta_data.merge!(blob: message_blob(options)).to_json,
72
- options[:producer_options] || {}
73
- ).send!
74
- end
75
-
76
- def message_meta_data
77
- {
78
- event: current_event,
79
- entity: self.class.name,
80
- timestamp: Time.now
81
- }
82
- end
83
10
 
84
- def current_event
85
- id_previously_changed? ? :create : :update
86
- end
87
-
88
- def message_blob(options)
89
- message = options[:message]
90
- return call_proc_or_instance_method(message) if message
91
- options[:serializer].new(self, options[:serializer_options] || {}).serializable_object
92
- end
93
-
94
- # This method has the :reek:ManualDispatch smell,
95
- # which is difficult to avoid since it handles
96
- # either a lambda/Proc or a named method from the including
97
- # class.
98
- def call_proc_or_instance_method(proc_or_symbol)
99
- return proc_or_symbol.call(self) if proc_or_symbol.respond_to?(:call)
100
- unless respond_to? proc_or_symbol
101
- raise "Method #{proc_or_symbol} not found for #{self.class.name}"
11
+ # Provides a block to override default config
12
+ def setup(&block)
13
+ Config.setup(&block)
102
14
  end
103
- __send__(proc_or_symbol)
104
15
  end
105
16
  end
17
+
data/pheromone.gemspec ADDED
@@ -0,0 +1,38 @@
1
+ # frozen_string_literal: true
2
+ # coding: utf-8
3
+ lib = File.expand_path('../lib', __FILE__)
4
+ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
5
+ require 'pheromone/version'
6
+
7
+ Gem::Specification.new do |spec|
8
+ spec.name = 'pheromone'
9
+ spec.version = Pheromone::VERSION
10
+ spec.authors = ['Ankita Gupta']
11
+ spec.email = ['ankitagupta12391@gmail.com']
12
+
13
+ spec.summary = 'Transmits messages to kafka from active record'
14
+ spec.description = 'Sends messages to kafka using different formats and strategies'
15
+ spec.homepage = 'https://github.com/ankitagupta12/pheromone'
16
+ spec.license = 'MIT'
17
+
18
+ spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(spec)/}) }
19
+ spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
20
+
21
+ spec.require_paths = ['lib']
22
+
23
+ spec.add_dependency 'active_model_serializers', '~> 0.9'
24
+ spec.add_dependency 'activerecord', '>= 4.2.5'
25
+ spec.add_dependency 'bundler', '>= 0'
26
+ spec.add_dependency 'dry-configurable', '~> 0.6'
27
+ spec.add_dependency 'waterdrop', '~> 0.3.2.1'
28
+
29
+ spec.add_development_dependency 'activesupport', '>= 4.2.5'
30
+ spec.add_development_dependency 'generator_spec', '~> 0.9.3'
31
+ spec.add_development_dependency 'rake', '~> 0'
32
+ spec.add_development_dependency 'resque', '~> 1.26'
33
+ spec.add_development_dependency 'rspec-rails', '~> 3.5'
34
+ spec.add_development_dependency 'sidekiq'
35
+ spec.add_development_dependency 'sqlite3'
36
+ spec.add_development_dependency 'timecop', '~> 0.8'
37
+ spec.add_development_dependency 'with_model', '~> 1.2'
38
+ end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: pheromone
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.2
4
+ version: 0.2.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Ankita Gupta
8
8
  autorequire:
9
- bindir: exe
9
+ bindir: bin
10
10
  cert_chain: []
11
- date: 2017-08-03 00:00:00.000000000 Z
11
+ date: 2017-08-25 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: active_model_serializers
@@ -16,14 +16,14 @@ dependencies:
16
16
  requirements:
17
17
  - - "~>"
18
18
  - !ruby/object:Gem::Version
19
- version: 0.9.3
19
+ version: '0.9'
20
20
  type: :runtime
21
21
  prerelease: false
22
22
  version_requirements: !ruby/object:Gem::Requirement
23
23
  requirements:
24
24
  - - "~>"
25
25
  - !ruby/object:Gem::Version
26
- version: 0.9.3
26
+ version: '0.9'
27
27
  - !ruby/object:Gem::Dependency
28
28
  name: activerecord
29
29
  requirement: !ruby/object:Gem::Requirement
@@ -39,19 +39,33 @@ dependencies:
39
39
  - !ruby/object:Gem::Version
40
40
  version: 4.2.5
41
41
  - !ruby/object:Gem::Dependency
42
- name: activesupport
42
+ name: bundler
43
43
  requirement: !ruby/object:Gem::Requirement
44
44
  requirements:
45
45
  - - ">="
46
46
  - !ruby/object:Gem::Version
47
- version: 4.2.5
47
+ version: '0'
48
48
  type: :runtime
49
49
  prerelease: false
50
50
  version_requirements: !ruby/object:Gem::Requirement
51
51
  requirements:
52
52
  - - ">="
53
53
  - !ruby/object:Gem::Version
54
- version: 4.2.5
54
+ version: '0'
55
+ - !ruby/object:Gem::Dependency
56
+ name: dry-configurable
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - "~>"
60
+ - !ruby/object:Gem::Version
61
+ version: '0.6'
62
+ type: :runtime
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - "~>"
67
+ - !ruby/object:Gem::Version
68
+ version: '0.6'
55
69
  - !ruby/object:Gem::Dependency
56
70
  name: waterdrop
57
71
  requirement: !ruby/object:Gem::Requirement
@@ -67,33 +81,33 @@ dependencies:
67
81
  - !ruby/object:Gem::Version
68
82
  version: 0.3.2.1
69
83
  - !ruby/object:Gem::Dependency
70
- name: bundler
84
+ name: activesupport
71
85
  requirement: !ruby/object:Gem::Requirement
72
86
  requirements:
73
- - - "~>"
87
+ - - ">="
74
88
  - !ruby/object:Gem::Version
75
- version: '1.12'
89
+ version: 4.2.5
76
90
  type: :development
77
91
  prerelease: false
78
92
  version_requirements: !ruby/object:Gem::Requirement
79
93
  requirements:
80
- - - "~>"
94
+ - - ">="
81
95
  - !ruby/object:Gem::Version
82
- version: '1.12'
96
+ version: 4.2.5
83
97
  - !ruby/object:Gem::Dependency
84
- name: timecop
98
+ name: generator_spec
85
99
  requirement: !ruby/object:Gem::Requirement
86
100
  requirements:
87
101
  - - "~>"
88
102
  - !ruby/object:Gem::Version
89
- version: '0.8'
103
+ version: 0.9.3
90
104
  type: :development
91
105
  prerelease: false
92
106
  version_requirements: !ruby/object:Gem::Requirement
93
107
  requirements:
94
108
  - - "~>"
95
109
  - !ruby/object:Gem::Version
96
- version: '0.8'
110
+ version: 0.9.3
97
111
  - !ruby/object:Gem::Dependency
98
112
  name: rake
99
113
  requirement: !ruby/object:Gem::Requirement
@@ -108,6 +122,20 @@ dependencies:
108
122
  - - "~>"
109
123
  - !ruby/object:Gem::Version
110
124
  version: '0'
125
+ - !ruby/object:Gem::Dependency
126
+ name: resque
127
+ requirement: !ruby/object:Gem::Requirement
128
+ requirements:
129
+ - - "~>"
130
+ - !ruby/object:Gem::Version
131
+ version: '1.26'
132
+ type: :development
133
+ prerelease: false
134
+ version_requirements: !ruby/object:Gem::Requirement
135
+ requirements:
136
+ - - "~>"
137
+ - !ruby/object:Gem::Version
138
+ version: '1.26'
111
139
  - !ruby/object:Gem::Dependency
112
140
  name: rspec-rails
113
141
  requirement: !ruby/object:Gem::Requirement
@@ -122,6 +150,20 @@ dependencies:
122
150
  - - "~>"
123
151
  - !ruby/object:Gem::Version
124
152
  version: '3.5'
153
+ - !ruby/object:Gem::Dependency
154
+ name: sidekiq
155
+ requirement: !ruby/object:Gem::Requirement
156
+ requirements:
157
+ - - ">="
158
+ - !ruby/object:Gem::Version
159
+ version: '0'
160
+ type: :development
161
+ prerelease: false
162
+ version_requirements: !ruby/object:Gem::Requirement
163
+ requirements:
164
+ - - ">="
165
+ - !ruby/object:Gem::Version
166
+ version: '0'
125
167
  - !ruby/object:Gem::Dependency
126
168
  name: sqlite3
127
169
  requirement: !ruby/object:Gem::Requirement
@@ -136,6 +178,20 @@ dependencies:
136
178
  - - ">="
137
179
  - !ruby/object:Gem::Version
138
180
  version: '0'
181
+ - !ruby/object:Gem::Dependency
182
+ name: timecop
183
+ requirement: !ruby/object:Gem::Requirement
184
+ requirements:
185
+ - - "~>"
186
+ - !ruby/object:Gem::Version
187
+ version: '0.8'
188
+ type: :development
189
+ prerelease: false
190
+ version_requirements: !ruby/object:Gem::Requirement
191
+ requirements:
192
+ - - "~>"
193
+ - !ruby/object:Gem::Version
194
+ version: '0.8'
139
195
  - !ruby/object:Gem::Dependency
140
196
  name: with_model
141
197
  requirement: !ruby/object:Gem::Requirement
@@ -153,17 +209,36 @@ dependencies:
153
209
  description: Sends messages to kafka using different formats and strategies
154
210
  email:
155
211
  - ankitagupta12391@gmail.com
156
- executables: []
212
+ executables:
213
+ - console
214
+ - setup
157
215
  extensions: []
158
216
  extra_rdoc_files: []
159
217
  files:
218
+ - ".gitignore"
219
+ - ".rspec"
160
220
  - CODE_OF_CONDUCT.md
221
+ - CONTRIBUTIONS.rst
222
+ - Gemfile
223
+ - LICENSE.txt
161
224
  - README.md
225
+ - Rakefile
162
226
  - bin/console
163
227
  - bin/setup
228
+ - lib/generators/pheromone/initializer_generator.rb
164
229
  - lib/pheromone.rb
165
- - lib/pheromone/options_validator.rb
230
+ - lib/pheromone/config.rb
231
+ - lib/pheromone/exceptions/invalid_publish_options.rb
232
+ - lib/pheromone/exceptions/unsupported_message_format.rb
233
+ - lib/pheromone/messaging/message_dispatcher.rb
234
+ - lib/pheromone/messaging/message_formatter.rb
235
+ - lib/pheromone/method_invoker.rb
236
+ - lib/pheromone/publishable.rb
237
+ - lib/pheromone/templates/resque_job.rb.example
238
+ - lib/pheromone/templates/sidekiq_job.rb.example
239
+ - lib/pheromone/validators/options_validator.rb
166
240
  - lib/pheromone/version.rb
241
+ - pheromone.gemspec
167
242
  homepage: https://github.com/ankitagupta12/pheromone
168
243
  licenses:
169
244
  - MIT
@@ -1,59 +0,0 @@
1
- # frozen_string_literal: true
2
- # validate message options provided to publish method in Publishable concern
3
- class OptionsValidator
4
- ACCEPTED_EVENT_TYPES = %i(create update).freeze
5
-
6
- def initialize(message_options)
7
- @errors = {}
8
- @message_options = message_options
9
- end
10
-
11
- def validate
12
- validate_message_options
13
- return @errors if @errors.present?
14
- validate_topic
15
- validate_event_types
16
- validate_message_attributes
17
- @errors
18
- end
19
-
20
- private
21
-
22
- def validate_message_options
23
- return if @message_options.is_a?(Array)
24
- add_error_message(:message_options, 'Message options should be an array')
25
- end
26
-
27
- def validate_topic
28
- return if @message_options.all? { |options| options[:topic].present? }
29
- add_error_message(:topic, 'Topic name missing')
30
- end
31
-
32
- # :reek:FeatureEnvy
33
- def validate_event_types
34
- return if @message_options.all? do |options|
35
- event_types = options[:event_types]
36
- next true unless event_types
37
- event_types.present? &&
38
- event_types.is_a?(Array) &&
39
- (event_types - ACCEPTED_EVENT_TYPES).empty?
40
- end
41
-
42
- add_error_message(
43
- :event_types,
44
- "Event types must be a non-empty array with types #{ACCEPTED_EVENT_TYPES.join(',')}"
45
- )
46
- end
47
-
48
- def validate_message_attributes
49
- return if @message_options.all? do |options|
50
- options[:serializer].present? || options[:message].present?
51
- end
52
-
53
- add_error_message(:message_attributes, 'Either serializer or message should be specified')
54
- end
55
-
56
- def add_error_message(key, value)
57
- @errors.merge!(key => value)
58
- end
59
- end