stapfen 2.2.1-java → 3.0.0-java

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: beee0b9ecf2b0bf8909a85b028254ebd1c3cb2e7
4
- data.tar.gz: 93521fe82ee23ca8fa9760521032db5f483a1012
3
+ metadata.gz: a15bccf07520be4677e0bc9522dc5ba3b89d1d4c
4
+ data.tar.gz: 0250d8554c15615b054ca7fa0e7cdba6a33ca298
5
5
  SHA512:
6
- metadata.gz: b12dfa7d9ff8417e02fe3dc7ca780dc256ff9dd832281178510e1d72d2a54679ab34db983491df3647458576d029b81c29a9b004cc18c1cc01b2d1974de0dab3
7
- data.tar.gz: b030c53f61d78525cddad2067b6fea5fcf2d389ea783c7f2d2ae76e3bb005fe26a668dfbece734886172f188a1a15b331a87778a7ed046d2655c44dd1fd900f8
6
+ metadata.gz: 79d452fea3dff4f7e1334b9ee833de9170e05e37718e6e22f8fe9db5b0a849be8d0ac2bdab31c06831b74566ee76297d4855af02a57d9b1fd177e6ed2613c80f
7
+ data.tar.gz: 609308d82ee199756efce1a9f7dc675ed8f8875101ee2c33b2f5297eb3d3cc615669b516f7a853d679d28b3ee2aebf82d6918941e2cbebfb5868dcdfcecdd35c
data/README.md CHANGED
@@ -20,10 +20,11 @@ Consider the following `myworker.rb` file:
20
20
 
21
21
  ```ruby
22
22
  class MyWorker < Stapfen::Worker
23
- use_stomp!
24
-
25
- configure do
26
- {
23
+ configure do |worker|
24
+ # You can also specify your own logger, but this is the default...
25
+ worker.logger = Logger.new(STDOUT)
26
+ worker.protocol = STOMP
27
+ worker.client_options = {
27
28
  :hosts => [
28
29
  {
29
30
  :host => 'localhost',
@@ -32,18 +33,14 @@ class MyWorker < Stapfen::Worker
32
33
  :passcode => 'guest',
33
34
  :ssl => false
34
35
  }
35
- ]
36
+ ],
37
+ :topic => 'thequeue',
38
+ :dead_letter_queue => '/queue/dlq',
39
+ :max_redeliveries => 0
36
40
  }
37
41
  end
38
42
 
39
- # [Optional] Set up a logger for each worker instance
40
- log do
41
- Logger.new(STDOUT)
42
- end
43
-
44
- consume 'thequeue', :dead_letter_queue => '/queue/dlq',
45
- :max_redeliveries => 0 do |message|
46
-
43
+ consume do |message|
47
44
  data = expensive_computation(message.body)
48
45
  # Save my data, or do something worker-specific with it
49
46
  persist(data)
@@ -57,13 +54,10 @@ end
57
54
  MyWorker.run!
58
55
  ```
59
56
 
57
+ When using the STOMP protocol, `worker.client_options` can be set with any of the attributes described in a `Stomp::Client` [connection
58
+ hash](https://github.com/stompgem/stomp#hash-login-example-usage-this-is-the-recommended-login-technique) as well as any `subscription` options.
60
59
 
61
- When using the STOMP protocol, the value returned from the `configure` block is expected to be a valid
62
- `Stomp::Client` [connection
63
- hash](https://github.com/stompgem/stomp#hash-login-example-usage-this-is-the-recommended-login-technique).
64
-
65
- In the case of the JMS protocol, the value returned from the `configure` block
66
- is expected to be a valid [configuration
60
+ When using the JMS protocol, `worker.client_options` can be set with any of the attributes described in [configuration
67
61
  hash](https://github.com/reidmorrison/jruby-jms#consumer) for the
68
62
  [jruby-jms](https://github.com/reidmorrison/jruby-jms) gem.
69
63
 
@@ -76,30 +70,46 @@ require 'stapfen'
76
70
  require 'stapfen/worker'
77
71
 
78
72
  class MyWorker < Stapfen::Worker
79
- use_kafka!
80
-
81
- configure do
82
- {
83
- :topic => 'test', # not required
73
+ configure do |worker|
74
+ # You can also specify your own logger, but this is the default...
75
+ worker.logger = Logger.new(STDOUT)
76
+ worker.protocol = KAFKA
77
+ worker.client_options = {
78
+ :topic => 'test',
84
79
  :groupId => 'groupId',
85
80
  :zookeepers => 'localhost:2181' # comma separated string of zookeepers
86
81
  }
87
82
  end
88
83
 
89
- # /topic/test - topic says its a topic, test is the actual topic name
90
- consume '/topic/test' do |message|
84
+ consume do |message|
91
85
  puts "Recv: #{message.body}"
92
86
  end
93
87
  end
94
88
 
95
89
  MyWorker.run!
96
90
  ```
91
+
92
+ ##### Notes
93
+ * Testing with Kafka
94
+ * Start Staphen worker first
95
+ * Using producer included with kafka
96
+ * Produce some messages
97
+ * ```echo foobar | bin/kafka-console-producer.sh --broker-list <brokers> --topic <topic>```
98
+ * Worker should be able to read the message
99
+ * using the same groupId a consumer will start reading from the last offset that was read by a consumer from the same group
100
+ * For example, Given 2 consumers belong to the same groupId
101
+ * Consumer1 reads a few messages and dies
102
+ * A producer produces 5 messages
103
+ * Consumer2 starts up and will receive the 5 messages produced because it started at the last offset of Consumer1
104
+
97
105
  ---
98
106
 
99
107
  It is also important to note that the `consume` block will be invoked inside an
100
108
  **instance** of `MyWorker` and will execute inside its own `Thread`, so take
101
109
  care when accessing other shared resources.
102
110
 
111
+ Also note you'll need to include the zk gem manually.
112
+
103
113
  ### Fallback and dead-letter-queue support
104
114
 
105
115
  The consume block accepts the usual subscriptions headers, as well as two
@@ -109,7 +119,6 @@ the block returns `false`; after `:max_redeliveries`, it will send the message
109
119
  to `:dead_letter_queue`. `consume` blocks without these headers will fail
110
120
  silently rather than unreceive.
111
121
 
112
-
113
122
  ## Installation
114
123
 
115
124
  Add this line to your application's Gemfile:
@@ -128,6 +137,7 @@ Or install it yourself as:
128
137
 
129
138
  Download this from jar from Maven Central
130
139
  * [activemq-all-5.8.0.jar](http://search.maven.org/#artifactdetails%7Corg.apache.activemq%7Cactivemq-all%7C5.8.0%7Cjar)
140
+ * `wget -O activemq-all-5.8.0.jar http://search.maven.org/remotecontent?filepath=org/apache/activemq/activemq-all/5.8.0/activemq-all-5.8.0.jar`
131
141
  * Put it in gem root
132
142
  * ```rake spec```
133
143
 
data/lib/stapfen.rb CHANGED
@@ -16,10 +16,22 @@ require 'stapfen/client'
16
16
  require 'stapfen/worker'
17
17
 
18
18
  module Stapfen
19
- class ConfigurationError < StandardError
19
+ class ConfigurationError < StandardError; end
20
+ class ConsumeError < StandardError; end
21
+ class InvalidMessageError < StandardError; end
22
+
23
+ def self.logger=(instance)
24
+ @logger = instance
20
25
  end
21
- class ConsumeError < StandardError
26
+
27
+ def self.logger
28
+ @logger ||= default_logger
22
29
  end
23
- class InvalidMessageError < StandardError
30
+
31
+ private
32
+
33
+ def self.default_logger
34
+ require 'logger'
35
+ Logger.new(STDOUT)
24
36
  end
25
37
  end
@@ -20,6 +20,7 @@ module Stapfen
20
20
  # @option configuration [String] :topic The kafka topic
21
21
  # @option configuration [String] :groupId The kafka groupId
22
22
  # @option configuration [String] :zookeepers Comma separated list of zookeepers
23
+ # @option configuration [Hash] :consumer_opts Options for Hermann consumer
23
24
  #
24
25
  # @raises [ConfigurationError] if required configs are not present
25
26
  def initialize(configuration)
@@ -28,8 +29,9 @@ module Stapfen
28
29
  @topic = @config[:topic]
29
30
  @groupId = @config[:groupId]
30
31
  @zookeepers = @config[:zookeepers]
32
+ opts = @config[:consumer_opts]
31
33
  raise ConfigurationError unless @groupId && @zookeepers
32
- @connection = Hermann::Consumer.new(@topic, @groupId, @zookeepers)
34
+ @connection = Hermann::Consumer.new(@topic, @groupId, @zookeepers, opts)
33
35
  end
34
36
 
35
37
  # This method is not implemenented
@@ -42,6 +44,14 @@ module Stapfen
42
44
  false
43
45
  end
44
46
 
47
+ # API compatibilty method, doesn't actually indicate that the connection
48
+ # is closed. Will only return true if no connection currently exists
49
+ #
50
+ # @return [Boolean]
51
+ def closed?
52
+ return connection.nil?
53
+ end
54
+
45
55
  # Closes the consumer threads created by kafka.
46
56
  #
47
57
  # @return [Boolean] True/false depending on whether we actually closed
@@ -1,3 +1,3 @@
1
1
  module Stapfen
2
- VERSION = '2.2.1'
2
+ VERSION = '3.0.0'
3
3
  end
@@ -1,58 +1,145 @@
1
- require 'stomp'
2
- require 'stapfen/logger'
3
1
  require 'stapfen/destination'
4
2
  require 'stapfen/message'
5
3
 
6
4
  module Stapfen
7
5
  class Worker
8
- include Stapfen::Logger
6
+ KAFKA = :kafka.freeze
7
+ STOMP = :stomp.freeze
8
+ JMS = :jms.freeze
9
9
 
10
- # Class variables!
11
- @@signals_handled = false
12
- @@workers = []
10
+ attr_accessor :client_options, :protocol, :logger, :stapfen_client
13
11
 
14
12
  class << self
15
- attr_accessor :configuration, :consumers, :logger, :destructor
16
- end
13
+ attr_accessor :instance_configuration, :consumers, :destructor
14
+
15
+ def configure(&configuration_block)
16
+ unless block_given?
17
+ raise Stapfen::ConfigurationError, "Method `configure` requires a block"
18
+ end
19
+ self.instance_configuration = configuration_block
20
+ end
21
+
22
+ # Instantiate a new +Worker+ instance and run it
23
+ def run!
24
+ worker = self.new
25
+
26
+ @@workers << worker
27
+
28
+ handle_signals
29
+
30
+ worker.run
31
+ end
17
32
 
18
- # Instantiate a new +Worker+ instance and run it
19
- def self.run!
20
- worker = self.new
33
+ # Main message consumption block
34
+ def consume(config_overrides={}, &consume_block)
35
+ unless block_given?
36
+ raise Stapfen::ConsumeError, "Method `consume` requires a block"
37
+ end
38
+ @consumers ||= []
39
+ @consumers << [config_overrides, consume_block]
40
+ end
21
41
 
22
- @@workers << worker
42
+ # Optional method, specifes a block to execute when the worker is shutting
43
+ # down.
44
+ def shutdown(&block)
45
+ @destructor = block
46
+ end
47
+
48
+ # Return all the currently running Stapfen::Worker instances in this
49
+ # process
50
+ def workers
51
+ @@workers
52
+ end
23
53
 
24
- handle_signals
54
+ # Invoke +exit_cleanly+ on each of the registered Worker instances that
55
+ # this class is keeping track of
56
+ #
57
+ # @return [Boolean] Whether or not we've exited/terminated cleanly
58
+ def exit_cleanly
59
+ return false if workers.empty?
60
+
61
+ cleanly = true
62
+ workers.each do |w|
63
+ begin
64
+ w.exit_cleanly
65
+ rescue StandardError => ex
66
+ $stderr.write("Failure while exiting cleanly #{ex.inspect}\n#{ex.backtrace}")
67
+ cleanly = false
68
+ end
69
+ end
70
+
71
+ if RUBY_PLATFORM == 'java'
72
+ Stapfen.logger.info 'Telling the JVM to exit cleanly'
73
+ Java::JavaLang::System.exit(0)
74
+ end
75
+
76
+ return cleanly
77
+ end
78
+
79
+ # Utility method to set up the proper worker signal handlers
80
+ def handle_signals
81
+ return if @@signals_handled
82
+
83
+ Signal.trap(:INT) do
84
+ self.exit_cleanly
85
+ exit!
86
+ end
87
+
88
+ Signal.trap(:TERM) do
89
+ self.exit_cleanly
90
+ end
91
+
92
+ @@signals_handled = true
93
+ end
94
+
95
+ # Class variables are put in this method to allow for "reset" style
96
+ # functionality if needed. Useful for testing (see worker_spec.rb).
97
+ def set_class_variable_defaults
98
+ @@signals_handled = false
99
+ @@workers = []
100
+ end
25
101
 
26
- worker.run
27
102
  end
28
103
 
29
- # Expects a block to be passed which will yield the appropriate
30
- # configuration for the Stomp gem. Whatever the block yields will be passed
31
- # directly into the {{Stomp::Client#new}} method
32
- def self.configure(&block)
33
- unless block_given?
34
- raise Stapfen::ConfigurationError
104
+ set_class_variable_defaults
105
+
106
+ ############################################################################
107
+ # Instance Methods
108
+ ############################################################################
109
+
110
+ def initialize
111
+ instance_configuration = self.class.instance_configuration
112
+ if instance_configuration
113
+ self.configure &instance_configuration
35
114
  end
36
- @configuration = block
115
+ self.client_options ||= {}
116
+ end
117
+
118
+ def configure(&configuration_block)
119
+ self.instance_eval &configuration_block
120
+ end
121
+
122
+ def logger
123
+ @logger ||= Stapfen.logger
37
124
  end
38
125
 
39
126
  # Force the worker to use STOMP as the messaging protocol (default)
40
127
  #
41
128
  # @return [Boolean]
42
- def self.use_stomp!
129
+ def use_stomp!
43
130
  begin
44
131
  require 'stomp'
45
132
  rescue LoadError
46
- puts "You need the `stomp` gem to be installed to use stomp!"
133
+ Stapfen.logger.info 'You need the `stomp` gem to be installed to use stomp!'
47
134
  raise
48
135
  end
49
136
 
50
- @protocol = 'stomp'
137
+ @protocol = STOMP
51
138
  return true
52
139
  end
53
140
 
54
- def self.stomp?
55
- @protocol.nil? || @protocol == 'stomp'
141
+ def stomp?
142
+ @protocol.nil? || @protocol == STOMP
56
143
  end
57
144
 
58
145
  # Force the worker to use JMS as the messaging protocol.
@@ -60,25 +147,25 @@ module Stapfen
60
147
  # *Note:* Only works under JRuby
61
148
  #
62
149
  # @return [Boolean]
63
- def self.use_jms!
150
+ def use_jms!
64
151
  unless RUBY_PLATFORM == 'java'
65
- raise Stapfen::ConfigurationError, "You cannot use JMS unless you're running under JRuby!"
152
+ raise Stapfen::ConfigurationError, 'You cannot use JMS unless you are running under JRuby!'
66
153
  end
67
154
 
68
155
  begin
69
156
  require 'java'
70
157
  require 'jms'
71
158
  rescue LoadError
72
- puts "You need the `jms` gem to be installed to use JMS!"
159
+ Stapfen.logger.info 'You need the `jms` gem to be installed to use JMS!'
73
160
  raise
74
161
  end
75
162
 
76
- @protocol = 'jms'
163
+ @protocol = JMS
77
164
  return true
78
165
  end
79
166
 
80
- def self.jms?
81
- @protocol == 'jms'
167
+ def jms?
168
+ @protocol == JMS
82
169
  end
83
170
 
84
171
  # Force the worker to use Kafka as the messaging protocol.
@@ -86,158 +173,81 @@ module Stapfen
86
173
  # *Note:* Only works under JRuby
87
174
  #
88
175
  # @return [Boolean]
89
- def self.use_kafka!
176
+ def use_kafka!
90
177
  unless RUBY_PLATFORM == 'java'
91
- raise Stapfen::ConfigurationError, "You cannot use Kafka unless you're running under JRuby!"
178
+ raise Stapfen::ConfigurationError, 'You cannot use Kafka unless you are running under JRuby!'
92
179
  end
93
180
 
94
181
  begin
95
182
  require 'java'
96
183
  require 'hermann'
97
184
  rescue LoadError
98
- puts "You need the `hermann` gem to be installed to use Kafka!"
185
+ Stapfen.logger.info 'You need the `hermann` gem to be installed to use Kafka!'
99
186
  raise
100
187
  end
101
188
 
102
- @protocol = 'kafka'
189
+ @protocol = KAFKA
103
190
  return true
104
191
  end
105
192
 
106
- def self.kafka?
107
- @protocol == 'kafka'
108
- end
109
-
110
- # Optional method, should be passed a block which will yield a {{Logger}}
111
- # instance for the Stapfen worker to use
112
- def self.log(&block)
113
- @logger = block
114
- end
115
-
116
- # Main message consumption block
117
- def self.consume(queue_name, headers={}, &block)
118
- unless block_given?
119
- raise Stapfen::ConsumeError, "Cannot consume #{queue_name} without a block!"
120
- end
121
- @consumers ||= []
122
- @consumers << [queue_name, headers, block]
123
- end
124
-
125
- # Optional method, specifes a block to execute when the worker is shutting
126
- # down.
127
- def self.shutdown(&block)
128
- @destructor = block
129
- end
130
-
131
- # Return all the currently running Stapfen::Worker instances in this
132
- # process
133
- def self.workers
134
- @@workers
193
+ def kafka?
194
+ @protocol == KAFKA
135
195
  end
136
196
 
137
- # Invoke +exit_cleanly+ on each of the registered Worker instances that
138
- # this class is keeping track of
139
- #
140
- # @return [Boolean] Whether or not we've exited/terminated cleanly
141
- def self.exit_cleanly
142
- return false if workers.empty?
143
-
144
- cleanly = true
145
- workers.each do |w|
146
- begin
147
- w.exit_cleanly
148
- rescue StandardError => ex
149
- $stderr.write("Failure while exiting cleanly #{ex.inspect}\n#{ex.backtrace}")
150
- cleanly = false
151
- end
152
- end
153
-
154
- if RUBY_PLATFORM == 'java'
155
- info "Telling the JVM to exit cleanly"
156
- Java::JavaLang::System.exit(0)
157
- end
158
-
159
- return cleanly
160
- end
161
-
162
- # Utility method to set up the proper worker signal handlers
163
- def self.handle_signals
164
- return if @@signals_handled
165
-
166
- Signal.trap(:INT) do
167
- self.exit_cleanly
168
- exit!
169
- end
170
-
171
- Signal.trap(:TERM) do
172
- self.exit_cleanly
173
- end
174
-
175
- @@signals_handled = true
176
- end
177
-
178
-
179
-
180
- ############################################################################
181
- # Instance Methods
182
- ############################################################################
183
-
184
- attr_accessor :client
185
-
186
197
  def run
187
- if self.class.stomp?
198
+ if stomp?
188
199
  require 'stapfen/client/stomp'
189
- @client = Stapfen::Client::Stomp.new(self.class.configuration.call)
190
- elsif self.class.jms?
200
+ stapfen_client = Stapfen::Client::Stomp.new(client_options)
201
+ elsif jms?
191
202
  require 'stapfen/client/jms'
192
- @client = Stapfen::Client::JMS.new(self.class.configuration.call)
193
- elsif self.class.kafka?
203
+ stapfen_client = Stapfen::Client::JMS.new(client_options)
204
+ elsif kafka?
194
205
  require 'stapfen/client/kafka'
195
- @client = Stapfen::Client::Kafka.new(self.class.configuration.call)
206
+ stapfen_client = Stapfen::Client::Kafka.new(client_options)
196
207
  end
197
208
 
198
- debug("Running with #{@client} inside of Thread:#{Thread.current.inspect}")
209
+ logger.info("Running with #{stapfen_client} inside of Thread:#{Thread.current.inspect}")
199
210
 
200
- @client.connect
211
+ stapfen_client.connect
201
212
 
202
- self.class.consumers.each do |name, headers, block|
203
- unreceive_headers = {}
204
- [:max_redeliveries, :dead_letter_queue].each do |sym|
205
- unreceive_headers[sym] = headers[sym] if headers.has_key? sym
206
- end
213
+ self.class.consumers.each do |config_overrides, block|
214
+ consumer_config = client_options.merge(config_overrides)
215
+ consumer_topic = consumer_config[:topic]
216
+ consumer_can_unreceive = !(consumer_config.keys & [:max_redeliveries, :dead_letter_queue]).empty?
207
217
 
208
218
  # We're taking each block and turning it into a method so that we can
209
219
  # use the instance scope instead of the blocks originally bound scope
210
220
  # which would be at a class level
211
- method_name = name.gsub(/[.|\-]/, '_').to_sym
212
- self.class.send(:define_method, method_name, &block)
221
+ methodized_topic = consumer_topic.gsub(/[.|\-]/, '_').to_sym
222
+ self.class.send(:define_method, methodized_topic, &block)
213
223
 
214
- client.subscribe(name, headers) do |m|
215
- message = nil
216
- if self.class.stomp?
217
- message = Stapfen::Message.from_stomp(m)
224
+ stapfen_client.subscribe(consumer_topic, consumer_config) do |message_entity|
225
+ stapfen_message = nil
226
+ if stomp?
227
+ stapfen_message = Stapfen::Message.from_stomp(message_entity)
218
228
  end
219
229
 
220
- if self.class.jms?
221
- message = Stapfen::Message.from_jms(m)
230
+ if jms?
231
+ stapfen_message = Stapfen::Message.from_jms(message_entity)
222
232
  end
223
233
 
224
- if self.class.kafka?
225
- message = Stapfen::Message.from_kafka(m)
234
+ if kafka?
235
+ stapfen_message = Stapfen::Message.from_kafka(message_entity)
226
236
  end
227
237
 
228
- success = self.send(method_name, message)
238
+ success = self.send(methodized_topic, stapfen_message)
229
239
 
230
240
  unless success
231
- if client.can_unreceive? && !unreceive_headers.empty?
232
- client.unreceive(m, unreceive_headers)
241
+ if stapfen_client.can_unreceive? && consumer_can_unreceive
242
+ stapfen_client.unreceive(message_entity, consumer_config)
233
243
  end
234
244
  end
235
245
  end
236
246
  end
237
247
 
238
248
  begin
239
- client.runloop
240
- warn("Exiting the runloop for #{self}")
249
+ stapfen_client.runloop
250
+ logger.info("Exiting the runloop for #{self}")
241
251
  rescue Interrupt
242
252
  exit_cleanly
243
253
  end
@@ -246,19 +256,19 @@ module Stapfen
246
256
  # Invokes the shutdown block if it has been created, and closes the
247
257
  # {{Stomp::Client}} connection unless it has already been shut down
248
258
  def exit_cleanly
249
- info("#{self} exiting ")
259
+ logger.info("#{self} exiting ")
250
260
  self.class.destructor.call if self.class.destructor
251
261
 
252
- info "Killing client"
262
+ logger.info 'Killing client'
253
263
  begin
254
264
  # Only close the client if we have one sitting around
255
- if client
256
- unless client.closed?
257
- client.close
265
+ if stapfen_client
266
+ unless stapfen_client.closed?
267
+ stapfen_client.close
258
268
  end
259
269
  end
260
270
  rescue StandardError => exc
261
- error "Exception received while trying to close client! #{exc.inspect}"
271
+ logger.error "Exception received while trying to close client! #{exc.inspect}"
262
272
  end
263
273
  end
264
274
  end