action_subscriber 2.5.0.pre-java → 3.0.0-java

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (41) hide show
  1. checksums.yaml +4 -4
  2. data/.travis.yml +1 -2
  3. data/README.md +15 -23
  4. data/action_subscriber.gemspec +1 -0
  5. data/lib/action_subscriber.rb +11 -23
  6. data/lib/action_subscriber/babou.rb +2 -29
  7. data/lib/action_subscriber/base.rb +0 -4
  8. data/lib/action_subscriber/bunny/subscriber.rb +18 -4
  9. data/lib/action_subscriber/configuration.rb +1 -11
  10. data/lib/action_subscriber/default_routing.rb +6 -4
  11. data/lib/action_subscriber/march_hare/subscriber.rb +20 -4
  12. data/lib/action_subscriber/message_retry.rb +1 -1
  13. data/lib/action_subscriber/middleware/env.rb +3 -1
  14. data/lib/action_subscriber/middleware/error_handler.rb +20 -4
  15. data/lib/action_subscriber/rabbit_connection.rb +24 -33
  16. data/lib/action_subscriber/route.rb +5 -1
  17. data/lib/action_subscriber/route_set.rb +13 -6
  18. data/lib/action_subscriber/router.rb +15 -3
  19. data/lib/action_subscriber/version.rb +1 -1
  20. data/spec/integration/around_filters_spec.rb +1 -1
  21. data/spec/integration/at_least_once_spec.rb +1 -1
  22. data/spec/integration/at_most_once_spec.rb +1 -1
  23. data/spec/integration/automatic_reconnect_spec.rb +3 -4
  24. data/spec/integration/basic_subscriber_spec.rb +2 -2
  25. data/spec/integration/custom_actions_spec.rb +1 -1
  26. data/spec/integration/custom_headers_spec.rb +2 -2
  27. data/spec/integration/decoding_payloads_spec.rb +2 -2
  28. data/spec/integration/manual_acknowledgement_spec.rb +1 -1
  29. data/spec/integration/multiple_connections_spec.rb +36 -0
  30. data/spec/integration/multiple_threadpools_spec.rb +3 -3
  31. data/spec/lib/action_subscriber/configuration_spec.rb +1 -5
  32. data/spec/lib/action_subscriber/middleware/error_handler_spec.rb +15 -0
  33. data/spec/spec_helper.rb +8 -4
  34. metadata +21 -16
  35. data/lib/action_subscriber/publisher.rb +0 -46
  36. data/lib/action_subscriber/publisher/async.rb +0 -31
  37. data/lib/action_subscriber/publisher/async/in_memory_adapter.rb +0 -153
  38. data/spec/integration/inferred_routes_spec.rb +0 -53
  39. data/spec/lib/action_subscriber/publisher/async/in_memory_adapter_spec.rb +0 -135
  40. data/spec/lib/action_subscriber/publisher/async_spec.rb +0 -40
  41. data/spec/lib/action_subscriber/publisher_spec.rb +0 -35
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 915c64745ca55a9a6f8fc148717856f3e89061bd
4
- data.tar.gz: 3e33152ee0f1c5116bdb6330368d01535c4c02be
3
+ metadata.gz: 9ad18409eae9a5a099a7263c0ade2678eab25342
4
+ data.tar.gz: 27072b9738bc2fe3cdd341092951ca9de3e6ac93
5
5
  SHA512:
6
- metadata.gz: 1804d484c794bb5abe9aea0ecd59956c6a3e3a8c07e96578c5706e590f09b619f333c7c3707cc9c00ddcd191770fb422da3c190aa302d13cffa786ebf11a1ca7
7
- data.tar.gz: e6cdcb2d5bfa138fa0d15501e3895add14cc3715c168688760c5ded0e8bb374affa71ca861844fcef5d05c3c2fd51df439d73638a3fb698282893dae87e17621
6
+ metadata.gz: 1419db5c28f91368548921aa60ad290544113c2580fcf5f63168cb025fad0969345ed62277c24850f47816893be95cc6542f05ab17681cc733d7d4760c4c3de6
7
+ data.tar.gz: 93858843bfd8de33cc95f4d85d3e671acd3abc010ada5b8b3ad3eb1a2d952f9e401d711492b0ae4d90cb02d41a3fcb0de2f75c5824c5ffff497daa1d69fca4b6
@@ -3,8 +3,8 @@ rvm:
3
3
  - 2.2.4
4
4
  - 2.3.0
5
5
  - jruby-9.0.4.0
6
+ - jruby-9.1.5.0
6
7
  - jruby-head
7
- - rbx-2
8
8
  services:
9
9
  - rabbitmq
10
10
  sudo: false
@@ -12,4 +12,3 @@ cache: bundler
12
12
  matrix:
13
13
  allow_failures:
14
14
  - rvm: jruby-head
15
- - rvm: rbx-2
data/README.md CHANGED
@@ -24,8 +24,6 @@ A subscriber is set up by creating a class that inherits from ActionSubscriber::
24
24
 
25
25
  ```ruby
26
26
  class UserSubscriber < ::ActionSubscriber::Base
27
- publisher :user_hq
28
-
29
27
  def created
30
28
  # do something when a user is created
31
29
  end
@@ -36,35 +34,29 @@ checkout the examples dir for more detailed examples.
36
34
 
37
35
  Usage
38
36
  -----------------
39
- ActionSubscriber is inspired by rails observers, and if you are familiar with rails
40
- observers the ActionSubscriber DSL should make you feel right at home!
41
-
42
- First, create a subscriber the inherits from ActionSubscriber::Base
43
37
 
44
- Then, when your app starts up, you will need to load your subscriber code and then do
38
+ In your application setup you will draw your subscription routes. In a rails app this is usually done in `config/initializers/action_subscriber.rb`.
45
39
 
46
40
  ```ruby
47
- ActionSubscriber.start_subscribers
48
- while true
49
- sleep 1.0
41
+ ::ActionSubscriber.draw_routes do
42
+ # you can define routes one-by-one for fine-grained controled
43
+ route UserSubscriber, :created
44
+
45
+ # or you can setup default routes for all the public methods in a subscriber
46
+ default_routes_for UserSubscriber
50
47
  end
51
48
  ```
52
49
 
53
- or
50
+ Now you can start your subscriber process with:
54
51
 
55
- ```ruby
56
- ::ActionSubscriber.start_queues
57
- while true
58
- ::ActionSubscriber.auto_pop!
59
- sleep 1.0
60
- end
52
+
53
+ ```
54
+ $ bundle exec action_subscriber start --mode=subscribe
61
55
  ```
62
56
 
63
- Any public methods on your subscriber will be registered as queues with rabbit with
64
- routing keys named intelligently.
57
+ This will start your subscribers in a mode where they connect to rabbitmq and let the broker push messages down to them.
65
58
 
66
- Once ActionSubscriber receives a message, it will call the associated method and the
67
- parameter you recieve will be a decoded message.
59
+ You can also start in `--mode=pop` where your process will poll the broker for messages.
68
60
 
69
61
  Configuration
70
62
  -----------------
@@ -80,11 +72,11 @@ In an initializer, you can set the host and the port like this :
80
72
  Other configuration options include :
81
73
 
82
74
  * config.add_decoder - add a custom decoder for a custom content type
83
- * config.allow_low_priority_methods - subscribe to queues for methods suffixed with "_low"
84
75
  * config.default_exchange - set the default exchange that your queues will use, using the default RabbitMQ exchange is not recommended
85
76
  * config.error_handler - handle error like you want to handle them!
86
77
  * config.heartbeat - number of seconds between hearbeats (default 5) [see bunny documentation for more details](http://rubybunny.info/articles/connecting.html)
87
- * config.hosts - an array of hostnames in your cluster
78
+ * config.hosts - an array of hostnames in your cluster (ie `["rabbit1.myapp.com", "rabbit2.myapp.com"]`)
79
+ * config.pop_interval - how long to wait between polling for messages in `--mode=pop`. It should be a number of milliseconds
88
80
  * config.threadpool_size - set the number of threads availiable to action_subscriber
89
81
  * config.timeout - how many seconds to allow rabbit to respond before timing out
90
82
  * config.times_to_pop - when using RabbitMQ's pull API, the number of messages we will grab each time we pool the broker
@@ -30,6 +30,7 @@ Gem::Specification.new do |spec|
30
30
  spec.add_dependency 'middleware'
31
31
  spec.add_dependency 'thor'
32
32
 
33
+ spec.add_development_dependency "active_publisher", "~> 0.1.5"
33
34
  spec.add_development_dependency "activerecord", ">= 3.2"
34
35
  spec.add_development_dependency "bundler", ">= 1.6"
35
36
  spec.add_development_dependency "pry-nav"
@@ -25,8 +25,6 @@ require "action_subscriber/subscribable"
25
25
  require "action_subscriber/bunny/subscriber"
26
26
  require "action_subscriber/march_hare/subscriber"
27
27
  require "action_subscriber/babou"
28
- require "action_subscriber/publisher"
29
- require "action_subscriber/publisher/async"
30
28
  require "action_subscriber/route"
31
29
  require "action_subscriber/route_set"
32
30
  require "action_subscriber/router"
@@ -80,19 +78,21 @@ module ActionSubscriber
80
78
  end
81
79
  end
82
80
 
83
- def self.setup_queues!
84
- route_set.setup_queues!
81
+ def self.setup_default_connection!
82
+ ::ActionSubscriber::RabbitConnection.setup_connection(:default, {})
83
+ end
84
+
85
+ def self.setup_subscriptions!
86
+ route_set.setup_subscriptions!
85
87
  end
86
88
 
87
89
  def self.start_queues
88
- ::ActionSubscriber::RabbitConnection.subscriber_connection
89
- setup_queues!
90
+ setup_subscriptions!
90
91
  print_subscriptions
91
92
  end
92
93
 
93
94
  def self.start_subscribers
94
- ::ActionSubscriber::RabbitConnection.subscriber_connection
95
- setup_queues!
95
+ setup_subscriptions!
96
96
  auto_subscribe!
97
97
  print_subscriptions
98
98
  end
@@ -105,21 +105,14 @@ module ActionSubscriber
105
105
  require "action_subscriber/railtie" if defined?(Rails)
106
106
  ::ActiveSupport.run_load_hooks(:action_subscriber, Base)
107
107
 
108
- # Intialize async publisher adapter
109
- ::ActionSubscriber::Publisher::Async.publisher_adapter
110
-
111
108
  ##
112
109
  # Private Implementation
113
110
  #
114
111
  def self.route_set
115
112
  @route_set ||= begin
116
- if @draw_routes_block
117
- routes = Router.draw_routes(&@draw_routes_block)
118
- RouteSet.new(routes)
119
- else
120
- logger.warn "DEPRECATION WARNING: We are inferring your routes by looking at your subscribers. This behavior is deprecated and will be removed in version 2.0. Please see the routing guide at https://github.com/mxenabled/action_subscriber/blob/master/routing.md"
121
- RouteSet.new(self.send(:default_routes))
122
- end
113
+ fail "cannot start because no routes have been defined. Please make sure that you call ActionSubscriber.draw_routes when your application loads" unless @draw_routes_block
114
+ routes = Router.draw_routes(&@draw_routes_block)
115
+ RouteSet.new(routes)
123
116
  end
124
117
  end
125
118
  private_class_method :route_set
@@ -131,8 +124,3 @@ module ActionSubscriber
131
124
  end
132
125
  private_class_method :default_routes
133
126
  end
134
-
135
- at_exit do
136
- ::ActionSubscriber::Publisher::Async.publisher_adapter.shutdown!
137
- ::ActionSubscriber::RabbitConnection.publisher_disconnect!
138
- end
@@ -7,7 +7,7 @@ module ActionSubscriber
7
7
  def self.auto_pop!
8
8
  @pop_mode = true
9
9
  reload_active_record
10
- load_subscribers unless subscribers_loaded?
10
+ ::ActionSubscriber.setup_default_connection!
11
11
  sleep_time = ::ActionSubscriber.configuration.pop_interval.to_i / 1000.0
12
12
 
13
13
  ::ActionSubscriber.start_queues
@@ -30,7 +30,7 @@ module ActionSubscriber
30
30
  def self.start_subscribers
31
31
  @prowl_mode = true
32
32
  reload_active_record
33
- load_subscribers unless subscribers_loaded?
33
+ ::ActionSubscriber.setup_default_connection!
34
34
 
35
35
  ::ActionSubscriber.start_subscribers
36
36
  logger.info "Action Subscriber connected"
@@ -45,29 +45,6 @@ module ActionSubscriber
45
45
  !!@prowl_mode
46
46
  end
47
47
 
48
- def self.load_subscribers
49
- subscription_paths = ["subscriptions", "subscribers"]
50
- path_prefixes = ["lib", "app"]
51
- cloned_paths = subscription_paths.dup
52
-
53
- path_prefixes.each do |prefix|
54
- cloned_paths.each { |path| subscription_paths << "#{prefix}/#{path}" }
55
- end
56
-
57
- absolute_subscription_paths = subscription_paths.map{ |path| ::File.expand_path(path) }
58
- absolute_subscription_paths.each do |path|
59
- if ::File.exists?("#{path}.rb")
60
- load("#{path}.rb")
61
- end
62
-
63
- if ::File.directory?(path)
64
- ::Dir[::File.join(path, "**", "*.rb")].sort.each do |file|
65
- load file
66
- end
67
- end
68
- end
69
- end
70
-
71
48
  def self.logger
72
49
  ::ActionSubscriber::Logging.logger
73
50
  end
@@ -108,9 +85,5 @@ module ActionSubscriber
108
85
 
109
86
  puts "threadpool empty. Shutting down"
110
87
  end
111
-
112
- def self.subscribers_loaded?
113
- !::ActionSubscriber::Base.inherited_classes.empty?
114
- end
115
88
  end
116
89
  end
@@ -26,10 +26,6 @@ module ActionSubscriber
26
26
  # Class Methods
27
27
  #
28
28
 
29
- def self.connection
30
- ::ActionSubscriber::RabbitConnection.subscriber_connection
31
- end
32
-
33
29
  # Inherited callback, save a reference to our descendents
34
30
  #
35
31
  def self.inherited(klass)
@@ -11,12 +11,18 @@ module ActionSubscriber
11
11
  bunny_consumers.each(&:cancel)
12
12
  end
13
13
 
14
+ def create_queue(channel, queue_name, queue_options)
15
+ ::Bunny::Queue.new(channel, queue_name, queue_options)
16
+ end
17
+
14
18
  def auto_pop!
15
19
  # Because threadpools can be large we want to cap the number
16
20
  # of times we will pop each time we poll the broker
17
21
  times_to_pop = [::ActionSubscriber::Threadpool.ready_size, ::ActionSubscriber.config.times_to_pop].min
18
22
  times_to_pop.times do
19
- queues.each do |route, queue|
23
+ subscriptions.each do |subscription|
24
+ route = subscription[:route]
25
+ queue = subscription[:queue]
20
26
  # Handle busy checks on a per threadpool basis
21
27
  next if route.threadpool.busy?
22
28
 
@@ -25,7 +31,6 @@ module ActionSubscriber
25
31
  ::ActiveSupport::Notifications.instrument "popped_event.action_subscriber", :payload_size => encoded_payload.bytesize, :queue => queue.name
26
32
  properties = {
27
33
  :action => route.action,
28
- :channel => queue.channel,
29
34
  :content_type => properties[:content_type],
30
35
  :delivery_tag => delivery_info.delivery_tag,
31
36
  :exchange => delivery_info.exchange,
@@ -41,7 +46,9 @@ module ActionSubscriber
41
46
  end
42
47
 
43
48
  def auto_subscribe!
44
- queues.each do |route, queue|
49
+ subscriptions.each do |subscription|
50
+ route = subscription[:route]
51
+ queue = subscription[:queue]
45
52
  channel = queue.channel
46
53
  channel.prefetch(route.prefetch) if route.acknowledgements?
47
54
  consumer = ::Bunny::Consumer.new(channel, queue, channel.generate_consumer_tag, !route.acknowledgements?)
@@ -59,7 +66,7 @@ module ActionSubscriber
59
66
  :queue => queue.name,
60
67
  }
61
68
  env = ::ActionSubscriber::Middleware::Env.new(route.subscriber, encoded_payload, properties)
62
- enqueue_env(route.threadpool, env)
69
+ run_env(env)
63
70
  end
64
71
  bunny_consumers << consumer
65
72
  queue.subscribe_with(consumer)
@@ -76,6 +83,13 @@ module ActionSubscriber
76
83
  end
77
84
  end
78
85
  end
86
+
87
+ def run_env(env)
88
+ logger.info "RECEIVED #{env.message_id} from #{env.queue}"
89
+ ::ActiveSupport::Notifications.instrument "process_event.action_subscriber", :subscriber => env.subscriber.to_s, :routing_key => env.routing_key, :queue => env.queue do
90
+ ::ActionSubscriber.config.middleware.call(env)
91
+ end
92
+ end
79
93
  end
80
94
  end
81
95
  end
@@ -4,10 +4,6 @@ require "action_subscriber/uri"
4
4
  module ActionSubscriber
5
5
  class Configuration
6
6
  attr_accessor :allow_low_priority_methods,
7
- :async_publisher,
8
- :async_publisher_drop_messages_when_queue_full,
9
- :async_publisher_max_queue_size,
10
- :async_publisher_supervisor_interval,
11
7
  :decoder,
12
8
  :default_exchange,
13
9
  :error_handler,
@@ -19,7 +15,6 @@ module ActionSubscriber
19
15
  :pop_interval,
20
16
  :port,
21
17
  :prefetch,
22
- :publisher_confirms,
23
18
  :seconds_to_wait_for_graceful_shutdown,
24
19
  :username,
25
20
  :threadpool_size,
@@ -31,10 +26,6 @@ module ActionSubscriber
31
26
 
32
27
  DEFAULTS = {
33
28
  :allow_low_priority_methods => false,
34
- :async_publisher => 'memory',
35
- :async_publisher_drop_messages_when_queue_full => false,
36
- :async_publisher_max_queue_size => 1_000_000,
37
- :async_publisher_supervisor_interval => 200, # in milliseconds
38
29
  :default_exchange => 'events',
39
30
  :heartbeat => 5,
40
31
  :host => 'localhost',
@@ -42,8 +33,7 @@ module ActionSubscriber
42
33
  :mode => 'subscribe',
43
34
  :pop_interval => 100, # in milliseconds
44
35
  :port => 5672,
45
- :prefetch => 5,
46
- :publisher_confirms => false,
36
+ :prefetch => 2,
47
37
  :seconds_to_wait_for_graceful_shutdown => 30,
48
38
  :threadpool_size => 8,
49
39
  :timeout => 1,
@@ -1,19 +1,21 @@
1
1
  module ActionSubscriber
2
2
  module DefaultRouting
3
- def routes
3
+ def routes(route_settings)
4
4
  @routes ||= begin
5
5
  routes = []
6
6
  exchange_names.each do |exchange_name|
7
7
  subscribable_methods.each do |method_name|
8
- routes << ActionSubscriber::Route.new({
8
+ settings = {
9
9
  acknowledgements: acknowledge_messages?,
10
10
  action: method_name,
11
- durable: false,
11
+ durable: false,
12
12
  exchange: exchange_name,
13
13
  routing_key: routing_key_name_for_method(method_name),
14
14
  subscriber: self,
15
15
  queue: queue_name_for_method(method_name),
16
- })
16
+ }
17
+ settings.merge!(route_settings)
18
+ routes << ActionSubscriber::Route.new(settings)
17
19
  end
18
20
  end
19
21
  routes
@@ -7,12 +7,20 @@ module ActionSubscriber
7
7
  march_hare_consumers.each(&:cancel)
8
8
  end
9
9
 
10
+ def create_queue(channel, queue_name, queue_options)
11
+ queue = ::MarchHare::Queue.new(channel, queue_name, queue_options)
12
+ queue.declare!
13
+ queue
14
+ end
15
+
10
16
  def auto_pop!
11
17
  # Because threadpools can be large we want to cap the number
12
18
  # of times we will pop each time we poll the broker
13
19
  times_to_pop = [::ActionSubscriber::Threadpool.ready_size, ::ActionSubscriber.config.times_to_pop].min
14
20
  times_to_pop.times do
15
- queues.each do |route,queue|
21
+ subscriptions.each do |subscription|
22
+ route = subscription[:route]
23
+ queue = subscription[:queue]
16
24
  # Handle busy checks on a per threadpool basis
17
25
  next if route.threadpool.busy?
18
26
 
@@ -21,7 +29,6 @@ module ActionSubscriber
21
29
  ::ActiveSupport::Notifications.instrument "popped_event.action_subscriber", :payload_size => encoded_payload.bytesize, :queue => queue.name
22
30
  properties = {
23
31
  :action => route.action,
24
- :channel => queue.channel,
25
32
  :content_type => metadata.content_type,
26
33
  :delivery_tag => metadata.delivery_tag,
27
34
  :exchange => metadata.exchange,
@@ -40,7 +47,9 @@ module ActionSubscriber
40
47
  end
41
48
 
42
49
  def auto_subscribe!
43
- queues.each do |route,queue|
50
+ subscriptions.each do |subscription|
51
+ route = subscription[:route]
52
+ queue = subscription[:queue]
44
53
  queue.channel.prefetch = route.prefetch if route.acknowledgements?
45
54
  consumer = queue.subscribe(route.queue_subscription_options) do |metadata, encoded_payload|
46
55
  ::ActiveSupport::Notifications.instrument "received_event.action_subscriber", :payload_size => encoded_payload.bytesize, :queue => queue.name
@@ -56,7 +65,7 @@ module ActionSubscriber
56
65
  :queue => queue.name,
57
66
  }
58
67
  env = ::ActionSubscriber::Middleware::Env.new(route.subscriber, encoded_payload, properties)
59
- enqueue_env(route.threadpool, env)
68
+ run_env(env)
60
69
  end
61
70
 
62
71
  march_hare_consumers << consumer
@@ -78,6 +87,13 @@ module ActionSubscriber
78
87
  end
79
88
  end
80
89
 
90
+ def run_env(env)
91
+ logger.info "RECEIVED #{env.message_id} from #{env.queue}"
92
+ ::ActiveSupport::Notifications.instrument "process_event.action_subscriber", :subscriber => env.subscriber.to_s, :routing_key => env.routing_key, :queue => env.queue do
93
+ ::ActionSubscriber.config.middleware.call(env)
94
+ end
95
+ end
96
+
81
97
  def _normalized_headers(metadata)
82
98
  return {} unless metadata.headers
83
99
  metadata.headers.each_with_object({}) do |(header,value), hash|
@@ -46,7 +46,7 @@ module ActionSubscriber
46
46
  end
47
47
 
48
48
  def self.with_exchange(env, ttl, retry_queue_name)
49
- channel = RabbitConnection.subscriber_connection.create_channel
49
+ channel = RabbitConnection.with_connection(:default){|connection| connection.create_channel}
50
50
  begin
51
51
  channel.confirm_select
52
52
  # an empty string is the default exchange [see bunny docs](http://rubybunny.info/articles/exchanges.html#default_exchange)
@@ -28,7 +28,7 @@ module ActionSubscriber
28
28
  # :routing_key => String
29
29
  def initialize(subscriber, encoded_payload, properties)
30
30
  @action = properties.fetch(:action)
31
- @channel = properties.fetch(:channel)
31
+ @channel = properties[:channel]
32
32
  @content_type = properties.fetch(:content_type)
33
33
  @delivery_tag = properties.fetch(:delivery_tag)
34
34
  @encoded_payload = encoded_payload
@@ -41,12 +41,14 @@ module ActionSubscriber
41
41
  end
42
42
 
43
43
  def acknowledge
44
+ fail ::RuntimeError, "you can't acknowledge messages under the polling API" unless @channel
44
45
  acknowledge_multiple_messages = false
45
46
  @channel.ack(@delivery_tag, acknowledge_multiple_messages)
46
47
  true
47
48
  end
48
49
 
49
50
  def reject
51
+ fail ::RuntimeError, "you can't acknowledge messages under the polling API" unless @channel
50
52
  requeue_message = true
51
53
  @channel.reject(@delivery_tag, requeue_message)
52
54
  true