action_subscriber 3.0.2 → 4.0.0.pre1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: bf91f9b2566e00305d9bcc1bb873c550390f29ad
-  data.tar.gz: e523407919e74f5af41f321e4a2bc64d00bcaf59
+  metadata.gz: 2ddc9cc72ddda0d121c1ffec7e99dca6d0ca323b
+  data.tar.gz: 401a3e22fd0bbbc48cfc414c55924b11b8146ac8
 SHA512:
-  metadata.gz: f4d5e7566aabf121149d7d2eb453f8892872b268e018a07b6afbada975ce1dcd132307d54a35416077413f8b05174bdd0e0eae869e15a96f7cf4b2bcb8a1aab3
-  data.tar.gz: 13c8d270b87969981b92f9777cde802ad1acc9b85c5d68577105730746561f6bc72de1d055efd0956a9cb9e5482dcbb1bf2930a147078af29ead94c6163c463a
+  metadata.gz: 1f8cca49c8ce821d1dca22951a41df636260a48d0a018c65ac3539dc27d7ce2ee200eb4d2e473d9c153a5d5ac01460efb0096bb5e94df850735c158c8ee3655f
+  data.tar.gz: 9eb940f6546ed1d35314aac1004eb87d00012a251fa7b61981b9dde4725be71317e8d7f0c4b83b6be495e584c679a9221dcb9d02d6b734c1e80b25e2b0df330d

data/README.md

@@ -13,6 +13,38 @@ I test on Ruby 2.2.1 and Jruby 9.x.  MRI 1.9 and jRuby 1.7 are still supported.
 
 If you want to use MRI 1.9 you will need to lock down the `amq-protocol` and `bunny` gems to `< 2.0` since they both require ruby 2.0+.
 
+Migrating from ActionSubscriber 3.X or earlier
+----------------------------------------------
+
+If you were using the `--mode=pop` from the 2.X or 3.X version of ActionSubscriber you can get the same sort of behavior by drawing your routes like this:
+
+```ruby
+::ActionSubscriber.draw_routes do
+  # instead of creating custom threadpools you set the threadpool size of your connection here in the routes
+  # you can set the threadpool size for the default connection via the `::ActionSubscriber.configuration.threadpool_size = 16`
+  route UserSubscriber, :created,
+    :prefetch => 1,
+    :concurrency => 16,
+    :acknowledgements => true
+
+  # in user_subscriber.rb make sure to set `at_most_once!` like this
+  #
+  # class UserSubscriber < ::ActionSubscriber::Base
+  #   at_most_once!
+  # end
+
+  # If you were previously using custom threadpools for different routes you can mimic that behavior by opening multiple connections
+  connection(:slow_work, :threadpool_size => 32) do
+    route UserSubscriber, :created,
+      :prefetch => 1,
+      :concurrency => 32,
+      :acknowledgements => true
+  end
+end
+```
+
+That will give you a similar behavior to the old `--mode=pop` where messages polled from the server, but with reduced latency.
+
 Supported Message Types
 -----------------
 ActionSubscriber support JSON and plain text out of the box, but you can easily
@@ -51,12 +83,10 @@ Now you can start your subscriber process with:
 
 
 ```
-$ bundle exec action_subscriber start --mode=subscribe
+$ bundle exec action_subscriber start
 ```
 
-This will start your subscribers in a mode where they connect to rabbitmq and let the broker push messages down to them.
-
-You can also start in `--mode=pop` where your process will poll the broker for messages.
+This will connect your subscribers to the rabbitmq broker and allow it to push messages down to your subscribers.
 
 Configuration
 -----------------
@@ -76,16 +106,18 @@ Other configuration options include :
 * config.error_handler - handle error like you want to handle them!
 * config.heartbeat - number of seconds between hearbeats (default 5) [see bunny documentation for more details](http://rubybunny.info/articles/connecting.html)
 * config.hosts - an array of hostnames in your cluster (ie `["rabbit1.myapp.com", "rabbit2.myapp.com"]`)
-* config.pop_interval - how long to wait between polling for messages in `--mode=pop`. It should be a number of milliseconds
 * config.threadpool_size - set the number of threads availiable to action_subscriber
 * config.timeout - how many seconds to allow rabbit to respond before timing out
-* config.times_to_pop - when using RabbitMQ's pull API, the number of messages we will grab each time we pool the broker
 
 Message Acknowledgment
 ----------------------
 ### no_acknolwedgement!
 
 This mode is the default. Rabbit is told to not expect any message acknowledgements so messages will be lost if an error occurs.
+This also allows the broker to send messages as quickly as it wants down to your subscriber.
+
+> Warning: If messages arrive very quickly this could cause your process to crash as your memory fills up with unprocessed message.
+> We highly recommend you use `at_least_once!` mode to provide a throttle so the broker does not overwhelm your process with messages.
 
 ### manual_acknowledgement!
 
@@ -97,7 +129,9 @@ Rabbit is told to expect message acknowledgements, but sending the acknowledgeme
 
 ### at_least_once!
 
-Rabbit is told to expect message acknowledgements, but sending the acknowledgement is left up to ActionSubscriber. We send the acknowledgement right after calling your subscriber. If an error is raised by your subscriber we reject the message instead of acknowledging it. Rejected messages go back to rabbit and will be re-delivered.
+Rabbit is told to expect message acknowledgements, but sending the acknowledgement is left up to ActionSubscriber.
+We send the acknowledgement right after calling your subscriber.
+If an error is raised your message will be retried on a sent back to rabbitmq and retried on an exponential backoff schedule.
 
 Testing
 -----------------

data/action_subscriber.gemspec

@@ -26,7 +26,6 @@ Gem::Specification.new do |spec|
   else
     spec.add_dependency 'bunny', '>= 1.5.0'
   end
-  spec.add_dependency 'lifeguard', '>= 0.0.9'
   spec.add_dependency 'middleware'
   spec.add_dependency 'thor'
 

data/bin/action_subscriber

@@ -7,7 +7,6 @@ module ActionSubscriber
   class CLI < ::Thor
     class_option :allow_low_priority_methods, :type => :boolean, :desc => "subscribe to low priority queues in addition to the normal queues", :default => false
     class_option :app, :default => "./config/environment.rb"
-    class_option :mode
     class_option :host
     class_option :hosts
     class_option :prefetch, :type => :numeric, :desc => "how many messages to hold in the local queue in subscribe mode"
@@ -36,14 +35,7 @@ module ActionSubscriber
       # Require action_subscriber if the application did not.
       require "action_subscriber"
 
-      case ::ActionSubscriber.configuration.mode
-      when /pop/i then
-        ::ActionSubscriber::Babou.auto_pop!
-      when /subscribe/i then
-        ::ActionSubscriber::Babou.start_subscribers
-      else
-        fail "ActionSubscriber.configuration.mode must be 'pop' or 'subscribe'.  Currently set to '#{::ActionSubscriber.configuration.mode}'"
-      end
+      ::ActionSubscriber::Babou.start_subscribers
     end
   end
 
@@ -62,15 +54,15 @@ module ActionSubscriber
 end
 
 trap(:TTIN) {
-  ::ActionSubscriber.print_subscriptions
+  ::Thread.new do
+    ::ActionSubscriber.print_subscriptions
+  end
 }
 
 trap(:USR2) {
-  puts <<-CONFIG.strip_heredoc
-  Action Subscriber Stats
-    Pool Size: #{ ::ActionSubscriber.config.threadpool_size }
-    Ready Size: #{ ::ActionSubscriber::Threadpool.ready_size }
-  CONFIG
+  ::Thread.new do
+    ::ActionSubscriber.print_threadpool_stats
+  end.join
 }
 
 ::ActionSubscriber::CLI.start(ARGV)

data/lib/action_subscriber.rb

@@ -5,7 +5,6 @@ if ::RUBY_PLATFORM == "java"
 else
   require "bunny"
 end
-require "lifeguard"
 require "middleware"
 require "thread"
 
@@ -28,7 +27,6 @@ require "action_subscriber/babou"
 require "action_subscriber/route"
 require "action_subscriber/route_set"
 require "action_subscriber/router"
-require "action_subscriber/threadpool"
 require "action_subscriber/base"
 
 module ActionSubscriber
@@ -36,21 +34,6 @@ module ActionSubscriber
   # Public Class Methods
   #
 
-  # Loop over all subscribers and pull messages if there are
-  # any waiting in the queue for us.
-  #
-  def self.auto_pop!
-    return if ::ActionSubscriber::Threadpool.busy?
-    route_set.auto_pop!
-  end
-
-  # Loop over all subscribers and register each as
-  # a subscriber.
-  #
-  def self.auto_subscribe!
-    route_set.auto_subscribe!
-  end
-
   def self.configure
     yield(configuration) if block_given?
   end
@@ -66,16 +49,11 @@ module ActionSubscriber
 
   def self.print_subscriptions
     logger.info configuration.inspect
-    route_set.routes.group_by(&:subscriber).each do |subscriber, routes|
-      logger.info subscriber.name
-      routes.each do |route|
-        logger.info "  -- method: #{route.action}"
-        logger.info "    --    exchange: #{route.exchange}"
-        logger.info "    --       queue: #{route.queue}"
-        logger.info "    -- routing_key: #{route.routing_key}"
-        logger.info "    --  threadpool: #{route.threadpool.name}, pool_size: #{route.threadpool.pool_size}"
-      end
-    end
+    route_set.print_subscriptions
+  end
+
+  def self.print_threadpool_stats
+    route_set.print_threadpool_stats
   end
 
   def self.setup_default_connection!
@@ -86,24 +64,14 @@ module ActionSubscriber
     route_set.setup_subscriptions!
   end
 
-  def self.start_queues
-    setup_subscriptions!
-    print_subscriptions
-  end
-
-  def self.start_subscribers
-    setup_subscriptions!
-    auto_subscribe!
-    print_subscriptions
+  def self.start_subscribers!
+    route_set.start_subscribers!
   end
 
-  def self.stop_subscribers!
+  def self.stop_subscribers!(timeout = nil)
+    timeout ||= ::ActionSubscriber.configuration.seconds_to_wait_for_graceful_shutdown
     route_set.cancel_consumers!
-  end
-
-  def self.wait_for_threadpools_to_finish_with_timeout(timeout)
-    puts "waiting for threadpools to empty (maximum wait of #{::ActionSubscriber.configuration.seconds_to_wait_for_graceful_shutdown}sec)"
-    ::ActionSubscriber::Threadpool.wait_to_finish_with_timeout(timeout)
+    puts "waiting for threadpools to empty (maximum wait of #{timeout}sec)"
     route_set.wait_to_finish_with_timeout(timeout)
   end
 
@@ -122,11 +90,4 @@ module ActionSubscriber
     end
   end
   private_class_method :route_set
-
-  def self.default_routes
-    ::ActionSubscriber::Base.inherited_classes.flat_map do |klass|
-      klass.routes
-    end
-  end
-  private_class_method :default_routes
 end

data/lib/action_subscriber/babou.rb

@@ -3,36 +3,12 @@ module ActionSubscriber
     ##
     # Class Methods
     #
-
-    def self.auto_pop!
-      @pop_mode = true
-      reload_active_record
-      ::ActionSubscriber.setup_default_connection!
-      sleep_time = ::ActionSubscriber.configuration.pop_interval.to_i / 1000.0
-
-      ::ActionSubscriber.start_queues
-      logger.info "Action Subscriber is popping messages every #{sleep_time} seconds."
-
-      # How often do we want the timer checking for new pops
-      # since we included an eager popper we decreased the
-      # default check interval to 100m
-      while true
-        ::ActionSubscriber.auto_pop! unless shutting_down?
-        sleep sleep_time
-        break if shutting_down?
-      end
-    end
-
-    def self.pop?
-      !!@pop_mode
-    end
-
     def self.start_subscribers
-      @prowl_mode = true
       reload_active_record
       ::ActionSubscriber.setup_default_connection!
-
-      ::ActionSubscriber.start_subscribers
+      ::ActionSubscriber.setup_subscriptions!
+      ::ActionSubscriber.print_subscriptions
+      ::ActionSubscriber.start_subscribers!
       logger.info "Action Subscriber connected"
 
       while true
@@ -41,10 +17,6 @@ module ActionSubscriber
       end
     end
 
-    def self.prowl?
-      !!@prowl_mode
-    end
-
     def self.logger
       ::ActionSubscriber::Logging.logger
     end
@@ -59,21 +31,13 @@ module ActionSubscriber
       !!@shutting_down
     end
 
-    def self.stop_receving_messages!
-      @shutting_down = true
-      ::Thread.new do
-        ::ActionSubscriber.stop_subscribers!
-        logger.info "stopped all subscribers"
-      end.join
-    end
-
     def self.stop_server!
       # this method is called from within a TRAP context so we can't use the logger
-      puts "Stopping server..."
-      ::ActionSubscriber::Babou.stop_receving_messages!
-      ::ActionSubscriber.wait_for_threadpools_to_finish_with_timeout(::ActionSubscriber.configuration.seconds_to_wait_for_graceful_shutdown)
-      puts "Shutting down"
+      @shutting_down = true
       ::Thread.new do
+        puts "Stopping subscribers..."
+        ::ActionSubscriber.stop_subscribers!
+        puts "Shutting down"
         ::ActionSubscriber::RabbitConnection.subscriber_disconnect!
       end.join
     end

data/lib/action_subscriber/base.rb

@@ -56,76 +56,6 @@ module ActionSubscriber
       env.acknowledge
     end
 
-    def _at_least_once_filter
-      processed_acknowledgement = false
-      yield
-      processed_acknowledgement = acknowledge
-    rescue => error
-      ::ActionSubscriber::MessageRetry.redeliver_message_with_backoff(env)
-      processed_acknowledgement = acknowledge
-      raise error
-    ensure
-      rejected_message = false
-      rejected_message = reject unless processed_acknowledgement
-
-      if !processed_acknowledgement && !rejected_message
-        Process.kill(:TTIN, Process.pid)
-        Process.kill(:USR2, Process.pid)
-
-        $stdout << <<-UNREJECTABLE
-          CANNOT ACKNOWLEDGE OR REJECT THE MESSAGE
-
-          This is a exceptional state for ActionSubscriber to enter and puts the current
-          Process in the position of "I can't get new work from RabbitMQ, but also
-          can't acknowledge or reject the work that I currently have" ... While rare
-          this state can happen.
-
-          Instead of continuing to try to process the message ActionSubscriber is
-          sending a Kill signal to the current running process to gracefully shutdown
-          so that the RabbitMQ server will purge any outstanding acknowledgements. If
-          you are running a process monitoring tool (like Upstart) the Subscriber
-          process will be restarted and be able to take on new work.
-
-          ** Running a process monitoring tool like Upstart is recommended for this reason **
-        UNREJECTABLE
-
-        Process.kill(:TERM, Process.pid)
-      end
-    end
-
-    def _at_most_once_filter
-      processed_acknowledgement = false
-      processed_acknowledgement = acknowledge
-      yield
-    ensure
-      rejected_message = false
-      rejected_message = reject unless processed_acknowledgement
-
-      if !processed_acknowledgement && !rejected_message
-        Process.kill(:TTIN, Process.pid)
-        Process.kill(:USR2, Process.pid)
-
-        $stdout << <<-UNREJECTABLE
-          CANNOT ACKNOWLEDGE OR REJECT THE MESSAGE
-
-          This is a exceptional state for ActionSubscriber to enter and puts the current
-          Process in the position of "I can't get new work from RabbitMQ, but also
-          can't acknowledge or reject the work that I currently have" ... While rare
-          this state can happen.
-
-          Instead of continuing to try to process the message ActionSubscriber is
-          sending a Kill signal to the current running process to gracefully shutdown
-          so that the RabbitMQ server will purge any outstanding acknowledgements. If
-          you are running a process monitoring tool (like Upstart) the Subscriber
-          process will be restarted and be able to take on new work.
-
-          ** Running a process monitoring tool like Upstart is recommended for this reason **
-        UNREJECTABLE
-
-        Process.kill(:TERM, Process.pid)
-      end
-    end
-
     def reject
       env.reject
     end

data/lib/action_subscriber/bunny/subscriber.rb

@@ -11,41 +11,51 @@ module ActionSubscriber
         bunny_consumers.each(&:cancel)
       end
 
-      def create_queue(channel, queue_name, queue_options)
-        ::Bunny::Queue.new(channel, queue_name, queue_options)
+      def print_subscriptions
+        routes.group_by(&:subscriber).each do |subscriber, routes|
+          logger.info subscriber.name
+          routes.each do |route|
+            logger.info "  -- method: #{route.action}"
+            logger.info "    --  connection: #{route.connection_name}"
+            logger.info "    -- concurrency: #{route.concurrency}"
+            logger.info "    --    exchange: #{route.exchange}"
+            logger.info "    --       queue: #{route.queue}"
+            logger.info "    -- routing_key: #{route.routing_key}"
+            logger.info "    --    prefetch: #{route.prefetch}"
+            logger.error "WARNING having a prefetch lower than your concurrency will prevent your subscriber from fully utilizing its threadpool" if route.prefetch < route.concurrency
+          end
+        end
       end
 
-      def auto_pop!
-        # Because threadpools can be large we want to cap the number
-        # of times we will pop each time we poll the broker
-        times_to_pop = [::ActionSubscriber::Threadpool.ready_size, ::ActionSubscriber.config.times_to_pop].min
-        times_to_pop.times do
+      def print_threadpool_stats
+        logger.info "*DISCLAIMER* the number of running jobs is just a best guess. We don't have a good way to introspect the bunny threadpools so jobs that are sleeping or waiting on IO won't show up as running"
+        subscriptions.group_by{|subscription| subscription[:route].subscriber}.each do |subscriber, subscriptions|
+          logger.info subscriber.name
           subscriptions.each do |subscription|
             route = subscription[:route]
-            queue = subscription[:queue]
-            # Handle busy checks on a per threadpool basis
-            next if route.threadpool.busy?
-
-            delivery_info, properties, encoded_payload = queue.pop(route.queue_subscription_options)
-            next unless encoded_payload # empty queue
-            ::ActiveSupport::Notifications.instrument "popped_event.action_subscriber", :payload_size => encoded_payload.bytesize, :queue => queue.name
-            properties = {
-              :action => route.action,
-              :content_type => properties[:content_type],
-              :delivery_tag => delivery_info.delivery_tag,
-              :exchange => delivery_info.exchange,
-              :headers => properties.headers,
-              :message_id => nil,
-              :routing_key => delivery_info.routing_key,
-              :queue => queue.name,
-            }
-            env = ::ActionSubscriber::Middleware::Env.new(route.subscriber, encoded_payload, properties)
-            enqueue_env(route.threadpool, env)
+            work_pool = subscription[:queue].channel.work_pool
+            running_threads = work_pool.threads.select{|thread| thread.status == "run"}.count
+            routes.each do |route|
+              logger.info "  -- method: #{route.action}"
+              logger.info "    --  concurrency: #{route.concurrency}"
+              logger.info "    -- running jobs: #{running_threads}"
+              logger.info "    --      backlog: #{work_pool.backlog}"
+            end
           end
         end
       end
 
-      def auto_subscribe!
+      def setup_subscriptions!
+        fail ::RuntimeError, "you cannot setup queues multiple times, this should only happen once at startup" unless subscriptions.empty?
+        routes.each do |route|
+          subscriptions << {
+            :route => route,
+            :queue => setup_queue(route),
+          }
+        end
+      end
+
+      def start_subscribers!
         subscriptions.each do |subscription|
           route = subscription[:route]
           queue = subscription[:queue]
@@ -84,20 +94,12 @@ module ActionSubscriber
 
       private
 
-      def enqueue_env(threadpool, env)
-        logger.info "RECEIVED #{env.message_id} from #{env.queue}"
-        threadpool.async(env) do |env|
-          ::ActiveSupport::Notifications.instrument "process_event.action_subscriber", :subscriber => env.subscriber.to_s, :routing_key => env.routing_key, :queue => env.queue do
-            ::ActionSubscriber.config.middleware.call(env)
-          end
-        end
-      end
-
-      def run_env(env)
-        logger.info "RECEIVED #{env.message_id} from #{env.queue}"
-        ::ActiveSupport::Notifications.instrument "process_event.action_subscriber", :subscriber => env.subscriber.to_s, :routing_key => env.routing_key, :queue => env.queue do
-          ::ActionSubscriber.config.middleware.call(env)
-        end
+      def setup_queue(route)
+        channel = ::ActionSubscriber::RabbitConnection.with_connection(route.connection_name){ |connection| connection.create_channel(nil, route.concurrency) }
+        exchange = channel.topic(route.exchange)
+        queue = channel.queue(route.queue, :durable => route.durable)
+        queue.bind(exchange, :routing_key => route.routing_key)
+        queue
       end
     end
   end