shoryuken 3.0.9 → 3.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: c88cb34dfa2eae0fbeea81ff493f88c389232edb
-  data.tar.gz: b271d7d9015868d1d5b344d4f338c3e603e57b50
+  metadata.gz: 779c3e9c7f708f0a6dccb3e8b0c9f7e2490b25e1
+  data.tar.gz: bf260a2793ff8c07ddfb670832ba5f12af8d85d9
 SHA512:
-  metadata.gz: 2a7a0ba02ee69d37ee8df087c9c260443cdb9559a48bd1bb58ea93e7c0281a8863acf9e91818b41ce91c211e3569d94fb88cf925ffbad579ccfa33dc917eb431
-  data.tar.gz: dbb15a8ff723305b3db21c01fa74eca91e9f782c07e5d65440eb71a3141969fd9a6bd38d0fe09b15f96260f7580b127c5ba78121f0444b5027ea011464793049
+  metadata.gz: 1f3e7a4b668c929daa1b5424b7a8ab7f028238b3f6dfe0e812351991bbc37923b58491760df6e4ed4921624cd4954761daeac257c6a540fd193d160d121044de
+  data.tar.gz: 02f2bf2e813cbe1624ae83d16936196ecf523fe8d3773f9b0c39813aa0822d3c71fd273ee01a63072c20d86bf05f4c1a775eda3fae41cec4bf55ea567f772aef

data/.travis.yml

@@ -4,6 +4,7 @@ rvm:
   - 2.1.0
   - 2.2.0
   - 2.3.3
+  - 2.4.1
 
 notifications:
   email:

data/CHANGELOG.md

@@ -1,3 +1,27 @@
+## [v3.1.0] - 2017-07-02
+
+- Add shoryuken sqs delete command
+ - [#395](https://github.com/phstc/shoryuken/pull/395)
+
+- Add processing groups support; Concurrency per queue support
+ - [#389](https://github.com/phstc/shoryuken/pull/389)
+
+- Terminate Shoryuken if the fetcher crashes
+ - [#389](https://github.com/phstc/shoryuken/pull/389)
+
+## [v3.0.11] - 2017-06-24
+
+- Add shoryuken sqs create command
+ - [#388](https://github.com/phstc/shoryuken/pull/388)
+
+## [v3.0.10] - 2017-06-24
+
+- Allow aws sdk v3
+ - [#381](https://github.com/phstc/shoryuken/pull/381)
+
+- Allow configuring Rails via the config file
+ - [#387](https://github.com/phstc/shoryuken/pull/387)
+
 ## [v3.0.9] - 2017-06-05
 
 - Allow configuring queue URLs instead of names

data/Gemfile

@@ -7,4 +7,5 @@ group :test do
   gem 'codeclimate-test-reporter', require: nil
   gem 'simplecov'
   gem 'multi_xml'
+  gem 'httparty'
 end

data/README.md

@@ -1,35 +1,23 @@
 # Shoryuken
 
-![](shoryuken.jpg)
+![Shoryuken](shoryuken.jpg)
 
-Shoryuken _sho-ryu-ken_ is a super-efficient [AWS SQS](https://aws.amazon.com/sqs/) thread-based message processor.
+Shoryuken _sho-ryu-ken_ is a super-efficient [Amazon SQS](https://aws.amazon.com/sqs/) thread-based message processor.
 
 [![Build Status](https://travis-ci.org/phstc/shoryuken.svg)](https://travis-ci.org/phstc/shoryuken)
 [![Code Climate](https://codeclimate.com/github/phstc/shoryuken/badges/gpa.svg)](https://codeclimate.com/github/phstc/shoryuken)
 
 ## Key features
 
-### Load balancing
-
-Yeah, Shoryuken load balances the messages consumption!
-
-```yaml
-concurrency: 25
-delay: 0
-queues:
-  - [high_priority, 6]
-  - [normal_priority, 2]
-  - [low_priority, 1]
-```
-
-Or you can set them all to 1 for having the same priorities.
-
-- `concurrency` (default 25) is the number of threads you want to make available for processing. 
-- `delay` (default to 0) is the number of seconds to pause fetching from an empty queue. SQS charges per request, pause checking empty queues for a while can be a cost efficient strategy.
-
-### Fetch in batches
-
-To be even more performant and cost effective, Shoryuken fetches SQS messages in batches, so a single SQS request can fetch up to 10 messages.
+- [Rails Active Job](https://github.com/phstc/shoryuken/wiki/Rails-Integration-Active-Job)
+- [Queue Load balancing](https://github.com/phstc/shoryuken/wiki/Shoryuken-options#load-balancing)
+- [Concurrency per queue](https://github.com/phstc/shoryuken/wiki/Processing-Groups)
+- [Long Polling](https://github.com/phstc/shoryuken/wiki/Long-Polling)
+- [Batch processing](https://github.com/phstc/shoryuken/wiki/Worker-options#batch)
+- [Auto extend visibility timeout](https://github.com/phstc/shoryuken/wiki/Worker-options#auto_visibility_timeout)
+- [Exponential backoff](https://github.com/phstc/shoryuken/wiki/Worker-options#retry_intervals)
+- [Middleware support](https://github.com/phstc/shoryuken/wiki/Middleware)
+- Amazon SQS CLI. See `shoryuken help sqs`
 
 ## Requirements
 
@@ -43,102 +31,19 @@ Add this line to your application's Gemfile:
 gem 'shoryuken'
 ```
 
-Or to get the latest updates:
-
-```ruby
-gem 'shoryuken', github: 'phstc/shoryuken', branch: 'master'
-```
-
 And then execute:
 
-    $ bundle
-
-Or install it yourself as:
-
-    $ gem install shoryuken
-
-## Usage
-
-### Worker class
-
-```ruby
-class MyWorker
-  include Shoryuken::Worker
-
-  shoryuken_options queue: 'default', auto_delete: true
-  # shoryuken_options queue: ->{ "#{ENV['environment']}_default" }
-
-  # shoryuken_options body_parser: :json
-  # shoryuken_options body_parser: ->(sqs_msg){ REXML::Document.new(sqs_msg.body) }
-  # shoryuken_options body_parser: JSON
-
-  def perform(sqs_msg, body)
-    puts body
-  end
-end
-```
-
-[Check the Worker options documention](https://github.com/phstc/shoryuken/wiki/Worker-options).
-
-### Sending a message
-
-[Check the Sending a message documentation](https://github.com/phstc/shoryuken/wiki/Sending-a-message)
-
-### Middleware
-
-```ruby
-class MyMiddleware
-  def call(worker_instance, queue, sqs_msg, body)
-    puts 'Before work'
-    yield
-    puts 'After work'
-  end
-end
-```
-
-[Check the Middleware documentation](https://github.com/phstc/shoryuken/wiki/Middleware).
-
-### Shoryuken Configuration
-
-Sample configuration file `shoryuken.yml`.
-
-```yaml
-concurrency: 25  # The number of allocated threads to process messages. Default 25
-delay: 0         # The delay in seconds to pause empty queues. Default 0
-queues:
-  - [queue1, 1]
-  - [queue2, 1]
-  - [queue3, 1]
-```
-
-#### AWS Configuration
-
-[Check the Configure AWS Client documentation](https://github.com/phstc/shoryuken/wiki/Configure-the-AWS-Client)
-
-### Rails Integration
-
-[Check the Rails Integration Active Job documention](https://github.com/phstc/shoryuken/wiki/Rails-Integration-Active-Job).
-
-### Start Shoryuken
-
 ```shell
-bundle exec shoryuken -r worker.rb -C shoryuken.yml
+$ bundle
 ```
 
-For other options check `bundle exec shoryuken help start`
-
-#### SQS commands
-
-Check also some available SQS commands `bundle exec shoryuken help sqs`, such as:
+## Usage
 
-- `ls` list queues
-- `mv` move messages from one queue to another
-- `dump` dump messages from a queue into a JSON lines file
-- `requeue` requeue messages from a dump file
+Check the [Getting Started](https://github.com/phstc/shoryuken/wiki/Getting-Started) page.
 
 ## More Information
 
-For more information on advanced topics such as signals (shutdown), ActiveJob integration, and so on please check the [Shoryuken Wiki](https://github.com/phstc/shoryuken/wiki).
+For more information check the [wiki page](https://github.com/phstc/shoryuken/wiki).
 
 ## Credits
 

data/bin/cli/sqs.rb

@@ -33,7 +33,7 @@ module Shoryuken
               entries: batch.map { |message| { id: message.message_id, receipt_handle: message.receipt_handle } }
             ).failed.any? do |failure|
               say(
-                "Could not delete #{failure.id}, code: '#{failure.code}', message: '#{failure.message}', sender_fault: #{failure.sender_fault}",
+                "Could not delete #{failure.id}, code: #{failure.code}, message: #{failure.message}, sender_fault: #{failure.sender_fault}",
                 :yellow
               )
             end
@@ -185,6 +185,20 @@ module Shoryuken
 
         say "Purge request sent for #{queue_name}. The message deletion process takes up to 60 seconds", :yellow
       end
+
+      desc 'create QUEUE-NAME', 'Create a queue'
+      def create(queue_name)
+        queue_url = sqs.create_queue(queue_name: queue_name).queue_url
+
+        say "Queue #{queue_name} was successfully created. Queue URL #{queue_url}", :green
+      end
+
+      desc 'delete QUEUE-NAME', 'delete a queue'
+      def delete(queue_name)
+        sqs.delete_queue(queue_url: find_queue_url(queue_name))
+
+        say "Queue #{queue_name} was successfully delete", :green
+      end
     end
   end
 end

data/lib/shoryuken/default_worker_registry.rb

@@ -24,7 +24,7 @@ module Shoryuken
                        @workers[queue]
                      end
 
-      worker_class.new
+      worker_class.new if worker_class
     end
 
     def queues
@@ -34,7 +34,7 @@ module Shoryuken
     def register_worker(queue, clazz)
       if (worker_class = @workers[queue])
         if worker_class.get_shoryuken_options['batch'] == true || clazz.get_shoryuken_options['batch'] == true
-          fail ArgumentError, "Could not register #{clazz} for '#{queue}', "\
+          fail ArgumentError, "Could not register #{clazz} for #{queue}, "\
             "because #{worker_class} is already registered for this queue, "\
             "and Shoryuken doesn't support a batchable worker for a queue with multiple workers"
         end

data/lib/shoryuken/environment_loader.rb

@@ -24,7 +24,7 @@ module Shoryuken
     end
 
     def load
-      load_rails if options[:rails]
+      load_rails if Shoryuken.options[:rails]
       prefix_active_job_queue_names
       parse_queues
       require_workers
@@ -44,7 +44,11 @@ module Shoryuken
 
       fail ArgumentError, "The supplied config file #{path} does not exist" unless File.exist?(path)
 
-      YAML.load(ERB.new(IO.read(path)).result).deep_symbolize_keys
+      if result = YAML.load(ERB.new(IO.read(path)).result)
+        result.deep_symbolize_keys
+      else
+        {}
+      end
     end
 
     def initialize_logger
@@ -96,13 +100,24 @@ module Shoryuken
       end
     end
 
-    def parse_queue(queue, weight = nil)
-      Shoryuken.add_queue(queue, [weight.to_i, 1].max)
+    def parse_queue(queue, weight = nil, group)
+      Shoryuken.add_queue(queue, [weight.to_i, 1].max, group)
     end
 
     def parse_queues
-      Shoryuken.options[:queues].to_a.each do |queue_and_weight|
-        parse_queue(*queue_and_weight)
+      if Shoryuken.options[:queues].to_a.any?
+        Shoryuken.add_group('default', Shoryuken.options.fetch(:concurrency, 25))
+
+        Shoryuken.options[:queues].to_a.each do |queue, weight|
+          parse_queue(queue, weight, 'default')
+        end
+      end
+
+      Shoryuken.options[:groups].to_a.each do |group, options|
+        Shoryuken.add_group(group, options.fetch(:concurrency, 25))
+        options[:queues].to_a.each do |queue, weight|
+          parse_queue(queue, weight, group)
+        end
       end
     end
 
@@ -119,25 +134,30 @@ module Shoryuken
     end
 
     def validate_queues
-      return Shoryuken.logger.warn { 'No queues supplied' } if Shoryuken.queues.empty?
+      return Shoryuken.logger.warn { 'No queues supplied' } if Shoryuken.ungrouped_queues.empty?
 
       non_existent_queues = []
 
-      Shoryuken.queues.uniq.each do |queue|
+      Shoryuken.ungrouped_queues.uniq.each do |queue|
         begin
           Shoryuken::Client.queues(queue)
-        rescue Aws::SQS::Errors::NonExistentQueue
+        rescue Aws::Errors::NoSuchEndpointError, Aws::SQS::Errors::NonExistentQueue
           non_existent_queues << queue
         end
       end
 
-      fail ArgumentError, "The specified queue(s) #{non_existent_queues.join(', ')} do not exist" if non_existent_queues.any?
+      return if non_existent_queues.none?
+
+      fail(
+        ArgumentError,
+        "The specified queue(s) #{non_existent_queues.join(', ')} do not exist.\nTry 'shoryuken sqs create QUEUE-NAME' for creating a queue with default settings"
+      )
     end
 
     def validate_workers
       return if defined?(::ActiveJob)
 
-      all_queues = Shoryuken.queues
+      all_queues = Shoryuken.ungrouped_queues
       queues_with_workers = Shoryuken.worker_registry.queues
 
       (all_queues - queues_with_workers).each do |queue|

data/lib/shoryuken/fetcher.rb

@@ -4,23 +4,23 @@ module Shoryuken
 
     FETCH_LIMIT = 10
 
-    def fetch(queue, available_processors)
+    attr_reader :group
+
+    def initialize(group)
+      @group = group
+    end
+
+    def fetch(queue, limit)
       started_at = Time.now
 
       logger.debug { "Looking for new messages in #{queue}" }
 
-      begin
-        limit = available_processors > FETCH_LIMIT ? FETCH_LIMIT : available_processors
-
-        sqs_msgs = Array(receive_messages(queue, limit))
-        logger.info { "Found #{sqs_msgs.size} messages for #{queue.name}" } unless sqs_msgs.empty?
-        logger.debug { "Fetcher for #{queue} completed in #{elapsed(started_at)} ms" }
-        sqs_msgs
-      rescue => ex
-        logger.error { "Error fetching message: #{ex.message}" }
-        logger.error { ex.backtrace.join("\n") } unless ex.backtrace.nil?
-        []
-      end
+      sqs_msgs = Array(receive_messages(queue, [FETCH_LIMIT, limit].min))
+
+      logger.info { "Found #{sqs_msgs.size} messages for #{queue.name}" } unless sqs_msgs.empty?
+      logger.debug { "Fetcher for #{queue} completed in #{elapsed(started_at)} ms" }
+
+      sqs_msgs
     end
 
     private
@@ -29,10 +29,11 @@ module Shoryuken
       # AWS limits the batch size by 10
       limit = limit > FETCH_LIMIT ? FETCH_LIMIT : limit
 
-      options = Shoryuken.sqs_client_receive_message_opts.to_h.dup
-      options[:max_number_of_messages] = limit
+      options = Shoryuken.sqs_client_receive_message_opts[group].to_h.dup
+
+      options[:max_number_of_messages]  = limit
       options[:message_attribute_names] = %w(All)
-      options[:attribute_names] = %w(All)
+      options[:attribute_names]         = %w(All)
 
       options.merge!(queue.options)
 

data/lib/shoryuken/launcher.rb

@@ -3,17 +3,96 @@ module Shoryuken
     include Util
 
     def initialize
-      @manager = Shoryuken::Manager.new(Shoryuken::Fetcher.new,
-                                        Shoryuken.options[:polling_strategy].new(Shoryuken.queues))
+      @managers    = create_managers
+      @shutdowning = Concurrent::AtomicBoolean.new(false)
     end
 
-    def stop(options = {})
-      @manager.stop(shutdown: !options[:shutdown].nil?,
-                    timeout: Shoryuken.options[:timeout])
+    def start
+      logger.info { 'Starting' }
+
+      start_callback
+      start_managers
+    end
+
+    def stop!
+      initiate_stop
+
+      executor.shutdown
+
+      return if executor.wait_for_termination(Shoryuken.options[:timeout])
+
+      executor.kill
+    end
+
+    def stop
+      fire_event(:quiet, true)
+
+      initiate_stop
+
+      executor.shutdown
+      executor.wait_for_termination
+    end
+
+    private
+
+    def executor
+      Concurrent.global_io_executor
+    end
+
+    def start_managers
+      @managers.each do |manager|
+        Concurrent::Promise.execute { manager.start }.rescue do |ex|
+          log_manager_failure(ex)
+          start_soft_shutdown
+        end
+      end
+    end
+
+    def start_soft_shutdown
+      Process.kill('USR1', Process.pid) if @shutdowning.make_true
+    end
+
+    def log_manager_failure(ex)
+      return unless ex
+
+      logger.error { "Manager failed: #{ex.message}" }
+      logger.error { ex.backtrace.join("\n") } unless ex.backtrace.nil?
+    end
+
+    def initiate_stop
+      logger.info { 'Shutting down' }
+
+      @managers.each(&:stop)
+
+      stop_callback
+    end
+
+    def start_callback
+      if (callback = Shoryuken.start_callback)
+        logger.debug { 'Calling start_callback' }
+        callback.call
+      end
+
+      fire_event(:startup)
+    end
+
+    def stop_callback
+      if (callback = Shoryuken.stop_callback)
+        logger.debug { 'Calling stop_callback' }
+        callback.call
+      end
+
+      fire_event(:shutdown, true)
     end
 
-    def run
-      @manager.start
+    def create_managers
+      Shoryuken.groups.map do |group, options|
+        Shoryuken::Manager.new(
+          Shoryuken::Fetcher.new(group),
+          Shoryuken.polling_strategy(group).new(options[:queues]),
+          options[:concurrency]
+        )
+      end
     end
   end
 end

data/lib/shoryuken/manager.rb

@@ -6,95 +6,71 @@ module Shoryuken
     # See https://github.com/phstc/shoryuken/issues/348#issuecomment-292847028
     MIN_DISPATCH_INTERVAL = 0.1
 
-    def initialize(fetcher, polling_strategy)
-      @count = Shoryuken.options.fetch(:concurrency, 25)
-
-      raise(ArgumentError, "Concurrency value #{@count} is invalid, it needs to be a positive number") unless @count > 0
-
-      @queues = Shoryuken.queues.dup.uniq
-
-      @done = Concurrent::AtomicBoolean.new(false)
-
-      @fetcher = fetcher
+    def initialize(fetcher, polling_strategy, concurrency)
+      @fetcher          = fetcher
       @polling_strategy = polling_strategy
-
-      @pool = Concurrent::FixedThreadPool.new(@count, max_queue: @count)
-      @dispatcher_executor = Concurrent::SingleThreadExecutor.new
+      @max_processors   = concurrency
+      @busy_processors  = Concurrent::AtomicFixnum.new(0)
+      @done             = Concurrent::AtomicBoolean.new(false)
     end
 
     def start
-      logger.info { 'Starting' }
-
-      dispatch_async
+      dispatch
     end
 
-    def stop(options = {})
+    def stop
       @done.make_true
-
-      if (callback = Shoryuken.stop_callback)
-        logger.info { 'Calling on_stop callback' }
-        callback.call
-      end
-
-      fire_event(:shutdown, true)
-
-      logger.info { 'Shutting down workers' }
-
-      @dispatcher_executor.kill
-
-      if options[:shutdown]
-        hard_shutdown_in(options[:timeout])
-      else
-        soft_shutdown
-      end
     end
 
-    def processor_failed(ex)
-      logger.error { "Processor failed: #{ex.message}" }
-      logger.error { ex.backtrace.join("\n") } unless ex.backtrace.nil?
-    end
+    private
 
-    def processor_done(queue)
-      logger.debug { "Process done for #{queue}" }
+    def stopped?
+      @done.true? || !Concurrent.global_io_executor.running?
     end
 
-    private
+    def dispatch
+      return if stopped?
 
-    def dispatch_async
-      @dispatcher_executor.post(&method(:dispatch_now))
-    end
+      if !ready.positive? || (queue = @polling_strategy.next_queue).nil?
+        return dispatch_later
+      end
 
-    def dispatch_now
-      return if @done.true?
+      fire_event(:dispatch)
 
-      begin
-        if ready.zero? || (queue = @polling_strategy.next_queue).nil?
-          sleep MIN_DISPATCH_INTERVAL
-          return
-        end
+      logger.info { "Ready: #{ready}, Busy: #{busy}, Active Queues: #{@polling_strategy.active_queues}" }
 
-        fire_event(:dispatch)
+      batched_queue?(queue) ? dispatch_batch(queue) : dispatch_single_messages(queue)
 
-        logger.debug { "Ready: #{ready}, Busy: #{busy}, Active Queues: #{@polling_strategy.active_queues}" }
+      dispatch
+    end
 
-        batched_queue?(queue) ? dispatch_batch(queue) : dispatch_single_messages(queue)
-      ensure
-        dispatch_async
-      end
+    def dispatch_later
+      sleep(MIN_DISPATCH_INTERVAL)
+      dispatch
     end
 
     def busy
-      @count - ready
+      @busy_processors.value
     end
 
     def ready
-      @pool.remaining_capacity
+      @max_processors - busy
     end
 
-    def assign(queue, sqs_msg)
+    def processor_done
+      @busy_processors.decrement
+    end
+
+    def assign(queue_name, sqs_msg)
+      return if stopped?
+
       logger.debug { "Assigning #{sqs_msg.message_id}" }
 
-      @pool.post { Processor.new(self).process(queue, sqs_msg) }
+      @busy_processors.increment
+
+      Concurrent::Promise.execute {
+        Processor.new(queue_name, sqs_msg).process
+      }.then { processor_done }.rescue { processor_done }
     end
 
     def dispatch_batch(queue)
@@ -105,6 +81,7 @@ module Shoryuken
 
     def dispatch_single_messages(queue)
       messages = @fetcher.fetch(queue, ready)
+
       @polling_strategy.messages_found(queue.name, messages.size)
       messages.each { |message| assign(queue.name, message) }
     end
@@ -113,24 +90,6 @@ module Shoryuken
       Shoryuken.worker_registry.batch_receive_messages?(queue.name)
     end
 
-    def soft_shutdown
-      @pool.shutdown
-      @pool.wait_for_termination
-    end
-
-    def hard_shutdown_in(delay)
-      if busy > 0
-        logger.info { "Pausing up to #{delay} seconds to allow workers to finish..." }
-      end
-
-      @pool.shutdown
-
-      return if @pool.wait_for_termination(delay)
-
-      logger.info { "Hard shutting down #{busy} busy workers" }
-      @pool.kill
-    end
-
     def patch_batch!(sqs_msgs)
       sqs_msgs.instance_eval do
         def message_id