shoryuken 1.0.2 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: c8c88a264b5df0c11cbcc440790224279c70176e
-  data.tar.gz: 57f9b92ee36f20d9cb385ea3b5a538f0bfd42f0d
+  metadata.gz: ee01f5dc2129537f640aba700e43c8be0f37e77c
+  data.tar.gz: 54c1fe1a997a29a77939336ca5f8f2c7e691a745
 SHA512:
-  metadata.gz: e8105e79af153637fb922ee9768aee12ddd69a9bb251229975f0aa54e19f277b9b06c3cbb1ee54d3ff2f865aef6adeeab0389a6e2c40225ceaa6714ae63ee527
-  data.tar.gz: 10e3cb7ffdab9d525bb2db06021e319ef56b18bdb81ce795d548f3a3f17959887df69cb9e642046f579d2040870ef1d8fdcc1e28a6256b4fbc5331eebc966b88
+  metadata.gz: 5543a21bcfadc457a3bb7444b27ba97d59c9f238bbc9633a100555154823132f7ee6f3cf233f84743ed553cc69a27f8ddf836515690a120b291ffb8212db8783
+  data.tar.gz: bcdc762ee9e02366ad4194e372ad76e2e7c23f7e5f15129d8c90d6abfa65af337c4900addaf47d14f8f6578e084a6fd84dad478878589e0c3e2912c5851c89f5

data/CHANGELOG.md

@@ -0,0 +1,11 @@
+## [v2.0.0] - 2015-09-22
+
+- Allow configuration of SQS/SNS endpoints via environment variables
+ - [#130](https://github.com/phstc/shoryuken/pull/130)
+
+- Expose queue_name in the message object
+  - [#127](https://github.com/phstc/shoryuken/pull/127)
+
+- README updates
+  - [#122](https://github.com/phstc/shoryuken/pull/122)
+  - [#120](https://github.com/phstc/shoryuken/pull/120)

data/Gemfile

@@ -3,4 +3,7 @@ source 'https://rubygems.org'
 # Specify your gem's dependencies in shoryuken.gemspec
 gemspec
 
-gem 'codeclimate-test-reporter', group: :test, require: nil
+group :test do
+  gem 'codeclimate-test-reporter', require: nil
+  gem 'multi_xml'
+end

data/README.md

@@ -36,7 +36,7 @@ If all queues get empty, all processors will be changed to the waiting state and
 
 ### Fetch in batches
 
-To be even more performance and cost efficient, Shoryuken fetches SQS messages in batches, so a single SQS request can fetch up to 10 messages.
+To be even more performant and cost effective, Shoryuken fetches SQS messages in batches, so a single SQS request can fetch up to 10 messages.
 
 ## Requirements
 
@@ -114,7 +114,7 @@ aws:
   access_key_id:      ...       # or <%= ENV['AWS_ACCESS_KEY_ID'] %>
   secret_access_key:  ...       # or <%= ENV['AWS_SECRET_ACCESS_KEY'] %>
   region:             us-east-1 # or <%= ENV['AWS_REGION'] %>
-  receive_message:              # See http://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/Queue.html#receive_messages-instance_method
+  receive_message:              # See http://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/Client.html#receive_message-instance_method
     # wait_time_seconds: N      # The number of seconds to wait for new messages when polling. Defaults to the #wait_time_seconds defined on the queue
     attribute_names:
       - ApproximateReceiveCount
@@ -132,59 +132,13 @@ The ```aws``` section is used to configure both the Aws objects used by Shoryuke
 - ```account_id``` is used when generating SNS ARNs
 - ```sns_endpoint``` can be used to explicitly override the SNS endpoint
 - ```sqs_endpoint``` can be used to explicitly override the SQS endpoint
-- ```receive_message``` can be used to define the options passed to the http://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/Queue.html#receive_messages-instance_method
+- ```receive_message``` can be used to define the options passed to the http://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/Client.html#receive_message-instance_method
 
-### Rails Integration
-
-You can tell Shoryuken to load your Rails application by passing the `-R` or `--rails` flag to the "shoryuken" command.
-
-If you load Rails, and assuming your workers are located in the `app/workers` directory, they will be auto-loaded. This means you don't need to require them explicitly with `-r`.
-
-For middleware and other configuration, you might want to create an initializer:
-
-```ruby
-Shoryuken.configure_server do |config|
-  # Replace Rails logger so messages are logged wherever Shoryuken is logging
-  # Note: this entire block is only run by the processor, so we don't overwrite
-  #       the logger when the app is running as usual.
-  Rails.logger = Shoryuken::Logging.logger
-
-  config.server_middleware do |chain|
-    chain.add Shoryuken::MyMiddleware
-  end
-end
-```
-
-*Note:* In the above case, since we are replacing the Rails logger, it's desired that this initializer runs before other initializers (in case they themselves use the logger). Since by Rails conventions initializers are executed in alphabetical order, this can be achieved by prepending the initializer filename with `00_` (assuming no other initializers alphabetically precede this one).
-
-This feature works for Rails 4+, but needs to be confirmed for older versions.
-
-#### ActiveJob Support
+The ```sns_endpoint``` and ```sqs_endpoint``` Shoryuken-specific options will also fallback to the environment variables ```AWS_SNS_ENDPOINT``` and ```AWS_SQS_ENDPOINT``` respectively, if they are set.
 
-Yes, Shoryuken supports ActiveJob! This means that you can put your jobs in processor-agnostic `ActiveJob::Base` subclasses, and change processors whenever you want (or better yet, switch to Shoryuken from another processor easily!).
-
-It works as expected. Just put your job in `app/jobs`. Here's an example:
-
-```ruby
-class ProcessPhotoJob < ActiveJob::Base
-  queue_as :default
-
-  rescue_from ActiveJob::DeserializationError do |e|
-    Shoryuken.logger.error ex
-    Shoryuken.logger.error ex.backtrace.join("\n")
-  end
-
-  def perform(photo)
-    photo.process_image!
-  end
-end
-```
-
-Delayed mailers, ActiveRecord serialization, etc. all work.
-
-See [ActiveJob docs](http://edgeguides.rubyonrails.org/active_job_basics.html) for more info.
+### Rails Integration
 
-*Note:* When queueing jobs to be performed in the future (e.g when setting the `wait` or `wait_until` ActiveJob options), SQS limits the amount of time to 15 minutes. Shoryuken will raise an exception if you attempt to schedule a job further into the future than this limit.
+[Check the Rails Integration Active Job documention](https://github.com/phstc/shoryuken/wiki/Rails-Integration-Active-Job).
 
 ### Start Shoryuken
 

data/lib/shoryuken/cli.rb

@@ -41,7 +41,7 @@ module Shoryuken
       @launcher = Shoryuken::Launcher.new
 
       if callback = Shoryuken.start_callback
-        logger.info "Calling Shoryuken.on_start block"
+        logger.info { 'Calling Shoryuken.on_start block' }
         callback.call
       end
 
@@ -158,7 +158,7 @@ module Shoryuken
 
       @parser.banner = 'shoryuken [options]'
       @parser.on_tail '-h', '--help', 'Show help' do
-        logger.info @parser
+        logger.info { @parser }
         exit 1
       end
       @parser.parse!(argv)
@@ -166,22 +166,22 @@ module Shoryuken
     end
 
     def handle_signal(sig)
-      logger.info "Got #{sig} signal"
+      logger.info { "Got #{sig} signal" }
 
       case sig
       when 'USR1'
-        logger.info 'Received USR1, will soft shutdown down'
+        logger.info { 'Received USR1, will soft shutdown down' }
 
         launcher.stop
 
         exit 0
       when 'TTIN'
         Thread.list.each do |thread|
-          logger.info "Thread TID-#{thread.object_id.to_s(36)} #{thread['label']}"
+          logger.info { "Thread TID-#{thread.object_id.to_s(36)} #{thread['label']}" }
           if thread.backtrace
-            logger.info thread.backtrace.join("\n")
+            logger.info { thread.backtrace.join("\n") }
           else
-            logger.info '<no backtrace available>'
+            logger.info { '<no backtrace available>' }
           end
         end
 
@@ -189,9 +189,9 @@ module Shoryuken
         busy   = launcher.manager.instance_variable_get(:@busy).size
         queues = launcher.manager.instance_variable_get(:@queues)
 
-        logger.info "Ready: #{ready}, Busy: #{busy}, Active Queues: #{unparse_queues(queues)}"
+        logger.info { "Ready: #{ready}, Busy: #{busy}, Active Queues: #{unparse_queues(queues)}" }
       else
-        logger.info "Received #{sig}, will shutdown down"
+        logger.info { "Received #{sig}, will shutdown down" }
 
         raise Interrupt
       end

data/lib/shoryuken/client.rb

@@ -30,7 +30,8 @@ module Shoryuken
       private
 
       def aws_client_options(service_endpoint_key)
-        explicit_endpoint = Shoryuken.options[:aws][service_endpoint_key]
+        environment_endpoint = ENV["AWS_#{service_endpoint_key.to_s.upcase}"]
+        explicit_endpoint = Shoryuken.options[:aws][service_endpoint_key] || environment_endpoint
         options = {}
         options[:endpoint] = explicit_endpoint unless explicit_endpoint.to_s.empty?
         options

data/lib/shoryuken/environment_loader.rb

@@ -19,6 +19,7 @@ module Shoryuken
       load_rails if options[:rails]
       Shoryuken.options.merge!(config_file_options)
       merge_cli_defined_queues
+      prefix_active_job_queue_names
       Shoryuken.options.merge!(options)
       parse_queues
       require_workers
@@ -33,7 +34,7 @@ module Shoryuken
     def config_file_options
       if (path = options[:config_file])
         unless File.exist?(path)
-          Shoryuken.logger.warn "Config file #{path} does not exist"
+          Shoryuken.logger.warn { "Config file #{path} does not exist" }
           path = nil
         end
       end
@@ -48,11 +49,11 @@ module Shoryuken
       # when not explicit supplied
       return if Shoryuken.options[:aws].empty?
 
-      shoryuken_keys = %i(
+      shoryuken_keys = %w(
         account_id
         sns_endpoint
         sqs_endpoint
-        receive_message)
+        receive_message).map(&:to_sym)
 
       aws_options = Shoryuken.options[:aws].reject do |k, v|
         shoryuken_keys.include?(k)
@@ -67,7 +68,7 @@ module Shoryuken
       aws_options = aws_options.merge(credentials: credentials) if credentials.set?
 
       if (callback = Shoryuken.aws_initialization_callback)
-        Shoryuken.logger.info 'Calling Shoryuken.on_aws_initialization block'
+        Shoryuken.logger.info { 'Calling Shoryuken.on_aws_initialization block' }
         callback.call(aws_options)
       end
 
@@ -96,7 +97,7 @@ module Shoryuken
         require File.expand_path('config/environment.rb')
       end
 
-      Shoryuken.logger.info 'Rails environment loaded'
+      Shoryuken.logger.info { 'Rails environment loaded' }
     end
 
     def merge_cli_defined_queues
@@ -111,6 +112,21 @@ module Shoryuken
       end
     end
 
+    def prefix_active_job_queue_names
+      return unless @options[:rails]
+      return unless Shoryuken.active_job_queue_name_prefixing
+
+      queue_name_prefix = ::ActiveJob::Base.queue_name_prefix
+      queue_name_delimiter = ::ActiveJob::Base.queue_name_delimiter
+
+      # See https://github.com/rails/rails/blob/master/activejob/lib/active_job/queue_name.rb#L27
+      Shoryuken.options[:queues].to_a.map! do |queue_name, weight|
+        name_parts = [queue_name_prefix.presence, queue_name]
+        prefixed_queue_name = name_parts.compact.join(queue_name_delimiter)
+        [prefixed_queue_name, weight]
+      end
+    end
+
     def parse_queue(queue, weight = nil)
       [weight.to_i, 1].max.times { Shoryuken.queues << queue }
     end
@@ -125,7 +141,7 @@ module Shoryuken
       Shoryuken.worker_registry.queues.each do |queue|
         Shoryuken.worker_registry.workers(queue).each do |worker_class|
           if worker_class.instance_method(:perform).arity == 1
-            Shoryuken.logger.warn "[DEPRECATION] #{worker_class.name}#perform(sqs_msg) is deprecated. Please use #{worker_class.name}#perform(sqs_msg, body)"
+            Shoryuken.logger.warn { "[DEPRECATION] #{worker_class.name}#perform(sqs_msg) is deprecated. Please use #{worker_class.name}#perform(sqs_msg, body)" }
 
             worker_class.class_eval do
               alias_method :deprecated_perform, :perform
@@ -144,13 +160,13 @@ module Shoryuken
     end
 
     def validate_queues
-      Shoryuken.logger.warn 'No queues supplied' if Shoryuken.queues.empty?
+      Shoryuken.logger.warn { 'No queues supplied' } if Shoryuken.queues.empty?
 
       Shoryuken.queues.uniq.each do |queue|
         begin
           Shoryuken::Client.queues queue
         rescue Aws::SQS::Errors::NonExistentQueue
-          Shoryuken.logger.warn "AWS Queue '#{queue}' does not exist"
+          Shoryuken.logger.warn { "AWS Queue '#{queue}' does not exist" }
         end
       end
     end
@@ -161,7 +177,7 @@ module Shoryuken
 
       unless defined?(::ActiveJob)
         (all_queues - queues_with_workers).each do |queue|
-          Shoryuken.logger.warn "No worker supplied for '#{queue}'"
+          Shoryuken.logger.warn { "No worker supplied for '#{queue}'" }
         end
       end
     end

data/lib/shoryuken/fetcher.rb

@@ -13,7 +13,7 @@ module Shoryuken
       # AWS limits the batch size by 10
       limit = limit > FETCH_LIMIT ? FETCH_LIMIT : limit
 
-      options = Shoryuken.options[:aws][:receive_message].to_h.dup
+      options = (Shoryuken.options[:aws][:receive_message] || {}).dup
       options[:max_number_of_messages] = limit
       options[:message_attribute_names] = %w(All)
       options[:attribute_names] = %w(All)
@@ -25,14 +25,14 @@ module Shoryuken
       watchdog('Fetcher#fetch died') do
         started_at = Time.now
 
-        logger.debug "Looking for new messages in '#{queue}'"
+        logger.debug { "Looking for new messages in '#{queue}'" }
 
         begin
           batch = Shoryuken.worker_registry.batch_receive_messages?(queue)
           limit = batch ? FETCH_LIMIT : available_processors
 
           if (sqs_msgs = Array(receive_messages(queue, limit))).any?
-            logger.info "Found #{sqs_msgs.size} messages for '#{queue}'"
+            logger.info { "Found #{sqs_msgs.size} messages for '#{queue}'" }
 
             if batch
               @manager.async.assign(queue, patch_sqs_msgs!(sqs_msgs))
@@ -42,7 +42,7 @@ module Shoryuken
 
             @manager.async.rebalance_queue_weight!(queue)
           else
-            logger.debug "No message found for '#{queue}'"
+            logger.debug { "No message found for '#{queue}'" }
 
             @manager.async.pause_queue!(queue)
           end
@@ -51,8 +51,8 @@ module Shoryuken
 
           logger.debug { "Fetcher for '#{queue}' completed in #{elapsed(started_at)} ms" }
         rescue => ex
-          logger.error "Error fetching message: #{ex}"
-          logger.error ex.backtrace.first
+          logger.error { "Error fetching message: #{ex}" }
+          logger.error { ex.backtrace.first }
 
           @manager.async.dispatch
         end

data/lib/shoryuken/launcher.rb

@@ -36,7 +36,7 @@ module Shoryuken
 
     def actor_died(actor, reason)
       return if @done
-      logger.warn 'Shoryuken died due to the following error, cannot recover, process exiting'
+      logger.warn { 'Shoryuken died due to the following error, cannot recover, process exiting' }
       exit 1
     end
   end

data/lib/shoryuken/manager.rb

@@ -23,7 +23,7 @@ module Shoryuken
     end
 
     def start
-      logger.info 'Starting'
+      logger.info { 'Starting' }
 
       dispatch
     end
@@ -33,7 +33,7 @@ module Shoryuken
         @done = true
 
         if (callback = Shoryuken.stop_callback)
-          logger.info 'Calling Shoryuken.on_stop block'
+          logger.info { 'Calling Shoryuken.on_stop block' }
           callback.call
         end
 
@@ -58,7 +58,7 @@ module Shoryuken
 
     def processor_done(queue, processor)
       watchdog('Manager#processor_done died') do
-        logger.info "Process done for '#{queue}'"
+        logger.debug { "Process done for '#{queue}'" }
 
         @threads.delete(processor.object_id)
         @busy.delete processor
@@ -73,7 +73,7 @@ module Shoryuken
 
     def processor_died(processor, reason)
       watchdog("Manager#processor_died died") do
-        logger.error "Process died, reason: #{reason}" unless reason.to_s.empty?
+        logger.error { "Process died, reason: #{reason}" unless reason.to_s.empty? }
 
         @threads.delete(processor.object_id)
         @busy.delete processor
@@ -90,7 +90,7 @@ module Shoryuken
 
     def assign(queue, sqs_msg)
       watchdog('Manager#assign died') do
-        logger.info "Assigning #{sqs_msg.message_id}"
+        logger.debug { "Assigning #{sqs_msg.message_id}" }
 
         processor = @ready.pop
         @busy << processor
@@ -102,7 +102,7 @@ module Shoryuken
     def rebalance_queue_weight!(queue)
       watchdog('Manager#rebalance_queue_weight! died') do
         if (original = original_queue_weight(queue)) > (current = current_queue_weight(queue))
-          logger.info "Increasing '#{queue}' weight to #{current + 1}, max: #{original}"
+          logger.info { "Increasing '#{queue}' weight to #{current + 1}, max: #{original}" }
 
           @queues << queue
         end
@@ -112,7 +112,7 @@ module Shoryuken
     def pause_queue!(queue)
       return if !@queues.include?(queue) || Shoryuken.options[:delay].to_f <= 0
 
-      logger.debug "Pausing '#{queue}' for #{Shoryuken.options[:delay].to_f} seconds, because it's empty"
+      logger.debug { "Pausing '#{queue}' for #{Shoryuken.options[:delay].to_f} seconds, because it's empty" }
 
       @queues.delete(queue)
 
@@ -158,7 +158,7 @@ module Shoryuken
       return if stopped?
 
       unless @queues.include? queue
-        logger.debug "Restarting '#{queue}'"
+        logger.debug { "Restarting '#{queue}'" }
 
         @queues << queue
 
@@ -192,7 +192,7 @@ module Shoryuken
 
       unless defined?(::ActiveJob) ||  !Shoryuken.worker_registry.workers(queue).empty?
         # when no worker registered pause the queue to avoid endless recursion
-        logger.debug "Pausing '#{queue}' for #{Shoryuken.options[:delay].to_f} seconds, because no workers registered"
+        logger.debug { "Pausing '#{queue}' for #{Shoryuken.options[:delay].to_f} seconds, because no workers registered" }
 
         after(Shoryuken.options[:delay].to_f) { async.restart_queue!(queue) }
 

data/lib/shoryuken/message.rb

@@ -1,11 +1,21 @@
 module Shoryuken
   class Message
-    attr_accessor :client, :queue_url, :data
+    attr_accessor :client, :queue_url, :queue_name, :data
 
-    def initialize(client, queue_url, data)
+    def initialize(client, queue, data)
       self.client = client
-      self.queue_url = queue_url
       self.data = data
+
+      if queue.is_a?(Shoryuken::Queue)
+        self.queue_url = queue.url
+        self.queue_name = queue.name
+      else
+        # TODO: Remove next major release
+        Shoryuken.loggger.warn do
+          '[DEPRECATION] Passing a queue url into Shoryuken::Message is deprecated, please pass the queue itself'
+        end
+        self.queue_url = queue
+      end
     end
 
     def delete

data/lib/shoryuken/middleware/chain.rb

@@ -117,7 +117,7 @@ module Shoryuken
 
       def patch_deprecated_middleware!(klass)
         if klass.instance_method(:call).arity == 3
-          Shoryuken.logger.warn "[DEPRECATION] #{klass.name}#call(worker_instance, queue, sqs_msg) is deprecated. Please use #{klass.name}#call(worker_instance, queue, sqs_msg, body)"
+          Shoryuken.logger.warn { "[DEPRECATION] #{klass.name}#call(worker_instance, queue, sqs_msg) is deprecated. Please use #{klass.name}#call(worker_instance, queue, sqs_msg, body)" }
 
           klass.class_eval do
             alias_method :deprecated_call, :call

data/lib/shoryuken/middleware/server/exponential_backoff_retry.rb

@@ -42,7 +42,7 @@ module Shoryuken
 
           sqs_msg.change_visibility(visibility_timeout: interval.to_i)
 
-          logger.info "Message #{sqs_msg.message_id} failed, will be retried in #{interval} seconds."
+          logger.info { "Message #{sqs_msg.message_id} failed, will be retried in #{interval} seconds." }
         end
       end
     end

data/lib/shoryuken/middleware/server/timing.rb

@@ -9,19 +9,19 @@ module Shoryuken
             begin
               started_at = Time.now
 
-              logger.info "started at #{started_at}"
+              logger.info { "started at #{started_at}" }
 
               yield
 
               total_time = elapsed(started_at)
 
               if (total_time / 1000.0) > (timeout = Shoryuken::Client.queues(queue).visibility_timeout)
-                logger.warn "exceeded the queue visibility timeout by #{total_time - (timeout * 1000)} ms"
+                logger.warn { "exceeded the queue visibility timeout by #{total_time - (timeout * 1000)} ms" }
               end
 
-              logger.info "completed in: #{total_time} ms"
+              logger.info { "completed in: #{total_time} ms" }
             rescue => e
-              logger.info "failed in: #{elapsed(started_at)} ms"
+              logger.info { "failed in: #{elapsed(started_at)} ms" }
               raise e
             end
           end

data/lib/shoryuken/processor.rb

@@ -19,12 +19,10 @@ module Shoryuken
       timer = auto_visibility_timeout(queue, sqs_msg, worker.class)
 
       begin
-        defer do
-          body = get_body(worker.class, sqs_msg)
+        body = get_body(worker.class, sqs_msg)
 
-          worker.class.server_middleware.invoke(worker, queue, sqs_msg, body) do
-            worker.perform(sqs_msg, body)
-          end
+        worker.class.server_middleware.invoke(worker, queue, sqs_msg, body) do
+          worker.perform(sqs_msg, body)
         end
 
         @manager.async.processor_done(queue, current_actor)
@@ -35,22 +33,31 @@ module Shoryuken
 
     private
 
-    def auto_visibility_timeout(queue, sqs_msg, worker_class)
-      if worker_class.auto_visibility_timeout?
+    class MessageVisibilityExtender
+      include Celluloid
+      include Util
+
+      def auto_extend(queue, sqs_msg, worker_class)
         queue_visibility_timeout = Shoryuken::Client.queues(queue).visibility_timeout
 
-        timer = every(queue_visibility_timeout - 5) do
+        every(queue_visibility_timeout - 5) do
           begin
-            logger.debug "Extending message #{worker_name(worker_class, sqs_msg)}/#{queue}/#{sqs_msg.message_id} visibility timeout by #{queue_visibility_timeout}s."
+            logger.debug { "Extending message #{worker_name(worker_class, sqs_msg)}/#{queue}/#{sqs_msg.message_id} visibility timeout by #{queue_visibility_timeout}s." }
 
             sqs_msg.visibility_timeout = queue_visibility_timeout
           rescue => e
-            logger.error "Could not auto extend the message #{worker_class}/#{queue}/#{sqs_msg.message_id} visibility timeout. Error: #{e.message}"
+            logger.error { "Could not auto extend the message #{worker_class}/#{queue}/#{sqs_msg.message_id} visibility timeout. Error: #{e.message}" }
           end
         end
       end
+    end
 
-      timer
+    def auto_visibility_timeout(queue, sqs_msg, worker_class)
+      return unless worker_class.auto_visibility_timeout?
+
+      @visibility_extender ||= MessageVisibilityExtender.new_link
+
+      @visibility_extender.auto_extend(queue, sqs_msg, worker_class)
     end
 
     def get_body(worker_class, sqs_msg)
@@ -72,11 +79,18 @@ module Shoryuken
       when :text, nil
         sqs_msg.body
       else
-        body_parser.parse(sqs_msg.body) if body_parser.respond_to?(:parse) # i.e. JSON.parse(...)
+        if body_parser.respond_to?(:parse)
+          # JSON.parse
+          body_parser.parse(sqs_msg.body)
+        elsif body_parser.respond_to?(:load)
+          # see https://github.com/phstc/shoryuken/pull/91
+          # JSON.load
+          body_parser.load(sqs_msg.body)
+        end
       end
     rescue => e
-      logger.error "Error parsing the message body: #{e.message}\nbody_parser: #{body_parser}\nsqs_msg.body: #{sqs_msg.body}"
-      nil
+      logger.error { "Error parsing the message body: #{e.message}\nbody_parser: #{body_parser}\nsqs_msg.body: #{sqs_msg.body}" }
+      raise
     end
   end
 end

data/lib/shoryuken/queue.rb

@@ -20,31 +20,62 @@ module Shoryuken
     end
 
     def send_message(options)
-      client.send_message(sanitize_message_body(options.merge(queue_url: url)))
+      options = sanitize_message!(options).merge(queue_url: url)
+
+      Shoryuken.client_middleware.invoke(options) do
+        client.send_message(options)
+      end
     end
 
     def send_messages(options)
-      client.send_message_batch(sanitize_message_body(options.merge(queue_url: url)))
+      client.send_message_batch(sanitize_messages!(options).merge(queue_url: url))
     end
 
     def receive_messages(options)
       client.receive_message(options.merge(queue_url: url)).
         messages.
-        map { |m| Message.new(client, url, m) }
+        map { |m| Message.new(client, self, m) }
     end
 
     private
 
-    def sanitize_message_body(options)
-      messages = options[:entries] || [options]
+    def sanitize_messages!(options)
+      options = case
+                when options.is_a?(Array)
+                  { entries: options.map.with_index { |m, index| { id: index.to_s, message_body: m } } }
+                when options.is_a?(Hash)
+                  options
+                end
+
+      validate_messages!(options)
+
+      options
+    end
+
+    def sanitize_message!(options)
+      options = case
+                when options.is_a?(String)
+                  # send_message('message')
+                  { message_body: options }
+                when options.is_a?(Hash)
+                  options
+                end
+
+      validate_message!(options)
+
+      options
+    end
+
+    def validate_messages!(options)
+      options[:entries].map { |m| validate_message!(m) }
+    end
 
-      messages.each do |m|
-        body = m[:message_body]
-        if body.is_a?(Hash)
-          m[:message_body] = JSON.dump(body)
-        elsif !body.is_a? String
-          fail ArgumentError, "The message body must be a String and you passed a #{body.class}"
-        end
+    def validate_message!(options)
+      body = options[:message_body]
+      if body.is_a?(Hash)
+        options[:message_body] = JSON.dump(body)
+      elsif !body.is_a?(String)
+        fail ArgumentError, "The message body must be a String and you passed a #{body.class}"
       end
 
       options

data/lib/shoryuken/util.rb

@@ -3,9 +3,9 @@ module Shoryuken
     def watchdog(last_words)
       yield
     rescue => ex
-      logger.error last_words
-      logger.error ex
-      logger.error ex.backtrace.join("\n")
+      logger.error { last_words }
+      logger.error { ex }
+      logger.error { ex.backtrace.join("\n") }
     end
 
     def logger
@@ -30,7 +30,7 @@ module Shoryuken
           && sqs_msg.message_attributes['shoryuken_class'] \
           && sqs_msg.message_attributes['shoryuken_class'][:string_value] == ActiveJob::QueueAdapters::ShoryukenAdapter::JobWrapper.to_s
 
-        "ActiveJob/#{body['job_class'].constantize}"
+        "ActiveJob/#{body['job_class']}"
       else
         worker_class.to_s
       end

data/lib/shoryuken/version.rb

@@ -1,3 +1,3 @@
 module Shoryuken
-  VERSION = '1.0.2'
+  VERSION = '2.0.0'
 end

data/lib/shoryuken/worker.rb

@@ -39,7 +39,7 @@ module Shoryuken
       end
 
       def shoryuken_options(opts = {})
-        @shoryuken_options = get_shoryuken_options.merge(stringify_keys(Hash(opts)))
+        @shoryuken_options = get_shoryuken_options.merge(stringify_keys(opts || {}))
         queue = @shoryuken_options['queue']
         if queue.respond_to? :call
           queue = queue.call

data/lib/shoryuken.rb

@@ -29,8 +29,9 @@ module Shoryuken
     timeout: 8
   }
 
-  @@queues          = []
+  @@queues = []
   @@worker_registry = DefaultWorkerRegistry.new
+  @@active_job_queue_name_prefixing = false
 
   class << self
     def options
@@ -57,11 +58,14 @@ module Shoryuken
       @@worker_registry
     end
 
-    # Shoryuken.configure_server do |config|
-    #   config.server_middleware do |chain|
-    #     chain.add MyServerHook
-    #   end
-    # end
+    def active_job_queue_name_prefixing
+      @@active_job_queue_name_prefixing
+    end
+
+    def active_job_queue_name_prefixing=(prefixing)
+      @@active_job_queue_name_prefixing = prefixing
+    end
+
     def configure_server
       yield self if server?
     end
@@ -72,6 +76,16 @@ module Shoryuken
       @server_chain
     end
 
+    def configure_client
+      yield self
+    end
+
+    def client_middleware
+      @client_chain ||= default_client_middleware
+      yield @client_chain if block_given?
+      @client_chain
+    end
+
     def default_worker_options
       @@default_worker_options ||= {
         'queue'                   => 'default',
@@ -116,6 +130,10 @@ module Shoryuken
       end
     end
 
+    def default_client_middleware
+      Middleware::Chain.new
+    end
+
     def server?
       defined?(Shoryuken::CLI)
     end

data/spec/shoryuken/client_spec.rb

@@ -5,12 +5,13 @@ describe Shoryuken::Client do
   let(:sqs)         { Aws::SQS::Client.new(stub_responses: true, credentials: credentials) }
   let(:queue_name)  { 'shoryuken' }
   let(:queue_url)   { 'https://eu-west-1.amazonaws.com:6059/123456789012/shoryuken' }
-
-  before do
-    described_class.sqs = sqs
-  end
+  let(:sqs_endpoint) { 'http://localhost:4568' }
+  let(:sns_endpoint) { 'http://0.0.0.0:4568'   }
 
   describe '.queue' do
+    before do
+      described_class.sqs = sqs
+    end
     it 'memoizes queues' do
       sqs.stub_responses(:get_queue_url, { queue_url: queue_url }, { queue_url: 'xyz' })
 
@@ -18,4 +19,44 @@ describe Shoryuken::Client do
       expect(Shoryuken::Client.queues(queue_name).url).to eq queue_url
     end
   end
+
+  describe 'environment variable endpoints' do
+    before do
+      ENV['AWS_SQS_ENDPOINT'] = sqs_endpoint
+      ENV['AWS_SNS_ENDPOINT'] = sns_endpoint
+      ENV['AWS_REGION'] = 'us-east-1'
+      Shoryuken.options[:aws] = {}
+    end
+
+    it 'will use config file settings if set' do
+      load_config_file_by_file_name('shoryuken_endpoint.yml')
+      expect(described_class.sqs.config.endpoint.to_s).to eql('https://github.com/phstc/shoryuken:4568')
+      expect(described_class.sns.config.endpoint.to_s).to eq('http://127.0.0.1:4568')
+    end
+
+    it 'should fallback to environment variable if config file not found or set' do
+      load_config_file_by_file_name(nil)
+      expect(described_class.sqs.config.endpoint.to_s).to eql(sqs_endpoint)
+      expect(described_class.sns.config.endpoint.to_s).to eq(sns_endpoint)
+    end
+
+    it 'should fallback to environment variable if config file found but settings not set' do
+      load_config_file_by_file_name('shoryuken.yml')
+      expect(described_class.sqs.config.endpoint.to_s).to eql(sqs_endpoint)
+      expect(described_class.sns.config.endpoint.to_s).to eq(sns_endpoint)
+    end
+
+    it 'will fallback to default settings if no config file settings or environment variables found' do
+      ENV['AWS_SQS_ENDPOINT'] = nil
+      ENV['AWS_SNS_ENDPOINT'] = nil
+      load_config_file_by_file_name('shoryuken.yml')
+      expect(described_class.sqs.config.endpoint.to_s).to eql('https://sqs.us-east-1.amazonaws.com')
+      expect(described_class.sns.config.endpoint.to_s).to eq('https://sns.us-east-1.amazonaws.com')
+    end
+  end
+
+  def load_config_file_by_file_name(file_name)
+    path_name = file_name ? File.join(File.expand_path('../../..', __FILE__), 'spec', file_name) : nil
+    Shoryuken::EnvironmentLoader.load(config_file: path_name)
+  end
 end

data/spec/shoryuken/middleware/chain_spec.rb

@@ -58,7 +58,9 @@ describe Shoryuken::Middleware::Chain do
   it 'patches deprecated middleware' do
     subject.clear
 
-    expect(Shoryuken.logger).to receive(:warn).with("[DEPRECATION] DeprecatedMiddleware#call(worker_instance, queue, sqs_msg) is deprecated. Please use DeprecatedMiddleware#call(worker_instance, queue, sqs_msg, body)")
+    expect(Shoryuken.logger).to receive(:warn) do |&block|
+      expect(block.call).to eq('[DEPRECATION] DeprecatedMiddleware#call(worker_instance, queue, sqs_msg) is deprecated. Please use DeprecatedMiddleware#call(worker_instance, queue, sqs_msg, body)')
+    end
 
     subject.add DeprecatedMiddleware
 

data/spec/shoryuken/middleware/server/timing_spec.rb

@@ -16,8 +16,12 @@ describe Shoryuken::Middleware::Server::Timing do
   end
 
   it 'logs timing' do
-    expect(Shoryuken.logger).to receive(:info).with(/started at/)
-    expect(Shoryuken.logger).to receive(:info).with(/completed in/)
+    expect(Shoryuken.logger).to receive(:info) do |&block|
+      expect(block.call).to match(/started at/)
+    end
+    expect(Shoryuken.logger).to receive(:info) do |&block|
+      expect(block.call).to match(/completed in/)
+    end
 
     subject.call(TestWorker.new, queue, sqs_msg, sqs_msg.body) {}
   end
@@ -26,9 +30,15 @@ describe Shoryuken::Middleware::Server::Timing do
     it 'logs exceeded' do
       allow(subject).to receive(:elapsed).and_return(120000)
 
-      expect(Shoryuken.logger).to receive(:info).with(/started at/)
-      expect(Shoryuken.logger).to receive(:info).with(/completed in/)
-      expect(Shoryuken.logger).to receive(:warn).with('exceeded the queue visibility timeout by 60000 ms')
+      expect(Shoryuken.logger).to receive(:info) do |&block|
+        expect(block.call).to match(/started at/)
+      end
+      expect(Shoryuken.logger).to receive(:info) do |&block|
+        expect(block.call).to match(/completed in/)
+      end
+      expect(Shoryuken.logger).to receive(:warn) do |&block|
+        expect(block.call).to match('exceeded the queue visibility timeout by 60000 ms')
+      end
 
       subject.call(TestWorker.new, queue, sqs_msg, sqs_msg.body) {}
     end
@@ -36,8 +46,12 @@ describe Shoryuken::Middleware::Server::Timing do
 
   context 'when exception' do
     it 'logs failed in' do
-      expect(Shoryuken.logger).to receive(:info).with(/started at/)
-      expect(Shoryuken.logger).to receive(:info).with(/failed in/)
+      expect(Shoryuken.logger).to receive(:info) do |&block|
+        expect(block.call).to match(/started at/)
+      end
+      expect(Shoryuken.logger).to receive(:info) do |&block|
+        expect(block.call).to match(/failed in/)
+      end
 
       expect {
         subject.call(TestWorker.new, queue, sqs_msg, sqs_msg.body) { raise }

data/spec/shoryuken/processor_spec.rb

@@ -59,8 +59,28 @@ describe Shoryuken::Processor do
       subject.process(queue, sqs_msg)
     end
 
+    it 'parses calling `.load`' do
+      TestWorker.get_shoryuken_options['body_parser'] = Class.new do
+        def self.load(*args)
+          JSON.load(*args)
+        end
+      end
+
+      body = { 'test' => 'hi' }
+
+      expect_any_instance_of(TestWorker).to receive(:perform).with(sqs_msg, body)
+
+      allow(sqs_msg).to receive(:body).and_return(JSON.dump(body))
+
+      subject.process(queue, sqs_msg)
+    end
+
     it 'parses calling `.parse`' do
-      TestWorker.get_shoryuken_options['body_parser'] = JSON
+      TestWorker.get_shoryuken_options['body_parser'] = Class.new do
+        def self.parse(*args)
+          JSON.parse(*args)
+        end
+      end
 
       body = { 'test' => 'hi' }
 
@@ -72,16 +92,22 @@ describe Shoryuken::Processor do
     end
 
     context 'when parse errors' do
-      it 'does not fail' do
+      before do
         TestWorker.get_shoryuken_options['body_parser'] = :json
 
-        expect_any_instance_of(TestWorker).to receive(:perform).with(sqs_msg, nil)
-
         allow(sqs_msg).to receive(:body).and_return('invalid json')
+      end
 
-        expect(subject.logger).to receive(:error).with("Error parsing the message body: 757: unexpected token at 'invalid json'\nbody_parser: json\nsqs_msg.body: invalid json")
+      it 'logs the error' do
+        expect(subject.logger).to receive(:error) do |&block|
+          expect(block.call).to eq("Error parsing the message body: 757: unexpected token at 'invalid json'\nbody_parser: json\nsqs_msg.body: invalid json")
+        end
 
-        subject.process(queue, sqs_msg)
+        subject.process(queue, sqs_msg) rescue nil
+      end
+
+      it 're raises the error' do
+        expect{ subject.process(queue, sqs_msg) }.to raise_error(JSON::ParserError, "757: unexpected token at 'invalid json'")
       end
     end
 

data/spec/shoryuken/queue_spec.rb

@@ -9,6 +9,19 @@ describe Shoryuken::Queue do
   subject { described_class.new(sqs, queue_name) }
 
   describe '#send_message' do
+    it 'accepts SQS request parameters' do
+      # https://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/Client.html#send_message-instance_method
+      expect(sqs).to receive(:send_message).with(hash_including(message_body: 'msg1'))
+
+      subject.send_message(message_body: 'msg1')
+    end
+
+    it 'accepts a string' do
+      expect(sqs).to receive(:send_message).with(hash_including(message_body: 'msg1'))
+
+      subject.send_message('msg1')
+    end
+
     context 'when body is invalid' do
       it 'raises ArgumentError for nil' do
         expect {
@@ -21,10 +34,55 @@ describe Shoryuken::Queue do
           subject.send_message(message_body: 1)
         }.to raise_error(ArgumentError, 'The message body must be a String and you passed a Fixnum')
       end
+
+      context 'when a client middleware' do
+        class MyClientMiddleware
+          def call(options)
+            options[:message_body] = 'changed'
+
+            yield
+          end
+        end
+
+        before do
+          Shoryuken.configure_client do |config|
+            config.client_middleware do |chain|
+              chain.add MyClientMiddleware
+            end
+          end
+        end
+
+        after do
+          Shoryuken.configure_client do |config|
+            config.client_middleware do |chain|
+              chain.remove MyClientMiddleware
+            end
+          end
+        end
+
+        it 'invokes MyClientMiddleware' do
+          expect(sqs).to receive(:send_message).with(hash_including(message_body: 'changed'))
+
+          subject.send_message(message_body: 'original')
+        end
+      end
     end
   end
 
   describe '#send_messages' do
+    it 'accepts SQS request parameters' do
+      # https://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/Client.html#send_message_batch-instance_method
+      expect(sqs).to receive(:send_message_batch).with(hash_including(entries: [{ id: '0', message_body: 'msg1'}, { id: '1', message_body: 'msg2' }]))
+
+      subject.send_messages(entries: [{ id: '0', message_body: 'msg1'}, { id: '1', message_body: 'msg2' }])
+    end
+
+    it 'accepts an array of string' do
+      expect(sqs).to receive(:send_message_batch).with(hash_including(entries: [{ id: '0', message_body: 'msg1'}, { id: '1', message_body: 'msg2' }]))
+
+      subject.send_messages(%w(msg1 msg2))
+    end
+
     context 'when body is invalid' do
       it 'raises ArgumentError for nil' do
         expect {

data/spec/shoryuken_endpoint.yml

@@ -0,0 +1,6 @@
+aws:
+  access_key_id:      <%= ENV['AWS_ACCESS_KEY_ID'] %>
+  secret_access_key:  <%= ENV['AWS_SECRET_ACCESS_KEY'] %>
+  region:             us-east-1
+  sqs_endpoint: https://github.com/phstc/shoryuken:4568
+  sns_endpoint: http://127.0.0.1:4568

data/spec/spec_helper.rb

@@ -39,7 +39,10 @@ class TestWorker
 end
 
 RSpec.configure do |config|
-  config.filter_run_excluding slow: true unless ENV['SPEC_ALL']
+  # Only run slow tests if SPEC_ALL=true and AWS_ACCESS_KEY_ID is present
+  # The AWS_ACCESS_KEY_ID checker is because Travis CI does not expose ENV variables to pull requests from forked repositories
+  # http://docs.travis-ci.com/user/pull-requests/
+  config.filter_run_excluding slow: true if ENV['SPEC_ALL'] != 'true' || ENV['AWS_ACCESS_KEY_ID'].nil?
 
   config.before do
     Shoryuken::Client.class_variable_set :@@queues, {}

metadata

@@ -1,125 +1,125 @@
 --- !ruby/object:Gem::Specification
 name: shoryuken
 version: !ruby/object:Gem::Version
-  version: 1.0.2
+  version: 2.0.0
 platform: ruby
 authors:
 - Pablo Cantero
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-04-08 00:00:00.000000000 Z
+date: 2015-09-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '1.6'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '1.6'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: pry-byebug
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: nokogiri
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: dotenv
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: aws-sdk-core
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 2.0.21
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 2.0.21
 - !ruby/object:Gem::Dependency
   name: celluloid
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 0.16.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 0.16.0
 description: Shoryuken is a super efficient AWS SQS thread based message processor
@@ -130,11 +130,12 @@ executables:
 extensions: []
 extra_rdoc_files: []
 files:
-- .gitignore
-- .hound.yml
-- .rspec
-- .rubocop.yml
-- .travis.yml
+- ".gitignore"
+- ".hound.yml"
+- ".rspec"
+- ".rubocop.yml"
+- ".travis.yml"
+- CHANGELOG.md
 - Gemfile
 - LICENSE.txt
 - README.md
@@ -186,6 +187,7 @@ files:
 - spec/shoryuken/topic_spec.rb
 - spec/shoryuken/util_spec.rb
 - spec/shoryuken/worker_spec.rb
+- spec/shoryuken_endpoint.yml
 - spec/shoryuken_spec.rb
 - spec/spec_helper.rb
 homepage: https://github.com/phstc/shoryuken
@@ -198,17 +200,17 @@ require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
-  - - '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.4.5
+rubygems_version: 2.2.5
 signing_key: 
 specification_version: 4
 summary: Shoryuken is a super efficient AWS SQS thread based message processor
@@ -230,5 +232,6 @@ test_files:
 - spec/shoryuken/topic_spec.rb
 - spec/shoryuken/util_spec.rb
 - spec/shoryuken/worker_spec.rb
+- spec/shoryuken_endpoint.yml
 - spec/shoryuken_spec.rb
 - spec/spec_helper.rb