shoryuken 3.0.11 → 3.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 245e3ba81adf78f01166faec0e8677ad325158c4
-  data.tar.gz: e6742defd6bf3b887783345ff4c282680c6e2ca0
+  metadata.gz: 779c3e9c7f708f0a6dccb3e8b0c9f7e2490b25e1
+  data.tar.gz: bf260a2793ff8c07ddfb670832ba5f12af8d85d9
 SHA512:
-  metadata.gz: ceb8790cd3b3ddc61ea36d09367e05cb42e5f195b90870e64642d1728ced6c2b94a5bb4d402db51b15da59c5db2f12852a1cc604ebf7fd6ac32259d8c71ab8f4
-  data.tar.gz: f26dc1d67b5f9a7093cd97640bd7fccea45fdfd37197791e189d02af9ab1a18c7a58b4f5f172e5acf1e4f8655a98dc01823684c6104adb8800741bba8a50173c
+  metadata.gz: 1f3e7a4b668c929daa1b5424b7a8ab7f028238b3f6dfe0e812351991bbc37923b58491760df6e4ed4921624cd4954761daeac257c6a540fd193d160d121044de
+  data.tar.gz: 02f2bf2e813cbe1624ae83d16936196ecf523fe8d3773f9b0c39813aa0822d3c71fd273ee01a63072c20d86bf05f4c1a775eda3fae41cec4bf55ea567f772aef

data/.travis.yml

@@ -4,6 +4,7 @@ rvm:
   - 2.1.0
   - 2.2.0
   - 2.3.3
+  - 2.4.1
 
 notifications:
   email:

data/CHANGELOG.md

@@ -1,6 +1,17 @@
+## [v3.1.0] - 2017-07-02
+
+- Add shoryuken sqs delete command
+ - [#395](https://github.com/phstc/shoryuken/pull/395)
+
+- Add processing groups support; Concurrency per queue support
+ - [#389](https://github.com/phstc/shoryuken/pull/389)
+
+- Terminate Shoryuken if the fetcher crashes
+ - [#389](https://github.com/phstc/shoryuken/pull/389)
+
 ## [v3.0.11] - 2017-06-24
 
-- Add shoryuken sqs create. Tell about sqs create, if queue not found.
+- Add shoryuken sqs create command
  - [#388](https://github.com/phstc/shoryuken/pull/388)
 
 ## [v3.0.10] - 2017-06-24
@@ -8,7 +19,7 @@
 - Allow aws sdk v3
  - [#381](https://github.com/phstc/shoryuken/pull/381)
 
-- Allow configuring Rails via the config file 
+- Allow configuring Rails via the config file
  - [#387](https://github.com/phstc/shoryuken/pull/387)
 
 ## [v3.0.9] - 2017-06-05

data/Gemfile

@@ -7,4 +7,5 @@ group :test do
   gem 'codeclimate-test-reporter', require: nil
   gem 'simplecov'
   gem 'multi_xml'
+  gem 'httparty'
 end

data/README.md

@@ -1,35 +1,23 @@
 # Shoryuken
 
-![](shoryuken.jpg)
+![Shoryuken](shoryuken.jpg)
 
-Shoryuken _sho-ryu-ken_ is a super-efficient [AWS SQS](https://aws.amazon.com/sqs/) thread-based message processor.
+Shoryuken _sho-ryu-ken_ is a super-efficient [Amazon SQS](https://aws.amazon.com/sqs/) thread-based message processor.
 
 [![Build Status](https://travis-ci.org/phstc/shoryuken.svg)](https://travis-ci.org/phstc/shoryuken)
 [![Code Climate](https://codeclimate.com/github/phstc/shoryuken/badges/gpa.svg)](https://codeclimate.com/github/phstc/shoryuken)
 
 ## Key features
 
-### Load balancing
-
-Yeah, Shoryuken load balances the messages consumption!
-
-```yaml
-concurrency: 25
-delay: 0
-queues:
-  - [high_priority, 6]
-  - [normal_priority, 2]
-  - [low_priority, 1]
-```
-
-Or you can set them all to 1 for having the same priorities.
-
-- `concurrency` (default 25) is the number of threads you want to make available for processing. 
-- `delay` (default to 0) is the number of seconds to pause fetching from an empty queue. SQS charges per request, pause checking empty queues for a while can be a cost efficient strategy.
-
-### Fetch in batches
-
-To be even more performant and cost effective, Shoryuken fetches SQS messages in batches, so a single SQS request can fetch up to 10 messages.
+- [Rails Active Job](https://github.com/phstc/shoryuken/wiki/Rails-Integration-Active-Job)
+- [Queue Load balancing](https://github.com/phstc/shoryuken/wiki/Shoryuken-options#load-balancing)
+- [Concurrency per queue](https://github.com/phstc/shoryuken/wiki/Processing-Groups)
+- [Long Polling](https://github.com/phstc/shoryuken/wiki/Long-Polling)
+- [Batch processing](https://github.com/phstc/shoryuken/wiki/Worker-options#batch)
+- [Auto extend visibility timeout](https://github.com/phstc/shoryuken/wiki/Worker-options#auto_visibility_timeout)
+- [Exponential backoff](https://github.com/phstc/shoryuken/wiki/Worker-options#retry_intervals)
+- [Middleware support](https://github.com/phstc/shoryuken/wiki/Middleware)
+- Amazon SQS CLI. See `shoryuken help sqs`
 
 ## Requirements
 
@@ -43,102 +31,19 @@ Add this line to your application's Gemfile:
 gem 'shoryuken'
 ```
 
-Or to get the latest updates:
-
-```ruby
-gem 'shoryuken', github: 'phstc/shoryuken', branch: 'master'
-```
-
 And then execute:
 
-    $ bundle
-
-Or install it yourself as:
-
-    $ gem install shoryuken
-
-## Usage
-
-### Worker class
-
-```ruby
-class MyWorker
-  include Shoryuken::Worker
-
-  shoryuken_options queue: 'default', auto_delete: true
-  # shoryuken_options queue: ->{ "#{ENV['environment']}_default" }
-
-  # shoryuken_options body_parser: :json
-  # shoryuken_options body_parser: ->(sqs_msg){ REXML::Document.new(sqs_msg.body) }
-  # shoryuken_options body_parser: JSON
-
-  def perform(sqs_msg, body)
-    puts body
-  end
-end
-```
-
-[Check the Worker options documention](https://github.com/phstc/shoryuken/wiki/Worker-options).
-
-### Sending a message
-
-[Check the Sending a message documentation](https://github.com/phstc/shoryuken/wiki/Sending-a-message)
-
-### Middleware
-
-```ruby
-class MyMiddleware
-  def call(worker_instance, queue, sqs_msg, body)
-    puts 'Before work'
-    yield
-    puts 'After work'
-  end
-end
-```
-
-[Check the Middleware documentation](https://github.com/phstc/shoryuken/wiki/Middleware).
-
-### Shoryuken Configuration
-
-Sample configuration file `shoryuken.yml`.
-
-```yaml
-concurrency: 25  # The number of allocated threads to process messages. Default 25
-delay: 0         # The delay in seconds to pause empty queues. Default 0
-queues:
-  - [queue1, 1]
-  - [queue2, 1]
-  - [queue3, 1]
-```
-
-#### AWS Configuration
-
-[Check the Configure AWS Client documentation](https://github.com/phstc/shoryuken/wiki/Configure-the-AWS-Client)
-
-### Rails Integration
-
-[Check the Rails Integration Active Job documention](https://github.com/phstc/shoryuken/wiki/Rails-Integration-Active-Job).
-
-### Start Shoryuken
-
 ```shell
-bundle exec shoryuken -r worker.rb -C shoryuken.yml
+$ bundle
 ```
 
-For other options check `bundle exec shoryuken help start`
-
-#### SQS commands
-
-Check also some available SQS commands `bundle exec shoryuken help sqs`, such as:
+## Usage
 
-- `ls` list queues
-- `mv` move messages from one queue to another
-- `dump` dump messages from a queue into a JSON lines file
-- `requeue` requeue messages from a dump file
+Check the [Getting Started](https://github.com/phstc/shoryuken/wiki/Getting-Started) page.
 
 ## More Information
 
-For more information on advanced topics such as signals (shutdown), ActiveJob integration, and so on please check the [Shoryuken Wiki](https://github.com/phstc/shoryuken/wiki).
+For more information check the [wiki page](https://github.com/phstc/shoryuken/wiki).
 
 ## Credits
 

data/bin/cli/sqs.rb

@@ -192,6 +192,13 @@ module Shoryuken
 
         say "Queue #{queue_name} was successfully created. Queue URL #{queue_url}", :green
       end
+
+      desc 'delete QUEUE-NAME', 'delete a queue'
+      def delete(queue_name)
+        sqs.delete_queue(queue_url: find_queue_url(queue_name))
+
+        say "Queue #{queue_name} was successfully delete", :green
+      end
     end
   end
 end

data/lib/shoryuken.rb

@@ -20,185 +20,51 @@ require 'shoryuken/middleware/server/auto_delete'
 Shoryuken::Middleware::Server.autoload :AutoExtendVisibility, 'shoryuken/middleware/server/auto_extend_visibility'
 require 'shoryuken/middleware/server/exponential_backoff_retry'
 require 'shoryuken/middleware/server/timing'
-require 'shoryuken/polling'
+require 'shoryuken/polling/base'
+require 'shoryuken/polling/weighted_round_robin'
+require 'shoryuken/polling/strict_priority'
 require 'shoryuken/manager'
 require 'shoryuken/launcher'
 require 'shoryuken/processor'
 require 'shoryuken/fetcher'
+require 'shoryuken/options'
 
 module Shoryuken
-  DEFAULTS = {
-    concurrency: 25,
-    queues: [],
-    aws: {},
-    delay: 0,
-    timeout: 8,
-    lifecycle_events: {
-      startup: [],
-      dispatch: [],
-      quiet: [],
-      shutdown: []
-    },
-    polling_strategy: Polling::WeightedRoundRobin
-  }.freeze
-
-  @@queues                          = []
-  @@worker_registry                 = DefaultWorkerRegistry.new
-  @@active_job_queue_name_prefixing = false
-  @@sqs_client                      = nil
-  @@sqs_client_receive_message_opts = {}
-  @@start_callback                  = nil
-  @@stop_callback                   = nil
-
-  class << self
-    def queues
-      @@queues
-    end
-
-    def add_queue(queue, priority = 1)
-      priority.times { queues << queue }
-    end
-
-    def worker_registry
-      @@worker_registry
-    end
-
-    def worker_registry=(worker_registry)
-      @@worker_registry = worker_registry
-    end
-
-    def start_callback
-      @@start_callback
-    end
-
-    def start_callback=(start_callback)
-      @@start_callback = start_callback
-    end
-
-    def stop_callback
-      @@stop_callback
-    end
-
-    def stop_callback=(stop_callback)
-      @@stop_callback = stop_callback
-    end
-
-    def active_job_queue_name_prefixing
-      @@active_job_queue_name_prefixing
-    end
-
-    def active_job_queue_name_prefixing=(active_job_queue_name_prefixing)
-      @@active_job_queue_name_prefixing = active_job_queue_name_prefixing
-    end
-
-    def sqs_client
-      @@sqs_client ||= Aws::SQS::Client.new
-    end
-
-    def sqs_client=(sqs_client)
-      @@sqs_client = sqs_client
-    end
-
-    def sqs_client_receive_message_opts
-      @@sqs_client_receive_message_opts
-    end
-
-    def sqs_client_receive_message_opts=(sqs_client_receive_message_opts)
-      @@sqs_client_receive_message_opts = sqs_client_receive_message_opts
-    end
-
-    def options
-      @@options ||= DEFAULTS.dup
-    end
-
-    def logger
-      Shoryuken::Logging.logger
-    end
-
-    def register_worker(*args)
-      @@worker_registry.register_worker(*args)
-    end
-
-    def configure_server
-      yield self if server?
-    end
-
-    def server_middleware
-      @@server_chain ||= default_server_middleware
-      yield @@server_chain if block_given?
-      @@server_chain
-    end
-
-    def configure_client
-      yield self unless server?
-    end
-
-    def client_middleware
-      @@client_chain ||= default_client_middleware
-      yield @@client_chain if block_given?
-      @@client_chain
-    end
-
-    def default_worker_options
-      @@default_worker_options ||= {
-        'queue'                   => 'default',
-        'delete'                  => false,
-        'auto_delete'             => false,
-        'auto_visibility_timeout' => false,
-        'retry_intervals'         => nil,
-        'batch'                   => false
-      }
-    end
-
-    def default_worker_options=(default_worker_options)
-      @@default_worker_options = default_worker_options
-    end
-
-    def on_start(&block)
-      @@start_callback = block
-    end
-
-    def on_stop(&block)
-      @@stop_callback = block
-    end
-
-    # Register a block to run at a point in the Shoryuken lifecycle.
-    # :startup, :quiet or :shutdown are valid events.
-    #
-    #   Shoryuken.configure_server do |config|
-    #     config.on(:shutdown) do
-    #       puts "Goodbye cruel world!"
-    #     end
-    #   end
-    def on(event, &block)
-      fail ArgumentError, "Symbols only please: #{event}" unless event.is_a?(Symbol)
-      fail ArgumentError, "Invalid event name: #{event}" unless options[:lifecycle_events].key?(event)
-      options[:lifecycle_events][event] << block
-    end
-
-    private
-
-    def default_server_middleware
-      Middleware::Chain.new do |m|
-        m.add Middleware::Server::Timing
-        m.add Middleware::Server::ExponentialBackoffRetry
-        m.add Middleware::Server::AutoDelete
-        m.add Middleware::Server::AutoExtendVisibility
-        if defined?(::ActiveRecord::Base)
-          require 'shoryuken/middleware/server/active_record'
-          m.add Middleware::Server::ActiveRecord
-        end
-      end
-    end
-
-    def default_client_middleware
-      Middleware::Chain.new
-    end
-
-    def server?
-      defined?(Shoryuken::CLI)
-    end
-  end
+  extend SingleForwardable
+
+  def_delegators(
+    :'Shoryuken::Options',
+    :add_group,
+    :groups,
+    :add_queue,
+    :ungrouped_queues,
+    :worker_registry,
+    :worker_registry=,
+    :polling_strategy,
+    :start_callback,
+    :start_callback=,
+    :stop_callback,
+    :stop_callback=,
+    :active_job_queue_name_prefixing,
+    :active_job_queue_name_prefixing=,
+    :sqs_client,
+    :sqs_client=,
+    :sqs_client_receive_message_opts,
+    :sqs_client_receive_message_opts=,
+    :options,
+    :logger,
+    :register_worker,
+    :configure_server,
+    :server?,
+    :server_middleware,
+    :configure_client,
+    :client_middleware,
+    :default_worker_options,
+    :default_worker_options=,
+    :on_start,
+    :on_stop,
+    :on
+  )
 end
 
 require 'shoryuken/extensions/active_job_adapter' if defined?(::ActiveJob)