shoryuken 2.1.3 → 3.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 84439f7b9ae680a0363a63e98c6cbba645541b86
-  data.tar.gz: 72e3f662b660f594d78bb4328b137c43774eaf9d
+  metadata.gz: 3810f509ebbedaf23dfcd756b17f487097d99ab7
+  data.tar.gz: 2dabc0be8dbab5536377b7fe6cb6e371b1c0a71c
 SHA512:
-  metadata.gz: 2c50f7e305327dc7e155c84a8e520579b3d72f0e5d0d224d6b9a0e346a6923c5348e66bd1c63e3a3cbd9ed22adc488680194266d80004556670250c7cc8db450
-  data.tar.gz: c0154a86dd1098ff213c55411fb6fcc50b6a5f2eebb5d207443cce9952355ed5f2ab191f5d321ac91322bb2f1e44b3099404d7b0018aaf300d450f87450d85c2
+  metadata.gz: 5fe6096c4df914e64b56dcbcad889bda4827cbf6882ab018d4e8568cab9caf61f548a6117ba3b5607da13abcad049be58c59d8498d76abcc0b6f366579eaa79d
+  data.tar.gz: 61c52ec5a273c3294679baff1b00f5938d1aec218f0a304ff7503c2c28f537407306380e8fce9e954ba4caa92113e682289a68de42dd26e3d5515f9714e30d92

data/.codeclimate.yml

@@ -11,6 +11,8 @@ engines:
     enabled: true
   rubocop:
     enabled: true
+    config:
+      file: .rubocop.yml
 ratings:
   paths:
   - "**.rb"

data/.rubocop.yml

@@ -1,9 +1,12 @@
-LineLength:
-  Max: 120
+AllCops:
+  TargetRubyVersion: 2.0
 
 Style/SignalException:
   Enabled: false
 
+Style/DoubleNegation:
+  Enabled: false
+
 Style/SpaceAroundEqualsInParameterDefault:
   Enabled: false
 
@@ -28,6 +31,9 @@ Metrics/ClassLength:
 Metrics/ParameterLists:
   Enabled: false
 
+Metrics/LineLength:
+  Max: 120
+
 Metrics/MethodLength :
   Enabled: false
 

data/.travis.yml

@@ -3,6 +3,7 @@ rvm:
   - 2.0.0
   - 2.1.0
   - 2.2.0
+  - 2.3.3
 
 notifications:
   email:

data/CHANGELOG.md

@@ -1,3 +1,22 @@
+## [v3.0.0] - 2017-03-12
+- Replace Celluloid with Concurrent Ruby
+ - [#291](https://github.com/phstc/shoryuken/pull/291)
+
+- Remove AWS configuration from Shoryuken. Now AWS should be configured from outside. Check [this](https://github.com/phstc/shoryuken/wiki/Configure-the-AWS-Client) for more details
+ - [#317](https://github.com/phstc/shoryuken/pull/291)
+
+- Remove deprecation warnings
+ - [#326](https://github.com/phstc/shoryuken/pull/326)
+
+- Allow dynamic adding queues
+ - [#322](https://github.com/phstc/shoryuken/pull/326)
+
+- Support retry_intervals passed in as a lambda. Auto coerce intervals into integer
+ - [#329](https://github.com/phstc/shoryuken/pull/329)
+
+- Add SQS commands `shoryuken help sqs`, such `ls`, `mv`, `dump` and `requeue`
+ - [#330](https://github.com/phstc/shoryuken/pull/330)
+
 ## [v2.1.3] - 2017-01-27
 - Show a warn message when batch isn't supported
  - [#302](https://github.com/phstc/shoryuken/pull/302)

data/README.md

@@ -20,18 +20,18 @@ concurrency: 25
 delay: 25
 queues:
   - [high_priority, 6]
-  - [default, 2]
+  - [normal_priority, 2]
   - [low_priority, 1]
 ```
 
-And supposing all the queues are full of messages, the configuration above will make Shoryuken to process `high_priority` 3 times more than `default` and 6 times more than `low_priority`,
-splitting the work among the `concurrency: 25` available processors.
+And supposing all the queues are full of messages, the configuration above will make Shoryuken to process `high_priority` 3 times more than `normal_priority` and 6 times more than `low_priority`,
+splitting the work load among all available processors `concurrency: 25` .
 
-If `high_priority` gets empty, Shoryuken will keep using the 25 processors, but only to process `default` (2 times more than `low_priority`) and `low_priority`.
+If `high_priority` gets empty, Shoryuken will keep using the 25 processors, but only to process `normal_priority` and `low_priority`.
 
-If `high_priority` receives a new message, Shoryuken will smoothly increase back the `high_priority` weight one by one until it reaches the weight of 6 again, which is the maximum configured for `high_priority`.
+If `high_priority` receives a new message, Shoryuken will smoothly increase back its weight one by one until it reaches the weight of 6 again.
 
-If all queues get empty, all processors will be changed to the waiting state and the queues will be checked every `delay: 25`. If any queue receives a new message, Shoryuken will start processing again. [Check the delay option documentation for more information](https://github.com/phstc/shoryuken/wiki/Shoryuken-options#delay).
+[If a queue gets empty, Shoryuken will pause checking it for `delay: 25`](https://github.com/phstc/shoryuken/wiki/Shoryuken-options#delay).
 
 
 ### Fetch in batches
@@ -40,7 +40,7 @@ To be even more performant and cost effective, Shoryuken fetches SQS messages in
 
 ## Requirements
 
-Ruby 2.0 or greater. Ruby 1.9 is no longer supported.
+Ruby 2.0 or greater.
 
 ## Installation
 
@@ -105,7 +105,7 @@ end
 
 [Check the Middleware documentation](https://github.com/phstc/shoryuken/wiki/Middleware).
 
-### Configuration (worker side)
+### Shoryuken Configuration
 
 Sample configuration file `shoryuken.yml`.
 
@@ -114,87 +114,13 @@ concurrency: 25  # The number of allocated threads to process messages. Default
 delay: 25        # The delay in seconds to pause a queue when it's empty. Default 0
 queues:
   - [high_priority, 6]
-  - [default, 2]
+  - [normal_priority, 2]
   - [low_priority, 1]
 ```
 
-And setup ```aws``` options to use ```configure_client``` in `config/initializers/shoryuken.rb`:
+#### AWS Configuration
 
-```ruby
-Shoryuken.configure_client do |config|
-  config.aws = {
-    secret_access_key: ...,         # or ENV["AWS_SECRET_ACCESS_KEY"]
-    access_key_id:     ...,         # or ENV["AWS_ACCESS_KEY_ID"]
-    region:            "us-east-1", # or ENV["AWS_REGION"]
-    receive_message: {              # See http://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/Client.html#receive_message-instance_method
-      # wait_time_seconds: N,       # The number of seconds to wait for new messages when polling. Defaults to the #wait_time_seconds defined on the queue
-      attribute_names: [
-        "ApproximateReceiveCount",
-        "SentTimestamp"
-      ]
-    }
-  }
-end
-```
-
-If you use Shoryuken with plain ruby worker class (not Rails), please call `configure_client` at the beginning of the worker file:
-
-```ruby
-Shoryuken.configure_client do |config|
-  config.aws = {
-    secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"],
-    access_key_id:     ENV["AWS_ACCESS_KEY_ID"],
-    region:            ENV["AWS_REGION"]
-  }
-end
-
-class MyWorker
-end
-```
-
-The `aws` section is used to configure both the Aws objects used by Shoryuken internally, and also to set up some Shoryuken-specific config. The Shoryuken-specific keys are listed below, and you can expect any other key defined in that block to be passed on untouched to `Aws::SQS::Client#initialize`:
-
-- `account_id` is used when generating SNS ARNs
-- `sns_endpoint` can be used to explicitly override the SNS endpoint
-- `sqs_endpoint` can be used to explicitly override the SQS endpoint
-- `receive_message` can be used to define the options passed to the http://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/Client.html#receive_message-instance_method
-
-The `sns_endpoint` and `sqs_endpoint` Shoryuken-specific options will also fallback to the environment variables `AWS_SNS_ENDPOINT` and `AWS_SQS_ENDPOINT` respectively, if they are set.
-
-### Configuration (producer side)
-
-'Producer' processes need permissions to put messages into SQS. There are a few ways:
-
-* Use the `configure_server` in Rails initializer
-* Ensure the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` env vars are set.
-* Create a `~/.aws/credentials` file.
-* Set `Aws.config[:credentials]` from Ruby code (e.g. in a Rails initializer)
-* Use the Instance Profiles feature. The IAM role of the targeted machine must have an adequate SQS Policy.
-
-For example, use the `configure_server` in `config/initializers/shoryuken.rb`:
-
-```ruby
-Shoryuken.configure_client do |config|
-  config.aws = {
-    secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"],
-    access_key_id:     ENV["AWS_ACCESS_KEY_ID"],
-    region:            ENV["AWS_REGION"]
-  }
-end
-
-Shoryuken.configure_server do |config|
-  config.aws = {
-    secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"],
-    access_key_id:     ENV["AWS_ACCESS_KEY_ID"],
-    region:            ENV["AWS_REGION"]
-  }
-end
-```
-
-
-Note that storing your credentials into Amazon instances represents a security risk. Instance Profiles tends to be the best choice.
-
-You can read about these in more detail [here](http://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/Client.html).
+[Check the Configure AWS Client documentation](https://github.com/phstc/shoryuken/wiki/Configure-the-AWS-Client)
 
 ### Rails Integration
 
@@ -206,26 +132,16 @@ You can read about these in more detail [here](http://docs.aws.amazon.com/sdkfor
 bundle exec shoryuken -r worker.rb -C shoryuken.yml
 ```
 
-Other options:
-
-```bash
-shoryuken --help
-
-shoryuken [options]
-    -c, --concurrency INT            Processor threads to use
-    -d, --daemon                     Daemonize process
-    -q, --queue QUEUE[,WEIGHT]...    Queues to process with optional weights
-    -r, --require [PATH|DIR]         Location of the worker
-    -C, --config PATH                Path to YAML config file
-    -R, --rails                      Attempts to load the containing Rails project
-    -L, --logfile PATH               Path to writable logfile
-    -P, --pidfile PATH               Path to pidfile
-    -v, --verbose                    Print more verbose output
-    -V, --version                    Print version and exit
-    -h, --help                       Show help
-    ...
-```
+For other options check `bundle exec shoryuken help start`
+
+#### SQS commands
+
+Check also some available SQS commands `bundle exec shoryuken help sqs`, such as:
 
+- `ls` list queues
+- `mv` move messages from one queue to another
+- `dump` dump messages from a queue into a JSON lines file
+- `requeue` requeue messages from a dump file
 
 ## More Information
 

data/Rakefile

@@ -10,7 +10,6 @@ end
 desc 'Open Shoryuken pry console'
 task :console do
   require 'pry'
-  require 'celluloid/current'
   require 'shoryuken'
 
   config_file = File.join File.expand_path('..', __FILE__), 'shoryuken.yml'

data/bin/cli/base.rb

@@ -0,0 +1,42 @@
+# rubocop:disable Metrics/BlockLength
+module Shoryuken
+  module CLI
+    class Base < Thor
+      no_commands do
+        def print_table(entries)
+          column_sizes = print_columns_size(entries)
+
+          entries.map do |entry|
+            puts entry.map.with_index { |e, i| print_format_column(e, column_sizes[i]) }.join
+          end
+        end
+
+        def print_columns_size(entries)
+          column_sizes = Hash.new(0)
+
+          entries.each do |entry|
+            entry.each_with_index do |e, i|
+              e = e.to_s
+              column_sizes[i] = e.size if column_sizes[i] < e.size
+            end
+          end
+
+          column_sizes
+        end
+
+        def print_format_column(column, size)
+          size = 40 if size > 40
+          size_with_padding = size + 4
+          column = column.to_s.ljust(size_with_padding)
+          column = "#{column[0...size - 2]}.." if column.size > size_with_padding
+          column
+        end
+
+        def fail_task(msg, quit = true)
+          say "[FAIL] #{msg}", :red
+          exit(1) if quit
+        end
+      end
+    end
+  end
+end

data/bin/cli/sqs.rb

@@ -0,0 +1,188 @@
+require 'date'
+
+# rubocop:disable Metrics/AbcSize, Metrics/BlockLength
+module Shoryuken
+  module CLI
+    class SQS < Base
+      namespace :sqs
+
+      no_commands do
+        def normalize_dump_message(message)
+          message[:id] = message.delete(:message_id)
+          message[:message_body] = message.delete(:body)
+          message.delete(:receipt_handle)
+          message.delete(:md5_of_body)
+          message.delete(:md5_of_message_attributes)
+          message
+        end
+
+        def sqs
+          @_sqs ||= Aws::SQS::Client.new
+        end
+
+        def find_queue_url(queue_name_prefix)
+          urls = sqs.list_queues(queue_name_prefix: queue_name_prefix).queue_urls
+
+          if urls.size > 1
+            fail_task "There's more than one queue starting with #{queue_name_prefix}: #{urls.join(', ')}"
+          end
+
+          url = urls.first
+
+          fail_task "Queue #{queue_name_prefix} not found" unless url
+
+          url
+        end
+
+        def batch_delete(url, messages)
+          messages.to_a.flatten.each_slice(10) do |batch|
+            sqs.delete_message_batch(
+              queue_url: url,
+              entries: batch.map { |message| { id: message.message_id, receipt_handle: message.receipt_handle } }
+            ).failed.any? do |failure|
+              say "Could not delete #{failure.id}, code: #{failure.code}", :yellow
+            end
+          end
+        end
+
+        def batch_send(url, messages)
+          messages.to_a.flatten.map(&method(:normalize_dump_message)).each_slice(10) do |batch|
+            sqs.send_message_batch(queue_url: url, entries: batch).failed.any? do |failure|
+              say "Could not requeue #{failure.id}, code: #{failure.code}", :yellow
+            end
+          end
+        end
+
+        def find_all(url, limit, &block)
+          count = 0
+          batch_size = limit > 10 ? 10 : limit
+
+          loop do
+            n = limit - count
+            batch_size = n if n < batch_size
+
+            messages = sqs.receive_message(
+              queue_url: url,
+              max_number_of_messages: batch_size,
+              message_attribute_names: ['All']
+            ).messages
+
+            messages.each { |m| yield m }
+
+            count += messages.size
+
+            break if count >= limit
+            break if messages.empty?
+          end
+
+          count
+        end
+
+        def list_and_print_queues(urls)
+          attrs = %w(QueueArn ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible LastModifiedTimestamp)
+
+          entries = urls.map { |u| sqs.get_queue_attributes(queue_url: u, attribute_names: attrs).attributes }.map do |q|
+            [
+              q['QueueArn'].split(':').last,
+              q['ApproximateNumberOfMessages'],
+              q['ApproximateNumberOfMessagesNotVisible'],
+              Time.at(q['LastModifiedTimestamp'].to_i)
+            ]
+          end
+
+          entries.unshift(['Queue', 'Messages Available', 'Messages Inflight', 'Last Modified'])
+
+          print_table(entries)
+        end
+
+        def dump_file(path, queue_name)
+          File.join(path, "#{queue_name}-#{Date.today}.jsonl")
+        end
+      end
+
+      desc 'ls [QUEUE-NAME-PREFIX]', 'List queues'
+      method_option :watch,          aliases: '-w',  type: :boolean,              desc: 'watch queues'
+      method_option :watch_interval,                 type: :numeric, default: 10, desc: 'watch interval'
+      def ls(queue_name_prefix = '')
+        trap('SIGINT', 'EXIT') # expect ctrl-c from loop
+
+        urls = sqs.list_queues(queue_name_prefix: queue_name_prefix).queue_urls
+
+        loop do
+          list_and_print_queues(urls)
+
+          break unless options.watch
+
+          sleep options.watch_interval
+          puts
+        end
+      end
+
+      desc 'dump QUEUE-NAME', 'Dump messages from a queue into a JSON lines file'
+      method_option :number, aliases: '-n', type: :numeric, default: Float::INFINITY, desc: 'number of messages to dump'
+      method_option :path,   aliases: '-p', type: :string,  default: './',            desc: 'path to save the dump file'
+      method_option :delete, aliases: '-d', type: :boolean, default: true,            desc: 'delete from the queue'
+      def dump(queue_name)
+        path = dump_file(options.path, queue_name)
+
+        fail_task "File #{path} already exists" if File.exist?(path)
+
+        url = find_queue_url(queue_name)
+
+        messages = []
+
+        file = nil
+
+        count = find_all(url, options.number) do |m|
+          file ||= File.open(path, 'w')
+
+          file.puts(JSON.dump(m.to_h))
+
+          messages << m if options.delete
+        end
+
+        batch_delete(url, messages) if options.delete
+
+        if count.zero?
+          say "Queue #{queue_name} is empty", :yellow
+        else
+          say "Dump saved in #{path} with #{count} messages", :green
+        end
+      ensure
+        file.close if file
+      end
+
+      desc 'requeue QUEUE-NAME PATH', 'Requeue messages from a dump file'
+      def requeue(queue_name, path)
+        fail_task "Path #{path} not found" unless File.exist?(path)
+
+        messages = File.readlines(path).map { |line| JSON.parse(line, symbolize_names: true) }
+
+        batch_send(find_queue_url(queue_name), messages)
+
+        say "Requeued #{messages.size} messages from #{path} to #{queue_name}", :green
+      end
+
+      desc 'mv QUEUE-NAME-SOURCE QUEUE-NAME-TARGET', 'Move messages from one queue (source) to another (target)'
+      method_option :number, aliases: '-n', type: :numeric, default: Float::INFINITY, desc: 'number of messages to move'
+      method_option :delete, aliases: '-d', type: :boolean, default: true,            desc: 'delete from the queue'
+      def mv(queue_name_source, queue_name_target)
+        url_source = find_queue_url(queue_name_source)
+        messages = []
+
+        count = find_all(url_source, options.number) do |m|
+          messages << m
+        end
+
+        batch_send(find_queue_url(queue_name_target), messages.map(&:to_h))
+        batch_delete(url_source, messages) if options.delete
+
+        if count.zero?
+          say "Queue #{queue_name_source} is empty", :yellow
+        else
+          say "Moved #{count} messages from #{queue_name_source} to #{queue_name_target}", :green
+        end
+      end
+    end
+  end
+end