shoryuken 3.0.6 → 4.0.0

data/Gemfile

@@ -4,7 +4,15 @@ source 'https://rubygems.org'
 gemspec
 
 group :test do
+  gem 'activejob', '~> 4'
+  gem 'aws-sdk-core', '~> 3'
+  gem 'aws-sdk-sqs'
   gem 'codeclimate-test-reporter', require: nil
-  gem 'simplecov'
+  gem 'httparty'
   gem 'multi_xml'
+  gem 'simplecov'
+end
+
+group :development do
+  gem 'rubocop'
 end

data/Gemfile.aws-sdk-core-v2

@@ -0,0 +1,13 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in shoryuken.gemspec
+gemspec
+
+group :test do
+  gem 'activejob', '~> 4'
+  gem 'aws-sdk-core', '~> 2'
+  gem 'codeclimate-test-reporter', require: nil
+  gem 'httparty'
+  gem 'multi_xml'
+  gem 'simplecov'
+end

data/README.md

@@ -1,42 +1,23 @@
 # Shoryuken
 
-![](shoryuken.jpg)
+![Shoryuken](shoryuken.jpg)
 
-Shoryuken _sho-ryu-ken_ is a super-efficient [AWS SQS](https://aws.amazon.com/sqs/) thread-based message processor.
+Shoryuken _sho-ryu-ken_ is a super-efficient [Amazon SQS](https://aws.amazon.com/sqs/) thread-based message processor.
 
 [![Build Status](https://travis-ci.org/phstc/shoryuken.svg)](https://travis-ci.org/phstc/shoryuken)
 [![Code Climate](https://codeclimate.com/github/phstc/shoryuken/badges/gpa.svg)](https://codeclimate.com/github/phstc/shoryuken)
 
 ## Key features
 
-### Load balancing
-
-Yeah, Shoryuken load balances the messages consumption!
-
-Given this configuration:
-
-```yaml
-concurrency: 25
-delay: 25
-queues:
-  - [high_priority, 6]
-  - [normal_priority, 2]
-  - [low_priority, 1]
-```
-
-And supposing all the queues are full of messages, the configuration above will make Shoryuken to process `high_priority` 3 times more than `normal_priority` and 6 times more than `low_priority`,
-splitting the work load among all available processors `concurrency: 25` .
-
-If `high_priority` gets empty, Shoryuken will keep using the 25 processors, but only to process `normal_priority` and `low_priority`.
-
-If `high_priority` receives a new message, Shoryuken will smoothly increase back its weight one by one until it reaches the weight of 6 again.
-
-[If a queue gets empty, Shoryuken will pause checking it for `delay: 25`](https://github.com/phstc/shoryuken/wiki/Shoryuken-options#delay).
-
-
-### Fetch in batches
-
-To be even more performant and cost effective, Shoryuken fetches SQS messages in batches, so a single SQS request can fetch up to 10 messages.
+- [Rails Active Job](https://github.com/phstc/shoryuken/wiki/Rails-Integration-Active-Job)
+- [Queue Load balancing](https://github.com/phstc/shoryuken/wiki/Shoryuken-options#load-balancing)
+- [Concurrency per queue](https://github.com/phstc/shoryuken/wiki/Processing-Groups)
+- [Long Polling](https://github.com/phstc/shoryuken/wiki/Long-Polling)
+- [Batch processing](https://github.com/phstc/shoryuken/wiki/Worker-options#batch)
+- [Auto extend visibility timeout](https://github.com/phstc/shoryuken/wiki/Worker-options#auto_visibility_timeout)
+- [Exponential backoff](https://github.com/phstc/shoryuken/wiki/Worker-options#retry_intervals)
+- [Middleware support](https://github.com/phstc/shoryuken/wiki/Middleware)
+- Amazon SQS CLI. See `shoryuken help sqs`
 
 ## Requirements
 
@@ -50,102 +31,27 @@ Add this line to your application's Gemfile:
 gem 'shoryuken'
 ```
 
-Or to get the latest updates:
-
-```ruby
-gem 'shoryuken', github: 'phstc/shoryuken', branch: 'master'
-```
-
-And then execute:
-
-    $ bundle
-
-Or install it yourself as:
-
-    $ gem install shoryuken
-
-## Usage
-
-### Worker class
-
-```ruby
-class MyWorker
-  include Shoryuken::Worker
-
-  shoryuken_options queue: 'default', auto_delete: true
-  # shoryuken_options queue: ->{ "#{ENV['environment']}_default" }
-
-  # shoryuken_options body_parser: :json
-  # shoryuken_options body_parser: ->(sqs_msg){ REXML::Document.new(sqs_msg.body) }
-  # shoryuken_options body_parser: JSON
-
-  def perform(sqs_msg, body)
-    puts body
-  end
-end
-```
-
-[Check the Worker options documention](https://github.com/phstc/shoryuken/wiki/Worker-options).
-
-### Sending a message
-
-[Check the Sending a message documentation](https://github.com/phstc/shoryuken/wiki/Sending-a-message)
-
-### Middleware
+If you are using AWS SDK version 3, please also add this line:
 
 ```ruby
-class MyMiddleware
-  def call(worker_instance, queue, sqs_msg, body)
-    puts 'Before work'
-    yield
-    puts 'After work'
-  end
-end
-```
-
-[Check the Middleware documentation](https://github.com/phstc/shoryuken/wiki/Middleware).
-
-### Shoryuken Configuration
-
-Sample configuration file `shoryuken.yml`.
-
-```yaml
-concurrency: 25  # The number of allocated threads to process messages. Default 25
-delay: 25        # The delay in seconds to pause a queue when it's empty. Default 0
-queues:
-  - [high_priority, 6]
-  - [normal_priority, 2]
-  - [low_priority, 1]
+gem 'aws-sdk-sqs'
 ```
 
-#### AWS Configuration
-
-[Check the Configure AWS Client documentation](https://github.com/phstc/shoryuken/wiki/Configure-the-AWS-Client)
-
-### Rails Integration
+The extra gem `aws-sdk-sqs` is required in order to keep Shoryuken compatible with AWS SDK version 2 and 3. 
 
-[Check the Rails Integration Active Job documention](https://github.com/phstc/shoryuken/wiki/Rails-Integration-Active-Job).
-
-### Start Shoryuken
+And then execute:
 
 ```shell
-bundle exec shoryuken -r worker.rb -C shoryuken.yml
+$ bundle
 ```
 
-For other options check `bundle exec shoryuken help start`
-
-#### SQS commands
-
-Check also some available SQS commands `bundle exec shoryuken help sqs`, such as:
+## Usage
 
-- `ls` list queues
-- `mv` move messages from one queue to another
-- `dump` dump messages from a queue into a JSON lines file
-- `requeue` requeue messages from a dump file
+Check the [Getting Started](https://github.com/phstc/shoryuken/wiki/Getting-Started) page.
 
 ## More Information
 
-For more information on advanced topics such as signals (shutdown), ActiveJob integration, and so on please check the [Shoryuken Wiki](https://github.com/phstc/shoryuken/wiki).
+For more information check the [wiki page](https://github.com/phstc/shoryuken/wiki).
 
 ## Credits
 

data/Rakefile

@@ -12,7 +12,7 @@ task :console do
   require 'pry'
   require 'shoryuken'
 
-  config_file = File.join File.expand_path('..', __FILE__), 'shoryuken.yml'
+  config_file = File.join File.expand_path(__dir__), 'shoryuken.yml'
 
   if File.exist? config_file
     config = YAML.load File.read(config_file)

data/bin/cli/base.rb

@@ -1,4 +1,3 @@
-# rubocop:disable Metrics/BlockLength
 module Shoryuken
   module CLI
     class Base < Thor
@@ -25,10 +24,8 @@ module Shoryuken
         end
 
         def print_format_column(column, size)
-          size = 40 if size > 40
           size_with_padding = size + 4
           column = column.to_s.ljust(size_with_padding)
-          column = "#{column[0...size - 2]}.." if column.size > size_with_padding
           column
         end
 

data/bin/cli/sqs.rb

@@ -1,6 +1,6 @@
 require 'date'
 
-# rubocop:disable Metrics/AbcSize, Metrics/BlockLength
+# rubocop:disable Metrics/BlockLength
 module Shoryuken
   module CLI
     class SQS < Base
@@ -8,12 +8,15 @@ module Shoryuken
 
       no_commands do
         def normalize_dump_message(message)
-          message[:id] = message.delete(:message_id)
-          message[:message_body] = message.delete(:body)
-          message.delete(:receipt_handle)
-          message.delete(:md5_of_body)
-          message.delete(:md5_of_message_attributes)
-          message
+          # symbolize_keys is needed for keeping it compatible with `requeue`
+          attributes = message[:attributes].symbolize_keys
+          {
+            id: message[:message_id],
+            message_body: message[:body],
+            message_attributes: message[:message_attributes],
+            message_deduplication_id: attributes[:MessageDeduplicationId],
+            message_group_id: attributes[:MessageGroupId]
+          }
         end
 
         def sqs
@@ -32,20 +35,23 @@ module Shoryuken
               queue_url: url,
               entries: batch.map { |message| { id: message.message_id, receipt_handle: message.receipt_handle } }
             ).failed.any? do |failure|
-              say "Could not delete #{failure.id}, code: #{failure.code}", :yellow
+              say(
+                "Could not delete #{failure.id}, code: #{failure.code}, message: #{failure.message}, sender_fault: #{failure.sender_fault}",
+                :yellow
+              )
             end
           end
         end
 
-        def batch_send(url, messages)
-          messages.to_a.flatten.map(&method(:normalize_dump_message)).each_slice(10) do |batch|
+        def batch_send(url, messages, messages_per_batch = 10)
+          messages.to_a.flatten.map(&method(:normalize_dump_message)).each_slice(messages_per_batch) do |batch|
             sqs.send_message_batch(queue_url: url, entries: batch).failed.any? do |failure|
               say "Could not requeue #{failure.id}, code: #{failure.code}", :yellow
             end
           end
         end
 
-        def find_all(url, limit, &block)
+        def find_all(url, limit)
           count = 0
           batch_size = limit > 10 ? 10 : limit
 
@@ -56,6 +62,7 @@ module Shoryuken
             messages = sqs.receive_message(
               queue_url: url,
               max_number_of_messages: batch_size,
+              attribute_names: ['All'],
               message_attribute_names: ['All']
             ).messages
 
@@ -71,7 +78,7 @@ module Shoryuken
         end
 
         def list_and_print_queues(urls)
-          attrs = %w(QueueArn ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible LastModifiedTimestamp)
+          attrs = %w[QueueArn ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible LastModifiedTimestamp]
 
           entries = urls.map { |u| sqs.get_queue_attributes(queue_url: u, attribute_names: attrs).attributes }.map do |q|
             [
@@ -93,8 +100,8 @@ module Shoryuken
       end
 
       desc 'ls [QUEUE-NAME-PREFIX]', 'Lists queues'
-      method_option :watch,          aliases: '-w',  type: :boolean,              desc: 'watch queues'
-      method_option :watch_interval,                 type: :numeric, default: 10, desc: 'watch interval'
+      method_option :watch,    aliases: '-w', type: :boolean, desc: 'watch queues'
+      method_option :interval, aliases: '-n', type: :numeric, default: 2, desc: 'watch interval in seconds'
       def ls(queue_name_prefix = '')
         trap('SIGINT', 'EXIT') # expect ctrl-c from loop
 
@@ -105,7 +112,7 @@ module Shoryuken
 
           break unless options[:watch]
 
-          sleep options[:watch_interval]
+          sleep options[:interval]
           puts
         end
       end
@@ -145,12 +152,13 @@ module Shoryuken
       end
 
       desc 'requeue QUEUE-NAME PATH', 'Requeues messages from a dump file'
+      method_option :batch_size, aliases: '-n', type: :numeric, default: 10, desc: 'number of messages per batch to send'
       def requeue(queue_name, path)
         fail_task "Path #{path} not found" unless File.exist?(path)
 
         messages = File.readlines(path).map { |line| JSON.parse(line, symbolize_names: true) }
 
-        batch_send(find_queue_url(queue_name), messages)
+        batch_send(find_queue_url(queue_name), messages, options[:batch_size])
 
         say "Requeued #{messages.size} messages from #{path} to #{queue_name}", :green
       end
@@ -182,6 +190,24 @@ module Shoryuken
 
         say "Purge request sent for #{queue_name}. The message deletion process takes up to 60 seconds", :yellow
       end
+
+      desc 'create QUEUE-NAME', 'Create a queue'
+      method_option :attributes, aliases: '-a', type: :hash, default: {}, desc: 'queue attributes'
+      def create(queue_name)
+        attributes = options[:attributes]
+        attributes['FifoQueue'] ||= 'true' if queue_name.end_with?('.fifo')
+
+        queue_url = sqs.create_queue(queue_name: queue_name, attributes: attributes).queue_url
+
+        say "Queue #{queue_name} was successfully created. Queue URL #{queue_url}", :green
+      end
+
+      desc 'delete QUEUE-NAME', 'delete a queue'
+      def delete(queue_name)
+        sqs.delete_queue(queue_url: find_queue_url(queue_name))
+
+        say "Queue #{queue_name} was successfully delete", :green
+      end
     end
   end
 end

data/bin/shoryuken

@@ -7,8 +7,6 @@ require 'aws-sdk-core'
 require_relative 'cli/base'
 require_relative 'cli/sqs'
 require_relative '../lib/shoryuken/runner'
-
-# rubocop:disable Metrics/AbcSize
 module Shoryuken
   module CLI
     class Runner < Base
@@ -21,6 +19,7 @@ module Shoryuken
       method_option :daemon,      aliases: '-d', type: :boolean, desc: 'Daemonize process'
       method_option :queues,      aliases: '-q', type: :array,   desc: 'Queues to process with optional weights'
       method_option :require,     aliases: '-r', type: :string,  desc: 'Dir or path of the workers'
+      method_option :timeout,     aliases: '-t', type: :numeric, desc: 'Hard shutdown timeout'
       method_option :config,      aliases: '-C', type: :string,  desc: 'Path to config file'
       method_option :config_file,                type: :string,  desc: 'Path to config file (backwards compatibility)'
       method_option :rails,       aliases: '-R', type: :boolean, desc: 'Load Rails'
@@ -31,18 +30,14 @@ module Shoryuken
       def start
         opts = options.to_h.symbolize_keys
 
-        if opts[:config_file]
-          say "[DEPRECATED] Please use --config instead of --config-file", :yellow
-        end
+        say '[DEPRECATED] Please use --config instead of --config-file', :yellow if opts[:config_file]
 
         opts[:config_file] = opts.delete(:config) if opts[:config]
 
         # Keep compatibility with old CLI queue format
-        opts[:queues] = options[:queues].map { |q| q.split(',') } if options[:queues]
+        opts[:queues] = opts[:queues].reject(&:empty?).map { |q| q.split(',') } if opts[:queues]
 
-        if options[:daemon] && options[:logfile].nil?
-          fail_task "You should set a logfile if you're going to daemonize"
-        end
+        fail_task "You should set a logfile if you're going to daemonize" if opts[:daemon] && opts[:logfile].nil?
 
         Shoryuken::Runner.instance.run(opts.freeze)
       end

data/examples/bootstrap_queues.rb

@@ -2,7 +2,7 @@ require 'yaml'
 require 'shoryuken'
 
 # load SQS credentials
-config = YAML.load File.read(File.join(File.expand_path('..', __FILE__), 'shoryuken.yml'))
+config = YAML.load File.read(File.join(File.expand_path(__dir__), 'shoryuken.yml'))
 
 Aws.config = config['aws']
 
@@ -18,11 +18,11 @@ if sqs.config['endpoint'] =~ /amazonaws.com/
 
   dead_letter_queue_arn = sqs.get_queue_attributes(
     queue_url: dead_letter_queue_url,
-    attribute_names: %w(QueueArn)
+    attribute_names: %w[QueueArn]
   ).attributes['QueueArn']
 
   attributes = {}
-  attributes['RedrivePolicy'] = %Q({"maxReceiveCount":"7", "deadLetterTargetArn":"#{dead_letter_queue_arn}"})
+  attributes['RedrivePolicy'] = %({"maxReceiveCount":"7", "deadLetterTargetArn":"#{dead_letter_queue_arn}"})
 
   sqs.set_queue_attributes queue_url: default_queue_url, attributes: attributes
 end

data/examples/default_worker.rb

@@ -3,7 +3,7 @@ class DefaultWorker
 
   shoryuken_options queue: 'default', auto_delete: true
 
-  def perform(sqs_msg, body)
-    Shoryuken.logger.debug("Received message: '#{body}'")
+  def perform(_sqs_msg, body)
+    Shoryuken.logger.debug("Received message: #{body}")
   end
 end

data/lib/shoryuken/body_parser.rb

@@ -0,0 +1,27 @@
+module Shoryuken
+  class BodyParser
+    class << self
+      def parse(worker_class, sqs_msg)
+        body_parser = worker_class.get_shoryuken_options['body_parser']
+
+        case body_parser
+        when :json
+          JSON.parse(sqs_msg.body)
+        when Proc
+          body_parser.call(sqs_msg)
+        when :text, nil
+          sqs_msg.body
+        else
+          if body_parser.respond_to?(:parse)
+            # JSON.parse
+            body_parser.parse(sqs_msg.body)
+          elsif body_parser.respond_to?(:load)
+            # see https://github.com/phstc/shoryuken/pull/91
+            # JSON.load
+            body_parser.load(sqs_msg.body)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/shoryuken/client.rb

@@ -8,11 +8,15 @@ module Shoryuken
       end
 
       def sqs
-        @@sqs ||= Shoryuken.sqs_client
+        Shoryuken.sqs_client
       end
 
       def sqs=(sqs)
-        @@sqs = sqs
+        # Since the @@queues values (Shoryuken::Queue objects) are built referencing @@sqs, if it changes, we need to
+        #   re-build them on subsequent calls to `.queues(name)`.
+        @@queues = {}
+
+        Shoryuken.sqs_client = sqs
       end
     end
   end

data/lib/shoryuken/core_ext.rb

@@ -34,7 +34,7 @@ module Shoryuken
   module StringExt
     module Constantize
       def constantize
-        names = self.split('::')
+        names = split('::')
         names.shift if names.empty? || names.first.empty?
 
         constant = Object

data/lib/shoryuken/default_worker_registry.rb

@@ -24,7 +24,7 @@ module Shoryuken
                        @workers[queue]
                      end
 
-      worker_class.new
+      worker_class.new if worker_class
     end
 
     def queues
@@ -34,7 +34,7 @@ module Shoryuken
     def register_worker(queue, clazz)
       if (worker_class = @workers[queue])
         if worker_class.get_shoryuken_options['batch'] == true || clazz.get_shoryuken_options['batch'] == true
-          fail ArgumentError, "Could not register #{clazz} for '#{queue}', "\
+          fail ArgumentError, "Could not register #{clazz} for #{queue}, "\
             "because #{worker_class} is already registered for this queue, "\
             "and Shoryuken doesn't support a batchable worker for a queue with multiple workers"
         end

data/lib/shoryuken/environment_loader.rb

@@ -24,7 +24,7 @@ module Shoryuken
     end
 
     def load
-      load_rails if options[:rails]
+      load_rails if Shoryuken.options[:rails]
       prefix_active_job_queue_names
       parse_queues
       require_workers
@@ -42,9 +42,13 @@ module Shoryuken
     def config_file_options
       return {} unless (path = options[:config_file])
 
-      fail ArgumentError, "The supplied config file '#{path}' does not exist" unless File.exist?(path)
+      fail ArgumentError, "The supplied config file #{path} does not exist" unless File.exist?(path)
 
-      YAML.load(ERB.new(IO.read(path)).result).deep_symbolize_keys
+      if (result = YAML.load(ERB.new(IO.read(path)).result))
+        result.deep_symbolize_keys
+      else
+        {}
+      end
     end
 
     def initialize_logger
@@ -62,10 +66,12 @@ module Shoryuken
       else
         # Painful contortions, see 1791 for discussion
         require File.expand_path('config/application.rb')
-        ::Rails::Application.initializer 'shoryuken.eager_load' do
-          ::Rails.application.config.eager_load = true
+        if ::Rails::VERSION::MAJOR == 4
+          ::Rails::Application.initializer 'shoryuken.eager_load' do
+            ::Rails.application.config.eager_load = true
+          end
         end
-        require 'shoryuken/extensions/active_job_adapter' if defined?(::ActiveJob)
+        require 'shoryuken/extensions/active_job_adapter' if Shoryuken.active_job?
         require File.expand_path('config/environment.rb')
       end
     end
@@ -81,28 +87,53 @@ module Shoryuken
       end
     end
 
-    def prefix_active_job_queue_names
-      return unless defined? ::ActiveJob
-      return unless Shoryuken.active_job_queue_name_prefixing
-
+    def prefix_active_job_queue_name(queue_name, weight)
       queue_name_prefix = ::ActiveJob::Base.queue_name_prefix
       queue_name_delimiter = ::ActiveJob::Base.queue_name_delimiter
 
       # See https://github.com/rails/rails/blob/master/activejob/lib/active_job/queue_name.rb#L27
+      name_parts = [queue_name_prefix.presence, queue_name]
+      prefixed_queue_name = name_parts.compact.join(queue_name_delimiter)
+      [prefixed_queue_name, weight]
+    end
+
+    def prefix_active_job_queue_names
+      return unless Shoryuken.active_job?
+      return unless Shoryuken.active_job_queue_name_prefixing
+
       Shoryuken.options[:queues].to_a.map! do |queue_name, weight|
-        name_parts = [queue_name_prefix.presence, queue_name]
-        prefixed_queue_name = name_parts.compact.join(queue_name_delimiter)
-        [prefixed_queue_name, weight]
+        prefix_active_job_queue_name(queue_name, weight)
+      end
+
+      Shoryuken.options[:groups].to_a.map! do |group, options|
+        if options[:queues]
+          options[:queues].map! do |queue_name, weight|
+            prefix_active_job_queue_name(queue_name, weight)
+          end
+        end
+
+        [group, options]
       end
     end
 
-    def parse_queue(queue, weight = nil)
-      Shoryuken.add_queue(queue, [weight.to_i, 1].max)
+    def parse_queue(queue, weight, group)
+      Shoryuken.add_queue(queue, [weight.to_i, 1].max, group)
     end
 
     def parse_queues
-      Shoryuken.options[:queues].to_a.each do |queue_and_weight|
-        parse_queue(*queue_and_weight)
+      if Shoryuken.options[:queues].to_a.any?
+        Shoryuken.add_group('default', Shoryuken.options.fetch(:concurrency, 25))
+
+        Shoryuken.options[:queues].to_a.each do |queue, weight|
+          parse_queue(queue, weight, 'default')
+        end
+      end
+
+      Shoryuken.options[:groups].to_a.each do |group, options|
+        Shoryuken.add_group(group, options.fetch(:concurrency, 25))
+        options[:queues].to_a.each do |queue, weight|
+          parse_queue(queue, weight, group)
+        end
       end
     end
 
@@ -119,29 +150,34 @@ module Shoryuken
     end
 
     def validate_queues
-      Shoryuken.logger.warn { 'No queues supplied' } if Shoryuken.queues.empty?
+      return Shoryuken.logger.warn { 'No queues supplied' } if Shoryuken.ungrouped_queues.empty?
 
       non_existent_queues = []
 
-      Shoryuken.queues.uniq.each do |queue|
+      Shoryuken.ungrouped_queues.uniq.each do |queue|
         begin
           Shoryuken::Client.queues(queue)
-        rescue Aws::SQS::Errors::NonExistentQueue
+        rescue Aws::Errors::NoSuchEndpointError, Aws::SQS::Errors::NonExistentQueue
           non_existent_queues << queue
         end
       end
 
-      fail ArgumentError, "The specified queue(s) #{non_existent_queues} do not exist" if non_existent_queues.any?
+      return if non_existent_queues.none?
+
+      fail(
+        ArgumentError,
+        "The specified queue(s) #{non_existent_queues.join(', ')} do not exist.\nTry 'shoryuken sqs create QUEUE-NAME' for creating a queue with default settings"
+      )
     end
 
     def validate_workers
-      return if defined?(::ActiveJob)
+      return if Shoryuken.active_job?
 
-      all_queues = Shoryuken.queues
+      all_queues = Shoryuken.ungrouped_queues
       queues_with_workers = Shoryuken.worker_registry.queues
 
       (all_queues - queues_with_workers).each do |queue|
-        Shoryuken.logger.warn { "No worker supplied for '#{queue}'" }
+        Shoryuken.logger.warn { "No worker supplied for #{queue}" }
       end
     end
   end