shoryuken 1.0.3 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: cbdf9963098403163b107d51bab8dc70384b9d92
-  data.tar.gz: 97ccedcd84c67dda49c66f96a21899ee5a2ce215
+  metadata.gz: ee01f5dc2129537f640aba700e43c8be0f37e77c
+  data.tar.gz: 54c1fe1a997a29a77939336ca5f8f2c7e691a745
 SHA512:
-  metadata.gz: a323f68eff71aa37fe470239fbcd1c67210200dc999d0d1e34f4b61e68c4da38f745613cf294d0b238708980e555875afa66d713c3ea2649468a39177dc96437
-  data.tar.gz: 8f7f72e1b4377614e9f7a01c5917b8e0601d59bb8e7669ee2258c7ca8b0adbeeae8f8ea2e5b5130122dcb4b546c671e4f95e295d2dd8bb697cfc0a1fa8a99ff5
+  metadata.gz: 5543a21bcfadc457a3bb7444b27ba97d59c9f238bbc9633a100555154823132f7ee6f3cf233f84743ed553cc69a27f8ddf836515690a120b291ffb8212db8783
+  data.tar.gz: bcdc762ee9e02366ad4194e372ad76e2e7c23f7e5f15129d8c90d6abfa65af337c4900addaf47d14f8f6578e084a6fd84dad478878589e0c3e2912c5851c89f5

data/CHANGELOG.md

@@ -0,0 +1,11 @@
+## [v2.0.0] - 2015-09-22
+
+- Allow configuration of SQS/SNS endpoints via environment variables
+ - [#130](https://github.com/phstc/shoryuken/pull/130)
+
+- Expose queue_name in the message object
+  - [#127](https://github.com/phstc/shoryuken/pull/127)
+
+- README updates
+  - [#122](https://github.com/phstc/shoryuken/pull/122)
+  - [#120](https://github.com/phstc/shoryuken/pull/120)

data/README.md

@@ -36,7 +36,7 @@ If all queues get empty, all processors will be changed to the waiting state and
 
 ### Fetch in batches
 
-To be even more performance and cost efficient, Shoryuken fetches SQS messages in batches, so a single SQS request can fetch up to 10 messages.
+To be even more performant and cost effective, Shoryuken fetches SQS messages in batches, so a single SQS request can fetch up to 10 messages.
 
 ## Requirements
 
@@ -134,64 +134,11 @@ The ```aws``` section is used to configure both the Aws objects used by Shoryuke
 - ```sqs_endpoint``` can be used to explicitly override the SQS endpoint
 - ```receive_message``` can be used to define the options passed to the http://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/Client.html#receive_message-instance_method
 
-### Rails Integration
-
-You can tell Shoryuken to load your Rails application by passing the `-R` or `--rails` flag to the "shoryuken" command.
-
-If you load Rails, and assuming your workers are located in the `app/workers` directory, they will be auto-loaded. This means you don't need to require them explicitly with `-r`.
-
-For middleware and other configuration, you might want to create an initializer:
-
-```ruby
-Shoryuken.configure_server do |config|
-  # Replace Rails logger so messages are logged wherever Shoryuken is logging
-  # Note: this entire block is only run by the processor, so we don't overwrite
-  #       the logger when the app is running as usual.
-  Rails.logger = Shoryuken::Logging.logger
-
-  config.server_middleware do |chain|
-    chain.add Shoryuken::MyMiddleware
-  end
-end
-```
-
-*Note:* In the above case, since we are replacing the Rails logger, it's desired that this initializer runs before other initializers (in case they themselves use the logger). Since by Rails conventions initializers are executed in alphabetical order, this can be achieved by prepending the initializer filename with `00_` (assuming no other initializers alphabetically precede this one).
-
-This feature works for Rails 4+, but needs to be confirmed for older versions.
-
-#### ActiveJob Support
-
-Yes, Shoryuken supports ActiveJob! This means that you can put your jobs in processor-agnostic `ActiveJob::Base` subclasses, and change processors whenever you want (or better yet, switch to Shoryuken from another processor easily!).
-
-It works as expected. Just put your job in `app/jobs`. Here's an example:
+The ```sns_endpoint``` and ```sqs_endpoint``` Shoryuken-specific options will also fallback to the environment variables ```AWS_SNS_ENDPOINT``` and ```AWS_SQS_ENDPOINT``` respectively, if they are set.
 
-```ruby
-class ProcessPhotoJob < ActiveJob::Base
-  queue_as :default
-
-  rescue_from ActiveJob::DeserializationError do |e|
-    Shoryuken.logger.error ex
-    Shoryuken.logger.error ex.backtrace.join("\n")
-  end
-
-  def perform(photo)
-    photo.process_image!
-  end
-end
-```
-
-Delayed mailers, ActiveRecord serialization, etc. all work.
-
-See [ActiveJob docs](http://edgeguides.rubyonrails.org/active_job_basics.html) for more info.
-
-*Note:* When queueing jobs to be performed in the future (e.g when setting the `wait` or `wait_until` ActiveJob options), SQS limits the amount of time to 15 minutes. Shoryuken will raise an exception if you attempt to schedule a job further into the future than this limit.
-
-*Note:* Active Job allows you to [prefix the queue names](http://edgeguides.rubyonrails.org/active_job_basics.html#queues) of all jobs. Shoryuken supports this behavior natively. By default, though, queue names defined in the config file (or passed to the CLI), are not prefixed in the same way. To have Shoryuken honor Active Job prefixes you must enable that option explicitly. A good place to do that in Rails is in an initializer:
+### Rails Integration
 
-```
-# config/initializers/shoryuken.rb
-Shoryuken.active_job_queue_name_prefixing = true
-```
+[Check the Rails Integration Active Job documention](https://github.com/phstc/shoryuken/wiki/Rails-Integration-Active-Job).
 
 ### Start Shoryuken
 

data/lib/shoryuken/client.rb

@@ -30,7 +30,8 @@ module Shoryuken
       private
 
       def aws_client_options(service_endpoint_key)
-        explicit_endpoint = Shoryuken.options[:aws][service_endpoint_key]
+        environment_endpoint = ENV["AWS_#{service_endpoint_key.to_s.upcase}"]
+        explicit_endpoint = Shoryuken.options[:aws][service_endpoint_key] || environment_endpoint
         options = {}
         options[:endpoint] = explicit_endpoint unless explicit_endpoint.to_s.empty?
         options

data/lib/shoryuken/message.rb

@@ -1,11 +1,21 @@
 module Shoryuken
   class Message
-    attr_accessor :client, :queue_url, :data
+    attr_accessor :client, :queue_url, :queue_name, :data
 
-    def initialize(client, queue_url, data)
+    def initialize(client, queue, data)
       self.client = client
-      self.queue_url = queue_url
       self.data = data
+
+      if queue.is_a?(Shoryuken::Queue)
+        self.queue_url = queue.url
+        self.queue_name = queue.name
+      else
+        # TODO: Remove next major release
+        Shoryuken.loggger.warn do
+          '[DEPRECATION] Passing a queue url into Shoryuken::Message is deprecated, please pass the queue itself'
+        end
+        self.queue_url = queue
+      end
     end
 
     def delete

data/lib/shoryuken/processor.rb

@@ -90,7 +90,7 @@ module Shoryuken
       end
     rescue => e
       logger.error { "Error parsing the message body: #{e.message}\nbody_parser: #{body_parser}\nsqs_msg.body: #{sqs_msg.body}" }
-      nil
+      raise
     end
   end
 end

data/lib/shoryuken/queue.rb

@@ -34,7 +34,7 @@ module Shoryuken
     def receive_messages(options)
       client.receive_message(options.merge(queue_url: url)).
         messages.
-        map { |m| Message.new(client, url, m) }
+        map { |m| Message.new(client, self, m) }
     end
 
     private

data/lib/shoryuken/version.rb

@@ -1,3 +1,3 @@
 module Shoryuken
-  VERSION = '1.0.3'
+  VERSION = '2.0.0'
 end

data/spec/shoryuken/client_spec.rb

@@ -5,12 +5,13 @@ describe Shoryuken::Client do
   let(:sqs)         { Aws::SQS::Client.new(stub_responses: true, credentials: credentials) }
   let(:queue_name)  { 'shoryuken' }
   let(:queue_url)   { 'https://eu-west-1.amazonaws.com:6059/123456789012/shoryuken' }
-
-  before do
-    described_class.sqs = sqs
-  end
+  let(:sqs_endpoint) { 'http://localhost:4568' }
+  let(:sns_endpoint) { 'http://0.0.0.0:4568'   }
 
   describe '.queue' do
+    before do
+      described_class.sqs = sqs
+    end
     it 'memoizes queues' do
       sqs.stub_responses(:get_queue_url, { queue_url: queue_url }, { queue_url: 'xyz' })
 
@@ -18,4 +19,44 @@ describe Shoryuken::Client do
       expect(Shoryuken::Client.queues(queue_name).url).to eq queue_url
     end
   end
+
+  describe 'environment variable endpoints' do
+    before do
+      ENV['AWS_SQS_ENDPOINT'] = sqs_endpoint
+      ENV['AWS_SNS_ENDPOINT'] = sns_endpoint
+      ENV['AWS_REGION'] = 'us-east-1'
+      Shoryuken.options[:aws] = {}
+    end
+
+    it 'will use config file settings if set' do
+      load_config_file_by_file_name('shoryuken_endpoint.yml')
+      expect(described_class.sqs.config.endpoint.to_s).to eql('https://github.com/phstc/shoryuken:4568')
+      expect(described_class.sns.config.endpoint.to_s).to eq('http://127.0.0.1:4568')
+    end
+
+    it 'should fallback to environment variable if config file not found or set' do
+      load_config_file_by_file_name(nil)
+      expect(described_class.sqs.config.endpoint.to_s).to eql(sqs_endpoint)
+      expect(described_class.sns.config.endpoint.to_s).to eq(sns_endpoint)
+    end
+
+    it 'should fallback to environment variable if config file found but settings not set' do
+      load_config_file_by_file_name('shoryuken.yml')
+      expect(described_class.sqs.config.endpoint.to_s).to eql(sqs_endpoint)
+      expect(described_class.sns.config.endpoint.to_s).to eq(sns_endpoint)
+    end
+
+    it 'will fallback to default settings if no config file settings or environment variables found' do
+      ENV['AWS_SQS_ENDPOINT'] = nil
+      ENV['AWS_SNS_ENDPOINT'] = nil
+      load_config_file_by_file_name('shoryuken.yml')
+      expect(described_class.sqs.config.endpoint.to_s).to eql('https://sqs.us-east-1.amazonaws.com')
+      expect(described_class.sns.config.endpoint.to_s).to eq('https://sns.us-east-1.amazonaws.com')
+    end
+  end
+
+  def load_config_file_by_file_name(file_name)
+    path_name = file_name ? File.join(File.expand_path('../../..', __FILE__), 'spec', file_name) : nil
+    Shoryuken::EnvironmentLoader.load(config_file: path_name)
+  end
 end

data/spec/shoryuken/processor_spec.rb

@@ -92,18 +92,22 @@ describe Shoryuken::Processor do
     end
 
     context 'when parse errors' do
-      it 'does not fail' do
+      before do
         TestWorker.get_shoryuken_options['body_parser'] = :json
 
-        expect_any_instance_of(TestWorker).to receive(:perform).with(sqs_msg, nil)
-
         allow(sqs_msg).to receive(:body).and_return('invalid json')
+      end
 
+      it 'logs the error' do
         expect(subject.logger).to receive(:error) do |&block|
           expect(block.call).to eq("Error parsing the message body: 757: unexpected token at 'invalid json'\nbody_parser: json\nsqs_msg.body: invalid json")
         end
 
-        subject.process(queue, sqs_msg)
+        subject.process(queue, sqs_msg) rescue nil
+      end
+
+      it 're raises the error' do
+        expect{ subject.process(queue, sqs_msg) }.to raise_error(JSON::ParserError, "757: unexpected token at 'invalid json'")
       end
     end
 

data/spec/shoryuken_endpoint.yml

@@ -0,0 +1,6 @@
+aws:
+  access_key_id:      <%= ENV['AWS_ACCESS_KEY_ID'] %>
+  secret_access_key:  <%= ENV['AWS_SECRET_ACCESS_KEY'] %>
+  region:             us-east-1
+  sqs_endpoint: https://github.com/phstc/shoryuken:4568
+  sns_endpoint: http://127.0.0.1:4568

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: shoryuken
 version: !ruby/object:Gem::Version
-  version: 1.0.3
+  version: 2.0.0
 platform: ruby
 authors:
 - Pablo Cantero
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-07-15 00:00:00.000000000 Z
+date: 2015-09-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -135,6 +135,7 @@ files:
 - ".rspec"
 - ".rubocop.yml"
 - ".travis.yml"
+- CHANGELOG.md
 - Gemfile
 - LICENSE.txt
 - README.md
@@ -186,6 +187,7 @@ files:
 - spec/shoryuken/topic_spec.rb
 - spec/shoryuken/util_spec.rb
 - spec/shoryuken/worker_spec.rb
+- spec/shoryuken_endpoint.yml
 - spec/shoryuken_spec.rb
 - spec/spec_helper.rb
 homepage: https://github.com/phstc/shoryuken
@@ -208,7 +210,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.2.3
+rubygems_version: 2.2.5
 signing_key: 
 specification_version: 4
 summary: Shoryuken is a super efficient AWS SQS thread based message processor
@@ -230,5 +232,6 @@ test_files:
 - spec/shoryuken/topic_spec.rb
 - spec/shoryuken/util_spec.rb
 - spec/shoryuken/worker_spec.rb
+- spec/shoryuken_endpoint.yml
 - spec/shoryuken_spec.rb
 - spec/spec_helper.rb