propono 1.0.0 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 134e98b5329c52e212887f279991341a009a46e2
-  data.tar.gz: 26589e1213fbc353ed00ffa48a630d7f2a16f9ed
+  metadata.gz: bc75e2e19f1d9e8fa2ca756f46652573e37e594c
+  data.tar.gz: 4c69d41b50a1a8789a914eff5f6c6bfed7cec2b4
 SHA512:
-  metadata.gz: c4adb5a4f0c46581418f1a1615ea9d8a22364fdfa5d92cb2a15d3506117474e13f68c87ec4c62338e2135953e2a5b34bac5dbe48d542160301383ed560b5cdb9
-  data.tar.gz: c0e345dadb1ba73c92017ba720ac83ee127b26b6436c3529b267571036a9b039b80f6d02c885107cb10340a9ad1fd5abf10767c9db5078d142748312e23a6caf
+  metadata.gz: 5b9b480054550f9b89702d960c0334ce4a379709e35789a2915da49de36239d9ede8f55b66675bc31b4ad628a738e9574b730b311ec1082b8a804529f5d55f9b
+  data.tar.gz: 5716632a5a337f4fe101a5ec9dc08e66d4fced1191fa77d479a28688e91f8a30ec3fc624494f4696c99537ebbb1754f42906c0a0cbbe6790ac82c65d51c05219

data/CHANGELOG.md

@@ -1,3 +1,6 @@
+# 1.1.0 / 2014-02-18
+* [FEATURE] Added slow queue to allow processing of lower priority messages.
+
 # 1.0.0.rc3 / 2013-12-20
 * [FEATURE] Create failed and corrupt queues when subscribe.
 

data/README.md

@@ -48,7 +48,7 @@ Propono.publish('some-topic', "Some string")
 Propono.publish('some-topic', {some: ['hash', 'or', 'array']})
 ```
 
-Listening for messages is easy too. Just tell Propono what your application is called and start listening. You'll get a block yieleded for each message.
+Listening for messages is easy too. Just tell Propono what your application is called and start listening. You'll get a block yielded for each message.
 
 ```ruby
 Propono.config.application_name = "application-name" # Something unique to this app.
@@ -56,11 +56,13 @@ Propono.listen_to_queue('some-topic') do |message|
   # ... Do something interesting with the message
 end
 ```
-In the background, Propono is automatically setting up a queue using SQS, a notification system using SNS, and glueing them all together for you. But you don't have to worry about any of that.
+In the background, Propono is automatically setting up a queue using SQS, a notification system using SNS, and gluing them all together for you. But you don't have to worry about any of that.
+
+**Note for using in Rake tasks and similar:** Propono spawns new threads for messages sent via SNS. If your application ends before the final thread is executed, then the last message might not send. It's therefore advisable to do a `Thread.join` on the thread that is returned from the final call to `publish`.
 
 ### Using TCP for messages
 
-Publishing directly to SNS takes about 15x longer than publishing over a simple TCP connection. It is therefore some times favourable to publish to a seperate machine listening for TCP messages, which will then proxy them on.
+Publishing directly to SNS takes about 15x longer than publishing over a simple TCP connection. It is therefore sometimes favourable to publish to a separate machine listening for TCP messages, which will then proxy them on.
 
 To send messages this way, you need to set up a little extra config:
 
@@ -91,7 +93,7 @@ Propono.proxy_tcp()
 
 ### Using UDP for messages
 
-If you want almost-zero performance impact, and don't mind the occasional message getting lost, you can use UDP. We use this for things like our live dashboard where we don't mind losing a piece of activity here and there, but any perforamnce impact on our Meducation itself is bad news.
+If you want almost-zero performance impact, and don't mind the occasional message getting lost, you can use UDP. We use this for things like our live dashboard where we don't mind losing a piece of activity here and there, but any performance impact on our Meducation itself is bad news.
 
 Sending messages in this way is very similar to using TCP. First add some config:
 
@@ -106,7 +108,7 @@ You then simply pass the `:udp` protocol into `publish`:
 Propono.publish('some-topic', message, protocol: :udp)
 ```
 
-As per the `listen_to_tcp` method explained above, you now listen to udp or use the proxy method:
+As per the `listen_to_tcp` method explained above, you now listen to UDP or use the proxy method:
 
 ```ruby
 Propono.listen_to_udp do |topic, message|
@@ -134,7 +136,7 @@ Propono.config do |config|
 end
 ```
 
-The can all also be setting using the `Propono.config.access_key = "..."` syntax.
+These can all also be set using the `Propono.config.access_key = "..."` syntax.
 
 ### Is it any good?
 
@@ -152,8 +154,9 @@ This project is managed by the [Meducation team](http://company.meducation.net/a
 
 These individuals have come up with the ideas and written the code that made this possible:
 
-- [Jeremy Walker](http://github.com/iHID)
-- [Malcom Landon](http://github.com/malcyL)
+- [Jeremy Walker](http://github.com/iHiD)
+- [Malcolm Landon](http://github.com/malcyL)
+- [Charles Care](http://github.com/ccare)
 - [Rob Styles](http://github.com/mmmmmrob)
 
 ## Licence

data/lib/propono/components/queue_subscription.rb

@@ -4,7 +4,7 @@ module Propono
     include Sns
     include Sqs
 
-    attr_reader :topic_arn, :queue_name, :queue, :failed_queue, :corrupt_queue
+    attr_reader :topic_arn, :queue_name, :queue, :failed_queue, :corrupt_queue, :slow_queue
 
     def self.create(topic_id, options = {})
       new(topic_id, options).tap do |subscription|
@@ -15,6 +15,7 @@ module Propono
     def initialize(topic_id, options = {})
       @topic_id = topic_id
       @suffixed_topic_id = "#{topic_id}#{Propono.config.queue_suffix}"
+      @suffixed_slow_topic_id = "#{topic_id}#{Propono.config.queue_suffix}-slow"
       @queue_name = "#{Propono.config.application_name.gsub(" ", "_")}-#{@suffixed_topic_id}"
     end
 
@@ -26,6 +27,11 @@ module Propono
       @corrupt_queue = QueueCreator.find_or_create("#{queue_name}-corrupt")
       sns.subscribe(@topic.arn, @queue.arn, 'sqs')
       sqs.set_queue_attributes(@queue.url, "Policy", generate_policy)
+
+      @slow_queue = QueueCreator.find_or_create("#{queue_name}-slow")
+      @slow_topic = TopicCreator.find_or_create(@suffixed_slow_topic_id)
+      sns.subscribe(@slow_topic.arn, @slow_queue.arn, 'sqs')
+      sqs.set_queue_attributes(@slow_queue.url, "Policy", generate_slow_policy)
     end
 
     private
@@ -46,6 +52,26 @@ module Propono
       "Resource": "#{@queue.arn}"
     }
   ]
+}
+      EOS
+    end
+
+    def generate_slow_policy
+      <<-EOS
+{
+  "Version": "2008-10-17",
+  "Id": "#{@slow_queue.arn}/SQSDefaultPolicy",
+  "Statement": [
+    {
+      "Sid": "#{@slow_queue.arn}-Sid",
+      "Effect": "Allow",
+      "Principal": {
+        "AWS": "*"
+      },
+      "Action": "SQS:*",
+      "Resource": "#{@slow_queue.arn}"
+    }
+  ]
 }
       EOS
     end

data/lib/propono/services/queue_listener.rb

@@ -24,12 +24,16 @@ module Propono
     private
 
     def read_messages
-      response = sqs.receive_message( queue_url, {'MaxNumberOfMessages' => 10} )
+      read_messages_from_queue(main_queue_url, 10) || read_messages_from_queue(slow_queue_url, 1)
+    end
+
+    def read_messages_from_queue(queue_url, num_messages)
+      response = sqs.receive_message(queue_url, {'MaxNumberOfMessages' => num_messages} )
       messages = response.body['Message']
       if messages.empty?
         false
       else
-        messages.each { |msg| process_raw_message(msg) }
+        messages.each { |msg| process_raw_message(msg, queue_url) }
       end
     rescue Excon::Errors::Forbidden
       Propono.config.logger.error "Forbidden error caught and re-raised. #{queue_url}"
@@ -44,21 +48,21 @@ module Propono
     # as to ensure the message is only deleted if the preceeding line
     # has completed succesfully. We do *not* want to ensure that the
     # message is deleted regardless of what happens in this method.
-    def process_raw_message(raw_sqs_message)
-      sqs_message = parse(raw_sqs_message)
+    def process_raw_message(raw_sqs_message, queue_url)
+      sqs_message = parse(raw_sqs_message, queue_url)
       unless sqs_message.nil?
         Propono.config.logger.info "Propono [#{sqs_message.context[:id]}]: Received from sqs."
         handle(sqs_message)
-        delete_message(raw_sqs_message)
+        delete_message(raw_sqs_message, queue_url)
       end
     end
 
-    def parse(raw_sqs_message)
+    def parse(raw_sqs_message, queue_url)
       SqsMessage.new(raw_sqs_message)
     rescue
       Propono.config.logger.error "Error parsing message, moving to corrupt queue", $!, $!.backtrace
       move_to_corrupt_queue(raw_sqs_message)
-      delete_message(raw_sqs_message)
+      delete_message(raw_sqs_message, queue_url)
       nil
     end
 
@@ -78,17 +82,17 @@ module Propono
     end
 
     def requeue_message_on_failure(sqs_message, exception)
-      next_queue = (sqs_message.failure_count < Propono.config.max_retries) ? queue_url : failed_queue_url
+      next_queue = (sqs_message.failure_count < Propono.config.max_retries) ? main_queue_url : failed_queue_url
       Propono.config.logger.error "Error proessing message, moving to queue: #{next_queue}"
       sqs.send_message(next_queue, sqs_message.to_json_with_exception(exception))
     end
 
-    def delete_message(raw_sqs_message)
+    def delete_message(raw_sqs_message, queue_url)
       sqs.delete_message(queue_url, raw_sqs_message['ReceiptHandle'])
     end
 
-    def queue_url
-      @queue_url ||= subscription.queue.url
+    def main_queue_url
+      @main_queue_url ||= subscription.queue.url
     end
 
     def failed_queue_url
@@ -99,8 +103,13 @@ module Propono
       @corrupt_queue_url ||= subscription.corrupt_queue.url
     end
 
+    def slow_queue_url
+      @slow_queue_url ||= subscription.slow_queue.url
+    end
+
     def subscription
       @subscription ||= QueueSubscription.create(@topic_id)
     end
+
   end
 end

data/lib/propono/version.rb

@@ -1,3 +1,3 @@
 module Propono
-  VERSION = "1.0.0"
+  VERSION = "1.1.0"
 end

data/test/components/queue_subscription_test.rb

@@ -16,7 +16,10 @@ module Propono
     def test_create_topic
       topic_id = 'foobar'
       topic = Topic.new(topic_id)
+      slow_topic_id = 'foobar-slow'
+      slow_topic = Topic.new(slow_topic_id)
       TopicCreator.expects(:find_or_create).with("#{topic_id}#{@suffix}").returns(topic)
+      TopicCreator.expects(:find_or_create).with("#{topic_id}#{@suffix}-slow").returns(slow_topic)
       QueueSubscription.create(topic_id)
     end
 
@@ -32,6 +35,7 @@ module Propono
       sqs.expects(:create_queue).with(queue_name).returns(mock(body: {'QueueUrl' => Fog::AWS::SQS::Mock::QueueUrl}))
       sqs.expects(:create_queue).with(queue_name + '-failed').returns(mock(body: {'QueueUrl' => Fog::AWS::SQS::Mock::QueueUrl}))
       sqs.expects(:create_queue).with(queue_name + '-corrupt').returns(mock(body: {'QueueUrl' => Fog::AWS::SQS::Mock::QueueUrl}))
+      sqs.expects(:create_queue).with(queue_name + '-slow').returns(mock(body: {'QueueUrl' => Fog::AWS::SQS::Mock::QueueUrl}))
       QueueCreator.any_instance.stubs(sqs: sqs)
 
       subscription.create
@@ -62,7 +66,7 @@ module Propono
       QueueCreator.stubs(find_or_create: Queue.new(Fog::AWS::SQS::Mock::QueueUrl))
 
       sns = mock()
-      sns.expects(:subscribe).with(arn, Fog::AWS::SQS::Mock::QueueArn, 'sqs')
+      sns.expects(:subscribe).with(arn, Fog::AWS::SQS::Mock::QueueArn, 'sqs').twice
       subscription = QueueSubscription.new("Some topic")
       subscription.stubs(sns: sns)
       subscription.create
@@ -71,15 +75,18 @@ module Propono
     def test_create_calls_set_queue_attributes
       arn = "arn123"
       policy = "{foobar: 123}"
+      slow_policy = "{foobar: 456}"
 
       TopicCreator.stubs(find_or_create: Topic.new(arn))
       QueueCreator.stubs(find_or_create: Queue.new(Fog::AWS::SQS::Mock::QueueUrl))
 
       sqs = mock()
       sqs.expects(:set_queue_attributes).with(Fog::AWS::SQS::Mock::QueueUrl, "Policy", policy)
+      sqs.expects(:set_queue_attributes).with(Fog::AWS::SQS::Mock::QueueUrl, "Policy", slow_policy)
       subscription = QueueSubscription.new("Some topic")
       subscription.stubs(sqs: sqs)
       subscription.stubs(generate_policy: policy)
+      subscription.stubs(generate_slow_policy: slow_policy)
       subscription.create
     end
 
@@ -87,10 +94,12 @@ module Propono
       queue = Queue.new(Fog::AWS::SQS::Mock::QueueUrl)
       failed_queue = Queue.new(Fog::AWS::SQS::Mock::QueueUrl)
       corrupt_queue = Queue.new(Fog::AWS::SQS::Mock::QueueUrl)
+      slow_queue = Queue.new(Fog::AWS::SQS::Mock::QueueUrl)
 
       QueueCreator.expects(:find_or_create).with('MyApp-SomeTopic-suf').returns(queue)
       QueueCreator.expects(:find_or_create).with('MyApp-SomeTopic-suf-failed').returns(failed_queue)
       QueueCreator.expects(:find_or_create).with('MyApp-SomeTopic-suf-corrupt').returns(corrupt_queue)
+      QueueCreator.expects(:find_or_create).with('MyApp-SomeTopic-suf-slow').returns(slow_queue)
       subscription = QueueSubscription.new("SomeTopic")
       subscription.create
       

data/test/integration/slow_queue_test.rb

@@ -0,0 +1,47 @@
+require File.expand_path('../integration_test', __FILE__)
+
+module Propono
+  class SlowQueueTest < IntegrationTest
+    def test_slow_messages_are_received
+      topic = "slow-test-topic"
+      slow_topic = "slow-test-topic-slow"
+      text = "This is my message #{DateTime.now} #{rand()}"
+      slow_text = "This is my slow message #{DateTime.now} #{rand()}"
+      flunks = []
+      message_received = false
+      slow_message_received = false
+
+      Propono.subscribe_by_queue(topic)
+
+      thread = Thread.new do
+        begin
+          Propono.listen_to_queue(topic) do |message, context|
+            flunks << "Wrong message" unless (message == text || message == slow_text)
+            message_received = true if message == text
+            slow_message_received = true if message == slow_text
+          end
+        rescue => e
+          flunks << e.message
+        ensure
+          thread.terminate
+        end
+      end
+
+      Thread.new do
+        sleep(1) while !message_received
+        sleep(1) while !slow_message_received
+        sleep(5) # Make sure all the message deletion clear up in the thread has happened
+        thread.terminate
+      end
+
+      sleep(1) # Make sure the listener has started
+
+      Propono.publish(slow_topic, slow_text)
+      Propono.publish(topic, text)
+      flunks << "Test Timeout" unless wait_for_thread(thread)
+      flunk(flunks.join("\n")) unless flunks.empty?
+    ensure
+      # thread.terminate
+    end        
+  end
+end

data/test/services/queue_listener_test.rb

@@ -47,44 +47,49 @@ module Propono
     end
 
     def test_read_message_from_sqs
-      queue_url = @listener.send(:queue_url)
+      queue_url = @listener.send(:main_queue_url)
       options = { 'MaxNumberOfMessages' => 10 }
       @sqs.expects(:receive_message).with(queue_url, options).returns(@sqs_response)
-      @listener.send(:read_messages)
+      @listener.send(:read_messages_from_queue, queue_url, 10)
     end
 
     def test_log_message_from_sqs
+      queue_url = @listener.send(:main_queue_url)
       Propono.config.logger.expects(:info).with() {|x| x == "Propono [#{@message1_id}]: Received from sqs."}
-      @listener.send(:read_messages)
+      @listener.send(:read_messages_from_queue, queue_url, 10)
     end
 
     def test_read_messages_calls_process_message_for_each_msg
-      @listener.expects(:process_raw_message).with(@sqs_message1)
-      @listener.expects(:process_raw_message).with(@sqs_message2)
-      @listener.send(:read_messages)
+      queue_url = @listener.send(:main_queue_url)
+      @listener.expects(:process_raw_message).with(@sqs_message1, queue_url)
+      @listener.expects(:process_raw_message).with(@sqs_message2, queue_url)
+      @listener.send(:read_messages_from_queue, queue_url, 10)
     end
 
     def test_read_messages_does_not_call_process_messages_if_there_are_none
+      queue_url = @listener.send(:main_queue_url)
       @sqs_response.stubs(body: {"Message" => []})
       @listener.expects(:process_message).never
-      @listener.send(:read_messages)
+      @listener.send(:read_messages_from_queue, queue_url, 10)
     end
 
     def test_exception_from_sqs_is_logged
-      @listener.stubs(queue_url: "http://example.com")
+      queue_url = "http://example.com"
+      @listener.stubs(main_queue_url: queue_url)
       @sqs.stubs(:receive_message).raises(StandardError)
       Propono.config.logger.expects(:error).with("Unexpected error reading from queue http://example.com")
       Propono.config.logger.expects(:error).with() {|x| x.is_a?(StandardError)}
-      @listener.send(:read_messages)
+      @listener.send(:read_messages_from_queue, queue_url, 10)
     end
 
     def test_forbidden_error_is_logged_and_re_raised
-      @listener.stubs(queue_url: "http://example.com")
+      queue_url = "http://example.com"
+      @listener.stubs(queue_url: queue_url)
       @sqs.stubs(:receive_message).raises(Excon::Errors::Forbidden.new(nil, nil, nil))
       Propono.config.logger.expects(:error).with("Forbidden error caught and re-raised. http://example.com")
       Propono.config.logger.expects(:error).with() {|x| x.is_a?(Excon::Errors::Forbidden)}
       assert_raises Excon::Errors::Forbidden do
-        @listener.send(:read_messages)
+        @listener.send(:read_messages_from_queue, queue_url, 10)
       end
     end
 
@@ -122,7 +127,7 @@ module Propono
       @sqs.expects(:delete_message).with(queue_url, @receipt_handle2)
 
       @listener.stubs(queue_url: queue_url)
-      @listener.send(:read_messages)
+      @listener.send(:read_messages_from_queue, queue_url, 10)
     end
 
     def test_messages_are_deleted_if_there_is_an_exception_processing
@@ -137,19 +142,23 @@ module Propono
       @listener.stubs(sqs: @sqs)
       @listener.stubs(:requeue_message_on_failure).with(SqsMessage.new(@sqs_message1), exception)
       @listener.stubs(:requeue_message_on_failure).with(SqsMessage.new(@sqs_message2), exception)
-      @listener.send(:read_messages)
+      @listener.send(:read_messages_from_queue, queue_url, 10)
     end
     
     def test_messages_are_retried_or_abandoned_on_failure
+      queue_url = "test-queue-url"
+
       exception = StandardError.new("Test Error")
       @listener = QueueListener.new(@topic_id) { raise exception }
       @listener.expects(:requeue_message_on_failure).with(SqsMessage.new(@sqs_message1), exception)
       @listener.expects(:requeue_message_on_failure).with(SqsMessage.new(@sqs_message2), exception)
       @listener.stubs(sqs: @sqs)
-      @listener.send(:read_messages)
+      @listener.send(:read_messages_from_queue, queue_url, 10)
     end
 
     def test_failed_on_moving_to_failed_queue_does_not_delete
+      queue_url = "test-queue-url"
+
       exception = StandardError.new("Test Error")
       @listener = QueueListener.new(@topic_id) { raise exception }
       @listener.stubs(:requeue_message_on_failure).with(SqsMessage.new(@sqs_message1), exception).raises(StandardError.new("failed to move"))
@@ -157,10 +166,11 @@ module Propono
       @listener.expects(:delete_message).with(@sqs_message1).never
       @listener.expects(:delete_message).with(@sqs_message2).never
       @listener.stubs(sqs: @sqs)
-      @listener.send(:read_messages)
+      @listener.send(:read_messages_from_queue, queue_url, 10)
     end
 
     def test_messages_are_moved_to_corrupt_queue_if_there_is_an_parsing_exception
+      queue_url = "test-queue-url"
       sqs_message1 = "foobar"
       sqs_message2 = "barfoo"
       @messages["Message"][0] = sqs_message1
@@ -168,7 +178,7 @@ module Propono
 
       @listener.expects(:move_to_corrupt_queue).with(sqs_message1)
       @listener.expects(:move_to_corrupt_queue).with(sqs_message2)
-      @listener.send(:read_messages)
+      @listener.send(:read_messages_from_queue, queue_url, 10)
     end
 
     def test_message_moved_to_failed_queue_if_there_is_an_exception_and_retry_count_is_zero
@@ -196,5 +206,24 @@ module Propono
       @sqs.expects(:send_message).with(regexp_matches(/https:\/\/queue.amazonaws.com\/[0-9]+\/MyApp-some-topic-corrupt/), anything)
       @listener.send(:move_to_corrupt_queue, @sqs_message1)
     end
+
+    def test_if_no_messages_read_from_normal_queue_read_from_slow_queue
+      main_queue_url = "http://normal.com"
+      @listener.stubs(main_queue_url: main_queue_url)
+      slow_queue_url = "http://slow.com"
+      @listener.stubs(slow_queue_url: slow_queue_url)
+
+      @listener.expects(:read_messages_from_queue).with(main_queue_url, 10).returns(false)
+      @listener.expects(:read_messages_from_queue).with(slow_queue_url, 1)
+      @listener.send(:read_messages)
+    end
+
+    def test_if_read_messages_from_normal_do_not_read_from_slow_queue
+      main_queue_url = "http://normal.com"
+      @listener.stubs(main_queue_url: main_queue_url)
+
+      @listener.expects(:read_messages_from_queue).with(main_queue_url, 10).returns(true)
+      @listener.send(:read_messages)
+    end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: propono
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.1.0
 platform: ruby
 authors:
 - MalcyL
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-01-22 00:00:00.000000000 Z
+date: 2014-02-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fog
@@ -144,6 +144,7 @@ files:
 - test/configuration_test.rb
 - test/helpers/hash_test.rb
 - test/integration/integration_test.rb
+- test/integration/slow_queue_test.rb
 - test/integration/sns_to_sqs_test.rb
 - test/integration/tcp_to_sqs_test.rb
 - test/integration/udp_proxy_test.rb
@@ -178,7 +179,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.1.9
+rubygems_version: 2.2.1
 signing_key: 
 specification_version: 4
 summary: General purpose pub/sub library built on top of AWS SNS and SQS
@@ -194,6 +195,7 @@ test_files:
 - test/configuration_test.rb
 - test/helpers/hash_test.rb
 - test/integration/integration_test.rb
+- test/integration/slow_queue_test.rb
 - test/integration/sns_to_sqs_test.rb
 - test/integration/tcp_to_sqs_test.rb
 - test/integration/udp_proxy_test.rb