propono 0.5.6 → 0.6.0

checksums.yaml

@@ -1,15 +1,15 @@
 ---
 !binary "U0hBMQ==":
   metadata.gz: !binary |-
-    OTg5ZWEwODk0YTc2YWVmNDA4NTAxMDMwMjM2M2ZhNWI0NGFjMmFjMg==
+    ZGRiZTQ4MGVmZDFmM2IzZTFjYjY4MDZkZTVlMjRkMzMyZmJjMGI1MQ==
   data.tar.gz: !binary |-
-    NjhlZjgyNDhlMTNmYWI2ODdjZTc5OTcyNmZlMzdiYmQ2M2NmZmVjMg==
+    NWYzZDE2YzEwMzk1YzJhZWIxY2QxNjE1NGU0NWViZDRkN2ZkNDk4YQ==
 SHA512:
   metadata.gz: !binary |-
-    YTlmNzBhNGQ0NGM3OTMyMzZmMmRiYWVkNTZjMTRjZDRhM2MwY2VkZjdhZGZl
-    ODU0MDJlNDk4MjE2YmQ4MTA4ZmRmOTEwNGFlZjJlZDE4OWIyOGZmYzlkODhl
-    NGE5YzY0NTg5YThkZTM1NjQ0MjRjNTQ3NTViZjY4YzMzOTQ5YjE=
+    MWNiNzQxOWE3YzY5YjVjMjhkNzYzYmI1ODRiYzU4ZDI1YThlMjE5YmE4ZGUy
+    NDUwZmY1NDE2ZGJjODNmZjE1Yzg1OWY1YzhjM2QyOTVmMDI2M2FlZmY0OGY4
+    ODI2NzRlN2RmYTkzMTg0ZjljMWM3Y2ZjYThmZGNjOGM2NTJjY2M=
   data.tar.gz: !binary |-
-    OWEyZWZjNDRjYzdjNTExYTk5NzI1M2U0NmI3OTdiYmI3ZGU0MmFiOThmZGVj
-    MmNlODQyZDNiMWFhZDQwNjljOGEwZjJiOWIzZGY5Njc2Y2JkZDY0NDc0ODE0
-    NWM1N2E0YWQzOWFlY2U5N2QwNTg4MmJkZmZlMDliYzQyYWM5NGM=
+    NzM4MjUwMzAzM2I5M2EyNjkxZDIyZjYzOGY5YmViYzJlODEwMjQwODNlZTdk
+    NjJjOTRjMjNjNDNmZTVjNWFlMDc2ZGU1ODVhNjI1YjgxMDMzNzllMDNkOGE2
+    ZTM0YTIzMTExNzMwYTA0NDNlNzUxMTJiNzNmY2IyMTc1NWYxYzg=

data/.gitignore

@@ -14,5 +14,6 @@ rdoc
 spec/reports
 test/tmp
 test/version_tmp
+test/config.yml
 tmp
 *.swp

data/.travis.yml

@@ -2,7 +2,7 @@ language: ruby
 rvm:
   - 2.0.0
 script:
-  - rake test
+  - bundle exec rake test:local
 addons:
     code_climate:
         repo_token: 88dd239bc2e0010742a42a4e0234b4decd19b46d0e9d3408d8b1fe0f96dd8fc1

data/CONTRIBUTING.md

@@ -9,19 +9,22 @@ Please file issues on the [GitHub issues list](https://github.com/meducation/pro
 
 If you want a feature implemented, the best way to get it done is to submit a pull request that implements it. Please make sure it has tests.
 
-You can run the tests with
+To get the implementation tests to run, you'll need to create a `test/config.yml` file. There's an example to get you started.
+
+You can run the tests with:
+
 ```
-rake test
+bundle exec rake test
 ```
 
 If you've not contributed to a repository before - this is the accepted pattern to use:
 
-1. Fork it
+1. Fork it (big button on Github at the top right)
 2. Create your feature branch (`git checkout -b my-new-feature`)
-3. Commit your changes (`git commit -am 'Add some feature'`)
-4. Push to the branch (`git push origin my-new-feature`)
-5. Create new Pull Request
+3. Make your changes (please add tests!)
+4. Commit your changes (`git commit -am 'Add some feature'`)
+5. Push to the branch (`git push origin my-new-feature`)
+6. Create new Pull Request on Github
 
 Thank you again!
 :heart: :sparkling_heart: :heart:
-

data/README.md

@@ -4,54 +4,128 @@
 [![Dependencies](https://gemnasium.com/meducation/propono.png?travis)](https://gemnasium.com/meducation/propono)
 [![Code Climate](https://codeclimate.com/github/meducation/propono.png)](https://codeclimate.com/github/meducation/propono)
 
+Propono is a [pub/sub](http://en.wikipedia.org/wiki/Publish-subscribe_pattern) gem built on top of Amazon Web Services (AWS). It uses Simple Notification Service (SNS) and Simple Queue Service (SQS) to seamlessly pass messages throughout your infrastructure.
+
+It's beautifully simple to use.
+
+```ruby
+# On Machine A
+Propono.listen_to_queue('some-topic') do |message|
+  puts "I just received: #{message}"
+end
+
+# On Machine B
+Propono.publish('some-topic', "The Best Message Ever")
+
+# Output on Machine A a second later.
+# - "I just recieved The Best Message Ever"
+```
+
 ## Installation
 
 Add this line to your application's Gemfile:
 
     gem 'propono'
 
-If you want to use the latest version from Github, you can do:
+And then execute:
+
+    $ bundle install
 
-    gem 'propono', github: "meducation/propono"
+## Usage
 
-And then execute:
+The first thing to do is setup some configuration keys for Propono. It's best to do this in an initializer, or at the start of your application.
+
+```ruby
+Propono.config.access_key       = "access-key"       # From AWS
+Propono.config.secret_key       = "secret-key"       # From AWS
+Propono.config.queue_region     = "queue-region"     # From AWS
+```
+
+You can then start publishing messages easily from anywhere in your codebase.
+
+```ruby
+Propono.publish('some-topic', "Some string")
+Propono.publish('some-topic', {some: ['hash', 'or', 'array']})
+```
+
+Listening for messages is easy too. Just tell Propono what your application is called and start listening. You'll get a block yieleded for each message.
+
+```ruby
+Propono.config.application_name = "application-name" # Something unique to this app.
+Propono.listen_to_queue('some-topic') do |message|
+  # ... Do something interesting with the message
+end
+```
+In the background, Propono is automatically setting up a queue using SQS, a notification system using SNS, and glueing them all together for you. But you don't have to worry about any of that.
+
+### Using UDP for messages
 
-    $ bundle
+If you want almost-zero performance impact, and don't mind the occasional message getting lost, you can use UDP. We use this for things like our live dashboard where we don't mind losing a piece of activity here and there, but any perforamnce impact on our Meducation itself is bad news.
 
-This script demonstrates usage:
+To send messages this way, you need to set up a little extra config:
 
 ```ruby
-require 'propono'
-
-class Toy
-  def play
-    configure
-    make_fun_stuff_happen
-  end
-
-  private
-  def make_fun_stuff_happen
-    Propono.publish("jez-test-topic", "A test message")
-    Propono.subscribe_by_queue("jez-test-topic")
-    Propono.subscribe_by_post("jez-test-topic", 'http://example.com/endpoint')
-  end
-
-  def configure
-    Propono.config.access_key = '...'
-    Propono.config.secret_key = '...'
-    Propono.config.queue_region = 'eu-west-1'
-    Propono.config.application_name = '...'
-  end
+Propono.config.udp_host = "some.host.running.a.propono.listener"
+Propono.config.udp_port = 12543
+```
+
+You then simply pass the `:udp` protocol into `publish`
+
+```ruby
+Propono.publish('some-topic', message, protocol: :udp)
+```
+
+You'll now need another application running Propono to listen to the UDP feed. You can use the same machine or a different one, just make sure the port config is the same in both applications, and you're good to go.
+
+```ruby
+Propono.listen_to_udp do |topic, message|
+  Propono.publish(topic, message) # Proxy the message to SNS
 end
+```
+
+This proxying of UDP to SQS is used so often that there's a simple shortcut. Just run this on the machine receiving the UDP packets.
+
+```ruby
+Propono.proxy_udp()
+```
+
+### Configuration
+
+The following configuration settings are available:
 
-Toy.new.play
 ```
+Propono.config do |config|
+  config.access_key = "An AWS access key"
+  config.secret_key = "A AWS secret key"
+  config.queue_region = "An AWS queue region"
+  config.application_name = "A name unique in your network"
+  config.udp_host = "The host of a machine used for UDP proxying"
+  config.udp_port = "The host of a machine used for UDP proxying"
+  config.logger = "A logger such as Log4r or Rails.logger"
+end
+```
+
+The can all also be setting using the `Propono.config.access_key = "..."` syntax.
+
+### Is it any good?
+
+[Yes.](http://news.ycombinator.com/item?id=3067434)
 
 ## Contributing
 
 Firstly, thank you!! :heart::sparkling_heart::heart:
 
-Please read our [contributing guide](https://github.com/meducation/propono/tree/master/CONTRIBUTING.md) for information on how to get stuck in.
+We'd love to have you involved. Please read our [contributing guide](https://github.com/meducation/propono/tree/master/CONTRIBUTING.md) for information on how to get stuck in.
+
+### Contributors
+
+This project is managed by the [Meducation team](http://company.meducation.net/about#team). 
+
+These individuals have come up with the ideas and written the code that made this possible:
+
+- [Jeremy Walker](http://github.com/iHID)
+- [Malcom Landon](http://github.com/malcyL)
+- [Rob Styles](http://github.com/mmmmmrob)
 
 ## Licence
 

data/Rakefile

@@ -1,8 +1,18 @@
 require "bundler/gem_tasks"
 require 'rake/testtask'
+require 'yard'
+
+YARD::Rake::YardocTask.new do |t|
+end
 
 Rake::TestTask.new do |t|
   t.pattern = "test/**/*_test.rb"
 end
 
+namespace :test do
+  Rake::TestTask.new(:local) do |t|
+    t.pattern = "test/{components/,services/,}*_test.rb"
+  end
+end
+
 task default: :test

data/lib/propono/components/queue_subscription.rb

@@ -24,15 +24,11 @@ module Propono
     end
 
     def queue_name
-      @queue_name ||= "#{config.application_name.gsub(" ", "_")}-#{@topic_id}"
+      @queue_name ||= "#{Propono.config.application_name.gsub(" ", "_")}-#{@topic_id}"
     end
 
     private
 
-    def config
-      Configuration.instance
-    end
-
     def generate_policy
       <<-EOS
 {

data/lib/propono/components/sns.rb

@@ -6,14 +6,10 @@ module Propono
 
     def sns
       @sns ||= Fog::AWS::SNS.new(
-        :aws_access_key_id => config.access_key,
-        :aws_secret_access_key => config.secret_key,
-        :region => config.queue_region
+        :aws_access_key_id => Propono.config.access_key,
+        :aws_secret_access_key => Propono.config.secret_key,
+        :region => Propono.config.queue_region
       )
     end
-
-    def config
-      Configuration.instance
-    end
   end
 end

data/lib/propono/components/sqs.rb

@@ -6,15 +6,11 @@ module Propono
 
     def sqs
       @sqs ||= Fog::AWS::SQS.new(
-        :aws_access_key_id => config.access_key,
-        :aws_secret_access_key => config.secret_key,
-        :region => config.queue_region
+        :aws_access_key_id => Propono.config.access_key,
+        :aws_secret_access_key => Propono.config.secret_key,
+        :region => Propono.config.queue_region
       )
     end
-
-    def config
-      Configuration.instance
-    end
   end
 end
 

data/lib/propono/configuration.rb

@@ -1,12 +1,9 @@
-require 'singleton'
-
 module Propono
 
-  class ConfigurationError < Exception
+  class ProponoConfigurationError < ProponoError
   end
 
   class Configuration
-    include Singleton
 
     SETTINGS = [
       :access_key, :secret_key, :queue_region,
@@ -17,7 +14,7 @@ module Propono
     attr_writer *SETTINGS
 
     def initialize
-      self.logger = $stderr
+      self.logger = Propono::Logger.new
     end
 
     SETTINGS.each do |setting|
@@ -29,11 +26,8 @@ module Propono
     private
 
     def get_or_raise(setting)
-      if val = instance_variable_get("@#{setting.to_s}")
-        val
-      else
-        raise ConfigurationError.new("Configuration for #{setting} is not set")
-      end
+      instance_variable_get("@#{setting.to_s}") || 
+        raise(ProponoConfigurationError.new("Configuration for #{setting} is not set"))
     end
   end
 end

data/lib/propono/logger.rb

@@ -0,0 +1,19 @@
+module Propono
+  class Logger
+
+    StdLevels = %W{debug info warn}
+    ErrorLevels = %W{error fatal}
+
+    StdLevels.each do |level|
+      define_method level do |*args|
+        $stdout.puts *args
+      end
+    end
+
+    ErrorLevels.each do |level|
+      define_method level do |*args|
+        $stderr.puts *args 
+      end
+    end
+  end
+end

data/lib/propono/propono_error.rb

@@ -0,0 +1,7 @@
+module Propono
+
+  # The exception from which all other exceptions in this library derive.
+  class ProponoError < StandardError
+  end
+
+end

data/lib/propono/services/publisher.rb

@@ -1,5 +1,5 @@
 module Propono
-  class PublisherError < Exception
+  class PublisherError < ProponoError
   end
 
   class Publisher
@@ -28,18 +28,15 @@ module Propono
 
     def publish_via_sns
       topic = TopicCreator.find_or_create(topic_id)
-      sns.publish(topic.arn, message)
+      msg = message.is_a?(String) ? message : message.to_json
+      sns.publish(topic.arn, msg)
     end
 
     def publish_via_udp
       payload = {topic: topic_id, message: message}.to_json
-      UDPSocket.new.send(payload, 0, config.udp_host, config.udp_port)
+      UDPSocket.new.send(payload, 0, Propono.config.udp_host, Propono.config.udp_port)
     rescue SocketError => e
-      config.logger.puts "Udp2sqs failed to send : #{e}"
-    end
-
-    def config
-      Configuration.instance
+      Propono.config.logger.error "Propono failed to send : #{e}"
     end
   end
 end

data/lib/propono/services/queue_creator.rb

@@ -1,5 +1,5 @@
 module Propono
-  class QueueCreatorError < Exception
+  class QueueCreatorError < ProponoError
   end
 
   class QueueCreator

data/lib/propono/services/queue_listener.rb

@@ -23,24 +23,23 @@ module Propono
     private
 
     def read_messages
-      response = sqs.receive_message( queue_url, options = { 'MaxNumberOfMessages' => 10 } )
+      #response = sqs.receive_message( queue_url, options = { 'MaxNumberOfMessages' => 10 } )
+      response = sqs.receive_message( queue_url, {'MaxNumberOfMessages' => 10} )
       messages = response.body['Message']
       if messages.empty?
         false
       else
-        process_messages(messages)
+        messages.each { |msg| process_sqs_message(msg) }
       end
     rescue
-      config.logger.puts "Unexpected error reading from queue #{queue_url}"
-      config.logger.puts $!
+      Propono.config.logger.error "Unexpected error reading from queue #{queue_url}"
+      Propono.config.logger.error $!
     end
 
-    def process_messages(messages)
-      messages.each do |message|
-        @message_processor.call(message)
-        sqs.delete_message(queue_url, message['ReceiptHandle'])
-      end
-      true
+    def process_sqs_message(sqs_message)
+      message = JSON.parse(sqs_message["Body"])["Message"]
+      @message_processor.call(message)
+      sqs.delete_message(queue_url, sqs_message['ReceiptHandle'])
     end
 
     def queue_url

data/lib/propono/services/topic_creator.rb

@@ -1,5 +1,5 @@
 module Propono
-  class TopicCreatorError < Exception
+  class TopicCreatorError < ProponoError
   end
 
   class TopicCreator

data/lib/propono/services/udp_listener.rb

@@ -1,17 +1,15 @@
 module Propono
-  class UdpListenerError < StandardError
+  class UdpListenerError < ProponoError
   end
 
   class UdpListener
 
-    def self.listen(host, port, &processor)
-      new(host, port, &processor).listen
+    def self.listen(&processor)
+      new(&processor).listen
     end
 
-    def initialize(host, port, &processor)
+    def initialize(&processor)
       raise UdpListenerError.new("Please provide a block to call for each message") unless block_given?
-      @host = host
-      @port = port
       @processor = processor
     end
 
@@ -22,22 +20,22 @@ module Propono
     private
 
     def receive_and_process
-      text = socket.recvfrom(1024)[0]
-      Thread.new { @processor.call(text) }
+      udp_data = socket.recvfrom(1024)[0]
+      Thread.new { process_udp_data(udp_data) }
+    end
+
+    def process_udp_data(udp_data)
+      json = JSON.parse(udp_data)
+      @processor.call(json['topic'], json['message'])
     end
 
     def socket
       @socket ||= begin
         socket = UDPSocket.new
-        socket.bind(@host, @port)
+        socket.bind(Propono.config.udp_host, Propono.config.udp_port)
         socket
       end
     end
-
-    def config
-      Configuration.instance
-    end
-
   end
 end
 

data/lib/propono/version.rb

@@ -1,3 +1,3 @@
 module Propono
-  VERSION = "0.5.6"
+  VERSION = "0.6.0"
 end

data/lib/propono.rb

@@ -1,5 +1,12 @@
+# Propono
+#
+# Propono is a pub/sub gem built on top of Amazon Web Services (AWS). It uses Simple Notification Service (SNS) and Simple Queue Service (SQS) to seamlessly pass messages throughout your infrastructure.
+
 require "propono/version"
+require 'propono/propono_error'
+require 'propono/logger'
 require 'propono/configuration'
+
 require 'propono/components/sns'
 require 'propono/components/sqs'
 require "propono/components/queue"
@@ -14,28 +21,105 @@ require "propono/services/subscriber"
 require "propono/services/topic_creator"
 require "propono/services/udp_listener"
 
+# Propono is a pub/sub gem built on top of Amazon Web Services (AWS).
+# It uses Simple Notification Service (SNS) and Simple Queue Service (SQS)
+# to seamlessly pass messages throughout your infrastructure.
 module Propono
+
+  # Propono configuration settings.
+  #
+  # Settings should be set in an initializer or using some
+  # other method that insures they are set before any
+  # Propono code is used. They can be set as followed:
+  #
+  #   Propono.config.access_key = "my-access-key"
+  #
+  # The following settings are allowed:
+  #
+  # * <tt>:access_key</tt> - The AWS access key
+  # * <tt>:secret_key</tt> - The AWS secret key
+  # * <tt>:queue_region</tt> - The AWS region
+  # * <tt>:application_name</tt> - The name of the application Propono
+  #   is included in.
+  # * <tt>:udp_host</tt> - If using UDP, the host to send to.
+  # * <tt>:udp_port</tt> - If using UDP, the port to send to.
+  # * <tt>:logger</tt> - A logger object that responds to puts.
   def self.config
-    Configuration.instance
+    @config ||= Configuration.new
+    if block_given?
+      yield @config
+    else
+      @config
+    end
+  end
+
+  # Publishes a new message into the Propono pub/sub network.
+  #
+  # This requires a topic and a message. By default this pushes
+  # out AWS SNS. The method optionally takes a :protocol key in
+  # options, which can be set to :udp for non-guaranteed but very
+  # fast delivery.
+  #
+  # @param [String] topic The name of the topic to publish to.
+  # @param [String] message The message to post.
+  # @param [Hash] options
+  #   * protocol: :udp
+  def self.publish(topic, message, options = {})
+    Publisher.publish(topic, message, options)
   end
 
-  def self.publish(*args)
-    Publisher.publish(*args)
+  # Creates a new SNS-SQS subscription on the specified topic.
+  #
+  # This is implicitly called by {#listen_to_queue}.
+  #
+  # @param [String] topic The name of the topic to subscribe to.
+  def self.subscribe_by_queue(topic)
+    Subscriber.subscribe_by_queue(topic)
   end
 
-  def self.subscribe_by_queue(*args)
-    Subscriber.subscribe_by_queue(*args)
+  # Creates a new SNS-POST subscription on the specified topic.
+  #
+  # The POST currently needs confirming before the subscription
+  # can be published to.
+  #
+  # @param [String] topic The name of the topic to subscribe to.
+  def self.subscribe_by_post(topic, endpoint)
+    Subscriber.subscribe_by_post(topic, endpoint)
   end
 
-  def self.subscribe_by_post(*args)
-    Subscriber.subscribe_by_post(*args)
+  # Listens on a queue and yields for each message
+  #
+  # Calling this will enter a queue-listening loop that
+  # yields the message_processor for each messages.
+  #
+  # This method will automatically create a subscription if
+  # one does not exist, so there is no need to call
+  # <tt>subscribe_by_queue</tt> in addition.
+  #
+  # @param [String] topic The topic to subscribe to.
+  # @param &message_processor The block to yield for each message.
+  def self.listen_to_queue(topic, &message_processor)
+    QueueListener.listen(topic, &message_processor)
   end
 
-  def self.listen_to_queue(*args, &block)
-    QueueListener.listen(*args, &block)
+  # Listens for UDP messages and yields for each.
+  #
+  # Calling this will enter a queue-listening loop that
+  # yields the message_processor for each UDP message received.
+  #
+  # @param &message_processor The block to yield for each message.
+  #   Is called with <tt>|topic, message|</tt>.
+  def self.listen_to_udp(&message_processor)
+    UdpListener.listen(&message_processor)
   end
 
-  def self.listen_to_udp(*args, &block)
-    UdpListener.listen(*args, &block)
+  # Listens for UDP messages and passes them onto the queue.
+  #
+  # This method uses #listen_to_udp and #publish to proxy
+  # messages from UDP onto the queue.
+  def self.proxy_udp
+    Propono.listen_to_udp do |topic, message|
+      Propono.publish(topic, message)
+    end
   end
 end

data/propono.gemspec

@@ -23,5 +23,6 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "bundler", "~> 1.3"
   spec.add_development_dependency "rake"
   spec.add_development_dependency "mocha"
+  spec.add_development_dependency "yard"
   spec.add_development_dependency "minitest", "~> 5.0.8"
 end

data/test/{post_subscription_test.rb → components/post_subscription_test.rb}

@@ -1,4 +1,4 @@
-require File.expand_path('../test_helper', __FILE__)
+require File.expand_path('../../test_helper', __FILE__)
 
 module Propono
   class PostSubscriptionTest < Minitest::Test