krakow 0.1.0 → 0.1.2

data/CHANGELOG.md

@@ -1,9 +1,18 @@
-	## v0.1.0
-	* Add logging support
-	* Include valid responses within commands
-	* Segregate responses from messages
-	* Manage connections in consumer (closed/reconnect)
-	* Add message distribution support
+## v0.1.2
+* Include backoff support
+* Remove `method_missing` magic
+* Force message redistribution when connection removed
+* Make discovery interval configurable
+* Add support for HTTP producer
+* Include namespace for custom exceptions #1 (thanks @copiousfreetime)
+* Fix timeout method access in req command #1 (thanks @copiousfreetime)
 
-	## v0.0.1
-	* Initial release
+## v0.1.0
+* Add logging support
+* Include valid responses within commands
+* Segregate responses from messages
+* Manage connections in consumer (closed/reconnect)
+* Add message distribution support
+
+## v0.0.1
+* Initial release

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    krakow (0.0.1)
+    krakow (0.1.1)
       celluloid-io
       http
       multi_json

data/README.md

@@ -7,7 +7,7 @@
 ```ruby
 require 'krakow'
 
-producer = Krakow::Producer(
+producer = Krakow::Producer.new(
   :host => 'HOST',
   :port => 'PORT',
   :topic => 'target'
@@ -20,7 +20,7 @@ producer.write('KRAKOW!', 'KRAKOW!')
 ```ruby
 require 'krakow'
 
-consumer = Krakow::Consumer(
+consumer = Krakow::Consumer.new(
   :nsqlookupd => 'http://HOST:PORT',
   :topic => 'target',
   :channel => 'ship'
@@ -36,7 +36,7 @@ end
 
 ## What is this?
 
-It's a Ruby library for NSQ[1] using Celluloid[2] under the hood.
+It's a Ruby library for [NSQ][1] using [Celluloid][2] under the hood.
 
 ## Information and FAQ that I totally made up
 
@@ -48,7 +48,7 @@ you want, so adjust it when you create your consumer instance.
 ```ruby
 require 'krakow'
 
-consumer = Krakow::Consumer(
+consumer = Krakow::Consumer.new(
   :nsqlookupd => 'http://HOST:PORT',
   :topic => 'target',
   :channel => 'ship',
@@ -56,11 +56,31 @@ consumer = Krakow::Consumer(
 )
 ```
 
+### Clean up after yourself
+
+Since [Celluloid][2] is in use under the hood, and the main interaction points are
+Actors (`Consumer` and `Producer`) you'll need to be sure you clean up. This simply
+means terminating the instance (since falling out of scope will not cause it to be
+garbage collected).
+
+```ruby
+consumer = Krakow::Consumer.new(
+  :nsqlookupd => 'http://HOST:PORT',
+  :topic => 'target',
+  :channel => 'ship',
+  :max_in_flight => 30
+)
+
+# do stuff
+
+consumer.terminate
+```
+
 ### Please make it shutup!
 
 Sure:
 
-```
+```ruby
 Krakow::Utils::Logging.level = :warn # :debug / :info / :warn / :error / :fatal
 ```
 
@@ -73,7 +93,7 @@ Because forcing starvation is mean.
 Fine!
 
 ```ruby
-consumer = Krakow::Consumer(
+consumer = Krakow::Consumer.new(
   :host => 'HOST',
   :port => 'PORT',
   :topic => 'target',
@@ -83,11 +103,32 @@ consumer = Krakow::Consumer(
 ```
 Great for testing, but you really should use the lookup service in the "real world"
 
+### Backoff support
+
+NSQ has this backoff notion. It's pretty swell. Basically, if messages from a specific
+producer get re-queued (fail), then message consumption from that producer is halted,
+and slowly ramped back up. It gives time for downstream issues to work themselves out,
+if possible, instead of just keeping the firehose of gasoline on. Neat.
+
+By default backoff support is disabled. It can be enabled by setting the `:backoff_interval`
+when constructing the `Consumer`. The interval is in seconds (and yes, floats are allowed
+for sub-second intervals):
+
+```ruby
+consumer = Krakow::Consumer.new(
+  :nsqlookupd => 'http://HOST:PORT',
+  :topic => 'target',
+  :channel => 'ship',
+  :max_in_flight => 30,
+  :backoff_interval => 1
+)
+```
+
 ### It doesn't work
 
 Create an issue on the github repository.
 
-#### It doesn't do `x`
+### It doesn't do `x`
 
 Create an issue, or even better, send a PR. Just base it off the `develop` branch.
 
@@ -95,5 +136,5 @@ Create an issue, or even better, send a PR. Just base it off the `develop` branc
 * Repo: https://github.com/chrisroberts/krakow
 * IRC: Freenode @ spox
 
-[1] https://github.com/bitly/nsq
-[2] https://github.com/celluloid/celluloid
+[1]: http://bitly.github.io/nsq/ "NSQ: a realtime distributed messaging platform"
+[2]: http://celluloid.io "Celluloid: Actor-based concurrent object framework for Ruby"

data/lib/krakow/command/req.rb

@@ -8,7 +8,7 @@ module Krakow
       end
 
       def to_line
-        "#{name} #{message_id} #{timeout}\n"
+        "#{name} #{message_id} #{self.timeout}\n"
       end
 
       class << self

data/lib/krakow/connection.rb

@@ -14,7 +14,7 @@ module Krakow
     def initialize(args={})
       super
       required! :host, :port
-      optional :version, :queue, :callback
+      optional :version, :queue, :callback, :responses
       arguments[:queue] ||= Queue.new
       arguments[:responses] ||= Queue.new
       arguments[:version] ||= 'v2'

data/lib/krakow/consumer.rb

@@ -12,17 +12,19 @@ module Krakow
     def initialize(args={})
       super
       required! :topic, :channel
-      optional :host, :port, :nslookupd, :max_in_flight
+      optional :host, :port, :nslookupd, :max_in_flight, :backoff_interval, :discovery_interval
       arguments[:max_in_flight] ||= 1
+      arguments[:discovery_interval] ||= 30
       @connections = {}
       @distribution = Distribution::Default.new(
-        :max_in_flight => max_in_flight
+        :max_in_flight => max_in_flight,
+        :backoff_interval => backoff_interval
       )
       @queue = Queue.new
       if(nslookupd)
         debug "Connections will be established via lookup #{nslookupd.inspect}"
         @discovery = Discovery.new(:nslookupd => nslookupd)
-        every(60){ init! }
+        every(discovery_interval){ init! }
         init!
       elsif(host && port)
         debug "Connection will be established via direct connection #{host}:#{port}"
@@ -126,10 +128,11 @@ module Krakow
       connections.delete_if do |key, value|
         if(value == con)
           warn "Connection failure detected. Removing connection: #{key}"
-          discovery.remove_connection(con)
+          distribution.remove_connection(con)
           true
         end
       end
+      distribution.redistribute!
     end
 
     # message_id:: Message ID
@@ -137,6 +140,7 @@ module Krakow
     def confirm(message_id)
       distribution.in_flight_lookup(message_id) do |connection|
         connection.transmit(Command::Fin.new(:message_id => message_id))
+        distribution.success(connection)
       end
       connection = distribution.unregister_message(message_id)
       update_ready!(connection)
@@ -154,8 +158,9 @@ module Krakow
             :timeout => timeout
           )
         )
+        distribution.failure(connection)
       end
-      connection = distributrion.unregister_message(message_id)
+      connection = distribution.unregister_message(message_id)
       update_ready!(connection)
       true
     end

data/lib/krakow/distribution/default.rb

@@ -18,9 +18,8 @@ module Krakow
           connections.each do |connection|
             set_ready_for(connection, :force)
           end
-          # TODO: Make interval configurable
           watch_dog.cancel if watch_dog
-          @watch_dog = every(5) do
+          @watch_dog = every(watch_dog_interval) do
             force_unready
           end
         else
@@ -46,10 +45,16 @@ module Krakow
 
       # Returns next connection to receive RDY count
       def less_than_ideal_ready!
-        if(less_than_ideal_stack.nil? || less_than_ideal_stack.empty?)
-          @less_than_ideal_stack = waiting_connections
+        admit_defeat = false
+        connection = nil
+        until(connection || (admit_defeat && less_than_ideal_stack.empty?))
+          if(less_than_ideal_stack.nil? || less_than_ideal_stack.empty?)
+            @less_than_ideal_stack = waiting_connections
+            admit_defeat = true
+          end
+          con = less_than_ideal_stack.pop
+          connection = con unless registry_lookup(con)[:backoff_until] > Time.now.to_i
         end
-        connection = less_than_ideal_stack.pop
         if(connection)
           registry_lookup(connection)[:ready] = 1
           connection
@@ -61,21 +66,16 @@ module Krakow
       # Provides customized RDY set when less than ideal to round
       # robin through connections
       def set_ready_for(connection, *args)
-        if(less_than_ideal?)
-          if(args.include?(:force))
-            super connection
+        super connection
+        if(less_than_ideal? && !args.include?(:force))
+          debug "RDY set ignored due to less than ideal state (con: #{connection})"
+          con = less_than_ideal_ready!
+          if(con)
+            watch_dog.reset if watch_dog
+            super con
           else
-            debug "RDY set ignored due to less than ideal state (con: #{connection})"
-            con = less_than_ideal_ready!
-            if(con)
-              watch_dog.reset if watch_dog
-              super con
-            else
-              warn 'Failed to set RDY state while less than ideal. Connection stack is empty!'
-            end
+            warn 'Failed to set RDY state while less than ideal. Connection stack is empty!'
           end
-        else
-          super connection
         end
       end
 
@@ -85,7 +85,14 @@ module Krakow
         registry_info = registry_lookup(connection)
         unless(less_than_ideal?)
           registry_info[:ready] = ideal - registry_info[:in_flight]
-          registry_info[:ready] = 0 if registry_info[:ready] < 0
+          if(registry_info[:ready] < 0 || registry_info[:backoff_until] > Time.now.to_i)
+            registry_info[:ready] = 0
+            registry_info[:backoff_timer].cancel if registry[:backoff_timer]
+            registry_info[:backoff_timer] = after(registry_info[:backoff_until] - Time.now.to_i) do
+              calculate_ready!(connection)
+              set_ready_for(conection) unless less_than_ideal?
+            end
+          end
           registry_info[:ready]
         else
           registry_info[:ready] = 0
@@ -95,7 +102,7 @@ module Krakow
       # Returns all connections without RDY state
       def waiting_connections
         registry.find_all do |connection, info|
-          info[:ready] < 1 && info[:in_flight] < 1
+          info[:ready] < 1 && info[:in_flight] < 1 && info[:backoff_until] < Time.now.to_i
         end.map(&:first).compact
       end
 

data/lib/krakow/distribution.rb

@@ -12,6 +12,8 @@ module Krakow
 
     def initialize(args={})
       super
+      optional :watch_dog_interval, :backoff_interval
+      arguments[:watch_dog_interval] ||= 5
       @max_in_flight = arguments[:max_in_flight] || 1
       @ideal = 0
       @flight_record = {}
@@ -35,8 +37,7 @@ module Krakow
     def unregister_message(message)
       msg_id = message.respond_to?(:message_id) ? message.message_id : message.to_s
       connection = flight_record[msg_id]
-      # TODO: Add lookup error
-      registry_info = registry[connection_key(connection)]
+      registry_info = registry_lookup(connection)
       flight_record.delete(msg_id)
       registry_info[:in_flight] -= 1
       calculate_ready!(connection)
@@ -51,7 +52,7 @@ module Krakow
 
     # connection:: Connection
     # Send RDY for given connection
-    def set_ready_for(connection)
+    def set_ready_for(connection, *_)
       connection.transmit(
         Command::Rdy.new(
           :count => ready_for(connection)
@@ -79,7 +80,9 @@ module Krakow
     def add_connection(connection)
       registry[connection_key(connection)] = {
         :ready => initial_ready,
-        :in_flight => 0
+        :in_flight => 0,
+        :failures => 0,
+        :backoff_until => 0
       }
       true
     end
@@ -90,7 +93,7 @@ module Krakow
       # remove connection from registry
       registry.delete(connection_key(connection))
       # remove any in flight messages
-      in_flight.delete_if do |k,v|
+      flight_record.delete_if do |k,v|
         v == connection_key(connection)
       end
       true
@@ -99,7 +102,7 @@ module Krakow
     # connection:: Connection
     # Return lookup key (actor reference)
     def connection_key(connection)
-      connection.current_actor
+      connection
     end
 
     # msg_id:: Message ID string
@@ -109,7 +112,7 @@ module Krakow
     def in_flight_lookup(msg_id)
       connection = flight_record[msg_id]
       unless(connection)
-        abort LookupFailed.new("Failed to locate in flight message (ID: #{msg_id})")
+        abort Krakow::Error::LookupFailed.new("Failed to locate in flight message (ID: #{msg_id})")
       end
       if(block_given?)
         yield connection
@@ -122,7 +125,7 @@ module Krakow
     # Return registry information for given connection
     def registry_lookup(connection)
       registry[connection_key(connection)] ||
-        abort(LookupFailed.new("Failed to locate connection information in registry (#{connection})"))
+        abort(Krakow::Error::LookupFailed.new("Failed to locate connection information in registry (#{connection})"))
     end
 
     # Return list of all connections in registry
@@ -130,5 +133,30 @@ module Krakow
       registry.keys
     end
 
+    # connection:: Connection
+    # Log failure of processed message
+    def failure(connection)
+      if(backoff_interval)
+        registry_info = registry_lookup(connection)
+        registry_info[:failures] += 1
+        registry_info[:backoff_until] = Time.now.to_i + (registry_info[:failures] * backoff_interval)
+      end
+      true
+    end
+
+    # connection:: Connection
+    # Log success of processed message
+    def success(connection)
+      if(backoff_interval)
+        registry_info = registry_lookup(connection)
+        if(registry_info[:failures] > 1)
+          registry_info[:failures] -= 1
+          registry_info[:backoff_until] = Time.now.to_i + (registry_info[:failures] * backoff_interval)
+        else
+          registry_info[:failures] = 0
+        end
+      end
+    end
+
   end
 end

data/lib/krakow/producer/http.rb

@@ -0,0 +1,115 @@
+require 'http'
+require 'uri'
+
+module Krakow
+  class Producer
+    class Http
+
+      include Utils::Lazy
+
+      attr_reader :uri
+
+      def initialize(args={})
+        required! :endpoint, :topic
+        @uri = URI.parse(endpoint)
+      end
+
+      def send_message(method, path, args={})
+        build = uri.dup
+        build.path = "/#{path}"
+        HTTP.send(method, build.to_s, args)
+      end
+
+      def write(*payload)
+        if(payload.size == 1)
+          payload = payload.first
+          send_message(:post, :pub,
+            :body => payload,
+            :params => {:topic => topic}
+          )
+        else
+          send_message(:post, :mpub,
+            :body => payload.join("\n"),
+            :params => {:topic => topic}
+          )
+        end
+      end
+
+      def create_topic
+        send_message(:post, :create_topic,
+          :params => {:topic => topic}
+        )
+      end
+
+      def delete_topic
+        send_message(:post, :delete_topic,
+          :params => {:topic => topic}
+        )
+      end
+
+      def create_channel(chan)
+        send_message(:post, :create_channel,
+          :params => {
+            :topic => topic,
+            :channel => chan
+          }
+        )
+      end
+
+      def delete_channel(chan)
+        send_message(:post, :delete_channel,
+          :params => {
+            :topic => topic,
+            :channel => chan
+          }
+        )
+      end
+
+      def empty_topic
+        send_message(:post, :empty_topic,
+          :params => {:topic => topic}
+        )
+      end
+
+      def empty_channel(chan)
+        send_message(:post, :empty_channel,
+          :params => {
+            :topic => topic,
+            :channel => chan
+          }
+        )
+      end
+
+      def pause_channel(chan)
+        send_message(:post, :pause_channel,
+          :params => {
+            :topic => topic,
+            :channel => chan
+          }
+        )
+      end
+
+      def unpause_channel(chan)
+        send_message(:post, :unpause_channel,
+          :params => {
+            :topic => topic,
+            :channel => chan
+          }
+        )
+      end
+
+      def stats(format='json')
+        send_message(:get, :stats,
+          :params => {
+            :format => format
+          }
+        )
+      end
+
+      def ping
+        send_message(:get, :ping)
+      end
+
+    end
+  end
+end

data/lib/krakow/producer.rb

@@ -1,6 +1,8 @@
 module Krakow
   class Producer
 
+    autoload :Http, 'krakow/producer/http'
+
     include Utils::Lazy
     include Celluloid
 

data/lib/krakow/utils/lazy.rb

@@ -19,9 +19,13 @@ module Krakow
       # error if not found
       def required!(*args)
         args.each do |key|
-          unless(arguments.has_key?(key.to_sym))
+          key = key.to_sym
+          unless(arguments.has_key?(key))
             raise ArgumentError.new "Missing required option `#{key}`!"
           end
+          define_singleton_method(key) do
+            arguments[key]
+          end
         end
       end
 
@@ -33,6 +37,9 @@ module Krakow
           unless(arguments.has_key?(key))
             arguments[key] = nil
           end
+          define_singleton_method(key) do
+            arguments[key]
+          end
         end
       end
 
@@ -44,20 +51,6 @@ module Krakow
         "<#{self.class.name}:#{object_id} [#{arguments.inspect}]>"
       end
 
-      def method_missing(*args)
-        key = args.first.to_sym
-        if(arguments.has_key?(key))
-          arguments[key]
-        else
-          super
-        end
-      end
-
-      def respond_to_missing?(key, *args)
-        key = key.to_sym
-        super || arguments.has_key?(key)
-      end
-
     end
   end
 end

data/lib/krakow/version.rb

@@ -3,5 +3,5 @@ require 'krakow'
 module Krakow
   class Version < Gem::Version
   end
-  VERSION = Version.new('0.1.0')
+  VERSION = Version.new('0.1.2')
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: krakow
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.1.2
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-01-28 00:00:00.000000000 Z
+date: 2014-02-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: celluloid-io
@@ -74,6 +74,7 @@ files:
 - lib/krakow/utils/lazy.rb
 - lib/krakow/utils/logging.rb
 - lib/krakow/producer.rb
+- lib/krakow/producer/http.rb
 - lib/krakow/distribution/default.rb
 - lib/krakow/frame_type/response.rb
 - lib/krakow/frame_type/error.rb