nsq-ruby 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 03b1479ff830b7fa68dcbccaab0e2b9a9866b433
+  data.tar.gz: 3df0a88a0c40d24c252e891e3d054a3278213492
+SHA512:
+  metadata.gz: 7548b29979d49e8b22d74b93a91b1f45eda87ef57d6211d6f4b8dac0fbe44c54d498c3a918a2d914e0cb3073abd63e537b104f0ee87b4540b4f04b57c2a6db4e
+  data.tar.gz: 5d15b617fd62fd51deb3b4902e51a1d60537c30b87fd0e36f76fc49e78329d69c3e8ab83e282d66853844447898d02f827134bf480bc57c4dd039e8bb03e0d08

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2014 Wistia, Inc.
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,276 @@
+# nsq-ruby
+
+nsq-ruby is a simple NSQ client library written in Ruby.
+
+- The code is straightforward.
+- It has no dependencies.
+- It's well tested.
+
+
+## Quick start
+
+### Publish messages
+
+```Ruby
+require 'nsq'
+producer = Nsq::Producer.new(
+  nsqd: '127.0.0.1:4150',
+  topic: 'some-topic'
+)
+
+# Write a message to NSQ
+producer.write('some-message')
+
+# Write a bunch of messages to NSQ (uses mpub)
+producer.write('one', 'two', 'three', 'four', 'five')
+
+# Close the connection
+producer.terminate
+```
+
+### Consume messages
+
+```Ruby
+require 'nsq'
+consumer = Nsq::Consumer.new(
+  nsqlookupd: '127.0.0.1:4161',
+  topic: 'some-topic',
+  channel: 'some-channel'
+)
+
+# Pop a message off the queue
+msg = consumer.pop
+puts msg.body
+msg.finish
+
+# Close the connections
+consumer.terminate
+```
+
+
+## Producer
+
+### Instantialization
+
+The Nsq::Producer constructor takes the following options:
+
+| Option        | Description                            | Default            |
+|---------------|----------------------------------------|--------------------|
+| `topic`       | Topic to which to publish messages     |                    |
+| `nsqd`        | Host and port of the nsqd instance     | '127.0.0.1:4150'   |
+
+For example:
+
+```Ruby
+producer = Nsq::Producer.new(
+  nsqd: '6.7.8.9:4150',
+  topic: 'topic-of-great-esteem'
+)
+```
+
+### `#write`
+
+Publishes one or more message to nsqd. If you give it a single argument, it will
+send it to nsqd via `PUB`. If you give it multiple arguments, it will send all
+those messages to nsqd via `MPUB`. It will automatically call `to_s` on any
+arguments you give it.
+
+```Ruby
+# Send a single message via PUB
+producer.write(123)
+
+# Send three messages via MPUB
+producer.write(456, 'another-message', { key: 'value' }.to_json)
+```
+
+If it's connection to nsqd fails, it will automatically try to reconnect with
+exponential backoff. Any messages that were sent to `#write` will be queued
+and transmitted after reconnecting.
+
+**Note** we don't wait for nsqd to acknowledge our writes. As a result, if the
+connection to nsqd fails, you can lose messages. This is acceptable for our use
+cases, mostly because we are sending messages to a local nsqd instance and
+failure is very rare.
+
+### `#connected?`
+
+Returns true if it's currently connected to nsqd and false if not.
+
+### `#terminate`
+
+Closes the connection to nsqd and stops it from trying to automatically
+reconnect.
+
+This is automatically called `at_exit`, but it's good practice to close your
+producers when you're done with them.
+
+
+## Consumer
+
+### Instantialization
+
+| Option               | Description                                   | Default            |
+|----------------------|-----------------------------------------------|--------------------|
+| `topic`              | Topic to consume messages from                |                    |
+| `channel`            | Channel name for this consumer                |                    |
+| `nsqlookupd`         | Use lookupd to automatically discover nsqds   |                    |
+| `nsqd`               | Connect directly to a single nsqd instance    | '127.0.0.1:4150'   |
+| `max_in_flight`      | Max number of messages for this consumer to have in flight at a time   | 1 |
+| `discovery_interval` | Seconds between queue discovery via nsqlookupd    | 60.0   |
+| `msg_timeout`        | Milliseconds before nsqd will timeout a message   | 60000  |
+
+
+For example:
+
+```Ruby
+consumer = Nsq::Consumer.new(
+  topic: 'the-topic',
+  channel: 'my-channel',
+  nsqlookupd: ['127.0.0.1:4161', '4.5.6.7:4161'],
+  max_in_flight: 100,
+  discovery_interval: 30,
+  msq_timeout: 120_000
+)
+```
+
+Notes:
+
+- `nsqlookupd` can be a string or array of strings for each nsqlookupd service
+  you'd like to use. The format is `"<host>:<http-port>"`. If you specify
+  `nsqlookupd`, it ignores the `nsqd` option.
+- `max_in_flight` is for the total max in flight across all the connections,
+  but to make the implementation of `nsq-ruby` as simple as possible, the minimum
+  `max_in_flight` _per_ connection is 1. So if you set `max_in_flight` to 1 and
+  are connected to 3 nsqds, you may have up to 3 messages in flight at a time.
+
+
+### `#pop`
+
+`nsq-ruby` works by maintaining a local queue of in flight messages from NSQ.
+To get at these messages, just call pop.
+
+```Ruby
+message = consumer.pop
+```
+
+If there are messages on the queue, `pop` will return one immediately. If there
+are no messages on the queue, `pop` will block execution until one arrives.
+
+
+### `#size`
+
+`size` returns the size of the local message queue.
+
+
+### `#terminate`
+
+Gracefully closes all connections and stops the consumer. You should call this
+when you're finished with a consumer object.
+
+
+## Message
+
+The `Message` object is what you get when you call `pop` on a consumer.
+Once you have a message, you'll likely want to get its contents using the `#body`
+method, and then call `#finish` once you're done with it.
+
+### `body`
+
+Returns the body of the message as a UTF-8 encoded string.
+
+### `attempts`
+
+Returns the number of times this message was attempted to be processed. For
+most messages this should be 1 (since it will be your first attempt processing
+them). If it's more than 1, that means that you requeued the message or it
+timed out in flight.
+
+### `#finish`
+
+Notify NSQ that you've completed processing of this message.
+
+### `#touch`
+
+Tells NSQ to reset the message timeout for this message so you have more time
+to process it.
+
+### `#requeue(timeout = 0)`
+
+Tells NSQ to requeue this message. Called with no arguments, this will requeue
+the message and it will be available to be received immediately.
+
+Optionally you can pass a number of milliseconds as an argument. This tells
+NSQ to delay its requeueing by that number of milliseconds.
+
+
+## Logging
+
+By default, `nsq-ruby` doesn't log anything. To enable logging, use
+`Nsq.logger=` and point it at a Ruby Logger instance. Like this:
+
+```Ruby
+Nsq.logger = Logger.new(STDOUT)
+```
+
+
+## Requirements
+
+NSQ v0.2.29 or later (due to IDENTITY metadata specification (0.2.28 and per-
+connection timeout support (0.2.29).
+
+
+### Supports
+
+- Discovery via nsqlookupd
+- Automatic reconnection to nsqd
+
+### Does not support
+
+- TLS
+- Compression
+- Backoff
+- Authentication
+
+If you need more advanced features, like these, you should check out
+[Krakow](https://github.com/chrisroberts/krakow), a more fully featured NSQ
+client for Ruby.
+
+
+## Testing
+
+Run the tests like this:
+
+```
+rake spec
+```
+
+Want a deluge of logging while running the specs to help determine what is
+going on?
+
+```
+VERBOSE=true rake spec
+```
+
+
+## MIT License
+
+Copyright (C) 2014 Wistia, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+of the Software, and to permit persons to whom the Software is furnished to do
+so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

data/lib/nsq.rb

@@ -0,0 +1,11 @@
+require_relative 'version'
+
+require_relative 'nsq/logger'
+
+require_relative 'nsq/frames/frame'
+require_relative 'nsq/frames/error'
+require_relative 'nsq/frames/response'
+require_relative 'nsq/frames/message'
+
+require_relative 'nsq/consumer'
+require_relative 'nsq/producer'

data/lib/nsq/connection.rb

@@ -0,0 +1,391 @@
+require 'json'
+require 'socket'
+require 'timeout'
+
+require_relative 'frames/error'
+require_relative 'frames/message'
+require_relative 'frames/response'
+require_relative 'logger'
+
+module Nsq
+  class Connection
+    include Nsq::AttributeLogger
+    @@log_attributes = [:host, :port]
+
+    attr_reader :host
+    attr_reader :port
+    attr_accessor :max_in_flight
+    attr_reader :presumed_in_flight
+
+    USER_AGENT = "nsq-ruby/#{Nsq::Version::STRING}"
+    RESPONSE_HEARTBEAT = '_heartbeat_'
+    RESPONSE_OK = 'OK'
+
+
+    def initialize(opts = {})
+      @host = opts[:host] || (raise ArgumentError, 'host is required')
+      @port = opts[:port] || (raise ArgumentError, 'host is required')
+      @queue = opts[:queue]
+      @topic = opts[:topic]
+      @channel = opts[:channel]
+      @msg_timeout = opts[:msg_timeout] || 60_000 # 60s
+      @max_in_flight = opts[:max_in_flight] || 1
+
+      if @msg_timeout < 1000
+        raise ArgumentError, 'msg_timeout cannot be less than 1000. it\'s in milliseconds.'
+      end
+
+      # for outgoing communication
+      @write_queue = Queue.new
+
+      # For indicating that the connection has died.
+      # We use a Queue so we don't have to poll. Used to communicate across
+      # threads (from write_loop and read_loop to connect_and_monitor).
+      @death_queue = Queue.new
+
+      @connected = false
+      @presumed_in_flight = 0
+
+      open_connection
+      start_monitoring_connection
+    end
+
+
+    def connected?
+      @connected
+    end
+
+
+    # close the connection and don't try to re-open it
+    def close
+      stop_monitoring_connection
+      close_connection
+    end
+
+
+    def sub(topic, channel)
+      write "SUB #{topic} #{channel}\n"
+    end
+
+
+    def rdy(count)
+      write "RDY #{count}\n"
+    end
+
+
+    def fin(message_id)
+      write "FIN #{message_id}\n"
+      decrement_in_flight
+    end
+
+
+    def req(message_id, timeout)
+      write "REQ #{message_id} #{timeout}\n"
+      decrement_in_flight
+    end
+
+
+    def touch(message_id)
+      write "TOUCH #{message_id}\n"
+    end
+
+
+    def pub(topic, message)
+      write ["PUB #{topic}\n", message.bytesize, message].pack('a*l>a*')
+    end
+
+
+    def mpub(topic, messages)
+      body = messages.map do |message|
+        [message.bytesize, message].pack('l>a*')
+      end.join
+
+      write ["MPUB #{topic}\n", body.bytesize, messages.size, body].pack('a*l>l>a*')
+    end
+
+
+    # Tell the server we are ready for more messages!
+    def re_up_ready
+      rdy(@max_in_flight)
+      # assume these messages are coming our way. yes, this might not be the
+      # case, but it's much easier to manage our RDY state with the server if
+      # we treat things this way.
+      @presumed_in_flight = @max_in_flight
+    end
+
+
+    private
+
+    def cls
+      write "CLS\n"
+    end
+
+
+    def nop
+      write "NOP\n"
+    end
+
+
+    def write(raw)
+      @write_queue.push(raw)
+    end
+
+
+    def write_to_socket(raw)
+      debug ">>> #{raw.inspect}"
+      @socket.write(raw)
+    end
+
+
+    # Block until we get an OK from nsqd
+    def wait_for_ok
+      frame = receive_frame
+      unless frame.is_a?(Response) && frame.data == RESPONSE_OK
+        raise "Received non-OK response while IDENTIFYing: #{frame.data}"
+      end
+    end
+
+
+    def identify
+      hostname = Socket.gethostname
+      metadata = {
+        client_id: Socket.gethostbyname(hostname).flatten.compact.first,
+        hostname: hostname,
+        feature_negotiation: false,
+        heartbeat_interval: 30_000, # 30 seconds
+        output_buffer: 16_000, # 16kb
+        output_buffer_timeout: 250, # 250ms
+        tls_v1: false,
+        snappy: false,
+        deflate: false,
+        sample_rate: 0, # disable sampling
+        user_agent: USER_AGENT,
+        msg_timeout: @msg_timeout
+      }.to_json
+      write_to_socket ["IDENTIFY\n", metadata.length, metadata].pack('a*l>a*')
+    end
+
+
+    def handle_response(frame)
+      if frame.data == RESPONSE_HEARTBEAT
+        debug 'Received heartbeat'
+        nop
+      elsif frame.data == RESPONSE_OK
+        debug 'Received OK'
+      else
+        die "Received response we don't know how to handle: #{frame.data}"
+      end
+    end
+
+
+    def receive_frame
+      if buffer = @socket.read(8)
+        size, type = buffer.unpack('l>l>')
+        size -= 4 # we want the size of the data part and type already took up 4 bytes
+        data = @socket.read(size)
+        frame_class = frame_class_for_type(type)
+        return frame_class.new(data, self)
+      end
+    end
+
+
+    FRAME_CLASSES = [Response, Error, Message]
+    def frame_class_for_type(type)
+      raise "Bad frame type specified: #{type}" if type > FRAME_CLASSES.length - 1
+      [Response, Error, Message][type]
+    end
+
+
+    def decrement_in_flight
+      @presumed_in_flight -= 1
+
+      # now that we're less than @max_in_flight we might need to re-up our RDY
+      # state
+      threshold = (@max_in_flight * 0.2).ceil
+      re_up_ready if @presumed_in_flight <= threshold
+    end
+
+
+    def start_read_loop
+      @read_loop_thread ||= Thread.new{read_loop}
+    end
+
+
+    def stop_read_loop
+      @read_loop_thread.kill if @read_loop_thread
+      @read_loop_thread = nil
+    end
+
+
+    def read_loop
+      loop do
+        frame = receive_frame
+        if frame.is_a?(Response)
+          handle_response(frame)
+        elsif frame.is_a?(Error)
+          error "Error received: #{frame.data}"
+        elsif frame.is_a?(Message)
+          debug "<<< #{frame.body}"
+          @queue.push(frame) if @queue
+        else
+          raise 'No data from socket'
+        end
+      end
+    rescue Exception => ex
+      die(ex)
+    end
+
+
+    def start_write_loop
+      @write_loop_thread ||= Thread.new{write_loop}
+    end
+
+
+    def stop_write_loop
+      @stop_write_loop = true
+      @write_loop_thread.join(1) if @write_loop_thread
+      @write_loop_thread = nil
+    end
+
+
+    def write_loop
+      @stop_write_loop = false
+      data = nil
+      loop do
+        data = @write_queue.pop
+        write_to_socket(data)
+        break if @stop_write_loop && @write_queue.size == 0
+      end
+    rescue Exception => ex
+      # requeue PUB and MPUB commands
+      if data =~ /^M?PUB/
+        debug "Requeueing to write_queue: #{data.inspect}"
+        @write_queue.push(data)
+      end
+      die(ex)
+    end
+
+
+    # Waits for death of connection
+    def start_monitoring_connection
+      @connection_monitor_thread ||= Thread.new{monitor_connection}
+      @connection_monitor_thread.abort_on_exception = true
+    end
+
+
+    def stop_monitoring_connection
+      @connection_monitor_thread.kill if @connection_monitor_thread
+      @connection_monitor = nil
+    end
+
+
+    def monitor_connection
+      loop do
+        # wait for death, hopefully it never comes
+        cause_of_death = @death_queue.pop
+        warn "Died from: #{cause_of_death}"
+
+        debug 'Reconnecting...'
+        reconnect
+        debug 'Reconnected!'
+
+        # clear all death messages, since we're now reconnected.
+        # we don't want to complete this loop and immediately reconnect again.
+        @death_queue.clear
+      end
+    end
+
+
+    # close the connection if it's not already closed and try to reconnect
+    # over and over until we succeed!
+    def reconnect
+      close_connection
+      with_retries do
+        open_connection
+      end
+    end
+
+
+    def open_connection
+      @socket = TCPSocket.new(@host, @port)
+      # write the version and IDENTIFY directly to the socket to make sure
+      # it gets to nsqd ahead of anything in the `@write_queue`
+      write_to_socket '  V2'
+      identify
+      wait_for_ok
+
+      start_read_loop
+      start_write_loop
+      @connected = true
+
+      # we need to re-subscribe if there's a topic specified
+      if @topic
+        debug "Subscribing to #{@topic}"
+        sub(@topic, @channel)
+        re_up_ready
+      end
+    end
+
+
+    # closes the connection and stops listening for messages
+    def close_connection
+      cls if connected?
+      stop_read_loop
+      stop_write_loop
+      @socket = nil
+      @connected = false
+    end
+
+
+    # this is called when there's a connection error in the read or write loop
+    # it triggers `connect_and_monitor` to try to reconnect
+    def die(reason)
+      @connected = false
+      @death_queue.push(reason)
+    end
+
+
+    # Retry the supplied block with exponential backoff.
+    #
+    # Borrowed liberally from:
+    # https://github.com/ooyala/retries/blob/master/lib/retries.rb
+    def with_retries(&block)
+      base_sleep_seconds = 0.5
+      max_sleep_seconds = 300 # 5 minutes
+
+      # Let's do this thing
+      attempts = 0
+      start_time = Time.now
+
+      begin
+        attempts += 1
+        return block.call(attempts)
+
+      rescue Errno::ECONNREFUSED, Errno::ECONNRESET, Errno::EHOSTUNREACH,
+             Errno::ENETDOWN, Errno::ENETUNREACH, Errno::ETIMEDOUT, Timeout::Error => ex
+
+        raise ex if attempts >= 100
+
+        # The sleep time is an exponentially-increasing function of base_sleep_seconds.
+        # But, it never exceeds max_sleep_seconds.
+        sleep_seconds = [base_sleep_seconds * (2 ** (attempts - 1)), max_sleep_seconds].min
+        # Randomize to a random value in the range sleep_seconds/2 .. sleep_seconds
+        sleep_seconds = sleep_seconds * (0.5 * (1 + rand()))
+        # But never sleep less than base_sleep_seconds
+        sleep_seconds = [base_sleep_seconds, sleep_seconds].max
+
+        warn "Failed to connect: #{ex}. Retrying in #{sleep_seconds.round(1)} seconds."
+
+        snooze(sleep_seconds)
+
+        retry
+      end
+    end
+
+
+    # Se we can stub for testing and reconnect in a tight loop
+    def snooze(t)
+      sleep(t)
+    end
+  end
+end

data/lib/nsq/consumer.rb

@@ -0,0 +1,146 @@
+require_relative 'connection'
+require_relative 'discovery'
+require_relative 'logger'
+
+module Nsq
+  class Consumer
+    include Nsq::AttributeLogger
+    @@log_attributes = [:topic]
+
+    attr_reader :topic
+    attr_reader :max_in_flight
+    attr_reader :discovery_interval
+    attr_reader :connections
+
+    def initialize(opts = {})
+      if opts[:nsqlookupd]
+        @nsqlookupds = [opts[:nsqlookupd]].flatten
+      else
+        @nsqlookupds = []
+      end
+
+      @topic = opts[:topic] || raise(ArgumentError, 'topic is required')
+      @channel = opts[:channel] || raise(ArgumentError, 'channel is required')
+      @max_in_flight = opts[:max_in_flight] || 1
+      @discovery_interval = opts[:discovery_interval] || 60
+      @msg_timeout = opts[:msg_timeout]
+
+      # This is where we queue up the messages we receive from each connection
+      @messages = Queue.new
+
+      # This is where we keep a record of our active nsqd connections
+      # The key is a string with the host and port of the instance (e.g.
+      # '127.0.0.1:4150') and the key is the Connection instance.
+      @connections = {}
+
+      if !@nsqlookupds.empty?
+        @discovery = Discovery.new(@nsqlookupds)
+        discover_repeatedly
+      else
+        # normally, we find nsqd instances to connect to via nsqlookupd(s)
+        # in this case let's connect to an nsqd instance directly
+        add_connection(opts[:nsqd] || '127.0.0.1:4150', @max_in_flight)
+      end
+
+      at_exit{terminate}
+    end
+
+
+    def terminate
+      @discovery_thread.kill if @discovery_thread
+      drop_all_connections
+    end
+
+
+    # pop the next message off the queue
+    def pop
+      @messages.pop
+    end
+
+
+    # returns the number of messages we have locally in the queue
+    def size
+      @messages.size
+    end
+
+
+    private
+    def discover_repeatedly
+      @discovery_thread = Thread.new do
+        loop do
+          discover
+          sleep @discovery_interval
+        end
+      end
+      @discovery_thread.abort_on_exception = true
+    end
+
+
+    def discover
+      nsqds = @discovery.nsqds_for_topic(@topic)
+
+      # drop nsqd connections that are no longer in lookupd
+      missing_nsqds = @connections.keys - nsqds
+      missing_nsqds.each do |nsqd|
+        drop_connection(nsqd)
+      end
+
+      # add new ones
+      new_nsqds = nsqds - @connections.keys
+      new_nsqds.each do |nsqd|
+        # Be conservative and start new connections with RDY 1
+        # This helps ensure we don't exceed @max_in_flight across all our
+        # connections momentarily.
+        add_connection(nsqd, 1)
+      end
+
+      # balance RDY state amongst the connections
+      redistribute_ready
+    end
+
+
+    def add_connection(nsqd, max_in_flight)
+      info "+ Adding connection #{nsqd}"
+      host, port = nsqd.split(':')
+      connection = Connection.new(
+        host: host,
+        port: port,
+        topic: @topic,
+        channel: @channel,
+        queue: @messages,
+        msg_timeout: @msg_timeout,
+        max_in_flight: max_in_flight
+      )
+      @connections[nsqd] = connection
+    end
+
+
+    def drop_connection(nsqd)
+      info "- Dropping connection #{nsqd}"
+      connection = @connections.delete(nsqd)
+      connection.close
+      redistribute_ready
+    end
+
+
+    def redistribute_ready
+      @connections.values.each do |connection|
+        connection.max_in_flight = max_in_flight_per_connection
+        connection.re_up_ready
+      end
+    end
+
+
+    def drop_all_connections
+      @connections.keys.each do |nsqd|
+        drop_connection(nsqd)
+      end
+    end
+
+
+    # Be conservative, but don't set a connection's max_in_flight below 1
+    def max_in_flight_per_connection(number_of_connections = @connections.length)
+      [@max_in_flight / number_of_connections, 1].max
+    end
+  end
+end

data/lib/nsq/discovery.rb

@@ -0,0 +1,56 @@
+require 'json'
+require 'net/http'
+require 'uri'
+
+require_relative 'logger'
+
+# Connects to nsqlookup's to find the nsqd instances for a given topic
+module Nsq
+  class Discovery
+    include Nsq::AttributeLogger
+
+    # lookupd addresses must be formatted like so: '<host>:<http-port>'
+    def initialize(lookupds)
+      @lookupds = lookupds
+    end
+
+    # Given a topic, returns an array of nsqds instances that have messages for
+    # that topic.
+    #
+    # nsqd instances returned are strings in this format: '<host>:<tcp-port>'
+    #
+    #     discovery.nsqds_for_topic('some-topic')
+    #     #=> ['127.0.0.1:4150', '127.0.0.1:4152']
+    #
+    def nsqds_for_topic(topic)
+      @lookupds.map do |lookupd|
+        get_nsqds_for_topic(lookupd, topic)
+      end.flatten.uniq
+    end
+
+
+    private
+
+    def get_nsqds_for_topic(lookupd, topic)
+      uri = URI.parse("http://#{lookupd}")
+      uri.path = '/lookup'
+      uri.query = "topic=#{topic}&ts=#{Time.now.to_i}"
+      begin
+        body = Net::HTTP.get(uri)
+        data = JSON.parse(body)
+
+        if data['data'] && data['data']['producers']
+          data['data']['producers'].map do |producer|
+            "#{producer['broadcast_address']}:#{producer['tcp_port']}"
+          end
+        else
+          []
+        end
+      rescue Exception => e
+        error "Error during discovery for #{lookupd}: #{e}"
+        []
+      end
+    end
+
+  end
+end

data/lib/nsq/frames/error.rb

@@ -0,0 +1,6 @@
+require_relative 'frame'
+
+module Nsq
+  class Error < Frame
+  end
+end

data/lib/nsq/frames/frame.rb

@@ -0,0 +1,16 @@
+require_relative '../logger'
+
+module Nsq
+  class Frame
+    include Nsq::AttributeLogger
+    @@log_attributes = [:connection]
+
+    attr_reader :data
+    attr_reader :connection
+
+    def initialize(data, connection)
+      @data = data
+      @connection = connection
+    end
+  end
+end

data/lib/nsq/frames/message.rb

@@ -0,0 +1,30 @@
+require_relative 'frame'
+
+module Nsq
+  class Message < Frame
+
+    attr_reader :timestamp
+    attr_reader :attempts
+    attr_reader :id
+    attr_reader :body
+
+    def initialize(data, connection)
+      super
+      @timestamp, @attempts, @id, @body = @data.unpack('Q>s>a16a*')
+      @body.force_encoding('UTF-8')
+    end
+
+    def finish
+      connection.fin(id)
+    end
+
+    def requeue(timeout = 0)
+      connection.req(id, timeout)
+    end
+
+    def touch
+      connection.touch(id)
+    end
+
+  end
+end

data/lib/nsq/frames/response.rb

@@ -0,0 +1,6 @@
+require_relative 'frame'
+
+module Nsq
+  class Response < Frame
+  end
+end

data/lib/nsq/logger.rb

@@ -0,0 +1,38 @@
+require 'logger'
+module Nsq
+  @@logger = Logger.new(nil)
+
+
+  def self.logger
+    @@logger
+  end
+
+
+  def self.logger=(new_logger)
+    @@logger = new_logger
+  end
+
+
+  module AttributeLogger
+    def self.included(klass)
+      klass.send :class_variable_set, :@@log_attributes, []
+    end
+
+    %w(fatal error warn info debug).map{|m| m.to_sym}.each do |level|
+      define_method level do |msg|
+        Nsq.logger.send(level, "#{prefix} #{msg}")
+      end
+    end
+
+
+    private
+    def prefix
+      attrs = self.class.send(:class_variable_get, :@@log_attributes)
+      if attrs.count > 0
+        "[#{attrs.map{|a| "#{a.to_s}: #{self.send(a)}"}.join(' ')}] "
+      else
+        ''
+      end
+    end
+  end
+end

data/lib/nsq/producer.rb

@@ -0,0 +1,48 @@
+require_relative 'connection'
+require_relative 'logger'
+
+module Nsq
+  class Producer
+    include Nsq::AttributeLogger
+    @@log_attributes = [:host, :port, :topic]
+
+    attr_reader :host
+    attr_reader :port
+    attr_reader :topic
+
+
+    def initialize(opts = {})
+      @nsqd = opts[:nsqd] || '127.0.0.1:4150'
+      @host, @port = @nsqd.split(':')
+
+      @topic = opts[:topic] || raise(ArgumentError, 'topic is required')
+
+      @connection = Connection.new(host: @host, port: @port)
+
+      at_exit{terminate}
+    end
+
+
+    def write(*raw_messages)
+      # stringify them
+      messages = raw_messages.map(&:to_s)
+
+      if messages.length > 1
+        @connection.mpub(@topic, messages)
+      else
+        @connection.pub(@topic, messages.first)
+      end
+    end
+
+
+    def connected?
+      @connection.connected?
+    end
+
+
+    def terminate
+      @connection.close
+    end
+
+  end
+end

data/lib/version.rb

@@ -0,0 +1,9 @@
+module Nsq
+  module Version
+    MAJOR = 0
+    MINOR = 1
+    PATCH = 0
+    BUILD = nil
+    STRING = [MAJOR, MINOR, PATCH, BUILD].compact.join('.')
+  end
+end

metadata

@@ -0,0 +1,114 @@
+--- !ruby/object:Gem::Specification
+name: nsq-ruby
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Wistia
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-07-28 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+- !ruby/object:Gem::Dependency
+  name: jeweler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 2.0.1
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 2.0.1
+- !ruby/object:Gem::Dependency
+  name: nsq-cluster
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.2.7
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.2.7
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+description: ''
+email: dev@wistia.com
+executables: []
+extensions: []
+extra_rdoc_files:
+- LICENSE.txt
+- README.md
+files:
+- LICENSE.txt
+- README.md
+- lib/nsq.rb
+- lib/nsq/connection.rb
+- lib/nsq/consumer.rb
+- lib/nsq/discovery.rb
+- lib/nsq/frames/error.rb
+- lib/nsq/frames/frame.rb
+- lib/nsq/frames/message.rb
+- lib/nsq/frames/response.rb
+- lib/nsq/logger.rb
+- lib/nsq/producer.rb
+- lib/version.rb
+homepage: http://github.com/wistia/nsq-ruby
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.2.2
+signing_key: 
+specification_version: 4
+summary: Ruby client library for NSQ
+test_files: []