true_queue 0.9.5

data/README.md

@@ -0,0 +1,185 @@
+
+## Overview
+
+TrueQueue (as in "the one true queue") is a proxy to multiple queueing backends.
+
+The most mature backend is one based on Redis which is a homegrown set of operations over Redis hashes and sorted sets that provides:
+
+* A fast in-memory queue, with constant backups to disk.
+* Atomic add and remove operations
+* An inspectable queue: you can see what's in the queue or peek into the head of the queue without changing it.
+* A reservable remove, where if the client quits halfway, the item is put back in.
+* Priority queues
+* And delayed retrieval for items, you can set a timestamp after which the entries are slated for removal.
+
+Not to mention the biggest advantage of all: continue to use your existing Redis install!
+
+There are other backends as well: 
+
+* memory: a simple in-process memory queue using a sorted set
+* zermoq: an experimental backend built on zeromq (see bin/zeromq-memory-queue.rb)
+* amqp: an AMQP backend to work with RabbitMQ
+
+There are a set of uniform conventions regardless of the queue backend used:
+
+* Queues are created when values are added to it. All input is encoded into JSON when stored and decoded when dequeued.
+* When a queue is empty, nil is returned on remove
+* There are always 9 standard methods: add, add\_bulk, remove, peek, list, empty, size, remove\_queues, list_queues.
+
+Certain features (for e.g. a reservable remove) might not be available on all queue backends.
+
+## Dependencies
+
+Ruby version 1.9.2p290
+
+All other dependencies are in the gemspec
+
+## Install
+
+    $ gem install true_queue
+
+## Spec
+    
+    $ bundle exec guard
+
+## Usage
+
+### Connect
+
+For the redis backend,             
+                                   
+    queue = TrueQueue.queue(:redis, options = { :redis_options => { } })
+
+For the AMQP backend using bunny,
+                                   
+    queue = TrueQueue.queue(:amqp, options = { :bunny_options => { } })
+
+For the in-memory backend that only stores keys within a process space,
+
+    queue = TrueQueue.queue(:memory, options = {})
+                                   
+For the zeromq backend,            
+                                   
+    queue = TrueQueue.queue(:zeromq, options = {})
+                                   
+### Add an item
+
+    queue.add("queue_name", {:jobid => 23, :url => 'http://example.com/' })
+    
+Items can also have arbitrary metadata. They are stored alongside items and returned on a dequeue. 
+
+    queue.add("publish", {:jobid => 23, :url => 'http://example.com/' }, {'importance' => low})
+
+Certain metadata have special meaning. If you set a dequeue-timestamp to a Time object, the item will only be dequeued *after* that time. Note that it won't be dequeued exactly *at* the time, but at any time afterwards.
+
+    # only dequeue 5s after queueing
+    queue.add("publish", {:jobid => 23, :url => 'http://example.com/' }, {'dequeue-timestamp' => Time.now + 5 })
+
+Another special metadata keyword is priority.
+
+    # priority is an integer from 1 to 100. Higher priority items are dequeued first.
+    queue.add("publish", {:jobid => 23, :url => 'http://example.com/' }, {'priority' => 5})
+
+Items with priority set (or a higher priority) are always dequeued first.
+
+Note that the AMQP backend doesn't support priorities or the dequeue timestamp.
+
+### Remove an item
+
+    # dequeue
+    queue.remove("publish")
+    
+\#remove returns an array. The first element is the Ruby object in the queue, the second is the associated metadata (always a Hash).
+
+    => {:jobid => 23, :url => 'http://example.com/' }, {'priority' => 5}
+    
+\#remove also can take a block. This is the recommended way to remove an item from a queue.
+
+    # dequeue into a block
+    queue.remove do |item|
+      #process item
+      ...
+    end
+    
+When a block is passed, Queue ensures that the item is put back in case of an error within the block.
+
+Inside a block, you can also manually raise {TrueQueue::RemoveAbort} to put back the item:
+
+    # dequeue into a block
+    queue.remove do |item|
+      #this item will be put back
+      raise TrueQueue::RemoveAbort
+    end
+    
+Note: you cannot pass in a block using the zeromq or amqp queue types.
+
+Another thing to note is that unlike in other queues, **remove does not block and returns nil when the queue is empty**. So you'll have to manually call sleep(delay) and re-poll the queue. This implementation might change in the future:
+
+    loop do	# dequeue into a block
+      queue.remove do |item|
+	  	next unless item
+        #this item will be put back
+        raise TrueQueue::RemoveAbort
+      end
+	  
+	  sleep 1
+	end
+	
+### List all items in a queue
+
+This is an expensive operation, but at times, very useful!
+
+    queue.list "queue"
+
+This is not supported for the amqp queue type.
+
+### List available queues
+
+    queue.list_queues
+
+Returns an array of all queues stored in the Redis instance.
+
+### Remove queues
+
+This empties and removes all queues:
+
+    queue.remove_queues
+
+To selectively remove queues:
+
+    queue.remove_queue "queue1"
+    queue.remove_queues "queue1", "queue2"
+
+## Performance & Memory Usage
+
+See detailed analysis in spec/performance.
+
+### The Redis Backend
+
+An indicative add performance is around 100,000 values stored in 20s: 5K/s write.
+
+An indicative normal workflow performance is 200,000 values stored and retrieved in 1 minute: ~3K/s read-write
+
+It's also reasonably memory efficient because it uses hashes instead of plain strings to store values. 200,000 values used 20MB (with each value 10 bytes).
+
+### The AMQP Backend
+
+The amqp backend uses the excellent bunny gem to connect to RabbitMQ.
+
+This is slightly slower than the Redis backend: 200,000 values read-write in around 1m30s (~2K/s read-write)
+
+### The Memory Backend
+
+The memory backend only stores keys within the process space.
+
+But performance is *very* good. It does 200,000 read/write in around 5s, which is ~40K/s read/write.
+
+### The ZeroMQ Backend
+
+The zeromq backend is currently experimental. It's meant to do these things:
+
+* Very fast queue adds (5s for 100,000 keys)
+* Consistent reads
+* Eventual consistency via a persistence server
+* A listener based queue interface where a client can request a message rather than messages being pushed down the wire (i.e. 'subscribe' to a queue) (not implemented yet)
+

data/TODO.md

@@ -0,0 +1,65 @@
+
+## TODO
+* zeromq-memory-queue:
+  * should store last ACK-ed at
+  * should expose a method to get at its stats
+    * read/s
+    * write/s
+    * transactions/s
+    * queue lengths
+    * memory used
+    * last ACK received time
+* selectively tweak a queue (remove or edit items with a specified lkey from a queue)
+* write tests that checks for contention cases, when we have a lot of queues, queue items and workers.
+* flesh out the tests more for every edge case.
+* write implementation tests and split tests into behavior and implementation.
+
+## CHANGES
+
+### 20120224 (vishnu@mobme.in)
+* TrueQueue
+
+### 20111203 (vishnu@mobme.in)
+* Renaming redis_queue to queue
+
+### 20111201 (vishnu@mobme.in)
+* A persistence server for the zeromq queue based on redis
+  * The persistence server requests backlogs from the memory queue and 
+    stores updates in Redis
+  * The persistence server sents an ACK for a backlog which says Redis
+    processing has been successful.
+* The memory queue maintains a backlog of updates for which it hasn't received
+  an ACK.
+
+### 20111130 (vishnu@mobme.in)
+* An experimental zeromq backend and queue server.
+
+### 20111128 (vishnu@mobme.in)
+* Modularizing the code so that we can now use multiple backends.
+* A memory based queue backend based on a C red-black tree implementation.
+* BREAKING. Queues are now created using:
+    MobME::Infrastructure::RedisQueue.queue(backend) where backend is either one of :memory or :redis now.
+
+### 20111118 (vishnu@mobme.in)
+* Store values inside small key-ed hashes for maximum memory efficiency.
+* Major reorganization, making everything far more modular & simple.
+
+### 20111113 (vishnu@mobme.in)
+* Initial documentation in Yardoc style.
+* Adding #delete_queue(s) to clear & delete any and all queues permanently
+* Adding #peek to peek at the first element in a queue without deleting it.
+* \#remove is now multi thread and evented systems friendly.
+* \#remove can now take a block that will auto add the item back into the queue when an error happens or a {MobME::Infrastructure::RedisQueueRemoveAbort} is raised.
+* \#list to list every element in the queue. This is an expensive operation.
+* Renaming #clear to #empty
+* Renaming #delete\_queue(s) to #remove\_queue(s)
+* More comprehensive documentation in README.md and TODO.md
+* Using Yajl instead of native JSON
+* Gemifying
+
+### 20111111 (vishnu@mobme.in)
+* Organized the project into the formal MobME ruby tdd template
+* Wrote initial specs for everything implemented.
+
+### Date Uncertain
+* -- initial release --

data/bin/zeromq-memory-queue.rb

@@ -0,0 +1,5 @@
+
+require "mobme/infrastructure/queue"
+require "mobme/infrastructure/queue/zeromq/server"
+
+server = MobME::Infrastructure::Queue::ZeroMQ::Server.new()

data/bin/zeromq-persistence-server.rb

@@ -0,0 +1,5 @@
+
+require "mobme/infrastructure/queue"
+require "mobme/infrastructure/queue/zeromq/persistence_server"
+
+server = MobME::Infrastructure::Queue::ZeroMQ::PersistenceServer.new()

data/lib/mobme/infrastructure/queue.rb

@@ -0,0 +1,37 @@
+require 'yajl'
+
+module MobME
+  module Infrastructure
+    module Queue
+    end
+  end
+end
+
+require_relative 'queue/backend'
+require_relative 'queue/exceptions'
+
+module MobME
+  module Infrastructure
+    module Queue
+      def self.queue(backend, options = {})
+        case backend
+        when :memory
+          require_relative 'queue/backends/memory'
+          MobME::Infrastructure::Queue::Backends::Memory.new(options)
+        when :redis
+          require_relative 'queue/backends/redis'
+          MobME::Infrastructure::Queue::Backends::Redis.new(options)
+        when :zeromq             
+          require_relative 'queue/backends/zeromq'
+          MobME::Infrastructure::Queue::Backends::ZeroMQ.new(options)
+        when :amqp               
+          require_relative 'queue/backends/amqp'
+          MobME::Infrastructure::Queue::Backends::AMQP.new(options)
+        end
+      end
+    end
+  end
+end
+
+                  
+

data/lib/mobme/infrastructure/queue/backend.rb

@@ -0,0 +1,45 @@
+
+module MobME::Infrastructure::Queue
+  module Backends
+  end
+end
+
+class MobME::Infrastructure::Queue::Backend  
+protected
+  def score_from_metadata(dequeue_timestamp, priority)
+    if dequeue_timestamp
+      (dequeue_timestamp.to_f * 1000000).to_i
+    else
+      ((Time.now.to_f * 1000000).to_i) / (priority || 1)
+    end
+  end
+  
+  def normalize_metadata(metadata)
+    dequeue_timestamp = metadata['dequeue-timestamp']
+    if dequeue_timestamp
+      unless dequeue_timestamp.is_a? Time
+        metadata['dequeue-timestamp'] = Time.now
+      end
+    end
+
+    priority = metadata['priority']
+    if priority
+      priority = priority.to_i
+      unless priority.between?(1, 100)
+        metadata['priority'] = 1
+      end
+    end
+    
+    metadata
+  end
+  
+  def serialize_item(item, metadata)
+    Yajl.dump([item, metadata])
+  end
+  
+  def unserialize_item(value)
+    json_value = value || Yajl.dump(value) #handle nil to null
+    Yajl.load(json_value)
+  end
+end
+

data/lib/mobme/infrastructure/queue/backends/amqp.rb

@@ -0,0 +1,84 @@
+
+require "bunny"
+
+class MobME::Infrastructure::Queue::Backends::AMQP < MobME::Infrastructure::Queue::Backend
+  def initialize(options = {})
+    @bunny_options = options[:bunny_options] || {}
+    @amqp_queues = {}
+    
+    configure
+  end
+  
+  def add(queue, item, metadata = {})
+    metadata = normalize_metadata(metadata)
+    
+    #register the queue if needed
+    queue_for(queue)
+    
+    @amqp_exchange.publish(serialize_item(item, metadata), :key => queue)
+  end
+  
+  # Adds many items together
+  def add_bulk(queue, items = [])
+    items.each do |item|
+      add(queue, item, {})
+    end
+  end
+  
+  def remove(queue)
+    item = queue_for(queue).pop[:payload]
+    (:queue_empty == item) ? nil : unserialize_item(item)
+  end
+  
+  def peek(queue)
+    raise NotImplementedError, "AMQP doesn't support peek!"
+  end
+  
+  def size(queue)
+    queue_for(queue).message_count
+  end
+  
+  def list(queue)
+    raise NotImplementedError, "AMQP doesn't support list!"
+  end
+  
+  def empty(queue)
+    queue_for(queue).purge
+  end
+  
+  def list_queues
+    @amqp_queues.keys
+  end
+  
+  def remove_queues(*queues)
+    queues = list_queues if queues.empty?
+    queues.each do |queue|
+      queue_for(queue).delete
+      @amqp_queues.delete(queue)
+    end
+  end
+  alias :remove_queue :remove_queues
+  
+  private
+  def queue_for(queue)
+    if @amqp_queues[queue]
+      @amqp_queues[queue]
+    else
+      @amqp_queues[queue] = @amqp_client.queue(queue)
+    end
+  end
+  
+  def configure
+    exchange = @bunny_options.delete(:exchange) || ''
+    @amqp_client ||= Bunny.new(@bunny_options)
+    
+    connect
+    @amqp_exchange ||= @amqp_client.exchange(exchange)
+  end
+  
+  def connect
+    if :not_connected == @amqp_client.status
+      @amqp_client.start
+    end
+  end
+end

data/lib/mobme/infrastructure/queue/backends/memory.rb

@@ -0,0 +1,95 @@
+
+require "algorithms"
+
+class MobME::Infrastructure::Queue::Backends::Memory < MobME::Infrastructure::Queue::Backend
+  attr_accessor :scores
+
+  # Initialises the Queue
+  # @param [Hash] options all options to pass to the queue
+  def initialize(options = {})
+    @@queues ||= {}
+  end
+
+  def queues
+    @@queues
+  end
+
+  def add(queue, item, metadata = {})
+    metadata = normalize_metadata(metadata)
+    score = score_from_metadata(metadata['dequeue-timestamp'], metadata['priority'])
+    
+    queues[queue] ||= Containers::CRBTreeMap.new
+    queues[queue][score] = serialize_item(item, metadata)
+  end
+  
+  # Adds many items together
+  def add_bulk(queue, items = [])
+    items.each do |item|
+      add(queue, item)
+    end
+  end
+  
+  def remove(queue, &block)
+    score = queues[queue].min_key
+    
+    item = item_with_score(queue, score)
+    
+    #If a block is given
+    if block_given?
+      begin
+        block.call(item)
+      rescue MobME::Infrastructure::Queue::RemoveAbort
+        return
+      end
+      queues[queue].delete(score) if item
+    else
+      queues[queue].delete(score) if item
+      return item
+    end
+  end
+  
+  def peek(queue)
+    score = queues[queue].min_key
+    
+    item_with_score(queue, score)
+  end
+  
+  def list(queue)
+    queues[queue].inject([]) { |collect, step| collect << item_with_score(queue, step[0]) }
+  end
+  
+  def empty(queue)
+    queues[queue] = nil
+    queues[queue] = Containers::CRBTreeMap.new
+    
+    true
+  end
+  
+  def size(queue)
+    queues[queue].size
+  end
+  
+  def remove_queues(*queues_to_delete)
+    queues_to_delete = list_queues if queues_to_delete.empty?
+    queues_to_delete.each do |queue|
+      queues.delete(queue)
+    end
+  end
+  alias :remove_queue :remove_queues
+  
+  def list_queues
+    queues.keys
+  end
+  
+  private
+  def item_with_score(queue, score)
+    item = if not score
+      nil
+    elsif score > (Time.now.to_f * 1000000).to_i # We don't return future items!
+      nil
+    else
+      value = queues[queue][score]
+      unserialize_item(value)
+    end
+  end
+end

data/lib/mobme/infrastructure/queue/backends/redis.rb

@@ -0,0 +1,255 @@
+require 'redis'
+
+class MobME::Infrastructure::Queue::Backends::Redis < MobME::Infrastructure::Queue::Backend
+
+  # The namespace that all redis queue keys live inside Redis
+  NAMESPACE = 'redis:queue:'
+
+  # The set used to store all keys
+  QUEUESET = 'redis:queue:set'
+
+  # The UUID suffix for keys that store values
+  UUID_SUFFIX = ':uuid'
+
+  # The sorted set suffix for the list of all keys in a queue
+  QUEUE_SUFFIX = ':queue'
+  
+  # The hash suffix for the hash that stores values of a queue
+  VALUE_SUFFIX = ':values'
+
+  # Initialises the Queue
+  # @param [Hash] options all options to pass to the queue
+  # @option options [Hash] :redis_options is passed on to the underlying Redis client
+  def initialize(options = {})
+    redis_options = options.delete(:redis_options) || {}
+
+    # Connect to Redis!
+    connect(redis_options)
+  end
+
+  # Connect to Redis
+  # @param [Hash] options to pass to the Redis client as is
+  # @option options :connection Instead of making a new connection, the queue will reuse this existing Redis connection
+  def connect(options)
+    @redis = options.delete(:connection)
+    @redis ||= Redis.new(options)
+  end
+
+  # Add a value to a queue
+  # @param [String] queue_name The queue name to add to. 
+  # @param [Object] item is the item to add
+  # @param [Hash] metadata is stored with the item and returned.
+  # @option metadata [Time] dequeue-timestamp An item with a dequeue-timestamp is only dequeued after this timestamp.
+  # @option metadata [Integer] priority An item with a higher priority is dequeued first. Always between 1 and 100.
+  # @return [String] A unique key in the queue name where the item is set.
+  def add(queue, item, metadata = {})
+    raise ArgumentError, "Metadata must be a hash, but #{metadata.class} given" unless metadata.is_a? Hash
+
+    metadata = normalize_metadata(metadata)
+    uuid = generate_uuid(queue)
+    
+    add_to_queueset(queue)
+    write_value(queue, uuid, item, metadata)
+    add_to_queue(queue, uuid, metadata['dequeue-timestamp'], metadata['priority'])
+    
+    uuid
+  end
+  
+  # Add values to the queue in bulk
+  # This works by pipelining writes to Redis, so results are generally much faster
+  # @param [String] queue The queue name to add to
+  # @param [Array] items The items to add
+  def add_bulk(queue, items = [])
+    metadata = {}
+    
+    # UUIDs have to be in sync!
+    uuids = []
+    items.each do |item|
+      uuids << generate_uuid(queue)
+    end
+    
+    add_to_queueset(queue)
+    
+    @redis.pipelined do
+      items.each do |item|
+        uuid = uuids.shift
+
+        # write value
+        value_hash = "#{NAMESPACE}#{queue}#{VALUE_SUFFIX}"
+        @redis.hset value_hash, uuid, serialize_item(item, metadata)
+
+        # add to queue
+        queue_key = NAMESPACE + queue.to_s + QUEUE_SUFFIX
+        @redis.zadd queue_key, score_from_metadata(metadata['dequeue_timestamp'], metadata['priority']), uuid
+      end
+    end
+  end
+
+  # Remove an item from a queue.
+  # When a block is passed, the item is reserved instead and automatically put back in case of an error.
+  # Raise MobME::Infrastructure::QueueRemoveAbort within the block to manually abort the remove.
+  #
+  # @param [String] queue_name is the queue name
+  # @yield [[Object, Hash]] An optional block that is passed the item being remove alongside metadata.
+  # @return [[Object, Hash]] The item plus the metadata in the queue
+  def remove(queue, &block)
+    begin
+      # Remove the first item!
+      uuid = first_in_queue(queue)
+      if uuid
+        # If we're not able to remove the key from the set here, it means that
+        # some other thread (or evented operation) has done it before us, so
+        # the current remove is invalid and we should retry!
+        raise MobME::Infrastructure::Queue::RemoveConflictException unless remove_from_queue(queue, uuid)
+      
+        queue_item = read_value(queue, uuid)
+        
+        # When a block is given, safely reserve the queue item
+        if block_given?
+          begin
+            block.call(queue_item)
+            remove_value(queue, uuid)
+          rescue #generic error
+            put_back_in_queue(queue, uuid, queue_item)
+            
+            # And now re-raise the error
+            raise
+          rescue MobME::Infrastructure::Queue::RemoveAbort
+            put_back_in_queue(queue, uuid, queue_item)
+          end
+        else
+          remove_value(queue, uuid)
+          queue_item
+        end
+      else
+        nil
+      end
+    rescue MobME::Infrastructure::Queue::RemoveConflictException
+      retry
+    end
+  end
+  
+  # Peek into the first item in a queue without removing it
+  # @param [String] queue_name is the queue name
+  # @return [[Object, Hash]] The item plus the metadata in the queue
+  def peek(queue)
+    uuid = first_in_queue(queue)
+    read_value(queue, uuid)
+  end
+
+  # Find the size of a queue
+  # @param [String] queue_name is the queue name
+  # @return [Integer] The size of the queue
+  def size(queue)
+    queue = NAMESPACE + queue.to_s + QUEUE_SUFFIX
+    length = (@redis.zcard queue)
+  end
+  
+  # Lists all items in the queue. This is an expensive operation
+  # @param [String] queue_name is the queue name
+  # @return [Array<Object, Hash>] An array of list items, the first element the object stored and the second, metadata
+  def list(queue)
+    batch_size = 1_000 # keep this low as the time complexity of zrangebyscore is O(log(N)+M) : M -> the size
+    
+    count = 0; values = []
+    (size(queue)/batch_size + 1).times do |i|
+      limit = [(batch_size * i), batch_size]
+      uuids = range_in_queue(queue, limit)
+      batch_values = uuids.map { |uuid| read_value(queue, uuid) }
+      values.push(*batch_values)
+    end
+    
+    values
+  end
+
+  # Clear the queue
+  # @param [String] queue_name is the queue name to clear
+  def empty(queue)
+    # Delete key and value stores.
+    @redis.del "#{NAMESPACE}#{queue}#{VALUE_SUFFIX}"
+    @redis.del "#{NAMESPACE}#{queue}#{QUEUE_SUFFIX}"
+  end
+  
+  # List all queues
+  # @return [Array] A list of queues (includes empty queues that were once available)
+  def list_queues
+    @redis.smembers QUEUESET
+  end
+  
+  # Delete queues
+  # @param [optional String ...] queues A list of queues to delete. If empty, all queues are deleted.
+  def remove_queues(*queues)
+    queues = list_queues if queues.empty?
+    queues.each do |queue_name|
+      empty(queue_name)
+      remove_from_queueset(queue_name)
+    end
+  end
+  alias :remove_queue :remove_queues
+  
+  private
+  def add_to_queueset(queue)
+    @redis.sadd QUEUESET, queue
+  end
+  
+  def remove_from_queueset(queue)
+    @redis.srem QUEUESET, queue
+  end
+  
+  def first_in_queue(queue)
+    queue = NAMESPACE + queue.to_s + QUEUE_SUFFIX
+    (@redis.zrangebyscore queue, "-inf", (Time.now.to_f * 1000000).to_i, {:limit => [0, 1]}).first
+  end
+  
+  def range_in_queue(queue, limit)
+    queue = NAMESPACE + queue.to_s + QUEUE_SUFFIX
+    (@redis.zrangebyscore queue, "-inf", (Time.now.to_f * 1000000).to_i, {:limit => limit})
+  end
+  
+  def add_to_queue(queue, uuid, dequeue_timestamp, priority)
+    # zadd adds to a sorted set, which is sorted by score.
+    # When set, the dequeue_timestamp is used as the score. If not, it's just the current timestamp.
+    # When set, current timestamp is divided by the integer priority.    
+    queue = NAMESPACE + queue.to_s + QUEUE_SUFFIX 
+    @redis.zadd queue, score_from_metadata(dequeue_timestamp, priority), uuid
+  end
+  
+  def remove_from_queue(queue, uuid)
+    queue = NAMESPACE + queue.to_s + QUEUE_SUFFIX
+    (@redis.zrem queue, uuid)
+  end
+  
+  def put_back_in_queue(queue, uuid, queue_item)
+    # Put the item back in the queue
+    metadata = queue_item[1]
+    metadata = normalize_metadata(metadata)
+    add_to_queue(queue, uuid, metadata['dequeue_timestamp'], metadata['priority'])
+  end
+  
+  def write_value(queue, uuid, item, metadata)
+    value_hash = "#{NAMESPACE}#{queue}#{VALUE_SUFFIX}"
+    
+    @redis.hset value_hash, uuid, serialize_item(item, metadata)
+  end
+  
+  def read_value(queue, uuid)
+    if uuid
+      value_hash = "#{NAMESPACE}#{queue}#{VALUE_SUFFIX}"
+      
+      value = @redis.hget value_hash, uuid
+      unserialize_item(value)
+    else
+      nil
+    end
+  end
+  
+  def remove_value(queue, uuid)
+    value_hash = "#{NAMESPACE}#{queue}#{VALUE_SUFFIX}"
+    
+    @redis.hdel value_hash, uuid
+  end
+  
+  def generate_uuid(queue)
+    @redis.incr NAMESPACE + queue.to_s + UUID_SUFFIX
+  end
+end

data/lib/mobme/infrastructure/queue/backends/zeromq.rb

@@ -0,0 +1,70 @@
+
+require 'em-zeromq'
+require 'em-synchrony'
+require 'mobme/infrastructure/queue/zeromq/connection_handler'
+
+class MobME::Infrastructure::Queue::Backends::ZeroMQ < MobME::Infrastructure::Queue::Backend
+  def initialize(options = {})
+    @socket = options[:socket] || "ipc:///tmp/mobme-infrastructure-queue-messages.sock"
+    connect
+  end
+  
+  def connect
+    @context = EM::ZeroMQ::Context.new(1)
+    @pool = EM::Synchrony::ConnectionPool.new(:size => 20) do
+      @context.connect(ZMQ::REQ, @socket)
+    end
+  end
+  
+  def add(queue, item, metadata = {})
+    dispatch(:add, queue, item, metadata)
+  end
+  
+  # Adds many items together
+  def add_bulk(queue, items = [])
+    dispatch(:add_bulk, queue, items)
+  end
+  
+  # Simple remove without reserving items
+  def remove(queue)
+    dispatch(:remove, queue)
+  end
+  
+  def peek(queue)
+    dispatch(:peek, queue)
+  end
+  
+  def size(queue)
+    dispatch(:size, queue)
+  end
+  
+  def list(queue)
+    dispatch(:list, queue)
+  end
+  
+  def empty(queue)
+    dispatch(:empty, queue)
+  end
+  
+  def list_queues
+    dispatch(:list_queues)
+  end
+  
+  def remove_queues(*queues)
+    dispatch(:remove_queues, *queues)
+  end
+  alias :remove_queue :remove_queues
+  
+  private
+  def dispatch(method, *args)
+    @pool.execute(false) do |connection|
+      handler = MobME::Infrastructure::Queue::ZeroMQ::ConnectionHandler.new(connection)
+    
+      message = Marshal.dump([method, args])
+    
+      handler.send_message(message)
+      
+      Marshal.load(handler.receive_message)
+    end
+  end
+end

data/lib/mobme/infrastructure/queue/exceptions.rb

@@ -0,0 +1,6 @@
+
+class MobME::Infrastructure::Queue::RemoveConflictException < Exception; end
+
+# Raise this to abort a remove and put back the item
+class MobME::Infrastructure::Queue::RemoveAbort < Exception; end
+

data/lib/mobme/infrastructure/queue/zeromq/connection_handler.rb

@@ -0,0 +1,50 @@
+
+module MobME::Infrastructure::Queue::ZeroMQ
+  class ConnectionHandler
+    attr_accessor :request
+
+    def initialize(connection, identity = "")
+      @connection = connection
+      @connection.handler = self
+      @connection.identity = "#{identity ? "#{identity}:" : ""}#{@client_fiber.object_id}"
+      @connection.notify_readable = false
+      @connection.notify_writable = false
+
+      @send_messages_buffer = nil
+    end
+
+    def receive_message
+      EM.next_tick { @connection.register_readable }
+
+      @client_fiber = Fiber.current
+      Fiber.yield 
+    end
+
+    def send_message(*messages)
+      @send_messages_buffer = messages
+      EM.next_tick { @connection.register_writable }
+
+      @client_fiber = Fiber.current
+      Fiber.yield 
+    end
+
+    def on_readable(connection, message)
+      request = message.map(&:copy_out_string).join
+      message.each { |part| part.close }
+
+      @connection.notify_readable = false
+      @connection.notify_writable = false
+
+      @client_fiber.resume(request)
+    end
+
+    def on_writable(connection)    
+      return_value = connection.send_msg *@send_messages_buffer
+
+      @connection.notify_readable = false
+      @connection.notify_writable = false
+
+      @client_fiber.resume(return_value)
+    end
+  end
+end

data/lib/mobme/infrastructure/queue/zeromq/persistence_server.rb

@@ -0,0 +1,80 @@
+
+require 'ffi-rzmq'
+require 'mobme/infrastructure/queue'
+require 'mobme/infrastructure/queue/zeromq/connection_handler'
+require 'redis/connection/synchrony'
+require 'digest/sha1'
+require 'fileutils'
+
+module MobME::Infrastructure::Queue::ZeroMQ
+  class PersistenceServer
+    def initialize(options = {})
+      @queue = MobME::Infrastructure::Queue.queue(:redis)
+      @persistence_socket = options[:persistence_socket] || "ipc:///tmp/mobme-infrastructure-queue-persistence.sock"
+      @persistence_store_path = options[:persistence_store_path] || "/tmp"
+      @backlog_interval = options[:backlog_interval] || 10
+      
+      EM.synchrony do
+        create_snapshot_directory
+        bind
+        
+        send_backlog_requests
+      end
+    end
+    
+    def bind
+      @context = EM::ZeroMQ::Context.new(1)
+      
+      @persistence_request_server = @context.connect(ZMQ::REQ, @persistence_socket)
+    end
+    
+    def send_backlog_requests
+      loop do
+        handler = MobME::Infrastructure::Queue::ZeroMQ::ConnectionHandler.new(@persistence_request_server)
+        handler.send_message Marshal.dump("BACKLOG")
+        puts "Sent BACKLOG"
+        
+        snapshot = handler.receive_message
+        snapshot = Marshal.load(snapshot) rescue nil
+        
+        case snapshot
+        when nil
+        when false
+        else
+          dump_snapshot_to_disk(snapshot)
+          
+          if snapshot and !snapshot.empty?
+            handler.send_message Marshal.dump("ACK #{ack_signature(snapshot)}")
+          
+            # We get an OK back from the server
+            handler.receive_message
+          end
+        end
+        
+        EM::Synchrony.sleep(@backlog_interval)
+      end
+    end
+    
+    private 
+    def create_snapshot_directory
+      @persistence_store_path = Pathname.new(@persistence_store_path).join("db")
+      FileUtils.mkdir_p(@persistence_store_path)
+    end
+    
+    def dump_snapshot_to_disk(snapshot)
+      snapshot.each do |queue, items|
+        puts "Snapshotting: #{queue}"
+        marshalled_items = StringIO.new(Marshal.dump(items))
+        File.open(@persistence_store_path.join("#{queue}.marshal"), "w+") do |file|
+          file.write marshalled_items.read
+        end
+        puts "Done."
+      end
+    end
+    
+    def ack_signature(snapshot)
+      marshaled_snapshot = Marshal.dump(snapshot)
+      Digest::SHA1.hexdigest(marshaled_snapshot)
+    end
+  end
+end

data/lib/mobme/infrastructure/queue/zeromq/server.rb

@@ -0,0 +1,107 @@
+
+require 'ffi-rzmq'
+require 'digest/sha1'
+
+require 'mobme/infrastructure/queue'
+require 'mobme/infrastructure/queue/zeromq/connection_handler'
+
+require 'em-synchrony'
+require 'em-zeromq'
+
+module MobME::Infrastructure::Queue::ZeroMQ
+  class Server
+    def initialize(options = {})
+      @queue = MobME::Infrastructure::Queue.queue(:memory)
+      @messages_socket = options[:messages_socket] || "ipc:///tmp/mobme-infrastructure-queue-messages.sock"
+      @persistence_socket = options[:persistence_socket] || "ipc:///tmp/mobme-infrastructure-queue-persistence.sock"
+      
+      EM.synchrony do
+        bind
+        
+        Fiber.new { listen_to_messages }.resume
+        Fiber.new { listen_to_backlog_requests }.resume
+      end
+    end
+    
+    def bind
+      @context = EM::ZeroMQ::Context.new(1)
+      
+      @messages_reply_server = @context.bind(ZMQ::REP, @messages_socket)
+      @persistence_reply_server = @context.bind(ZMQ::REP, @persistence_socket)
+    end
+    
+    def listen_to_messages
+      loop do
+        handler = MobME::Infrastructure::Queue::ZeroMQ::ConnectionHandler.new(@messages_reply_server)
+        message = @messages_reply_server.handler.receive_message
+        
+        message = Marshal.load(message) rescue nil
+        
+        queue_return = if message
+          route_to_queue(message)
+        end
+        
+        @messages_reply_server.handler.send_message(Marshal.dump(queue_return))
+      end
+    end
+    
+    def listen_to_backlog_requests
+      loop do
+        handler = MobME::Infrastructure::Queue::ZeroMQ::ConnectionHandler.new(@persistence_reply_server)
+        message = @persistence_reply_server.handler.receive_message
+        
+        
+        message = Marshal.load(message) rescue nil
+      
+        queue_return = if message == "BACKLOG"
+          queues_snapshot
+        elsif ack_message?(message)
+          puts "Got ACK: #{signature_from_ack_message(message)}"
+          true
+        else
+          false
+        end
+        
+        @persistence_reply_server.handler.send_message(Marshal.dump(queue_return))
+      end
+    end
+    
+    private
+    def route_to_queue(message)
+      method = method_from_message(message)
+      args = arguments_from_message(message)
+      
+      begin
+        @queue.send(method, *args)
+      rescue NoMethodError
+        false
+      end
+    end
+    
+    def queues_snapshot
+      snapshot = {}
+      queues = @queue.list_queues
+      queues.each do |queue|
+        snapshot[queue] = @queue.list queue
+      end
+      
+      snapshot
+    end
+    
+    def method_from_message(message)
+      message[0]
+    end
+    
+    def arguments_from_message(message)
+      message[1]
+    end
+    
+    def ack_message?(message)
+      message.match(/^ACK (.*)/)
+    end
+    
+    def signature_from_ack_message(message)
+      message.match(/^ACK (.*)/).to_a[1]
+    end
+  end
+end

data/lib/true_queue.rb

@@ -0,0 +1,3 @@
+require_relative 'mobme/infrastructure/queue'
+
+TrueQueue = MobME::Infrastructure::Queue

metadata

@@ -0,0 +1,249 @@
+--- !ruby/object:Gem::Specification
+name: true_queue
+version: !ruby/object:Gem::Version
+  version: 0.9.5
+  prerelease: 
+platform: ruby
+authors:
+- Vishnu Gopal
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-02-24 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: &70255280344720 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *70255280344720
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: &70255280344240 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *70255280344240
+- !ruby/object:Gem::Dependency
+  name: guard
+  requirement: &70255280343780 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *70255280343780
+- !ruby/object:Gem::Dependency
+  name: guard-rspec
+  requirement: &70255280343280 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *70255280343280
+- !ruby/object:Gem::Dependency
+  name: simplecov
+  requirement: &70255280342820 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *70255280342820
+- !ruby/object:Gem::Dependency
+  name: flog
+  requirement: &70255280342340 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *70255280342340
+- !ruby/object:Gem::Dependency
+  name: yard
+  requirement: &70255280341900 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *70255280341900
+- !ruby/object:Gem::Dependency
+  name: simplecov-rcov
+  requirement: &70255280341440 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *70255280341440
+- !ruby/object:Gem::Dependency
+  name: diff-lcs
+  requirement: &70255280340980 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *70255280340980
+- !ruby/object:Gem::Dependency
+  name: rdiscount
+  requirement: &70255280340520 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *70255280340520
+- !ruby/object:Gem::Dependency
+  name: hiredis
+  requirement: &70255280340000 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.3.1
+  type: :development
+  prerelease: false
+  version_requirements: *70255280340000
+- !ruby/object:Gem::Dependency
+  name: redis
+  requirement: &70255280339440 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.2.0
+  type: :development
+  prerelease: false
+  version_requirements: *70255280339440
+- !ruby/object:Gem::Dependency
+  name: algorithms
+  requirement: &70255280339040 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *70255280339040
+- !ruby/object:Gem::Dependency
+  name: em-synchrony
+  requirement: &70255280338540 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *70255280338540
+- !ruby/object:Gem::Dependency
+  name: em-zeromq
+  requirement: &70255280337980 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *70255280337980
+- !ruby/object:Gem::Dependency
+  name: bunny
+  requirement: &70255280337200 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.7.4
+  type: :development
+  prerelease: false
+  version_requirements: *70255280337200
+- !ruby/object:Gem::Dependency
+  name: yajl-ruby
+  requirement: &70255280336680 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: *70255280336680
+description: ! 'Queue is a proxy to several queueing libraries: a homegrown queue
+  on top of Redis, an in-process memory queue for use in testing, a robust AMQP backend
+  based on Bunny, and an experimental zeromq backend'
+email:
+- vishnu@mobme.in
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- lib/mobme/infrastructure/queue/backend.rb
+- lib/mobme/infrastructure/queue/backends/amqp.rb
+- lib/mobme/infrastructure/queue/backends/memory.rb
+- lib/mobme/infrastructure/queue/backends/redis.rb
+- lib/mobme/infrastructure/queue/backends/zeromq.rb
+- lib/mobme/infrastructure/queue/exceptions.rb
+- lib/mobme/infrastructure/queue/zeromq/connection_handler.rb
+- lib/mobme/infrastructure/queue/zeromq/persistence_server.rb
+- lib/mobme/infrastructure/queue/zeromq/server.rb
+- lib/mobme/infrastructure/queue.rb
+- lib/true_queue.rb
+- bin/zeromq-memory-queue.rb
+- bin/zeromq-persistence-server.rb
+- README.md
+- TODO.md
+homepage: http://42.mobme.in/
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: 1.3.6
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.10
+signing_key: 
+specification_version: 3
+summary: A simple proxy to various queue backends.
+test_files: []