quredis 0.5.1

data/.gitignore

@@ -0,0 +1,21 @@
+*.gem
+*.rbc
+.bundle
+.config
+coverage
+InstalledFiles
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+
+# YARD artifacts
+.yardoc
+_yardoc
+doc/
+
+.DS_Store
+

data/Procfile

@@ -0,0 +1 @@
+web: ruby -Ilib lib/quredis/web.rb

data/README.md

@@ -0,0 +1,93 @@
+quredis
+=======
+
+quredis (pronounced ‘kurdis’) is a little DSL for redis safe queues using `brpoplpush`. `brpoplpush` command in Redis atomically pops a message from one list and pushes onto another list. This primitive is exactly what we need to build fail-safe queues.
+
+A quredis queue is identified by its name. A queredis queue is employs three redis lists. The names of redis lists (queues) can be configured for these lists but also follow a naming convention described below.
+
+
+Name | Default Name | Description
+------------- | ------------------ | ------------
+Ingress Queue | ingress:queue_name | Messages are enqueued
+Transit Queue | transit:queue_name | Messages are held while they are being processed
+Escape Queue  | escape:queue_name  | Failed messages which are not successfully processed
+
+
+Producer
+========
+
+Messages are enqueued using redis `lpush` command.
+
+    redis.lpush('ingress:queue_name', message)
+
+Consumer
+========
+To handle messages, include Quredis module. `quredis(name, options = {}, &block)` method sets things up. The block passed to the `quredis` method handles the message. Call to `start` method and let the messages flow.
+
+    class Shop
+      include Quredis
+    
+      def initialize
+        quredis :shop do |customer|
+          puts "Now serving #{customer}."
+        end
+      end
+    end
+
+    Shop.new.start
+
+Custom queue names can be passed as options to `quredis` method:
+
+    quredis :mail, 
+      {:ingress => :read_later, :transit => :reading_now, :escape => :trash} do |message|
+      puts "now reading #{message}"
+    end
+
+
+
+Configuring Redis
+-----------
+To configure redis connection, either provide a method called `redis`.
+
+Configuring Logger
+-----------
+Provide a method called `logger` to plug a custom logger.
+
+Under the Hood
+==============
+Quredis moves messages from the ingress list, `ingress:queue_name`, to a transit list, `transit:queue_name` atomically using `brpoplpush`. The operation returns the message that was moved from the ingress list to the transit list. Next, the application code processes the message and deletes the message from the transit list using red is `lrem` command.
+
+    message = redis.brpoplpush(ingress, transit, timeout)
+    if message
+      handle_message(message)
+      redis.lrem(transit, -1, message)
+    end
+
+This code snippet above handles normal execution. To handle the the cases when application code crashes or there is a hardware failure, etc, Quredis introduces a third list, called escape list. If `handle_message(message)` fails because it throws an exception, then the message is atomically removed from `transit` list and added to the `escape` list. If the application crashes or hardware fails while handling the message, the message stays in transit list to be processed later. When the application starts again, quredis runs the recovery operation where we use `rpoplpush` to move messages from transit list the the escape list. If a message is handled normally, it’s removed from the escape queue, otherwise it stays in the escape queue.
+
+    while message = redis.rpoplpush(transit, escape)
+      handle_message(message)
+      redis.lrem(escape, -1, message)
+    end
+
+Also the recovery on start automates the processing of any messages in flight during deployment, application migation, etc. shutdown and restarts.
+
+Fault Tolerence
+===============
+Quredis is tolerant against following faults:
+
+Bad Message
+-----------
+Queredis ensures that a bad message will not cause the system from moving forward. During normal execution, if StandardError or Timeout::Error is raised while handling a message, quredis removes the message from the transit queue and moves it to the escape queue atomically. It can be later handled using admin tool which allow you to either delete the message from escape queue or reenqueue to ingress queue.
+
+Application Failure
+-----------
+Queredis ensures that a non standard error, e.g., NoMemoryError, will not cause the system from moving forward. Suppose a message is causing NoMemoryError which crashes the app and the monitoring process like monit, blue pill, god, supervisor, upstart, etc, restart the application. On restart, the recovery operation will move the message to the escape queue from the transit queue and will call `handle_message(message)`. If the app crashes again, then the message stays in the escape queue and the next application restart will not fail due to this error.
+
+Hardware Failure
+-----------
+Queredis ensures that messages are moved between queues atomically which makes it fault tolrent against the application hardware failure. Quredis doesn’t protect againt any hardware problem on the redis server itself.
+
+Quredis admin tool allows deleting and enqueuing messages from transit and escape queues.
+
+ 

data/Rakefile

@@ -0,0 +1,11 @@
+require 'rake/testtask'
+
+Rake::TestTask.new do |t|
+  t.libs << 'spec'
+  t.pattern = "spec/*_spec.rb"
+end
+
+desc "Run tests"
+task :default => :test
+
+task :spec => :test

data/bin/quredis

@@ -0,0 +1,3 @@
+#!/usr/bin/env ruby
+require 'quredis/cli'
+Quredis::CLI.start

data/examples/mail.rb

@@ -0,0 +1,12 @@
+require File.join(File.dirname(__FILE__), '..', 'lib', 'quredis')
+
+class Mail
+  include Quredis
+  def initialize
+    quredis :mail, {:ingress => :read_later, :transit => :reading_now, :escape => :trash} do |message|
+      puts "now reading #{message}"
+    end
+  end	
+end
+
+Mail.new.start

data/examples/store.rb

@@ -0,0 +1,12 @@
+require File.join(File.dirname(__FILE__), '..', 'lib', 'quredis')
+
+class Store
+  include Quredis
+  def initialize
+    quredis :shop do |customer|
+      puts "servicing #{customer}"
+    end
+  end
+end
+
+Store.new.start

data/lib/quredis/admin.rb

@@ -0,0 +1,80 @@
+require 'redis'
+require 'json'
+
+
+module Quredis
+  class Admin
+
+    def initialize(opts={})
+      @options = opts
+    end
+    
+    def redis
+      @redis ||= Redis.new(@options)
+    end
+
+    def queues(params={})
+      offset = params.fetch(:offset, 0).to_i
+      limit = params.fetch(:limit, 1000).to_i
+      queue_names, total = redis.multi do |multi|
+        multi.zrange("quredis:queues", offset, offset+limit)
+        multi.zcard("quredis:queues")
+      end
+      queue_keys = queue_names.map {|queue_name| "quredis:queue:#{queue_name}"}
+      queues = redis.mget(*queue_keys).map {|str| JSON.parse(str)}
+
+      queues.each do |queue| 
+        ingress_count, transit_count, escape_count = redis.multi do |multi|
+          multi.llen(queue['ingress'])
+          multi.llen(queue['transit'])
+          multi.llen(queue['escape'])
+        end
+        queue['ingress_count'] = ingress_count
+        queue['transit_count'] = transit_count
+        queue['escape_count'] = escape_count
+      end
+      {:total => total, :offset => offset, :items => queues}
+    end
+
+    def queue(name, type)
+      str = redis.get("quredis:queue:#{name}")
+      queue = JSON.parse(str)
+      list = queue[type]
+
+      total, items = redis.multi do |multi|
+        multi.llen(list)
+        multi.lrange(list, 0, 100)
+      end
+      {:type => type, :queue => name, :list => list, :total => total, :offset => 0, :items => items}    
+    end
+
+    def destroy_queue(name)
+      queue_key = "quredis:queue:#{name}"
+      str = redis.get(queue_key)
+      queue = JSON.parse(str)
+      redis.multi do |multi|
+        multi.del(queue['escape'])
+        multi.del(queue['transit'])
+        multi.del(queue['ingress'])
+        multi.del(queue_key)
+        multi.zrem("quredis:queues", name)
+      end
+    end
+
+    def purge(name, type)
+      queue_key = "quredis:queue:#{name}"
+      str = redis.get(queue_key)
+      queue = JSON.parse(str)
+      redis.del(queue[type])
+    end
+
+    def retry(name, type)
+      queue_key = "quredis:queue:#{name}"
+      str = redis.get(queue_key)
+      queue = JSON.parse(str)
+      from = queue[type]
+      to = queue['ingress']
+      message = redis.rpoplpush(from, to) while message
+    end
+  end
+end

data/lib/quredis/cli.rb

@@ -0,0 +1,64 @@
+require 'thor'
+require 'quredis/admin'
+require 'quredis/web'
+require 'quredis/version'
+
+module Quredis
+  class CLI < Thor
+    default_task("ls")
+
+    desc "ls", "list queues"
+    method_option :host, :type => :string, :aliases => '-h'
+    method_option :port, :type => :numeric, :aliases => '-p'
+    def ls
+      table = Quredis::Admin.new(options).queues[:items].map do |q|
+        [q['name'], 
+        "#{q['ingress']} (#{q['ingress_count']})", 
+        "#{q['transit']} (#{q['transit_count']})",
+        "#{q['escape']} (#{q['escape_count']})"]
+      end
+      table.unshift ["Name", "Ingress", "Transit", "Escape"]
+
+      Thor::Base.shell.new.print_table(table, :indent => 2, :truncate => true)
+    end
+
+    desc "destroy queue_name", "destroy the specified queue queue"
+    method_option :host, :type => :string, :aliases => '-h'
+    method_option :port, :type => :numeric, :aliases => '-p'
+    def destroy(queue_name)
+      Quredis::Admin.new(options).destroy_queue(queue_name)
+    end
+
+    desc "purge queue_name queue_type", "purges queue_name queue_type"
+    method_option :host, :type => :string, :aliases => '-h'
+    method_option :port, :type => :numeric, :aliases => '-p'
+    def purge(queue_name, queue_type)
+      Quredis::Admin.new(options).purge(queue_name, queue_type)
+    end
+
+    desc "retry queue_name queue_type", "re-enqueues queue type"
+    method_option :host, :type => :string, :aliases => '-h'
+    method_option :port, :type => :numeric, :aliases => '-p'
+    def retry(queue_name, queue_type)
+      Quredis::Admin.new(options).retry(queue_name, queue_type)
+    end
+
+    desc "web", "start webapp"
+    method_option :redis_host, :type => :string, :aliases => '-H', :desc => 'redis server host'
+    method_option :redis_port, :type => :numeric, :aliases => '-P', :desc => 'redis server port'
+    method_option :bind, :type => :string, :aliases => '-b', :desc => 'webapp IP address to bind to (default: 0.0.0.0)'
+    method_option :port, :type => :numeric, :aliases => '-p', :desc => 'webapp Port to listen on'
+    method_option :server, :type => :string, :aliases => '-s', :desc => 'specify rack server/handler (default is thin)'
+    def web
+      [:redis_host, :redis_port, :bind, :port, :server].each do |option|
+        Quredis::Web.set(option, options[option]) if options[option]
+      end
+      Quredis::Web.run!
+    end
+
+    desc "version", "version"
+    def version
+      puts "Quredis #{Quredis::VERSION}"
+    end
+  end
+end