em-hiredis 0.1.1 → 0.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: e19e2546bb52eb3e8f9f4a969d5666cd04eb0f39
+  data.tar.gz: 80435303e1cf652b489855c8e8f83228e614918e
+SHA512:
+  metadata.gz: aeea4ec51cc1a5ca3f0cf1ccecc5f00513d7840f1af34460ee5c19768e14d832d3460ada718927ac92c467d95a68f9e5f170bb318ea2b19510ef9b9c01de8158
+  data.tar.gz: 15d493eb81dfe7d044c8a55d25b3b269dd2af36d9d129fd5bfbd9d598f30d828a8ca7643a7a7f82a8f264b21d4ca8a2d5f5e43f25c48029792ff8024ba1a8062

data/CHANGELOG.md

@@ -0,0 +1,19 @@
+# Changelog
+
+## 0.2.0 (2013-04-05)
+
+[NEW] Richer interface for pubsub (accessible via `client.pubsub`). See example in `examples/pubsub.rb`.
+
+[NEW] Better failure handling:
+
+  * Clients now emit the following events: connected, reconnected, disconnected, reconnect_failed (passes the number of consecutive failures)
+  * Client is considered failed after 4 consecutive failures
+  * Fails all queued commands when client failed
+  * Can now reconfiure and reconnect an exising client
+  * Reconnect timeout can be configured (defaults to 0.5s)
+
+[NEW] Added `EM::Hiredis::Lock` and `EM::Hiredis::PersistentLock`
+
+[CHANGE] When a redis command fails, the errback is now always passed an `EM::Hiredis::Error`.
+
+[FIX] Fixed info parsing for Redis 2.6

data/README.md

@@ -1,5 +1,20 @@
-Getting started
-===============
+# em-hiredis
+
+## What
+
+A Redis client for EventMachine designed to be fast and simple.
+
+## Why
+
+I wanted a client which:
+
+* used the C hiredis library to parse redis replies
+* had a convenient API for pubsub
+* exposed the state of the underlying redis connections so that custom failover logic could be written outside the library
+
+Also, <https://github.com/madsimian/em-redis> is no longer maintained.
+
+## Getting started
 
 Connect to redis:
 
@@ -10,11 +25,9 @@ Or, connect to redis with a redis URL (for a different host, port, password, DB)
 
     redis = EM::Hiredis.connect("redis://:secretpassword@example.com:9000/4")
 
-The client is a deferrable which succeeds when the underlying connection is established so you can bind to this. This isn't necessary however - any commands sent before the connection is established (or while reconnecting) will be sent to redis on connect.
-
-    redis.callback { puts "Redis now connected" }
+Commands may be sent immediately. Any commands sent while connecting to redis will be queued.
 
-All redis commands are available without any remapping of names
+All redis commands are available without any remapping of names, and return a deferrable
 
     redis.set('foo', 'bar').callback {
       redis.get('foo').callback { |value|
@@ -22,40 +35,54 @@ All redis commands are available without any remapping of names
       }
     }
 
+If redis replies with an error (for example you called a hash operation against a set or the database is full), or if the redis connection disconnects before the command returns, the deferrable will fail.
+
+    redis.sadd('aset', 'member').callback {
+      response_deferrable = redis.hget('aset', 'member')
+      response_deferrable.errback { |e|
+        p e # => #<EventMachine::Hiredis::RedisError: Error reply from redis (wrapped in redis_error)>
+        p e.redis_error # => #<RuntimeError: ERR Operation against a key holding the wrong kind of value>
+      }
+    }
+
 As a shortcut, if you're only interested in binding to the success case you can simply provide a block to any command
 
     redis.get('foo') { |value|
       p [:returned, value]
     }
 
-Handling failure
-----------------
+## Understanding the state of the connection
 
-All commands return a deferrable. In the case that redis replies with an error (for example you called a hash operation against a set), or in the case that the redis connection is broken before the command returns, the deferrable will fail. If you care about the failure case you should bind to the errback - for example:
+When a connection to redis server closes, a `:disconnected` event will be emitted and the connection will be immediately reconnect. If the connection reconnects a `:connected` event will be emitted.
 
-    redis.sadd('aset', 'member').callback {
-      response_deferrable = redis.hget('aset', 'member')
-      response_deferrable.errback { |e|
-        p e # => #<RuntimeError: ERR Operation against a key holding the wrong kind of value>
-      }
-    }
+If a reconnect fails to connect, a `:reconnect_failed` event will be emitted (rather than `:disconnected`) with the number of consecutive failures, and the connection will be retried after a timeout (defaults to 0.5s, can be set via `EM::Hiredis.reconnect_timeout=`).
 
-Pubsub
-------
+If a client fails to reconnect 4 consecutive times then a `:failed` event will be emitted, and any queued redis commands will be failed (otherwise they would be queued forever waiting for a reconnect).
 
-This example should explain things. Once a redis connection is in a pubsub state, you must make sure you only send pubsub commands.
+## Pubsub
 
+The way pubsub works in redis is that once a subscribe has been made on a connection, it's only possible to send (p)subscribe or (p)unsubscribe commands on that connection. The connection will also receive messages which are not replies to commands.
+
+The regular `EM::Hiredis::Client` no longer understands pubsub messages - this logic has been moved to `EM::Hiredis::PubsubClient`. The pubsub client can either be initialized directly (see code) or you can get one connected to the same redis server by calling `#pubsub` on an existing `EM::Hiredis::Client` instance.
+
+Pubsub can either be used in em-hiredis in a close-to-the-metal fashion, or you can use the convenience functionality for binding blocks to subscriptions if you prefer (recommended).
+
+### Close to the metal
+
+Basically just bind to `:message` and `:pmessage` events:
+
+    # Create two connections, one will be used for subscribing
     redis = EM::Hiredis.connect
-    subscriber = EM::Hiredis.connect
+    pubsub = redis.pubsub
 
-    subscriber.subscribe('bar.0')
-    subscriber.psubscribe('bar.*')
+    pubsub.subscribe('bar.0').callback { puts "Subscribed" }
+    pubsub.psubscribe('bar.*')
 
-    subscriber.on(:message) { |channel, message|
+    pubsub.on(:message) { |channel, message|
       p [:message, channel, message]
     }
 
-    subscriber.on(:pmessage) { |key, channel, message|
+    pubsub.on(:pmessage) { |key, channel, message|
       p [:pmessage, key, channel, message]
     }
 
@@ -65,17 +92,43 @@ This example should explain things. Once a redis connection is in a pubsub state
       }
     }
 
-Hacking
--------
+### Richer interface to pubsub
+
+If you pass a block to `subscribe` or `psubscribe`, the passed block will be called whenever a message arrives on that subscription:
+
+    redis = EM::Hiredis.connect
+
+    puts "Subscribing"
+    redis.pubsub.subscribe("foo") { |msg|
+      p [:sub1, msg]
+    }
+
+    redis.pubsub.psubscribe("f*") { |msg|
+      p [:sub2, msg]
+    }
+
+    EM.add_periodic_timer(1) {
+      redis.publish("foo", "Hello")
+    }
+
+    EM.add_timer(5) {
+      puts "Unsubscribing sub1"
+      redis.pubsub.unsubscribe("foo")
+    }
+
+It's possible to subscribe to the same channel multiple time and just unsubscribe a single callback using `unsubscribe_proc` or `punsubscribe_proc`.
+
+## Developing
 
 Hacking on em-hiredis is pretty simple, make sure you have Bundler installed:
 
     gem install bundler
     bundle
 
-To run all the tests:
+In order to run the tests you need to have a local redis server running on port 6379. Run all the tests:
 
-    bundle exec rake
+    # WARNING: The tests call flushdb on db 9 - this clears all keys!
+    bundle exec rake 
 
 To run an individual test:
 

data/examples/getting_started.rb

@@ -0,0 +1,14 @@
+$:.unshift(File.expand_path('../../lib', __FILE__))
+require 'em-hiredis'
+
+EM.run {
+  redis = EM::Hiredis.connect
+  
+  redis.sadd('aset', 'member').callback {
+    response_deferrable = redis.hget('aset', 'member')
+    response_deferrable.errback { |e|
+      p e # => #<RuntimeError: ERR Operation against a key holding the wrong kind of value>
+      p e.redis_error
+    }
+  }
+}

data/examples/pubsub_basics.rb

@@ -0,0 +1,24 @@
+$:.unshift(File.expand_path('../../lib', __FILE__))
+require 'em-hiredis'
+
+EM.run {
+  redis = EM::Hiredis.connect
+  
+  puts "Subscribing"
+  redis.pubsub.subscribe("foo") { |msg|
+    p [:sub1, msg]
+  }
+  
+  redis.pubsub.psubscribe("f*") { |msg|
+    p [:sub2, msg]
+  }
+  
+  EM.add_periodic_timer(1) {
+    redis.publish("foo", "Hello")
+  }
+  
+  EM.add_timer(5) {
+    puts "Unsubscribing sub1"
+    redis.pubsub.unsubscribe("foo")
+  }
+}

data/examples/pubsub_more.rb

@@ -0,0 +1,51 @@
+$:.unshift(File.expand_path('../../lib', __FILE__))
+require 'em-hiredis'
+
+EM.run {
+  redis = EM::Hiredis.connect
+  
+  # If you pass a block to subscribe it will be called whenever a message
+  # is received on this channel
+  redis.pubsub.subscribe('foo') { |message|
+    puts "Block received #{message}"
+  }
+  
+  # You can also pass any other object which responds to call if you wish
+  callback = Proc.new { |message|
+    "Proc received #{message}"
+  }
+  df = redis.pubsub.subscribe('foo', callback)
+  
+  # All calls return a deferrable
+  df.callback { |reply|
+    p [:subscription_succeeded, reply]
+  }
+  
+  # Passing such an object is useful if you want to unsubscribe
+  redis.pubsub.unsubscribe_proc('foo', callback)
+  
+  # Or if you want to call a method on a certain object
+  class Thing
+    def receive_message(message)
+      puts "Thing received #{message}"
+    end
+  end
+  redis.pubsub.subscribe('bar', Thing.new.method(:receive_message))
+  
+  # You can also get all the following raw events:
+  # message pmessage subscribe unsubscribe psubscribe punsubscribe
+  redis.pubsub.on(:message) { |channel, message|
+    p [:message_received, channel, message]
+  }
+  redis.pubsub.on(:unsubscribe) { |channel, remaining_subscriptions|
+    p [:unsubscribe_happened, channel, remaining_subscriptions]
+  }
+  
+  EM.add_timer(1) {
+    # You can also unsubscribe completely from a channel
+    redis.pubsub.unsubscribe('foo')
+    
+    # Publishing events
+    redis.publish('bar', 'Hello')
+  }
+}

data/examples/pubsub_raw.rb

@@ -0,0 +1,25 @@
+$:.unshift(File.expand_path('../../lib', __FILE__))
+require 'em-hiredis'
+
+EM.run {
+  # Create two connections, one will be used for subscribing
+  redis = EM::Hiredis.connect
+  pubsub = redis.pubsub
+
+  pubsub.subscribe('bar.0').callback { puts "Subscribed" }
+  pubsub.psubscribe('bar.*')
+
+  pubsub.on(:message) { |channel, message|
+    p [:message, channel, message]
+  }
+
+  pubsub.on(:pmessage) { |key, channel, message|
+    p [:pmessage, key, channel, message]
+  }
+
+  EM.add_periodic_timer(1) {
+    redis.publish("bar.#{rand(2)}", "hello").errback { |e|
+      p [:publisherror, e]
+    }
+  }
+}

data/lib/em-hiredis.rb

@@ -1,11 +1,26 @@
 require 'eventmachine'
-require 'uri'
 
 module EventMachine
   module Hiredis
+    # All em-hiredis errors should descend from EM::Hiredis::Error
+    class Error < RuntimeError; end
+
+    # In the case of error responses from Redis, the RuntimeError returned
+    # by ::Hiredis will be wrapped
+    class RedisError < Error
+      attr_accessor :redis_error
+    end
+
+    class << self
+      attr_accessor :reconnect_timeout
+    end
+    self.reconnect_timeout = 0.5
+
     def self.setup(uri = nil)
-      url = URI(uri || ENV["REDIS_URL"] || "redis://127.0.0.1:6379/0")
-      Client.new(url.host, url.port, url.password, url.path[1..-1])
+      uri = uri || ENV["REDIS_URL"] || "redis://127.0.0.1:6379/0"
+      client = Client.new
+      client.configure(uri)
+      client
     end
 
     def self.connect(uri = nil)
@@ -26,9 +41,14 @@ module EventMachine
         log
       end
     end
+
+    autoload :Lock, 'em-hiredis/lock'
+    autoload :PersistentLock, 'em-hiredis/persistent_lock'
   end
 end
 
 require 'em-hiredis/event_emitter'
 require 'em-hiredis/connection'
+require 'em-hiredis/base_client'
 require 'em-hiredis/client'
+require 'em-hiredis/pubsub_client'

data/lib/em-hiredis/base_client.rb

@@ -0,0 +1,200 @@
+require 'uri'
+
+module EventMachine::Hiredis
+  # Emits the following events
+  #
+  # * :connected - on successful connection or reconnection
+  # * :reconnected - on successful reconnection
+  # * :disconnected - no longer connected, when previously in connected state
+  # * :reconnect_failed(failure_number) - a reconnect attempt failed
+  #     This event is passed number of failures so far (1,2,3...)
+  # * :monitor
+  #
+  class BaseClient
+    include EventEmitter
+    include EM::Deferrable
+
+    attr_reader :host, :port, :password, :db
+
+    def initialize(host='localhost', port='6379', password=nil, db=nil)
+      @host, @port, @password, @db = host, port, password, db
+      @defs = []
+      @command_queue = []
+
+      @closing_connection = false
+      @reconnect_failed_count = 0
+      @reconnect_timer = nil
+      @failed = false
+
+      self.on(:failed) {
+        @failed = true
+        @command_queue.each do |df, _, _|
+          df.fail(Error.new("Redis connection in failed state"))
+        end
+        @command_queue = []
+      }
+    end
+
+    # Configure the redis connection to use
+    #
+    # In usual operation, the uri should be passed to initialize. This method
+    # is useful for example when failing over to a slave connection at runtime
+    #
+    def configure(uri_string)
+      uri = URI(uri_string)
+      @host = uri.host
+      @port = uri.port
+      @password = uri.password
+      path = uri.path[1..-1]
+      @db = path.to_i # Empty path => 0
+    end
+
+    def connect
+      @connection = EM.connect(@host, @port, Connection, @host, @port)
+
+      @connection.on(:closed) do
+        if @connected
+          @defs.each { |d| d.fail(Error.new("Redis disconnected")) }
+          @defs = []
+          @deferred_status = nil
+          @connected = false
+          unless @closing_connection
+            # Next tick avoids reconnecting after for example EM.stop
+            EM.next_tick { reconnect }
+          end
+          emit(:disconnected)
+          EM::Hiredis.logger.info("#{@connection} Disconnected")
+        else
+          unless @closing_connection
+            @reconnect_failed_count += 1
+            @reconnect_timer = EM.add_timer(EM::Hiredis.reconnect_timeout) {
+              @reconnect_timer = nil
+              reconnect
+            }
+            emit(:reconnect_failed, @reconnect_failed_count)
+            EM::Hiredis.logger.info("#{@connection} Reconnect failed")
+
+            if @reconnect_failed_count >= 4
+              emit(:failed)
+              self.fail(Error.new("Could not connect after 4 attempts"))
+            end
+          end
+        end
+      end
+
+      @connection.on(:connected) do
+        @connected = true
+        @reconnect_failed_count = 0
+        @failed = false
+
+        select(@db) unless @db == 0
+        auth(@password) if @password
+
+        @command_queue.each do |df, command, args|
+          @connection.send_command(command, args)
+          @defs.push(df)
+        end
+        @command_queue = []
+
+        emit(:connected)
+        EM::Hiredis.logger.info("#{@connection} Connected")
+        succeed
+
+        if @reconnecting
+          @reconnecting = false
+          emit(:reconnected)
+        end
+      end
+
+      @connection.on(:message) do |reply|
+        if RuntimeError === reply
+          raise "Replies out of sync: #{reply.inspect}" if @defs.empty?
+          deferred = @defs.shift
+          error = RedisError.new("Error reply from redis (wrapped in redis_error)")
+          error.redis_error = reply
+          deferred.fail(error) if deferred
+        else
+          handle_reply(reply)
+        end
+      end
+
+      @connected = false
+      @reconnecting = false
+
+      return self
+    end
+
+    # Indicates that commands have been sent to redis but a reply has not yet
+    # been received
+    #
+    # This can be useful for example to avoid stopping the
+    # eventmachine reactor while there are outstanding commands
+    #
+    def pending_commands?
+      @connected && @defs.size > 0
+    end
+
+    def connected?
+      @connected
+    end
+
+    def select(db, &blk)
+      @db = db
+      method_missing(:select, db, &blk)
+    end
+
+    def auth(password, &blk)
+      @password = password
+      method_missing(:auth, password, &blk)
+    end
+
+    def close_connection
+      EM.cancel_timer(@reconnect_timer) if @reconnect_timer
+      @closing_connection = true
+      @connection.close_connection_after_writing
+    end
+
+    def reconnect_connection
+      EM.cancel_timer(@reconnect_timer) if @reconnect_timer
+      reconnect
+    end
+
+    private
+
+    def method_missing(sym, *args)
+      deferred = EM::DefaultDeferrable.new
+      # Shortcut for defining the callback case with just a block
+      deferred.callback { |result| yield(result) } if block_given?
+
+      if @connected
+        @connection.send_command(sym, args)
+        @defs.push(deferred)
+      elsif @failed
+        deferred.fail(Error.new("Redis connection in failed state"))
+      else
+        @command_queue << [deferred, sym, args]
+      end
+
+      deferred
+    end
+
+    def reconnect
+      @reconnecting = true
+      @connection.reconnect @host, @port
+      EM::Hiredis.logger.info("#{@connection} Reconnecting")
+    end
+
+    def handle_reply(reply)
+      if @defs.empty?
+        if @monitoring
+          emit(:monitor, reply)
+        else
+          raise "Replies out of sync: #{reply.inspect}"
+        end
+      else
+        deferred = @defs.shift
+        deferred.succeed(reply) if deferred
+      end
+    end
+  end
+end