dalli 0.11.0 → 0.11.1

data/History.md

@@ -1,6 +1,12 @@
 Dalli Changelog
 =====================
 
+0.11.1
+======
+
+ - Minor fixes, doc updates.
+ - Add optional support for kgio sockets, gives a 10-15% performance boost.
+
 0.11.0
 ======
 

data/Performance.md

@@ -4,6 +4,10 @@ Performance
 Caching is all about performance, so I carefully track Dalli performance to ensure no regressions.
 Times are from a Unibody MBP 2.4Ghz Core 2 Duo running Snow Leopard.
 
+You can optionally use kgio to give Dalli a small, 10-20% performance boost: gem install kgio.
+
+*memcache-client*:
+
     Testing 1.8.5 with ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-darwin10.4.0]
                                          user     system      total        real
     set:plain:memcache-client        2.070000   0.590000   2.660000 (  2.669744)
@@ -14,6 +18,8 @@ Times are from a Unibody MBP 2.4Ghz Core 2 Duo running Snow Leopard.
     missing:ruby:memcache-client     1.900000   0.370000   2.270000 (  2.282264)
     mixed:ruby:memcache-client       4.430000   0.950000   5.380000 (  5.420251)
 
+*dalli*:
+
     Testing 0.9.0 with ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-darwin10.4.0]
                                          user     system      total        real
     set:plain:dalli                  1.610000   0.360000   1.970000 (  2.032947)
@@ -33,3 +39,31 @@ Times are from a Unibody MBP 2.4Ghz Core 2 Duo running Snow Leopard.
     multiget:ruby:dalli              0.870000   0.300000   1.170000 (  1.295592)
     missing:ruby:dalli               1.420000   0.370000   1.790000 (  1.925094)
     mixed:ruby:dalli                 2.800000   0.680000   3.480000 (  3.820694)
+
+    Testing 0.11.1 with ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-darwin10.4.0]
+    Using standard socket IO
+                                         user     system      total        real
+    set:plain:dalli                  1.570000   0.380000   1.950000 (  1.990252)
+    setq:plain:dalli                 0.460000   0.140000   0.600000 (  0.600362)
+    set:ruby:dalli                   1.630000   0.380000   2.010000 (  2.050056)
+    get:plain:dalli                  1.710000   0.410000   2.120000 (  2.156428)
+    get:ruby:dalli                   1.680000   0.410000   2.090000 (  2.120228)
+    multiget:ruby:dalli              0.860000   0.310000   1.170000 (  1.182857)
+    missing:ruby:dalli               1.540000   0.390000   1.930000 (  1.976637)
+    mixed:ruby:dalli                 3.300000   0.810000   4.110000 (  4.166230)
+    mixedq:ruby:dalli                2.530000   0.640000   3.170000 (  3.214916)
+    incr:ruby:dalli                  0.540000   0.140000   0.680000 (  0.691829)
+
+    Testing 0.11.1 with ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-darwin10.4.0]
+    Using kgio socket IO
+                                         user     system      total        real
+    set:plain:dalli                  0.800000   0.370000   1.170000 (  1.694842)
+    setq:plain:dalli                 0.460000   0.150000   0.610000 (  0.618146)
+    set:ruby:dalli                   0.860000   0.370000   1.230000 (  1.760995)
+    get:plain:dalli                  0.910000   0.390000   1.300000 (  1.860499)
+    get:ruby:dalli                   0.900000   0.390000   1.290000 (  1.809426)
+    multiget:ruby:dalli              0.720000   0.300000   1.020000 (  1.044887)
+    missing:ruby:dalli               0.770000   0.400000   1.170000 (  1.649516)
+    mixed:ruby:dalli                 1.750000   0.760000   2.510000 (  3.563845)
+    mixedq:ruby:dalli                1.730000   0.650000   2.380000 (  2.883827)
+    incr:ruby:dalli                  0.280000   0.130000   0.410000 (  0.603845)

data/README.md

@@ -44,6 +44,9 @@ Remember, Dalli **requires** memcached 1.4+.  You can check the version with `me
 
 The test suite requires memcached 1.4.3+ with SASL enabled (./configure --enable-sasl).  Currently only supports the PLAIN mechanism.
 
+Dalli has no runtime dependencies and never will.  You can optionally install the 'kgio' gem to
+give Dalli a 10-20% performance boost.
+
 
 Usage with Rails 3.0
 ---------------------------
@@ -127,6 +130,8 @@ If you have a fix you wish to provide, please fork the code, fix in your local p
 Thanks
 ------------
 
+Eric Wong - for help using his [kgio](http://unicorn.bogomips.org/kgio/index.html) library.
+
 Brian Mitchell - for his remix-stash project which was helpful when implementing and testing the binary protocol support.
 
 [NorthScale](http://northscale.com) - for their project sponsorship

data/dalli.gemspec

@@ -14,7 +14,6 @@ Gem::Specification.new do |s|
      "LICENSE",
      "README.md",
      "History.md",
-     "TODO.md",
      "Rakefile",
      "Gemfile",
      "dalli.gemspec",

data/lib/dalli.rb

@@ -1,6 +1,7 @@
 require 'dalli/client'
 require 'dalli/ring'
 require 'dalli/server'
+require 'dalli/socket'
 require 'dalli/version'
 require 'dalli/options'
 

data/lib/dalli/client.rb

@@ -1,3 +1,4 @@
+# encoding: ascii
 module Dalli
   class Client
     
@@ -43,9 +44,12 @@ module Dalli
 
     def get(key, options=nil)
       resp = perform(:get, key)
-      (!resp || resp == 'Not found') ? nil : deserialize(resp, options)
+      (!resp || resp == 'Not found') ? nil : resp
     end
 
+    ##
+    # Fetch multiple keys efficiently.
+    # Returns a hash of { 'key' => 'value', 'key2' => 'value1' }
     def get_multi(*keys)
       return {} if keys.empty?
       options = nil
@@ -55,7 +59,7 @@ module Dalli
           perform(:getkq, key)
         end
         values = ring.servers.inject({}) { |hash, s| hash.merge!(s.request(:noop)); hash }
-        values.inject(values) { |memo, (k,v)| memo[k] = deserialize(v, options); memo }
+        values.inject(values) { |memo, (k,v)| memo[k] = v; memo }
       end
     end
 
@@ -69,29 +73,46 @@ module Dalli
       val
     end
 
+    ##
+    # compare and swap values using optimistic locking.
+    # Fetch the existing value for key.
+    # If it exists, yield the value to the block.
+    # Add the block's return value as the new value for the key.
+    # Add will fail if someone else changed the value.
+    #
+    # Returns:
+    # - nil if the key did not exist.
+    # - false if the value was changed by someone else.
+    # - true if the value was successfully updated.
     def cas(key, ttl=nil, options=nil, &block)
       ttl ||= @options[:expires_in]
       (value, cas) = perform(:cas, key)
-      value = (!value || value == 'Not found') ? nil : deserialize(value, options)
+      value = (!value || value == 'Not found') ? nil : value
       if value
         newvalue = block.call(value)
-        perform(:add, key, serialize(newvalue, options), ttl, cas, options)
+        perform(:add, key, newvalue, ttl, cas, options)
       end
     end
 
     def set(key, value, ttl=nil, options=nil)
       ttl ||= @options[:expires_in]
-      perform(:set, key, serialize(value, options), ttl, options)
+      perform(:set, key, value, ttl, options)
     end
-    
+
+    ##
+    # Conditionally add a key/value pair, if the key does not already exist
+    # on the server.  Returns true if the operation succeeded.
     def add(key, value, ttl=nil, options=nil)
       ttl ||= @options[:expires_in]
-      perform(:add, key, serialize(value, options), ttl, 0, options)
+      perform(:add, key, value, ttl, 0, options)
     end
 
+    ##
+    # Conditionally add a key/value pair, only if the key already exists
+    # on the server.  Returns true if the operation succeeded.
     def replace(key, value, ttl=nil, options=nil)
       ttl ||= @options[:expires_in]
-      perform(:replace, key, serialize(value, options), ttl, options)
+      perform(:replace, key, value, ttl, options)
     end
 
     def delete(key)
@@ -148,10 +169,16 @@ module Dalli
       perform(:decr, key, amt, ttl, default)
     end
 
+    ##
+    # Collect the stats for each server.
+    # Returns a hash like { 'hostname:port' => { 'stat1' => 'value1', ... }, 'hostname2:port' => { ... } }
     def stats
       ring.servers.inject({}) { |memo, s| memo["#{s.hostname}:#{s.port}"] = s.request(:stats); memo }
     end
 
+    ##
+    # Close our connection to each server.
+    # If you perform another operation after this, the connections will be re-established.
     def close
       if @ring
         @ring.servers.map { |s| s.close }
@@ -170,18 +197,6 @@ module Dalli
       )
     end
 
-    def serialize(value, options)
-      value
-#      options && options[:raw] ? value.to_s : ::Marshal.dump(value)
-    end
-
-    def deserialize(value, options)
-      value
-    #   options && options[:raw] ? value : ::Marshal.load(value)
-    # rescue TypeError
-    #   raise Dalli::DalliError, "Invalid marshalled data in memcached: #{value}"
-    end
-
     def env_servers
       ENV['MEMCACHE_SERVERS'] ? ENV['MEMCACHE_SERVERS'].split(',') : nil
     end

data/lib/dalli/server.rb

@@ -91,14 +91,22 @@ module Dalli
     def down!(toss_exception=false)
       close
       @down_at = Time.now.to_i
-      @msg = @msg || ($! && $!.message) || ''
-      raise Dalli::NetworkError, "#{self.hostname}:#{self.port} is currently down: #{@msg}" if toss_exception
+      @error = $! && $!.class.name
+      @msg = @msg || ($! && $!.message && !$!.message.empty? && $!.message)
+      @trace = @trace || $!.backtrace
+
+      if toss_exception
+        x = Dalli::NetworkError.new("#{self.hostname}:#{self.port} is currently down: #{@error} #{@msg}")
+        x.set_backtrace @trace
+        raise x
+      end
       nil
     end
 
     def up!
       @down_at = nil
       @msg = nil
+      @trace = nil
     end
 
     def multi?
@@ -325,7 +333,21 @@ module Dalli
       end
     end
 
-    TIMEOUT = 0.5
+    def write(bytes, socket=connection)
+      begin
+        socket.write(bytes)
+      rescue SystemCallError
+        down!(true)
+      end
+    end
+
+    def read(count, socket=connection)
+      begin
+        socket.readfull(count)
+      rescue SystemCallError, Timeout::Error, EOFError
+        down!(true)
+      end
+    end
 
     def connection
       @sock ||= begin
@@ -333,60 +355,17 @@ module Dalli
           raise Dalli::NetworkError, "#{self.hostname}:#{self.port} is currently down: #{@msg}"
         end
 
-        # All this ugly code to ensure proper Socket connect timeout
         begin
-          addr = Socket.getaddrinfo(self.hostname, nil)
-          sock = Socket.new(Socket.const_get(addr[0][0]), Socket::SOCK_STREAM, 0)
-          begin
-            sock.connect_nonblock(Socket.pack_sockaddr_in(port, addr[0][3]))
-          rescue Errno::EINPROGRESS
-            resp = IO.select(nil, [sock], nil, TIMEOUT)
-            begin
-              sock.connect_nonblock(Socket.pack_sockaddr_in(port, addr[0][3]))
-            rescue Errno::EISCONN
-            end
-          end
+          sock = KSocket.open(hostname, port)
         rescue
           down!(true)
         end
-        # end ugly code
-
-        sock.setsockopt Socket::IPPROTO_TCP, Socket::TCP_NODELAY, 1
         sasl_authentication(sock) if Dalli::Server.need_auth?
         up!
         sock
       end
     end
 
-    def write(bytes)
-      begin
-        connection.write(bytes)
-      rescue SystemCallError
-        down!(true)
-      end
-    end
-
-    def read(count, socket=connection)
-      begin
-        value = ''
-        begin
-          loop do
-            value << socket.sysread(count - value.bytesize)
-            break if value.bytesize == count
-          end
-        rescue Errno::EAGAIN, Errno::EWOULDBLOCK
-          if IO.select([socket], nil, nil, TIMEOUT)
-            retry
-          else
-            raise Timeout::Error, "IO timeout"
-          end
-        end
-        value
-      rescue SystemCallError, Timeout::Error, EOFError
-        down!(true)
-      end
-    end
-
     def split(n)
       [n >> 32, 0xFFFFFFFF & n]
     end
@@ -487,7 +466,7 @@ module Dalli
 
       # negotiate
       req = [REQUEST, OPCODES[:auth_negotiation], 0, 0, 0, 0, 0, 0, 0].pack(FORMAT[:noop])
-      socket.write(req)
+      write(req, socket)
       header = read(24, socket)
       raise Dalli::NetworkError, 'No response' if !header
       (extras, type, status, count) = header.unpack(NORMAL_HEADER)

data/lib/dalli/socket.rb

@@ -0,0 +1,80 @@
+begin
+  require 'kgio'
+  puts "Using kgio socket IO" if $TESTING
+
+  class Dalli::Server::KSocket < Kgio::Socket
+    TIMEOUT = 0.5
+
+    def wait_readable
+      IO.select([self], nil, nil, TIMEOUT) || raise(Timeout::Error, "IO timeout")
+    end
+
+    def wait_writable
+      IO.select(nil, [self], nil, TIMEOUT) || raise(Timeout::Error, "IO timeout")
+    end
+
+    def self.open(host, port)
+      addr = Socket.pack_sockaddr_in(port, host)
+      sock = start(addr)
+      sock.wait_writable
+      sock
+    end
+
+    alias :write :kgio_write
+
+    def readfull(count)
+      value = ''
+      loop do
+        value << kgio_read!(count - value.bytesize)
+        break if value.bytesize == count
+      end 
+      value
+    end
+
+  end
+
+  ::Kgio.wait_readable = :wait_readable
+  ::Kgio.wait_writable = :wait_writable
+
+rescue LoadError
+  puts "Using standard socket IO" if $TESTING
+
+  class Dalli::Server::KSocket < Socket
+    TIMEOUT = 0.5
+
+    def self.open(host, port)
+      # All this ugly code to ensure proper Socket connect timeout
+      addr = Socket.getaddrinfo(host, nil)
+      sock = new(Socket.const_get(addr[0][0]), Socket::SOCK_STREAM, 0)
+      begin
+        sock.connect_nonblock(Socket.pack_sockaddr_in(port, addr[0][3]))
+      rescue Errno::EINPROGRESS
+        resp = IO.select(nil, [sock], nil, TIMEOUT)
+        begin
+          sock.connect_nonblock(Socket.pack_sockaddr_in(port, addr[0][3]))
+        rescue Errno::EISCONN
+        end
+      end
+      sock
+    end
+  
+    def readfull(count)
+      value = ''
+      begin
+        loop do
+          value << sysread(count - value.bytesize)
+          break if value.bytesize == count
+        end
+      rescue Errno::EAGAIN, Errno::EWOULDBLOCK
+        if IO.select([self], nil, nil, TIMEOUT)
+          retry
+        else
+          raise Timeout::Error, "IO timeout"
+        end
+      end
+      value
+    end
+
+  end
+
+end

data/lib/dalli/version.rb

@@ -1,3 +1,3 @@
 module Dalli
-  VERSION = '0.11.0'
+  VERSION = '0.11.1'
 end

data/test/helper.rb

@@ -1,3 +1,4 @@
+$TESTING = true
 require 'rubygems'
 # require 'simplecov'
 # SimpleCov.start

data/test/test_active_support.rb

@@ -72,7 +72,7 @@ class TestActiveSupport < Test::Unit::TestCase
           connect
           @mc.write("abc", 5, :raw => true)
           @mc.write("cba", 5, :raw => true)
-          if RAILS_VERSION < '3.0.0'
+          if RAILS_VERSION =~ /2\.3/
             assert_raise ArgumentError do
               @mc.read_multi("abc", "cba")
             end
@@ -82,13 +82,9 @@ class TestActiveSupport < Test::Unit::TestCase
 
           @dalli.write("abc", 5, :raw => true)
           @dalli.write("cba", 5, :raw => true)
-          if RAILS_VERSION < '3.0.0'
-            assert_raise ArgumentError do
-              @dalli.read_multi("abc", "cba")
-            end
-          else
-            assert_equal({'abc' => '5', 'cba' => '5' }, @dalli.read_multi("abc", "cba"))
-          end
+          # XXX: API difference between m-c and dalli.  Dalli is smarter about
+          # what it needs to unmarshal.
+          assert_equal({'abc' => '5', 'cba' => '5' }, @dalli.read_multi("abc", "cba"))
         end
       end
     end

data/test/test_network.rb

@@ -18,7 +18,7 @@ class TestNetwork < Test::Unit::TestCase
 
       should 'handle connection reset' do
         memcached_mock(lambda {|sock| sock.close }) do
-          assert_error Dalli::NetworkError, /Connection reset|end of file/ do
+          assert_error Dalli::NetworkError, /Connection reset|EOFError/ do
             dc = Dalli::Client.new('localhost:19123')
             dc.get('abc')
           end
@@ -27,7 +27,7 @@ class TestNetwork < Test::Unit::TestCase
 
       should 'handle malformed response' do
         memcached_mock(lambda {|sock| sock.write('123') }) do
-          assert_error Dalli::NetworkError, /end of file/ do
+          assert_error Dalli::NetworkError, /EOFError/ do
             dc = Dalli::Client.new('localhost:19123')
             dc.get('abc')
           end

metadata

@@ -5,8 +5,8 @@ version: !ruby/object:Gem::Version
   segments: 
   - 0
   - 11
-  - 0
-  version: 0.11.0
+  - 1
+  version: 0.11.1
 platform: ruby
 authors: 
 - Mike Perham
@@ -14,7 +14,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2010-11-03 00:00:00 -07:00
+date: 2010-11-14 00:00:00 -08:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -93,12 +93,12 @@ files:
 - lib/dalli/sasl/base.rb
 - lib/dalli/sasl/plain.rb
 - lib/dalli/server.rb
+- lib/dalli/socket.rb
 - lib/dalli/version.rb
 - lib/dalli.rb
 - LICENSE
 - README.md
 - History.md
-- TODO.md
 - Rakefile
 - Gemfile
 - dalli.gemspec

data/TODO.md

@@ -1,5 +0,0 @@
-TODO
-========
-
- * More of the memcached instruction set.  This will be done based on user demand.  Email me if the API is missing a feature you'd like to use.
- * Better API documentation