dalli 2.7.2 → 2.7.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e3197932a3392dbaf7e6c59789506a28a61e3ace
-  data.tar.gz: f4d9b560f4294773b1555beee774c94def079f81
+  metadata.gz: d251aba11eb6809a22a244864c5d8d509347dac9
+  data.tar.gz: 5d3cb28222688e09fc202179a7bc98b2546268d6
 SHA512:
-  metadata.gz: 07c253819a5a2f40002b48f44439bc4eaa784cf0154cf769d852b28ec26657e24696bc97832fe47f6c34ffcb81a4dbdeb643f53bca491085d298053d77c3098e
-  data.tar.gz: 5da4300af76b9899673044ee73fbd011cfe5ce2962028692f9e47a983a0bceaeee60493a91464be44f3b05f28409f163d5d9801aa73f81e395e3f041ae61e3fd
+  metadata.gz: 6741b307e4311f199c8bcbaf8981838f5e3a82f920e37d138663f35fc96900b5f5f040be67358381c3b4b90893cb7e30e8871b0a6bd5fe0c10d604741d0e5a1e
+  data.tar.gz: 40f41b7bfb6d9de9e99cf4fcafc06112c3776c297bfe52f9525ef45209ff92e3aa52ee8831de66ea4e07f2ab3165481dae8c8d03175a0e3bda78453df4145845

data/History.md

@@ -1,6 +1,20 @@
 Dalli Changelog
 =====================
 
+2.7.3
+==========
+
+- Assorted spec improvements
+- README changes to specify defaults for failover and compress options (keen99, #470)
+- SASL authentication changes to deal with Unicode characters (flypiggy, #477)
+- Call to_i on ttl to accomodate ActiveSupport::Duration (#494)
+- Change to implicit blocks for performance (glaucocustodio, #495)
+- Change to each_key for performance (jastix, #496)
+- Support stats settings - (dterei, #500)
+- Raise DallError if hostname canno be parsed (dannyfallon, #501)
+- Fix instrumentation for falsey values (AlexRiedler, #514)
+- Support UNIX socket configurations (r-stu31, #515)
+
 2.7.2
 ==========
 

data/README.md

@@ -44,7 +44,7 @@ If you have problems, please enter an issue.
 Installation and Usage
 ------------------------
 
-Remember, Dalli **requires** memcached 1.4+. You can check the version with `memcached -h`. Please note that memcached that Mac OS X Snow Leopard ships with is 1.2.8 and won't work. Install 1.4.x using Homebrew with
+Remember, Dalli **requires** memcached 1.4+. You can check the version with `memcached -h`. Please note that the memcached version that *Mac OS X Snow Leopard* ships with is 1.2.8 and it won't work. Install memcached 1.4.x using Homebrew with
 
     brew install memcached
 
@@ -130,7 +130,7 @@ add :pool\_size to your `dalli_store` config:
 config.cache_store = :dalli_store, 'cache-1.example.com', { :pool_size => 5 }
 ```
 
-You can then use the Rails cache as normal or check out a Dalli client directly from the pool:
+You can then use the Rails cache as normal and Rails.cache will use the pool transparently under the covers, or you can check out a Dalli client directly from the pool:
 
 ```ruby
 Rails.cache.fetch('foo', :expires_in => 300) do
@@ -151,16 +151,26 @@ Dalli::Client accepts the following options. All times are in seconds.
 
 **expires_in**: Global default for key TTL.  Default is 0, which means no expiry.
 
-**failover**: Boolean, if true Dalli will failover to another server if the main server for a key is down.
+**namespace**: If specified, prepends each key with this value to provide simple namespacing.  Default is nil.
 
-**compress**: Boolean, if true Dalli will gzip-compress values larger than 1K.
+**failover**: Boolean, if true Dalli will failover to another server if the main server for a key is down.  Default is true.
+
+**threadsafe**: Boolean.  If true Dalli ensures that only one thread is using a socket at a given time.  Default is true.  Set to false at your own peril.
+
+**serializer**: The serializer to use for objects being stored (ex. JSON).
+Default is Marshal.
+
+**compress**: Boolean, if true Dalli will gzip-compress values larger than 1K. Default is false.
 
 **compression_min_size**: Minimum value byte size for which to attempt compression. Default is 1K.
 
 **compression_max_size**: Maximum value byte size for which to attempt compression. Default is unlimited.
 
-**serializer**: The serializer to use for objects being stored (ex. JSON).
-Default is Marshal.
+**compressor**: The compressor to use for objects being stored.
+Default is zlib, implemented under `Dalli::Compressor`.
+If serving compressed data using nginx's HttpMemcachedModule, set `memcached_gzip_flag 2` and use `Dalli::GzipCompressor`
+
+**keepalive**: Boolean. If true, Dalli will enable keep-alive for socket connections.  Default is true.
 
 **socket_timeout**: Timeout for all socket operations (connect, read, write). Default is 0.5.
 
@@ -168,7 +178,7 @@ Default is Marshal.
 
 **socket_failure_delay**: Before retrying a socket operation, the process sleeps for this amount of time. Default is 0.01.  Set to nil for no delay.
 
-**down_retry_delay**: When a server has been marked down due to many failures, the server will be checked again for being alive only after this amount of time. Don't set this value to low, otherwise each request which tries the failed server might hang for the maximum **socket_timeout**. Default is 1 second.
+**down_retry_delay**: When a server has been marked down due to many failures, the server will be checked again for being alive only after this amount of time. Don't set this value too low, otherwise each request which tries the failed server might hang for the maximum **socket_timeout**. Default is 1 second.
 
 **value_max_bytes**: The maximum size of a value in memcached.  Defaults to 1MB, this can be increased with memcached's -I parameter.  You must also configure Dalli to allow the larger size here.
 
@@ -176,12 +186,6 @@ Default is Marshal.
 
 **password**: The password to use for authenticating this client instance against a SASL-enabled memcached server.  Heroku users should not need to use this normally.
 
-**keepalive**: Boolean. If true, Dalli will enable keep-alive for socket connections.  Default is true.
-
-**compressor**: The compressor to use for objects being stored.
-Default is zlib, implemented under `Dalli::Compressor`.
-If serving compressed data using nginx's HttpMemcachedModule, set `memcached_gzip_flag 2` and use `Dalli::GzipCompressor`
-
 Features and Changes
 ------------------------
 
@@ -205,7 +209,7 @@ We're not accepting new compressors. They are trivial to add in an initializer.
 Thanks
 ------------
 
-Eric Wong - for help using his [kgio](http://unicorn.bogomips.org/kgio/index.html) library.
+Eric Wong - for help using his [kgio](http://bogomips.org/kgio/) library.
 
 Brian Mitchell - for his remix-stash project which was helpful when implementing and testing the binary protocol support.
 

data/Rakefile

@@ -1,3 +1,4 @@
+require 'bundler/gem_tasks'
 require 'appraisal'
 require 'rake/testtask'
 Rake::TestTask.new(:test) do |test|

data/lib/active_support/cache/dalli_store.rb

@@ -83,15 +83,15 @@ module ActiveSupport
 
       def fetch(name, options=nil)
         options ||= {}
-        name = expanded_key name
+        namespaced_name = namespaced_key(name, options)
 
         if block_given?
           unless options[:force]
-            entry = instrument(:read, name, options) do |payload|
-              read_entry(name, options).tap do |result|
+            entry = instrument(:read, namespaced_name, options) do |payload|
+              read_entry(namespaced_name, options).tap do |result|
                 if payload
                   payload[:super_operation] = :fetch
-                  payload[:hit] = !!result
+                  payload[:hit] = !result.nil?
                 end
               end
             end
@@ -114,18 +114,18 @@ module ActiveSupport
 
       def read(name, options=nil)
         options ||= {}
-        name = expanded_key name
+        name = namespaced_key(name, options)
 
         instrument(:read, name, options) do |payload|
           entry = read_entry(name, options)
-          payload[:hit] = !!entry if payload
+          payload[:hit] = !entry.nil? if payload
           entry
         end
       end
 
       def write(name, value, options=nil)
         options ||= {}
-        name = expanded_key name
+        name = namespaced_key(name, options)
 
         instrument(:write, name, options) do |payload|
           with do |connection|
@@ -137,7 +137,7 @@ module ActiveSupport
 
       def exist?(name, options=nil)
         options ||= {}
-        name = expanded_key name
+        name = namespaced_key(name, options)
 
         log(:exist, name, options)
         !read_entry(name, options).nil?
@@ -145,7 +145,7 @@ module ActiveSupport
 
       def delete(name, options=nil)
         options ||= {}
-        name = expanded_key name
+        name = namespaced_key(name, options)
 
         instrument(:delete, name, options) do |payload|
           delete_entry(name, options)
@@ -155,12 +155,12 @@ module ActiveSupport
       # Reads multiple keys from the cache using a single call to the
       # servers for all keys. Keys must be Strings.
       def read_multi(*names)
-        names.extract_options!
-        mapping = names.inject({}) { |memo, name| memo[expanded_key(name)] = name; memo }
+        options  = names.extract_options!
+        mapping = names.inject({}) { |memo, name| memo[namespaced_key(name, options)] = name; memo }
         instrument(:read_multi, names) do
           results = {}
           if local_cache
-            mapping.keys.each do |key|
+            mapping.each_key do |key|
               if value = local_cache.read_entry(key, options)
                 results[key] = value
               end
@@ -186,7 +186,7 @@ module ActiveSupport
       # and the result will be written to the cache and returned.
       def fetch_multi(*names)
         options = names.extract_options!
-        mapping = names.inject({}) { |memo, name| memo[expanded_key(name)] = name; memo }
+        mapping = names.inject({}) { |memo, name| memo[namespaced_key(name, options)] = name; memo }
 
         instrument(:fetch_multi, names) do
           with do |connection|
@@ -216,7 +216,7 @@ module ActiveSupport
       # memcached counters cannot hold negative values.
       def increment(name, amount = 1, options=nil)
         options ||= {}
-        name = expanded_key name
+        name = namespaced_key(name, options)
         initial = options.has_key?(:initial) ? options[:initial] : amount
         expires_in = options[:expires_in]
         instrument(:increment, name, :amount => amount) do
@@ -235,7 +235,7 @@ module ActiveSupport
       # memcached counters cannot hold negative values.
       def decrement(name, amount = 1, options=nil)
         options ||= {}
-        name = expanded_key name
+        name = namespaced_key(name, options)
         initial = options.has_key?(:initial) ? options[:initial] : 0
         expires_in = options[:expires_in]
         instrument(:decrement, name, :amount => amount) do
@@ -317,6 +317,15 @@ module ActiveSupport
       end
 
       private
+
+      def namespaced_key(key, options)
+        key = expanded_key(key)
+        namespace = options[:namespace] if options
+        prefix = namespace.is_a?(Proc) ? namespace.call : namespace
+        key = "#{prefix}:#{key}" if prefix
+        key
+      end
+
       # Expand key to be a consistent string value. Invoke +cache_key+ if
       # object responds to +cache_key+. Otherwise, to_param method will be
       # called. If the key is a Hash, then keys will be sorted alphabetically.

data/lib/dalli/client.rb

@@ -9,12 +9,15 @@ module Dalli
     # Dalli::Client is the main class which developers will use to interact with
     # the memcached server.  Usage:
     #
-    #   Dalli::Client.new(['localhost:11211:10', 'cache-2.example.com:11211:5', '192.168.0.1:22122:5'],
+    #   Dalli::Client.new(['localhost:11211:10', 'cache-2.example.com:11211:5', '192.168.0.1:22122:5', '/var/run/memcached/socket'],
     #                   :threadsafe => true, :failover => true, :expires_in => 300)
     #
     # servers is an Array of "host:port:weight" where weight allows you to distribute cache unevenly.
     # Both weight and port are optional.  If you pass in nil, Dalli will use the <tt>MEMCACHE_SERVERS</tt>
-    # environment variable or default to 'localhost:11211' if it is not present.
+    # environment variable or default to 'localhost:11211' if it is not present.  Dalli also supports
+    # the ability to connect to Memcached on localhost through a UNIX socket.  To use this functionality,
+    # use a full pathname (beginning with a slash character '/') in place of the "host:port" pair in
+    # the server configuration.
     #
     # Options:
     # - :namespace - prepend each key with this value to provide simple namespacing.
@@ -58,10 +61,11 @@ module Dalli
     # If a block is given, yields key/value pairs one at a time.
     # Otherwise returns a hash of { 'key' => 'value', 'key2' => 'value1' }
     def get_multi(*keys)
+      return {} if keys.flatten.compact.empty?
       if block_given?
         get_multi_yielder(keys) {|k, data| yield k, data.first}
       else
-         Hash.new.tap do |hash|
+        Hash.new.tap do |hash|
           get_multi_yielder(keys) {|k, data| hash[k] = data.first}
         end
       end
@@ -88,12 +92,12 @@ module Dalli
     # - nil if the key did not exist.
     # - false if the value was changed by someone else.
     # - true if the value was successfully updated.
-    def cas(key, ttl=nil, options=nil, &block)
+    def cas(key, ttl=nil, options=nil)
       ttl ||= @options[:expires_in].to_i
       (value, cas) = perform(:cas, key)
       value = (!value || value == 'Not found') ? nil : value
       if value
-        newvalue = block.call(value)
+        newvalue = yield(value)
         perform(:set, key, newvalue, ttl, cas, options)
       end
     end
@@ -193,13 +197,13 @@ module Dalli
 
     ##
     # Collect the stats for each server.
-    # You can optionally pass a type including :items or :slabs to get specific stats
+    # You can optionally pass a type including :items, :slabs or :settings to get specific stats
     # Returns a hash like { 'hostname:port' => { 'stat1' => 'value1', ... }, 'hostname2:port' => { ... } }
     def stats(type=nil)
-      type = nil if ![nil, :items,:slabs].include? type
+      type = nil if ![nil, :items,:slabs,:settings].include? type
       values = {}
       ring.servers.each do |server|
-        values["#{server.hostname}:#{server.port}"] = server.alive? ? server.request(:stats,type.to_s) : nil
+        values["#{server.name}"] = server.alive? ? server.request(:stats,type.to_s) : nil
       end
       values
     end
@@ -223,7 +227,7 @@ module Dalli
     def version
       values = {}
       ring.servers.each do |server|
-        values["#{server.hostname}:#{server.port}"] = server.alive? ? server.request(:version) : nil
+        values["#{server.name}"] = server.alive? ? server.request(:version) : nil
       end
       values
     end
@@ -272,7 +276,7 @@ module Dalli
           server.request(:send_multiget, keys_for_server)
         rescue DalliError, NetworkError => e
           Dalli.logger.debug { e.inspect }
-          Dalli.logger.debug { "unable to get keys for server #{server.hostname}:#{server.port}" }
+          Dalli.logger.debug { "unable to get keys for server #{server.name}" }
         end
       end
     end

data/lib/dalli/ring.rb

@@ -15,7 +15,7 @@ module Dalli
         continuum = []
         servers.each do |server|
           entry_count_for(server, servers.size, total_weight).times do |idx|
-            hash = Digest::SHA1.hexdigest("#{server.hostname}:#{server.port}:#{idx}")
+            hash = Digest::SHA1.hexdigest("#{server.name}:#{idx}")
             value = Integer("0x#{hash[0..7]}")
             continuum << Dalli::Ring::Entry.new(value, server)
           end

data/lib/dalli/server.rb

@@ -8,7 +8,10 @@ module Dalli
     attr_accessor :weight
     attr_accessor :options
     attr_reader :sock
+    attr_reader :socket_type  # possible values: :unix, :tcp
 
+    DEFAULT_PORT = 11211
+    DEFAULT_WEIGHT = 1
     DEFAULTS = {
       # seconds between trying to contact a remote server
       :down_retry_delay => 1,
@@ -32,11 +35,7 @@ module Dalli
     }
 
     def initialize(attribs, options = {})
-      (@hostname, @port, @weight) = parse_hostname(attribs)
-      @port ||= 11211
-      @port = Integer(@port)
-      @weight ||= 1
-      @weight = Integer(@weight)
+      @hostname, @port, @weight, @socket_type = parse_hostname(attribs)
       @fail_count = 0
       @down_at = nil
       @last_down_at = nil
@@ -49,13 +48,17 @@ module Dalli
     end
 
     def name
-      "#{@hostname}:#{@port}"
+      if socket_type == :unix
+        hostname
+      else
+        "#{hostname}:#{port}"
+      end
     end
 
     # Chokepoint method for instrumentation
     def request(op, *args)
       verify_state
-      raise Dalli::NetworkError, "#{hostname}:#{port} is down: #{@error} #{@msg}. If you are sure it is running, ensure memcached version is > 1.4." unless alive?
+      raise Dalli::NetworkError, "#{name} is down: #{@error} #{@msg}. If you are sure it is running, ensure memcached version is > 1.4." unless alive?
       begin
         send(op, *args)
       rescue Dalli::NetworkError
@@ -67,9 +70,10 @@ module Dalli
         false
       rescue Dalli::DalliError
         raise
+      rescue Timeout::Error
+        raise
       rescue => ex
-        Dalli.logger.error "Unexpected exception in Dalli: #{ex.class.name}: #{ex.message}"
-        Dalli.logger.error "This is a bug in Dalli, please enter an issue in Github if it does not already exist."
+        Dalli.logger.error "Unexpected exception during Dalli request: #{ex.class.name}: #{ex.message}"
         Dalli.logger.error ex.backtrace.join("\n\t")
         down!
       end
@@ -80,7 +84,7 @@ module Dalli
 
       if @last_down_at && @last_down_at + options[:down_retry_delay] >= Time.now
         time = @last_down_at + options[:down_retry_delay] - Time.now
-        Dalli.logger.debug { "down_retry_delay not reached for #{hostname}:#{port} (%.3f seconds left)" % time }
+        Dalli.logger.debug { "down_retry_delay not reached for #{name} (%.3f seconds left)" % time }
         return false
       end
 
@@ -203,7 +207,7 @@ module Dalli
     end
 
     def failure!(exception)
-      message = "#{hostname}:#{port} failed (count: #{@fail_count}) #{exception.class}: #{exception.message}"
+      message = "#{name} failed (count: #{@fail_count}) #{exception.class}: #{exception.message}"
       Dalli.logger.info { message }
 
       @fail_count += 1
@@ -223,21 +227,21 @@ module Dalli
 
       if @down_at
         time = Time.now - @down_at
-        Dalli.logger.debug { "#{hostname}:#{port} is still down (for %.3f seconds now)" % time }
+        Dalli.logger.debug { "#{name} is still down (for %.3f seconds now)" % time }
       else
         @down_at = @last_down_at
-        Dalli.logger.warn { "#{hostname}:#{port} is down" }
+        Dalli.logger.warn { "#{name} is down" }
       end
 
       @error = $! && $!.class.name
       @msg = @msg || ($! && $!.message && !$!.message.empty? && $!.message)
-      raise Dalli::NetworkError, "#{hostname}:#{port} is down: #{@error} #{@msg}"
+      raise Dalli::NetworkError, "#{name} is down: #{@error} #{@msg}"
     end
 
     def up!
       if @down_at
         time = Time.now - @down_at
-        Dalli.logger.warn { "#{hostname}:#{port} is back (downtime was %.3f seconds)" % time }
+        Dalli.logger.warn { "#{name} is back (downtime was %.3f seconds)" % time }
       end
 
       @fail_count = 0
@@ -396,6 +400,8 @@ module Dalli
         marshalled = true
         begin
           self.serializer.dump(value)
+        rescue TimeoutError => e
+          raise e
         rescue => ex
           # Marshalling can throw several different types of generic Ruby exceptions.
           # Convert to a specific exception so we can special case it higher up the stack.
@@ -467,9 +473,9 @@ module Dalli
     def sanitize_ttl(ttl)
       if ttl > MAX_ACCEPTABLE_EXPIRATION_INTERVAL
         Dalli.logger.debug "Expiration interval too long for Memcached, converting to an expiration timestamp"
-        Time.now.to_i + ttl
+        Time.now.to_i + ttl.to_i
       else
-        ttl
+        ttl.to_i
       end
     end
 
@@ -555,13 +561,17 @@ module Dalli
     end
 
     def connect
-      Dalli.logger.debug { "Dalli::Server#connect #{hostname}:#{port}" }
+      Dalli.logger.debug { "Dalli::Server#connect #{name}" }
 
       begin
         @pid = Process.pid
-        @sock = KSocket.open(hostname, port, self, options)
-        @version = version # trigger actual connect
+        if socket_type == :unix
+          @sock = KSocket::UNIX.open(hostname, self, options)
+        else
+          @sock = KSocket::TCP.open(hostname, port, self, options)
+        end
         sasl_authentication if need_auth?
+        @version = version # trigger actual connect
         up!
       rescue Dalli::DalliError # SASL auth failure
         raise
@@ -666,7 +676,7 @@ module Dalli
 
       (extras, type, status, count) = read_header.unpack(NORMAL_HEADER)
       raise Dalli::NetworkError, "Unexpected message format: #{extras} #{count}" unless extras == 0 && count > 0
-      content = read(count)
+      content = read(count).gsub(/\u0000/, ' ')
       return (Dalli.logger.debug("Authentication not required/supported by server")) if status == 0x81
       mechanisms = content.split(' ')
       raise NotImplementedError, "Dalli only supports the PLAIN authentication mechanism" if !mechanisms.include?('PLAIN')
@@ -689,8 +699,23 @@ module Dalli
     end
 
     def parse_hostname(str)
-      res = str.match(/\A(\[([\h:]+)\]|[^:]+)(:(\d+))?(:(\d+))?\z/)
-      return res[2] || res[1], res[4], res[6]
+      res = str.match(/\A(\[([\h:]+)\]|[^:]+)(?::(\d+))?(?::(\d+))?\z/)
+      raise Dalli::DalliError, "Could not parse hostname #{str}" if res.nil? || res[1] == '[]'
+      hostnam = res[2] || res[1]
+      if hostnam =~ /\A\//
+        socket_type = :unix
+        # in case of unix socket, allow only setting of weight, not port
+        raise Dalli::DalliError, "Could not parse hostname #{str}" if res[4]
+        weigh = res[3]
+      else
+        socket_type = :tcp
+        por = res[3] || DEFAULT_PORT
+        por = Integer(por)
+        weigh = res[4]
+      end
+      weigh ||= DEFAULT_WEIGHT
+      weigh = Integer(weigh)
+      return hostnam, por, weigh, socket_type
     end
   end
 end