dalli 2.7.4 → 2.7.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 91e347f30ee656fe31027c0eab1b45885a2adbec
-  data.tar.gz: 55755a2802168338fb9da8507ab8338436994bb6
+  metadata.gz: 29b86664473df52d0ca07fbf21015a39260962a1
+  data.tar.gz: 5061af271bef6e1db864d5e6de4f9464412e0c67
 SHA512:
-  metadata.gz: 48faaa3c18f65110ec7f7bb826139d0df0ac4662b0b29458fe9a9bdc1b219aaaa838fd933b8c3ca4cf8d4201d68e5f73a9915721f3c5be62402379627bbcd024
-  data.tar.gz: 31494501ba0963d4e6ff62998c9da03a9576659c69ddab17b5f916a0778a7fb0c6f1a3c38ec3c26d6b096fac5e88f9167c53aec911d8567ce42df980b29a1cfa
+  metadata.gz: 7ccec720347f06a142f74d1a500948390f0dcd3520c73922363267c393d99067cb3682f4c2577cc2cbbd99a18a93f800041eaec7570303c73d5e7ab451b9396e
+  data.tar.gz: a68483e89e28c82d52852d73c921aa40578abad55a2f3ec9106b01f1fd5dfc6f98104e9b2f8ab0006306685a4b3ecafca3d768ebf81223c213171482ee00d1f1

data/Gemfile

@@ -2,11 +2,4 @@ source 'https://rubygems.org'
 
 gemspec
 
-gem 'rake'
 gem 'kgio', :platform => :mri
-gem 'appraisal'
-gem 'connection_pool'
-
-group :test do
-  gem 'simplecov'
-end

data/History.md

@@ -1,6 +1,21 @@
 Dalli Changelog
 =====================
 
+2.7.5
+==========
+
+- Support rcvbuff and sndbuff byte configuration. (btatnall)
+- Add `:cache_nils` option to support nil values in `DalliStore#fetch` and `Dalli::Client#fetch` (wjordan, #559)
+- Log retryable server errors with 'warn' instead of 'info' (phrinx)
+- Fix timeout issue with Dalli::Client#get_multi_yielder (dspeterson)
+- Escape namespaces with special regexp characters (Steven Peckins)
+- Ensure LocalCache supports the `:raw` option and Entry unwrapping (sj26)
+- Ensure bad ttl values don't cause Dalli::RingError (eagletmt, petergoldstein)
+- Always pass namespaced key to instrumentation API (kaorimatz)
+- Replace use of deprecated TimeoutError with Timeout::Error (eagletmt)
+- Clean up gemspec, and use Bundler for loading (grosser)
+- Dry up local cache testing (grosser)
+
 2.7.4
 ==========
 

data/LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) Mike Perham
+Copyright (c) Peter M. Goldstein, Mike Perham
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.md

@@ -1,4 +1,4 @@
-Dalli [![Build Status](https://secure.travis-ci.org/mperham/dalli.png)](http://travis-ci.org/mperham/dalli) [![Dependency Status](https://gemnasium.com/mperham/dalli.png)](https://gemnasium.com/mperham/dalli) [![Code Climate](https://codeclimate.com/github/mperham/dalli.png)](https://codeclimate.com/github/mperham/dalli)
+Dalli [![Build Status](https://secure.travis-ci.org/petergoldstein/dalli.png)](http://travis-ci.org/petergoldstein/dalli) [![Dependency Status](https://gemnasium.com/petergoldstein/dalli.png)](https://gemnasium.com/petergoldstein/dalli) [![Code Climate](https://codeclimate.com/github/petergoldstein/dalli.png)](https://codeclimate.com/github/petergoldstein/dalli)
 =====
 
 Dalli is a high performance pure Ruby client for accessing memcached servers.  It works with memcached 1.4+ only as it uses the newer binary protocol.  It should be considered a replacement for the memcache-client gem.
@@ -13,7 +13,7 @@ Dalli's initial development was sponsored by [CouchBase](http://www.couchbase.co
 Design
 ------------
 
-I decided to write Dalli after maintaining memcache-client for two years for a few specific reasons:
+Mike Perham decided to write Dalli after maintaining memcache-client for two years for a few specific reasons:
 
  0. The code is mostly old and gross.  The bulk of the code is a single 1000 line .rb file.
  1. It has a lot of options that are infrequently used which complicate the codebase.
@@ -186,6 +186,12 @@ If serving compressed data using nginx's HttpMemcachedModule, set `memcached_gzi
 
 **password**: The password to use for authenticating this client instance against a SASL-enabled memcached server.  Heroku users should not need to use this normally.
 
+**sndbuf**: In bytes, set the socket SO_SNDBUF. Defaults to operating system default.
+
+**rcvbuf**: In bytes, set the socket SO_RCVBUF. Defaults to operating system default.
+
+**cache_nils**: Boolean. If true Dalli will not treat cached `nil` values as 'not found' for `#fetch` operations. Default is false.
+
 Features and Changes
 ------------------------
 
@@ -209,20 +215,22 @@ We're not accepting new compressors. They are trivial to add in an initializer.
 Thanks
 ------------
 
+Mike Perham - for originally authoring the Dalli project and serving as maintainer and primary contributor
+
 Eric Wong - for help using his [kgio](http://bogomips.org/kgio/) library.
 
 Brian Mitchell - for his remix-stash project which was helpful when implementing and testing the binary protocol support.
 
 [CouchBase](http://couchbase.com) - for their project sponsorship
 
-
-Author
+Authors
 ----------
 
-Mike Perham, [mikeperham.com](http://mikeperham.com), [@mperham](http://twitter.com/mperham)
+* [Peter M. Goldstein](https://github.com/petergoldstein) - current maintainer
+* [Mike Perham](https://github.com/mperham) and contributors
 
 
 Copyright
 -----------
 
-Copyright (c) Mike Perham. See LICENSE for details.
+Copyright (c) Mike Perham, Peter M. Goldstein. See LICENSE for details.

data/lib/active_support/cache/dalli_store.rb

@@ -68,6 +68,7 @@ module ActiveSupport
         end
 
         extend Strategy::LocalCache
+        extend LocalCacheEntryUnwrapAndRaw
       end
 
       ##
@@ -81,31 +82,41 @@ module ActiveSupport
         @data.with(&block)
       end
 
+      # Fetch the value associated with the key.
+      # If a value is found, then it is returned.
+      #
+      # If a value is not found and no block is given, then nil is returned.
+      #
+      # If a value is not found (or if the found value is nil and :cache_nils is false)
+      # and a block is given, the block will be invoked and its return value
+      # written to the cache and returned.
       def fetch(name, options=nil)
         options ||= {}
+        options[:cache_nils] = true if @options[:cache_nils]
         namespaced_name = namespaced_key(name, options)
-
+        not_found = options[:cache_nils] ? Dalli::Server::NOT_FOUND : nil
         if block_given?
+          entry = not_found
           unless options[:force]
             entry = instrument(:read, namespaced_name, options) do |payload|
               read_entry(namespaced_name, options).tap do |result|
                 if payload
                   payload[:super_operation] = :fetch
-                  payload[:hit] = !result.nil?
+                  payload[:hit] = result != not_found
                 end
               end
             end
           end
 
-          if !entry.nil?
-            instrument(:fetch_hit, name, options) { |payload| }
-            entry
-          else
-            result = instrument(:generate, name, options) do |payload|
+          if entry == not_found
+            result = instrument(:generate, namespaced_name, options) do |payload|
               yield
             end
             write(name, result, options)
             result
+          else
+            instrument(:fetch_hit, namespaced_name, options) { |payload| }
+            entry
           end
         else
           read(name, options)
@@ -157,7 +168,7 @@ module ActiveSupport
       def read_multi(*names)
         options  = names.extract_options!
         mapping = names.inject({}) { |memo, name| memo[namespaced_key(name, options)] = name; memo }
-        instrument(:read_multi, names) do
+        instrument(:read_multi, mapping.keys) do
           results = {}
           if local_cache
             mapping.each_key do |key|
@@ -188,7 +199,7 @@ module ActiveSupport
         options = names.extract_options!
         mapping = names.inject({}) { |memo, name| memo[namespaced_key(name, options)] = name; memo }
 
-        instrument(:fetch_multi, names) do
+        instrument(:fetch_multi, mapping.keys) do
           with do |connection|
             results = connection.get_multi(mapping.keys)
 
@@ -367,6 +378,25 @@ module ActiveSupport
       def raise_errors?
         !!@options[:raise_errors]
       end
+
+      # Make sure LocalCache is giving raw values, not `Entry`s, and
+      # respect `raw` option.
+      module LocalCacheEntryUnwrapAndRaw # :nodoc:
+        protected
+          def read_entry(key, options)
+            retval = super
+            if retval.is_a? ActiveSupport::Cache::Entry
+              # Must have come from LocalStore, unwrap it
+              if options[:raw]
+                retval.value.to_s
+              else
+                retval.value
+              end
+            else
+              retval
+            end
+          end
+      end
     end
   end
 end

data/lib/dalli/client.rb

@@ -27,6 +27,7 @@ module Dalli
     # - :compress - defaults to false, if true Dalli will compress values larger than 1024 bytes before sending them to memcached.
     # - :serializer - defaults to Marshal
     # - :compressor - defaults to zlib
+    # - :cache_nils - defaults to false, if true Dalli will not treat cached nil values as 'not found' for #fetch operations.
     #
     def initialize(servers=nil, options={})
       @servers = normalize_servers(servers || ENV["MEMCACHE_SERVERS"] || '127.0.0.1:11211')
@@ -52,8 +53,9 @@ module Dalli
 
     ##
     # Get the value associated with the key.
+    # If a value is not found, then +nil+ is returned.
     def get(key, options=nil)
-      perform(:get, key)
+      perform(:get, key, options)
     end
 
     ##
@@ -71,12 +73,25 @@ module Dalli
       end
     end
 
+    CACHE_NILS = {cache_nils: true}.freeze
+
+    # Fetch the value associated with the key.
+    # If a value is found, then it is returned.
+    #
+    # If a value is not found and no block is given, then nil is returned.
+    #
+    # If a value is not found (or if the found value is nil and :cache_nils is false)
+    # and a block is given, the block will be invoked and its return value
+    # written to the cache and returned.
     def fetch(key, ttl=nil, options=nil)
-      ttl ||= @options[:expires_in].to_i
+      options = options.nil? ? CACHE_NILS : options.merge(CACHE_NILS) if @options[:cache_nils]
       val = get(key, options)
-      if val.nil? && block_given?
+      not_found = @options[:cache_nils] ?
+        val == Dalli::Server::NOT_FOUND :
+        val.nil?
+      if not_found && block_given?
         val = yield
-        add(key, val, ttl, options)
+        add(key, val, ttl_or_default(ttl), options)
       end
       val
     end
@@ -93,34 +108,30 @@ module Dalli
     # - false if the value was changed by someone else.
     # - true if the value was successfully updated.
     def cas(key, ttl=nil, options=nil)
-      ttl ||= @options[:expires_in].to_i
       (value, cas) = perform(:cas, key)
       value = (!value || value == 'Not found') ? nil : value
       if value
         newvalue = yield(value)
-        perform(:set, key, newvalue, ttl, cas, options)
+        perform(:set, key, newvalue, ttl_or_default(ttl), cas, options)
       end
     end
 
     def set(key, value, ttl=nil, options=nil)
-      ttl ||= @options[:expires_in].to_i
-      perform(:set, key, value, ttl, 0, options)
+      perform(:set, key, value, ttl_or_default(ttl), 0, options)
     end
 
     ##
     # Conditionally add a key/value pair, if the key does not already exist
     # on the server.  Returns truthy if the operation succeeded.
     def add(key, value, ttl=nil, options=nil)
-      ttl ||= @options[:expires_in].to_i
-      perform(:add, key, value, ttl, options)
+      perform(:add, key, value, ttl_or_default(ttl), options)
     end
 
     ##
     # Conditionally add a key/value pair, only if the key already exists
     # on the server.  Returns truthy if the operation succeeded.
     def replace(key, value, ttl=nil, options=nil)
-      ttl ||= @options[:expires_in].to_i
-      perform(:replace, key, value, ttl, 0, options)
+      perform(:replace, key, value, ttl_or_default(ttl), 0, options)
     end
 
     def delete(key)
@@ -161,8 +172,7 @@ module Dalli
     # #cas.
     def incr(key, amt=1, ttl=nil, default=nil)
       raise ArgumentError, "Positive values only: #{amt}" if amt < 0
-      ttl ||= @options[:expires_in].to_i
-      perform(:incr, key, amt.to_i, ttl, default)
+      perform(:incr, key, amt.to_i, ttl_or_default(ttl), default)
     end
 
     ##
@@ -181,8 +191,7 @@ module Dalli
     # #cas.
     def decr(key, amt=1, ttl=nil, default=nil)
       raise ArgumentError, "Positive values only: #{amt}" if amt < 0
-      ttl ||= @options[:expires_in].to_i
-      perform(:decr, key, amt.to_i, ttl, default)
+      perform(:decr, key, amt.to_i, ttl_or_default(ttl), default)
     end
 
     ##
@@ -190,8 +199,7 @@ module Dalli
     #
     # Returns true if key exists, otherwise nil.
     def touch(key, ttl=nil)
-      ttl ||= @options[:expires_in].to_i
-      resp = perform(:touch, key, ttl)
+      resp = perform(:touch, key, ttl_or_default(ttl))
       resp.nil? ? nil : true
     end
 
@@ -250,6 +258,12 @@ module Dalli
 
     private
 
+    def ttl_or_default(ttl)
+      (ttl || @options[:expires_in]).to_i
+    rescue NoMethodError
+      raise ArgumentError, "Cannot convert ttl (#{ttl}) to an integer"
+    end
+
     def groups_for_keys(*keys)
       groups = mapped_keys(keys).flatten.group_by do |key|
         begin
@@ -296,8 +310,9 @@ module Dalli
     end
 
     ##
-    # Normalizes the argument into an array of servers. If the argument is a string, it's expected to be of
-    # the format "memcache1.example.com:11211[,memcache2.example.com:11211[,memcache3.example.com:11211[...]]]
+    # Normalizes the argument into an array of servers.
+    # If the argument is a string, it's expected that the URIs are comma separated e.g.
+    # "memcache1.example.com:11211,memcache2.example.com:11211,memcache3.example.com:11211"
     def normalize_servers(servers)
       if servers.is_a? String
         return servers.split(",")
@@ -354,7 +369,7 @@ module Dalli
     end
 
     def key_without_namespace(key)
-      (ns = namespace) ? key.sub(%r(\A#{ns}:), '') : key
+      (ns = namespace) ? key.sub(%r(\A#{Regexp.escape ns}:), '') : key
     end
 
     def namespace
@@ -401,12 +416,10 @@ module Dalli
               # calculate remaining timeout
               elapsed = Time.now - start
               timeout = servers.first.options[:socket_timeout]
-              if elapsed > timeout
-                readable = nil
-              else
-                sockets = servers.map(&:sock)
-                readable, _ = IO.select(sockets, nil, nil, timeout - elapsed)
-              end
+              time_left = (elapsed > timeout) ? 0 : timeout - elapsed
+
+              sockets = servers.map(&:sock)
+              readable, _ = IO.select(sockets, nil, nil, time_left)
 
               if readable.nil?
                 # no response within timeout; abort pending connections

data/lib/dalli/server.rb

@@ -31,7 +31,11 @@ module Dalli
       :serializer => Marshal,
       :username => nil,
       :password => nil,
-      :keepalive => true
+      :keepalive => true,
+      # max byte size for SO_SNDBUF
+      :sndbuf => nil,
+      # max byte size for SO_RCVBUF
+      :rcvbuf => nil
     }
 
     def initialize(attribs, options = {})
@@ -203,20 +207,28 @@ module Dalli
 
     def verify_state
       failure!(RuntimeError.new('Already writing to socket')) if @inprogress
-      failure!(RuntimeError.new('Cannot share client between multiple processes')) if @pid && @pid != Process.pid
+      if @pid && @pid != Process.pid
+        message = 'Fork detected, re-connecting child process...'
+        Dalli.logger.info { message }
+        reconnect! message
+      end
+    end
+
+    def reconnect!(message)
+      close
+      sleep(options[:socket_failure_delay]) if options[:socket_failure_delay]
+      raise Dalli::NetworkError, message
     end
 
     def failure!(exception)
       message = "#{name} failed (count: #{@fail_count}) #{exception.class}: #{exception.message}"
-      Dalli.logger.info { message }
+      Dalli.logger.warn { message }
 
       @fail_count += 1
       if @fail_count >= options[:socket_max_failures]
         down!
       else
-        close
-        sleep(options[:socket_failure_delay]) if options[:socket_failure_delay]
-        raise Dalli::NetworkError, "Socket operation failed, retrying..."
+        reconnect! 'Socket operation failed, retrying...'
       end
     end
 
@@ -255,10 +267,10 @@ module Dalli
       Thread.current[:dalli_multi]
     end
 
-    def get(key)
+    def get(key, options=nil)
       req = [REQUEST, OPCODES[:get], key.bytesize, 0, 0, 0, key.bytesize, 0, 0, key].pack(FORMAT[:get])
       write(req)
-      generic_response(true)
+      generic_response(true, !!(options && options.is_a?(Hash) && options[:cache_nils]))
     end
 
     def send_multiget(keys)
@@ -400,7 +412,7 @@ module Dalli
         marshalled = true
         begin
           self.serializer.dump(value)
-        rescue TimeoutError => e
+        rescue Timeout::Error => e
           raise e
         rescue => ex
           # Marshalling can throw several different types of generic Ruby exceptions.
@@ -413,7 +425,8 @@ module Dalli
         value.to_s
       end
       compressed = false
-      if @options[:compress] && value.bytesize >= @options[:compression_min_size] &&
+      set_compress_option = true if options && options[:compress]
+      if (@options[:compress] || set_compress_option) && value.bytesize >= @options[:compression_min_size] &&
         (!@options[:compression_max_size] || value.bytesize <= @options[:compression_max_size])
         value = self.compressor.compress(value)
         compressed = true
@@ -471,19 +484,21 @@ module Dalli
     # > An expiration time, in seconds. Can be up to 30 days. After 30 days, is treated as a unix timestamp of an exact date.
     MAX_ACCEPTABLE_EXPIRATION_INTERVAL = 30*24*60*60 # 30 days
     def sanitize_ttl(ttl)
-      if ttl > MAX_ACCEPTABLE_EXPIRATION_INTERVAL
-        Dalli.logger.debug "Expiration interval too long for Memcached, converting to an expiration timestamp"
-        Time.now.to_i + ttl.to_i
-      else
-        ttl.to_i
-      end
+      ttl_as_i = ttl.to_i
+      return ttl_as_i if ttl_as_i <= MAX_ACCEPTABLE_EXPIRATION_INTERVAL
+      Dalli.logger.debug "Expiration interval (#{ttl_as_i}) too long for Memcached, converting to an expiration timestamp"
+      Time.now.to_i + ttl_as_i
     end
 
-    def generic_response(unpack=false)
+    # Implements the NullObject pattern to store an application-defined value for 'Key not found' responses.
+    class NilObject; end
+    NOT_FOUND = NilObject.new
+
+    def generic_response(unpack=false, cache_nils=false)
       (extras, _, status, count) = read_header.unpack(NORMAL_HEADER)
       data = read(count) if count > 0
       if status == 1
-        nil
+        cache_nils ? NOT_FOUND : nil
       elsif status == 2 || status == 5
         false # Not stored, normal status for add operation
       elsif status != 0