dalli 2.6.4 → 2.7.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 4e1d2bd1cdadde91a31a3e1311e4189e54b57623
+  data.tar.gz: 6e2fbab2dd27e1cf5f62eef39d7292d1ac6f93b8
+SHA512:
+  metadata.gz: ad6caea0b89835f15f7ddc4fd4ecab66eaa87ed969020c41fae9324da520b3e9f3b15afe53d1b14e515eafcb29cbc76c62267ce12a8c708f525ab2863e62f682
+  data.tar.gz: 8646c41bab51faa972bd16c5e06157c1c201a862f911f3f961b252ce9fd96c1b583345068090b2606f1a302f5d106df072eb7a91acb2ecb1c73464b3ca2f6f58

data/Gemfile

@@ -5,6 +5,7 @@ gemspec
 gem 'rake'
 gem 'kgio', :platform => :mri
 gem 'appraisal'
+gem 'connection_pool'
 
 group :test do
   gem 'simplecov'

data/History.md

@@ -1,11 +1,35 @@
 Dalli Changelog
 =====================
 
-HEAD
+2.7.0
+==========
+
+- Multithreading support with dalli\_store:
+  Use :pool\_size to create a pool of shared, threadsafe Dalli clients in Rails:
+```ruby
+    config.cache_store = :dalli_store, "cache-1.example.com", "cache-2.example.com", :compress => true, :pool_size => 5, :expires_in => 300
+```
+  This will ensure the Rails.cache singleton does not become a source of contention.
+  **PLEASE NOTE** Rails's :mem\_cache\_store does not support pooling as of
+Rails 4.0.  You must use :dalli\_store.
+
+- Implement `version` for retrieving version of connected servers [dterei, #384]
+- Implement `fetch_multi` for batched read/write [sorentwo, #380]
+- Add more support for safe updates with multiple writers: [philipmw, #395]
+  `require 'dalli/cas/client'` augments Dalli::Client with the following methods:
+  * Get value with CAS:            `[value, cas] = get_cas(key)`
+                                   `get_cas(key) {|value, cas| ...}`
+  * Get multiple values with CAS:  `get_multi_cas(k1, k2, ...) {|value, metadata| cas = metadata[:cas]}`
+  * Set value with CAS:            `new_cas = set_cas(key, value, cas, ttl, options)`
+  * Replace value with CAS:        `replace_cas(key, new_value, cas, ttl, options)`
+  * Delete value with CAS:         `delete_cas(key, cas)`
+- Fix bug with get key with "Not found" value [uzzz, #375]
+
+2.6.4
 =======
 
 - Fix ADD command, aka `write(unless_exist: true)` (pitr, #365)
-- Upgrade test suite from mini_shoulda to minitest.
+- Upgrade test suite from mini\_shoulda to minitest.
 - Even more performance improvements for get\_multi (xaop, #331)
 
 2.6.3

data/LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2012 Mike Perham
+Copyright (c) Mike Perham
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.md

@@ -1,4 +1,4 @@
-Dalli [![Build Status](https://secure.travis-ci.org/mperham/dalli.png)](http://travis-ci.org/mperham/dalli) [![Dependency Status](https://gemnasium.com/mperham/dalli.png)](https://gemnasium.com/mperham/dalli)
+Dalli [![Build Status](https://secure.travis-ci.org/mperham/dalli.png)](http://travis-ci.org/mperham/dalli) [![Dependency Status](https://gemnasium.com/mperham/dalli.png)](https://gemnasium.com/mperham/dalli) [![Code Climate](https://codeclimate.com/github/mperham/dalli.png)](https://codeclimate.com/github/mperham/dalli)
 =====
 
 Dalli is a high performance pure Ruby client for accessing memcached servers.  It works with memcached 1.4+ only as it uses the newer binary protocol.  It should be considered a replacement for the memcache-client gem.
@@ -35,8 +35,7 @@ Supported Ruby versions and implementations
 Dalli should work identically on:
 
  * JRuby 1.6+
- * Ruby 1.9.2+
- * Ruby 1.8.7+
+ * Ruby 1.9.3+
  * Rubinius 2.0
 
 If you have problems, please enter an issue.
@@ -73,7 +72,7 @@ Dalli has no runtime dependencies and never will.  You can optionally install th
 give Dalli a 20-30% performance boost.
 
 
-Usage with Rails 3.x
+Usage with Rails 3.x and 4.x
 ---------------------------
 
 In your Gemfile:
@@ -113,8 +112,22 @@ Rails.application.config.session_store :dalli_store, :memcache_server => ['host1
 Dalli does not support Rails 2.x.
 
 
+Multithreading and Rails
+--------------------------
+
+If you use Puma or another threaded app server, as of Dalli 2.7, you can use a pool
+of Dalli clients with Rails to ensure the `Rails.cache` singleton does not become a
+source of thread contention.  You must add `gem 'connection_pool'` to your Gemfile and
+add :pool\_size to your `dalli_store` config:
+
+```ruby
+config.cache_store = :dalli_store, 'cache-1.example.com', { :pool_size => 5 }
+```
+
+
 Configuration
 ------------------------
+
 Dalli::Client accepts the following options. All times are in seconds.
 
 **expires_in**: Global default for key TTL.  Default is 0, which means no expiry.
@@ -161,13 +174,14 @@ socket.
 
 Note that Dalli does not require ActiveSupport or Rails.  You can safely use it in your own Ruby projects.
 
-[View the API](http://www.ruby-doc.org/gems/docs/d/dalli-2.6.2/Dalli/Client.html)
+[View the Client API](http://www.rubydoc.info/github/mperham/dalli/Dalli/Client)
 
 Helping Out
 -------------
 
 If you have a fix you wish to provide, please fork the code, fix in your local project and then send a pull request on github.  Please ensure that you include a test which verifies your fix and update History.md with a one sentence description of your fix so you get credit as a contributor.
 
+We're not accepting new compressors. They are trivial to add in an initializer. See #385 (LZ4), #406 (Snappy)
 
 Thanks
 ------------
@@ -182,12 +196,10 @@ Brian Mitchell - for his remix-stash project which was helpful when implementing
 Author
 ----------
 
-Mike Perham, mperham@gmail.com, [mikeperham.com](http://mikeperham.com), [@mperham](http://twitter.com/mperham)  If you like and use this project, please give me a recommendation at [WWR](http://workingwithrails.com/person/10797-mike-perham) or send a few bucks my way via my Pledgie page below.  Happy caching!
-
-<a href='http://www.pledgie.com/campaigns/16623'><img alt='Click here to lend your support to Open Source and make a donation at www.pledgie.com !'     src='http://www.pledgie.com/campaigns/16623.png?skin_name=chrome' border='0' /></a>
+Mike Perham, [mikeperham.com](http://mikeperham.com), [@mperham](http://twitter.com/mperham)
 
 
 Copyright
 -----------
 
-Copyright (c) 2012 Mike Perham. See LICENSE for details.
+Copyright (c) Mike Perham. See LICENSE for details.

data/dalli.gemspec

@@ -22,7 +22,7 @@ Gem::Specification.new do |s|
   s.require_paths = ["lib"]
   s.summary = %q{High performance memcached client for Ruby}
   s.test_files = Dir.glob("test/**/*")
-  s.add_development_dependency(%q<minitest>, [">= 5.0.0"])
+  s.add_development_dependency(%q<minitest>, [">= 4.2.0"])
   s.add_development_dependency(%q<mocha>, [">= 0"])
   s.add_development_dependency(%q<rails>, ["~> 3"])
 end

data/lib/active_support/cache/dalli_store.rb

@@ -33,29 +33,52 @@ module ActiveSupport
       # If no addresses are specified, then DalliStore will connect to
       # localhost port 11211 (the default memcached port).
       #
+      # Connection Pool support
+      #
+      # If you are using multithreaded Rails, the Rails.cache singleton can become a source
+      # of contention.  You can use a connection pool of Dalli clients with Rails.cache by
+      # passing :pool_size and/or :pool_timeout:
+      #
+      # config.cache_store = :dalli_store, 'localhost:11211', :pool_size => 10
+      #
+      # Both pool options default to 5.  You must include the `connection_pool` gem if you
+      # wish to use pool support.
+      #
       def initialize(*addresses)
         addresses = addresses.flatten
         options = addresses.extract_options!
         @options = options.dup
+
+        pool_options = {}
+        pool_options[:size] = options[:pool_size] if options[:pool_size]
+        pool_options[:timeout] = options[:pool_timeout] if options[:pool_timeout]
+
         @options[:compress] ||= @options[:compression]
-        @raise_errors = !!@options[:raise_errors]
         servers = if addresses.empty?
                     nil # use the default from Dalli::Client
                   else
                     addresses
                   end
-        @data = Dalli::Client.new(servers, @options)
+        if pool_options.empty?
+          @data = Dalli::Client.new(servers, @options)
+        else
+          @data = ::ConnectionPool.new(pool_options) { Dalli::Client.new(servers, @options.merge(:threadsafe => false)) }
+        end
 
         extend Strategy::LocalCache
       end
 
       ##
-      # Access the underlying Dalli::Client instance for
+      # Access the underlying Dalli::Client or ConnectionPool instance for
       # access to get_multi, etc.
       def dalli
         @data
       end
 
+      def with(&block)
+        @data.with(&block)
+      end
+
       def fetch(name, options=nil)
         options ||= {}
         name = expanded_key name
@@ -103,7 +126,10 @@ module ActiveSupport
         name = expanded_key name
 
         instrument(:write, name, options) do |payload|
-          write_entry(name, value, options)
+          with do |connection|
+            options = options.merge(:connection => connection)
+            write_entry(name, value, options)
+          end
         end
       end
 
@@ -139,7 +165,8 @@ module ActiveSupport
             end
           end
 
-          results.merge!(@data.get_multi(mapping.keys - results.keys))
+          data = with { |c| c.get_multi(mapping.keys - results.keys) }
+          results.merge!(data)
           results.inject({}) do |memo, (inner, _)|
             entry = results[inner]
             # NB Backwards data compatibility, to be removed at some point
@@ -151,6 +178,35 @@ module ActiveSupport
         end
       end
 
+      # Fetches data from the cache, using the given keys. If there is data in
+      # the cache with the given keys, then that data is returned. Otherwise,
+      # the supplied block is called for each key for which there was no data,
+      # and the result will be written to the cache and returned.
+      def fetch_multi(*names)
+        options = names.extract_options!
+        mapping = names.inject({}) { |memo, name| memo[expanded_key(name)] = name; memo }
+
+        instrument(:fetch_multi, names) do
+          with do |connection|
+            results = connection.get_multi(mapping.keys)
+
+            connection.multi do
+              mapping.inject({}) do |memo, (expanded, name)|
+                memo[name] = results[expanded]
+                if memo[name].nil?
+                  value = yield(name)
+                  memo[name] = value
+                  options = options.merge(:connection => connection)
+                  write_entry(expanded, value, options)
+                end
+
+                memo
+              end
+            end
+          end
+        end
+      end
+
       # Increment a cached value. This method uses the memcached incr atomic
       # operator and can only be used on values written with the :raw option.
       # Calling it on a value not stored with :raw will fail.
@@ -162,11 +218,11 @@ module ActiveSupport
         initial = options.has_key?(:initial) ? options[:initial] : amount
         expires_in = options[:expires_in]
         instrument(:increment, name, :amount => amount) do
-          @data.incr(name, amount, expires_in, initial)
+          with { |c| c.incr(name, amount, expires_in, initial) }
         end
       rescue Dalli::DalliError => e
         logger.error("DalliError: #{e.message}") if logger
-        raise if @raise_errors
+        raise if raise_errors?
         nil
       end
 
@@ -181,11 +237,11 @@ module ActiveSupport
         initial = options.has_key?(:initial) ? options[:initial] : 0
         expires_in = options[:expires_in]
         instrument(:decrement, name, :amount => amount) do
-          @data.decr(name, amount, expires_in, initial)
+          with { |c| c.decr(name, amount, expires_in, initial) }
         end
       rescue Dalli::DalliError => e
         logger.error("DalliError: #{e.message}") if logger
-        raise if @raise_errors
+        raise if raise_errors?
         nil
       end
 
@@ -193,11 +249,11 @@ module ActiveSupport
       # be used with care when using a shared cache.
       def clear(options=nil)
         instrument(:clear, 'flushing all keys') do
-          @data.flush_all
+          with { |c| c.flush_all }
         end
       rescue Dalli::DalliError => e
         logger.error("DalliError: #{e.message}") if logger
-        raise if @raise_errors
+        raise if raise_errors?
         nil
       end
 
@@ -207,11 +263,11 @@ module ActiveSupport
 
       # Get the statistics from the memcached servers.
       def stats
-        @data.stats
+        with { |c| c.stats }
       end
 
       def reset
-        @data.reset
+        with { |c| c.reset }
       end
 
       def logger
@@ -226,12 +282,12 @@ module ActiveSupport
 
       # Read an entry from the cache.
       def read_entry(key, options) # :nodoc:
-        entry = @data.get(key, options)
+        entry = with { |c| c.get(key, options) }
         # NB Backwards data compatibility, to be removed at some point
         entry.is_a?(ActiveSupport::Cache::Entry) ? entry.value : entry
       rescue Dalli::DalliError => e
         logger.error("DalliError: #{e.message}") if logger
-        raise if @raise_errors
+        raise if raise_errors?
         nil
       end
 
@@ -241,19 +297,20 @@ module ActiveSupport
         cleanup if options[:unless_exist]
         method = options[:unless_exist] ? :add : :set
         expires_in = options[:expires_in]
-        @data.send(method, key, value, expires_in, options)
+        connection = options.delete(:connection)
+        connection.send(method, key, value, expires_in, options)
       rescue Dalli::DalliError => e
         logger.error("DalliError: #{e.message}") if logger
-        raise if @raise_errors
+        raise if raise_errors?
         false
       end
 
       # Delete an entry from the cache.
       def delete_entry(key, options) # :nodoc:
-        @data.delete(key)
+        with { |c| c.delete(key) }
       rescue Dalli::DalliError => e
         logger.error("DalliError: #{e.message}") if logger
-        raise if @raise_errors
+        raise if raise_errors?
         false
       end
 
@@ -296,6 +353,9 @@ module ActiveSupport
         logger.debug("Cache #{operation}: #{key}#{options.blank? ? "" : " (#{options.inspect})"}")
       end
 
+      def raise_errors?
+        !!@options[:raise_errors]
+      end
     end
   end
 end

data/lib/dalli/cas/client.rb

@@ -0,0 +1,58 @@
+require 'dalli/client'
+
+module Dalli
+  class Client
+    ##
+    # Get the value and CAS ID associated with the key.  If a block is provided,
+    # value and CAS will be passed to the block.
+    def get_cas(key)
+      (value, cas) = perform(:cas, key)
+      value = (!value || value == 'Not found') ? nil : value
+      if block_given?
+        yield value, cas
+      else
+        [value, cas]
+      end
+    end
+
+    ##
+    # Fetch multiple keys efficiently, including available metadata such as CAS.
+    # If a block is given, yields key/data pairs one a time.  Data is an array:
+    # [value, cas_id]
+    # If no block is given, returns a hash of
+    #   { 'key' => [value, cas_id] }
+    def get_multi_cas(*keys)
+      if block_given?
+        get_multi_yielder(keys) {|*args| yield(*args)}
+      else
+        Hash.new.tap do |hash|
+          get_multi_yielder(keys) {|k, data| hash[k] = data}
+        end
+      end
+    end
+
+    ##
+    # Set the key-value pair, verifying existing CAS.
+    # Returns the resulting CAS value if succeeded, and false otherwise.
+    def set_cas(key, value, cas, ttl=nil, options=nil)
+      ttl ||= @options[:expires_in].to_i
+      perform(:set, key, value, ttl, cas, options)
+    end
+
+    ##
+    # Conditionally add a key/value pair, verifying existing CAS, only if the
+    # key already exists on the server.  Returns the new CAS value if the
+    # operation succeeded, or false otherwise.
+    def replace_cas(key, value, cas, ttl=nil, options=nil)
+      ttl ||= @options[:expires_in].to_i
+      perform(:replace, key, value, ttl, cas, options)
+    end
+
+    # Delete a key/value pair, verifying existing CAS.
+    # Returns true if succeeded, and false otherwise.
+    def delete_cas(key, cas=0)
+      perform(:delete, key, cas)
+    end
+
+  end
+end

data/lib/dalli/client.rb

@@ -31,17 +31,6 @@ module Dalli
       @ring = nil
     end
 
-    ##
-    # Normalizes the argument into an array of servers. If the argument is a string, it's expected to be of
-    # the format "memcache1.example.com:11211[,memcache2.example.com:11211[,memcache3.example.com:11211[...]]]
-    def normalize_servers(servers)
-      if servers.is_a? String
-        return servers.split(",")
-      else
-        return servers
-      end
-    end
-
     #
     # The standard memcached instruction set
     #
@@ -58,107 +47,22 @@ module Dalli
       Thread.current[:dalli_multi] = old
     end
 
+    ##
+    # Get the value associated with the key.
     def get(key, options=nil)
-      resp = perform(:get, key)
-      resp.nil? || 'Not found' == resp ? nil : resp
+      perform(:get, key)
     end
 
     ##
     # Fetch multiple keys efficiently.
-    # Returns a hash of { 'key' => 'value', 'key2' => 'value1' }
+    # If a block is given, yields key/value pairs one at a time.
+    # Otherwise returns a hash of { 'key' => 'value', 'key2' => 'value1' }
     def get_multi(*keys)
-      perform do
-        return {} if keys.empty?
-        options = nil
-        options = keys.pop if keys.last.is_a?(Hash) || keys.last.nil?
-        ring.lock do
-          begin
-            mapped_keys = keys.flatten.map {|a| validate_key(a.to_s)}
-            groups = mapped_keys.flatten.group_by do |key|
-              begin
-                ring.server_for_key(key)
-              rescue Dalli::RingError
-                Dalli.logger.debug { "unable to get key #{key}" }
-                nil
-              end
-            end
-            if unfound_keys = groups.delete(nil)
-              Dalli.logger.debug { "unable to get keys for #{unfound_keys.length} keys because no matching server was found" }
-            end
-
-            groups.each do |server, keys_for_server|
-              begin
-                # TODO: do this with the perform chokepoint?
-                # But given the fact that fetching the response doesn't take place
-                # in that slot it's misleading anyway. Need to move all of this method
-                # into perform to be meaningful
-                server.request(:send_multiget, keys_for_server)
-              rescue DalliError, NetworkError => e
-                Dalli.logger.debug { e.inspect }
-                Dalli.logger.debug { "unable to get keys for server #{server.hostname}:#{server.port}" }
-              end
-            end
-
-            servers = groups.keys
-            values = {}
-            return values if servers.empty?
-
-            servers.each do |server|
-              next unless server.alive?
-              begin
-                server.multi_response_start
-              rescue DalliError, NetworkError => e
-                Dalli.logger.debug { e.inspect }
-                Dalli.logger.debug { "results from this server will be missing" }
-                servers.delete(server)
-              end
-            end
-
-            start = Time.now
-            loop do
-              # remove any dead servers
-              servers.delete_if { |s| s.sock.nil? }
-              break if servers.empty?
-
-              # calculate remaining timeout
-              elapsed = Time.now - start
-              timeout = servers.first.options[:socket_timeout]
-              if elapsed > timeout
-                readable = nil
-              else
-                sockets = servers.map(&:sock)
-                readable, _ = IO.select(sockets, nil, nil, timeout - elapsed)
-              end
-
-              if readable.nil?
-                # no response within timeout; abort pending connections
-                servers.each do |server|
-                  Dalli.logger.debug { "memcached at #{server.name} did not response within timeout" }
-                  server.multi_response_abort
-                end
-                break
-
-              else
-                readable.each do |sock|
-                  server = sock.server
-
-                  begin
-                    server.multi_response_nonblock.each do |key, value|
-                      values[key_without_namespace(key)] = value
-                    end
-
-                    if server.multi_response_completed?
-                      servers.delete(server)
-                    end
-                  rescue NetworkError
-                    servers.delete(server)
-                  end
-                end
-              end
-            end
-
-            values
-          end
+      if block_given?
+        get_multi_yielder(keys) {|k, data| yield k, data.first}
+      else
+         Hash.new.tap do |hash|
+          get_multi_yielder(keys) {|k, data| hash[k] = data.first}
         end
       end
     end
@@ -212,11 +116,11 @@ module Dalli
     # on the server.  Returns true if the operation succeeded.
     def replace(key, value, ttl=nil, options=nil)
       ttl ||= @options[:expires_in].to_i
-      perform(:replace, key, value, ttl, options)
+      perform(:replace, key, value, ttl, 0, options)
     end
 
     def delete(key)
-      perform(:delete, key)
+      perform(:delete, key, 0)
     end
 
     ##
@@ -308,6 +212,16 @@ module Dalli
       end
     end
 
+    ##
+    ## Version of the memcache servers.
+    def version
+      values = {}
+      ring.servers.each do |server|
+        values["#{server.hostname}:#{server.port}"] = server.alive? ? server.request(:version) : nil
+      end
+      values
+    end
+
     ##
     # Close our connection to each server.
     # If you perform another operation after this, the connections will be re-established.
@@ -319,8 +233,69 @@ module Dalli
     end
     alias_method :reset, :close
 
+    # Stub method so a bare Dalli client can pretend to be a connection pool.
+    def with
+      yield self
+    end
+
     private
 
+    def groups_for_keys(*keys)
+      groups = mapped_keys(keys).flatten.group_by do |key|
+        begin
+          ring.server_for_key(key)
+        rescue Dalli::RingError
+          Dalli.logger.debug { "unable to get key #{key}" }
+          nil
+        end
+      end
+      return groups
+    end
+
+    def mapped_keys(keys)
+      keys.flatten.map {|a| validate_key(a.to_s)}
+    end
+
+    def make_multi_get_requests(groups)
+      groups.each do |server, keys_for_server|
+        begin
+          # TODO: do this with the perform chokepoint?
+          # But given the fact that fetching the response doesn't take place
+          # in that slot it's misleading anyway. Need to move all of this method
+          # into perform to be meaningful
+          server.request(:send_multiget, keys_for_server)
+        rescue DalliError, NetworkError => e
+          Dalli.logger.debug { e.inspect }
+          Dalli.logger.debug { "unable to get keys for server #{server.hostname}:#{server.port}" }
+        end
+      end
+    end
+
+    def perform_multi_response_start(servers)
+      servers.each do |server|
+        next unless server.alive?
+        begin
+          server.multi_response_start
+        rescue DalliError, NetworkError => e
+          Dalli.logger.debug { e.inspect }
+          Dalli.logger.debug { "results from this server will be missing" }
+          servers.delete(server)
+        end
+      end
+      servers
+    end
+
+    ##
+    # Normalizes the argument into an array of servers. If the argument is a string, it's expected to be of
+    # the format "memcache1.example.com:11211[,memcache2.example.com:11211[,memcache3.example.com:11211[...]]]
+    def normalize_servers(servers)
+      if servers.is_a? String
+        return servers.split(",")
+      else
+        return servers
+      end
+    end
+
     def ring
       @ring ||= Dalli::Ring.new(
         @servers.map do |s|
@@ -388,5 +363,70 @@ module Dalli
       end
       opts
     end
+
+    ##
+    # Yields, one at a time, keys and their values+attributes.
+    def get_multi_yielder(keys)
+      perform do
+        return {} if keys.empty?
+        ring.lock do
+          begin
+            groups = groups_for_keys(keys)
+            if unfound_keys = groups.delete(nil)
+              Dalli.logger.debug { "unable to get keys for #{unfound_keys.length} keys because no matching server was found" }
+            end
+            make_multi_get_requests(groups)
+
+            servers = groups.keys
+            return if servers.empty?
+            servers = perform_multi_response_start(servers)
+
+            start = Time.now
+            loop do
+              # remove any dead servers
+              servers.delete_if { |s| s.sock.nil? }
+              break if servers.empty?
+
+              # calculate remaining timeout
+              elapsed = Time.now - start
+              timeout = servers.first.options[:socket_timeout]
+              if elapsed > timeout
+                readable = nil
+              else
+                sockets = servers.map(&:sock)
+                readable, _ = IO.select(sockets, nil, nil, timeout - elapsed)
+              end
+
+              if readable.nil?
+                # no response within timeout; abort pending connections
+                servers.each do |server|
+                  Dalli.logger.debug { "memcached at #{server.name} did not response within timeout" }
+                  server.multi_response_abort
+                end
+                break
+
+              else
+                readable.each do |sock|
+                  server = sock.server
+
+                  begin
+                    server.multi_response_nonblock.each_pair do |key, value_list|
+                      yield key_without_namespace(key), value_list
+                    end
+
+                    if server.multi_response_completed?
+                      servers.delete(server)
+                    end
+                  rescue NetworkError
+                    servers.delete(server)
+                  end
+                end
+              end
+            end
+          end
+        end
+      end
+    end
+
   end
 end