dalli 5.2.2 → 5.2.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cd910ad2e0c6e5aa47b8591e9ba8c5806b6208fbab0ae0852637aecc48939051
-  data.tar.gz: bd0db96211060011acb2135feb337d8acc88638b08c57b3fa875445695a09b59
+  metadata.gz: 64bd21546442078425398b10e3baa0b75000a4cc8388dcac08c834909bc29731
+  data.tar.gz: 67568b0a0d030ce81c657cb98a2957f0a6fc12d5ed98f485305e4a93573474f8
 SHA512:
-  metadata.gz: 82da7aebd50661e1ab9af25dab7d66e5f70cb15260ff9ca4b3476e63eeb60cbb2ad1299718e56312f9a6a61d7a2b14d394fcc470325329f1152db42ac921ee6d
-  data.tar.gz: a036421fac121aa342adc3beced711ce64a1bd241673a9314f1f1454a951515890732c4c9651e52baf65c94cd01c4b5cbb655c8dcca4fa4db03d656c3cfc4abe
+  metadata.gz: b069ac0229a977d837ba26755ba279c9eda6b3295bb6c68fabc9baf07ae4c7367cc6acd6d1132f0a5e2913f4b000c8881df0094562b7dd773e34877fab8c21b3
+  data.tar.gz: 84fbaba58b48b46b7be72fbef38006259ce90268d8eb2ccfa3a1ceeee57a69e9f05472db76d2382d0a8d2e2965e63c363484f351a08bd99b9f9d65e7524d294e

data/CHANGELOG.md

@@ -4,6 +4,81 @@ Dalli Changelog
 Unreleased
 ==========
 
+5.2.5
+==========
+
+Bug fixes:
+
+- `set_multi` across more than one server stored `nil` for a Symbol key or a key over 250 bytes, in place of the value given
+
+5.2.4
+==========
+
+Security:
+
+- Keep a connection shared by several threads or fibers in step during a `quiet` block (GHSA-5xvx-r8rg-5qm2)
+  - A `quiet` (or `multi`) block is quiet only for the thread or fiber that runs it. memcached still replies to a quiet request that fails (for example an `add` of a key that exists), and another thread or fiber using the same client while the block was open read that reply as its own, so its later requests could return another key's value. A non-quiet request now reads any pending quiet replies first
+- Don't let a `get_multi` that stopped early leave replies for a later request (GHSA-5xvx-r8rg-5qm2)
+  - A pipelined `get_multi` stopped by a non-network exception (its block raising or breaking, `Timeout.timeout`, or a value that can't be deserialized) left its unread replies in the response buffer, and a later `get_multi` large enough to use the interleaved path returned them as its own. The buffer is now cleared when a connection closes or a pipelined get starts, and unfinished pipelines are aborted
+  - Affects every supported line (the `get_multi` issue, 4.3 and later); fixed in 5.2.4, 5.1.6, 5.0.12, 4.3.9 and 3.2.15. On 5.2, clients with `defer_drain: true` weren't affected by the `quiet` issue
+
+Bug fixes:
+
+- A failed TLS handshake is now handled as a network error even when openssl wasn't loaded before Dalli
+  - The fix in 5.2.3 (#1215) took effect only when the application had already required openssl: otherwise Dalli's list of TLS errors was empty, so a failed handshake still raised `OpenSSL::SSL::SSLError` and left the socket open
+
+5.2.3
+==========
+
+Security:
+
+- Don't rely on memcached's auto-create for keys that need base64 encoding (GHSA-r47x-6p6r-mvcp)
+  - memcached stores an item made by auto-create (`ma` with `N`/`J`, or `mg` with `N`) for a base64-encoded key without its binary-key flag, and later replies name the item by its raw key bytes. A key containing spaces or CR/LF could then make a multi-get read the replies after it as different ones
+  - `incr` and `decr` with an initial value now create such counters with an add instead. Return values are unchanged
+  - The new `verify_reply_keys` option makes `get_multi`, `get_multi_cas` and `get_multi_with_metadata` ignore replies for keys they didn't request, which guards against items created by older Dalli versions or other clients until memcached is upgraded. It's off by default, since it costs a multi-get about 15%
+  - Affects the meta protocol since 3.2.0; fixed in 5.2.3, 5.1.5, 5.0.11, 4.3.8 and 3.2.14. memcached 1.6.45 and earlier still name these items by their raw key bytes
+
+Bug fixes:
+
+- `get_multi_with_metadata` and its `CacheResult` adapter now retry transient network errors and raise `Dalli::NetworkError` on terminal failures instead of returning misses or partial results (#1222, @jaideeppyne)
+- A per-request `raw: true` write no longer compresses the value (#1213)
+  - Values at or above the compression threshold (4 KB by default) were compressed and flagged even with `raw: true`. Raw reads ignore flags, so reading them back with `raw: true` returned the compressed bytes
+  - Raw values are now stored exactly as given. Callers that pass `raw: true` without `compress: false` store large values uncompressed; Rails' `MemCacheStore` passes `compress: false` and isn't affected
+- Fix `cas` and `cas!` edge cases, and make `flush` attempt every server (#1214)
+  - `cas!` on a missing key used an unconditional set, so a value another client created while the block ran was overwritten. It now adds the key and returns `false` if another client got there first
+  - With `cache_nils: true`, `cas` treated a stored `nil` as a miss and didn't call the block
+  - The docs now say what `cas` and `cas!` return on success: the new CAS value, not `true`. The return values are unchanged
+  - `flush` stopped at the first server that raised, so later servers weren't flushed. It now flushes every server it can reach, then raises the first error
+- Handle failed TLS handshakes like other connection failures, and warn when TLS verification is off (#1215)
+  - A failed or timed-out handshake left the TCP socket open until GC and raised `OpenSSL::SSL::SSLError` directly, so the server was never marked down. The socket is now closed, and the error goes through the usual retry and `socket_max_failures` handling, raising `Dalli::NetworkError` (or `Dalli::RingError` once no server is available)
+  - Certificate and hostname checks come only from the `ssl_context` you pass, and a bare `OpenSSL::SSL::SSLContext.new` checks neither. Dalli now logs a warning once per process when `verify_mode` doesn't include `VERIFY_PEER` or `verify_hostname` is false; see the new TLS section in the README
+- Connection and configuration fixes (#1216)
+  - `memcached://` URIs with a bracketed IPv6 host (`memcached://[::1]:11211`) now connect
+  - A server weight below 1 raises `ArgumentError`. Before, a ring where every weight was 0 raised `FloatDomainError`
+  - A reply line longer than 8 KiB without a CRLF, or one cut short by the connection closing, is now a network error that closes the connection. Before, the line had no length limit
+  - On CRuby, DNS resolution is now limited by `socket_timeout` (`resolv_timeout:`) as well as the connect
+- `Dalli::Client.new` no longer modifies the options hash it's given, and accepts a frozen one (#1217)
+- `value_max_bytes` now counts the key and memcached's per-item overhead (#1217)
+  - memcached's `-I` limit covers the whole item, not only the value. A value of exactly 1 MB passed Dalli's check and then failed in memcached with `Dalli::ServerError`, which `quiet` blocks and `set_multi` silently discarded
+  - Dalli now checks value + key + 63 bytes, the overhead measured on memcached 1.6, and raises `Dalli::ValueOverMaxSize` before sending. A custom `value_max_bytes` is now that much stricter
+  - A String `value_max_bytes` is converted to an Integer instead of raising on every set
+- `Rack::Session::Dalli` fixes (#1217)
+  - It no longer requires the `connection_pool` gem unless `pool_size` or `pool_timeout` is given
+  - `pool_size` and `pool_timeout` are no longer passed to `Dalli::Client`, and the options hash isn't modified
+  - A failed session delete is logged as a warning, not only when `$VERBOSE` is set
+- One value that can't be stored no longer discards other buffered writes (#1218)
+  - A value that raised `Dalli::MarshalError` or `Dalli::ValueOverMaxSize` closed the connection, which dropped writes still buffered from the same `quiet` block or `set_multi`. Multi-server `set_multi` did this without raising
+  - Both `set_multi` paths now store every other key, skip the bad one, and then raise. Single-server `set_multi` used to store nothing
+  - When single-server `set_multi` failed to marshal a value, the error log included every value in the batch. It now lists only the keys
+- `Rack::Session::Dalli`'s 401 for a session deleted mid-request is now a valid Rack response (#1220)
+  - When a session is deleted while a request is using it (for example, by a logout in another request), the store still returns 401 instead of writing the session back. The 401 now has a `content-type` and expires the session cookie, so the browser stops sending the dead session ID
+  - The app's response body is now closed, as the Rack spec requires. Before, it was dropped without `close`
+  - The app has already run when the 401 is returned, so its side effects have happened. This is documented in the README
+
+Improvements:
+
+- Add a `connect_timeout` option to set a separate timeout for DNS resolution and TCP/UNIX socket connects (including reconnects); it defaults to `socket_timeout`. TLS handshakes continue to use `socket_timeout` (#1221, @underwoo16)
+
 5.2.2
 ==========
 

data/README.md

@@ -59,6 +59,17 @@ Dalli::Client.new('localhost:11211', namespace: 'myapp', namespace_separator: '/
 
 The separator must be a single non-alphanumeric character. Valid examples: `:`, `/`, `|`, `.`, `-`, `_`, `#`
 
+### Maximum Item Size
+
+memcached rejects items larger than its `-I` setting (1MB by default). Set `value_max_bytes` to the same value if you change it:
+
+```ruby
+# memcached started with -I 2m
+Dalli::Client.new('localhost:11211', value_max_bytes: 2 * 1024 * 1024)
+```
+
+memcached's limit covers the whole item, so Dalli checks the stored (serialized and compressed) value plus the key plus 63 bytes of per-item overhead against `value_max_bytes`, and raises `Dalli::ValueOverMaxSize` before sending anything larger. With the defaults, the largest value you can store under a 10-byte key is 1,048,576 - 10 - 63 = 1,048,503 bytes.
+
 ### Deferred Draining
 
 By default, a `quiet` (or `multi`) block ends by waiting for the replies memcached still sends for its quiet requests, one round trip for each server the block wrote to. With `defer_drain: true`, the block returns without waiting:
@@ -112,6 +123,35 @@ The digest is MD5 by default. `digest_class:` takes any object that responds to
 
 See the [5.0-Upgrade.md](5.0-Upgrade.md) guide for upgrade information.
 
+### Counters and verify_reply_keys
+
+memcached names an item made by auto-create for a base64-encoded key (one with whitespace, control or non-ASCII characters) by its raw key bytes in later replies. Since 5.2.3, Dalli creates counters for such keys with an add rather than auto-create. Items created before then, or by other clients, can still be named that way until memcached releases a fix (1.6.45 and earlier behave this way). If keys passed to `incr` or `decr` with an initial value can contain untrusted input, set `verify_reply_keys: true` until then: multi-gets then ignore replies for keys they didn't request, at a cost of about 15% per multi-get.
+
+## TLS
+
+Pass an `OpenSSL::SSL::SSLContext` as `ssl_context:` to connect over TLS. Certificate and hostname verification come only from that context: Dalli adds no checks of its own. A bare `OpenSSL::SSL::SSLContext.new` verifies nothing, so it accepts any certificate from any server. Configure the context to verify both:
+
+```ruby
+ssl_context = OpenSSL::SSL::SSLContext.new
+ssl_context.set_params(
+  verify_mode: OpenSSL::SSL::VERIFY_PEER,
+  verify_hostname: true,
+  ca_file: '/path/to/ca.crt' # omit to use the system's trusted certificates
+)
+
+Dalli::Client.new('memcached.example.com:11211', ssl_context: ssl_context)
+```
+
+Dalli logs a warning once per process when the context's `verify_mode` is `VERIFY_NONE`, or when it is `VERIFY_PEER` but `verify_hostname` is false.
+
+A failed TLS handshake counts as a failed connection attempt, like a refused connection: once `socket_max_failures` attempts have failed, the server is marked down and requests fail over to the other servers.
+
+## Rack Sessions
+
+`Rack::Session::Dalli` stores Rack sessions in memcached. If a session is deleted while a request is using it (for example, when the user logs out in another tab during a slow request), the store doesn't write the session back, so a logged-out session can't come back. It replaces the app's response with a `401 Wrong session ID` that expires the session cookie.
+
+The app has already run by then, so whatever it did (database writes, emails, and so on) has happened, and only its response is lost. If a request needs a different outcome, check that the session still exists before making the changes.
+
 ## OpenTelemetry Tracing
 
 Dalli automatically instruments operations with [OpenTelemetry](https://opentelemetry.io/) when the SDK is present. No configuration is required - just add the OpenTelemetry gems to your application:

data/lib/dalli/client.rb

@@ -35,15 +35,20 @@ module Dalli
     #               to memcached.  Default: true.
     # - :compression_min_size - the minimum size (in bytes) for which Dalli will compress values sent to Memcached.
     #                           Defaults to 4K.
+    # - :value_max_bytes - the largest item memcached will accept, i.e. its -I setting. Defaults to 1MB. Like
+    #                     memcached, Dalli counts the key and 63 bytes of item overhead against this limit along
+    #                     with the (serialized, compressed) value, so the largest value that can be stored is
+    #                     value_max_bytes - key bytesize - 63. Larger values raise Dalli::ValueOverMaxSize.
     # - :serializer - defaults to Marshal
     # - :compressor - defaults to Dalli::Compressor, a Zlib-based implementation
     # - :cache_nils - defaults to false, if true Dalli will not treat cached nil values as 'not found' for
     #                 #fetch operations.
     # - :raw        - If set, disables serialization and compression entirely at the client level.
     #                 Only String values are supported. This is useful when the caller handles its own
-    #                 serialization (e.g., Rails' ActiveSupport::Cache). Note: this is different from
-    #                 the per-request :raw option which converts values to strings but still uses the
-    #                 serialization pipeline.
+    #                 serialization (e.g., Rails' ActiveSupport::Cache). The per-request :raw option
+    #                 (e.g. set(key, value, ttl, raw: true)) does the same for a single call: String
+    #                 values are stored as-is, uncompressed, and non-String values raise
+    #                 Dalli::MarshalError.
     # - :digest_class - defaults to Digest::MD5, allows you to pass in an object that responds to the hexdigest method,
     #                   useful for injecting a FIPS compliant hash object.
     # - :otel_db_statement - controls the +db.query.text+ span attribute when OpenTelemetry is loaded.
@@ -58,6 +63,17 @@ module Dalli
     #                  quiet requests (NF, NS, errors) wait on the connection until then, so a client that only makes
     #                  quiet requests should call #drain_deferred_responses periodically. See "Deferred Draining" in
     #                  the README.
+    # - :socket_timeout - timeout in seconds for reads and writes on a connection, and for the TLS
+    #                     handshake. Also used for connecting unless :connect_timeout is set.
+    #                     Default: 1.
+    # - :connect_timeout - timeout in seconds for DNS resolution and TCP/UNIX socket connects,
+    #                      including reconnects after a failure. Defaults to :socket_timeout.
+    #                      TLS handshakes use :socket_timeout.
+    # - :verify_reply_keys - defaults to false. When true, get_multi, get_multi_cas and get_multi_with_metadata only
+    #                        accept replies for the keys they requested; after a reply naming another key, the rest of
+    #                        that server's replies are treated as misses. memcached can name an item by its raw key
+    #                        bytes when the item was created by auto-create from a base64-encoded key, which Dalli no
+    #                        longer does. Costs a multi-get about 15%.
     #
     def initialize(servers = nil, options = {})
       @normalized_servers = ::Dalli::ServersArgNormalizer.normalize_servers(servers)
@@ -226,9 +242,14 @@ module Dalli
     #
     # `req_options` accepts :p_token/:l_token, applied to every key in the batch.
     #
+    # A transient network error is retried automatically. If a server remains
+    # unreachable after retrying, raises Dalli::NetworkError rather than
+    # returning its keys as misses or yielding a partial result.
+    #
     # @param keys [Array<String>] the keys to fetch
     # @param req_options [Hash, nil] routing-token options
     # @return [Hash] key => { value:, cas:, stale:, miss: }
+    # @raise [Dalli::NetworkError] if a server is unreachable after retrying
     def get_multi_with_metadata(*keys, req_options: nil, &block)
       keys.flatten!
       keys.compact!
@@ -371,24 +392,26 @@ module Dalli
     # compare and swap values using optimistic locking.
     # Fetch the existing value for key.
     # If it exists, yield the value to the block.
-    # Add the block's return value as the new value for the key.
-    # Add will fail if someone else changed the value.
+    # Store the block's return value as the new value for the key.
+    # The store will fail if someone else changed the value.
     #
     # Returns:
-    # - nil if the key did not exist.
+    # - nil if the key did not exist (or held nil and :cache_nils is false);
+    #   the block is not called.
     # - false if the value was changed by someone else.
-    # - true if the value was successfully updated.
+    # - the new CAS value (a positive Integer) if the value was successfully updated.
     def cas(key, ttl = nil, req_options = nil, &)
       cas_core(key, false, ttl, req_options, &)
     end
 
     ##
     # like #cas, but will yield to the block whether or not the value
-    # already exists.
+    # already exists. If the key did not exist, the block receives nil.
     #
     # Returns:
-    # - false if the value was changed by someone else.
-    # - true if the value was successfully updated.
+    # - false if the value was changed by someone else, including another
+    #   client creating a missing key while the block ran.
+    # - the new CAS value (a positive Integer) if the value was successfully updated.
     def cas!(key, ttl = nil, req_options = nil, &)
       cas_core(key, true, ttl, req_options, &)
     end
@@ -411,6 +434,11 @@ module Dalli
     # the defer_drain option, before the next non-quiet request), and
     # Dalli client methods invoked inside the block will not
     # have return values
+    #
+    # The block is quiet only for the thread or fiber that runs it. Another
+    # thread or fiber sharing this client can make requests meanwhile; a
+    # non-quiet one first sends the block's buffered requests and discards
+    # their replies, so it reads only its own.
     def quiet
       old = Thread.current[::Dalli::QUIET]
       Thread.current[::Dalli::QUIET] = true
@@ -650,9 +678,20 @@ module Dalli
     ##
     # Flush the memcached server, at 'delay' seconds in the future.
     # Delay defaults to zero seconds, which means an immediate flush.
+    # Every server is attempted. If any of them fails, the first error is
+    # raised once all the others have been flushed.
     ##
     def flush(delay = 0)
-      ring.servers.map { |s| s.request(:flush, delay) }
+      first_error = nil
+      results = ring.servers.map do |s|
+        s.request(:flush, delay)
+      rescue StandardError => e
+        first_error ||= e
+        nil
+      end
+      raise first_error if first_error
+
+      results
     end
     alias flush_all flush
 
@@ -805,16 +844,26 @@ module Dalli
       Dalli.logger.debug { e.inspect }
       Dalli.logger.debug { 'retrying single-server get_multi because of network error' }
       retry
+    rescue Dalli::UnexpectedReplyKeyError => e
+      # The connection is closed and the keys are treated as missing, as the
+      # pipelined get_multi does for a server whose replies are out of step
+      Dalli.logger.warn { e.message }
+      {}
     end
 
     def single_server_get_multi_with_metadata(keys, req_options = nil)
-      keys.map! { |k| @key_manager.validate_key(k.to_s) }
+      validated_keys = keys.map { |k| @key_manager.validate_key(k.to_s) }
       return {} unless (server = single_server)
 
-      result = server.request(:read_multi_with_metadata_req, keys, req_options)
+      result = server.request(:read_multi_with_metadata_req, validated_keys, req_options)
       result.transform_keys! { |k| @key_manager.key_without_namespace(k) }
       result
-    rescue Dalli::NetworkError
+    rescue Dalli::RetryableNetworkError => e
+      Dalli.logger.debug { e.inspect }
+      Dalli.logger.debug { 'retrying single-server get_multi_with_metadata because of network error' }
+      retry
+    rescue Dalli::UnexpectedReplyKeyError => e
+      Dalli.logger.warn { e.message }
       {}
     end
 
@@ -913,12 +962,23 @@ module Dalli
     def cas_core(key, always_set, ttl = nil, req_options = nil)
       validate_routing_tokens!(req_options)
       (value, cas) = perform(:cas, key, req_options)
-      return if value.nil? && !always_set
+      return if !always_set && cas_miss?(value, cas)
 
       newvalue = yield(value)
+      # A CAS of 0 means the key was missing. A set with CAS 0 is unconditional,
+      # so add instead: it fails if another client created the key meanwhile.
+      return perform(:add, key, newvalue, ttl_or_default(ttl), req_options) if cas.zero?
+
       perform(:set, key, newvalue, ttl_or_default(ttl), cas, req_options)
     end
 
+    # The CAS lookup returns [nil, 0] on a miss rather than NOT_FOUND, and a
+    # stored item always has a non-zero CAS. So under cache_nils a zero CAS
+    # marks the miss, and a stored nil is a hit, as in #not_found?.
+    def cas_miss?(value, cas)
+      cache_nils ? cas.zero? : value.nil?
+    end
+
     def fetch_with_lock_request(key, ttl, lock_ttl, recache_threshold, req_options)
       server = ring.server_for_key(key)
       # req_options is the base, not the override: fetch_with_lock's own
@@ -986,7 +1046,10 @@ module Dalli
       retry
     end
 
-    def normalize_options(opts)
+    # Works on a copy so the caller's hash is neither modified here nor able
+    # to change this client's configuration later.
+    def normalize_options(options)
+      opts = options.dup
       opts[:expires_in] = opts[:expires_in].to_i if opts[:expires_in]
       opts
     rescue NoMethodError

data/lib/dalli/options.rb

@@ -43,7 +43,7 @@ module Dalli
       end
     end
 
-    def pipeline_abort
+    def pipeline_abort(...)
       @lock.synchronize do
         super
       end

data/lib/dalli/pipelined_getter.rb

@@ -32,15 +32,21 @@ module Dalli
       return {} if keys.empty?
 
       @ring.lock do
-        # Stores partial results collected during interleaved send phase
-        @partial_results = {}
-        servers = setup_requests(keys, req_options, return_cas: return_cas)
-        start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-
-        # First yield any partial results collected during interleaved send
-        yield_partial_results(&block)
-
-        servers = fetch_responses(servers, start_time, @ring.socket_timeout, &block) until servers.empty?
+        # The servers with replies still to read, updated in place
+        servers = []
+        begin
+          # Stores partial results collected during interleaved send phase
+          @partial_results = {}
+          setup_requests(servers, keys, req_options, return_cas: return_cas)
+          start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
+          # First yield any partial results collected during interleaved send
+          yield_partial_results(&block)
+
+          fetch_responses(servers, start_time, @ring.socket_timeout, &block) until servers.empty?
+        ensure
+          abort_unfinished(servers)
+        end
       end
     rescue Dalli::RetryableNetworkError => e
       Dalli.logger.debug { e.inspect }
@@ -60,6 +66,9 @@ module Dalli
     #
     # `req_options` accepts :p_token/:l_token, applied to every key in the batch.
     #
+    # A transient network error is retried automatically. A terminal
+    # Dalli::NetworkError propagates instead of returning partial metadata.
+    #
     def process_with_metadata(keys, req_options = nil)
       return {} if keys.empty?
 
@@ -67,9 +76,9 @@ module Dalli
         results = {}
         groups_for_keys(keys).each do |server, keys_for_server|
           results.merge!(server.request(:read_multi_with_metadata_req, keys_for_server, req_options))
-        rescue Dalli::RetryableNetworkError
+        rescue Dalli::NetworkError
           raise
-        rescue DalliError, NetworkError => e
+        rescue DalliError => e
           Dalli.logger.debug { e.inspect }
           Dalli.logger.debug { "unable to get keys for server #{server.name}" }
         end
@@ -93,9 +102,9 @@ module Dalli
 
     # Sends each server's queries and its terminating noop before building
     # the next server's, so memcached is already answering the first servers
-    # while the rest are prepared. Returns the servers with a pending response.
-    def setup_requests(keys, req_options = nil, return_cas: true)
-      started = []
+    # while the rest are prepared. Adds the servers with a pending response
+    # to started.
+    def setup_requests(started, keys, req_options = nil, return_cas: true)
       groups_for_keys(keys).each do |server, keys_for_server|
         make_getkq_request(server, keys_for_server, req_options, return_cas: return_cas)
         next unless server.connected?
@@ -108,7 +117,6 @@ module Dalli
       rescue Dalli::DalliError
         started.delete(server)
       end
-      started
     end
 
     ##
@@ -152,29 +160,39 @@ module Dalli
       raise
     end
 
-    # Swallows Dalli::NetworkError
+    # Swallows Dalli::NetworkError. Empties servers, so they aren't aborted
+    # again.
     def abort_without_timeout(servers)
       servers.each(&:pipeline_abort)
+      servers.clear
+    end
+
+    # When #process stops before reading every server's replies (the caller's
+    # block raises or breaks, a Timeout.timeout fires, a value can't be
+    # deserialized), the rest are still on the socket and some may be in the
+    # response buffer. Closing those connections keeps them from being read
+    # as a later request's replies. Aborts that #process already handled
+    # emptied servers.
+    def abort_unfinished(servers)
+      servers.each { |server| server.pipeline_abort(timed_out: false) }
+      servers.clear
     end
 
+    # Reads the replies available, removing finished servers from servers
     def fetch_responses(servers, start_time, timeout, &block)
       # Remove any servers which are not connected
       servers.select!(&:connected?)
-      return [] if servers.empty?
+      return if servers.empty?
 
       time_left = remaining_time(start_time, timeout)
       readable_servers = servers_with_response(servers, time_left)
-      if readable_servers.empty?
-        abort_with_timeout(servers)
-        return []
-      end
+      return abort_with_timeout(servers) if readable_servers.empty?
 
       # Loop through the servers with responses, and
       # delete any from our list that are finished
       readable_servers.each do |server|
         servers.delete(server) if process_server(server, &block)
       end
-      servers
     rescue NetworkError
       # Abort and raise if we encountered a network error.  This triggers
       # a retry at the top level on RetryableNetworkError.
@@ -191,12 +209,10 @@ module Dalli
 
     # Swallows Dalli::NetworkError
     def abort_with_timeout(servers)
-      abort_without_timeout(servers)
       servers.each do |server|
         Dalli.logger.debug { "memcached at #{server.name} did not response within timeout" }
       end
-
-      true # Required to simplify caller
+      abort_without_timeout(servers)
     end
 
     # Processes responses from a server.  Returns true if there are no

data/lib/dalli/pipelined_setter.rb

@@ -18,6 +18,9 @@ module Dalli
     # A transient network error is retried automatically. If a server remains
     # unreachable after retrying, raises Dalli::NetworkError.
     #
+    # A value that can't be marshalled or is over value_max_bytes is skipped;
+    # the other keys are still stored, then the first such error is raised.
+    #
     # @param hash [Hash] key-value pairs to set
     # @param ttl [Integer] time-to-live in seconds
     # @param req_options [Hash] options passed to each set operation
@@ -26,10 +29,12 @@ module Dalli
     def process(hash, ttl, req_options)
       return if hash.empty?
 
+      @value_error = nil
       @ring.lock do
         servers = setup_requests(hash, ttl, req_options)
         finish_requests(servers)
       end
+      raise @value_error if @value_error
     rescue Dalli::RetryableNetworkError => e
       Dalli.logger.debug { e.inspect }
       Dalli.logger.debug { 'retrying pipelined sets because of network error' }
@@ -38,9 +43,12 @@ module Dalli
 
     private
 
+    # Values are looked up by each key as sent. The caller's key can't be
+    # rebuilt from it: a long key is digested, and a Symbol becomes a String.
     def setup_requests(hash, ttl, req_options)
-      groups = groups_for_keys(hash.keys)
-      make_set_requests(groups, hash, ttl, req_options)
+      pairs = hash.transform_keys { |k| @key_manager.validate_key(k.to_s) }
+      groups = groups_for_keys(pairs.keys)
+      make_set_requests(groups, pairs, ttl, req_options)
       groups.keys
     end
 
@@ -54,14 +62,16 @@ module Dalli
     # RetryableNetworkError too -- since NetworkError < DalliError -- dropping
     # this server's keys on a transient hiccup instead of retrying. Only a
     # non-network DalliError should be swallowed here.
-    def make_set_requests(groups, hash, ttl, req_options)
+    def make_set_requests(groups, pairs, ttl, req_options)
       groups.each do |server, keys_for_server|
         keys_for_server.each do |key|
-          original_key = @key_manager.key_without_namespace(key)
-          value = hash[original_key]
-          server.request(:pipelined_set, key, value, ttl, req_options)
+          server.request(:pipelined_set, key, pairs[key], ttl, req_options)
         rescue Dalli::NetworkError
           raise
+        rescue Dalli::MarshalError, Dalli::ValueOverMaxSize => e
+          # Nothing was written for this key, so the server's other keys are
+          # unaffected; raised by #process once the batch is complete
+          @value_error ||= e
         rescue DalliError => e
           Dalli.logger.debug { e.inspect }
           Dalli.logger.debug { "unable to set key #{key} for server #{server.name}" }
@@ -84,8 +94,7 @@ module Dalli
     end
 
     def groups_for_keys(keys)
-      validated_keys = keys.map { |k| @key_manager.validate_key(k.to_s) }
-      groups = @ring.keys_grouped_by_server(validated_keys)
+      groups = @ring.keys_grouped_by_server(keys)
 
       if (unfound_keys = groups.delete(nil))
         Dalli.logger.debug do