dalli 5.2.1 → 5.2.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 42431b1f305b3fe3d3923c2b66f3d1edfb402da378c434a587bb2c85d6688055
-  data.tar.gz: 39bc473979fb49cd488853d39a64675884ce89951e96a7b5925dc88c8307219a
+  metadata.gz: 1a31a4159ee5cd637dca6676f343595f653798864bef28b109fbc329debe2aff
+  data.tar.gz: efba0a1a4bb20eff2fd4f6b26c15359351484401502a798fa398b5bf1301d046
 SHA512:
-  metadata.gz: 810da4f09ea75bfa616471a48686cb0804b820b3a629c1ec53c3ecc7146c76d94e7afa729ce3361287ccafaa71d97a0ed5053f0b6adc0d8cea91c5de81070231
-  data.tar.gz: c18a6a1f3d1dac3fcf5f640c8714ff68e70e07dc8d154d1471fc62c9ccb5ee6a970a3d079b042d4c5337ad11f39090e69022ef6eb34dac281906e9834863b285
+  metadata.gz: 38f6d3ba18790b94875f18bf6239a520f57144ba763826063471bea2047aa6def2cc528d37de35f29037a0c33ca9bb8b46d7ed0ddc5dccebf8914888c71dafd7
+  data.tar.gz: 9f091240e20313b7cbb9250e900cbf3aa02f9aedbffceabeedc7355bc780cd3895e231c77bb091ab97680dbdac85e093635e3cfbca904fd7f095512181e4a59d

data/CHANGELOG.md

@@ -4,6 +4,87 @@ Dalli Changelog
 Unreleased
 ==========
 
+5.2.3
+==========
+
+Security:
+
+- Don't rely on memcached's auto-create for keys that need base64 encoding (GHSA-r47x-6p6r-mvcp)
+  - memcached stores an item made by auto-create (`ma` with `N`/`J`, or `mg` with `N`) for a base64-encoded key without its binary-key flag, and later replies name the item by its raw key bytes. A key containing spaces or CR/LF could then make a multi-get read the replies after it as different ones
+  - `incr` and `decr` with an initial value now create such counters with an add instead. Return values are unchanged
+  - The new `verify_reply_keys` option makes `get_multi`, `get_multi_cas` and `get_multi_with_metadata` ignore replies for keys they didn't request, which guards against items created by older Dalli versions or other clients until memcached is upgraded. It's off by default, since it costs a multi-get about 15%
+  - Affects the meta protocol since 3.2.0; fixed in 5.2.3, 5.1.5, 5.0.11, 4.3.8 and 3.2.14. memcached 1.6.45 and earlier still name these items by their raw key bytes
+
+Bug fixes:
+
+- `get_multi_with_metadata` and its `CacheResult` adapter now retry transient network errors and raise `Dalli::NetworkError` on terminal failures instead of returning misses or partial results (#1222, @jaideeppyne)
+- A per-request `raw: true` write no longer compresses the value (#1213)
+  - Values at or above the compression threshold (4 KB by default) were compressed and flagged even with `raw: true`. Raw reads ignore flags, so reading them back with `raw: true` returned the compressed bytes
+  - Raw values are now stored exactly as given. Callers that pass `raw: true` without `compress: false` store large values uncompressed; Rails' `MemCacheStore` passes `compress: false` and isn't affected
+- Fix `cas` and `cas!` edge cases, and make `flush` attempt every server (#1214)
+  - `cas!` on a missing key used an unconditional set, so a value another client created while the block ran was overwritten. It now adds the key and returns `false` if another client got there first
+  - With `cache_nils: true`, `cas` treated a stored `nil` as a miss and didn't call the block
+  - The docs now say what `cas` and `cas!` return on success: the new CAS value, not `true`. The return values are unchanged
+  - `flush` stopped at the first server that raised, so later servers weren't flushed. It now flushes every server it can reach, then raises the first error
+- Handle failed TLS handshakes like other connection failures, and warn when TLS verification is off (#1215)
+  - A failed or timed-out handshake left the TCP socket open until GC and raised `OpenSSL::SSL::SSLError` directly, so the server was never marked down. The socket is now closed, and the error goes through the usual retry and `socket_max_failures` handling, raising `Dalli::NetworkError` (or `Dalli::RingError` once no server is available)
+  - Certificate and hostname checks come only from the `ssl_context` you pass, and a bare `OpenSSL::SSL::SSLContext.new` checks neither. Dalli now logs a warning once per process when `verify_mode` doesn't include `VERIFY_PEER` or `verify_hostname` is false; see the new TLS section in the README
+- Connection and configuration fixes (#1216)
+  - `memcached://` URIs with a bracketed IPv6 host (`memcached://[::1]:11211`) now connect
+  - A server weight below 1 raises `ArgumentError`. Before, a ring where every weight was 0 raised `FloatDomainError`
+  - A reply line longer than 8 KiB without a CRLF, or one cut short by the connection closing, is now a network error that closes the connection. Before, the line had no length limit
+  - On CRuby, DNS resolution is now limited by `socket_timeout` (`resolv_timeout:`) as well as the connect
+- `Dalli::Client.new` no longer modifies the options hash it's given, and accepts a frozen one (#1217)
+- `value_max_bytes` now counts the key and memcached's per-item overhead (#1217)
+  - memcached's `-I` limit covers the whole item, not only the value. A value of exactly 1 MB passed Dalli's check and then failed in memcached with `Dalli::ServerError`, which `quiet` blocks and `set_multi` silently discarded
+  - Dalli now checks value + key + 63 bytes, the overhead measured on memcached 1.6, and raises `Dalli::ValueOverMaxSize` before sending. A custom `value_max_bytes` is now that much stricter
+  - A String `value_max_bytes` is converted to an Integer instead of raising on every set
+- `Rack::Session::Dalli` fixes (#1217)
+  - It no longer requires the `connection_pool` gem unless `pool_size` or `pool_timeout` is given
+  - `pool_size` and `pool_timeout` are no longer passed to `Dalli::Client`, and the options hash isn't modified
+  - A failed session delete is logged as a warning, not only when `$VERBOSE` is set
+- One value that can't be stored no longer discards other buffered writes (#1218)
+  - A value that raised `Dalli::MarshalError` or `Dalli::ValueOverMaxSize` closed the connection, which dropped writes still buffered from the same `quiet` block or `set_multi`. Multi-server `set_multi` did this without raising
+  - Both `set_multi` paths now store every other key, skip the bad one, and then raise. Single-server `set_multi` used to store nothing
+  - When single-server `set_multi` failed to marshal a value, the error log included every value in the batch. It now lists only the keys
+- `Rack::Session::Dalli`'s 401 for a session deleted mid-request is now a valid Rack response (#1220)
+  - When a session is deleted while a request is using it (for example, by a logout in another request), the store still returns 401 instead of writing the session back. The 401 now has a `content-type` and expires the session cookie, so the browser stops sending the dead session ID
+  - The app's response body is now closed, as the Rack spec requires. Before, it was dropped without `close`
+  - The app has already run when the 401 is returned, so its side effects have happened. This is documented in the README
+
+Improvements:
+
+- Add a `connect_timeout` option to set a separate timeout for DNS resolution and TCP/UNIX socket connects (including reconnects); it defaults to `socket_timeout`. TLS handshakes continue to use `socket_timeout` (#1221, @underwoo16)
+
+5.2.2
+==========
+
+Security:
+
+- Keep the caller's key when retrying a request (GHSA-m252-9cgf-vx2w)
+  - With a `namespace`, a request retried after a transient network error (a timeout, or a connection closed by memcached or a proxy) applied the namespace a second time, so a retried read could return a different key's value and a retried write could overwrite a different key
+  - Affects single-key operations since 5.0.3, single-server `get_multi` since 5.1.0, and `get_with_metadata` and `fetch_with_lock` since 4.1.0; fixed in 5.2.2, 5.1.4, 5.0.10 and 4.3.7. Clients without a namespace aren't affected
+- Complete the fix for per-request `raw: true` on reads (GHSA-wr87-m4jw-29x5)
+  - A read made with `raw: true` doesn't ask for flags, but a reply carrying them anyway (from a proxy or a hostile server) still had its value deserialized. Raw reads now ignore flags in the reply
+  - Affects 5.2.1 and earlier; fixed in 5.2.2, 5.1.4, 5.0.10, 4.3.7 and 3.2.13
+
+Bug fixes:
+
+- `get_with_metadata` and `fetch_with_lock` retried the final error raised when a server is marked down, so with `down_retry_delay: 0` they retried an unresponsive server forever. They now retry only retryable errors, like other operations
+- Fix a regression in 5.2.1: a `get_multi` that didn't finish within `socket_timeout` counted toward `socket_max_failures`, so two slow `get_multi` calls in a row marked a healthy server down. It now just closes the connection
+- Fix a regression in 5.2.1: over TLS, every request buffered in a `quiet` block was sent as its own TLS record and system call (2.5 times slower for a block of 2000 deletes). Each flush is one write again
+- Handle malformed replies from a broken or hostile server or proxy: a negative value size, a hit with no key, or a hit with no `s` flag could leave the connection out of step or raise a non-Dalli error
+- With `decompressed_max_bytes` in effect, data after the end of a compressed value's stream was returned as part of the value. It's now ignored, as without the limit
+- With a `digest_class` whose digests aren't short hex strings, shortening a long key could loop forever. It now raises `ArgumentError`
+
+Notes:
+
+- Document how long keys are shortened, and that a short key written in the same form names the same item, in the README's security note (#1200)
+
+Development:
+
+- Run the Tests, RuboCop and Profiles workflows on pushes to `main` and the `*-stable` branches only, so a pull request's branch isn't tested twice (once for the push and once for the pull request) (#1198)
+
 5.2.1
 ==========
 

data/README.md

@@ -59,6 +59,17 @@ Dalli::Client.new('localhost:11211', namespace: 'myapp', namespace_separator: '/
 
 The separator must be a single non-alphanumeric character. Valid examples: `:`, `/`, `|`, `.`, `-`, `_`, `#`
 
+### Maximum Item Size
+
+memcached rejects items larger than its `-I` setting (1MB by default). Set `value_max_bytes` to the same value if you change it:
+
+```ruby
+# memcached started with -I 2m
+Dalli::Client.new('localhost:11211', value_max_bytes: 2 * 1024 * 1024)
+```
+
+memcached's limit covers the whole item, so Dalli checks the stored (serialized and compressed) value plus the key plus 63 bytes of per-item overhead against `value_max_bytes`, and raises `Dalli::ValueOverMaxSize` before sending anything larger. With the defaults, the largest value you can store under a 10-byte key is 1,048,576 - 10 - 63 = 1,048,503 bytes.
+
 ### Deferred Draining
 
 By default, a `quiet` (or `multi`) block ends by waiting for the replies memcached still sends for its quiet requests, one round trip for each server the block wrote to. With `defer_drain: true`, the block returns without waiting:
@@ -97,8 +108,50 @@ Dalli::Client.new('localhost:11211', serializer: Dalli::JSONSerializer)
 
 Per-request `raw: true` returns the stored bytes without deserializing or decompressing them, on every read method (`get`, `get_multi`, `get_cas`, `get_with_metadata` and `fetch`).
 
+### Long keys
+
+memcached limits keys to 250 bytes. Dalli shortens a longer key (counting its namespace) to the start of the key, followed by `:md5:` and a hex digest of the whole key:
+
+```ruby
+dc.set("report:#{'x' * 300}", 'data')
+# stored under "report:xxx...xxx:md5:<32 hex characters>"
+```
+
+The shortened key is an ordinary memcached key, so a short key written in exactly that form names the same item, and reading or writing it reads or overwrites the long key's value. This only matters if untrusted input can choose an entire cache key. If it can, hash the untrusted part yourself so the key never needs shortening, or reject keys that contain `:md5:`.
+
+The digest is MD5 by default. `digest_class:` takes any object that responds to `hexdigest`, such as `Digest::SHA256`. Changing it renames every shortened key, so existing entries for long keys are missed once.
+
 See the [5.0-Upgrade.md](5.0-Upgrade.md) guide for upgrade information.
 
+### Counters and verify_reply_keys
+
+memcached names an item made by auto-create for a base64-encoded key (one with whitespace, control or non-ASCII characters) by its raw key bytes in later replies. Since 5.2.3, Dalli creates counters for such keys with an add rather than auto-create. Items created before then, or by other clients, can still be named that way until memcached releases a fix (1.6.45 and earlier behave this way). If keys passed to `incr` or `decr` with an initial value can contain untrusted input, set `verify_reply_keys: true` until then: multi-gets then ignore replies for keys they didn't request, at a cost of about 15% per multi-get.
+
+## TLS
+
+Pass an `OpenSSL::SSL::SSLContext` as `ssl_context:` to connect over TLS. Certificate and hostname verification come only from that context: Dalli adds no checks of its own. A bare `OpenSSL::SSL::SSLContext.new` verifies nothing, so it accepts any certificate from any server. Configure the context to verify both:
+
+```ruby
+ssl_context = OpenSSL::SSL::SSLContext.new
+ssl_context.set_params(
+  verify_mode: OpenSSL::SSL::VERIFY_PEER,
+  verify_hostname: true,
+  ca_file: '/path/to/ca.crt' # omit to use the system's trusted certificates
+)
+
+Dalli::Client.new('memcached.example.com:11211', ssl_context: ssl_context)
+```
+
+Dalli logs a warning once per process when the context's `verify_mode` is `VERIFY_NONE`, or when it is `VERIFY_PEER` but `verify_hostname` is false.
+
+A failed TLS handshake counts as a failed connection attempt, like a refused connection: once `socket_max_failures` attempts have failed, the server is marked down and requests fail over to the other servers.
+
+## Rack Sessions
+
+`Rack::Session::Dalli` stores Rack sessions in memcached. If a session is deleted while a request is using it (for example, when the user logs out in another tab during a slow request), the store doesn't write the session back, so a logged-out session can't come back. It replaces the app's response with a `401 Wrong session ID` that expires the session cookie.
+
+The app has already run by then, so whatever it did (database writes, emails, and so on) has happened, and only its response is lost. If a request needs a different outcome, check that the session still exists before making the changes.
+
 ## OpenTelemetry Tracing
 
 Dalli automatically instruments operations with [OpenTelemetry](https://opentelemetry.io/) when the SDK is present. No configuration is required - just add the OpenTelemetry gems to your application:

data/lib/dalli/client.rb

@@ -35,15 +35,20 @@ module Dalli
     #               to memcached.  Default: true.
     # - :compression_min_size - the minimum size (in bytes) for which Dalli will compress values sent to Memcached.
     #                           Defaults to 4K.
+    # - :value_max_bytes - the largest item memcached will accept, i.e. its -I setting. Defaults to 1MB. Like
+    #                     memcached, Dalli counts the key and 63 bytes of item overhead against this limit along
+    #                     with the (serialized, compressed) value, so the largest value that can be stored is
+    #                     value_max_bytes - key bytesize - 63. Larger values raise Dalli::ValueOverMaxSize.
     # - :serializer - defaults to Marshal
     # - :compressor - defaults to Dalli::Compressor, a Zlib-based implementation
     # - :cache_nils - defaults to false, if true Dalli will not treat cached nil values as 'not found' for
     #                 #fetch operations.
     # - :raw        - If set, disables serialization and compression entirely at the client level.
     #                 Only String values are supported. This is useful when the caller handles its own
-    #                 serialization (e.g., Rails' ActiveSupport::Cache). Note: this is different from
-    #                 the per-request :raw option which converts values to strings but still uses the
-    #                 serialization pipeline.
+    #                 serialization (e.g., Rails' ActiveSupport::Cache). The per-request :raw option
+    #                 (e.g. set(key, value, ttl, raw: true)) does the same for a single call: String
+    #                 values are stored as-is, uncompressed, and non-String values raise
+    #                 Dalli::MarshalError.
     # - :digest_class - defaults to Digest::MD5, allows you to pass in an object that responds to the hexdigest method,
     #                   useful for injecting a FIPS compliant hash object.
     # - :otel_db_statement - controls the +db.query.text+ span attribute when OpenTelemetry is loaded.
@@ -58,6 +63,17 @@ module Dalli
     #                  quiet requests (NF, NS, errors) wait on the connection until then, so a client that only makes
     #                  quiet requests should call #drain_deferred_responses periodically. See "Deferred Draining" in
     #                  the README.
+    # - :socket_timeout - timeout in seconds for reads and writes on a connection, and for the TLS
+    #                     handshake. Also used for connecting unless :connect_timeout is set.
+    #                     Default: 1.
+    # - :connect_timeout - timeout in seconds for DNS resolution and TCP/UNIX socket connects,
+    #                      including reconnects after a failure. Defaults to :socket_timeout.
+    #                      TLS handshakes use :socket_timeout.
+    # - :verify_reply_keys - defaults to false. When true, get_multi, get_multi_cas and get_multi_with_metadata only
+    #                        accept replies for the keys they requested; after a reply naming another key, the rest of
+    #                        that server's replies are treated as misses. memcached can name an item by its raw key
+    #                        bytes when the item was created by auto-create from a base64-encoded key, which Dalli no
+    #                        longer does. Costs a multi-get about 15%.
     #
     def initialize(servers = nil, options = {})
       @normalized_servers = ::Dalli::ServersArgNormalizer.normalize_servers(servers)
@@ -144,14 +160,18 @@ module Dalli
     #
     def get_with_metadata(key, options = {})
       validate_routing_tokens!(options)
-      key = key.to_s
-      key = @key_manager.validate_key(key)
+      # A retry re-runs this method with its original arguments, so `key` must
+      # stay as the caller gave it; reassigning it would add the namespace again.
+      validated_key = @key_manager.validate_key(key.to_s)
 
-      server = ring.server_for_key(key)
-      Instrumentation.trace('get_with_metadata', trace_attrs('get_with_metadata', key, server)) do
-        server.request(:meta_get, key, options)
+      server = ring.server_for_key(validated_key)
+      Instrumentation.trace('get_with_metadata', trace_attrs('get_with_metadata', validated_key, server)) do
+        server.request(:meta_get, validated_key, options)
       end
-    rescue NetworkError => e
+    # Only a retryable error is retried. The NetworkError raised when a server
+    # is marked down is final, as in #perform; retrying it looped forever once
+    # down_retry_delay let the same server be tried again.
+    rescue RetryableNetworkError => e
       Dalli.logger.debug { e.inspect }
       Dalli.logger.debug { 'retrying get_with_metadata with new server' }
       retry
@@ -222,9 +242,14 @@ module Dalli
     #
     # `req_options` accepts :p_token/:l_token, applied to every key in the batch.
     #
+    # A transient network error is retried automatically. If a server remains
+    # unreachable after retrying, raises Dalli::NetworkError rather than
+    # returning its keys as misses or yielding a partial result.
+    #
     # @param keys [Array<String>] the keys to fetch
     # @param req_options [Hash, nil] routing-token options
     # @return [Hash] key => { value:, cas:, stale:, miss: }
+    # @raise [Dalli::NetworkError] if a server is unreachable after retrying
     def get_multi_with_metadata(*keys, req_options: nil, &block)
       keys.flatten!
       keys.compact!
@@ -347,14 +372,17 @@ module Dalli
       validate_integer!(:lock_ttl, lock_ttl)
       validate_integer!(:recache_threshold, recache_threshold)
       validate_routing_tokens!(req_options)
-      key = key.to_s
-      key = @key_manager.validate_key(key)
+      # Left unchanged for the retry below; see get_with_metadata
+      validated_key = @key_manager.validate_key(key.to_s)
 
-      server = ring.server_for_key(key)
-      Instrumentation.trace('fetch_with_lock', trace_attrs('fetch_with_lock', key, server)) do
-        fetch_with_lock_request(key, ttl, lock_ttl, recache_threshold, req_options, &block)
+      server = ring.server_for_key(validated_key)
+      Instrumentation.trace('fetch_with_lock', trace_attrs('fetch_with_lock', validated_key, server)) do
+        fetch_with_lock_request(validated_key, ttl, lock_ttl, recache_threshold, req_options, &block)
       end
-    rescue NetworkError => e
+    # Only a retryable error is retried. The NetworkError raised when a server
+    # is marked down is final, as in #perform; retrying it looped forever once
+    # down_retry_delay let the same server be tried again.
+    rescue RetryableNetworkError => e
       Dalli.logger.debug { e.inspect }
       Dalli.logger.debug { 'retrying fetch_with_lock with new server' }
       retry
@@ -364,24 +392,26 @@ module Dalli
     # compare and swap values using optimistic locking.
     # Fetch the existing value for key.
     # If it exists, yield the value to the block.
-    # Add the block's return value as the new value for the key.
-    # Add will fail if someone else changed the value.
+    # Store the block's return value as the new value for the key.
+    # The store will fail if someone else changed the value.
     #
     # Returns:
-    # - nil if the key did not exist.
+    # - nil if the key did not exist (or held nil and :cache_nils is false);
+    #   the block is not called.
     # - false if the value was changed by someone else.
-    # - true if the value was successfully updated.
+    # - the new CAS value (a positive Integer) if the value was successfully updated.
     def cas(key, ttl = nil, req_options = nil, &)
       cas_core(key, false, ttl, req_options, &)
     end
 
     ##
     # like #cas, but will yield to the block whether or not the value
-    # already exists.
+    # already exists. If the key did not exist, the block receives nil.
     #
     # Returns:
-    # - false if the value was changed by someone else.
-    # - true if the value was successfully updated.
+    # - false if the value was changed by someone else, including another
+    #   client creating a missing key while the block ran.
+    # - the new CAS value (a positive Integer) if the value was successfully updated.
     def cas!(key, ttl = nil, req_options = nil, &)
       cas_core(key, true, ttl, req_options, &)
     end
@@ -643,9 +673,20 @@ module Dalli
     ##
     # Flush the memcached server, at 'delay' seconds in the future.
     # Delay defaults to zero seconds, which means an immediate flush.
+    # Every server is attempted. If any of them fails, the first error is
+    # raised once all the others have been flushed.
     ##
     def flush(delay = 0)
-      ring.servers.map { |s| s.request(:flush, delay) }
+      first_error = nil
+      results = ring.servers.map do |s|
+        s.request(:flush, delay)
+      rescue StandardError => e
+        first_error ||= e
+        nil
+      end
+      raise first_error if first_error
+
+      results
     end
     alias flush_all flush
 
@@ -787,26 +828,37 @@ module Dalli
     # to route to at all is still silent, matching Ring#keys_grouped_by_server
     # dropping a key it can't route on both the single- and multi-server paths.
     def single_server_get_multi(keys, req_options = nil)
-      keys.map! { |k| @key_manager.validate_key(k.to_s) }
+      # A new array, so the retry below starts again from the caller's keys
+      validated_keys = keys.map { |k| @key_manager.validate_key(k.to_s) }
       return {} unless (server = single_server)
 
-      result = server.request(:read_multi_req, keys, req_options)
+      result = server.request(:read_multi_req, validated_keys, req_options)
       result.transform_keys! { |k| @key_manager.key_without_namespace(k) }
       result
     rescue Dalli::RetryableNetworkError => e
       Dalli.logger.debug { e.inspect }
       Dalli.logger.debug { 'retrying single-server get_multi because of network error' }
       retry
+    rescue Dalli::UnexpectedReplyKeyError => e
+      # The connection is closed and the keys are treated as missing, as the
+      # pipelined get_multi does for a server whose replies are out of step
+      Dalli.logger.warn { e.message }
+      {}
     end
 
     def single_server_get_multi_with_metadata(keys, req_options = nil)
-      keys.map! { |k| @key_manager.validate_key(k.to_s) }
+      validated_keys = keys.map { |k| @key_manager.validate_key(k.to_s) }
       return {} unless (server = single_server)
 
-      result = server.request(:read_multi_with_metadata_req, keys, req_options)
+      result = server.request(:read_multi_with_metadata_req, validated_keys, req_options)
       result.transform_keys! { |k| @key_manager.key_without_namespace(k) }
       result
-    rescue Dalli::NetworkError
+    rescue Dalli::RetryableNetworkError => e
+      Dalli.logger.debug { e.inspect }
+      Dalli.logger.debug { 'retrying single-server get_multi_with_metadata because of network error' }
+      retry
+    rescue Dalli::UnexpectedReplyKeyError => e
+      Dalli.logger.warn { e.message }
       {}
     end
 
@@ -905,12 +957,23 @@ module Dalli
     def cas_core(key, always_set, ttl = nil, req_options = nil)
       validate_routing_tokens!(req_options)
       (value, cas) = perform(:cas, key, req_options)
-      return if value.nil? && !always_set
+      return if !always_set && cas_miss?(value, cas)
 
       newvalue = yield(value)
+      # A CAS of 0 means the key was missing. A set with CAS 0 is unconditional,
+      # so add instead: it fails if another client created the key meanwhile.
+      return perform(:add, key, newvalue, ttl_or_default(ttl), req_options) if cas.zero?
+
       perform(:set, key, newvalue, ttl_or_default(ttl), cas, req_options)
     end
 
+    # The CAS lookup returns [nil, 0] on a miss rather than NOT_FOUND, and a
+    # stored item always has a non-zero CAS. So under cache_nils a zero CAS
+    # marks the miss, and a stored nil is a hit, as in #not_found?.
+    def cas_miss?(value, cas)
+      cache_nils ? cas.zero? : value.nil?
+    end
+
     def fetch_with_lock_request(key, ttl, lock_ttl, recache_threshold, req_options)
       server = ring.server_for_key(key)
       # req_options is the base, not the override: fetch_with_lock's own
@@ -957,18 +1020,20 @@ module Dalli
       # rubocop:enable Naming/MethodParameterName
       return yield if block_given?
 
-      key = key.to_s
-      key = @key_manager.validate_key(key)
+      # `retry` below re-runs this method with its original arguments, so `key`
+      # must stay as the caller gave it: reassigning it would apply the
+      # namespace a second time and send the retry to a different key.
+      validated_key = @key_manager.validate_key(key.to_s)
 
-      server = ring.server_for_key(key)
+      server = ring.server_for_key(validated_key)
 
       if Instrumentation.enabled?
         op_name = op.name
-        Instrumentation.trace(op_name, trace_attrs(op_name, key, server)) do
-          server.request(op, key, ...)
+        Instrumentation.trace(op_name, trace_attrs(op_name, validated_key, server)) do
+          server.request(op, validated_key, ...)
         end
       else
-        server.request(op, key, ...)
+        server.request(op, validated_key, ...)
       end
     rescue RetryableNetworkError => e
       Dalli.logger.debug { e.inspect }
@@ -976,7 +1041,10 @@ module Dalli
       retry
     end
 
-    def normalize_options(opts)
+    # Works on a copy so the caller's hash is neither modified here nor able
+    # to change this client's configuration later.
+    def normalize_options(options)
+      opts = options.dup
       opts[:expires_in] = opts[:expires_in].to_i if opts[:expires_in]
       opts
     rescue NoMethodError

data/lib/dalli/compressor.rb

@@ -39,7 +39,12 @@ module Dalli
       out = String.new(encoding: Encoding::BINARY)
       0.step(data.bytesize - 1, INFLATE_SLICE_BYTES) do |pos|
         inflate_slice(inflater, data.byteslice(pos, INFLATE_SLICE_BYTES), out, max_bytes)
+        # Anything after the end of the stream isn't part of the value, as
+        # with Zlib::Inflate.inflate; finish would return it as output
+        break if inflater.finished?
       end
+      return out if inflater.finished?
+
       append_within_limit(out, inflater.finish, max_bytes)
     ensure
       inflater.close

data/lib/dalli/key_manager.rb

@@ -122,15 +122,18 @@ module Dalli
     ##
     def truncated_key(key)
       digest = digest_class.hexdigest(key)
-      prefix = key[0, prefix_length(digest)]
+      prefix = key[0, [prefix_length(digest), 0].max]
       truncated = "#{prefix}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
       # A prefix with whitespace or non-ASCII characters sends the whole key
       # base64-encoded, which can still be too long on the wire; shorten it.
-      while wire_length(truncated) > MAX_KEY_LENGTH
+      while wire_length(truncated) > MAX_KEY_LENGTH && !prefix.empty?
         prefix = prefix[0...-1]
         truncated = "#{prefix}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
       end
-      truncated
+      return truncated if wire_length(truncated) <= MAX_KEY_LENGTH
+
+      # Only possible with a digest_class whose digests aren't short hex strings
+      raise ArgumentError, "#{digest_class} digests are too long to fit in a truncated key"
     end
 
     # memcached's key limit applies to the key as sent: in bytes, and after

data/lib/dalli/pipelined_getter.rb

@@ -60,6 +60,9 @@ module Dalli
     #
     # `req_options` accepts :p_token/:l_token, applied to every key in the batch.
     #
+    # A transient network error is retried automatically. A terminal
+    # Dalli::NetworkError propagates instead of returning partial metadata.
+    #
     def process_with_metadata(keys, req_options = nil)
       return {} if keys.empty?
 
@@ -67,9 +70,9 @@ module Dalli
         results = {}
         groups_for_keys(keys).each do |server, keys_for_server|
           results.merge!(server.request(:read_multi_with_metadata_req, keys_for_server, req_options))
-        rescue Dalli::RetryableNetworkError
+        rescue Dalli::NetworkError
           raise
-        rescue DalliError, NetworkError => e
+        rescue DalliError => e
           Dalli.logger.debug { e.inspect }
           Dalli.logger.debug { "unable to get keys for server #{server.name}" }
         end

data/lib/dalli/pipelined_setter.rb

@@ -18,6 +18,9 @@ module Dalli
     # A transient network error is retried automatically. If a server remains
     # unreachable after retrying, raises Dalli::NetworkError.
     #
+    # A value that can't be marshalled or is over value_max_bytes is skipped;
+    # the other keys are still stored, then the first such error is raised.
+    #
     # @param hash [Hash] key-value pairs to set
     # @param ttl [Integer] time-to-live in seconds
     # @param req_options [Hash] options passed to each set operation
@@ -26,10 +29,12 @@ module Dalli
     def process(hash, ttl, req_options)
       return if hash.empty?
 
+      @value_error = nil
       @ring.lock do
         servers = setup_requests(hash, ttl, req_options)
         finish_requests(servers)
       end
+      raise @value_error if @value_error
     rescue Dalli::RetryableNetworkError => e
       Dalli.logger.debug { e.inspect }
       Dalli.logger.debug { 'retrying pipelined sets because of network error' }
@@ -62,6 +67,10 @@ module Dalli
           server.request(:pipelined_set, key, value, ttl, req_options)
         rescue Dalli::NetworkError
           raise
+        rescue Dalli::MarshalError, Dalli::ValueOverMaxSize => e
+          # Nothing was written for this key, so the server's other keys are
+          # unaffected; raised by #process once the batch is complete
+          @value_error ||= e
         rescue DalliError => e
           Dalli.logger.debug { e.inspect }
           Dalli.logger.debug { "unable to set key #{key} for server #{server.name}" }