dalli 5.1.4 → 5.1.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 70f17667b9c623f9233be342d920de385bab797b44a4bc9f1e85c233f766404f
-  data.tar.gz: 527b9ff3dd4699ad5d307107668b31d3a739675825ad4d0b2cfb35ac09581a81
+  metadata.gz: a14acb855194ffb54b0395ba7e9a4785a69e3d00068088e42aca53d91dcc3929
+  data.tar.gz: 3017450cd4da74f426885ae2684f562b42983ca83f85b13242611b7d32f76c02
 SHA512:
-  metadata.gz: 9e54f454a1bca3b79f229a219b458533d2778c1521043c1ee7c2d203ae0d25e1449fc375b59754aaadac79de6ea484170ba52c10008da6abd40451821300a60c
-  data.tar.gz: 1dea61992e48d8786dbb3ef929e30c53d3544bafed895334cf4316ad0710791c4ebb0887fb4fa63b79358c0029be11b3495a2875a24956a296246fe9d5e4eaa0
+  metadata.gz: 0b6f8bd702bb10c9f4d47ad981fbc702ac00dda484a84f82449fb0874aae98127b96c8518aad7768c7996f8c95c2772394771e84128d0c4d65278257a28540c4
+  data.tar.gz: 766ebee713280bc7ec98ec53becf3ece1a73c2734171f27379e13e2343f2dfce16649a77104254084cf9b948679646a8b6f4cbb0cb55529de354e1686d0e8b31

data/CHANGELOG.md

@@ -4,6 +4,42 @@ Dalli Changelog
 Unreleased
 ==========
 
+5.1.6
+==========
+
+Security:
+
+- Keep a connection shared by several threads or fibers in step during a `quiet` block (GHSA-5xvx-r8rg-5qm2)
+  - A `quiet` (or `multi`) block is quiet only for the thread or fiber that runs it. memcached still replies to a quiet request that fails (for example an `add` of a key that exists), and another thread or fiber using the same client while the block was open read that reply as its own, so its later requests could return another key's value. A non-quiet request now reads any pending quiet replies first
+- Don't let a `get_multi` that stopped early leave replies for a later request (GHSA-5xvx-r8rg-5qm2)
+  - A pipelined `get_multi` stopped by a non-network exception (its block raising or breaking, `Timeout.timeout`, or a value that can't be deserialized) left its unread replies in the response buffer, and a later `get_multi` large enough to use the interleaved path returned them as its own. The buffer is now cleared when a connection closes or a pipelined get starts, and unfinished pipelines are aborted
+  - Affects every supported line (the `get_multi` issue, 4.3 and later); fixed in 5.2.4, 5.1.6, 5.0.12, 4.3.9 and 3.2.15
+
+Bug fixes:
+
+- A per-request `raw: true` write no longer compresses the value (backport of #1213)
+  - Values at or above the compression threshold (4 KB by default) were compressed and flagged even with `raw: true`. Raw reads ignore flags, so reading them back with `raw: true` returned the compressed bytes
+  - Raw values are now stored exactly as given. Callers that pass `raw: true` without `compress: false` store large values uncompressed; Rails' `MemCacheStore` passes `compress: false` and isn't affected
+- Handle failed TLS handshakes like other connection failures, and warn when TLS verification is off (backport of #1215)
+  - A failed or timed-out handshake left the TCP socket open until GC and raised `OpenSSL::SSL::SSLError` directly, so the server was never marked down. The socket is now closed, and the error goes through the usual retry and `socket_max_failures` handling, raising `Dalli::NetworkError` (or `Dalli::RingError` once no server is available)
+  - Certificate and hostname checks come only from the `ssl_context` you pass, and a bare `OpenSSL::SSL::SSLContext.new` checks neither. Dalli now logs a warning once per process when `verify_mode` doesn't include `VERIFY_PEER` or `verify_hostname` is false; see the new TLS section in the README
+- A reply line longer than 8 KiB without a CRLF, or one cut short by the connection closing, is now a network error that closes the connection. Before, the line had no length limit (backport of #1216)
+- One value that can't be stored no longer discards other buffered writes (backport of #1218)
+  - A value that raised `Dalli::MarshalError` or `Dalli::ValueOverMaxSize` closed the connection, which dropped writes still buffered from the same `quiet` block or `set_multi`. Multi-server `set_multi` did this without raising
+  - Both `set_multi` paths now store every other key, skip the bad one, and then raise. Single-server `set_multi` used to store nothing
+  - When single-server `set_multi` failed to marshal a value, the error log included every value in the batch. It now lists only the keys
+
+5.1.5
+==========
+
+Security:
+
+- Don't rely on memcached's auto-create for keys that need base64 encoding (GHSA-r47x-6p6r-mvcp)
+  - memcached stores an item made by auto-create (`ma` with `N`/`J`, or `mg` with `N`) for a base64-encoded key without its binary-key flag, and later replies name the item by its raw key bytes. A key containing spaces or CR/LF could then make a multi-get read the replies after it as different ones
+  - `incr` and `decr` with an initial value now create such counters with an add instead. Return values are unchanged
+  - The new `verify_reply_keys` option makes `get_multi`, `get_multi_cas` and `get_multi_with_metadata` ignore replies for keys they didn't request, which guards against items created by older Dalli versions or other clients until memcached is upgraded. It's off by default, since it costs a multi-get about 15%
+  - Affects the meta protocol since 3.2.0; fixed in 5.2.3, 5.1.5, 5.0.11, 4.3.8 and 3.2.14. memcached 1.6.45 and earlier still name these items by their raw key bytes
+
 5.1.4
 ==========
 

data/README.md

@@ -63,6 +63,29 @@ Per-request `raw: true` returns the stored bytes without deserializing or decomp
 
 See the [5.0-Upgrade.md](5.0-Upgrade.md) guide for upgrade information.
 
+### Counters and verify_reply_keys
+
+memcached names an item made by auto-create for a base64-encoded key (one with whitespace, control or non-ASCII characters) by its raw key bytes in later replies. Since 5.1.5, Dalli creates counters for such keys with an add rather than auto-create. Items created before then, or by other clients, can still be named that way until memcached releases a fix (1.6.45 and earlier behave this way). If keys passed to `incr` or `decr` with an initial value can contain untrusted input, set `verify_reply_keys: true` until then: multi-gets then ignore replies for keys they didn't request, at a cost of about 15% per multi-get.
+
+## TLS
+
+Pass an `OpenSSL::SSL::SSLContext` as `ssl_context:` to connect over TLS. Certificate and hostname verification come only from that context: Dalli adds no checks of its own. A bare `OpenSSL::SSL::SSLContext.new` verifies nothing, so it accepts any certificate from any server. Configure the context to verify both:
+
+```ruby
+ssl_context = OpenSSL::SSL::SSLContext.new
+ssl_context.set_params(
+  verify_mode: OpenSSL::SSL::VERIFY_PEER,
+  verify_hostname: true,
+  ca_file: '/path/to/ca.crt' # omit to use the system's trusted certificates
+)
+
+Dalli::Client.new('memcached.example.com:11211', ssl_context: ssl_context)
+```
+
+Dalli logs a warning once per process when the context's `verify_mode` is `VERIFY_NONE`, or when it is `VERIFY_PEER` but `verify_hostname` is false.
+
+A failed TLS handshake counts as a failed connection attempt, like a refused connection: once `socket_max_failures` attempts have failed, the server is marked down and requests fail over to the other servers.
+
 ## OpenTelemetry Tracing
 
 Dalli automatically instruments operations with [OpenTelemetry](https://opentelemetry.io/) when the SDK is present. No configuration is required - just add the OpenTelemetry gems to your application:

data/lib/dalli/client.rb

@@ -41,15 +41,21 @@ module Dalli
     #                 #fetch operations.
     # - :raw        - If set, disables serialization and compression entirely at the client level.
     #                 Only String values are supported. This is useful when the caller handles its own
-    #                 serialization (e.g., Rails' ActiveSupport::Cache). Note: this is different from
-    #                 the per-request :raw option which converts values to strings but still uses the
-    #                 serialization pipeline.
+    #                 serialization (e.g., Rails' ActiveSupport::Cache). The per-request :raw option
+    #                 (e.g. set(key, value, ttl, raw: true)) does the same for a single call: String
+    #                 values are stored as-is, uncompressed, and non-String values raise
+    #                 Dalli::MarshalError.
     # - :digest_class - defaults to Digest::MD5, allows you to pass in an object that responds to the hexdigest method,
     #                   useful for injecting a FIPS compliant hash object.
     # - :otel_db_statement - controls the +db.query.text+ span attribute when OpenTelemetry is loaded.
     #                        +:include+ logs the full operation and key(s), +:obfuscate+ replaces keys with "?",
     #                        +nil+ (default) omits the attribute entirely.
     # - :otel_peer_service - when set, adds a +peer.service+ span attribute with this value for logical service naming.
+    # - :verify_reply_keys - defaults to false. When true, get_multi, get_multi_cas and get_multi_with_metadata only
+    #                        accept replies for the keys they requested; after a reply naming another key, the rest of
+    #                        that server's replies are treated as misses. memcached can name an item by its raw key
+    #                        bytes when the item was created by auto-create from a base64-encoded key, which Dalli no
+    #                        longer does. Costs a multi-get about 15%.
     #
     def initialize(servers = nil, options = {})
       @normalized_servers = ::Dalli::ServersArgNormalizer.normalize_servers(servers)
@@ -365,6 +371,11 @@ module Dalli
     # Any error replies will be discarded at the end of the block, and
     # Dalli client methods invoked inside the block will not
     # have return values
+    #
+    # The block is quiet only for the thread or fiber that runs it. Another
+    # thread or fiber sharing this client can make requests meanwhile; a
+    # non-quiet one first sends the block's buffered requests and discards
+    # their replies, so it reads only its own.
     def quiet
       old = Thread.current[::Dalli::QUIET]
       Thread.current[::Dalli::QUIET] = true
@@ -745,6 +756,11 @@ module Dalli
       Dalli.logger.debug { e.inspect }
       Dalli.logger.debug { 'retrying single-server get_multi because of network error' }
       retry
+    rescue Dalli::UnexpectedReplyKeyError => e
+      # The connection is closed and the keys are treated as missing, as the
+      # pipelined get_multi does for a server whose replies are out of step
+      Dalli.logger.warn { e.message }
+      {}
     end
 
     def single_server_get_multi_with_metadata(keys, req_options = nil)
@@ -756,6 +772,9 @@ module Dalli
       result
     rescue Dalli::NetworkError
       {}
+    rescue Dalli::UnexpectedReplyKeyError => e
+      Dalli.logger.warn { e.message }
+      {}
     end
 
     def single_server_set_multi(hash, ttl, req_options)

data/lib/dalli/options.rb

@@ -43,7 +43,7 @@ module Dalli
       end
     end
 
-    def pipeline_abort
+    def pipeline_abort(...)
       @lock.synchronize do
         super
       end

data/lib/dalli/pipelined_getter.rb

@@ -32,15 +32,21 @@ module Dalli
       return {} if keys.empty?
 
       @ring.lock do
-        # Stores partial results collected during interleaved send phase
-        @partial_results = {}
-        servers = setup_requests(keys, req_options, return_cas: return_cas)
-        start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-
-        # First yield any partial results collected during interleaved send
-        yield_partial_results(&block)
-
-        servers = fetch_responses(servers, start_time, @ring.socket_timeout, &block) until servers.empty?
+        # The servers with replies still to read, updated in place
+        servers = []
+        begin
+          # Stores partial results collected during interleaved send phase
+          @partial_results = {}
+          setup_requests(servers, keys, req_options, return_cas: return_cas)
+          start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
+          # First yield any partial results collected during interleaved send
+          yield_partial_results(&block)
+
+          fetch_responses(servers, start_time, @ring.socket_timeout, &block) until servers.empty?
+        ensure
+          abort_unfinished(servers)
+        end
       end
     rescue Dalli::RetryableNetworkError => e
       Dalli.logger.debug { e.inspect }
@@ -93,9 +99,9 @@ module Dalli
 
     # Sends each server's queries and its terminating noop before building
     # the next server's, so memcached is already answering the first servers
-    # while the rest are prepared. Returns the servers with a pending response.
-    def setup_requests(keys, req_options = nil, return_cas: true)
-      started = []
+    # while the rest are prepared. Adds the servers with a pending response
+    # to started.
+    def setup_requests(started, keys, req_options = nil, return_cas: true)
       groups_for_keys(keys).each do |server, keys_for_server|
         make_getkq_request(server, keys_for_server, req_options, return_cas: return_cas)
         next unless server.connected?
@@ -108,7 +114,6 @@ module Dalli
       rescue Dalli::DalliError
         started.delete(server)
       end
-      started
     end
 
     ##
@@ -152,29 +157,39 @@ module Dalli
       raise
     end
 
-    # Swallows Dalli::NetworkError
+    # Swallows Dalli::NetworkError. Empties servers, so they aren't aborted
+    # again.
     def abort_without_timeout(servers)
       servers.each(&:pipeline_abort)
+      servers.clear
+    end
+
+    # When #process stops before reading every server's replies (the caller's
+    # block raises or breaks, a Timeout.timeout fires, a value can't be
+    # deserialized), the rest are still on the socket and some may be in the
+    # response buffer. Closing those connections keeps them from being read
+    # as a later request's replies. Aborts that #process already handled
+    # emptied servers.
+    def abort_unfinished(servers)
+      servers.each { |server| server.pipeline_abort(timed_out: false) }
+      servers.clear
     end
 
+    # Reads the replies available, removing finished servers from servers
     def fetch_responses(servers, start_time, timeout, &block)
       # Remove any servers which are not connected
       servers.select!(&:connected?)
-      return [] if servers.empty?
+      return if servers.empty?
 
       time_left = remaining_time(start_time, timeout)
       readable_servers = servers_with_response(servers, time_left)
-      if readable_servers.empty?
-        abort_with_timeout(servers)
-        return []
-      end
+      return abort_with_timeout(servers) if readable_servers.empty?
 
       # Loop through the servers with responses, and
       # delete any from our list that are finished
       readable_servers.each do |server|
         servers.delete(server) if process_server(server, &block)
       end
-      servers
     rescue NetworkError
       # Abort and raise if we encountered a network error.  This triggers
       # a retry at the top level on RetryableNetworkError.
@@ -191,12 +206,10 @@ module Dalli
 
     # Swallows Dalli::NetworkError
     def abort_with_timeout(servers)
-      abort_without_timeout(servers)
       servers.each do |server|
         Dalli.logger.debug { "memcached at #{server.name} did not response within timeout" }
       end
-
-      true # Required to simplify caller
+      abort_without_timeout(servers)
     end
 
     # Processes responses from a server.  Returns true if there are no

data/lib/dalli/pipelined_setter.rb

@@ -18,6 +18,9 @@ module Dalli
     # A transient network error is retried automatically. If a server remains
     # unreachable after retrying, raises Dalli::NetworkError.
     #
+    # A value that can't be marshalled or is over value_max_bytes is skipped;
+    # the other keys are still stored, then the first such error is raised.
+    #
     # @param hash [Hash] key-value pairs to set
     # @param ttl [Integer] time-to-live in seconds
     # @param req_options [Hash] options passed to each set operation
@@ -26,10 +29,12 @@ module Dalli
     def process(hash, ttl, req_options)
       return if hash.empty?
 
+      @value_error = nil
       @ring.lock do
         servers = setup_requests(hash, ttl, req_options)
         finish_requests(servers)
       end
+      raise @value_error if @value_error
     rescue Dalli::RetryableNetworkError => e
       Dalli.logger.debug { e.inspect }
       Dalli.logger.debug { 'retrying pipelined sets because of network error' }
@@ -62,6 +67,10 @@ module Dalli
           server.request(:pipelined_set, key, value, ttl, req_options)
         rescue Dalli::NetworkError
           raise
+        rescue Dalli::MarshalError, Dalli::ValueOverMaxSize => e
+          # Nothing was written for this key, so the server's other keys are
+          # unaffected; raised by #process once the batch is complete
+          @value_error ||= e
         rescue DalliError => e
           Dalli.logger.debug { e.inspect }
           Dalli.logger.debug { "unable to set key #{key} for server #{server.name}" }

data/lib/dalli/protocol/base.rb

@@ -41,6 +41,7 @@ module Dalli
         warn_uri_credentials(user_creds)
         @options = client_options.merge(user_creds)
         @raw_mode = client_options[:raw]
+        @verify_reply_keys = client_options[:verify_reply_keys] ? true : false
         @value_marshaller = @raw_mode ? StringMarshaller.new(@options) : ValueMarshaller.new(@options)
         @connection_manager = ConnectionManager.new(hostname, port, socket_type, @options)
       end
@@ -66,15 +67,25 @@ module Dalli
         begin
           request_completed = false
           @connection_manager.start_request!
+          drain_pending_quiet_responses(opkey)
           response = send(opkey, *args)
+          # Quiet requests can leave replies on the socket; remember that so a
+          # later non-quiet request knows to read them first
+          @connection_manager.mark_quiet_responses_pending! if quiet? && opkey != :noop
 
           # pipelined_get/pipelined_get_interleaved emit query but don't read the response(s)
           @connection_manager.finish_request! unless %i[pipelined_get pipelined_get_interleaved].include?(opkey)
 
           request_completed = true
           response
-        rescue Dalli::MarshalError => e
-          log_marshal_err(args.first, e)
+        rescue Dalli::MarshalError, Dalli::ValueOverMaxSize => e
+          log_marshal_err(args.first, e) if e.is_a?(Dalli::MarshalError)
+          # Raised only while the connection is clean: a value is marshalled
+          # before any of its request is written, and write_multi_req raises
+          # only after reading every reply. Closing the connection would
+          # discard earlier quiet writes still waiting in the write buffer.
+          @connection_manager.abort_request!
+          request_completed = true
           raise
         rescue Dalli::DalliError
           raise
@@ -134,7 +145,7 @@ module Dalli
         verify_pipelined_state(:getkq)
         write_noop
         # Use ensure_ready instead of reset to preserve any data already buffered
-        # during interleaved pipelined get draining
+        # while this get_multi's interleaved requests were sent
         response_buffer.ensure_ready
       end
 
@@ -147,6 +158,8 @@ module Dalli
       # avoiding intermediate Hash allocation. Returns nil.
       # Without a block, returns a Hash of { key => [value, cas] }.
       def pipeline_next_responses(&block)
+        return abandon_pipeline if @pipeline_out_of_step
+
         reconnect_on_pipeline_complete!
         values = nil
 
@@ -168,6 +181,13 @@ module Dalli
           # to the noop at the end of the pipeline
           finish_pipeline && break if status && key.nil?
 
+          # A key that wasn't requested means the replies are out of step
+          # (see expect_reply_keys): keep what was read so far, drop the rest
+          if unexpected_reply_key?(key)
+            abandon_pipeline
+            break
+          end
+
           # If the status is ok and the key is not nil, then this is a
           # getkq response with a value that we want to set in the response hash
           unless key.nil?
@@ -179,6 +199,10 @@ module Dalli
             end
           end
 
+          # A request to this server from inside the block closes the
+          # connection, and with it the rest of this get_multi's replies
+          raise_pipeline_closed unless response_buffer.in_progress?
+
           # Get the next response from the buffer
           status, cas, key, value = response_buffer.process_single_getk_response
         end
@@ -188,21 +212,24 @@ module Dalli
       # rubocop:enable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
 
       # Abort current pipelined get. Generally used to signal an external
-      # timeout during pipelined get.  The underlying socket is
+      # timeout during pipelined get, or that the get_multi stopped before
+      # reading all its replies (timed_out: false).  The underlying socket is
       # disconnected, and the exception is swallowed.
       #
       # Returns nothing.
-      def pipeline_abort
+      def pipeline_abort(timed_out: true)
         response_buffer.clear
+        @pipeline_out_of_step = false
+        @expected_reply_keys = nil
         @connection_manager.abort_request!
         return true unless connected?
 
         # Closes the connection, which ensures that our connection is in a
         # clean state for future requests. The get_multi ran out of time
-        # overall rather than hitting a socket error, so it doesn't count
-        # toward socket_max_failures: two slow get_multis in a row would
-        # otherwise mark a healthy server down.
-        Dalli.logger.warn { "#{name} didn't finish a get_multi within socket_timeout; closing the connection" }
+        # overall, or stopped early, rather than hitting a socket error, so it
+        # doesn't count toward socket_max_failures: two slow get_multis in a
+        # row would otherwise mark a healthy server down.
+        log_pipeline_abort(timed_out)
         @connection_manager.close
         true
       end
@@ -234,6 +261,18 @@ module Dalli
       ALLOWED_QUIET_OPS = %i[add replace set delete incr decr append prepend flush noop].freeze
       private_constant :ALLOWED_QUIET_OPS
 
+      # Reads the replies still pending from quiet requests before any
+      # non-quiet request, which reads a reply of its own. A quiet block reads
+      # them at its end, but the connection is shared: another thread or fiber
+      # (quiet? is fiber-local) can make a request while the block is still
+      # open, and would otherwise read the block's replies as its own. The
+      # noop also sends any quiet requests still buffered.
+      def drain_pending_quiet_responses(opkey)
+        return unless @connection_manager.quiet_responses_pending? && opkey != :noop && !quiet?
+
+        noop
+      end
+
       def verify_allowed_quiet!(opkey)
         return if ALLOWED_QUIET_OPS.include?(opkey)
 
@@ -332,6 +371,7 @@ module Dalli
         # the expected state before pipeline_response_setup is called.
         response_buffer.clear
         response_buffer.raw = raw_request?(options)
+        expect_reply_keys(keys)
 
         # The terminating noop is sent by pipeline_response_setup
         write(quiet_get_requests(keys, options, return_cas: return_cas))
@@ -341,9 +381,11 @@ module Dalli
       # This prevents socket buffer deadlock when sending many keys.
       # Populates the provided results hash with any responses drained during send.
       def pipelined_get_interleaved(keys, chunk_size, results, options = nil)
-        # Initialize the response buffer for draining during send phase
-        response_buffer.ensure_ready
+        # Start with an empty buffer for draining during the send phase. Bytes
+        # left by an earlier get_multi that stopped early belong to its replies.
+        response_buffer.reset
         response_buffer.raw = raw_request?(options)
+        expect_reply_keys(keys)
 
         keys.each_slice(chunk_size) do |chunk|
           # Build and write this chunk of requests
@@ -376,6 +418,12 @@ module Dalli
           status, cas, key, value = response_buffer.process_single_getk_response
           break if status.nil? # No complete response available
 
+          # Handled by the next pipeline_next_responses, once sending is done
+          if unexpected_reply_key?(key)
+            @pipeline_out_of_step = true
+            break
+          end
+
           results[key] = [value, cas] unless key.nil?
         end
       rescue SystemCallError, Dalli::NetworkError
@@ -383,8 +431,48 @@ module Dalli
         nil
       end
 
+      # Registered with the connection manager, which clears it whenever the
+      # connection closes, so a new connection never parses an old one's bytes
       def response_buffer
-        @response_buffer ||= ResponseBuffer.new(@connection_manager, response_processor)
+        @response_buffer ||= ResponseBuffer.new(@connection_manager, response_processor).tap do |buffer|
+          @connection_manager.response_buffer = buffer
+        end
+      end
+
+      # A multi-get only accepts replies for the keys it asked for. memcached
+      # names an item created by auto-create (ma or mg with N) for a
+      # base64-encoded key by its raw bytes, without the b flag; such a key
+      # can contain spaces or CR/LF, which split the reply header and make the
+      # replies after it parse as different ones. A key that wasn't requested
+      # is how that shows up.
+      # Only with the verify_reply_keys option, since building the set costs a
+      # multi-get about 15%. Dalli itself no longer creates such items (see
+      # Meta#decr_incr); the check guards against ones that older versions or
+      # other clients created, on memcached versions with the behavior.
+      def expect_reply_keys(keys)
+        @pipeline_out_of_step = false
+        @expected_reply_keys = (keys.to_set { |k| reply_key_form(k) } if @verify_reply_keys)
+      end
+
+      def unexpected_reply_key?(key)
+        !key.nil? && @expected_reply_keys && !@expected_reply_keys.include?(reply_key_form(key))
+      end
+
+      # Keys compared as bytes; ASCII keys compare equal in any encoding as is
+      def reply_key_form(key)
+        key.ascii_only? ? key : key.b
+      end
+
+      # Keeps what was read, drops the rest of this server's replies, and
+      # closes the connection so nothing unread is left on it. Not a socket
+      # failure, so it doesn't count toward socket_max_failures; the keys not
+      # read are simply missing from the result.
+      def abandon_pipeline
+        Dalli.logger.warn { "#{name} replied to a get_multi with a key that wasn't requested; closing the connection" }
+        @pipeline_out_of_step = false
+        response_buffer.clear
+        @connection_manager.close
+        {}
       end
 
       # Called after the noop response is received at the end of a set
@@ -400,7 +488,23 @@ module Dalli
         @connection_manager.reconnect! 'pipelined get has completed' if pipeline_complete?
       end
 
+      def raise_pipeline_closed
+        raise Dalli::DalliError, "The connection to #{name} closed while a get_multi was reading its replies " \
+                                 '(a request made from inside the get_multi block?)'
+      end
+
+      def log_pipeline_abort(timed_out)
+        if timed_out
+          Dalli.logger.warn { "#{name} didn't finish a get_multi within socket_timeout; closing the connection" }
+        else
+          Dalli.logger.debug { "#{name} get_multi stopped before reading all replies; closing the connection" }
+        end
+      end
+
+      # The single-server set_multi path passes its whole pairs hash as the
+      # first argument; log only its keys, never the values
       def log_marshal_err(key, err)
+        key = key.keys.join("', '") if key.is_a?(Hash)
         Dalli.logger.error "Marshalling error for key '#{key}': #{err.message}"
         Dalli.logger.error 'You are trying to cache a Ruby object which cannot be serialized to memcached.'
       end

data/lib/dalli/protocol/connection_manager.rb

@@ -26,6 +26,9 @@ module Dalli
 
       attr_accessor :hostname, :port, :socket_type, :options
       attr_reader :sock
+      # The buffer replies are read into (see Base#response_buffer). It's
+      # cleared with the write buffer whenever the connection closes.
+      attr_writer :response_buffer
 
       def initialize(hostname, port, socket_type, client_options)
         @hostname = hostname
@@ -33,10 +36,12 @@ module Dalli
         @socket_type = socket_type
         @options = DEFAULTS.merge(client_options)
         @request_in_progress = false
+        @quiet_responses_pending = false
         @sock = nil
         @pid = nil
         @write_buffer = []
         @write_buffer_bytes = 0
+        @response_buffer = nil
 
         @fail_count = 0
         reset_down_info
@@ -58,8 +63,9 @@ module Dalli
         @sock.sync = true
         @pid = Process.pid
         @request_in_progress = false
-      rescue SystemCallError, *TIMEOUT_ERRORS, EOFError, SocketError => e
-        # SocketError = DNS resolution failure
+        @quiet_responses_pending = false
+      rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError, SocketError => e
+        # SocketError = DNS resolution failure; SSL_ERRORS = failed TLS handshake
         error_on_request!(e)
       end
 
@@ -129,7 +135,9 @@ module Dalli
           # @request_in_progress == true.
           @sock = nil
           @pid = nil
+          @quiet_responses_pending = false
           discard_write_buffer
+          @response_buffer&.clear
           abort_request!
         end
       end
@@ -145,6 +153,21 @@ module Dalli
         !@sock.nil?
       end
 
+      # True when a quiet request was written on this connection and its
+      # replies (memcached only suppresses some of them) haven't been
+      # consumed by a noop yet.
+      def quiet_responses_pending?
+        @quiet_responses_pending
+      end
+
+      def mark_quiet_responses_pending!
+        @quiet_responses_pending = true
+      end
+
+      def clear_quiet_responses_pending!
+        @quiet_responses_pending = false
+      end
+
       def request_in_progress?
         @request_in_progress
       end
@@ -170,10 +193,22 @@ module Dalli
         @request_in_progress = false
       end
 
+      # The longest reply line Dalli reads, in bytes, CRLF included. A meta
+      # reply header is at most about 500 bytes: "VA <size>" plus its flags,
+      # the largest being a 250-byte key base64-encoded to 336 bytes, a
+      # 20-digit CAS, a 32-byte opaque and a few numeric fields. The longest
+      # stats line (stats settings) carries a file path such as ext_path, up
+      # to PATH_MAX (4096). 8 KiB leaves room for both, while a server that
+      # never sends CRLF can't grow the read buffer without bound.
+      MAX_LINE_BYTES = 8 * 1024
+
       def read_line
         flush_write_buffer
-        data = @sock.gets("\r\n")
+        data = @sock.gets("\r\n", MAX_LINE_BYTES)
         error_on_request!('EOF in read_line') if data.nil?
+        # Without its CRLF the line either ran past MAX_LINE_BYTES or was cut
+        # short by EOF; either way the connection is out of step
+        error_on_request!("Reply line without CRLF in the first #{MAX_LINE_BYTES} bytes") unless data.end_with?("\r\n")
         data
       rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError => e
         error_on_request!(e)

data/lib/dalli/protocol/meta.rb

@@ -224,12 +224,45 @@ module Dalli
       # rubocop:disable-next Metrics/ParameterLists
       def decr_incr(incr, key, delta, ttl, initial, options = nil)
         ttl = initial ? TtlSanitizer.sanitize(ttl) : nil # Only set a TTL if we want to set a value on miss
+        # memcached stores an item created by ma's auto-create (N/J) for a
+        # base64-encoded key without its binary-key flag, and later replies
+        # name it with the raw key bytes. Create such counters with an add.
+        return decr_incr_with_add(incr, key, delta, ttl, initial, options) if initial && KeyRegularizer.required?(key)
+
         write(RequestFormatter.meta_arithmetic(key: key, delta: delta, initial: initial, incr: incr, ttl: ttl,
                                                quiet: quiet?, **routing_token_kwargs(options)))
         @connection_manager.flush unless quiet?
         response_processor.decr_incr unless quiet?
       end
 
+      # Like decr_incr with an initial value, without memcached's auto-create:
+      # on a miss the counter is added with the initial value (which is
+      # returned, as auto-create would), and the add losing a race to another
+      # client means the counter now exists, so the incr is tried again. In a
+      # quiet block both requests are sent and whichever applies takes effect.
+      # rubocop:disable-next Metrics/ParameterLists
+      def decr_incr_with_add(incr, key, delta, ttl, initial, options)
+        add_options = (options || {}).merge(raw: true) # counters are stored raw
+        return quiet_decr_incr_with_add(incr, key, delta, ttl, initial, add_options) if quiet?
+
+        2.times do
+          value = decr_incr(incr, key, delta, nil, nil, options)
+          return value unless value.nil?
+
+          write_storage_req(:add, key, initial.to_s, ttl, nil, add_options, quiet: false)
+          return initial if response_processor.meta_set_with_cas
+        end
+        decr_incr(incr, key, delta, nil, nil, options)
+      end
+
+      # rubocop:disable-next Metrics/ParameterLists
+      def quiet_decr_incr_with_add(incr, key, delta, ttl, initial, add_options)
+        write(RequestFormatter.meta_arithmetic(key: key, delta: delta, initial: nil, incr: incr, ttl: nil,
+                                               quiet: true, **routing_token_kwargs(add_options)))
+        # Sent with the rest of the quiet block, by its closing noop
+        write_storage_req(:add, key, initial.to_s, ttl, nil, add_options, quiet: true)
+      end
+
       # Other Commands
       def flush(delay = 0)
         write(RequestFormatter.flush(delay: delay))
@@ -241,7 +274,10 @@ module Dalli
       # We need to read all the responses at once.
       def noop
         write_noop
-        response_processor.consume_all_responses_until_mn
+        result = response_processor.consume_all_responses_until_mn
+        # Everything up to and including the MN has been read
+        @connection_manager.clear_quiet_responses_pending!
+        result
       end
 
       def stats(info = nil)
@@ -271,7 +307,7 @@ module Dalli
         buffer = RequestFormatter.multi_meta_get(keys, skip_flags: is_raw, **routing_token_kwargs(options))
         flushed_write(buffer)
         buffer.clear
-        read_multi_get_responses(is_raw)
+        read_multi_get_responses(is_raw, keys)
       end
 
       # Stale-aware bulk get.  Returns { key => { value:, cas:, stale:, miss: } }
@@ -288,12 +324,13 @@ module Dalli
                                                        **routing_token_kwargs(options))
         flushed_write(buffer)
         buffer.clear
-        read_multi_metadata_responses(is_raw)
+        read_multi_metadata_responses(is_raw, keys)
       end
 
       # Unlike read_multi_get_responses this locates tokens by flag rather than
       # by position, because the c flag shifts the key's index.
-      def read_multi_metadata_responses(is_raw)
+      def read_multi_metadata_responses(is_raw, keys)
+        expect_reply_keys(keys)
         hash = {}
         while (line = @connection_manager.read_line)
           break if line.start_with?('MN')
@@ -309,12 +346,15 @@ module Dalli
           key = response_processor.key_from_tokens(tokens)
           next if key.nil?
 
+          raise_unexpected_reply_key(key) if unexpected_reply_key?(key)
+
           hash[key] = { value: @value_marshaller.retrieve(value, bitflags), cas: cas, stale: stale, miss: false }
         end
         hash
       end
 
-      def read_multi_get_responses(is_raw)
+      def read_multi_get_responses(is_raw, keys)
+        expect_reply_keys(keys)
         hash = {}
         key_index = is_raw ? 2 : 3
         while (line = @connection_manager.read_line)
@@ -322,11 +362,19 @@ module Dalli
           next unless line.start_with?('VA ')
 
           key, value = parse_multi_get_value(line, key_index, is_raw)
+          raise_unexpected_reply_key(key) if unexpected_reply_key?(key)
           hash[key] = value if key
         end
         hash
       end
 
+      # The replies after a key that wasn't requested can't be trusted (see
+      # Base#expect_reply_keys). Raising ends the request, which closes the
+      # connection; the caller treats this server's keys as missing.
+      def raise_unexpected_reply_key(key)
+        raise Dalli::UnexpectedReplyKeyError, "#{name} replied with a key that wasn't requested: #{key.inspect}"
+      end
+
       def parse_multi_get_value(line, key_index, is_raw)
         tokens = line.chomp!(TERMINATOR).split
         size = tokens[1].to_i
@@ -343,16 +391,27 @@ module Dalli
 
       # Single-server fast path for set_multi. Inlines request formatting to
       # minimize per-key overhead. Avoids PipelinedSetter server grouping.
+      #
+      # A value that can't be marshalled or is over value_max_bytes is skipped;
+      # the other keys are still stored, then the first such error is raised
+      # (as PipelinedSetter does on the multi-server path).
       def write_multi_req(pairs, ttl, req_options)
         ttl = TtlSanitizer.sanitize(ttl) if ttl
-        entries = pairs.map do |key, raw_value|
+        value_error = nil
+        entries = pairs.filter_map do |key, raw_value|
           [key, @value_marshaller.store(key, raw_value, req_options)]
+        rescue Dalli::MarshalError, Dalli::ValueOverMaxSize => e
+          value_error ||= e
+          nil
         end
 
-        buffer = RequestFormatter.multi_meta_set(entries, ttl: ttl, **routing_token_kwargs(req_options))
-        flushed_write(buffer)
-        buffer.clear
-        response_processor.consume_all_responses_until_mn
+        unless entries.empty?
+          buffer = RequestFormatter.multi_meta_set(entries, ttl: ttl, **routing_token_kwargs(req_options))
+          flushed_write(buffer)
+          buffer.clear
+          response_processor.consume_all_responses_until_mn
+        end
+        raise value_error if value_error
       end
 
       # Single-server fast path for delete_multi. Writes all quiet delete requests

data/lib/dalli/protocol/value_compressor.rb

@@ -84,11 +84,13 @@ module Dalli
       end
 
       # Checks whether we should apply compression when serializing a value
-      # based on the specified options.  Returns false unless the value
-      # is greater than the minimum compression size.  Otherwise returns
-      # based on a method-level option if specified, falling back to the
-      # server default.
+      # based on the specified options.  Returns false for a raw request
+      # (raw reads ignore bitflags, so the value must be stored as given) or
+      # unless the value is greater than the minimum compression size.
+      # Otherwise returns based on a method-level option if specified,
+      # falling back to the server default.
       def compress_value?(value, req_options)
+        return false if req_options&.dig(:raw)
         return false unless value.bytesize >= compression_min_size
         return compress_by_default? unless req_options && !req_options[:compress].nil?
 

data/lib/dalli/protocol.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require 'openssl'
 require 'timeout'
 
 module Dalli
@@ -16,14 +17,11 @@ module Dalli
         [Timeout::Error]
       end
 
-    # SSL errors that occur during read/write operations (not during initial
-    # handshake) should trigger reconnection. These indicate transient network
-    # issues, not configuration problems.
-    SSL_ERRORS =
-      if defined?(OpenSSL::SSL::SSLError)
-        [OpenSSL::SSL::SSLError]
-      else
-        []
-      end
+    # SSL errors during the handshake or during read/write operations count as
+    # a failed request: they trigger reconnection and, after
+    # socket_max_failures, mark the server down.
+    # openssl is required above: this file loads before dalli/socket, so a
+    # defined? check here would find OpenSSL missing and leave this empty.
+    SSL_ERRORS = [OpenSSL::SSL::SSLError].freeze
   end
 end

data/lib/dalli/socket.rb

@@ -203,12 +203,41 @@ module Dalli
         [seconds, microseconds].pack(timeval_pack_format(sock))
       end
 
+      # Closes the TCP socket if the handshake fails or times out, so a failed
+      # attempt doesn't hold a file descriptor until GC.
       def self.wrapping_ssl_socket(tcp_socket, host, ssl_context)
+        warn_if_unverified(ssl_context)
+        connected = false
         ssl_socket = Dalli::Socket::SSLSocket.new(tcp_socket, ssl_context)
         ssl_socket.hostname = host
         ssl_socket.sync_close = true
         ssl_socket.connect
+        connected = true
         ssl_socket
+      ensure
+        tcp_socket.close unless connected
+      end
+
+      # Class variable to track whether the TLS verification warning has been logged
+      @@ssl_verification_warning_logged = false # rubocop:disable Style/ClassVars
+
+      # Verification comes entirely from the caller's ssl_context, and a bare
+      # SSLContext.new verifies nothing, so warn (once per process) when the
+      # certificate or the hostname won't be checked.
+      def self.warn_if_unverified(ssl_context)
+        return if @@ssl_verification_warning_logged
+
+        problem = if !ssl_context.verify_mode.to_i.anybits?(::OpenSSL::SSL::VERIFY_PEER)
+                    'verify_mode is VERIFY_NONE, so the server certificate is not checked'
+                  elsif !ssl_context.verify_hostname
+                    'verify_hostname is false, so the certificate is not checked against the hostname'
+                  end
+        return unless problem
+
+        Dalli.logger.warn "SECURITY WARNING: Dalli's ssl_context #{problem}. " \
+                          'Use ssl_context.set_params(verify_mode: OpenSSL::SSL::VERIFY_PEER, ' \
+                          'verify_hostname: true) to verify the server.'
+        @@ssl_verification_warning_logged = true # rubocop:disable Style/ClassVars
       end
     end
 

data/lib/dalli/version.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module Dalli
-  VERSION = '5.1.4'
+  VERSION = '5.1.6'
 
   MIN_SUPPORTED_MEMCACHED_VERSION = '1.6.27'
 end

data/lib/dalli.rb

@@ -31,6 +31,10 @@ module Dalli
   # socket/server communication error that can be retried
   class RetryableNetworkError < NetworkError; end
 
+  # a multi-get reply named a key that wasn't requested, so the rest of that
+  # server's replies can't be trusted
+  class UnexpectedReplyKeyError < DalliError; end
+
   # Implements the NullObject pattern to store an application-defined value for 'Key not found' responses.
   class NilObject; end # rubocop:disable Lint/EmptyClass
   NOT_FOUND = NilObject.new

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: dalli
 version: !ruby/object:Gem::Version
-  version: 5.1.4
+  version: 5.1.6
 platform: ruby
 authors:
 - Peter M. Goldstein
@@ -88,7 +88,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.21
+rubygems_version: 4.0.23
 specification_version: 4
 summary: High performance memcached client for Ruby
 test_files: []