dalli 3.2.14 → 3.2.15

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5b52d92d26a7277b6f65b402c082be2f99b98ca9cb7596af709ac70848167a53
-  data.tar.gz: 2cc59e69e536f07ec79ec6e37ffcae997c1e47231aaa213e32fdce02b3b3e123
+  metadata.gz: 1282508d83c234919dfb2c8a05c08d7069186663c83d4e3aaf040e6f503a2e9f
+  data.tar.gz: cb10108ea1d22c25630b5cfa36b6e038515e3f053e2354e3921718050138535b
 SHA512:
-  metadata.gz: 8db03f1265942f604b68bdded36f70aa00e859269950367df91c62ef1b2d1011ba6166269125f4ab377c4b4acd4acfd763c3f17a867d2e357479b259c0592ffc
-  data.tar.gz: 7f15a910062c45a776e3bc3964a3555b4a3c0c0a5897877dc170ee5a80d16a313bb4eb0da64821ca14d930ffbea1445855e2f1ca7fe2a068a33765aabc064636
+  metadata.gz: 614a4eebdaa9cee1d9dc6afdfd45c96ff1ec7eda12f471a12c6c1eca7866af971d907a7c288aeb5a72734b3f4f72bfb685ba58337da2ac50146ed74a018c0d0a
+  data.tar.gz: 8f577fef557f593d9cb6327a54ec1b5d3df8e554a5935ba7a7faadf86259248a60f29204b3ad5a295024b676ebc481e925f81f330095a73e6649e11d182b1127

data/CHANGELOG.md

@@ -4,6 +4,23 @@ Dalli Changelog
 Unreleased
 ==========
 
+3.2.15
+==========
+
+- Keep a connection shared by several threads or fibers in step during a `quiet` block (GHSA-5xvx-r8rg-5qm2)
+  - A `quiet` (or `multi`) block is quiet only for the thread or fiber that runs it. memcached still replies to a quiet request that fails (for example an `add` of a key that exists), and another thread or fiber using the same client while the block was open read that reply as its own, so its later requests could return another key's value. A non-quiet request now reads any pending quiet replies first
+  - Affects every supported line, with both protocols; with the binary protocol the connection stayed out of step after the block ended. Fixed in 5.2.4, 5.1.6, 5.0.12, 4.3.9 and 3.2.15
+- With `protocol: :meta`, read values by their size instead of up to the first CRLF (GHSA-5xvx-r8rg-5qm2)
+  - `get`, `gat` and `cas` read a value as one line, so a value containing CR/LF was cut short and the rest was parsed as the next reply. A stored value containing a crafted reply was then returned for a later read of a different key, with flags that could cause it to be deserialized
+  - Affects 3.2.0 to 3.2.14 with `protocol: :meta`; fixed in 3.2.15. The binary protocol (3.2's default) and 4.0 and later aren't affected
+- A per-request `raw: true` write no longer compresses the value (backport of #1213)
+  - Values at or above the compression threshold (4 KB by default) were compressed and flagged even with `raw: true`. Raw reads ignore flags, so reading them back with `raw: true` returned the compressed bytes
+  - Raw values are now stored exactly as given. Callers that pass `raw: true` without `compress: false` store large values uncompressed; Rails' `MemCacheStore` passes `compress: false` and isn't affected
+- Handle failed TLS handshakes like other connection failures, and warn when TLS verification is off (backport of #1215)
+  - A failed or timed-out handshake left the TCP socket open until GC and raised `OpenSSL::SSL::SSLError` directly, so the server was never marked down. The socket is now closed, and the error goes through the usual retry and `socket_max_failures` handling, raising `Dalli::NetworkError` (or `Dalli::RingError` once no server is available)
+  - Certificate and hostname checks come only from the `ssl_context` you pass, and a bare `OpenSSL::SSL::SSLContext.new` checks neither. Dalli now logs a warning once per process when `verify_mode` doesn't include `VERIFY_PEER` or `verify_hostname` is false; see the new TLS section in the README
+- A reply line longer than 8 KiB without a CRLF, or one cut short by the connection closing, is now a network error that closes the connection. Before, the line had no length limit (backport of #1216)
+
 3.2.14
 ==========
 

data/README.md

@@ -41,6 +41,25 @@ Per-request `raw: true` returns the stored bytes without deserializing or decomp
 
 memcached names an item made by auto-create for a base64-encoded key (one with whitespace, control or non-ASCII characters) by its raw key bytes in later replies (with `protocol: :meta`). Since 3.2.14, Dalli creates counters for such keys with an add rather than auto-create. Items created before then, or by other clients, can still be named that way until memcached releases a fix (1.6.45 and earlier behave this way). If keys passed to `incr` or `decr` with an initial value can contain untrusted input, set `verify_reply_keys: true` until then: multi-gets then ignore replies for keys they didn't request, at a cost of about 15% per multi-get.
 
+## TLS
+
+Pass an `OpenSSL::SSL::SSLContext` as `ssl_context:` to connect over TLS. Certificate and hostname verification come only from that context: Dalli adds no checks of its own. A bare `OpenSSL::SSL::SSLContext.new` verifies nothing, so it accepts any certificate from any server. Configure the context to verify both:
+
+```ruby
+ssl_context = OpenSSL::SSL::SSLContext.new
+ssl_context.set_params(
+  verify_mode: OpenSSL::SSL::VERIFY_PEER,
+  verify_hostname: true,
+  ca_file: '/path/to/ca.crt' # omit to use the system's trusted certificates
+)
+
+Dalli::Client.new('memcached.example.com:11211', ssl_context: ssl_context)
+```
+
+Dalli logs a warning once per process when the context's `verify_mode` is `VERIFY_NONE`, or when it is `VERIFY_PEER` but `verify_hostname` is false.
+
+A failed TLS handshake counts as a failed connection attempt, like a refused connection: once `socket_max_failures` attempts have failed, the server is marked down and requests fail over to the other servers.
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. You can run `bin/console` for an interactive prompt that will allow you to experiment.

data/lib/dalli/client.rb

@@ -192,6 +192,11 @@ module Dalli
     # Any error replies will be discarded at the end of the block, and
     # Dalli client methods invoked inside the block will not
     # have return values
+    #
+    # The block is quiet only for the thread or fiber that runs it. Another
+    # thread or fiber sharing this client can make requests meanwhile; a
+    # non-quiet one first reads the block's pending replies with a noop and
+    # discards them, so it reads only its own.
     def quiet
       old = Thread.current[::Dalli::QUIET]
       Thread.current[::Dalli::QUIET] = true

data/lib/dalli/protocol/base.rb

@@ -41,7 +41,11 @@ module Dalli
 
         begin
           @connection_manager.start_request!
+          drain_pending_quiet_responses(opkey)
           response = send(opkey, *args)
+          # Quiet requests can leave replies on the socket; remember that so a
+          # later non-quiet request knows to read them first
+          @connection_manager.quiet_responses_pending = true if quiet? && opkey != :noop
 
           # pipelined_get emit query but doesn't read the response(s)
           @connection_manager.finish_request! unless opkey == :pipelined_get
@@ -165,6 +169,18 @@ module Dalli
       private
 
       ALLOWED_QUIET_OPS = %i[add replace set delete incr decr append prepend flush noop].freeze
+
+      # Reads the replies still pending from quiet requests before any
+      # non-quiet request, which reads a reply of its own. A quiet block reads
+      # them at its end, but the connection is shared: another thread or fiber
+      # (quiet? is fiber-local) can make a request while the block is still
+      # open, and would otherwise read the block's replies as its own.
+      def drain_pending_quiet_responses(opkey)
+        return unless @connection_manager.quiet_responses_pending? && opkey != :noop && !quiet?
+
+        noop
+      end
+
       def verify_allowed_quiet!(opkey)
         return if ALLOWED_QUIET_OPS.include?(opkey)
 

data/lib/dalli/protocol/binary.rb

@@ -139,7 +139,10 @@ module Dalli
       # We need to read all the responses at once.
       def noop
         write_noop
-        response_processor.consume_all_responses_until_noop
+        result = response_processor.consume_all_responses_until_noop
+        # Everything up to and including the noop reply has been read
+        @connection_manager.quiet_responses_pending = false
+        result
       end
 
       def stats(info = '')

data/lib/dalli/protocol/connection_manager.rb

@@ -28,6 +28,7 @@ module Dalli
 
       attr_accessor :hostname, :port, :socket_type, :options
       attr_reader :sock
+      attr_writer :quiet_responses_pending
 
       def initialize(hostname, port, socket_type, client_options)
         @hostname = hostname
@@ -35,6 +36,7 @@ module Dalli
         @socket_type = socket_type
         @options = DEFAULTS.merge(client_options)
         @request_in_progress = false
+        @quiet_responses_pending = false
         @sock = nil
         @pid = nil
 
@@ -56,8 +58,9 @@ module Dalli
         @sock = memcached_socket
         @pid = Process.pid
         @request_in_progress = false
-      rescue SystemCallError, *TIMEOUT_ERRORS, EOFError, SocketError => e
-        # SocketError = DNS resolution failure
+        @quiet_responses_pending = false
+      rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError, SocketError => e
+        # SocketError = DNS resolution failure; SSL_ERRORS = failed TLS handshake
         error_on_request!(e)
       end
 
@@ -122,6 +125,7 @@ module Dalli
         end
         @sock = nil
         @pid = nil
+        @quiet_responses_pending = false
         abort_request!
       end
 
@@ -136,6 +140,13 @@ module Dalli
         !@sock.nil?
       end
 
+      # True when a quiet request was written on this connection and its
+      # replies (memcached only suppresses some of them) haven't been
+      # consumed by a noop yet.
+      def quiet_responses_pending?
+        @quiet_responses_pending
+      end
+
       def request_in_progress?
         @request_in_progress
       end
@@ -161,9 +172,21 @@ module Dalli
         @request_in_progress = false
       end
 
+      # The longest reply line Dalli reads, in bytes, CRLF included. A meta
+      # reply header is at most about 500 bytes: "VA <size>" plus its flags,
+      # the largest being a 250-byte key base64-encoded to 336 bytes, a
+      # 20-digit CAS, a 32-byte opaque and a few numeric fields. The longest
+      # stats line (stats settings) carries a file path such as ext_path, up
+      # to PATH_MAX (4096). 8 KiB leaves room for both, while a server that
+      # never sends CRLF can't grow the read buffer without bound.
+      MAX_LINE_BYTES = 8 * 1024
+
       def read_line
-        data = @sock.gets("\r\n")
+        data = @sock.gets("\r\n", MAX_LINE_BYTES)
         error_on_request!('EOF in read_line') if data.nil?
+        # Without its CRLF the line either ran past MAX_LINE_BYTES or was cut
+        # short by EOF; either way the connection is out of step
+        error_on_request!("Reply line without CRLF in the first #{MAX_LINE_BYTES} bytes") unless data.end_with?("\r\n")
         data
       rescue SystemCallError, *TIMEOUT_ERRORS, EOFError => e
         error_on_request!(e)

data/lib/dalli/protocol/meta/response_processor.rb

@@ -34,7 +34,7 @@ module Dalli
           return cache_nils ? ::Dalli::NOT_FOUND : nil if tokens.first == EN
           return true unless tokens.first == VA
 
-          value = read_line
+          value = read_data(tokens[1].to_i)
           raw ? value : @value_marshaller.retrieve(value, bitflags_from_tokens(tokens))
         end
 
@@ -46,7 +46,7 @@ module Dalli
           cas = cas_from_tokens(tokens)
           return [nil, cas] unless tokens.first == VA
 
-          value = read_line
+          value = read_data(tokens[1].to_i)
           [raw ? value : @value_marshaller.retrieve(value, bitflags_from_tokens(tokens)), cas]
         end
 
@@ -242,6 +242,15 @@ module Dalli
           line = read_line
           line&.split || []
         end
+
+        # Values are read by the size in the VA line, not up to the next CRLF:
+        # a value can contain CRLF and can be longer than MAX_LINE_BYTES
+        def read_data(data_size)
+          # Checked before the terminator is added: -1 or -2 would otherwise
+          # read 1 or 0 bytes and leave the connection out of step
+          check_value_size!(data_size)
+          @io_source.read(data_size + TERMINATOR.bytesize)&.chomp!(TERMINATOR)
+        end
       end
     end
   end

data/lib/dalli/protocol/meta.rb

@@ -176,7 +176,10 @@ module Dalli
       # We need to read all the responses at once.
       def noop
         write_noop
-        response_processor.consume_all_responses_until_mn
+        result = response_processor.consume_all_responses_until_mn
+        # Everything up to and including the MN has been read
+        @connection_manager.quiet_responses_pending = false
+        result
       end
 
       def stats(info = nil)

data/lib/dalli/protocol/value_compressor.rb

@@ -97,11 +97,13 @@ module Dalli
       end
 
       # Checks whether we should apply compression when serializing a value
-      # based on the specified options.  Returns false unless the value
-      # is greater than the minimum compression size.  Otherwise returns
-      # based on a method-level option if specified, falling back to the
-      # server default.
+      # based on the specified options.  Returns false for a raw request
+      # (raw reads ignore bitflags, so the value must be stored as given) or
+      # unless the value is greater than the minimum compression size.
+      # Otherwise returns based on a method-level option if specified,
+      # falling back to the server default.
       def compress_value?(value, req_options)
+        return false if req_options&.dig(:raw)
         return false unless value.bytesize >= compression_min_size
         return compress_by_default? unless req_options && !req_options[:compress].nil?
 

data/lib/dalli/protocol.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require 'openssl'
 require 'timeout'
 
 module Dalli
@@ -15,5 +16,11 @@ module Dalli
       else
         [Timeout::Error]
       end
+
+    # SSL errors during the TLS handshake count as a failed request: they
+    # trigger reconnection and, after socket_max_failures, mark the server
+    # down. openssl is required above because this file loads before
+    # dalli/socket, so OpenSSL may not be defined yet.
+    SSL_ERRORS = [OpenSSL::SSL::SSLError].freeze
   end
 end

data/lib/dalli/socket.rb

@@ -132,12 +132,42 @@ module Dalli
         sock.setsockopt(::Socket::SOL_SOCKET, ::Socket::SO_SNDTIMEO, timeval)
       end
 
+      # Closes the TCP socket if the handshake fails or times out, so a failed
+      # attempt doesn't hold a file descriptor until GC.
       def self.wrapping_ssl_socket(tcp_socket, host, ssl_context)
+        warn_if_unverified(ssl_context)
+        connected = false
         ssl_socket = Dalli::Socket::SSLSocket.new(tcp_socket, ssl_context)
         ssl_socket.hostname = host
         ssl_socket.sync_close = true
         ssl_socket.connect
+        connected = true
         ssl_socket
+      ensure
+        tcp_socket.close unless connected
+      end
+
+      # Verification comes entirely from the caller's ssl_context, and a bare
+      # SSLContext.new verifies nothing, so warn (once per process) when the
+      # certificate or the hostname won't be checked.
+      def self.warn_if_unverified(ssl_context)
+        return if @ssl_verification_warning_logged
+
+        problem = unverified_problem(ssl_context)
+        return unless problem
+
+        Dalli.logger.warn "SECURITY WARNING: Dalli's ssl_context #{problem}. " \
+                          'Use ssl_context.set_params(verify_mode: OpenSSL::SSL::VERIFY_PEER, ' \
+                          'verify_hostname: true) to verify the server.'
+        @ssl_verification_warning_logged = true
+      end
+
+      def self.unverified_problem(ssl_context)
+        if !ssl_context.verify_mode.to_i.anybits?(::OpenSSL::SSL::VERIFY_PEER)
+          'verify_mode is VERIFY_NONE, so the server certificate is not checked'
+        elsif !ssl_context.verify_hostname
+          'verify_hostname is false, so the certificate is not checked against the hostname'
+        end
       end
     end
 

data/lib/dalli/version.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module Dalli
-  VERSION = '3.2.14'
+  VERSION = '3.2.15'
 
   MIN_SUPPORTED_MEMCACHED_VERSION = '1.4'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: dalli
 version: !ruby/object:Gem::Version
-  version: 3.2.14
+  version: 3.2.15
 platform: ruby
 authors:
 - Peter M. Goldstein