dalli 3.2.13 → 3.2.15

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7f7bcbe37e631c3c733d1487e519501497702b873e81e6588d6dfad1ab96a65b
-  data.tar.gz: d7a953d2c0052334d8edb7fa45306724ffa25cec64436dcceccc9847d057e8bd
+  metadata.gz: 1282508d83c234919dfb2c8a05c08d7069186663c83d4e3aaf040e6f503a2e9f
+  data.tar.gz: cb10108ea1d22c25630b5cfa36b6e038515e3f053e2354e3921718050138535b
 SHA512:
-  metadata.gz: 8d7b0ed94102d333f611736c48d944b0f1010760ae646003db60cd02c90b1a047a5fd10f7a5ef077527a88594f8844cbe08f9647259749e78ade74d09d2e851b
-  data.tar.gz: bf86e4604fc18eaf2bdf57d6abad82258745238112a9866e5da9e05c0f45b44d62f82ebe8b5519ecc83d00ec6ee94e2d11bb931aa14171081a0b731745633e6d
+  metadata.gz: 614a4eebdaa9cee1d9dc6afdfd45c96ff1ec7eda12f471a12c6c1eca7866af971d907a7c288aeb5a72734b3f4f72bfb685ba58337da2ac50146ed74a018c0d0a
+  data.tar.gz: 8f577fef557f593d9cb6327a54ec1b5d3df8e554a5935ba7a7faadf86259248a60f29204b3ad5a295024b676ebc481e925f81f330095a73e6649e11d182b1127

data/CHANGELOG.md

@@ -4,6 +4,32 @@ Dalli Changelog
 Unreleased
 ==========
 
+3.2.15
+==========
+
+- Keep a connection shared by several threads or fibers in step during a `quiet` block (GHSA-5xvx-r8rg-5qm2)
+  - A `quiet` (or `multi`) block is quiet only for the thread or fiber that runs it. memcached still replies to a quiet request that fails (for example an `add` of a key that exists), and another thread or fiber using the same client while the block was open read that reply as its own, so its later requests could return another key's value. A non-quiet request now reads any pending quiet replies first
+  - Affects every supported line, with both protocols; with the binary protocol the connection stayed out of step after the block ended. Fixed in 5.2.4, 5.1.6, 5.0.12, 4.3.9 and 3.2.15
+- With `protocol: :meta`, read values by their size instead of up to the first CRLF (GHSA-5xvx-r8rg-5qm2)
+  - `get`, `gat` and `cas` read a value as one line, so a value containing CR/LF was cut short and the rest was parsed as the next reply. A stored value containing a crafted reply was then returned for a later read of a different key, with flags that could cause it to be deserialized
+  - Affects 3.2.0 to 3.2.14 with `protocol: :meta`; fixed in 3.2.15. The binary protocol (3.2's default) and 4.0 and later aren't affected
+- A per-request `raw: true` write no longer compresses the value (backport of #1213)
+  - Values at or above the compression threshold (4 KB by default) were compressed and flagged even with `raw: true`. Raw reads ignore flags, so reading them back with `raw: true` returned the compressed bytes
+  - Raw values are now stored exactly as given. Callers that pass `raw: true` without `compress: false` store large values uncompressed; Rails' `MemCacheStore` passes `compress: false` and isn't affected
+- Handle failed TLS handshakes like other connection failures, and warn when TLS verification is off (backport of #1215)
+  - A failed or timed-out handshake left the TCP socket open until GC and raised `OpenSSL::SSL::SSLError` directly, so the server was never marked down. The socket is now closed, and the error goes through the usual retry and `socket_max_failures` handling, raising `Dalli::NetworkError` (or `Dalli::RingError` once no server is available)
+  - Certificate and hostname checks come only from the `ssl_context` you pass, and a bare `OpenSSL::SSL::SSLContext.new` checks neither. Dalli now logs a warning once per process when `verify_mode` doesn't include `VERIFY_PEER` or `verify_hostname` is false; see the new TLS section in the README
+- A reply line longer than 8 KiB without a CRLF, or one cut short by the connection closing, is now a network error that closes the connection. Before, the line had no length limit (backport of #1216)
+
+3.2.14
+==========
+
+- Don't rely on memcached's auto-create for keys that need base64 encoding (GHSA-r47x-6p6r-mvcp)
+  - memcached stores an item made by auto-create (`ma` with `N`/`J`, or `mg` with `N`) for a base64-encoded key without its binary-key flag, and later replies name the item by its raw key bytes. A key containing spaces or CR/LF could then make a multi-get read the replies after it as different ones
+  - `incr` and `decr` with an initial value now create such counters with an add instead. Return values are unchanged
+  - The new `verify_reply_keys` option makes `get_multi` ignore replies for keys they didn't request, which guards against items created by older Dalli versions or other clients until memcached is upgraded. It's off by default, since it costs a multi-get about 15%
+  - Affects `protocol: :meta` since 3.2.0; fixed in 5.2.3, 5.1.5, 5.0.11, 4.3.8 and 3.2.14. memcached 1.6.45 and earlier still name these items by their raw key bytes
+
 3.2.13
 ==========
 

data/README.md

@@ -37,6 +37,29 @@ Dalli::Client.new('localhost:11211', serializer: Dalli::JSONSerializer)
 
 Per-request `raw: true` returns the stored bytes without deserializing or decompressing them, on every read method that accepts it (`get` and `fetch`).
 
+### Counters and verify_reply_keys
+
+memcached names an item made by auto-create for a base64-encoded key (one with whitespace, control or non-ASCII characters) by its raw key bytes in later replies (with `protocol: :meta`). Since 3.2.14, Dalli creates counters for such keys with an add rather than auto-create. Items created before then, or by other clients, can still be named that way until memcached releases a fix (1.6.45 and earlier behave this way). If keys passed to `incr` or `decr` with an initial value can contain untrusted input, set `verify_reply_keys: true` until then: multi-gets then ignore replies for keys they didn't request, at a cost of about 15% per multi-get.
+
+## TLS
+
+Pass an `OpenSSL::SSL::SSLContext` as `ssl_context:` to connect over TLS. Certificate and hostname verification come only from that context: Dalli adds no checks of its own. A bare `OpenSSL::SSL::SSLContext.new` verifies nothing, so it accepts any certificate from any server. Configure the context to verify both:
+
+```ruby
+ssl_context = OpenSSL::SSL::SSLContext.new
+ssl_context.set_params(
+  verify_mode: OpenSSL::SSL::VERIFY_PEER,
+  verify_hostname: true,
+  ca_file: '/path/to/ca.crt' # omit to use the system's trusted certificates
+)
+
+Dalli::Client.new('memcached.example.com:11211', ssl_context: ssl_context)
+```
+
+Dalli logs a warning once per process when the context's `verify_mode` is `VERIFY_NONE`, or when it is `VERIFY_PEER` but `verify_hostname` is false.
+
+A failed TLS handshake counts as a failed connection attempt, like a refused connection: once `socket_max_failures` attempts have failed, the server is marked down and requests fail over to the other servers.
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. You can run `bin/console` for an interactive prompt that will allow you to experiment.

data/lib/dalli/client.rb

@@ -45,6 +45,11 @@ module Dalli
     #                   useful for injecting a FIPS compliant hash object.
     # - :protocol - one of either :binary or :meta, defaulting to :binary.  This sets the protocol that Dalli uses
     #               to communicate with memcached.
+    # - :verify_reply_keys - defaults to false. When true, get_multi only accepts replies for the keys it requested;
+    #                        after a reply naming another key, the rest of that server's replies are treated as misses.
+    #                        With the meta protocol, memcached can name an item by its raw key bytes when the item was
+    #                        created by auto-create from a base64-encoded key, which Dalli no longer does. Costs a
+    #                        multi-get about 15%.
     #
     def initialize(servers = nil, options = {})
       @normalized_servers = ::Dalli::ServersArgNormalizer.normalize_servers(servers)
@@ -187,6 +192,11 @@ module Dalli
     # Any error replies will be discarded at the end of the block, and
     # Dalli client methods invoked inside the block will not
     # have return values
+    #
+    # The block is quiet only for the thread or fiber that runs it. Another
+    # thread or fiber sharing this client can make requests meanwhile; a
+    # non-quiet one first reads the block's pending replies with a noop and
+    # discards them, so it reads only its own.
     def quiet
       old = Thread.current[::Dalli::QUIET]
       Thread.current[::Dalli::QUIET] = true

data/lib/dalli/protocol/base.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require 'forwardable'
+require 'set'
 require 'socket'
 require 'timeout'
 
@@ -23,6 +24,7 @@ module Dalli
       def initialize(attribs, client_options = {})
         hostname, port, socket_type, @weight, user_creds = ServerConfigParser.parse(attribs)
         @options = client_options.merge(user_creds)
+        @verify_reply_keys = client_options[:verify_reply_keys] ? true : false
         @value_marshaller = ValueMarshaller.new(@options)
         @connection_manager = ConnectionManager.new(hostname, port, socket_type, @options)
       end
@@ -39,7 +41,11 @@ module Dalli
 
         begin
           @connection_manager.start_request!
+          drain_pending_quiet_responses(opkey)
           response = send(opkey, *args)
+          # Quiet requests can leave replies on the socket; remember that so a
+          # later non-quiet request knows to read them first
+          @connection_manager.quiet_responses_pending = true if quiet? && opkey != :noop
 
           # pipelined_get emit query but doesn't read the response(s)
           @connection_manager.finish_request! unless opkey == :pipelined_get
@@ -105,7 +111,7 @@ module Dalli
 
           # If the status is ok and the key is not nil, then this is a
           # getkq response with a value that we want to set in the response hash
-          values[key] = [value, cas] unless key.nil?
+          break unless keep_reading?(values, key, value, cas)
 
           # Get the next response from the buffer
           status, cas, key, value = response_buffer.process_single_getk_response
@@ -163,6 +169,18 @@ module Dalli
       private
 
       ALLOWED_QUIET_OPS = %i[add replace set delete incr decr append prepend flush noop].freeze
+
+      # Reads the replies still pending from quiet requests before any
+      # non-quiet request, which reads a reply of its own. A quiet block reads
+      # them at its end, but the connection is shared: another thread or fiber
+      # (quiet? is fiber-local) can make a request while the block is still
+      # open, and would otherwise read the block's replies as its own.
+      def drain_pending_quiet_responses(opkey)
+        return unless @connection_manager.quiet_responses_pending? && opkey != :noop && !quiet?
+
+        noop
+      end
+
       def verify_allowed_quiet!(opkey)
         return if ALLOWED_QUIET_OPS.include?(opkey)
 
@@ -220,6 +238,7 @@ module Dalli
       end
 
       def pipelined_get(keys)
+        expect_reply_keys(keys)
         req = +''
         keys.each do |key|
           req << quiet_get_request(key)
@@ -232,6 +251,50 @@ module Dalli
         @response_buffer ||= ResponseBuffer.new(@connection_manager, response_processor)
       end
 
+      # A multi-get only accepts replies for the keys it asked for. With the
+      # meta protocol, memcached names an item created by auto-create (ma or mg
+      # with N) for a base64-encoded key by its raw bytes, without the b flag;
+      # such a key can contain spaces or CR/LF, which split the reply header and
+      # make the replies after it parse as different ones. A key that wasn't
+      # requested is how that shows up. Only with the verify_reply_keys option,
+      # since building the set costs a multi-get about 15%.
+      def expect_reply_keys(keys)
+        @expected_reply_keys = (keys.to_set { |k| reply_key_form(k) } if @verify_reply_keys)
+      end
+
+      # Stores a getkq reply's value and returns true, unless its key wasn't
+      # requested: then the replies are out of step, so what was read so far is
+      # kept, the rest dropped, and false returned to stop reading
+      def keep_reading?(values, key, value, cas)
+        return true if key.nil?
+
+        if unexpected_reply_key?(key)
+          abandon_pipeline
+          return false
+        end
+
+        values[key] = [value, cas]
+        true
+      end
+
+      def unexpected_reply_key?(key)
+        !key.nil? && @expected_reply_keys && !@expected_reply_keys.include?(reply_key_form(key))
+      end
+
+      # Keys compared as bytes; ASCII keys compare equal in any encoding as is
+      def reply_key_form(key)
+        key.ascii_only? ? key : key.b
+      end
+
+      # Drops the rest of this server's replies and closes the connection so
+      # nothing unread is left on it. Not a socket failure, so it doesn't count
+      # toward socket_max_failures; the keys not read are simply missing.
+      def abandon_pipeline
+        Dalli.logger.warn { "#{name} replied to a get_multi with a key that wasn't requested; closing the connection" }
+        response_buffer.clear
+        @connection_manager.close
+      end
+
       # Called after the noop response is received at the end of a set
       # of pipelined gets
       def finish_pipeline

data/lib/dalli/protocol/binary.rb

@@ -139,7 +139,10 @@ module Dalli
       # We need to read all the responses at once.
       def noop
         write_noop
-        response_processor.consume_all_responses_until_noop
+        result = response_processor.consume_all_responses_until_noop
+        # Everything up to and including the noop reply has been read
+        @connection_manager.quiet_responses_pending = false
+        result
       end
 
       def stats(info = '')

data/lib/dalli/protocol/connection_manager.rb

@@ -28,6 +28,7 @@ module Dalli
 
       attr_accessor :hostname, :port, :socket_type, :options
       attr_reader :sock
+      attr_writer :quiet_responses_pending
 
       def initialize(hostname, port, socket_type, client_options)
         @hostname = hostname
@@ -35,6 +36,7 @@ module Dalli
         @socket_type = socket_type
         @options = DEFAULTS.merge(client_options)
         @request_in_progress = false
+        @quiet_responses_pending = false
         @sock = nil
         @pid = nil
 
@@ -56,8 +58,9 @@ module Dalli
         @sock = memcached_socket
         @pid = Process.pid
         @request_in_progress = false
-      rescue SystemCallError, *TIMEOUT_ERRORS, EOFError, SocketError => e
-        # SocketError = DNS resolution failure
+        @quiet_responses_pending = false
+      rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError, SocketError => e
+        # SocketError = DNS resolution failure; SSL_ERRORS = failed TLS handshake
         error_on_request!(e)
       end
 
@@ -122,6 +125,7 @@ module Dalli
         end
         @sock = nil
         @pid = nil
+        @quiet_responses_pending = false
         abort_request!
       end
 
@@ -136,6 +140,13 @@ module Dalli
         !@sock.nil?
       end
 
+      # True when a quiet request was written on this connection and its
+      # replies (memcached only suppresses some of them) haven't been
+      # consumed by a noop yet.
+      def quiet_responses_pending?
+        @quiet_responses_pending
+      end
+
       def request_in_progress?
         @request_in_progress
       end
@@ -161,9 +172,21 @@ module Dalli
         @request_in_progress = false
       end
 
+      # The longest reply line Dalli reads, in bytes, CRLF included. A meta
+      # reply header is at most about 500 bytes: "VA <size>" plus its flags,
+      # the largest being a 250-byte key base64-encoded to 336 bytes, a
+      # 20-digit CAS, a 32-byte opaque and a few numeric fields. The longest
+      # stats line (stats settings) carries a file path such as ext_path, up
+      # to PATH_MAX (4096). 8 KiB leaves room for both, while a server that
+      # never sends CRLF can't grow the read buffer without bound.
+      MAX_LINE_BYTES = 8 * 1024
+
       def read_line
-        data = @sock.gets("\r\n")
+        data = @sock.gets("\r\n", MAX_LINE_BYTES)
         error_on_request!('EOF in read_line') if data.nil?
+        # Without its CRLF the line either ran past MAX_LINE_BYTES or was cut
+        # short by EOF; either way the connection is out of step
+        error_on_request!("Reply line without CRLF in the first #{MAX_LINE_BYTES} bytes") unless data.end_with?("\r\n")
         data
       rescue SystemCallError, *TIMEOUT_ERRORS, EOFError => e
         error_on_request!(e)

data/lib/dalli/protocol/meta/response_processor.rb

@@ -34,7 +34,7 @@ module Dalli
           return cache_nils ? ::Dalli::NOT_FOUND : nil if tokens.first == EN
           return true unless tokens.first == VA
 
-          value = read_line
+          value = read_data(tokens[1].to_i)
           raw ? value : @value_marshaller.retrieve(value, bitflags_from_tokens(tokens))
         end
 
@@ -46,7 +46,7 @@ module Dalli
           cas = cas_from_tokens(tokens)
           return [nil, cas] unless tokens.first == VA
 
-          value = read_line
+          value = read_data(tokens[1].to_i)
           [raw ? value : @value_marshaller.retrieve(value, bitflags_from_tokens(tokens)), cas]
         end
 
@@ -242,6 +242,15 @@ module Dalli
           line = read_line
           line&.split || []
         end
+
+        # Values are read by the size in the VA line, not up to the next CRLF:
+        # a value can contain CRLF and can be longer than MAX_LINE_BYTES
+        def read_data(data_size)
+          # Checked before the terminator is added: -1 or -2 would otherwise
+          # read 1 or 0 bytes and leave the connection out of step
+          check_value_size!(data_size)
+          @io_source.read(data_size + TERMINATOR.bytesize)&.chomp!(TERMINATOR)
+        end
       end
     end
   end

data/lib/dalli/protocol/meta.rb

@@ -13,6 +13,7 @@ module Dalli
     ##
     class Meta < Base
       TERMINATOR = "\r\n"
+      COUNTER_ADD_OPTIONS = { raw: true }.freeze # counters are stored raw
 
       def response_processor
         @response_processor ||= ResponseProcessor.new(@connection_manager, @value_marshaller)
@@ -129,11 +130,42 @@ module Dalli
       def decr_incr(incr, key, delta, ttl, initial)
         ttl = initial ? TtlSanitizer.sanitize(ttl) : nil # Only set a TTL if we want to set a value on miss
         encoded_key, base64 = KeyRegularizer.encode(key)
+        # memcached stores an item created by ma's auto-create (N/J) for a
+        # base64-encoded key without its binary-key flag, and later replies
+        # name it with the raw key bytes. Create such counters with an add.
+        return decr_incr_with_add(incr, key, delta, ttl, initial) if initial && base64
+
         write(RequestFormatter.meta_arithmetic(key: encoded_key, delta: delta, initial: initial, incr: incr, ttl: ttl,
                                                quiet: quiet?, base64: base64))
         response_processor.decr_incr unless quiet?
       end
 
+      # Like decr_incr with an initial value, without memcached's auto-create:
+      # on a miss the counter is added with the initial value (which is
+      # returned, as auto-create would), and the add losing a race to another
+      # client means the counter now exists, so the incr is tried again. In a
+      # quiet block both requests are sent and whichever applies takes effect.
+      def decr_incr_with_add(incr, key, delta, ttl, initial)
+        return quiet_decr_incr_with_add(incr, key, delta, ttl, initial) if quiet?
+
+        2.times do
+          value = decr_incr(incr, key, delta, nil, nil)
+          return value unless value.nil?
+
+          write_storage_req(:add, key, initial.to_s, ttl, nil, COUNTER_ADD_OPTIONS)
+          return initial if response_processor.meta_set_with_cas
+        end
+        decr_incr(incr, key, delta, nil, nil)
+      end
+
+      def quiet_decr_incr_with_add(incr, key, delta, ttl, initial)
+        encoded_key, base64 = KeyRegularizer.encode(key)
+        write(RequestFormatter.meta_arithmetic(key: encoded_key, delta: delta, initial: nil, incr: incr, ttl: nil,
+                                               quiet: true, base64: base64))
+        # Sent with the rest of the quiet block, by its closing noop
+        write_storage_req(:add, key, initial.to_s, ttl, nil, COUNTER_ADD_OPTIONS)
+      end
+
       # Other Commands
       def flush(delay = 0)
         write(RequestFormatter.flush(delay: delay))
@@ -144,7 +176,10 @@ module Dalli
       # We need to read all the responses at once.
       def noop
         write_noop
-        response_processor.consume_all_responses_until_mn
+        result = response_processor.consume_all_responses_until_mn
+        # Everything up to and including the MN has been read
+        @connection_manager.quiet_responses_pending = false
+        result
       end
 
       def stats(info = nil)

data/lib/dalli/protocol/value_compressor.rb

@@ -97,11 +97,13 @@ module Dalli
       end
 
       # Checks whether we should apply compression when serializing a value
-      # based on the specified options.  Returns false unless the value
-      # is greater than the minimum compression size.  Otherwise returns
-      # based on a method-level option if specified, falling back to the
-      # server default.
+      # based on the specified options.  Returns false for a raw request
+      # (raw reads ignore bitflags, so the value must be stored as given) or
+      # unless the value is greater than the minimum compression size.
+      # Otherwise returns based on a method-level option if specified,
+      # falling back to the server default.
       def compress_value?(value, req_options)
+        return false if req_options&.dig(:raw)
         return false unless value.bytesize >= compression_min_size
         return compress_by_default? unless req_options && !req_options[:compress].nil?
 

data/lib/dalli/protocol.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require 'openssl'
 require 'timeout'
 
 module Dalli
@@ -15,5 +16,11 @@ module Dalli
       else
         [Timeout::Error]
       end
+
+    # SSL errors during the TLS handshake count as a failed request: they
+    # trigger reconnection and, after socket_max_failures, mark the server
+    # down. openssl is required above because this file loads before
+    # dalli/socket, so OpenSSL may not be defined yet.
+    SSL_ERRORS = [OpenSSL::SSL::SSLError].freeze
   end
 end

data/lib/dalli/socket.rb

@@ -132,12 +132,42 @@ module Dalli
         sock.setsockopt(::Socket::SOL_SOCKET, ::Socket::SO_SNDTIMEO, timeval)
       end
 
+      # Closes the TCP socket if the handshake fails or times out, so a failed
+      # attempt doesn't hold a file descriptor until GC.
       def self.wrapping_ssl_socket(tcp_socket, host, ssl_context)
+        warn_if_unverified(ssl_context)
+        connected = false
         ssl_socket = Dalli::Socket::SSLSocket.new(tcp_socket, ssl_context)
         ssl_socket.hostname = host
         ssl_socket.sync_close = true
         ssl_socket.connect
+        connected = true
         ssl_socket
+      ensure
+        tcp_socket.close unless connected
+      end
+
+      # Verification comes entirely from the caller's ssl_context, and a bare
+      # SSLContext.new verifies nothing, so warn (once per process) when the
+      # certificate or the hostname won't be checked.
+      def self.warn_if_unverified(ssl_context)
+        return if @ssl_verification_warning_logged
+
+        problem = unverified_problem(ssl_context)
+        return unless problem
+
+        Dalli.logger.warn "SECURITY WARNING: Dalli's ssl_context #{problem}. " \
+                          'Use ssl_context.set_params(verify_mode: OpenSSL::SSL::VERIFY_PEER, ' \
+                          'verify_hostname: true) to verify the server.'
+        @ssl_verification_warning_logged = true
+      end
+
+      def self.unverified_problem(ssl_context)
+        if !ssl_context.verify_mode.to_i.anybits?(::OpenSSL::SSL::VERIFY_PEER)
+          'verify_mode is VERIFY_NONE, so the server certificate is not checked'
+        elsif !ssl_context.verify_hostname
+          'verify_hostname is false, so the certificate is not checked against the hostname'
+        end
       end
     end
 

data/lib/dalli/version.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module Dalli
-  VERSION = '3.2.13'
+  VERSION = '3.2.15'
 
   MIN_SUPPORTED_MEMCACHED_VERSION = '1.4'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: dalli
 version: !ruby/object:Gem::Version
-  version: 3.2.13
+  version: 3.2.15
 platform: ruby
 authors:
 - Peter M. Goldstein
@@ -90,7 +90,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.21
+rubygems_version: 4.0.23
 specification_version: 4
 summary: High performance memcached client for Ruby
 test_files: []