dalli 4.3.5 → 4.3.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 96ac26bd7ef748d2407efaa5d1bcfe8f762c2eb47d67f85fb6e19f789926abae
-  data.tar.gz: 617744189a342d6b39bd741a685b997bb0fbbae82afb3aeaafbf87ac79f6fe29
+  metadata.gz: dd77fe4e8ef65f531c5d9eb4a8e638feb6e443493cd57125bad104867399cd19
+  data.tar.gz: 814cc63401565f306a2aa6045476579f93b028fff8802ea443cde65375328636
 SHA512:
-  metadata.gz: 7c0f37ce570d445484045722da9d5463879d825cb0ff2c8436e59852e0a996b44be61ebf942eb7bfc14846fb64991edf4cbf75c5bfa5871d8f3409b8a3fae25d
-  data.tar.gz: 3a96488a8bb4da71afbb724745da958bec01ce5e9e229d7b5f0be0f5bd56513a0de7acefba6feab012efab5cd8d9971a5956a4aa5f1c9b154f503721bd703b72
+  metadata.gz: c983d9e656540f313aaea157b8d0290c3610b7c03575625a6e8cfba4ec5be99a893955506b409867689de1a5f4acc7715eee07928df8ef5c8724141ffae243b2
+  data.tar.gz: 62d38b83c8ba58ec871613b25e5f197d14c768436cde9f3aa382189dd6af19802832c545474add6a8b8b9edb373c948cde2db15599b5bba1b15720db71d1cfed

data/CHANGELOG.md

@@ -4,6 +4,54 @@ Dalli Changelog
 Unreleased
 ==========
 
+4.3.6
+==========
+
+Security:
+
+- Fix pipelined `get_multi` returning one key's value for another after an error reply (GHSA-p6pm-ch9v-44vx)
+  - The pipelined reply parser took any reply without a body for the end of the batch. An error reply for one key (`CLIENT_ERROR`, `SERVER_ERROR`, or `EN` from a proxy) ended it early, and the replies still on the connection were read as the replies to later commands, so a `get` could return another key's value
+  - Keys are now measured in bytes as sent (after base64 encoding when the meta protocol needs it). A key under 250 characters but over 250 bytes, for example one with many non-ASCII characters, passed validation:
+    - with the meta protocol, memcached rejected it with the error reply above
+    - with the binary protocol, memcached dropped the connection, and `get` or `get_multi` with that key retried forever
+  - Such keys are now truncated like other long keys; keys that worked before are unchanged
+  - The reply parser affects `protocol: :meta` since 3.2.0; the key length affects both protocols; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
+- Stop retrying an unresponsive server forever (GHSA-4qp6-2jcr-596v)
+  - A server that accepted connections but never answered (or dropped the connection on a request) was retried forever, hanging the caller, because each successful reconnect reset the failure count. Requests now fail after `socket_max_failures` attempts and the server is marked down, as when it can't be reached at all; it gets a full set of attempts again after `down_retry_delay`
+  - Affects all versions; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
+- Limit decompressed and reply value sizes (GHSA-3553-vcg5-72jw)
+  - Values flagged as compressed were inflated without limit, so a small stored value could expand to gigabytes on read, whatever the client's `compress` or `serializer` settings. The new `decompressed_max_bytes` option (default 128 MiB; `nil` disables it) makes a read that would pass it raise `Dalli::UnmarshalError`. Custom compressors whose `decompress` takes only the data keep working, without the limit
+  - The value size in a reply (a meta `VA` size or a binary body length) was used to read or buffer that many bytes, so a malicious server could make the client allocate gigabytes. Sizes over 1 GiB, memcached's largest item, now raise `Dalli::DalliError` before reading
+  - Affects all versions; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
+- Honor per-request `raw: true` on reads, and add `Dalli::JSONSerializer` (GHSA-wr87-m4jw-29x5)
+  - With the binary protocol, `get`, `gat` and `fetch`, and with the meta protocol, `get_with_metadata`, ignored a per-request `raw: true` and deserialized the value according to its stored flags, so a caller who asked for raw bytes could still have `Marshal.load` run on data someone else wrote to memcached. They now return the stored bytes
+  - The `raw` part affects binary `get` and `gat` in all versions, and `get_with_metadata` since 4.2.0
+  - `serializer: JSON` reads values with `JSON.load`, which on json gem versions before 3.0 creates an object of the class named in a stored `json_class` key when the json additions are loaded. The new `Dalli::JSONSerializer` reads with `JSON.parse` and only returns plain JSON types. The README and the Marshal security warning now recommend it; `serializer: JSON` itself is unchanged
+  - The `serializer: JSON` part affects any version used with the json gem before 3.0
+- Keep a forked child from writing to or closing the parent's connection (GHSA-w39f-xq2m-4g8x)
+  - A child forked while the parent had requests buffered but not yet sent, such as quiet writes inside a `quiet` block, sent them again when it closed the client or reconnected: quiet writes ran twice, and the parent's later replies could be read as the replies to other requests, returning one key's value for another. Dalli now buffers requests itself and a forked child discards them
+  - Affects 4.2.0 and later; fixed in 5.2.1, 5.1.3, 5.0.9 and 4.3.6
+  - With TLS, a forked child that closed the client (or reconnected after detecting the fork) sent a TLS close on the connection the parent was still using, ending the parent's session. A forked child now closes only its own copy of the socket
+  - Affects all versions with TLS; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
+
+Bug fixes:
+
+- An exception raised by a `get_multi` block (such as `Timeout::Error`) was treated as a network failure: swallowed, with the whole `get_multi` retried and keys yielded twice. It now reaches the caller
+- Fix `set_multi` with the binary protocol
+  - Each quiet `setq` waited for a reply memcached never sends for a successful write, so `set_multi` timed out on its first key without storing anything (and, with retries, hung or marked the server down). Affects 4.2.0 and later
+
+Notes:
+
+- Document [Ruby bug 21195](https://bugs.ruby-lang.org/issues/21195), a VM crash (`[BUG] rb_sys_fail_path_in(io_fillbuf, ...) - errno == 0`) when a socket read using `IO#timeout` is interrupted by a signal, in the README (#1188)
+  - Affects Ruby 3.2.x, 3.3.0–3.3.7 and 3.4.0–3.4.2. Fixed in 3.3.8 and 3.4.3; Ruby 3.2 reached end of life without the fix
+  - Ruby 3.1 doesn't use `IO#timeout` and isn't affected
+
+Development:
+
+- Drop Ruby 3.2 from the CI matrix, since that bug intermittently crashed its test runs (#1188)
+- Fix three flaky tests: the `KeyManager` namespace key-length test, the Rack session freshness test, and `MemcachedManager`'s start and stop handling (backport of #1141, #1182)
+- Fix flaky failover tests: move their ports out of Linux's ephemeral port range, and wait for memcached to accept connections after starting it (backport of #1184)
+
 4.3.5
 ==========
 

data/README.md

@@ -20,6 +20,18 @@ The name is a variant of Salvador Dali for his famous painting [The Persistence
 * Ruby 3.1 or later
 * memcached 1.4 or later (1.6+ recommended for meta protocol support)
 
+### Known Ruby issue: crash on an interrupted socket read
+
+On Ruby 3.2 and later, Dalli sets socket timeouts with `IO#timeout`. Ruby 3.2.x, 3.3.0–3.3.7 and 3.4.0–3.4.2 have a bug ([Ruby #21195](https://bugs.ruby-lang.org/issues/21195)) where a timed read interrupted by a signal aborts the whole process:
+
+```
+[BUG] rb_sys_fail_path_in(io_fillbuf, fd:N ) - errno == 0
+```
+
+The crash is inside Ruby itself, so it can't be rescued. It was fixed in Ruby 3.3.8 and 3.4.3. Ruby 3.2 reached end of life without the fix. If you see this crash, upgrade to Ruby 3.3.8+ or 3.4.3+. Ruby 3.1 doesn't use `IO#timeout` and isn't affected.
+
+Ruby 3.2 is no longer part of this branch's CI matrix for the same reason.
+
 ## Protocol Options
 
 Dalli supports two protocols for communicating with memcached:
@@ -61,9 +73,13 @@ The separator must be a single non-alphanumeric character. Valid examples: `:`,
 By default, Dalli uses Ruby's Marshal for serialization. Deserializing untrusted data with Marshal can lead to remote code execution. If you cache user-controlled data, consider using a safer serializer:
 
 ```ruby
-Dalli::Client.new('localhost:11211', serializer: JSON)
+Dalli::Client.new('localhost:11211', serializer: Dalli::JSONSerializer)
 ```
 
+`Dalli::JSONSerializer` reads values with `JSON.parse`, so it only ever returns plain hashes, arrays, strings, numbers, booleans and `nil`. Passing the `JSON` module itself (`serializer: JSON`) reads values with `JSON.load`, which can create objects of other classes from a stored `json_class` key and is not recommended for data an attacker might write.
+
+Per-request `raw: true` returns the stored bytes without deserializing or decompressing them, on every read method that accepts it (`get`, `get_with_metadata` and `fetch`).
+
 See the [4.0-Upgrade.md](4.0-Upgrade.md) guide for more information.
 
 ## OpenTelemetry Tracing

data/lib/dalli/compressor.rb

@@ -13,8 +13,49 @@ module Dalli
       Zlib::Deflate.deflate(data)
     end
 
-    def self.decompress(data)
-      Zlib::Inflate.inflate(data)
+    # DEFLATE can't expand data by more than about 1032:1
+    MAX_EXPANSION = 1032
+    # Compressed input is fed to zlib this many bytes at a time. MRI's zlib
+    # yields the output in small chunks; one that returns it instead still
+    # produces at most MAX_EXPANSION times this per step.
+    INFLATE_SLICE_BYTES = 64 * 1024
+
+    # max_bytes caps the decompressed size; inflating stops and raises as soon
+    # as the output passes it, so a small compressed value can't expand into
+    # gigabytes of memory.
+    def self.decompress(data, max_bytes: nil)
+      return Zlib::Inflate.inflate(data) unless may_exceed?(data, max_bytes)
+
+      inflate_within_limit(Zlib::Inflate.new, data, max_bytes)
+    end
+
+    # False when even the largest possible expansion stays within max_bytes,
+    # so the value can be inflated in one step
+    def self.may_exceed?(data, max_bytes)
+      max_bytes ? data.bytesize * MAX_EXPANSION > max_bytes : false
+    end
+
+    def self.inflate_within_limit(inflater, data, max_bytes)
+      out = String.new(encoding: Encoding::BINARY)
+      0.step(data.bytesize - 1, INFLATE_SLICE_BYTES) do |pos|
+        inflate_slice(inflater, data.byteslice(pos, INFLATE_SLICE_BYTES), out, max_bytes)
+      end
+      append_within_limit(out, inflater.finish, max_bytes)
+    ensure
+      inflater.close
+    end
+
+    # Both yielded and returned output count toward the limit
+    def self.inflate_slice(inflater, slice, out, max_bytes)
+      rest = inflater.inflate(slice) { |chunk| append_within_limit(out, chunk, max_bytes) }
+      append_within_limit(out, rest, max_bytes)
+    end
+
+    def self.append_within_limit(out, chunk, max_bytes)
+      out << chunk if chunk
+      return out if out.bytesize <= max_bytes
+
+      raise Dalli::UnmarshalError, "Decompressed value exceeds #{max_bytes} bytes (decompressed_max_bytes)"
     end
   end
 
@@ -32,9 +73,11 @@ module Dalli
       io.string
     end
 
-    def self.decompress(data)
-      io = StringIO.new(data, 'rb')
-      Zlib::GzipReader.new(io).read
+    def self.decompress(data, max_bytes: nil)
+      return Zlib::GzipReader.new(StringIO.new(data, 'rb')).read unless Compressor.may_exceed?(data, max_bytes)
+
+      # MAX_WBITS + 16 makes zlib read the gzip format (and check its CRC)
+      Compressor.inflate_within_limit(Zlib::Inflate.new(Zlib::MAX_WBITS + 16), data, max_bytes)
     end
   end
 end

data/lib/dalli/json_serializer.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+require 'json'
+
+module Dalli
+  ##
+  # A serializer that stores values as JSON and never creates Ruby objects
+  # from what it reads.
+  #
+  # Passing the JSON module itself as the serializer (serializer: JSON) reads
+  # values back with JSON.load, which on json gem versions before 3.0 honors
+  # "json_class" and instantiates other classes when the json gem's additions
+  # are loaded. This serializer reads with JSON.parse, so a value written to
+  # memcached by someone else can only come back as plain hashes, arrays,
+  # strings, numbers, booleans and nil.
+  #
+  #   Dalli::Client.new(servers, serializer: Dalli::JSONSerializer)
+  ##
+  module JSONSerializer
+    def self.dump(value)
+      JSON.generate(value)
+    end
+
+    def self.load(data)
+      JSON.parse(data)
+    end
+  end
+end

data/lib/dalli/key_manager.rb

@@ -41,6 +41,9 @@ module Dalli
 
       @namespace = namespace_from_options
       @namespace_separator = @key_options[:namespace_separator]
+      # The meta protocol base64-encodes keys with whitespace or non-ASCII
+      # characters, so their length on the wire differs from their length here
+      @meta_protocol = client_options[:protocol]&.to_s == 'meta'
     end
 
     ##
@@ -58,7 +61,7 @@ module Dalli
       raise ArgumentError, 'key cannot be blank' unless key&.length&.positive?
 
       key = key_with_namespace(key)
-      key.length > MAX_KEY_LENGTH ? truncated_key(key) : key
+      wire_length(key) > MAX_KEY_LENGTH ? truncated_key(key) : key
     end
 
     ##
@@ -122,7 +125,27 @@ module Dalli
     ##
     def truncated_key(key)
       digest = digest_class.hexdigest(key)
-      "#{key[0, prefix_length(digest)]}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
+      prefix = key[0, prefix_length(digest)]
+      truncated = "#{prefix}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
+      # A prefix with whitespace or non-ASCII characters sends the whole key
+      # base64-encoded, which can still be too long on the wire; shorten it.
+      while wire_length(truncated) > MAX_KEY_LENGTH
+        prefix = prefix[0...-1]
+        truncated = "#{prefix}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
+      end
+      truncated
+    end
+
+    # memcached's key limit applies to the key as sent: in bytes, and, with
+    # the meta protocol, after base64 encoding when the key needs it. A key
+    # within the limit in characters can still be too long. With the meta
+    # protocol memcached rejects it with a CLIENT_ERROR that must never reach
+    # a pipelined get; with the binary protocol it drops the connection.
+    def wire_length(key)
+      return key.bytesize unless @meta_protocol
+
+      encoded, = Dalli::Protocol::Meta::KeyRegularizer.encode(key)
+      encoded.bytesize
     end
 
     def prefix_length(digest)

data/lib/dalli/protocol/base.rb

@@ -34,6 +34,13 @@ module Dalli
         @raw_mode
       end
 
+      # True when values should come back as stored: the client is in raw mode,
+      # or this request passed raw: true. A stored value is then never
+      # deserialized or decompressed for a caller who asked for raw bytes.
+      def raw_request?(options)
+        raw_mode? || (options.is_a?(Hash) && options[:raw]) ? true : false
+      end
+
       # Chokepoint method for error handling and ensuring liveness
       def request(opkey, *args)
         verify_state(opkey)
@@ -99,7 +106,15 @@ module Dalli
         reconnect_on_pipeline_complete!
         values = nil
 
-        response_buffer.read
+        # Only the read is socket I/O. The caller's block runs below, outside
+        # this rescue, so an Errno or Timeout::Error it raises reaches the
+        # caller instead of being mistaken for a network failure (which closed
+        # the connection and retried the whole get_multi, yielding keys twice).
+        begin
+          response_buffer.read
+        rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError => e
+          @connection_manager.error_on_request!(e)
+        end
 
         status, cas, key, value = response_buffer.process_single_getk_response
         # status is not nil only if we have a full response to parse
@@ -125,8 +140,6 @@ module Dalli
         end
 
         values || {}
-      rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError => e
-        @connection_manager.error_on_request!(e)
       end
 
       # Abort current pipelined get. Generally used to signal an external

data/lib/dalli/protocol/binary/response_processor.rb

@@ -66,7 +66,9 @@ module Dalli
           raise Dalli::DalliError, "Response error #{resp_header.status}: #{RESPONSE_CODES[resp_header.status]}"
         end
 
-        def get(cache_nils: false)
+        # raw: true returns the value as stored, without deserializing or
+        # decompressing it, whatever flags it was stored with
+        def get(cache_nils: false, raw: false)
           resp_header, body = read_response
 
           return false if resp_header.not_stored? # Not stored, normal status for add operation
@@ -75,7 +77,7 @@ module Dalli
           raise_on_not_ok!(resp_header)
           return true unless body
 
-          unpack_response_body(resp_header, body, true).last
+          unpack_response_body(resp_header, body, !raw).last
         end
 
         ##
@@ -199,6 +201,17 @@ module Dalli
         # The remaining three values in the array are the ResponseHeader,
         # key, and value.
         ##
+        # memcached can't store an item larger than 1 GiB (its -I maximum). A
+        # pipelined reply claiming an impossible size would otherwise have the
+        # buffer wait for (and accumulate) that many bytes.
+        MAX_BODY_BYTES = 1024 * 1024 * 1024
+
+        def check_body_size!(size)
+          return if size <= MAX_BODY_BYTES
+
+          raise Dalli::DalliError, "Reply body size #{size} is out of range"
+        end
+
         def getk_response_from_buffer(buf, offset = 0)
           # There's no header in the buffer, so don't advance
           return [0, nil, nil, nil, nil] unless buf && buf.bytesize >= offset + ResponseHeader::SIZE
@@ -212,6 +225,8 @@ module Dalli
           # error response that needs to be discarded.
           return [ResponseHeader::SIZE, resp_header.ok?, resp_header.cas, nil, nil] if body_len.zero?
 
+          check_body_size!(body_len)
+
           resp_size = ResponseHeader::SIZE + body_len
           # The header is in the buffer, but the body is not.  As we don't have
           # a complete response, don't advance the buffer

data/lib/dalli/protocol/binary.rb

@@ -23,7 +23,7 @@ module Dalli
         req = RequestFormatter.standard_request(opkey: :get, key: key)
         write(req)
         @connection_manager.flush
-        response_processor.get(cache_nils: cache_nils?(options))
+        response_processor.get(cache_nils: cache_nils?(options), raw: raw_request?(options))
       end
 
       def quiet_get_request(key)
@@ -35,7 +35,7 @@ module Dalli
         req = RequestFormatter.standard_request(opkey: :gat, key: key, ttl: ttl)
         write(req)
         @connection_manager.flush
-        response_processor.get(cache_nils: cache_nils?(options))
+        response_processor.get(cache_nils: cache_nils?(options), raw: raw_request?(options))
       end
 
       def touch(key, ttl)
@@ -63,7 +63,7 @@ module Dalli
       # Pipelined set - writes a quiet set request without reading response.
       # Used by PipelinedSetter for bulk operations.
       def pipelined_set(key, value, ttl, options)
-        storage_req(:setq, key, value, ttl, 0, options)
+        storage_req(:setq, key, value, ttl, 0, options, quiet: true)
       end
 
       def add(key, value, ttl, options)
@@ -77,7 +77,7 @@ module Dalli
       end
 
       # rubocop:disable-next Metrics/ParameterLists
-      def storage_req(opkey, key, value, ttl, cas, options)
+      def storage_req(opkey, key, value, ttl, cas, options, quiet: quiet?)
         (value, bitflags) = @value_marshaller.store(key, value, options)
         ttl = TtlSanitizer.sanitize(ttl)
 
@@ -85,8 +85,8 @@ module Dalli
                                                 value: value, bitflags: bitflags,
                                                 ttl: ttl, cas: cas)
         write(req)
-        @connection_manager.flush unless quiet?
-        response_processor.storage_response unless quiet?
+        @connection_manager.flush unless quiet
+        response_processor.storage_response unless quiet
       end
 
       def append(key, value)

data/lib/dalli/protocol/connection_manager.rb

@@ -37,7 +37,10 @@ module Dalli
         @request_in_progress = false
         @sock = nil
         @pid = nil
+        @write_buffer = []
+        @write_buffer_bytes = 0
 
+        @fail_count = 0
         reset_down_info
       end
 
@@ -53,7 +56,8 @@ module Dalli
         Dalli.logger.debug { "Dalli::Server#connect #{name}" }
 
         @sock = memcached_socket
-        @sock.sync = false # Enable buffered I/O for better performance
+        # Writes are buffered in @write_buffer instead; see WRITE_BUFFER_FLUSH_BYTES
+        @sock.sync = true
         @pid = PIDCache.pid
         @request_in_progress = false
       rescue SystemCallError, *TIMEOUT_ERRORS, EOFError, SocketError => e
@@ -85,6 +89,8 @@ module Dalli
       def down!
         close
         log_down_detected
+        # Once down_retry_delay passes, the server gets a full set of attempts
+        @fail_count = 0
 
         @error = $ERROR_INFO&.class&.name
         @msg ||= $ERROR_INFO&.message
@@ -114,15 +120,23 @@ module Dalli
         return unless @sock
 
         begin
-          @sock.close
+          close_socket
         rescue StandardError
           nil
         end
         @sock = nil
         @pid = nil
+        discard_write_buffer
         abort_request!
       end
 
+      # A forked child shares the parent's connection, so it closes only its
+      # own file descriptor. Closing the TLS socket itself would send
+      # close_notify and end the parent's TLS session.
+      def close_socket
+        fork_detected? ? @sock.to_io.close : @sock.close
+      end
+
       def connected?
         !@sock.nil?
       end
@@ -141,6 +155,11 @@ module Dalli
         raise '[Dalli] No request in progress. This may be a bug in Dalli.' unless @request_in_progress
 
         @request_in_progress = false
+        # A completed request is what proves the server healthy again, so the
+        # failure count resets here rather than on reconnect: a server that
+        # accepts connections but never answers would otherwise reset it on
+        # every retry and never reach socket_max_failures.
+        @fail_count = 0
       end
 
       def abort_request!
@@ -148,6 +167,7 @@ module Dalli
       end
 
       def read_line
+        flush_write_buffer
         data = @sock.gets("\r\n")
         error_on_request!('EOF in read_line') if data.nil?
         data
@@ -155,27 +175,53 @@ module Dalli
         error_on_request!(e)
       end
 
+      # memcached can't store an item larger than 1 GiB (its -I maximum), so a
+      # reply claiming more (or a negative size) is malformed or hostile.
+      # Reading allocates the full count up front, so check before reading.
+      MAX_READ_BYTES = (1024 * 1024 * 1024) + 2 # plus the value's trailing "\r\n"
+
+      def check_read_size!(count)
+        return if count.between?(0, MAX_READ_BYTES)
+
+        raise Dalli::DalliError, "Reply size #{count} from #{name} is out of range"
+      end
+
       def read(count)
+        check_read_size!(count)
+        flush_write_buffer
         @sock.readfull(count)
       rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError => e
         error_on_request!(e)
       end
 
+      # Requests are buffered here rather than in the socket's own IO buffer
+      # (the socket is sync). Ruby flushes an IO's write buffer when any process
+      # holding it closes or finalizes the IO, so after a fork a child could
+      # send bytes the parent had buffered but not sent yet: a quiet write would
+      # run twice, or the parent's replies would shift onto the wrong requests.
+      # A forked child discards this buffer instead. Like IO's own buffer, it
+      # is sent once it grows past this size, or when a reply is read.
+      WRITE_BUFFER_FLUSH_BYTES = 64 * 1024
+
       def write(bytes)
-        @sock.write(bytes)
-      rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS => e
+        @write_buffer << bytes
+        @write_buffer_bytes += bytes.bytesize
+        flush_write_buffer if @write_buffer_bytes >= WRITE_BUFFER_FLUSH_BYTES
+        bytes.bytesize
+      rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, IOError => e
         error_on_request!(e)
       end
 
       def flush
-        @sock.flush
-      rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS => e
+        flush_write_buffer
+      rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, IOError => e
         error_on_request!(e)
       end
 
       # Non-blocking read.  Here to support the operation
       # of the get_multi operation
       def read_nonblock
+        flush_write_buffer
         @sock.read_available
       end
 
@@ -202,8 +248,9 @@ module Dalli
         raise Dalli::RetryableNetworkError, message
       end
 
+      # Called on connect. Deliberately leaves @fail_count alone; see
+      # finish_request!.
       def reset_down_info
-        @fail_count = 0
         @down_at = nil
         @last_down_at = nil
         @msg = nil
@@ -228,7 +275,8 @@ module Dalli
       def reconnect_on_fork
         message = 'Fork detected, re-connecting child process...'
         Dalli.logger.info { message }
-        # Close socket on a fork and reconnect immediately
+        # Drops anything the parent had buffered, closes the inherited socket
+        # without touching the parent's connection, and reconnects immediately
         close
         establish_connection
       end
@@ -255,6 +303,20 @@ module Dalli
         time = Time.now - @down_at
         Dalli.logger.warn { format('%<name>s is back (downtime was %<time>.3f seconds)', name: name, time: time) }
       end
+
+      private
+
+      def flush_write_buffer
+        return if @write_buffer.empty?
+
+        @sock.write(*@write_buffer)
+        discard_write_buffer
+      end
+
+      def discard_write_buffer
+        @write_buffer.clear
+        @write_buffer_bytes = 0
+      end
     end
   end
 end

data/lib/dalli/protocol/meta/response_processor.rb

@@ -172,13 +172,14 @@ module Dalli
           header_len = header.bytesize + TERMINATOR.length
           body_len = body_len_from_tokens(tokens)
 
-          # We have a complete response that has no body.
-          # This is either the response to the terminating
-          # noop or, if the status is not MN, an intermediate
-          # error response that needs to be discarded. A hit
-          # on an empty value (VA 0) still has a body -- just
-          # its terminator -- so it's parsed below as a value.
-          return [header_len, true, nil, nil, nil] if no_body?(tokens, body_len)
+          # A complete response with no body. Only the terminating noop's MN
+          # ends the pipeline. Any other bodyless reply (CLIENT_ERROR,
+          # SERVER_ERROR, ...) answers one key's request: report it with a
+          # false status so the caller skips it and keeps reading. Treating it
+          # as the end would leave the remaining replies on the connection for
+          # later commands to read as their own. A hit on an empty value (VA 0)
+          # still has a body -- just its terminator -- so it's parsed below.
+          return [header_len, tokens.first == MN, nil, nil, nil] if no_body?(tokens, body_len)
 
           resp_size = header_len + body_len + TERMINATOR.length
           # The header is in the buffer, but the body is not.  As we don't have
@@ -191,6 +192,17 @@ module Dalli
           full_response_from_buffer(tokens, body, resp_size)
         end
 
+        # memcached can't store an item larger than 1 GiB (its -I maximum). A
+        # pipelined reply claiming an impossible size would otherwise have the
+        # buffer wait for (and accumulate) that many bytes.
+        MAX_VALUE_BYTES = 1024 * 1024 * 1024
+
+        def check_value_size!(size)
+          return if size.between?(0, MAX_VALUE_BYTES)
+
+          raise Dalli::DalliError, "Reply value size #{size} is out of range"
+        end
+
         # A zero-size reply has no body unless it's a VA (a hit on an empty value)
         def no_body?(tokens, body_len)
           body_len.zero? && tokens.first != VA
@@ -236,7 +248,9 @@ module Dalli
         end
 
         def body_len_from_tokens(tokens)
-          value_from_tokens(tokens, 's')&.to_i
+          size = value_from_tokens(tokens, 's')&.to_i
+          check_value_size!(size) if size
+          size
         end
 
         def value_from_tokens(tokens, flag)

data/lib/dalli/protocol/meta.rb

@@ -96,7 +96,7 @@ module Dalli
       def meta_get(key, options = {})
         encoded_key, base64 = KeyRegularizer.encode(key)
         req = RequestFormatter.meta_get(
-          key: encoded_key, value: true, return_cas: true, base64: base64,
+          key: encoded_key, value: true, return_cas: true, base64: base64, skip_flags: raw_request?(options),
           vivify_ttl: options[:vivify_ttl], recache_ttl: options[:recache_ttl],
           return_hit_status: options[:return_hit_status],
           return_last_access: options[:return_last_access], skip_lru_bump: options[:skip_lru_bump]

data/lib/dalli/protocol/value_compressor.rb

@@ -15,11 +15,17 @@ module Dalli
         compress: true,
         compressor: ::Dalli::Compressor,
         # min byte size to attempt compression
-        compression_min_size: 4 * 1024 # 4K
+        compression_min_size: 4 * 1024, # 4K
+        # max size a stored value may decompress to (nil: no limit). Guards
+        # against a small compressed value expanding into gigabytes on read.
+        decompressed_max_bytes: 128 * 1024 * 1024 # 128 MiB
       }.freeze
 
       OPTIONS = DEFAULTS.keys.freeze
 
+      KEYWORD_PARAMETER_TYPES = %i[key keyreq].freeze
+      private_constant :KEYWORD_PARAMETER_TYPES
+
       # https://www.hjp.at/zettel/m/memcached_flags.rxml
       # Looks like most clients use bit 1 to indicate gzip compression.
       FLAG_COMPRESSED = 0x2
@@ -38,8 +44,16 @@ module Dalli
       end
 
       def retrieve(value, bitflags)
-        compressed = bitflags.anybits?(FLAG_COMPRESSED)
-        compressed ? compressor.decompress(value) : value
+        return value unless bitflags.anybits?(FLAG_COMPRESSED)
+
+        max_bytes = @compression_options[:decompressed_max_bytes]
+        # Custom compressors that only define decompress(data) keep working,
+        # without the limit
+        if max_bytes && compressor_accepts_max_bytes?
+          compressor.decompress(value, max_bytes: max_bytes)
+        else
+          compressor.decompress(value)
+        end
 
       # TODO: We likely want to move this rescue into the Dalli::Compressor / Dalli::GzipCompressor
       # itself, since not all compressors necessarily use Zlib.  For now keep it here, so the behavior
@@ -48,6 +62,19 @@ module Dalli
         raise UnmarshalError, "Unable to uncompress value: #{$ERROR_INFO.message}"
       end
 
+      def compressor_accepts_max_bytes?
+        return @compressor_accepts_max_bytes if defined?(@compressor_accepts_max_bytes)
+
+        @compressor_accepts_max_bytes =
+          begin
+            compressor.method(:decompress).parameters.any? do |type, name|
+              name == :max_bytes && KEYWORD_PARAMETER_TYPES.include?(type)
+            end
+          rescue NameError # an object without a reflectable decompress method
+            false
+          end
+      end
+
       def compress_by_default?
         @compression_options[:compress]
       end

data/lib/dalli/protocol/value_serializer.rb

@@ -102,7 +102,7 @@ module Dalli
         Dalli.logger.warn 'SECURITY WARNING: Dalli is using Marshal for serialization. ' \
                           'Marshal can execute arbitrary code during deserialization. ' \
                           'If your memcached server could be compromised, consider using ' \
-                          'a safer serializer like JSON: Dalli::Client.new(servers, serializer: JSON)'
+                          'a safer serializer: Dalli::Client.new(servers, serializer: Dalli::JSONSerializer)'
         @@marshal_warning_logged = true # rubocop:disable Style/ClassVars
       end
     end

data/lib/dalli/version.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module Dalli
-  VERSION = '4.3.5'
+  VERSION = '4.3.6'
 
   MIN_SUPPORTED_MEMCACHED_VERSION = '1.4'
 end

data/lib/dalli.rb

@@ -62,6 +62,7 @@ require_relative 'dalli/version'
 require_relative 'dalli/instrumentation'
 
 require_relative 'dalli/compressor'
+require_relative 'dalli/json_serializer'
 require_relative 'dalli/protocol_deprecations'
 require_relative 'dalli/client'
 require_relative 'dalli/key_manager'

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: dalli
 version: !ruby/object:Gem::Version
-  version: 4.3.5
+  version: 4.3.6
 platform: ruby
 authors:
 - Peter M. Goldstein
@@ -41,6 +41,7 @@ files:
 - lib/dalli/client.rb
 - lib/dalli/compressor.rb
 - lib/dalli/instrumentation.rb
+- lib/dalli/json_serializer.rb
 - lib/dalli/key_manager.rb
 - lib/dalli/options.rb
 - lib/dalli/pid_cache.rb