dalli 3.2.11 → 3.2.13

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 17d814f4043cf5882166c1519dd436fb4a7d0caef32c86789dd5da3c5e51e52c
-  data.tar.gz: 27ba5ff384558a7ab08c7dcb3e6f2cf43af4c04dddc1ce2e5ca4ab1d35b01001
+  metadata.gz: 7f7bcbe37e631c3c733d1487e519501497702b873e81e6588d6dfad1ab96a65b
+  data.tar.gz: d7a953d2c0052334d8edb7fa45306724ffa25cec64436dcceccc9847d057e8bd
 SHA512:
-  metadata.gz: 63914cbb89b5a5d517dfd11f72b6bd56b22c9cb0700d9d53a037e12c7d9872150038f5a50b20f959bda47160644a30a433720e99da54ad048b15fee03ddb9baf
-  data.tar.gz: 11ede1a2e918d269cd084f171c97e5bad156b96fe0384e366cd531cc68a543bd1061b34ac437a7c2794e27ad6f63a3661e035c01bb4e8172d6753cfbca839fd1
+  metadata.gz: 8d7b0ed94102d333f611736c48d944b0f1010760ae646003db60cd02c90b1a047a5fd10f7a5ef077527a88594f8844cbe08f9647259749e78ade74d09d2e851b
+  data.tar.gz: bf86e4604fc18eaf2bdf57d6abad82258745238112a9866e5da9e05c0f45b44d62f82ebe8b5519ecc83d00ec6ee94e2d11bb931aa14171081a0b731745633e6d

data/CHANGELOG.md

@@ -4,6 +4,52 @@ Dalli Changelog
 Unreleased
 ==========
 
+3.2.13
+==========
+
+- Complete the fix for per-request `raw: true` on reads (GHSA-wr87-m4jw-29x5)
+  - `cas` and `cas!` accepted `raw: true` but didn't pass it to the read, so the value was still deserialized. They now honor it, in both protocols
+  - Affects 3.2.12 and earlier; fixed in 5.2.2, 5.1.4, 5.0.10, 4.3.7 and 3.2.13
+- Complete the fork-safety fix (GHSA-w39f-xq2m-4g8x)
+  - Fork detection used a pid cache refreshed by a `Process._fork` hook. When another library's fork hook ran first in the child (connection_pool, loaded before Dalli as in Rails, closes its connections there), Dalli didn't know it was in a forked child and ended the parent's TLS session. It now checks `Process.pid`
+  - Fixed in 4.3.7 and 3.2.13
+
+- Fix a regression in 3.2.12: a `get_multi` that didn't finish within `socket_timeout` counted toward `socket_max_failures`, so two slow `get_multi` calls in a row marked a healthy server down. It now just closes the connection
+- Handle malformed pipelined replies from a broken or hostile server or proxy: a hit with no key, or with no `s` flag, could leave the connection out of step or raise an error
+- With `decompressed_max_bytes` in effect, data after the end of a compressed value's stream was returned as part of the value. It's now ignored, as without the limit
+- With a `digest_class` whose digests aren't short hex strings, shortening a long key could loop forever. It now raises `ArgumentError`
+
+- Run the Tests and RuboCop workflows on pushes to `main` and the `*-stable` branches only, so a pull request's branch isn't tested twice (backport of #1198)
+
+3.2.12
+==========
+
+- Fix pipelined `get_multi` returning one key's value for another after an error reply (GHSA-p6pm-ch9v-44vx)
+  - The pipelined reply parser took any reply without a body for the end of the batch. An error reply for one key (`CLIENT_ERROR`, `SERVER_ERROR`, or `EN` from a proxy) ended it early, and the replies still on the connection were read as the replies to later commands, so a `get` could return another key's value
+  - Keys are now measured in bytes as sent (after base64 encoding when the meta protocol needs it). A key under 250 characters but over 250 bytes, for example one with many non-ASCII characters, passed validation:
+    - with the meta protocol, memcached rejected it with the error reply above
+    - with the binary protocol, memcached dropped the connection, and `get` or `get_multi` with that key retried forever
+  - Such keys are now truncated like other long keys; keys that worked before are unchanged
+  - The reply parser affects `protocol: :meta` since 3.2.0; the key length affects both protocols; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
+- Stop retrying an unresponsive server forever (GHSA-4qp6-2jcr-596v)
+  - A server that accepted connections but never answered (or dropped the connection on a request) was retried forever, hanging the caller, because each successful reconnect reset the failure count. Requests now fail after `socket_max_failures` attempts and the server is marked down, as when it can't be reached at all; it gets a full set of attempts again after `down_retry_delay`
+  - Affects all versions; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
+- Limit decompressed and reply value sizes (GHSA-3553-vcg5-72jw)
+  - Values flagged as compressed were inflated without limit, so a small stored value could expand to gigabytes on read, whatever the client's `compress` or `serializer` settings. The new `decompressed_max_bytes` option (default 128 MiB; `nil` disables it) makes a read that would pass it raise `Dalli::UnmarshalError`. Custom compressors whose `decompress` takes only the data keep working, without the limit
+  - The value size in a reply (a meta `VA` size or a binary body length) was used to read or buffer that many bytes, so a malicious server could make the client allocate gigabytes. Sizes over 1 GiB, memcached's largest item, now raise `Dalli::DalliError` before reading
+  - Affects all versions; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
+- Honor per-request `raw: true` on reads, and add `Dalli::JSONSerializer` (GHSA-wr87-m4jw-29x5)
+  - `get`, `gat` and `fetch` ignored a per-request `raw: true` and deserialized the value according to its stored flags, so a caller who asked for raw bytes could still have `Marshal.load` run on data someone else wrote to memcached. They now return the stored bytes
+  - The `raw` part affects all versions
+  - `serializer: JSON` reads values with `JSON.load`, which on json gem versions before 3.0 creates an object of the class named in a stored `json_class` key when the json additions are loaded. The new `Dalli::JSONSerializer` reads with `JSON.parse` and only returns plain JSON types. The README now recommends it; `serializer: JSON` itself is unchanged
+  - The `serializer: JSON` part affects any version used with the json gem before 3.0
+- Keep a forked child from closing the parent's TLS session (GHSA-w39f-xq2m-4g8x)
+  - With TLS, a forked child that closed the client (or reconnected after detecting the fork) sent a TLS close on the connection the parent was still using, ending the parent's session. A forked child now closes only its own copy of the socket
+  - Affects all versions with TLS; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
+
+- Fix three flaky tests: the `KeyManager` namespace key-length test, the Rack session freshness test, and `MemcachedManager`'s start and stop handling (backport of #1141, #1183)
+- Fix flaky failover tests: move their ports out of Linux's ephemeral port range, and wait for memcached to accept connections after starting it (backport of #1184)
+
 3.2.11
 ==========
 

data/README.md

@@ -25,6 +25,18 @@ The name is a variant of Salvador Dali for his famous painting [The Persistence
 * [Forum](https://github.com/petergoldstein/dalli/discussions/categories/q-a) - If you have questions about Dalli, please post them here.
 * [Client API](https://www.rubydoc.info/gems/dalli) - Ruby documentation for the `Dalli::Client` API
 
+## Security Note
+
+By default, Dalli uses Ruby's Marshal for serialization. Deserializing untrusted data with Marshal can lead to remote code execution. If you cache user-controlled data, or your memcached server could be written to by someone else, consider using a safer serializer:
+
+```ruby
+Dalli::Client.new('localhost:11211', serializer: Dalli::JSONSerializer)
+```
+
+`Dalli::JSONSerializer` reads values with `JSON.parse`, so it only ever returns plain hashes, arrays, strings, numbers, booleans and `nil`. Passing the `JSON` module itself (`serializer: JSON`) reads values with `JSON.load`, which can create objects of other classes from a stored `json_class` key and is not recommended for data an attacker might write.
+
+Per-request `raw: true` returns the stored bytes without deserializing or decompressing them, on every read method that accepts it (`get` and `fetch`).
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. You can run `bin/console` for an interactive prompt that will allow you to experiment.

data/lib/dalli/client.rb

@@ -388,7 +388,8 @@ module Dalli
     end
 
     def cas_core(key, always_set, ttl = nil, req_options = nil)
-      (value, cas) = perform(:cas, key)
+      # req_options reaches the read too, so raw: true returns the stored bytes
+      (value, cas) = perform(:cas, key, req_options)
       return if value.nil? && !always_set
 
       newvalue = yield(value)

data/lib/dalli/compressor.rb

@@ -13,8 +13,54 @@ module Dalli
       Zlib::Deflate.deflate(data)
     end
 
-    def self.decompress(data)
-      Zlib::Inflate.inflate(data)
+    # DEFLATE can't expand data by more than about 1032:1
+    MAX_EXPANSION = 1032
+    # Compressed input is fed to zlib this many bytes at a time. MRI's zlib
+    # yields the output in small chunks; one that returns it instead still
+    # produces at most MAX_EXPANSION times this per step.
+    INFLATE_SLICE_BYTES = 64 * 1024
+
+    # max_bytes caps the decompressed size; inflating stops and raises as soon
+    # as the output passes it, so a small compressed value can't expand into
+    # gigabytes of memory.
+    def self.decompress(data, max_bytes: nil)
+      return Zlib::Inflate.inflate(data) unless may_exceed?(data, max_bytes)
+
+      inflate_within_limit(Zlib::Inflate.new, data, max_bytes)
+    end
+
+    # False when even the largest possible expansion stays within max_bytes,
+    # so the value can be inflated in one step
+    def self.may_exceed?(data, max_bytes)
+      max_bytes ? data.bytesize * MAX_EXPANSION > max_bytes : false
+    end
+
+    def self.inflate_within_limit(inflater, data, max_bytes)
+      out = String.new(encoding: Encoding::BINARY)
+      0.step(data.bytesize - 1, INFLATE_SLICE_BYTES) do |pos|
+        inflate_slice(inflater, data.byteslice(pos, INFLATE_SLICE_BYTES), out, max_bytes)
+        # Anything after the end of the stream isn't part of the value, as
+        # with Zlib::Inflate.inflate; finish would return it as output
+        break if inflater.finished?
+      end
+      return out if inflater.finished?
+
+      append_within_limit(out, inflater.finish, max_bytes)
+    ensure
+      inflater.close
+    end
+
+    # Both yielded and returned output count toward the limit
+    def self.inflate_slice(inflater, slice, out, max_bytes)
+      rest = inflater.inflate(slice) { |chunk| append_within_limit(out, chunk, max_bytes) }
+      append_within_limit(out, rest, max_bytes)
+    end
+
+    def self.append_within_limit(out, chunk, max_bytes)
+      out << chunk if chunk
+      return out if out.bytesize <= max_bytes
+
+      raise Dalli::UnmarshalError, "Decompressed value exceeds #{max_bytes} bytes (decompressed_max_bytes)"
     end
   end
 
@@ -32,9 +78,11 @@ module Dalli
       io.string
     end
 
-    def self.decompress(data)
-      io = StringIO.new(data, 'rb')
-      Zlib::GzipReader.new(io).read
+    def self.decompress(data, max_bytes: nil)
+      return Zlib::GzipReader.new(StringIO.new(data, 'rb')).read unless Compressor.may_exceed?(data, max_bytes)
+
+      # MAX_WBITS + 16 makes zlib read the gzip format (and check its CRC)
+      Compressor.inflate_within_limit(Zlib::Inflate.new(Zlib::MAX_WBITS + 16), data, max_bytes)
     end
   end
 end

data/lib/dalli/json_serializer.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+require 'json'
+
+module Dalli
+  ##
+  # A serializer that stores values as JSON and never creates Ruby objects
+  # from what it reads.
+  #
+  # Passing the JSON module itself as the serializer (serializer: JSON) reads
+  # values back with JSON.load, which on json gem versions before 3.0 honors
+  # "json_class" and instantiates other classes when the json gem's additions
+  # are loaded. This serializer reads with JSON.parse, so a value written to
+  # memcached by someone else can only come back as plain hashes, arrays,
+  # strings, numbers, booleans and nil.
+  #
+  #   Dalli::Client.new(servers, serializer: Dalli::JSONSerializer)
+  ##
+  module JSONSerializer
+    def self.dump(value)
+      JSON.generate(value)
+    end
+
+    def self.load(data)
+      JSON.parse(data)
+    end
+  end
+end

data/lib/dalli/key_manager.rb

@@ -34,6 +34,9 @@ module Dalli
       validate_digest_class_option(@key_options)
 
       @namespace = namespace_from_options
+      # The meta protocol base64-encodes keys with whitespace or non-ASCII
+      # characters, so their length on the wire differs from their length here
+      @meta_protocol = client_options[:protocol]&.to_s == 'meta'
     end
 
     ##
@@ -51,7 +54,7 @@ module Dalli
       raise ArgumentError, 'key cannot be blank' unless key&.length&.positive?
 
       key = key_with_namespace(key)
-      key.length > MAX_KEY_LENGTH ? truncated_key(key) : key
+      wire_length(key) > MAX_KEY_LENGTH ? truncated_key(key) : key
     end
 
     ##
@@ -107,7 +110,30 @@ module Dalli
     ##
     def truncated_key(key)
       digest = digest_class.hexdigest(key)
-      "#{key[0, prefix_length(digest)]}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
+      prefix = key[0, [prefix_length(digest), 0].max]
+      truncated = "#{prefix}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
+      # A prefix with whitespace or non-ASCII characters sends the whole key
+      # base64-encoded, which can still be too long on the wire; shorten it.
+      while wire_length(truncated) > MAX_KEY_LENGTH && !prefix.empty?
+        prefix = prefix[0...-1]
+        truncated = "#{prefix}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
+      end
+      return truncated if wire_length(truncated) <= MAX_KEY_LENGTH
+
+      # Only possible with a digest_class whose digests aren't short hex strings
+      raise ArgumentError, "#{digest_class} digests are too long to fit in a truncated key"
+    end
+
+    # memcached's key limit applies to the key as sent: in bytes, and, with
+    # the meta protocol, after base64 encoding when the key needs it. A key
+    # within the limit in characters can still be too long. With the meta
+    # protocol memcached rejects it with a CLIENT_ERROR that must never reach
+    # a pipelined get; with the binary protocol it drops the connection.
+    def wire_length(key)
+      return key.bytesize unless @meta_protocol
+
+      encoded, = Dalli::Protocol::Meta::KeyRegularizer.encode(key)
+      encoded.bytesize
     end
 
     def prefix_length(digest)

data/lib/dalli/protocol/base.rb

@@ -27,6 +27,12 @@ module Dalli
         @connection_manager = ConnectionManager.new(hostname, port, socket_type, @options)
       end
 
+      # True when this request passed raw: true. A stored value is then
+      # returned as stored, never deserialized or decompressed.
+      def raw_request?(options)
+        options.is_a?(Hash) && options[:raw] ? true : false
+      end
+
       # Chokepoint method for error handling and ensuring liveness
       def request(opkey, *args)
         verify_state(opkey)
@@ -118,13 +124,16 @@ module Dalli
       def pipeline_abort
         response_buffer.clear
         @connection_manager.abort_request!
-        return true unless connected?
-
-        # Closes the connection, which ensures that our connection
-        # is in a clean state for future requests
-        @connection_manager.error_on_request!('External timeout')
-      rescue NetworkError
-        true
+        return unless connected?
+
+        # Closes the connection, which ensures that our connection is in a
+        # clean state for future requests. The get_multi ran out of time
+        # overall rather than hitting a socket error, so it doesn't count
+        # toward socket_max_failures: two slow get_multis in a row would
+        # otherwise mark a healthy server down.
+        Dalli.logger.warn { "#{name} didn't finish a get_multi within socket_timeout; closing the connection" }
+        @connection_manager.close
+        nil
       end
 
       # Did the last call to #pipeline_response_setup complete successfully?

data/lib/dalli/protocol/binary/response_processor.rb

@@ -66,7 +66,9 @@ module Dalli
           raise Dalli::DalliError, "Response error #{resp_header.status}: #{RESPONSE_CODES[resp_header.status]}"
         end
 
-        def get(cache_nils: false)
+        # raw: true returns the value as stored, without deserializing or
+        # decompressing it, whatever flags it was stored with
+        def get(cache_nils: false, raw: false)
           resp_header, body = read_response
 
           return false if resp_header.not_stored? # Not stored, normal status for add operation
@@ -75,7 +77,7 @@ module Dalli
           raise_on_not_ok!(resp_header)
           return true unless body
 
-          unpack_response_body(resp_header, body, true).last
+          unpack_response_body(resp_header, body, !raw).last
         end
 
         ##
@@ -100,7 +102,8 @@ module Dalli
           true
         end
 
-        def data_cas_response
+        # raw: true returns the value as stored; see #get
+        def data_cas_response(raw: false)
           resp_header, body = read_response
           return [nil, resp_header.cas] if resp_header.not_found?
           return [nil, false] if resp_header.not_stored?
@@ -108,7 +111,7 @@ module Dalli
           raise_on_not_ok!(resp_header)
           return [nil, resp_header.cas] unless body
 
-          [unpack_response_body(resp_header, body, true).last, resp_header.cas]
+          [unpack_response_body(resp_header, body, !raw).last, resp_header.cas]
         end
 
         # Returns the new value for the key, if found and updated
@@ -209,6 +212,17 @@ module Dalli
         # The remaining three values in the array are the ResponseHeader,
         # key, and value.
         ##
+        # memcached can't store an item larger than 1 GiB (its -I maximum). A
+        # pipelined reply claiming an impossible size would otherwise have the
+        # buffer wait for (and accumulate) that many bytes.
+        MAX_BODY_BYTES = 1024 * 1024 * 1024
+
+        def check_body_size!(size)
+          return if size <= MAX_BODY_BYTES
+
+          raise Dalli::DalliError, "Reply body size #{size} is out of range"
+        end
+
         def getk_response_from_buffer(buf)
           # There's no header in the buffer, so don't advance
           return [0, nil, nil, nil, nil] unless contains_header?(buf)
@@ -222,6 +236,8 @@ module Dalli
           # error response that needs to be discarded.
           return [ResponseHeader::SIZE, resp_header.ok?, resp_header.cas, nil, nil] if body_len.zero?
 
+          check_body_size!(body_len)
+
           resp_size = ResponseHeader::SIZE + body_len
           # The header is in the buffer, but the body is not.  As we don't have
           # a complete response, don't advance the buffer

data/lib/dalli/protocol/binary.rb

@@ -22,7 +22,7 @@ module Dalli
       def get(key, options = nil)
         req = RequestFormatter.standard_request(opkey: :get, key: key)
         write(req)
-        response_processor.get(cache_nils: cache_nils?(options))
+        response_processor.get(cache_nils: cache_nils?(options), raw: raw_request?(options))
       end
 
       def quiet_get_request(key)
@@ -33,7 +33,7 @@ module Dalli
         ttl = TtlSanitizer.sanitize(ttl)
         req = RequestFormatter.standard_request(opkey: :gat, key: key, ttl: ttl)
         write(req)
-        response_processor.get(cache_nils: cache_nils?(options))
+        response_processor.get(cache_nils: cache_nils?(options), raw: raw_request?(options))
       end
 
       def touch(key, ttl)
@@ -44,10 +44,10 @@ module Dalli
 
       # TODO: This is confusing, as there's a cas command in memcached
       # and this isn't it.  Maybe rename?  Maybe eliminate?
-      def cas(key)
+      def cas(key, options = nil)
         req = RequestFormatter.standard_request(opkey: :get, key: key)
         write(req)
-        response_processor.data_cas_response
+        response_processor.data_cas_response(raw: raw_request?(options))
       end
 
       # Storage Commands

data/lib/dalli/protocol/connection_manager.rb

@@ -38,6 +38,7 @@ module Dalli
         @sock = nil
         @pid = nil
 
+        @fail_count = 0
         reset_down_info
       end
 
@@ -53,7 +54,7 @@ module Dalli
         Dalli.logger.debug { "Dalli::Server#connect #{name}" }
 
         @sock = memcached_socket
-        @pid = PIDCache.pid
+        @pid = Process.pid
         @request_in_progress = false
       rescue SystemCallError, *TIMEOUT_ERRORS, EOFError, SocketError => e
         # SocketError = DNS resolution failure
@@ -84,6 +85,8 @@ module Dalli
       def down!
         close
         log_down_detected
+        # Once down_retry_delay passes, the server gets a full set of attempts
+        @fail_count = 0
 
         @error = $ERROR_INFO&.class&.name
         @msg ||= $ERROR_INFO&.message
@@ -113,7 +116,7 @@ module Dalli
         return unless @sock
 
         begin
-          @sock.close
+          close_socket
         rescue StandardError
           nil
         end
@@ -122,6 +125,13 @@ module Dalli
         abort_request!
       end
 
+      # A forked child shares the parent's connection, so it closes only its
+      # own file descriptor. Closing the TLS socket itself would send
+      # close_notify and end the parent's TLS session.
+      def close_socket
+        fork_detected? ? @sock.to_io.close : @sock.close
+      end
+
       def connected?
         !@sock.nil?
       end
@@ -140,6 +150,11 @@ module Dalli
         raise '[Dalli] No request in progress. This may be a bug in Dalli.' unless @request_in_progress
 
         @request_in_progress = false
+        # A completed request is what proves the server healthy again, so the
+        # failure count resets here rather than on reconnect: a server that
+        # accepts connections but never answers would otherwise reset it on
+        # every retry and never reach socket_max_failures.
+        @fail_count = 0
       end
 
       def abort_request!
@@ -154,7 +169,19 @@ module Dalli
         error_on_request!(e)
       end
 
+      # memcached can't store an item larger than 1 GiB (its -I maximum), so a
+      # reply claiming more (or a negative size) is malformed or hostile.
+      # Reading allocates the full count up front, so check before reading.
+      MAX_READ_BYTES = (1024 * 1024 * 1024) + 2 # plus the value's trailing "\r\n"
+
+      def check_read_size!(count)
+        return if count.between?(0, MAX_READ_BYTES)
+
+        raise Dalli::DalliError, "Reply size #{count} from #{name} is out of range"
+      end
+
       def read(count)
+        check_read_size!(count)
         @sock.readfull(count)
       rescue SystemCallError, *TIMEOUT_ERRORS, EOFError => e
         error_on_request!(e)
@@ -195,8 +222,9 @@ module Dalli
         raise Dalli::NetworkError, message
       end
 
+      # Called on connect. Deliberately leaves @fail_count alone; see
+      # finish_request!.
       def reset_down_info
-        @fail_count = 0
         @down_at = nil
         @last_down_at = nil
         @msg = nil
@@ -222,14 +250,18 @@ module Dalli
       def close_on_fork
         message = 'Fork detected, re-connecting child process...'
         Dalli.logger.info { message }
-        # Close socket on a fork, setting us up for reconnect
-        # on next request.
+        # Closes the inherited socket without touching the parent's
+        # connection, setting us up for reconnect on next request.
         close
         raise Dalli::NetworkError, message
       end
 
       def fork_detected?
-        @pid && @pid != PIDCache.pid
+        # Process.pid rather than PIDCache: PIDCache is refreshed by a
+        # Process._fork hook, and another library's fork hook (such as
+        # connection_pool closing its connections) can run in the child before
+        # it, which would close the parent's TLS session as if it were ours.
+        @pid && @pid != Process.pid
       end
 
       def log_down_detected

data/lib/dalli/protocol/meta/response_processor.rb

@@ -27,22 +27,27 @@ module Dalli
           @value_marshaller = value_marshaller
         end
 
-        def meta_get_with_value(cache_nils: false)
+        # raw: true returns the value as stored, without deserializing or
+        # decompressing it, whatever flags it was stored with
+        def meta_get_with_value(cache_nils: false, raw: false)
           tokens = error_on_unexpected!([VA, EN, HD])
           return cache_nils ? ::Dalli::NOT_FOUND : nil if tokens.first == EN
           return true unless tokens.first == VA
 
-          @value_marshaller.retrieve(read_line, bitflags_from_tokens(tokens))
+          value = read_line
+          raw ? value : @value_marshaller.retrieve(value, bitflags_from_tokens(tokens))
         end
 
-        def meta_get_with_value_and_cas
+        # raw: true returns the value as stored; see #meta_get_with_value
+        def meta_get_with_value_and_cas(raw: false)
           tokens = error_on_unexpected!([VA, EN, HD])
           return [nil, 0] if tokens.first == EN
 
           cas = cas_from_tokens(tokens)
           return [nil, cas] unless tokens.first == VA
 
-          [@value_marshaller.retrieve(read_line, bitflags_from_tokens(tokens)), cas]
+          value = read_line
+          [raw ? value : @value_marshaller.retrieve(value, bitflags_from_tokens(tokens)), cas]
         end
 
         def meta_get_without_value
@@ -120,6 +125,10 @@ module Dalli
         end
 
         def full_response_from_buffer(tokens, body, resp_size)
+          # A hit with no key can't be matched to a request: skip it (false
+          # status) rather than yield it, or take it for the end of the pipeline
+          return [resp_size, false, nil, nil, nil] unless tokens.any? { |t| t.start_with?('k') }
+
           value = @value_marshaller.retrieve(body, bitflags_from_tokens(tokens))
           [resp_size, tokens.first == VA, cas_from_tokens(tokens), key_from_tokens(tokens), value]
         end
@@ -140,13 +149,14 @@ module Dalli
 
           tokens, header_len, body_len = tokens_from_header_buffer(buf)
 
-          # We have a complete response that has no body.
-          # This is either the response to the terminating
-          # noop or, if the status is not MN, an intermediate
-          # error response that needs to be discarded. A hit
-          # on an empty value (VA 0) still has a body -- just
-          # its terminator -- so it's parsed below as a value.
-          return [header_len, true, nil, nil, nil] if no_body?(tokens, body_len)
+          # A complete response with no body. Only the terminating noop's MN
+          # ends the pipeline. Any other bodyless reply (CLIENT_ERROR,
+          # SERVER_ERROR, ...) answers one key's request: report it with a
+          # false status so the caller skips it and keeps reading. Treating it
+          # as the end would leave the remaining replies on the connection for
+          # later commands to read as their own. A hit on an empty value (VA 0)
+          # still has a body -- just its terminator -- so it's parsed below.
+          return [header_len, tokens.first == MN, nil, nil, nil] if no_body?(tokens, body_len)
 
           resp_size = header_len + body_len + TERMINATOR.length
           # The header is in the buffer, but the body is not.  As we don't have
@@ -159,6 +169,22 @@ module Dalli
           full_response_from_buffer(tokens, body, resp_size)
         end
 
+        # memcached can't store an item larger than 1 GiB (its -I maximum). A
+        # pipelined reply claiming an impossible size would otherwise have the
+        # buffer wait for (and accumulate) that many bytes.
+        MAX_VALUE_BYTES = 1024 * 1024 * 1024
+
+        # The size given right after "VA", or 0 when it isn't a number
+        def positional_size(tokens)
+          tokens.first == VA && tokens[1]&.match?(/\A\d+\z/) ? tokens[1].to_i : 0
+        end
+
+        def check_value_size!(size)
+          return if size.between?(0, MAX_VALUE_BYTES)
+
+          raise Dalli::DalliError, "Reply value size #{size} is out of range"
+        end
+
         # A zero-size reply has no body unless it's a VA (a hit on an empty value)
         def no_body?(tokens, body_len)
           body_len.zero? && tokens.first != VA
@@ -194,7 +220,11 @@ module Dalli
         end
 
         def body_len_from_tokens(tokens)
-          value_from_tokens(tokens, 's')&.to_i
+          # memcached also gives the size positionally ("VA <size> ..."); use it
+          # when there's no s flag rather than reading the value as replies
+          size = tokens.any? { |t| t.start_with?('s') } ? value_from_tokens(tokens, 's')&.to_i : positional_size(tokens)
+          check_value_size!(size) if size
+          size
         end
 
         def value_from_tokens(tokens, flag)

data/lib/dalli/protocol/meta.rb

@@ -27,7 +27,7 @@ module Dalli
         encoded_key, base64 = KeyRegularizer.encode(key)
         req = RequestFormatter.meta_get(key: encoded_key, base64: base64)
         write(req)
-        response_processor.meta_get_with_value(cache_nils: cache_nils?(options))
+        response_processor.meta_get_with_value(cache_nils: cache_nils?(options), raw: raw_request?(options))
       end
 
       def quiet_get_request(key)
@@ -40,7 +40,7 @@ module Dalli
         encoded_key, base64 = KeyRegularizer.encode(key)
         req = RequestFormatter.meta_get(key: encoded_key, ttl: ttl, base64: base64)
         write(req)
-        response_processor.meta_get_with_value(cache_nils: cache_nils?(options))
+        response_processor.meta_get_with_value(cache_nils: cache_nils?(options), raw: raw_request?(options))
       end
 
       def touch(key, ttl)
@@ -53,11 +53,11 @@ module Dalli
 
       # TODO: This is confusing, as there's a cas command in memcached
       # and this isn't it.  Maybe rename?  Maybe eliminate?
-      def cas(key)
+      def cas(key, options = nil)
         encoded_key, base64 = KeyRegularizer.encode(key)
         req = RequestFormatter.meta_get(key: encoded_key, value: true, return_cas: true, base64: base64)
         write(req)
-        response_processor.meta_get_with_value_and_cas
+        response_processor.meta_get_with_value_and_cas(raw: raw_request?(options))
       end
 
       # Storage Commands

data/lib/dalli/protocol/value_compressor.rb

@@ -15,11 +15,17 @@ module Dalli
         compress: true,
         compressor: ::Dalli::Compressor,
         # min byte size to attempt compression
-        compression_min_size: 4 * 1024 # 4K
+        compression_min_size: 4 * 1024, # 4K
+        # max size a stored value may decompress to (nil: no limit). Guards
+        # against a small compressed value expanding into gigabytes on read.
+        decompressed_max_bytes: 128 * 1024 * 1024 # 128 MiB
       }.freeze
 
       OPTIONS = DEFAULTS.keys.freeze
 
+      KEYWORD_PARAMETER_TYPES = %i[key keyreq].freeze
+      private_constant :KEYWORD_PARAMETER_TYPES
+
       # https://www.hjp.at/zettel/m/memcached_flags.rxml
       # Looks like most clients use bit 1 to indicate gzip compression.
       FLAG_COMPRESSED = 0x2
@@ -47,8 +53,16 @@ module Dalli
       end
 
       def retrieve(value, bitflags)
-        compressed = (bitflags & FLAG_COMPRESSED) != 0
-        compressed ? compressor.decompress(value) : value
+        return value unless bitflags.anybits?(FLAG_COMPRESSED)
+
+        max_bytes = @compression_options[:decompressed_max_bytes]
+        # Custom compressors that only define decompress(data) keep working,
+        # without the limit
+        if max_bytes && compressor_accepts_max_bytes?
+          compressor.decompress(value, max_bytes: max_bytes)
+        else
+          compressor.decompress(value)
+        end
 
       # TODO: We likely want to move this rescue into the Dalli::Compressor / Dalli::GzipCompressor
       # itself, since not all compressors necessarily use Zlib.  For now keep it here, so the behavior
@@ -57,6 +71,19 @@ module Dalli
         raise UnmarshalError, "Unable to uncompress value: #{$ERROR_INFO.message}"
       end
 
+      def compressor_accepts_max_bytes?
+        return @compressor_accepts_max_bytes if defined?(@compressor_accepts_max_bytes)
+
+        @compressor_accepts_max_bytes =
+          begin
+            compressor.method(:decompress).parameters.any? do |type, name|
+              name == :max_bytes && KEYWORD_PARAMETER_TYPES.include?(type)
+            end
+          rescue NameError # an object without a reflectable decompress method
+            false
+          end
+      end
+
       def compress_by_default?
         @compression_options[:compress]
       end

data/lib/dalli/version.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module Dalli
-  VERSION = '3.2.11'
+  VERSION = '3.2.13'
 
   MIN_SUPPORTED_MEMCACHED_VERSION = '1.4'
 end

data/lib/dalli.rb

@@ -57,6 +57,7 @@ end
 require_relative 'dalli/version'
 
 require_relative 'dalli/compressor'
+require_relative 'dalli/json_serializer'
 require_relative 'dalli/client'
 require_relative 'dalli/key_manager'
 require_relative 'dalli/pipelined_getter'

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: dalli
 version: !ruby/object:Gem::Version
-  version: 3.2.11
+  version: 3.2.13
 platform: ruby
 authors:
 - Peter M. Goldstein
@@ -40,6 +40,7 @@ files:
 - lib/dalli/cas/client.rb
 - lib/dalli/client.rb
 - lib/dalli/compressor.rb
+- lib/dalli/json_serializer.rb
 - lib/dalli/key_manager.rb
 - lib/dalli/options.rb
 - lib/dalli/pid_cache.rb