dalli 4.3.4 → 4.3.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d296cbb71a7d6d7f57969a942e7270bd3b7145160c2ba4c971426dfbdb352368
-  data.tar.gz: 14929b5d79fa9d709dd7ccf5b633a44a91fdb2cbc7139eecaa1687e0e63751db
+  metadata.gz: 4b1a68fbdf9ea89024d6e9bc5803f94d3342c2f7b8bd7cfea3a97f7a873f7b76
+  data.tar.gz: 93d1c2fcb71fa739e6a00808cddb11567ad9fed3caa4d0751c924576bc04bfe0
 SHA512:
-  metadata.gz: 04d254a2ea7d37a29634aab3441634ec660fe430efcdf64e3ae547bd456f53a2d9a0c47eb17ae8b8a5407d0fdd6c811d8e4b681213f5089011b217465265388f
-  data.tar.gz: 55928535640b642c258efecfbede5a423638d0a45b1c814b7aaff03af67ae65feb5d344e9ab31fa963c25fcb318a25538c7e6e495dec1ae9e2b85c6bcfcc06aa
+  metadata.gz: bc42cde8b6f61ba60768bf43a9ebf8b1088dc8b2efbb4e32b5ed1abedd3c3be81c66aa02dc1a2558fd319debcc8074e61471996ad8825afed32d4fd741f8a312
+  data.tar.gz: 3a8320685a39edef196feaa387ef954caea41650b540c7f37d32d4821b5804f41631e5346e14b21912de0963f147c7d55e8c1ad08e24d2f7d25073975ba64d72

data/CHANGELOG.md

@@ -4,6 +4,100 @@ Dalli Changelog
 Unreleased
 ==========
 
+4.3.7
+==========
+
+Security:
+
+- Keep the caller's key when retrying a request (GHSA-m252-9cgf-vx2w)
+  - With a `namespace`, a request retried after a transient network error (a timeout, or a connection closed by memcached or a proxy) applied the namespace a second time, so a retried read could return a different key's value and a retried write could overwrite a different key
+  - Affects `get_with_metadata` and `fetch_with_lock` since 4.1.0 (single-key operations on 4.x aren't affected); fixed in 5.2.2, 5.1.4, 5.0.10 and 4.3.7. Clients without a namespace aren't affected
+- Complete the fix for per-request `raw: true` on reads (GHSA-wr87-m4jw-29x5)
+  - `cas` and `cas!` (both protocols) and `fetch_with_lock` (meta protocol) accepted `raw: true` but didn't pass it to the read, so the value was still deserialized. They now honor it
+  - A read made with `raw: true` doesn't ask for flags, but a reply carrying them anyway (from a proxy or a hostile server) still had its value deserialized. Meta protocol raw reads now ignore flags in the reply
+  - Affects 4.3.6 and earlier; fixed in 5.2.2, 5.1.4, 5.0.10, 4.3.7 and 3.2.13
+- Complete the fork-safety fix (GHSA-w39f-xq2m-4g8x)
+  - Fork detection used a pid cache refreshed by a `Process._fork` hook. When another library's fork hook ran first in the child (connection_pool, loaded before Dalli as in Rails, closes its connections there), Dalli didn't know it was in a forked child and ended the parent's TLS session. It now checks `Process.pid`
+  - 4.3.6's write buffer kept a reference to the caller's value string with the meta protocol, so changing the string before the buffer was sent (for example, reusing one buffer inside a `quiet` block) changed the bytes sent, losing values or adding commands. The buffer now holds a copy
+  - Fixed in 4.3.7 and 3.2.13
+
+Bug fixes:
+
+- `get_with_metadata` and `fetch_with_lock` retried the final error raised when a server is marked down, so with `down_retry_delay: 0` they retried an unresponsive server forever. They now retry only retryable errors, like other operations
+- Fix a regression in 4.3.6: a `get_multi` that didn't finish within `socket_timeout` counted toward `socket_max_failures`, so two slow `get_multi` calls in a row marked a healthy server down. It now just closes the connection
+- Fix a regression in 4.3.6: over TLS, every request buffered in a `quiet` block was sent as its own TLS record and system call (2.5 times slower for a block of 2000 deletes). Each flush is one write again
+- Handle malformed replies from a broken or hostile server or proxy: a negative value size, a hit with no key, or a hit with no `s` flag could leave the connection out of step or raise a non-Dalli error
+- With `decompressed_max_bytes` in effect, data after the end of a compressed value's stream was returned as part of the value. It's now ignored, as without the limit
+- With a `digest_class` whose digests aren't short hex strings, shortening a long key could loop forever. It now raises `ArgumentError`
+
+Development:
+
+- Run the Tests, RuboCop and Profiles workflows on pushes to `main` and the `*-stable` branches only, so a pull request's branch isn't tested twice (backport of #1198)
+
+4.3.6
+==========
+
+Security:
+
+- Fix pipelined `get_multi` returning one key's value for another after an error reply (GHSA-p6pm-ch9v-44vx)
+  - The pipelined reply parser took any reply without a body for the end of the batch. An error reply for one key (`CLIENT_ERROR`, `SERVER_ERROR`, or `EN` from a proxy) ended it early, and the replies still on the connection were read as the replies to later commands, so a `get` could return another key's value
+  - Keys are now measured in bytes as sent (after base64 encoding when the meta protocol needs it). A key under 250 characters but over 250 bytes, for example one with many non-ASCII characters, passed validation:
+    - with the meta protocol, memcached rejected it with the error reply above
+    - with the binary protocol, memcached dropped the connection, and `get` or `get_multi` with that key retried forever
+  - Such keys are now truncated like other long keys; keys that worked before are unchanged
+  - The reply parser affects `protocol: :meta` since 3.2.0; the key length affects both protocols; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
+- Stop retrying an unresponsive server forever (GHSA-4qp6-2jcr-596v)
+  - A server that accepted connections but never answered (or dropped the connection on a request) was retried forever, hanging the caller, because each successful reconnect reset the failure count. Requests now fail after `socket_max_failures` attempts and the server is marked down, as when it can't be reached at all; it gets a full set of attempts again after `down_retry_delay`
+  - Affects all versions; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
+- Limit decompressed and reply value sizes (GHSA-3553-vcg5-72jw)
+  - Values flagged as compressed were inflated without limit, so a small stored value could expand to gigabytes on read, whatever the client's `compress` or `serializer` settings. The new `decompressed_max_bytes` option (default 128 MiB; `nil` disables it) makes a read that would pass it raise `Dalli::UnmarshalError`. Custom compressors whose `decompress` takes only the data keep working, without the limit
+  - The value size in a reply (a meta `VA` size or a binary body length) was used to read or buffer that many bytes, so a malicious server could make the client allocate gigabytes. Sizes over 1 GiB, memcached's largest item, now raise `Dalli::DalliError` before reading
+  - Affects all versions; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
+- Honor per-request `raw: true` on reads, and add `Dalli::JSONSerializer` (GHSA-wr87-m4jw-29x5)
+  - With the binary protocol, `get`, `gat` and `fetch`, and with the meta protocol, `get_with_metadata`, ignored a per-request `raw: true` and deserialized the value according to its stored flags, so a caller who asked for raw bytes could still have `Marshal.load` run on data someone else wrote to memcached. They now return the stored bytes
+  - The `raw` part affects binary `get` and `gat` in all versions, and `get_with_metadata` since 4.2.0
+  - `serializer: JSON` reads values with `JSON.load`, which on json gem versions before 3.0 creates an object of the class named in a stored `json_class` key when the json additions are loaded. The new `Dalli::JSONSerializer` reads with `JSON.parse` and only returns plain JSON types. The README and the Marshal security warning now recommend it; `serializer: JSON` itself is unchanged
+  - The `serializer: JSON` part affects any version used with the json gem before 3.0
+- Keep a forked child from writing to or closing the parent's connection (GHSA-w39f-xq2m-4g8x)
+  - A child forked while the parent had requests buffered but not yet sent, such as quiet writes inside a `quiet` block, sent them again when it closed the client or reconnected: quiet writes ran twice, and the parent's later replies could be read as the replies to other requests, returning one key's value for another. Dalli now buffers requests itself and a forked child discards them
+  - Affects 4.2.0 and later; fixed in 5.2.1, 5.1.3, 5.0.9 and 4.3.6
+  - With TLS, a forked child that closed the client (or reconnected after detecting the fork) sent a TLS close on the connection the parent was still using, ending the parent's session. A forked child now closes only its own copy of the socket
+  - Affects all versions with TLS; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
+
+Bug fixes:
+
+- An exception raised by a `get_multi` block (such as `Timeout::Error`) was treated as a network failure: swallowed, with the whole `get_multi` retried and keys yielded twice. It now reaches the caller
+- Fix `set_multi` with the binary protocol
+  - Each quiet `setq` waited for a reply memcached never sends for a successful write, so `set_multi` timed out on its first key without storing anything (and, with retries, hung or marked the server down). Affects 4.2.0 and later
+
+Notes:
+
+- Document [Ruby bug 21195](https://bugs.ruby-lang.org/issues/21195), a VM crash (`[BUG] rb_sys_fail_path_in(io_fillbuf, ...) - errno == 0`) when a socket read using `IO#timeout` is interrupted by a signal, in the README (#1188)
+  - Affects Ruby 3.2.x, 3.3.0–3.3.7 and 3.4.0–3.4.2. Fixed in 3.3.8 and 3.4.3; Ruby 3.2 reached end of life without the fix
+  - Ruby 3.1 doesn't use `IO#timeout` and isn't affected
+
+Development:
+
+- Drop Ruby 3.2 from the CI matrix, since that bug intermittently crashed its test runs (#1188)
+- Fix three flaky tests: the `KeyManager` namespace key-length test, the Rack session freshness test, and `MemcachedManager`'s start and stop handling (backport of #1141, #1182)
+- Fix flaky failover tests: move their ports out of Linux's ephemeral port range, and wait for memcached to accept connections after starting it (backport of #1184)
+
+4.3.5
+==========
+
+Bug fixes:
+
+- Fix multi-server `get_multi` stopping at an empty value (#1170)
+  - The pipelined reply parser took a hit on a zero-length value (`VA 0`) for the terminating `MN`, so that server's remaining keys were silently missing from the result
+  - Its value terminator and any replies not yet parsed stayed on the connection and were read as the replies to later commands on it. Those commands could fail with `Dalli::DalliError: Response error`, or a single-key `get` could silently return another key's value
+  - An empty value is now returned as `''`, like a single-key `get` and single-server `get_multi`
+  - Affects only clients using `protocol: :meta`; the default binary protocol is not affected
+  - Thanks to Julian Richard Contreras for this contribution
+
+Development:
+
+- Fix offenses reported by RuboCop 1.91 and require `rubocop >= 1.91` (backport of #1162)
+
 4.3.4
 ==========
 

data/Gemfile

@@ -18,7 +18,7 @@ group :development, :test do
   gem 'rack', '~> 3'
   gem 'rack-session'
   gem 'rake', '~> 13.0'
-  gem 'rubocop'
+  gem 'rubocop', '>= 1.91' # disable-next directives require 1.91+
   gem 'rubocop-minitest'
   gem 'rubocop-performance'
   gem 'rubocop-rake'

data/README.md

@@ -20,6 +20,18 @@ The name is a variant of Salvador Dali for his famous painting [The Persistence
 * Ruby 3.1 or later
 * memcached 1.4 or later (1.6+ recommended for meta protocol support)
 
+### Known Ruby issue: crash on an interrupted socket read
+
+On Ruby 3.2 and later, Dalli sets socket timeouts with `IO#timeout`. Ruby 3.2.x, 3.3.0–3.3.7 and 3.4.0–3.4.2 have a bug ([Ruby #21195](https://bugs.ruby-lang.org/issues/21195)) where a timed read interrupted by a signal aborts the whole process:
+
+```
+[BUG] rb_sys_fail_path_in(io_fillbuf, fd:N ) - errno == 0
+```
+
+The crash is inside Ruby itself, so it can't be rescued. It was fixed in Ruby 3.3.8 and 3.4.3. Ruby 3.2 reached end of life without the fix. If you see this crash, upgrade to Ruby 3.3.8+ or 3.4.3+. Ruby 3.1 doesn't use `IO#timeout` and isn't affected.
+
+Ruby 3.2 is no longer part of this branch's CI matrix for the same reason.
+
 ## Protocol Options
 
 Dalli supports two protocols for communicating with memcached:
@@ -61,9 +73,13 @@ The separator must be a single non-alphanumeric character. Valid examples: `:`,
 By default, Dalli uses Ruby's Marshal for serialization. Deserializing untrusted data with Marshal can lead to remote code execution. If you cache user-controlled data, consider using a safer serializer:
 
 ```ruby
-Dalli::Client.new('localhost:11211', serializer: JSON)
+Dalli::Client.new('localhost:11211', serializer: Dalli::JSONSerializer)
 ```
 
+`Dalli::JSONSerializer` reads values with `JSON.parse`, so it only ever returns plain hashes, arrays, strings, numbers, booleans and `nil`. Passing the `JSON` module itself (`serializer: JSON`) reads values with `JSON.load`, which can create objects of other classes from a stored `json_class` key and is not recommended for data an attacker might write.
+
+Per-request `raw: true` returns the stored bytes without deserializing or decompressing them, on every read method that accepts it (`get`, `get_with_metadata` and `fetch`).
+
 See the [4.0-Upgrade.md](4.0-Upgrade.md) guide for more information.
 
 ## OpenTelemetry Tracing

data/lib/dalli/client.rb

@@ -3,7 +3,6 @@
 require 'digest/md5'
 require 'set'
 
-# encoding: ascii
 module Dalli
   ##
   # Dalli::Client is the main class which developers will use to interact with
@@ -135,14 +134,18 @@ module Dalli
     def get_with_metadata(key, options = {})
       raise_unless_meta_protocol!
 
-      key = key.to_s
-      key = @key_manager.validate_key(key)
+      # A retry re-runs this method with its original arguments, so `key` must
+      # stay as the caller gave it; reassigning it would add the namespace again.
+      validated_key = @key_manager.validate_key(key.to_s)
 
-      server = ring.server_for_key(key)
-      Instrumentation.trace('get_with_metadata', trace_attrs('get_with_metadata', key, server)) do
-        server.request(:meta_get, key, options)
+      server = ring.server_for_key(validated_key)
+      Instrumentation.trace('get_with_metadata', trace_attrs('get_with_metadata', validated_key, server)) do
+        server.request(:meta_get, validated_key, options)
       end
-    rescue NetworkError => e
+    # Only a retryable error is retried. The NetworkError raised when a server
+    # is marked down is final, as in #perform; retrying it looped forever once
+    # down_retry_delay let the same server be tried again.
+    rescue RetryableNetworkError => e
       Dalli.logger.debug { e.inspect }
       Dalli.logger.debug { 'retrying get_with_metadata with new server' }
       retry
@@ -152,7 +155,7 @@ module Dalli
     # Fetch multiple keys efficiently.
     # If a block is given, yields key/value pairs one at a time.
     # Otherwise returns a hash of { 'key' => 'value', 'key2' => 'value1' }
-    # rubocop:disable Style/ExplicitBlockArgument
+    # rubocop:disable-next Style/ExplicitBlockArgument
     def get_multi(*keys)
       keys.flatten!
       keys.compact!
@@ -164,7 +167,6 @@ module Dalli
         get_multi_hash(keys)
       end
     end
-    # rubocop:enable Style/ExplicitBlockArgument
 
     ##
     # Fetch multiple keys efficiently, including available metadata such as CAS.
@@ -240,14 +242,17 @@ module Dalli
       validate_integer!(:lock_ttl, lock_ttl)
       validate_integer!(:recache_threshold, recache_threshold)
 
-      key = key.to_s
-      key = @key_manager.validate_key(key)
+      # Left unchanged for the retry below; see get_with_metadata
+      validated_key = @key_manager.validate_key(key.to_s)
 
-      server = ring.server_for_key(key)
-      Instrumentation.trace('fetch_with_lock', trace_attrs('fetch_with_lock', key, server)) do
-        fetch_with_lock_request(key, ttl, lock_ttl, recache_threshold, req_options, &block)
+      server = ring.server_for_key(validated_key)
+      Instrumentation.trace('fetch_with_lock', trace_attrs('fetch_with_lock', validated_key, server)) do
+        fetch_with_lock_request(validated_key, ttl, lock_ttl, recache_threshold, req_options, &block)
       end
-    rescue NetworkError => e
+    # Only a retryable error is retried. The NetworkError raised when a server
+    # is marked down is final, as in #perform; retrying it looped forever once
+    # down_retry_delay let the same server be tried again.
+    rescue RetryableNetworkError => e
       Dalli.logger.debug { e.inspect }
       Dalli.logger.debug { 'retrying fetch_with_lock with new server' }
       retry
@@ -594,7 +599,8 @@ module Dalli
     end
 
     def cas_core(key, always_set, ttl = nil, req_options = nil)
-      (value, cas) = perform(:cas, key)
+      # req_options reaches the read too, so raw: true returns the stored bytes
+      (value, cas) = perform(:cas, key, req_options)
       return if value.nil? && !always_set
 
       newvalue = yield(value)
@@ -603,7 +609,12 @@ module Dalli
 
     def fetch_with_lock_request(key, ttl, lock_ttl, recache_threshold, req_options)
       server = ring.server_for_key(key)
-      result = server.request(:meta_get, key, { vivify_ttl: lock_ttl, recache_ttl: recache_threshold })
+      # req_options (e.g. raw: true) reaches the read too. It's the base, not
+      # the override: fetch_with_lock's own lock_ttl/recache_threshold win.
+      meta_options = req_options.is_a?(Hash) ? req_options.dup : {}
+      meta_options[:vivify_ttl] = lock_ttl
+      meta_options[:recache_ttl] = recache_threshold
+      result = server.request(:meta_get, key, meta_options)
 
       return result[:value] unless result[:won_recache]
 

data/lib/dalli/compressor.rb

@@ -13,8 +13,54 @@ module Dalli
       Zlib::Deflate.deflate(data)
     end
 
-    def self.decompress(data)
-      Zlib::Inflate.inflate(data)
+    # DEFLATE can't expand data by more than about 1032:1
+    MAX_EXPANSION = 1032
+    # Compressed input is fed to zlib this many bytes at a time. MRI's zlib
+    # yields the output in small chunks; one that returns it instead still
+    # produces at most MAX_EXPANSION times this per step.
+    INFLATE_SLICE_BYTES = 64 * 1024
+
+    # max_bytes caps the decompressed size; inflating stops and raises as soon
+    # as the output passes it, so a small compressed value can't expand into
+    # gigabytes of memory.
+    def self.decompress(data, max_bytes: nil)
+      return Zlib::Inflate.inflate(data) unless may_exceed?(data, max_bytes)
+
+      inflate_within_limit(Zlib::Inflate.new, data, max_bytes)
+    end
+
+    # False when even the largest possible expansion stays within max_bytes,
+    # so the value can be inflated in one step
+    def self.may_exceed?(data, max_bytes)
+      max_bytes ? data.bytesize * MAX_EXPANSION > max_bytes : false
+    end
+
+    def self.inflate_within_limit(inflater, data, max_bytes)
+      out = String.new(encoding: Encoding::BINARY)
+      0.step(data.bytesize - 1, INFLATE_SLICE_BYTES) do |pos|
+        inflate_slice(inflater, data.byteslice(pos, INFLATE_SLICE_BYTES), out, max_bytes)
+        # Anything after the end of the stream isn't part of the value, as
+        # with Zlib::Inflate.inflate; finish would return it as output
+        break if inflater.finished?
+      end
+      return out if inflater.finished?
+
+      append_within_limit(out, inflater.finish, max_bytes)
+    ensure
+      inflater.close
+    end
+
+    # Both yielded and returned output count toward the limit
+    def self.inflate_slice(inflater, slice, out, max_bytes)
+      rest = inflater.inflate(slice) { |chunk| append_within_limit(out, chunk, max_bytes) }
+      append_within_limit(out, rest, max_bytes)
+    end
+
+    def self.append_within_limit(out, chunk, max_bytes)
+      out << chunk if chunk
+      return out if out.bytesize <= max_bytes
+
+      raise Dalli::UnmarshalError, "Decompressed value exceeds #{max_bytes} bytes (decompressed_max_bytes)"
     end
   end
 
@@ -32,9 +78,11 @@ module Dalli
       io.string
     end
 
-    def self.decompress(data)
-      io = StringIO.new(data, 'rb')
-      Zlib::GzipReader.new(io).read
+    def self.decompress(data, max_bytes: nil)
+      return Zlib::GzipReader.new(StringIO.new(data, 'rb')).read unless Compressor.may_exceed?(data, max_bytes)
+
+      # MAX_WBITS + 16 makes zlib read the gzip format (and check its CRC)
+      Compressor.inflate_within_limit(Zlib::Inflate.new(Zlib::MAX_WBITS + 16), data, max_bytes)
     end
   end
 end

data/lib/dalli/json_serializer.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+require 'json'
+
+module Dalli
+  ##
+  # A serializer that stores values as JSON and never creates Ruby objects
+  # from what it reads.
+  #
+  # Passing the JSON module itself as the serializer (serializer: JSON) reads
+  # values back with JSON.load, which on json gem versions before 3.0 honors
+  # "json_class" and instantiates other classes when the json gem's additions
+  # are loaded. This serializer reads with JSON.parse, so a value written to
+  # memcached by someone else can only come back as plain hashes, arrays,
+  # strings, numbers, booleans and nil.
+  #
+  #   Dalli::Client.new(servers, serializer: Dalli::JSONSerializer)
+  ##
+  module JSONSerializer
+    def self.dump(value)
+      JSON.generate(value)
+    end
+
+    def self.load(data)
+      JSON.parse(data)
+    end
+  end
+end

data/lib/dalli/key_manager.rb

@@ -41,6 +41,9 @@ module Dalli
 
       @namespace = namespace_from_options
       @namespace_separator = @key_options[:namespace_separator]
+      # The meta protocol base64-encodes keys with whitespace or non-ASCII
+      # characters, so their length on the wire differs from their length here
+      @meta_protocol = client_options[:protocol]&.to_s == 'meta'
     end
 
     ##
@@ -58,7 +61,7 @@ module Dalli
       raise ArgumentError, 'key cannot be blank' unless key&.length&.positive?
 
       key = key_with_namespace(key)
-      key.length > MAX_KEY_LENGTH ? truncated_key(key) : key
+      wire_length(key) > MAX_KEY_LENGTH ? truncated_key(key) : key
     end
 
     ##
@@ -122,7 +125,30 @@ module Dalli
     ##
     def truncated_key(key)
       digest = digest_class.hexdigest(key)
-      "#{key[0, prefix_length(digest)]}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
+      prefix = key[0, [prefix_length(digest), 0].max]
+      truncated = "#{prefix}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
+      # A prefix with whitespace or non-ASCII characters sends the whole key
+      # base64-encoded, which can still be too long on the wire; shorten it.
+      while wire_length(truncated) > MAX_KEY_LENGTH && !prefix.empty?
+        prefix = prefix[0...-1]
+        truncated = "#{prefix}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
+      end
+      return truncated if wire_length(truncated) <= MAX_KEY_LENGTH
+
+      # Only possible with a digest_class whose digests aren't short hex strings
+      raise ArgumentError, "#{digest_class} digests are too long to fit in a truncated key"
+    end
+
+    # memcached's key limit applies to the key as sent: in bytes, and, with
+    # the meta protocol, after base64 encoding when the key needs it. A key
+    # within the limit in characters can still be too long. With the meta
+    # protocol memcached rejects it with a CLIENT_ERROR that must never reach
+    # a pipelined get; with the binary protocol it drops the connection.
+    def wire_length(key)
+      return key.bytesize unless @meta_protocol
+
+      encoded, = Dalli::Protocol::Meta::KeyRegularizer.encode(key)
+      encoded.bytesize
     end
 
     def prefix_length(digest)

data/lib/dalli/protocol/base.rb

@@ -34,6 +34,13 @@ module Dalli
         @raw_mode
       end
 
+      # True when values should come back as stored: the client is in raw mode,
+      # or this request passed raw: true. A stored value is then never
+      # deserialized or decompressed for a caller who asked for raw bytes.
+      def raw_request?(options)
+        raw_mode? || (options.is_a?(Hash) && options[:raw]) ? true : false
+      end
+
       # Chokepoint method for error handling and ensuring liveness
       def request(opkey, *args)
         verify_state(opkey)
@@ -94,12 +101,20 @@ module Dalli
       # When a block is given, yields (key, value, cas) for each response,
       # avoiding intermediate Hash allocation. Returns nil.
       # Without a block, returns a Hash of { key => [value, cas] }.
-      # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
+      # rubocop:disable-next Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
       def pipeline_next_responses(&block)
         reconnect_on_pipeline_complete!
         values = nil
 
-        response_buffer.read
+        # Only the read is socket I/O. The caller's block runs below, outside
+        # this rescue, so an Errno or Timeout::Error it raises reaches the
+        # caller instead of being mistaken for a network failure (which closed
+        # the connection and retried the whole get_multi, yielding keys twice).
+        begin
+          response_buffer.read
+        rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError => e
+          @connection_manager.error_on_request!(e)
+        end
 
         status, cas, key, value = response_buffer.process_single_getk_response
         # status is not nil only if we have a full response to parse
@@ -125,10 +140,7 @@ module Dalli
         end
 
         values || {}
-      rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError => e
-        @connection_manager.error_on_request!(e)
       end
-      # rubocop:enable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
 
       # Abort current pipelined get. Generally used to signal an external
       # timeout during pipelined get.  The underlying socket is
@@ -140,10 +152,13 @@ module Dalli
         @connection_manager.abort_request!
         return true unless connected?
 
-        # Closes the connection, which ensures that our connection
-        # is in a clean state for future requests
-        @connection_manager.error_on_request!('External timeout')
-      rescue NetworkError
+        # Closes the connection, which ensures that our connection is in a
+        # clean state for future requests. The get_multi ran out of time
+        # overall rather than hitting a socket error, so it doesn't count
+        # toward socket_max_failures: two slow get_multis in a row would
+        # otherwise mark a healthy server down.
+        Dalli.logger.warn { "#{name} didn't finish a get_multi within socket_timeout; closing the connection" }
+        @connection_manager.close
         true
       end
 

data/lib/dalli/protocol/binary/request_formatter.rb

@@ -88,7 +88,7 @@ module Dalli
         }.freeze
         FORMAT = BODY_FORMATS.transform_values { |v| REQ_HEADER_FORMAT + v }
 
-        # rubocop:disable Metrics/ParameterLists
+        # rubocop:disable-next Metrics/ParameterLists
         def self.standard_request(opkey:, key: nil, value: nil, opaque: 0, cas: 0, bitflags: nil, ttl: nil)
           extra_len = (bitflags.nil? ? 0 : 4) + (ttl.nil? ? 0 : 4)
           key_len = key.nil? ? 0 : key.bytesize
@@ -97,7 +97,6 @@ module Dalli
           body = [bitflags, ttl, key, value].compact
           (header + body).pack(FORMAT[opkey])
         end
-        # rubocop:enable Metrics/ParameterLists
 
         def self.decr_incr_request(opkey:, key: nil, count: nil, initial: nil, expiry: nil)
           extra_len = 20

data/lib/dalli/protocol/binary/response_processor.rb

@@ -66,7 +66,9 @@ module Dalli
           raise Dalli::DalliError, "Response error #{resp_header.status}: #{RESPONSE_CODES[resp_header.status]}"
         end
 
-        def get(cache_nils: false)
+        # raw: true returns the value as stored, without deserializing or
+        # decompressing it, whatever flags it was stored with
+        def get(cache_nils: false, raw: false)
           resp_header, body = read_response
 
           return false if resp_header.not_stored? # Not stored, normal status for add operation
@@ -75,7 +77,7 @@ module Dalli
           raise_on_not_ok!(resp_header)
           return true unless body
 
-          unpack_response_body(resp_header, body, true).last
+          unpack_response_body(resp_header, body, !raw).last
         end
 
         ##
@@ -100,7 +102,8 @@ module Dalli
           true
         end
 
-        def data_cas_response
+        # raw: true returns the value as stored; see #get
+        def data_cas_response(raw: false)
           resp_header, body = read_response
           return [nil, resp_header.cas] if resp_header.not_found?
           return [nil, false] if resp_header.not_stored?
@@ -108,7 +111,7 @@ module Dalli
           raise_on_not_ok!(resp_header)
           return [nil, resp_header.cas] unless body
 
-          [unpack_response_body(resp_header, body, true).last, resp_header.cas]
+          [unpack_response_body(resp_header, body, !raw).last, resp_header.cas]
         end
 
         # Returns the new value for the key, if found and updated
@@ -199,6 +202,17 @@ module Dalli
         # The remaining three values in the array are the ResponseHeader,
         # key, and value.
         ##
+        # memcached can't store an item larger than 1 GiB (its -I maximum). A
+        # pipelined reply claiming an impossible size would otherwise have the
+        # buffer wait for (and accumulate) that many bytes.
+        MAX_BODY_BYTES = 1024 * 1024 * 1024
+
+        def check_body_size!(size)
+          return if size <= MAX_BODY_BYTES
+
+          raise Dalli::DalliError, "Reply body size #{size} is out of range"
+        end
+
         def getk_response_from_buffer(buf, offset = 0)
           # There's no header in the buffer, so don't advance
           return [0, nil, nil, nil, nil] unless buf && buf.bytesize >= offset + ResponseHeader::SIZE
@@ -212,6 +226,8 @@ module Dalli
           # error response that needs to be discarded.
           return [ResponseHeader::SIZE, resp_header.ok?, resp_header.cas, nil, nil] if body_len.zero?
 
+          check_body_size!(body_len)
+
           resp_size = ResponseHeader::SIZE + body_len
           # The header is in the buffer, but the body is not.  As we don't have
           # a complete response, don't advance the buffer

data/lib/dalli/protocol/binary.rb

@@ -23,7 +23,7 @@ module Dalli
         req = RequestFormatter.standard_request(opkey: :get, key: key)
         write(req)
         @connection_manager.flush
-        response_processor.get(cache_nils: cache_nils?(options))
+        response_processor.get(cache_nils: cache_nils?(options), raw: raw_request?(options))
       end
 
       def quiet_get_request(key)
@@ -35,7 +35,7 @@ module Dalli
         req = RequestFormatter.standard_request(opkey: :gat, key: key, ttl: ttl)
         write(req)
         @connection_manager.flush
-        response_processor.get(cache_nils: cache_nils?(options))
+        response_processor.get(cache_nils: cache_nils?(options), raw: raw_request?(options))
       end
 
       def touch(key, ttl)
@@ -47,11 +47,11 @@ module Dalli
 
       # TODO: This is confusing, as there's a cas command in memcached
       # and this isn't it.  Maybe rename?  Maybe eliminate?
-      def cas(key)
+      def cas(key, options = nil)
         req = RequestFormatter.standard_request(opkey: :get, key: key)
         write(req)
         @connection_manager.flush
-        response_processor.data_cas_response
+        response_processor.data_cas_response(raw: raw_request?(options))
       end
 
       # Storage Commands
@@ -63,7 +63,7 @@ module Dalli
       # Pipelined set - writes a quiet set request without reading response.
       # Used by PipelinedSetter for bulk operations.
       def pipelined_set(key, value, ttl, options)
-        storage_req(:setq, key, value, ttl, 0, options)
+        storage_req(:setq, key, value, ttl, 0, options, quiet: true)
       end
 
       def add(key, value, ttl, options)
@@ -76,8 +76,8 @@ module Dalli
         storage_req(opkey, key, value, ttl, cas, options)
       end
 
-      # rubocop:disable Metrics/ParameterLists
-      def storage_req(opkey, key, value, ttl, cas, options)
+      # rubocop:disable-next Metrics/ParameterLists
+      def storage_req(opkey, key, value, ttl, cas, options, quiet: quiet?)
         (value, bitflags) = @value_marshaller.store(key, value, options)
         ttl = TtlSanitizer.sanitize(ttl)
 
@@ -85,10 +85,9 @@ module Dalli
                                                 value: value, bitflags: bitflags,
                                                 ttl: ttl, cas: cas)
         write(req)
-        @connection_manager.flush unless quiet?
-        response_processor.storage_response unless quiet?
+        @connection_manager.flush unless quiet
+        response_processor.storage_response unless quiet
       end
-      # rubocop:enable Metrics/ParameterLists
 
       def append(key, value)
         opkey = quiet? ? :appendq : :append

data/lib/dalli/protocol/connection_manager.rb

@@ -37,7 +37,10 @@ module Dalli
         @request_in_progress = false
         @sock = nil
         @pid = nil
+        @write_buffer = []
+        @write_buffer_bytes = 0
 
+        @fail_count = 0
         reset_down_info
       end
 
@@ -53,8 +56,9 @@ module Dalli
         Dalli.logger.debug { "Dalli::Server#connect #{name}" }
 
         @sock = memcached_socket
-        @sock.sync = false # Enable buffered I/O for better performance
-        @pid = PIDCache.pid
+        # Writes are buffered in @write_buffer instead; see WRITE_BUFFER_FLUSH_BYTES
+        @sock.sync = true
+        @pid = Process.pid
         @request_in_progress = false
       rescue SystemCallError, *TIMEOUT_ERRORS, EOFError, SocketError => e
         # SocketError = DNS resolution failure
@@ -85,6 +89,8 @@ module Dalli
       def down!
         close
         log_down_detected
+        # Once down_retry_delay passes, the server gets a full set of attempts
+        @fail_count = 0
 
         @error = $ERROR_INFO&.class&.name
         @msg ||= $ERROR_INFO&.message
@@ -114,15 +120,23 @@ module Dalli
         return unless @sock
 
         begin
-          @sock.close
+          close_socket
         rescue StandardError
           nil
         end
         @sock = nil
         @pid = nil
+        discard_write_buffer
         abort_request!
       end
 
+      # A forked child shares the parent's connection, so it closes only its
+      # own file descriptor. Closing the TLS socket itself would send
+      # close_notify and end the parent's TLS session.
+      def close_socket
+        fork_detected? ? @sock.to_io.close : @sock.close
+      end
+
       def connected?
         !@sock.nil?
       end
@@ -141,6 +155,11 @@ module Dalli
         raise '[Dalli] No request in progress. This may be a bug in Dalli.' unless @request_in_progress
 
         @request_in_progress = false
+        # A completed request is what proves the server healthy again, so the
+        # failure count resets here rather than on reconnect: a server that
+        # accepts connections but never answers would otherwise reset it on
+        # every retry and never reach socket_max_failures.
+        @fail_count = 0
       end
 
       def abort_request!
@@ -148,6 +167,7 @@ module Dalli
       end
 
       def read_line
+        flush_write_buffer
         data = @sock.gets("\r\n")
         error_on_request!('EOF in read_line') if data.nil?
         data
@@ -155,27 +175,56 @@ module Dalli
         error_on_request!(e)
       end
 
+      # memcached can't store an item larger than 1 GiB (its -I maximum), so a
+      # reply claiming more (or a negative size) is malformed or hostile.
+      # Reading allocates the full count up front, so check before reading.
+      MAX_READ_BYTES = (1024 * 1024 * 1024) + 2 # plus the value's trailing "\r\n"
+
+      def check_read_size!(count)
+        return if count.between?(0, MAX_READ_BYTES)
+
+        raise Dalli::DalliError, "Reply size #{count} from #{name} is out of range"
+      end
+
       def read(count)
+        check_read_size!(count)
+        flush_write_buffer
         @sock.readfull(count)
       rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError => e
         error_on_request!(e)
       end
 
+      # Requests are buffered here rather than in the socket's own IO buffer
+      # (the socket is sync). Ruby flushes an IO's write buffer when any process
+      # holding it closes or finalizes the IO, so after a fork a child could
+      # send bytes the parent had buffered but not sent yet: a quiet write would
+      # run twice, or the parent's replies would shift onto the wrong requests.
+      # A forked child discards this buffer instead. Like IO's own buffer, it
+      # is sent once it grows past this size, or when a reply is read.
+      WRITE_BUFFER_FLUSH_BYTES = 64 * 1024
+
       def write(bytes)
-        @sock.write(bytes)
-      rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS => e
+        # A copy: the caller may change its string before the buffer is sent
+        # (the meta protocol writes a value string as its own part), as Ruby's
+        # IO buffer copied on write
+        @write_buffer << (bytes.frozen? ? bytes : bytes.dup)
+        @write_buffer_bytes += bytes.bytesize
+        flush_write_buffer if @write_buffer_bytes >= WRITE_BUFFER_FLUSH_BYTES
+        bytes.bytesize
+      rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, IOError => e
         error_on_request!(e)
       end
 
       def flush
-        @sock.flush
-      rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS => e
+        flush_write_buffer
+      rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, IOError => e
         error_on_request!(e)
       end
 
       # Non-blocking read.  Here to support the operation
       # of the get_multi operation
       def read_nonblock
+        flush_write_buffer
         @sock.read_available
       end
 
@@ -202,8 +251,9 @@ module Dalli
         raise Dalli::RetryableNetworkError, message
       end
 
+      # Called on connect. Deliberately leaves @fail_count alone; see
+      # finish_request!.
       def reset_down_info
-        @fail_count = 0
         @down_at = nil
         @last_down_at = nil
         @msg = nil
@@ -228,13 +278,18 @@ module Dalli
       def reconnect_on_fork
         message = 'Fork detected, re-connecting child process...'
         Dalli.logger.info { message }
-        # Close socket on a fork and reconnect immediately
+        # Drops anything the parent had buffered, closes the inherited socket
+        # without touching the parent's connection, and reconnects immediately
         close
         establish_connection
       end
 
       def fork_detected?
-        @pid && @pid != PIDCache.pid
+        # Process.pid rather than PIDCache: PIDCache is refreshed by a
+        # Process._fork hook, and another library's fork hook (such as
+        # connection_pool closing its connections) can run in the child before
+        # it, which would close the parent's TLS session as if it were ours.
+        @pid && @pid != Process.pid
       end
 
       def log_down_detected
@@ -255,6 +310,39 @@ module Dalli
         time = Time.now - @down_at
         Dalli.logger.warn { format('%<name>s is back (downtime was %<time>.3f seconds)', name: name, time: time) }
       end
+
+      private
+
+      # One write per flush. A TLS socket sends each argument of
+      # write(*parts) as its own record and system call, so the parts are
+      # joined first.
+      def flush_write_buffer
+        return if @write_buffer.empty?
+
+        @sock.write(@write_buffer.size == 1 ? @write_buffer.first : joined_write_buffer)
+        discard_write_buffer
+      end
+
+      # Joined as bytes: requests can mix UTF-8 and binary strings, which
+      # String#<< refuses to combine when both hold non-ASCII bytes
+      if String.method_defined?(:append_as_bytes)
+        def joined_write_buffer
+          out = String.new(capacity: @write_buffer_bytes, encoding: Encoding::BINARY)
+          @write_buffer.each { |part| out.append_as_bytes(part) }
+          out
+        end
+      else
+        def joined_write_buffer
+          out = String.new(capacity: @write_buffer_bytes, encoding: Encoding::BINARY)
+          @write_buffer.each { |part| out << part.b }
+          out
+        end
+      end
+
+      def discard_write_buffer
+        @write_buffer.clear
+        @write_buffer_bytes = 0
+      end
     end
   end
 end

data/lib/dalli/protocol/meta/response_processor.rb

@@ -28,22 +28,25 @@ module Dalli
           @value_marshaller = value_marshaller
         end
 
-        def meta_get_with_value(cache_nils: false)
+        # raw: true (a request made with raw: true) returns the value as stored,
+        # ignoring any flags in the reply. The request didn't ask for flags, so
+        # a reply carrying them anyway must not get the value deserialized.
+        def meta_get_with_value(cache_nils: false, raw: false)
           tokens = error_on_unexpected!([VA, EN, HD])
           return cache_nils ? ::Dalli::NOT_FOUND : nil if tokens.first == EN
           return true unless tokens.first == VA
 
-          @value_marshaller.retrieve(read_data(tokens[1].to_i), bitflags_from_tokens(tokens))
+          retrieve(read_data(tokens[1].to_i), raw ? 0 : bitflags_from_tokens(tokens), raw)
         end
 
-        def meta_get_with_value_and_cas
+        def meta_get_with_value_and_cas(raw: false)
           tokens = error_on_unexpected!([VA, EN, HD])
           return [nil, 0] if tokens.first == EN
 
           cas = cas_from_tokens(tokens)
           return [nil, cas] unless tokens.first == VA
 
-          [@value_marshaller.retrieve(read_data(tokens[1].to_i), bitflags_from_tokens(tokens)), cas]
+          [retrieve(read_data(tokens[1].to_i), raw ? 0 : bitflags_from_tokens(tokens), raw), cas]
         end
 
         def meta_get_without_value
@@ -62,12 +65,13 @@ module Dalli
         #
         # Used by meta_get for comprehensive metadata retrieval.
         # Supports thundering herd protection (N/R flags) and metadata flags (h/l/u).
-        def meta_get_with_metadata(cache_nils: false, return_hit_status: false, return_last_access: false)
+        def meta_get_with_metadata(cache_nils: false, return_hit_status: false, return_last_access: false,
+                                   raw: false)
           tokens = error_on_unexpected!([VA, EN, HD])
           result = build_metadata_result(tokens)
           result[:hit_before] = hit_status_from_tokens(tokens) if return_hit_status
           result[:last_access] = last_access_from_tokens(tokens) if return_last_access
-          result[:value] = parse_value_from_tokens(tokens, cache_nils)
+          result[:value] = parse_value_from_tokens(tokens, cache_nils, raw)
           result
         end
 
@@ -79,11 +83,16 @@ module Dalli
           }
         end
 
-        def parse_value_from_tokens(tokens, cache_nils)
+        def parse_value_from_tokens(tokens, cache_nils, raw = false) # rubocop:disable Style/OptionalBooleanParameter
           return cache_nils ? ::Dalli::NOT_FOUND : nil if tokens.first == EN
           return unless tokens.first == VA
 
-          @value_marshaller.retrieve(read_data(tokens[1].to_i), bitflags_from_tokens(tokens))
+          retrieve(read_data(tokens[1].to_i), raw ? 0 : bitflags_from_tokens(tokens), raw)
+        end
+
+        # Values from a raw request are returned as stored
+        def retrieve(value, bitflags, raw)
+          raw ? value : @value_marshaller.retrieve(value, bitflags)
         end
 
         def meta_set_with_cas
@@ -148,6 +157,10 @@ module Dalli
         end
 
         def full_response_from_buffer(tokens, body, resp_size)
+          # A hit with no key can't be matched to a request: skip it (false
+          # status) rather than yield it, or take it for the end of the pipeline
+          return [resp_size, false, nil, nil, nil] unless tokens.any? { |t| t.start_with?('k') }
+
           value = @value_marshaller.retrieve(body, bitflags_from_tokens(tokens))
           [resp_size, tokens.first == VA, cas_from_tokens(tokens), key_from_tokens(tokens), value]
         end
@@ -172,11 +185,14 @@ module Dalli
           header_len = header.bytesize + TERMINATOR.length
           body_len = body_len_from_tokens(tokens)
 
-          # We have a complete response that has no body.
-          # This is either the response to the terminating
-          # noop or, if the status is not MN, an intermediate
-          # error response that needs to be discarded.
-          return [header_len, true, nil, nil, nil] if body_len.zero?
+          # A complete response with no body. Only the terminating noop's MN
+          # ends the pipeline. Any other bodyless reply (CLIENT_ERROR,
+          # SERVER_ERROR, ...) answers one key's request: report it with a
+          # false status so the caller skips it and keeps reading. Treating it
+          # as the end would leave the remaining replies on the connection for
+          # later commands to read as their own. A hit on an empty value (VA 0)
+          # still has a body -- just its terminator -- so it's parsed below.
+          return [header_len, tokens.first == MN, nil, nil, nil] if no_body?(tokens, body_len)
 
           resp_size = header_len + body_len + TERMINATOR.length
           # The header is in the buffer, but the body is not.  As we don't have
@@ -189,6 +205,27 @@ module Dalli
           full_response_from_buffer(tokens, body, resp_size)
         end
 
+        # memcached can't store an item larger than 1 GiB (its -I maximum). A
+        # pipelined reply claiming an impossible size would otherwise have the
+        # buffer wait for (and accumulate) that many bytes.
+        MAX_VALUE_BYTES = 1024 * 1024 * 1024
+
+        # The size given right after "VA", or 0 when it isn't a number
+        def positional_size(tokens)
+          tokens.first == VA && tokens[1]&.match?(/\A\d+\z/) ? tokens[1].to_i : 0
+        end
+
+        def check_value_size!(size)
+          return if size.between?(0, MAX_VALUE_BYTES)
+
+          raise Dalli::DalliError, "Reply value size #{size} is out of range"
+        end
+
+        # A zero-size reply has no body unless it's a VA (a hit on an empty value)
+        def no_body?(tokens, body_len)
+          body_len.zero? && tokens.first != VA
+        end
+
         def error_on_unexpected!(expected_codes)
           tokens = next_line_to_tokens
 
@@ -229,7 +266,11 @@ module Dalli
         end
 
         def body_len_from_tokens(tokens)
-          value_from_tokens(tokens, 's')&.to_i
+          # memcached also gives the size positionally ("VA <size> ..."); use it
+          # when there's no s flag rather than reading the value as replies
+          size = tokens.any? { |t| t.start_with?('s') } ? value_from_tokens(tokens, 's')&.to_i : positional_size(tokens)
+          check_value_size!(size) if size
+          size
         end
 
         def value_from_tokens(tokens, flag)
@@ -249,6 +290,9 @@ module Dalli
         end
 
         def read_data(data_size)
+          # Checked before the terminator is added: -1 or -2 would otherwise
+          # read 1 or 0 bytes and leave the connection out of step
+          check_value_size!(data_size)
           @io_source.read(data_size + TERMINATOR.bytesize)&.chomp!(TERMINATOR)
         end
       end

data/lib/dalli/protocol/meta.rb

@@ -30,7 +30,7 @@ module Dalli
         req = RequestFormatter.meta_get(key: encoded_key, base64: base64, skip_flags: skip_flags)
         write(req)
         @connection_manager.flush
-        response_processor.meta_get_with_value(cache_nils: cache_nils?(options))
+        response_processor.meta_get_with_value(cache_nils: cache_nils?(options), raw: raw_request?(options))
       end
 
       def quiet_get_request(key)
@@ -47,7 +47,7 @@ module Dalli
         req = RequestFormatter.meta_get(key: encoded_key, ttl: ttl, base64: base64, skip_flags: skip_flags)
         write(req)
         @connection_manager.flush
-        response_processor.meta_get_with_value(cache_nils: cache_nils?(options))
+        response_processor.meta_get_with_value(cache_nils: cache_nils?(options), raw: raw_request?(options))
       end
 
       def touch(key, ttl)
@@ -61,12 +61,13 @@ module Dalli
 
       # TODO: This is confusing, as there's a cas command in memcached
       # and this isn't it.  Maybe rename?  Maybe eliminate?
-      def cas(key)
+      def cas(key, options = nil)
         encoded_key, base64 = KeyRegularizer.encode(key)
-        req = RequestFormatter.meta_get(key: encoded_key, value: true, return_cas: true, base64: base64)
+        req = RequestFormatter.meta_get(key: encoded_key, value: true, return_cas: true, base64: base64,
+                                        skip_flags: raw_request?(options))
         write(req)
         @connection_manager.flush
-        response_processor.meta_get_with_value_and_cas
+        response_processor.meta_get_with_value_and_cas(raw: raw_request?(options))
       end
 
       # Comprehensive meta get with support for all metadata flags.
@@ -96,7 +97,7 @@ module Dalli
       def meta_get(key, options = {})
         encoded_key, base64 = KeyRegularizer.encode(key)
         req = RequestFormatter.meta_get(
-          key: encoded_key, value: true, return_cas: true, base64: base64,
+          key: encoded_key, value: true, return_cas: true, base64: base64, skip_flags: raw_request?(options),
           vivify_ttl: options[:vivify_ttl], recache_ttl: options[:recache_ttl],
           return_hit_status: options[:return_hit_status],
           return_last_access: options[:return_last_access], skip_lru_bump: options[:skip_lru_bump]
@@ -105,7 +106,7 @@ module Dalli
         @connection_manager.flush
         response_processor.meta_get_with_metadata(
           cache_nils: cache_nils?(options), return_hit_status: options[:return_hit_status],
-          return_last_access: options[:return_last_access]
+          return_last_access: options[:return_last_access], raw: raw_request?(options)
         )
       end
 
@@ -146,7 +147,7 @@ module Dalli
         response_processor.meta_set_with_cas unless quiet?
       end
 
-      # rubocop:disable Metrics/ParameterLists
+      # rubocop:disable-next Metrics/ParameterLists
       def write_storage_req(mode, key, raw_value, ttl = nil, cas = nil, options = {}, quiet: quiet?)
         (value, bitflags) = @value_marshaller.store(key, raw_value, options)
         ttl = TtlSanitizer.sanitize(ttl) if ttl
@@ -159,7 +160,6 @@ module Dalli
         write(TERMINATOR)
         @connection_manager.flush unless quiet
       end
-      # rubocop:enable Metrics/ParameterLists
 
       def append(key, value)
         write_append_prepend_req(:append, key, value)
@@ -171,7 +171,7 @@ module Dalli
         response_processor.meta_set_append_prepend unless quiet?
       end
 
-      # rubocop:disable Metrics/ParameterLists
+      # rubocop:disable-next Metrics/ParameterLists
       def write_append_prepend_req(mode, key, value, ttl = nil, cas = nil, _options = {})
         ttl = TtlSanitizer.sanitize(ttl) if ttl
         encoded_key, base64 = KeyRegularizer.encode(key)
@@ -182,7 +182,6 @@ module Dalli
         write(TERMINATOR)
         @connection_manager.flush unless quiet?
       end
-      # rubocop:enable Metrics/ParameterLists
 
       # Delete Commands
       def delete(key, cas)

data/lib/dalli/protocol/value_compressor.rb

@@ -15,11 +15,17 @@ module Dalli
         compress: true,
         compressor: ::Dalli::Compressor,
         # min byte size to attempt compression
-        compression_min_size: 4 * 1024 # 4K
+        compression_min_size: 4 * 1024, # 4K
+        # max size a stored value may decompress to (nil: no limit). Guards
+        # against a small compressed value expanding into gigabytes on read.
+        decompressed_max_bytes: 128 * 1024 * 1024 # 128 MiB
       }.freeze
 
       OPTIONS = DEFAULTS.keys.freeze
 
+      KEYWORD_PARAMETER_TYPES = %i[key keyreq].freeze
+      private_constant :KEYWORD_PARAMETER_TYPES
+
       # https://www.hjp.at/zettel/m/memcached_flags.rxml
       # Looks like most clients use bit 1 to indicate gzip compression.
       FLAG_COMPRESSED = 0x2
@@ -38,8 +44,16 @@ module Dalli
       end
 
       def retrieve(value, bitflags)
-        compressed = bitflags.anybits?(FLAG_COMPRESSED)
-        compressed ? compressor.decompress(value) : value
+        return value unless bitflags.anybits?(FLAG_COMPRESSED)
+
+        max_bytes = @compression_options[:decompressed_max_bytes]
+        # Custom compressors that only define decompress(data) keep working,
+        # without the limit
+        if max_bytes && compressor_accepts_max_bytes?
+          compressor.decompress(value, max_bytes: max_bytes)
+        else
+          compressor.decompress(value)
+        end
 
       # TODO: We likely want to move this rescue into the Dalli::Compressor / Dalli::GzipCompressor
       # itself, since not all compressors necessarily use Zlib.  For now keep it here, so the behavior
@@ -48,6 +62,19 @@ module Dalli
         raise UnmarshalError, "Unable to uncompress value: #{$ERROR_INFO.message}"
       end
 
+      def compressor_accepts_max_bytes?
+        return @compressor_accepts_max_bytes if defined?(@compressor_accepts_max_bytes)
+
+        @compressor_accepts_max_bytes =
+          begin
+            compressor.method(:decompress).parameters.any? do |type, name|
+              name == :max_bytes && KEYWORD_PARAMETER_TYPES.include?(type)
+            end
+          rescue NameError # an object without a reflectable decompress method
+            false
+          end
+      end
+
       def compress_by_default?
         @compression_options[:compress]
       end

data/lib/dalli/protocol/value_serializer.rb

@@ -102,7 +102,7 @@ module Dalli
         Dalli.logger.warn 'SECURITY WARNING: Dalli is using Marshal for serialization. ' \
                           'Marshal can execute arbitrary code during deserialization. ' \
                           'If your memcached server could be compromised, consider using ' \
-                          'a safer serializer like JSON: Dalli::Client.new(servers, serializer: JSON)'
+                          'a safer serializer: Dalli::Client.new(servers, serializer: Dalli::JSONSerializer)'
         @@marshal_warning_logged = true # rubocop:disable Style/ClassVars
       end
     end

data/lib/dalli/version.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module Dalli
-  VERSION = '4.3.4'
+  VERSION = '4.3.7'
 
   MIN_SUPPORTED_MEMCACHED_VERSION = '1.4'
 end

data/lib/dalli.rb

@@ -62,6 +62,7 @@ require_relative 'dalli/version'
 require_relative 'dalli/instrumentation'
 
 require_relative 'dalli/compressor'
+require_relative 'dalli/json_serializer'
 require_relative 'dalli/protocol_deprecations'
 require_relative 'dalli/client'
 require_relative 'dalli/key_manager'

data/lib/rack/session/dalli.rb

@@ -16,11 +16,10 @@ module Rack
       attr_reader :data
 
       # Don't freeze this until we fix the specs/implementation
-      # rubocop:disable Style/MutableConstant
+      # rubocop:disable-next Style/MutableConstant
       DEFAULT_DALLI_OPTIONS = {
         namespace: 'rack:session'
       }
-      # rubocop:enable Style/MutableConstant
 
       # Brings in a new Rack::Session::Dalli middleware with the given
       # `:memcache_server`. The server is either a hostname, or a

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: dalli
 version: !ruby/object:Gem::Version
-  version: 4.3.4
+  version: 4.3.7
 platform: ruby
 authors:
 - Peter M. Goldstein
@@ -41,6 +41,7 @@ files:
 - lib/dalli/client.rb
 - lib/dalli/compressor.rb
 - lib/dalli/instrumentation.rb
+- lib/dalli/json_serializer.rb
 - lib/dalli/key_manager.rb
 - lib/dalli/options.rb
 - lib/dalli/pid_cache.rb