dalli 5.2.0 → 5.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 23a3890b1ca95b3668eea5b0f2f6eabfb3ad6062daebf5723e5b99c2c7a74854
-  data.tar.gz: 4f0c1775badbf133dab91d32a19f84ffafab7f1c0a6c9d7c195f968bed01c089
+  metadata.gz: 42431b1f305b3fe3d3923c2b66f3d1edfb402da378c434a587bb2c85d6688055
+  data.tar.gz: 39bc473979fb49cd488853d39a64675884ce89951e96a7b5925dc88c8307219a
 SHA512:
-  metadata.gz: b1624d8d1024956e90ab86a5d5b7f56833d9d195b95d2e8f7ed92774775eebb5ae19baf798773e17ce8fdb7b8fa5abf728a5899d085ecd11335fe11daac52bdb
-  data.tar.gz: 76c4e96a6ac88ec7ce86d213d58bf4f709f6ebc52e747ffa05661343173c654696e6e2625090b14d31874f160f3778a750e9cd524cb4f8985ea86716f267793e
+  metadata.gz: 810da4f09ea75bfa616471a48686cb0804b820b3a629c1ec53c3ecc7146c76d94e7afa729ce3361287ccafaa71d97a0ed5053f0b6adc0d8cea91c5de81070231
+  data.tar.gz: c18a6a1f3d1dac3fcf5f640c8714ff68e70e07dc8d154d1471fc62c9ccb5ee6a970a3d079b042d4c5337ad11f39090e69022ef6eb34dac281906e9834863b285

data/CHANGELOG.md

@@ -4,6 +4,53 @@ Dalli Changelog
 Unreleased
 ==========
 
+5.2.1
+==========
+
+Security:
+
+- Fix pipelined `get_multi` returning one key's value for another after an error reply (GHSA-p6pm-ch9v-44vx)
+  - The pipelined reply parser took any reply without a body for the end of the batch. An error reply for one key (`CLIENT_ERROR`, `SERVER_ERROR`, or `EN` from a proxy) ended it early, and the replies still on the connection were read as the replies to later commands, so a `get` could return another key's value
+  - Keys are now measured in bytes as sent, after base64 encoding when the key needs it. A key under 250 characters but over 250 bytes on the wire (for example, one with many non-ASCII characters) was the easiest way to cause that error reply. Such keys are now truncated like other long keys; keys that worked before are unchanged
+  - Affects the meta protocol since 3.2.0 (the default since 5.0.0); fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
+- Reject routing tokens that add meta flags, and stop retrying an unresponsive server forever (GHSA-4qp6-2jcr-596v)
+  - `p_token` and `l_token` now reject whitespace and control characters, not just CR, LF and NUL. A space let a token add meta flags to the request it was sent with: changing an item's TTL on a read, creating stub items on a miss, turning a delete into a stale tombstone, or reading a different key
+  - Affects 5.1.0 and later; fixed in 5.2.1 and 5.1.3
+  - A server that accepted connections but never answered (or dropped the connection on a request) was retried forever, hanging the caller, because each successful reconnect reset the failure count. Requests now fail after `socket_max_failures` attempts and the server is marked down, as when it can't be reached at all; it gets a full set of attempts again after `down_retry_delay`
+  - Affects all versions; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
+- Limit decompressed and reply value sizes (GHSA-3553-vcg5-72jw)
+  - Values flagged as compressed were inflated without limit, so a small stored value could expand to gigabytes on read, whatever the client's `compress` or `serializer` settings. The new `decompressed_max_bytes` option (default 128 MiB; `nil` disables it) makes a read that would pass it raise `Dalli::UnmarshalError`. Custom compressors whose `decompress` takes only the data keep working, without the limit
+  - The value size in a reply was used to read or buffer that many bytes, so a malicious server could make the client allocate gigabytes. Sizes over 1 GiB, memcached's largest item, now raise `Dalli::DalliError` before reading
+  - Affects all versions; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
+- Honor per-request `raw: true` on reads, and add `Dalli::JSONSerializer` (GHSA-wr87-m4jw-29x5)
+  - `get_multi`, `get_multi_cas`, `get_multi_with_metadata`, `get_cas` and `get_with_metadata` ignored a per-request `raw: true` and deserialized the value according to its stored flags, so a caller who asked for raw bytes could still have `Marshal.load` run on data someone else wrote to memcached. They now return the stored bytes
+  - The `raw` part affects `get_with_metadata` since 4.2.0, and the other methods since they gained per-request options in 5.1.0
+  - `serializer: JSON` reads values with `JSON.load`, which on json gem versions before 3.0 creates an object of the class named in a stored `json_class` key when the json additions are loaded. The new `Dalli::JSONSerializer` reads with `JSON.parse` and only returns plain JSON types. The README and the Marshal security warning now recommend it; `serializer: JSON` itself is unchanged
+  - The `serializer: JSON` part affects any version used with the json gem before 3.0
+- Keep a forked child from writing to or closing the parent's connection (GHSA-w39f-xq2m-4g8x)
+  - A child forked while the parent had requests buffered but not yet sent, such as quiet writes inside a `quiet` block, sent them again when it closed the client or reconnected: quiet writes ran twice, and the parent's later replies could be read as the replies to other requests, returning one key's value for another. Dalli now buffers requests itself and a forked child discards them
+  - Affects 4.2.0 and later; fixed in 5.2.1, 5.1.3, 5.0.9 and 4.3.6
+  - With TLS, a forked child that closed the client (or reconnected after detecting the fork) sent a TLS close on the connection the parent was still using, ending the parent's session. A forked child now closes only its own copy of the socket
+  - Affects all versions with TLS; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
+
+Bug fixes:
+
+- An exception raised by a `get_multi` block (such as `Timeout::Error`) was treated as a network failure: swallowed, with the whole `get_multi` retried and keys yielded twice. It now reaches the caller
+
+Notes:
+
+- Document [Ruby bug 21195](https://bugs.ruby-lang.org/issues/21195), a VM crash (`[BUG] rb_sys_fail_path_in(io_fillbuf, ...) - errno == 0`) when a socket read using `IO#timeout` is interrupted by a signal, in the README (#1189)
+  - Affects Ruby 3.3.0–3.3.7 and 3.4.0–3.4.2; fixed in 3.3.8 and 3.4.3
+
+Development:
+
+- Add property tests for request building and fuzz tests for the reply parsers
+- Add a security policy (`SECURITY.md`), give every workflow a least-privilege token, and pin third-party actions by commit SHA (#1190)
+- Fix flaky failover tests (#1184)
+  - Their ports were inside Linux's ephemeral port range (32768-60999), so an earlier client connection could hold one as its local port. memcached then couldn't bind it on IPv4 and listened on IPv6 only, and the test's client marked that server down. They now use ports in the 26xxx range
+  - The test helper now waits until memcached accepts connections, instead of sleeping a fixed 0.1s after starting it
+- Refresh apt's package index before installing libevent in CI, so a stale runner image can't break every memcached build (#1172)
+
 5.2.0
 ==========
 

data/README.md

@@ -24,6 +24,16 @@ latest release. Earlier 1.6.x servers are not supported: the meta protocol
 rejects unknown flags outright, so features added after a server's release fail
 with `CLIENT_ERROR invalid flag` rather than degrading.
 
+### Known Ruby issue: crash on an interrupted socket read
+
+Dalli sets socket timeouts with `IO#timeout`. Ruby 3.3.0–3.3.7 and 3.4.0–3.4.2 have a bug ([Ruby #21195](https://bugs.ruby-lang.org/issues/21195)) where a timed read interrupted by a signal aborts the whole process:
+
+```
+[BUG] rb_sys_fail_path_in(io_fillbuf, fd:N ) - errno == 0
+```
+
+The crash is inside Ruby itself, so it can't be rescued. It's fixed in Ruby 3.3.8 and 3.4.3. If you see this crash, upgrade to one of those versions or later.
+
 ## Configuration Options
 
 ### Namespace
@@ -80,9 +90,13 @@ end
 By default, Dalli uses Ruby's Marshal for serialization. Deserializing untrusted data with Marshal can lead to remote code execution. If you cache user-controlled data, consider using a safer serializer:
 
 ```ruby
-Dalli::Client.new('localhost:11211', serializer: JSON)
+Dalli::Client.new('localhost:11211', serializer: Dalli::JSONSerializer)
 ```
 
+`Dalli::JSONSerializer` reads values with `JSON.parse`, so it only ever returns plain hashes, arrays, strings, numbers, booleans and `nil`. Passing the `JSON` module itself (`serializer: JSON`) reads values with `JSON.load`, which can create objects of other classes from a stored `json_class` key and is not recommended for data an attacker might write.
+
+Per-request `raw: true` returns the stored bytes without deserializing or decompressing them, on every read method (`get`, `get_multi`, `get_cas`, `get_with_metadata` and `fetch`).
+
 See the [5.0-Upgrade.md](5.0-Upgrade.md) guide for upgrade information.
 
 ## OpenTelemetry Tracing

data/lib/dalli/client.rb

@@ -881,7 +881,7 @@ module Dalli
     # check means unwinding through Protocol::Base#request, which logs the
     # failure as unexpected and closes the connection -- a caller passing a
     # bad token should get a clean ArgumentError and keep its connection.
-    ROUTING_TOKEN_FORBIDDEN = /[\r\n\0]/
+    ROUTING_TOKEN_FORBIDDEN = /[\x00-\x20\x7F]/
     private_constant :ROUTING_TOKEN_FORBIDDEN
 
     def validate_routing_tokens!(req_options)
@@ -897,7 +897,9 @@ module Dalli
       # here (it would also excuse [] / {} from the type check below).
       return if value.nil? || (value.is_a?(String) && value.empty?)
       raise ArgumentError, "#{name} must be a String, got #{value.class}" unless value.is_a?(String)
-      raise ArgumentError, "#{name} must not contain CRLF or null bytes" if value.match?(ROUTING_TOKEN_FORBIDDEN)
+      return unless value.match?(ROUTING_TOKEN_FORBIDDEN)
+
+      raise ArgumentError, "#{name} must not contain whitespace or control characters"
     end
 
     def cas_core(key, always_set, ttl = nil, req_options = nil)

data/lib/dalli/compressor.rb

@@ -13,8 +13,49 @@ module Dalli
       Zlib::Deflate.deflate(data)
     end
 
-    def self.decompress(data)
-      Zlib::Inflate.inflate(data)
+    # DEFLATE can't expand data by more than about 1032:1
+    MAX_EXPANSION = 1032
+    # Compressed input is fed to zlib this many bytes at a time. MRI's zlib
+    # yields the output in small chunks; one that returns it instead still
+    # produces at most MAX_EXPANSION times this per step.
+    INFLATE_SLICE_BYTES = 64 * 1024
+
+    # max_bytes caps the decompressed size; inflating stops and raises as soon
+    # as the output passes it, so a small compressed value can't expand into
+    # gigabytes of memory.
+    def self.decompress(data, max_bytes: nil)
+      return Zlib::Inflate.inflate(data) unless may_exceed?(data, max_bytes)
+
+      inflate_within_limit(Zlib::Inflate.new, data, max_bytes)
+    end
+
+    # False when even the largest possible expansion stays within max_bytes,
+    # so the value can be inflated in one step
+    def self.may_exceed?(data, max_bytes)
+      max_bytes ? data.bytesize * MAX_EXPANSION > max_bytes : false
+    end
+
+    def self.inflate_within_limit(inflater, data, max_bytes)
+      out = String.new(encoding: Encoding::BINARY)
+      0.step(data.bytesize - 1, INFLATE_SLICE_BYTES) do |pos|
+        inflate_slice(inflater, data.byteslice(pos, INFLATE_SLICE_BYTES), out, max_bytes)
+      end
+      append_within_limit(out, inflater.finish, max_bytes)
+    ensure
+      inflater.close
+    end
+
+    # Both yielded and returned output count toward the limit
+    def self.inflate_slice(inflater, slice, out, max_bytes)
+      rest = inflater.inflate(slice) { |chunk| append_within_limit(out, chunk, max_bytes) }
+      append_within_limit(out, rest, max_bytes)
+    end
+
+    def self.append_within_limit(out, chunk, max_bytes)
+      out << chunk if chunk
+      return out if out.bytesize <= max_bytes
+
+      raise Dalli::UnmarshalError, "Decompressed value exceeds #{max_bytes} bytes (decompressed_max_bytes)"
     end
   end
 
@@ -32,9 +73,11 @@ module Dalli
       io.string
     end
 
-    def self.decompress(data)
-      io = StringIO.new(data, 'rb')
-      Zlib::GzipReader.new(io).read
+    def self.decompress(data, max_bytes: nil)
+      return Zlib::GzipReader.new(StringIO.new(data, 'rb')).read unless Compressor.may_exceed?(data, max_bytes)
+
+      # MAX_WBITS + 16 makes zlib read the gzip format (and check its CRC)
+      Compressor.inflate_within_limit(Zlib::Inflate.new(Zlib::MAX_WBITS + 16), data, max_bytes)
     end
   end
 end

data/lib/dalli/json_serializer.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+require 'json'
+
+module Dalli
+  ##
+  # A serializer that stores values as JSON and never creates Ruby objects
+  # from what it reads.
+  #
+  # Passing the JSON module itself as the serializer (serializer: JSON) reads
+  # values back with JSON.load, which on json gem versions before 3.0 honors
+  # "json_class" and instantiates other classes when the json gem's additions
+  # are loaded. This serializer reads with JSON.parse, so a value written to
+  # memcached by someone else can only come back as plain hashes, arrays,
+  # strings, numbers, booleans and nil.
+  #
+  #   Dalli::Client.new(servers, serializer: Dalli::JSONSerializer)
+  ##
+  module JSONSerializer
+    def self.dump(value)
+      JSON.generate(value)
+    end
+
+    def self.load(data)
+      JSON.parse(data)
+    end
+  end
+end

data/lib/dalli/key_manager.rb

@@ -58,7 +58,7 @@ module Dalli
       raise ArgumentError, 'key cannot be blank' unless key&.length&.positive?
 
       key = key_with_namespace(key)
-      key.length > MAX_KEY_LENGTH ? truncated_key(key) : key
+      wire_length(key) > MAX_KEY_LENGTH ? truncated_key(key) : key
     end
 
     ##
@@ -122,7 +122,25 @@ module Dalli
     ##
     def truncated_key(key)
       digest = digest_class.hexdigest(key)
-      "#{key[0, prefix_length(digest)]}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
+      prefix = key[0, prefix_length(digest)]
+      truncated = "#{prefix}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
+      # A prefix with whitespace or non-ASCII characters sends the whole key
+      # base64-encoded, which can still be too long on the wire; shorten it.
+      while wire_length(truncated) > MAX_KEY_LENGTH
+        prefix = prefix[0...-1]
+        truncated = "#{prefix}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
+      end
+      truncated
+    end
+
+    # memcached's key limit applies to the key as sent: in bytes, and after
+    # base64 encoding when the key needs it. A key can be within the limit in
+    # characters and still be rejected, and memcached's CLIENT_ERROR for it
+    # must never reach a pipelined get.
+    def wire_length(key)
+      return key.bytesize unless Dalli::Protocol::Meta::KeyRegularizer.required?(key)
+
+      ((key.bytesize + 2) / 3) * 4
     end
 
     def prefix_length(digest)

data/lib/dalli/protocol/base.rb

@@ -53,6 +53,14 @@ module Dalli
         @raw_mode
       end
 
+      # True when values should come back as stored: the client is in raw mode,
+      # or this request passed raw: true. Retrieval paths use it to skip
+      # requesting flags, so a stored value is never deserialized or
+      # decompressed for a caller who asked for raw bytes.
+      def raw_request?(options)
+        raw_mode? || (options.is_a?(Hash) && options[:raw]) ? true : false
+      end
+
       # Chokepoint method for error handling and ensuring liveness
       def request(opkey, *args)
         verify_state(opkey)
@@ -148,7 +156,15 @@ module Dalli
         reconnect_on_pipeline_complete!
         values = nil
 
-        response_buffer.read
+        # Only the read is socket I/O. The caller's block runs below, outside
+        # this rescue, so an Errno or Timeout::Error it raises reaches the
+        # caller instead of being mistaken for a network failure (which closed
+        # the connection and retried the whole get_multi, yielding keys twice).
+        begin
+          response_buffer.read
+        rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError => e
+          @connection_manager.error_on_request!(e)
+        end
 
         status, cas, key, value = response_buffer.process_single_getk_response
         # status is not nil only if we have a full response to parse
@@ -174,8 +190,6 @@ module Dalli
         end
 
         values || {}
-      rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError => e
-        @connection_manager.error_on_request!(e)
       end
       # rubocop:enable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
 

data/lib/dalli/protocol/connection_manager.rb

@@ -36,7 +36,10 @@ module Dalli
         @quiet_responses_pending = false
         @sock = nil
         @pid = nil
+        @write_buffer = []
+        @write_buffer_bytes = 0
 
+        @fail_count = 0
         reset_down_info
       end
 
@@ -52,7 +55,8 @@ module Dalli
         Dalli.logger.debug { "Dalli::Server#connect #{name}" }
 
         @sock = memcached_socket
-        @sock.sync = false # Enable buffered I/O for better performance
+        # Writes are buffered in @write_buffer instead; see WRITE_BUFFER_FLUSH_BYTES
+        @sock.sync = true
         @pid = Process.pid
         @request_in_progress = false
         @quiet_responses_pending = false
@@ -85,6 +89,8 @@ module Dalli
       def down!
         close
         log_down_detected
+        # Once down_retry_delay passes, the server gets a full set of attempts
+        @fail_count = 0
 
         @error = $ERROR_INFO&.class&.name
         @msg ||= $ERROR_INFO&.message
@@ -114,7 +120,7 @@ module Dalli
         return unless @sock
 
         begin
-          @sock.close
+          close_socket
         rescue StandardError
           nil
         ensure
@@ -126,10 +132,18 @@ module Dalli
           @sock = nil
           @pid = nil
           @quiet_responses_pending = false
+          discard_write_buffer
           abort_request!
         end
       end
 
+      # A forked child shares the parent's connection, so it closes only its
+      # own file descriptor. Closing the TLS socket itself would send
+      # close_notify and end the parent's TLS session.
+      def close_socket
+        fork_detected? ? @sock.to_io.close : @sock.close
+      end
+
       def connected?
         !@sock.nil?
       end
@@ -163,6 +177,11 @@ module Dalli
         raise '[Dalli] No request in progress. This may be a bug in Dalli.' unless @request_in_progress
 
         @request_in_progress = false
+        # A completed request is what proves the server healthy again, so the
+        # failure count resets here rather than on reconnect: a server that
+        # accepts connections but never answers would otherwise reset it on
+        # every retry and never reach socket_max_failures.
+        @fail_count = 0
       end
 
       def abort_request!
@@ -170,6 +189,7 @@ module Dalli
       end
 
       def read_line
+        flush_write_buffer
         data = @sock.gets("\r\n")
         error_on_request!('EOF in read_line') if data.nil?
         data
@@ -179,14 +199,29 @@ module Dalli
 
       # JRuby doesn't support IO#timeout=, so use custom readfull implementation
       # CRuby 3.3+ has IO#timeout= which makes IO#read work with timeouts
+      # memcached can't store an item larger than 1 GiB (its -I maximum), so a
+      # reply claiming more (or a negative size) is malformed or hostile.
+      # IO#read allocates the full count up front, so check before reading.
+      MAX_READ_BYTES = (1024 * 1024 * 1024) + 2 # plus the value's trailing "\r\n"
+
+      def check_read_size!(count)
+        return if count.between?(0, MAX_READ_BYTES)
+
+        raise Dalli::DalliError, "Reply size #{count} from #{name} is out of range"
+      end
+
       if RUBY_ENGINE == 'jruby'
         def read(count)
+          check_read_size!(count)
+          flush_write_buffer
           @sock.readfull(count)
         rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError => e
           error_on_request!(e)
         end
       else
         def read(count)
+          check_read_size!(count)
+          flush_write_buffer
           read_bytes(count)
         rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError => e
           error_on_request!(e)
@@ -197,22 +232,34 @@ module Dalli
       # at the call site.
       alias read_exact read
 
+      # Requests are buffered here rather than in the socket's own IO buffer
+      # (the socket is sync). Ruby flushes an IO's write buffer when any process
+      # holding it closes or finalizes the IO, so after a fork a child could
+      # send bytes the parent had buffered but not sent yet: a quiet write would
+      # run twice, or the parent's replies would shift onto the wrong requests.
+      # A forked child discards this buffer instead. Like IO's own buffer, it
+      # is sent once it grows past this size, or when a reply is read.
+      WRITE_BUFFER_FLUSH_BYTES = 64 * 1024
+
       def write(bytes)
-        @sock.write(bytes)
+        @write_buffer << bytes
+        @write_buffer_bytes += bytes.bytesize
+        flush_write_buffer if @write_buffer_bytes >= WRITE_BUFFER_FLUSH_BYTES
+        bytes.bytesize
       rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, IOError => e
         error_on_request!(e)
       end
 
       def flush
-        @sock.flush
+        flush_write_buffer
       rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, IOError => e
         error_on_request!(e)
       end
 
       def flushed_write(bytes)
-        written = @sock.write(bytes)
-        @sock.flush
-        written
+        @write_buffer << bytes
+        flush_write_buffer
+        bytes.bytesize
       rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, IOError => e
         error_on_request!(e)
       end
@@ -220,6 +267,7 @@ module Dalli
       # Non-blocking read.  Here to support the operation
       # of the get_multi operation
       def read_available(...)
+        flush_write_buffer
         @sock.read_available(...)
       end
 
@@ -246,8 +294,9 @@ module Dalli
         raise Dalli::RetryableNetworkError, message
       end
 
+      # Called on connect. Deliberately leaves @fail_count alone; see
+      # finish_request!.
       def reset_down_info
-        @fail_count = 0
         @down_at = nil
         @last_down_at = nil
         @msg = nil
@@ -272,7 +321,8 @@ module Dalli
       def reconnect_on_fork
         message = 'Fork detected, re-connecting child process...'
         Dalli.logger.info { message }
-        # Close socket on a fork and reconnect immediately
+        # Drops anything the parent had buffered, closes the inherited socket
+        # without touching the parent's connection, and reconnects immediately
         close
         establish_connection
       end
@@ -302,6 +352,18 @@ module Dalli
 
       private
 
+      def flush_write_buffer
+        return if @write_buffer.empty?
+
+        @sock.write(*@write_buffer)
+        discard_write_buffer
+      end
+
+      def discard_write_buffer
+        @write_buffer.clear
+        @write_buffer_bytes = 0
+      end
+
       # Reads exactly `count` bytes. IO#read(count) on a blocking socket blocks
       # until it has `count` bytes, accumulating across TCP chunks internally,
       # and only hands back a shorter (or nil) buffer when the stream hits EOF.

data/lib/dalli/protocol/meta.rb

@@ -43,15 +43,15 @@ module Dalli
 
       def quiet_get_request(key, options = nil)
         # Skip bitflags in raw mode - saves 2 bytes per request and skips parsing
-        RequestFormatter.meta_get(key: key, return_cas: true, quiet: true, skip_flags: raw_mode?,
+        RequestFormatter.meta_get(key: key, return_cas: true, quiet: true, skip_flags: raw_request?(options),
                                   **routing_token_kwargs(options))
       end
 
       # Same requests as quiet_get_request for each key, built in one pass
       # and without the trailing noop.
       def quiet_get_requests(keys, options = nil, return_cas: true)
-        RequestFormatter.multi_meta_get(keys, skip_flags: raw_mode?, return_cas: return_cas, terminate: false,
-                                              **routing_token_kwargs(options))
+        RequestFormatter.multi_meta_get(keys, skip_flags: raw_request?(options), return_cas: return_cas,
+                                              terminate: false, **routing_token_kwargs(options))
       end
 
       def gat(key, ttl, options = nil)
@@ -72,7 +72,8 @@ module Dalli
       # TODO: This is confusing, as there's a cas command in memcached
       # and this isn't it.  Maybe rename?  Maybe eliminate?
       def cas(key, options = nil)
-        req = RequestFormatter.meta_get(key: key, value: true, return_cas: true, **routing_token_kwargs(options))
+        req = RequestFormatter.meta_get(key: key, value: true, return_cas: true, skip_flags: raw_request?(options),
+                                        **routing_token_kwargs(options))
         flushed_write(req)
         response_processor.meta_get_with_value_and_cas
       end
@@ -103,7 +104,7 @@ module Dalli
       #   - :last_access - seconds since last access (only if return_last_access: true)
       def meta_get(key, options = {})
         req = RequestFormatter.meta_get(
-          key: key, value: true, return_cas: true,
+          key: key, value: true, return_cas: true, skip_flags: raw_request?(options),
           vivify_ttl: options[:vivify_ttl], recache_ttl: options[:recache_ttl],
           return_hit_status: options[:return_hit_status],
           return_last_access: options[:return_last_access],
@@ -288,7 +289,7 @@ module Dalli
       # response parsing to minimize per-key overhead. Avoids the PipelinedGetter
       # machinery (IO.select, response buffering, server grouping).
       def read_multi_req(keys, options = nil)
-        is_raw = raw_mode?
+        is_raw = raw_request?(options)
         buffer = RequestFormatter.multi_meta_get(keys, skip_flags: is_raw, **routing_token_kwargs(options))
         flushed_write(buffer)
         buffer.clear
@@ -304,7 +305,7 @@ module Dalli
       # Shared by both the single-server fast path and PipelinedGetter's
       # per-server-group request, so this one change covers both routes.
       def read_multi_with_metadata_req(keys, options = nil)
-        is_raw = raw_mode?
+        is_raw = raw_request?(options)
         buffer = RequestFormatter.multi_meta_get(keys, skip_flags: is_raw, return_cas: true,
                                                        **routing_token_kwargs(options))
         flushed_write(buffer)

data/lib/dalli/protocol/request_formatter.rb

@@ -218,11 +218,11 @@ module Dalli
         # specifications to a proxy or router inbetween a client and a
         # memcached daemon."
         #
-        # Empty / nil tokens are treated as no-ops. CRLF and null bytes are
-        # rejected with ArgumentError to prevent the token from being used as a
-        # wire-protocol injection vector (e.g. "foo\r\nflush_all\r\n" would
-        # otherwise be parsed as a second command by memcached or any
-        # intermediate proxy/LB).
+        # Empty / nil tokens are treated as no-ops. Whitespace and control
+        # characters are rejected with ArgumentError so a token can't be used
+        # for wire-protocol injection: CRLF would add a second command (e.g.
+        # "foo\r\nflush_all\r\n"), and a space would add meta flags to this one
+        # (e.g. "foo T1" changing an item's TTL on a read).
         def routing_tokens(p_token: nil, l_token: nil)
           # Only an empty *String* is a no-op. Checking respond_to?(:empty?)
           # instead would also swallow p_token: [] / {} before the type check
@@ -279,15 +279,18 @@ module Dalli
 
         private
 
-        # Disallowed bytes: CR, LF, NUL. Any of these embedded in a routing
-        # token would let the caller inject a second wire-protocol command.
-        ROUTING_TOKEN_FORBIDDEN = /[\r\n\0]/
+        # Disallowed bytes: whitespace and control characters. CR, LF or NUL
+        # would let a token inject a second command; a space would let it add
+        # meta flags (T, N, I, q, ...) to the request it rides on.
+        ROUTING_TOKEN_FORBIDDEN = /[\x00-\x20\x7F]/
         private_constant :ROUTING_TOKEN_FORBIDDEN
 
         def validate_routing_token!(name, value)
           return if value.nil?
           raise ArgumentError, "#{name} must be a String, got #{value.class}" unless value.is_a?(String)
-          raise ArgumentError, "#{name} must not contain CRLF or null bytes" if value.match?(ROUTING_TOKEN_FORBIDDEN)
+          return unless value.match?(ROUTING_TOKEN_FORBIDDEN)
+
+          raise ArgumentError, "#{name} must not contain whitespace or control characters"
         end
 
         # Numeric flag values are written straight into the command line, so a

data/lib/dalli/protocol/response_processor.rb

@@ -24,6 +24,8 @@ module Dalli
         SERVER_ERROR = 'SERVER_ERROR'
 
         VA_PREFIX = 'VA '
+        # memcached can't store an item larger than 1 GiB (its -I maximum)
+        MAX_VALUE_BYTES = 1024 * 1024 * 1024
         HD_PREFIX = 'HD '
         FLAGS_TOKEN_PREFIX = ' f'
         CAS_TOKEN_PREFIX = ' c'
@@ -256,14 +258,16 @@ module Dalli
           end
           body_len = flag_int(size, 's')
 
-          # We have a complete response that has no body.
-          # This is either the response to the terminating
-          # noop or, if the status is not MN, an intermediate
-          # error response that needs to be discarded. A hit
-          # on an empty value (VA 0) still has a body -- just
-          # its terminator -- so it's parsed below as a value.
-          return [true, header_len] if body_len.zero? && tokens.first != VA
+          # A complete response with no body. Only the terminating noop's MN
+          # ends the pipeline. Any other bodyless reply (CLIENT_ERROR,
+          # SERVER_ERROR, ...) answers one key's request: report it with a
+          # false status so the caller skips it and keeps reading. Treating it
+          # as the end would leave the remaining replies on the connection for
+          # later commands to read as their own. A hit on an empty value (VA 0)
+          # still has a body -- just its terminator -- so it's parsed below.
+          return [tokens.first == MN, header_len] if body_len.zero? && tokens.first != VA
 
+          check_value_size!(body_len)
           resp_size = header_len + body_len + TERMINATOR.length
           # The header is in the buffer, but the body is not.  As we don't have
           # a complete response, don't advance the buffer
@@ -300,6 +304,7 @@ module Dalli
           end
           return nil if size.nil? || size.zero?
 
+          check_value_size!(size)
           header_len = term_idx - offset + TERMINATOR.length
           resp_size = header_len + size + TERMINATOR.length
           return [0] unless buf.bytesize >= offset + resp_size
@@ -424,6 +429,14 @@ module Dalli
           end
         end
 
+        # A pipelined reply claiming an impossible size would otherwise have the
+        # buffer wait for (and accumulate) that many bytes
+        def check_value_size!(size)
+          return if size.between?(0, MAX_VALUE_BYTES)
+
+          raise Dalli::DalliError, "Reply value size #{size} is out of range"
+        end
+
         def read_line
           @io_source.read_line&.chomp!(TERMINATOR)
         end

data/lib/dalli/protocol/value_compressor.rb

@@ -15,11 +15,17 @@ module Dalli
         compress: true,
         compressor: ::Dalli::Compressor,
         # min byte size to attempt compression
-        compression_min_size: 4 * 1024 # 4K
+        compression_min_size: 4 * 1024, # 4K
+        # max size a stored value may decompress to (nil: no limit). Guards
+        # against a small compressed value expanding into gigabytes on read.
+        decompressed_max_bytes: 128 * 1024 * 1024 # 128 MiB
       }.freeze
 
       OPTIONS = DEFAULTS.keys.freeze
 
+      KEYWORD_PARAMETER_TYPES = %i[key keyreq].freeze
+      private_constant :KEYWORD_PARAMETER_TYPES
+
       def initialize(client_options)
         @compression_options =
           DEFAULTS.merge(client_options.slice(*OPTIONS))
@@ -34,8 +40,16 @@ module Dalli
       end
 
       def retrieve(value, bitflags)
-        compressed = bitflags.anybits?(Flags::COMPRESSED)
-        compressed ? compressor.decompress(value) : value
+        return value unless bitflags.anybits?(Flags::COMPRESSED)
+
+        max_bytes = @compression_options[:decompressed_max_bytes]
+        # Custom compressors that only define decompress(data) keep working,
+        # without the limit
+        if max_bytes && compressor_accepts_max_bytes?
+          compressor.decompress(value, max_bytes: max_bytes)
+        else
+          compressor.decompress(value)
+        end
 
       # TODO: We likely want to move this rescue into the Dalli::Compressor / Dalli::GzipCompressor
       # itself, since not all compressors necessarily use Zlib.  For now keep it here, so the behavior
@@ -44,6 +58,19 @@ module Dalli
         raise UnmarshalError, "Unable to uncompress value: #{$ERROR_INFO.message}"
       end
 
+      def compressor_accepts_max_bytes?
+        return @compressor_accepts_max_bytes if defined?(@compressor_accepts_max_bytes)
+
+        @compressor_accepts_max_bytes =
+          begin
+            compressor.method(:decompress).parameters.any? do |type, name|
+              name == :max_bytes && KEYWORD_PARAMETER_TYPES.include?(type)
+            end
+          rescue NameError # an object without a reflectable decompress method
+            false
+          end
+      end
+
       def compress_by_default?
         @compression_options[:compress]
       end

data/lib/dalli/protocol/value_serializer.rb

@@ -100,7 +100,7 @@ module Dalli
         Dalli.logger.warn 'SECURITY WARNING: Dalli is using Marshal for serialization. ' \
                           'Marshal can execute arbitrary code during deserialization. ' \
                           'If your memcached server could be compromised, consider using ' \
-                          'a safer serializer like JSON: Dalli::Client.new(servers, serializer: JSON)'
+                          'a safer serializer: Dalli::Client.new(servers, serializer: Dalli::JSONSerializer)'
         @@marshal_warning_logged = true # rubocop:disable Style/ClassVars
       end
     end

data/lib/dalli/version.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module Dalli
-  VERSION = '5.2.0'
+  VERSION = '5.2.1'
 
   MIN_SUPPORTED_MEMCACHED_VERSION = '1.6.27'
 end

data/lib/dalli.rb

@@ -62,6 +62,7 @@ require_relative 'dalli/version'
 require_relative 'dalli/instrumentation'
 
 require_relative 'dalli/flags'
+require_relative 'dalli/json_serializer'
 require_relative 'dalli/compressor'
 require_relative 'dalli/cache_result'
 require_relative 'dalli/client'

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: dalli
 version: !ruby/object:Gem::Version
-  version: 5.2.0
+  version: 5.2.1
 platform: ruby
 authors:
 - Peter M. Goldstein
@@ -43,6 +43,7 @@ files:
 - lib/dalli/compressor.rb
 - lib/dalli/flags.rb
 - lib/dalli/instrumentation.rb
+- lib/dalli/json_serializer.rb
 - lib/dalli/key_manager.rb
 - lib/dalli/options.rb
 - lib/dalli/pipelined_deleter.rb